Evaluation Report Health Development Africa (HDA) Circles of Support (COS) Project Prepared by Umhlaba Development Services Evaluation Team Danielle Dass Marinda Weideman (Dr) Cyril Turton July 2012 Health Development Africa Circles of Support (COS) OVC Project Evaluation Report Commissioned by Pact South Africa under Associate Award No. 674-A-00-08-00001 Evaluation Team Director Danielle Dass Associates Marinda Weideman (Dr) Cyril Turton Umhlaba Development Services 43 Birchwood Court, Montrose Street, Vorna Valley Midrand 1686 South Africa Cell: +27 82 411 1193 Phone/Fax: +27 86 530 5732 Email Address: dannyd@dbn.umhlaba.com Disclaimer: This publication was made possible through support provided by the Office of HIV/AIDS, Bureau for Health, U.S. Agency for International Development, under the terms of USAID South Africa Associate Award No. 674-A-00-08-00001-00. The opinions expressed herein are those of the author(s) and do not necessarily reflect the views of Pact or the U.S. Agency for International Development. Page | iii TABLE OF CONTENTS SECTION 1 : EXECUTIVE SUMMARY ............................................................................ 1 SECTION 2 : INTRODUCTION ...................................................................................... 6 2.1 PROJECT BACKGROUND, RATIONALE AND FORM .............................................................................. 6 2.1.1 Project Background ..................................................................................................... 6 2.1.2 Situating the COS Project in the Lejweleputswa District Municipality .......................... 7 2.1.3 COS Project Rationale ................................................................................................. 8 2.1.4 Project Description .................................................................................................... 11 2.1.5 Project Structure ....................................................................................................... 14 SECTION 3 : EVALUATION PURPOSE AND DESIGN ..................................................... 15 3.1 OVERVIEW OF THE EVALUATION METHODOLOGY ........................................................................... 15 3.2 SAMPLING AND FIELDWORK EXPERIENCES .................................................................................... 16 3.2.1 Sampling ................................................................................................................... 16 3.2.2 Pilot .......................................................................................................................... 17 3.2.3 Fieldworker Training ................................................................................................. 17 3.2.4 Respondent Co-ordination ........................................................................................ 17 SECTION 4 : KEY FINDINGS ....................................................................................... 19 4.1 KEY EVALUATION QUESTION (KEQ) 1 ......................................................................................... 19 4.1.1 KEQ 1 Strengths and Opportunities ........................................................................... 19 4.1.1.1 Community outreach was a catalyst for change ................................................................. 19 4.1.1.2 Strong relationships with government were established .................................................... 20 4.1.1.3 Community based stakeholder networks created .............................................................. 22 4.1.1.4 Greater focus towards resource optimization .................................................................... 22 4.1.2 KEQ 1 Weaknesses and Threats................................................................................. 25 4.1.2.1 Incorrect understanding and expectations of the COS project ........................................... 25 4.1.2.2 Volunteer and stipend model not ideal for indigent communities ..................................... 26 4.1.2.3 Community perception that COS does not respond to actual needs .................................. 27 4.1.2.4 Child participation in project life cycle ................................................................................ 27 4.1.3 Recommendations for Key Evaluation Question 1 ..................................................... 28 4.2 KEY EVALUATION QUESTION (KEQ) 2 ......................................................................................... 28 4.2.1 KEQ 2 Strengths and Opportunities ........................................................................... 29 4.2.1.1 Training content and recipients are relevant to the project ............................................... 29 4.2.1.2 Monitoring and Evaluation Skills and Training improved project delivery .......................... 31 4.2.1.3 Leveraging quality training from external providers to minimize cost of training .............. 31 4.2.1.4 Training builds skills, instills confidence and creates employability opportunities in other sectors 31 4.2.2 KEQ 2 Weaknesses and Threats................................................................................. 33 4.2.2.1 Training spend is not enough for consistent training of new recruits ................................. 33 4.2.2.2 Newly recruited SBFs lack basic workplace skills ................................................................ 33 4.2.2.3 Most training offered was not SAQA accredited ................................................................. 34 4.2.2.4 Data Quality Issues Identified in RDQA process: ................................................................. 34 4.2.2.5 Inadequate Monitoring of Activities ................................................................................... 35 4.2.3 Recommendations for Key Evaluation Question 2 ..................................................... 36 iv | P a g e 4.3 KEY EVALUATION QUESTION (KEQ) 3 ......................................................................................... 37 4.3.1 KEQ 3 Strengths and Opportunities ........................................................................... 37 4.3.1.1 Accessing community based circles of support ................................................................... 37 4.3.1.2 Improved psychosocial wellbeing of OVCs and their families ............................................. 38 4.3.1.3 Improved Nutrition and health ........................................................................................... 38 4.3.1.4 Increased access to government grants .............................................................................. 38 4.3.1.5 Improved Education Support .............................................................................................. 39 4.3.2 KEQ 3 Weaknesses and Threats................................................................................. 39 4.3.3 Recommendation for Key Evaluation Question 3 ...................................................... 40 SECTION 5 : CONCLUSION ........................................................................................ 41 SECTION 6 : KEY LESSONS LEARNT AND OVERALL RECOMMENDATIONS .................... 44 6.1 COMMUNITY WIDE INTERVENTIONS FOR OVCS WORK ..................................................................... 44 6.2 THE VOLUNTEER MODEL IN INDIGENT COMMUNITIES CAUSES PROJECT INSTABILITY ................................ 44 6.3 COS CAN BE SCALED DOWN TO FOCUS ON SCHOOLS WITH GREATEST NEEDS .......................................... 45 SECTION 7 : ACKNOWLEDGEMENTS.......................................................................... 46 SECTION 8 : APPENDICES ......................................................................................... 47 Appendix 1: Basic Situational Analysis of target schools and reduction criteria used............................. 47 Appendix 2: Roles and Responsibilities of the COS team ........................................................................ 48 Appendix 3: Transcripts of Interviews ..................................................................................................... 49 Appendix 4: Sampling Strategy ............................................................................................................... 50 Appendix 5A: Absenteeism Rates: Parents Questionnaire ................................................................... 55 Appendix 5B: Child Afraid to Go to School .......................................................................................... 55 Appendix 6: Actual Services Provided to 16 sampled schools ................................................................ 56 Appendix 7: Food Security at Home ........................................................................................................ 57 SECTION 9 : ATTACHMENTS ...................................................................................... 58 List of In-depth Interviewees .................................................................................................................... 58 Number of Participating Community Stakeholders per School ................................................................ 59 Types of Training Offered to COS ............................................................................................................. 60 Achievement of Output Indicators in Schools under study ...................................................................... 62 CASE STUDY ............................................................................................................................ 63 v | P a g e List of Graphs Graph 1: HDA baseline data on socio economic profile of the Lejweleputswa District Graph 2: Geographical location of schools visited Graph 3: COS assistance in accessing grants List of Tables Table 1: Geographical Location and School Type Table 2: Baseline Data on OVC age groups to be targeted for assistance Table 3: Number of Respondents Interviewed Table 4: Service Provision trends for 4 years Table 5: Training Conducted over Project Cycle period Table 6: Systemic Problems revealed through RDQA Process List of Diagrams Diagram 1: Theory of Change Diagram 2: COS Organogram Diagram 3: Stakeholder Involvement before COS Diagram 4: Stakeholder involvement after COS at Phomolong, Oziel Selele and Senzile Schools vi | P a g e Abbreviations ABET Adult Basic Education and Training CCF Community Care Forums COS Circles of Support CCW Community Care Worker CDW Community Development Worker DoE Department of Education DoHA Department of Home Affairs DoJ Department of Justice DoRA Department of Rural Affairs and Agriculture DoSD Department of Social Development FET Further Education and Training HDA Health and Development Africa LTF Learner Training Facilitator OVCs Orphans and Vulnerable Children PACTSA PACT South Africa PEPFAR Presidents Emergency Plan For Aids Relief RDQA Routine Data Quality Assessment SASA South African Schools Act SASSA South African Social Security Agency SAPS South African Police Services SBF School Based Facilitators SBST School Based Support Team VCT Voluntary Counselling and Testing 1 | P a g e Section 1 : EXECUTIVE SUMMARY Rationale for this OVC intervention The socio economic decline in the Lejweleputswa District Municipality in the Free State Province created an environment with unacceptably high unemployment rates (40%) and families living below the national poverty indices (56.1%)1. To further compound this socio-economic distress, the district municipality urgently re-launched its Lejweleputswa District AIDS Council in October 2011 in response to the provincial finding that ‘Lejweleputswa has the highest HIV and AIDS prevalence rates in the Free State2. HDA conducted district level baseline research in 2009 and found that children in this district were in chronic distress and that support services were needed as an emergency stop gap. Circles of Support Theory and Approach The Circles of Support (COS) project is a community based response driven project that uses low￾cost interventions, and focuses primarily on systems strengthening at the local level. Schools are a central point because most indigent communities will have schools even though other services may not be available. COS seeks to lessen the impact of HIV/AIDS on orphans and vulnerable children (OVCs) by implementing practical methodologies to establish community/neighbourhood circles of support to provide assistance. The intention is to maximize skills and resources from existing stakeholders such as government, Non-Profit Organisations (NPOs) and private sectors and to train these local support groups to continue the service beyond the project life cycle. Evaluation Approach The evaluation design included a retrospective “before” and “after” analysis of key characteristics among targeted beneficiaries of the program. A mixed methodology was used including qualitative (i.e. desktop research, in-depth interviews and focus groups) and quantitative (survey) methods were used to gather data from a wide range of stakeholders. The combination of qualitative and quantitative data was considered highly suitable for the evaluation as it enables the assessment of trends, perceptions and behavior while at the same time enabling the quantification of program results based on a survey of beneficiaries. Evaluation Phases included a Literature Review from which the development of 4 questionnaires ensued. This was followed by a Pilot test of the questionnaires at Ikemetsiseng School which alerted researchers to the need for learner and caregiver questionnaire translation into Sesotho and multilingual research teams. Prior to the fieldwork phase, the entire team and two COS staff were trained in fieldwork research techniques. A team of 11 fieldworkers spent three weeks gathering data from the 16 sampled schools and surrounding communities. Quantitative data was captured on the Statistical Package for Social Sciences (SPSS) database and this can be used as updated baseline data for future use by this project. 1 Lejweleputswa District Municipality Final Reviewed IDP for 2011/2012, data quoted from Census 2007 2 Volksblad, 6 October 2011. 2 | P a g e Key Evaluation Questions 1. What are the key characteristics of the Circles of Support (COS) school-based model at its best, and what factors promote or hinder success of the model in responding to and addressing needs of children? 2. How effective was the project in enhancing the capacity of schools based facilitators (SBFs) and SBSTs in identifying and addressing the needs of vulnerable children? 3. In which areas of child wellbeing did the project make the biggest change? Key findings The COS project has excelled in most of its objectives and has implemented the theory of change so well that in some schools, communities and other stakeholders can provide some services independently of the COS team. The project has trained unskilled community members such as school based facilitators (SBFs) as well as the management and teachers in targeted schools, resulting in strengthened school-based support teams which have initiated a series of in-school support services to OVCs. Key services provided by the SBSTs include homework support classes, memory work, HIV prevention education and programmes for learners with special needs. Most importantly, the COS project has enabled service delivery to be fast tracked from various government departments so that indigent families could access social grants and other services required to improve quality of life. The uncertainty about the COS project’s future beyond PEPFAR’s funding cycle ending in September 2012, is not well received by staff and the community and the absence of a clearly documented exit strategy further compounds the uncertainty felt by stakeholders interviewed. Answering the Key Evaluation Questions • The COS model at its very best was seen in three schools with the other 13 schools showing mixed degrees of success. At the three schools, the networks forged with government, NPOs and private sector stakeholders is so effective that the community feels confident enough to continue this project even after COS exits. SBF’s were well capacitated to consistently collect good quality data about OVCs and the actual needs at household level. Their strong networking skills expanded the community consultations to such an extent that community stakeholders now actively seek out OVCs to refer to COS for assistance. A best practice case study in Phomolong shows how one efficient SBF was able to facilitate a circle of support to 300 pregnant teens, thereby improving the quality of life to the mothers and securing the health of a new generation within this community. • Capacity development had a mixed review as there was clear evidence of the commitment to training and development and evidence that appropriate and relevant training was conducted. This effective training however did result in the high turnover of SBFs in particular because high performing staff were often attracted to and offered better paying full time jobs by schools and other NGO’s. Three key weaknesses which could affect long term sustainability were however identified only two training courses were SAQA accredited, including the NACCW Child and Youth Care worker training and the Thomogelo Psycho Social Care course. However, conclusions about the success rate among trainee and actual skills transfer cannot be made at this stage and the NACCW training is yet to be completed, and 3 | P a g e trainees who attended the Thomogelo training are yet to receive competency certificates from the HWSETA. Community members were not trained in generic project management skills. Although community members are able to do OVC referrals they are not sufficiently skilled to take over this project and manage it to the current COS standards should COS choose to exit in September 2012. Long term sustainability is likely to be compromised as a result. a)High staff turnover, especially high performing SBF’s who were able to find full time salaried employment (as a direct result of effective COS training) has created critical skills gaps, loss of institutional memory, inconsistencies in quality of services. Staff vacancies at project and leadership level have not helped this situation either . The biggest lifestyle changes as reported by caregivers interviewed is that COS had excelled in its core focus to set up and grow community networks and multi-sector stakeholder involvement. SBF’s are generally reported to be most effective in facilitating access to child protection services. Many respondents were able to provide specific examples of assistance provided by COS to abused children particularly in enabling access to government services such as the police and statutory work by government social workers. Additionally, awareness of the rights of children has also increased within communities as a whole. This has resulted in caregivers being more knowledgeable about aid services available within the community and having the confidence to approach them. Records of changes were collected through a pre and post intervention characteristic analysis, where questionnaires asked Before COS and After COS questions. Furthermore, Focus Group respondents provided unprompted examples of changes observed in their lifestyles and these are recorded in great detail in the qualitative report attached to this report. From these accounts, we conclude that the greatest achievement of the COS project was to integrate OVCs and their families into the community and set up strong networks for them to: • Access services such as grants. • Access child protection services. 93% of program beneficiaries were provided with child protection support • Address food security -food gardens were established in distressed homes and at schools enabling families to eat more frequent and better quality meals per day, 74% of OVC’s requiring Clinical Nutrition Support have been assisted and this is in due to COS facilitating greater efficiency from the Department of Health (DoH) • Retain OVC in school by ensuring that children were exempted from school fees and provided the necessary uniforms and stationery. These efforts contributed to improved school performance among program beneficiaries • Access free medical care through exemptions at clinics as well as through pro bono services offered by retired medical professionals The evaluation findings revealed however that the program had been less successful in enhancing access to household economic strengthening , with only 22% OVCs assisted and psychological care, with only 36% OVCs assisted. 4 | P a g e Overall Conclusions  The COS programme has put in place valuable safety nets for OVC’s and these activities must continue. As a result of its exemplary achievements in this community, HDA must make an immediate decision about its plan for COS post September 2012 and then put in place an exit strategy for the smooth transition. The four overall key learning from this evaluation are: 1. Community wide interventions for OVCs work - The COS model and approach of assisting OVCs and their families, using schools as a base, is effective and it reflects best practice observed in the international OVC arena. Community events which included non COS schools, government, the NPO sector and the private sector has embedded the COS project within the community very well and this approach is a best practice for replication for similar projects operating throughout the country. 2. COS can be scaled down to focus on schools with greatest OVC needs - COS has succeeded in mobilising services offered by NGOs and government and in the three best practice schools under study (and others that COS should assess), COS should play a monitoring role rather than an implementing agency. These duties can be handed over to schools, community organisations and volunteers who – as a result of the COS project, now have the basic skills to continue providing support to their OVCs. The organisational management capacity of these schools and volunteers is however still limited for COS to exit immediately but through additional management training, COS will be able to reduce its management role and physical presence in better performing schools, and redirect their resources to poor performing schools. 3. The Volunteer Model in Indigent Communities does not work - The dependence on volunteers from indigent communities to ensure a daily operational presence at schools is affected by high staff turnover which leads to skills and capacity gaps, loss of institutional memory and disruptions in service levels. Indigent communities also have a low skills base and those volunteers who have little or no workplace skills constitute the recruitment pool from which COS had to source replacement SBFs – the investment in training new recruits to required standards is high and negatively affects the budget and the achievement of goals within a short time frame. 4. Strategic Planning with communities and target setting can improve. The indicators set by COS at project inception did not include actual numerical targets - the project aimed to achieve increases in all 10 Indicators, without stipulating numerical percentages or numbers. Furthermore, the initial project plan did not include input from communities and OVC beneficiaries. Community Management Forums who are ready to take over the COS project at their schools should be assisted by COS with developing specific numerical targets - for example “20% increase in access to grants per annum for X municipality” and OVCs, their families and community stakeholders must give input to future project plans and preferred approaches. 5 | P a g e For each of these key learnings, the recommendations are: Recommendations for key learning 1:  COS Welkom staff and the schools coordinators should assess each of their schools according to level of independence from COS, thereby determining the assistance they still require from COS.  Full time coordinators should be employed for each of the four municipalities in this district to manage the service levels of the schools in those areas. Having a municipal rather than a district focus will enable rapid responses to poor performing schools.  The M & E Officer should be based in Welkom and conduct SBF and volunteer training once a month in each of the four municipal areas. Recommendations for key learning 2:  COS achieved building community networks in all schools therefore the project should assess all community management forums for readiness to manage the COS model themselves.  The second phase of the COS programme should focus on training communities in organisational management so that the future role of COS would be to focus on training, monitoring and supervision only. In this way, COS can focus on assisting with implementation in only the poorest performing schools. Recommendations for key learning 3:  COS should allocate a larger budget to employ full time co-ordinators for each of the four local municipalities in this district. This will ensure that highly skilled SBF’s are retained at least at municipal level.  COS should also explore creative incentives to retain effective staff either through financing specialised training (career development); convening annual recognition awards ceremonies; providing once-off cash bonuses to high performers or brokering a full time employment arrangement with schools which the DOE and the private sector could jointly fund. Recommendations for Key learning 4:  Community interventions remain relevant only insofar as they respond to needs identified by beneficiaries therefore project targets must be customized for each school and their local challenges. Child and caregiver participation from project design phase to project conclusion phase must become part of the overall project culture. This can be achieved through a wider and all inclusive stakeholder strategic planning process.  Specific targets and indicators should be set by all stakeholders using this report as a baseline data source.  Refresher training and sharing forums should be held monthly for all SBFs and community volunteers. This will ensure that service levels remain consistent at all schools and that community volunteers can manage the project with limited involvement from COS. 6 | P a g e Section 2 : INTRODUCTION The Circles of Support (COS) project in the Lejweleputswa District Municipality of the Free State Province has been implemented by HDA for three years and is slated to end on 30 September 2012. PACTSA, which acts as the projects’ grant administrator on behalf of USAID, PEPFAR, has consequently commissioned a Summative Evaluation to assess whether COS was able to bring about any changes to the lives of targeted Orphans and Vulnerable Children (OVCs), their families and the community in general. This evaluation therefore, has as its main aim, to determine the extent to which the COS project was able effectively address the needs of the vulnerable children in targeted communities. Based on the ASSA model (2003) it’s estimated that there are nearly 70 000 OVCs in the Free State District of Lejweleputswa.3 2.1 Project Background, Rationale and Form 2.1.1 Project Background The COS project falls within the business ambit of Health and Development Africa (HDA) which is a leading South African health consulting company, with extensive experience in HIV/AIDS development projects in sub Saharan Africa. Between 2000 and 2003 HDA staff led and participated in a number of HIV Impact Studies for Ministries of Education across Southern Africa. As a result HDA developed the Circles of Support (COS). The model was designed as a community based response driven project that uses low-cost interventions, and focuses primarily on systems strengthening at community and school levels. COS utilizes schools as an entry point, precisely because schools are central points and are located in all communities. The intervention seeks to lessen the impact of HIV/AIDS by implementing practical methodologies to support vulnerable children and through establishing community/neighbourhood circles of support resident on school premises to protect OVC education and wellbeing. To this end, concerned adults, community members, teachers, local leaders, other professionals and older children volunteer to provide assistance. These groups allow for coordination and sharing of expertise, resources and capacity, and can tap into external resources provided by government, NGOs and private sector sources. The COS project has continued to encourage the establishment of lateral and horizontal resource networks that include relevant government services that are able to respond to the needs raised by schools and communities. COS provides training and support to these local organisations, promoting their capacity to understand children vulnerabilities and to plan for appropriate action to mitigate the problems. In this way, COS seeks to secure project sustainability. Initially the COS programme selected 90 schools for support but during the penultimate year of the project cycle, this number was reduced to 60 of the most needy schools – Appendix 1 tabulates a basic situational analysis of schools and provides reasons for the reduction in target numbers. 3 ASSA 2003 Model – cited in COS Child Participation Report, January 2011 and cited in ANC data for this district in 2009 7 | P a g e The table below indicates that COS directs its resources and services mainly towards primary schools (68%). Due to accessibility and proximity constraints, rural schools targeted by the COS program, accounts for only 17% of the total schools served. Table 1: Geographical Location and School Types Geographical Location Primary School Secondary School Combined School Total Rural 6 1 3 10 Semi Urban 21 12 1 34 Urban 14 2 0 16 TOTAL 41 15 4 60 Based on HDA’s extensive contextual research, the project defines vulnerability as: …any school-going child whose home/schooling environment and livelihoods are threatened by chronically ill parents, households that have recently experienced a death of a parent, households headed by a grandparent or a child and children who are marginalised, stigmatised or discriminated against as a result of their HIV status. This definition was important in conceptualising the aim and structure of COS project which could now clearly identify vulnerable children directly affected by AIDS-related morbidity and/or mortality and immediately put in place systems to continue their schooling. As articulated by an HDA respondent, “our research has shown that any disruption in a child’s home life has an immediate disruption to their schooling, and this often has a devastating long term effect on the ability of the child to reach his or her full potential.” To this end, the project emphasised the strengthening of community resources and networks to meet the specific needs of their OVCs, beginning with capacity building and network linkages with the schools where OVCs make up a sizeable percentage. 2.1.2 Situating the COS Project in the Lejweleputswa District Municipality The Lejweleputswa District Municipality was selected as the primary site for the COS project because of its severe socio-economic profile caused by the rapid decline in mining activities. Large scale retrenchments contributed towards the unemployment (40%) and poverty rates (56.1%) - far exceeding the national averages of 27% and 40% respectively.4 In addition to this socio-economic decline the district municipality urgently re-launched its Lejweleputswa District AIDS Council in October 2011 in response to the provincial finding that ‘Lejweleputswa has the 4 Free State Province IDP 2010, pp.10-11 8 | P a g e highest HIV and AIDS prevalence rates in the Free State.’5 . HDA conducted district level baseline research in 2009 and found that children in this district were in chronic distress and that support services needed as an immediate stop gap, were vast and complex:6 The graph below indicates that OVCs were in immediate need of urgent and multi-stakeholder intervention. Graph 1: HDA 2009 Baseline Study indicating socio economic issues in Lejweleputswa District A number of the community descriptions cited in this baseline report refer to the demise of the social fabric where alcohol and substance abuse is increasing amongst adults, youth and even children. Another description highlighted the problem of gangs that disrupt children’s ability to access the school and a few others make reference to “thugs” and increasing crime rates. Teenage pregnancy and a high birth rate were also noted as features of the community in distress. This baseline study also asked schools to indicate which 3 areas of intervention would immediately make the greatest difference to the lives of their OVCs and the top 3 responses were “government grants (66%), “food security/ nutritional support (61%), and “counseling and psychosocial support” (57%). 7 2.1.3 COS Project Rationale Drawing on international best practices highlighted in the Literature Review, the following constructs were adopted by HDA to customise a model to militate against the impact of the HIV and AIDS epidemic on children’s social, psychological, economic, cultural, educational and health status: 5 Volksblad, 6 October 2011. 6 HDA Baseline Report, May 2009, pages 24 to 26 7 Ibid pp 25 9 | P a g e a) Interventions should be child-centered The most compelling effect of HIV and AIDS is the one it continues to have on children made vulnerable by the prolonged illness and/or death of one or both of their parents from AIDS￾related illnesses and [in many cases] their own HIV-positive status; and the many negative circumstances and consequences flowing from these situations. It is now commonly accepted that children tend to bear the brunt of the epidemic’s impact and are made extremely vulnerable when confronted with poverty and livelihood insecurities in the context of HIV and AIDS. They also become more prone to victimization, marginalization, exploitation and abuse, including sexual abuse, which obviously has further implications of HIV-infection and even greater levels of vulnerability.8 HDA’s baseline research shows that the majority of OVCs who need support from this project range between the ages of 5 and 18 years. Table 2: Baseline Data on OVC age groups to be targeted for assistance OVC members of the household Age 0 to <2 Age 2 to <5 Age 5 to <12 Age 12 to <18 Total Male 2 1 532 648 1,183 Female 5 5 676 819 1,505 Total 7 6 1,208 1,467 2,688 Source: COS Annual Report 2011 Actual needs of OVCs in this district as presented to the South African AIDS Conference (SAAIDS)9 indicated that the demand for services per COS thematic area was: Needs Total OVCs identified for assistance Clinical Nutritional Support 573 Household Economic Strengthening 2,368 Child Protection 2,777 General Health Care 5,116 Psycho Social Care 6,282 Education Support 7,047 In addition to this baseline information, the Handover Report10 records that this district has more female children (54%) compared to male children (46%); more children are orphaned 8,105 (70%) 8 De Bruin Cardoso, I., 2010, National plan of Action for Orphans and Vulnerable Children – Establishing, reviewing and implementing National Plans of Action for OVC in Southern and Eastern Africa: Lessons learnt and challenges. Save the Children UK. Report on a workshop, held in Pretoria in April 2010. p.4 9 Presented to SA AIDS Conference by Ms Bertha Magoge on July 2011, Figure 3, page 2 of the presentation 10 Data obtained from attachment to the Handover Report submitted December 2011 by the outgoing Project Manager 10 | P a g e than those children with both parents alive 3,452 (30%); the majority of these children reside in the rural areas, and they are mostly taken care of by their grandparents. Due to high illiteracy rates of aged caregivers, support with homework and general health care has become overwhelming for these caregivers. b) Linking local OVC support services to global Frameworks The COS project replicated international best practices related to OVC support in that it focused on a more holistic ‘OVC sensitive programming’, some of which includes coordinating services for social protection and responding to the needs of all vulnerable children. The Framework for the Protection, Care and Support of Orphans and Vulnerable Children living in a World of HIV and AIDS adopted in 2004, has become the standard set of policy and strategy guidelines on the provision of OVC support internationally.11 This Framework contains five key strategies which COS has successfully included in its own project approach, starting with the need to strengthen family and community-based responses and [only] thereafter progressing to governments’ responsibilities with regard to the needs of OVC: • Strengthen the capacity of families to protect and care for OVC by prolonging the lives of parents and providing economic, psycho-social and other support • Mobilise and support community-based responses • Ensure access for OVC to essential services, including education, healthcare, birth registration and others • Ensure that governments protect the most vulnerable children through improved policy and legislation and by channeling resources to families and communities • Raise awareness at all levels [of society] through advocacy and social mobilisation to create a supportive environment for children and families affected by HIV and AIDS’12 c) Implementing and sustaining an unprecedented Project response to HIV and AIDS. The sheer scale of this pandemic and its complexity is unprecedented and therefore requires an unprecedented response that has to be sustained over at least the next two to three generations.13 It is therefore essential that comprehensive and multi-sectoral policies directed at the care of OVCs is kept in the national and international spotlight and that their translation into well￾designed, coordinated and properly resourced programmes be closely monitored by institutions appointed to do this. Furthermore, it is also imperative that such policies and programmes be fundamentally informed by the expressed needs, problems and suggestions of vulnerable children, their caretakers and community structures. Throughout the literature surveyed (including data from the project under review), there are repeated references to the lack of consultation with/ and participation of children and community structures in OVC project.14 This problem persists, despite the fact that the basic departure point of most international OVC policy 11 For more detail, see the Foreword in the Framework document, ibid. 12 Ibid., p.14. For an in-depth and detailed discussion of the strategies, see pp. 14-27. 13 The Framework for the Protection, Care and Support of Orphans and Vulnerable Children living in World or HIV and AIDS, 2004, UNICEF and UNAIDS, p.10. (generally referred to as “The Framework”) 14Loening-Voysey and Wilson: Approaches to Caring for OVC’s, for UNICEF, February 2011, pp 22. 11 | P a g e documents is to start with what already exists in a community in terms of taking care of OVC and to then strengthen, supplement and scale-up such processes and practices.15 The COS project design therefore took its lead from these international best practices and was formed around the following results framework hypotheses: • If educators and community members are sufficiently trained to identify OVCs, then OVCs can be brought to the attention of local service providers to access basic livelihood assistance. • If it is a school-based project, this is a valuable catchment area to closely monitor levels of vulnerability and to prevent any interruptions in learning whilst OVCs receive other basic needs support. 2.1.4 Project Description In order to respond to the specific experiences in this district and as reported by DoE respondents who had already put in place a schools-based intervention COS was developed with broader project scope and founded on a clear Theory of Change16,. This theory and approach is premised on a detailed roadmap (diagram and explanation follows below) setting out clearly what change is required and how it will be achieved. It explains how change happens, and what is required to succeed, showing the causal relationship between specific activities and the predicted outcomes for each activity. Diagram 1: Theory of Change 15 PEPFAR Framework on HIV and AIDS 16 This Theory of Change was presented to the PACTSA HDA Implementation meeting on 20 February 2012 Level 5 Indicators Level 4 Penultimate Outcomes Level 4 Penultimate Outcomes Level 3 Secondary Interventions Level 2 Intermediate Outcomes Level 2 Intermediate Outcomes Level 1 Primary Interventions OVERALL GOAL 12 | P a g e Level 1: Primary Interventions Each school is allocated a Schools Based Facilitator (SBF) to identify vulnerable children. The SBF is then linked with the Schools Based Support Team (SBST) to further support and refer these learners. Then home visits are organized to establish the circumstances under which the child is living, usually conducted by SBFs and Community Care Workers (CCWs). These visits are critical to ensure parental or caregiver involvement. Home visits also provide an opportunity to identify other siblings who may require support. On matters requiring special attention, such as learners with specific learning disabilities, learners are referred to the Learner Support Facilitator’s (LSFs) from the Inclusive Education Unit of the DoE, for further intervention. Typical primary interventions include homework support, HIV and AIDS prevention education, birth certificate and ID registration and memory work. Level 2: Intermediate Outcomes Once the household needs assessment is complete the COS team and the SBST arrange the necessary service(s) to be provided to the child. The SBST works with class teachers, neighbours, community groups and organizations to make sure that these children get continued holistic support from a wide range of service providers. In this case, the anticipated outcomes of the project were to: 1. Improve and strengthen school based support to OVC’s 2. Strengthen community support and networks to improve support to OVCs in schools 3. Build and strengthen partnerships with local, district, provincial and national stakeholders for learner care and support in project schools 4. Advocate and lobby multi-stakeholder involvement and commitment to supporting schools as centres of care and support for vulnerable children and vulnerable communities Level 3: Secondary Interventions The project identifies secondary interventions necessary to assist each OVC with household specific needs. These include access to social grants, provision of psychosocial support, addressing material needs and access to health care. In this phase, extreme cases are identified where children require alternative housing or food (food gardens and parcels from local municipalities), or protection from abuse. In this case the COS team and SBST will work closely with the Department of Social Development (DoSD), the South African Police (SAPS), Department of Health, the Department of Agriculture and local magistrates to ensure the safety of that child within the home. Level 4: Penultimate Outcomes Through the use of strategies and tools, OVCs are continually monitored through utilizing household level tools to collect data at regular intervals. This data is collated on a central database and the results used to ascertain whether health, education, psycho-social support and household economic strengthening are indeed contributing to an improved quality of life. Where this is not the case, SBF’s are trained to seek remedies to deal with challenges at an individual, household and school level. 13 | P a g e Level 5: Indicators for achieving the overall goal The indicators17 for this project are: • Increased % of OVC who are of school-going age and whose schooling is uninterrupted • Increased % of OVC who are in the programme, who are eligible for grants and are receiving grants • Increased % of OVCs who are not malnourished • Increased number of community / government-funded interventions established as a result of COS intervention • Increased # of OVCs receiving care and support • Increased # of OVCs served per service category • Increased # of family members of OVC provided with care and support • Increased % of schools based support teams • Increased % of established community forums. Whether formal or informal, to create dialogue between community members, leaders and government departments • Increased # of volunteers will be capacitated through workshops and meetings Although the DoE and other district and local government departments were active in the community, the services were not integrated and focused on OVCs. One HDA respondent explained that “each government department has a clear mandate to contribute and implement government’s strategy on HIV/AIDS, yet they continue to work in isolation from each other...this COS project was necessary to bring together community level partnerships so that resources are used optimally to benefit OVCs in the shortest possible time.” The outgoing Project Manager also confirmed in her interview that relationships with the DoE were often challenging and strategic focus areas would clash “(and) they would understandably give precedence to their own work rather than COS focus areas” Hence, HDA with the assistance from PEPFAR and PACTSA put in place a Monitoring and Evaluation reporting system accompanied by highly efficient tools to consistently measure whether the following results questions could be answered in the affirmative at the end of the project life cycle18 : • Has quality services been delivered to OVCs? • How many OVC have received the services rendered (looking at the planned versus the actual number and account for any discrepancies)? • What needs have been identified and serviced in the extended family? • How was the target population been positively affected by these interventions? • Do targeted stakeholders perceive the project as essential? • Is the project sustainable? 17 COS MERP document of October 2010 to September 2011, pp9 18 ibid, pp 7 14 | P a g e 2.1.5 Project Structure In order to achieve the objectives and indicators listed above a structure was set up where HDA is responsible for the management of four personnel who are based in Welkom so that access to the targeted schools is enhanced. Depicted below, is the relatively flat organizational structure outlined by those interviewed. The reporting lines indicate that the project is therefore not subject to bureaucratic impediments. Diagram 2: Broad Project Structure The COS project was organized according to the following roles and responsibilities: • PEPFAR provided annual funding to the value of $ 645 651 and instituted internationally accepted planning, reporting and monitoring systems. • PACTSA was appointed to provide grant management functions and capacity building to the HDA staff in furtherance of the effective implementation of the COS project. • HDA set up an organizational structure partly in Johannesburg and wholly in the district to implement the COS Project. Support staff include an M&E Officer, finance and data capturing support. • A 4 person COS team was set up in Welkom to oversee the daily operations of the project. Appendix 1 shows the specific roles and responsibilities of volunteers and full time personnel working directly in schools and within the community. School Based Support Teams School Based Facilitators Welkom Based Coordinator/s and Admin Support JHB based M&E Officer and Project Support staff JHB based Programme Manager PEPFAR PACTSA 15 | P a g e Section 3 : EVALUATION PURPOSE AND DESIGN The aim of this Summative Evaluation is to assess whether COS was able to bring about any changes to the lives of targeted Orphans and Vulnerable Children (OVCs), their families and the community in general. The extent to which various services were provided has been recorded annually by the project on a comprehensive database. This evaluation there focused on gathering data related to program outcomes as opposed to service delivery result. The goal of the evaluators was therefore to utilize an appropriate evaluation methodology that will reveal the impact (i.e. outcomes) that access to support services has had on the quality of life of these children and their families. It is therefore essential to understand the rationale and design of the COS project in order to assess whether it indeed made a difference to the children it had targeted for support. The evaluation design and methodology was formulated around the following three key evaluation questions: 1. What are the key characteristics of the Circles of Support (COS) school-based project at its best, and what factors promote or hinder success of the model in responding to and addressing needs of children? 2. How effective was the project in enhancing the capacity of schools based facilitators (SBFs) and SBSTs in identifying and addressing the needs of vulnerable children? 3. In which areas of child wellbeing did the project make the biggest change? 3.1 Overview of the Evaluation Methodology Guided by the Terms of Reference and subsequent meetings with the PACTSA and HDA, a mixed methodology was supported for this evaluation. This method is largely preferred because it can yield richer, more reliable and valid findings than a single quantitative method would19. Secondly, in this case where trends, perceptions and behavior are measured, a qualitative analysis was deemed to be more appropriate. Several senior representatives from HDA were clear that the intention to measure “numbers” (quantitative evaluation) instead of quality (qualitative evaluation) is “contradictory to the ethos of the COS Project”. As a result, this evaluation incorporated the following mixed methods conducted in the following distinct phases: • Assessment of the project documentation and international best practices related to OVC support, this was largely qualitative and culminated in a Literature Review Report. • Individual and Focus Group interviews with PACTSA, HDA/ COS, school, community and government stakeholders. These were conducted qualitatively using 2 respondent interview guides and resulted in the Transcripts (listed in Appendix 2) analysed according 19 Sharp L and Frechtling J: Overview of the Design Process for Mixed Method Evaluations, Chapter 5 16 | P a g e to the themes in the questionnaires, and using the AtlasTI programme to produce data relevant to the three key evaluation questions. • Two separate guided questionnaires were developed for completion by learners and their parents and these took place over a three week period. These were largely quantitative in nature and captured using the Statistical Package for Social Sciences (SPSS) program - the data generated here is used in the analysis of the three key evaluation questions in subsequent sections of this report. 3.2 Sampling and Fieldwork Experiences 3.2.1 Sampling The sample was drawn using a randomized stratified multi-stage sampling with probability proportionate to size (PPS). The HDA COS project schools were stratified according to urban/Rural/Semi-Urban settings; following which a two stage sampling technique was utilized for the selection of eligible respondents. The selection of schools was the first stage of the sample selection procedure. Probability proportionate to size (PPS) was used with the total OVC served in the period October 2010 – September 2011 as measure of size (MOS). The eligible respondents (children) were sampled using simple random sampling (SRS) from the selected schools on the second stage. With this procedure schools were selected, and within them varying numbers of children were selected proportionate with the annual total OVC served number. The full version of the sampling strategy is included as Appendix 4. Sample Size The sample size was calculated using statistical package nQuery Advisor Version 7. It took into consideration the fact that the evaluation in general intends to assess the extent to which HDA COS project contributed to improved wellbeing of OVC. In calculating the sample size several points were therefore taken into consideration; the magnitude of change differences expected to be reliably measured and the degree of confidence with which it is desired to be certain that the observed change would not have occurred by chance (the level of statistical significance). The sample size allocated for each selected schools was based on proportionate to OVC served in the period Oct’10 – Sept’1 17 | P a g e Graph 2: Geographical location of schools visited 3.2.2 Pilot Prior to the commencement of fieldwork, each of the research instruments above and the methodology were piloted at Ikemisetseng Public School in Bothaville. Based on the findings, it was determined that it was essential to use bilingual fieldworkers and that the questionnaires needed to be translated from English to seSotho. Even though this pilot was informative, the extent of the challenges associated with interviewing younger learners and high levels of parent and learner illiteracy were only encountered once in the field. Facilitators therefore had to simplify a very lengthy questionnaire even further and extend the period of the interviews by at least 2 additional hours in order to obtain good quality information. 3.2.3 Fieldworker Training Our fieldworkers were trained on the purpose and structure of the COS project; the administering of the questionnaires; in research ethics related to interviewing children and OVCs and in approaches to interviewing community informants with low literacy and English proficiency levels. Two COS Coordinators also attended this training and benefited from the opportunity to learn about research and evaluation methodologies. 3.2.4 Respondent Co-ordination The COS team based in Welkom used the following methods to encourage participation from school based informants: • Schools were informed by letter regarding the timing and purpose of the evaluation, and requesting their cooperation in the process. • Parents were informed about the process by letter a week prior to the scheduled fieldwork informing them about the evaluation and requesting their cooperation. Furthermore, parents were issued consent forms to allow their children to participate. 13% 58% 29% Rural Peri Urban Urban 18 | P a g e • The learners were informed during the homework support sessions about this research and encouraged to attend the after school interview sessions. All participants were given refreshments and participating learners received educational books as a token of thanks. The research team coordinated individual interviews with Project staff and government respondents and all took place within a short time frame, indicating a genuine support for this evaluation. This co-ordination approach was very effective as is indicated in the total stakeholder respondents which exceeded the initial sample sizes proposed for this evaluation: Table 3: Number of Respondents Stakeholder Number interviewed Learners 245 Parents/Caregivers 253 Project Staff (PACTSA, HDA, COS, SBF’s) 42 Government 18 NGO’s and Community Members 93 TOTAL RESPONDENTS 651 19 | P a g e Major Conclusion: Key results achieved here are demonstrated by three high performing schools who participated in this evaluation, namely Oziel Selele, Phomolong and Senzile Combined School. The best practices were as a result of COS being able to work closely with a wide range of community based stakeholders to provide a circle of support to OVCs. Key strengths of the project are mobilizing community involvement which resulted in increased awareness and willingness of the community to identify and respond to OVC needs; community based service providers had a central point to offer and provide free remedial action; strong partnerships are built and strengthened with local, district, provincial and national Government. Regular home visits provide valuable information about the actual demand for services from OVCs and most importantly, the daily presence at schools enabled SBF’s to respond to urgent needs as and when they arise - thus able to fast-track access to support services at short notice. This is a structure that the DoE cannot put in place because of its own capacity limitations. The key weaknesses related to high turnover of staff which resulted in inconsistent service delivery to all schools sampled and COS creating high expectations in communities which far exceeded what the project was designed to accomplish. Section 4 : KEY FINDINGS 4.1 Key Evaluation Question (KEQ) 1 What are the key characteristics of the Circles of Support (COS) school based model at its best, and what factors promote or hinder success of the model in responding to and addressing needs of children? 4.1.1 KEQ 1 Strengths and Opportunities 4.1.1.1 Community outreach was a catalyst for change The COS project is rooted in the Theory of Change which espouses that all activities and interventions must link to the desired outcome of improving the quality of life for OVCs and their families through accessing a network of support. This was clearly achieved in three schools where in a single jamboree event; the project was able to register undocumented indigent families. Based on this information, COS was able to conduct home visits to assess the conditions under which OVCs lived. This was followed by the registration of all indigent families on the local municipality database as earmarked for free basic services. One caregiver stated that “all of this was done within 4 weeks of the Jamboree and I’ve been struggling to get free services from the municipality for years” Although COS reports show many other community outreach activities such as the Psychosocial services camps, Community Care Forums and joint events with local NGOs – these are not mentioned as regularly as the Jamborees which were more popular amongst respondents. 20 | P a g e COS captured data collected at Jamborees on their COS 1 and COS 2 Household Forms to identify those homes in greatest need in terms of nutrition and other material needs. In Senzile for example, these were set up within eight weeks of COS identifying this chronic need. In Oziel Selele, respondents cited that the Jamboree brought the OVC problem to the attention of strategic partners and the SBF was contacted by the Unicorn Farmer’s Project who offered and set up a food bank at the school; Dr Molausi in Welkom started to treat OVCs referred by the COS project, for free and Lovelife began workshops on sexual health and VCT. Observing this theory of change in action, researchers noted that in the three best performing schools learners were clean and looked happy and healthy, the school environment, although under resourced, was well kept and the commitment from teachers and management to identify and report newly vulnerable children to the COS project, was clearly evident in the comprehensive school reports submitted to COS. 4.1.1.2 Strong relationships with government were established Respondents from Oziel Selele reported that as a result of the COS project, the Department of Home Affairs has since opened a satellite office at the local Municipal offices and were hopeful that ”With the relationships that COS assisted us to establish with other government departments, we hope to see more of them opening satellite offices within the community”. According to another best practice school, in Senzile, it was reported that “the referral system to Department of Social Development is working much faster now that COS is around”. The two diagrams below represent how government involvement at school level has improved since the beginning of the COS project: Diagram 3: Stakeholder Involvement before COS School Management, teachers and parents OVCs DOE, DOHA, Municipalities DOA, Justice, private sector, NGOs Relational distance 21 | P a g e Despite an MOU provided by HDA/COS to Government stakeholders, the Department of Social Development (DoSD) and the Department of Education (DoE) initially understood the COS Project primarily as an intervention aimed at enabling OVCs to access schools and to be retained for prolonged periods. These respondents therefore initially did not play an active role in OVC wellbeing beyond receiving data about school attendance, drop-out rates and/or applications for grants. This view changed after COS established a foothold in the community as evidenced in the three best practice schools: Diagram 4: Stakeholder involvement after COS at Phomolong, Oziel Selele, and Senzile Schools Although these cited best practice cases indicate strong community and government involvement in the wellbeing of the OVC, there were other examples of a growing circle of support cited by poorer performing schools too. For example, positive relationships have also been built with the Departments of Home Affairs, Agriculture and Social Development (in Kweetsa), with the Department of Justice (according to DoSD representatives) and with Education (in Makeleketla). In some instances20, it is reported that the relationships with local municipalities have also improved and that more frequent meetings are taking place. Based on this oral evidence, whereas most community stakeholders initially had sporadic contact with schools, the trend is shifting towards greater daily contact. 20 Refer to additional examples captured in the Qualitative Report attached BUSINESS AND FARMERS LOCAL MUNIC NGOS DOSD DOA DOHA DOE HDA and COS SCHOOLS AND HOUSE HOLDS 22 | P a g e 4.1.1.3 Community based stakeholder networks created COS has successfully implemented a model where a wide range of community stakeholders are now actively contributing to the wellbeing of OVCs. Since the COS project began focusing on community awareness on the types of services offered by the various government and NPO organisations, the result has been an increase in utilization of services by families. The DoE respondents stated that prior to the COS project, the department was “hindered by its own inability to create community networks and multi-sector stakeholder involvement because it lacked the physical presence at all schools.” In other words the physical presence of SBF’s tasked with dealing specifically with OVC’s overcame the capacity constraints faced by the DoE’s in the identification and addressing of needs of vulnerable children and their families – teachers are unable to perform these tasks due to their teaching commitments. This success was also noted in varying degrees within most schools under evaluation, for example NGO's, CBO’s and FBO’s began working with indigent families in Monyakeng and Kweetsa. Larger and more well-known organizations such as labour unions (at Oziel Selele) and LoveLife (at Oziel Selele & Monyakeng), FAMSA (Morobe) have been incorporated into the project implementation ambit. At one of the poorer performing schools (Morobe), Community Care Forums and the South African Police Services have reportedly begun regularly attending COS meetings and community events. Thus the 93 NGO respondents generally agreed that the COS project was successful in creating circles of support around vulnerable families by mobilizing a wide range of stakeholders who actively responded to needs of vulnerable children and their families. Teachers and principals noted that NGO’s regularly visit schools to see how they can assist OVCs and these are visible examples demonstrating how stakeholders now interact more often and more personally with schools, OVCs and their families. 4.1.1.4 Greater focus towards resource optimization The COS ability to work within its means is a strong show of organizational focus, realism and flexibility - this is evidenced by the letter to the DoE submitted in 2011 which cited “As we move into this final phase we would like to consolidate as much as we can in order to improve the sustainability prospect of the project after September 2012.” This ability to reorganize the project and direct valuable yet scarce resources to the neediest schools is a strong indication that management at COS/HDA understood their limitations well. The achievement of continuous growth is demonstrated in the table below which shows that the project was able to expand its reach to OVCs from 3.6% to an estimated 20% by the end of this year. 23 | P a g e Table 4: Service Provision trends for 4 years Year October 2008 to September 2009 October 2009 to September 2010 October 2010 to September 2011 October 2011 to September 2012 Total to date Number of OVCs served Target: 5,400 Target: 9,000 Target: 12000 Target: 15,000 26,394 Actual: 2,507 Actual: 4,330 Actual : 11,557 Actual: 8,000 (at time of evaluation but expected to exceed Year 3 total) Projected growth of OVC’s per annum 21 69,950 73,950 77,950 81,950 81,950 COS support provided to 3.6% of OVC’s COS support provided to 5.9% COS support provided to 15% of OVC’s COS support provided to OVC’s predicted close to 20% (or 16 390 OVC’s served) Achievements per service category The key change agents are the highly motivated SBF’s with a daily presence at COS targeted schools. This physical presence at schools encourages regular updates of changes in vulnerability, consistent reporting to relevant government departments and the ability to respond in a short space of time to urgent matters related to the safety and care of the OVC – and based on this information, COS was able to make informed decisions about compacting its services from 90 schools to the current 60 schools. Whereas most of the 16 schools interviewed registered success in some of the service categories below, the three best practice schools registered quality services in each of them. This is the strongest evidence that the COS model of change is achievable. The following table shows the actual services provided to OVCs identified per school, for the 3 years of the project. Note that some learners received more than one service from COS but the purpose of this table is to demonstrate the service levels per school22 21 ASSA 2003 Model – cited in COS Child Participation Report, January 2011 and cited in ANC data for this district in 2009 22 Detailed results for all 60 schools and the number of services per child, is captured in the comprehensive database collated by PACTSA. 24 | P a g e Table 5: Annual achievements per thematic area for all schools Services provided October 2008 to September 2009 Baseline Year October 2009 to September 2010 October 2010 to September 2011 Percentage increase from Baseline Year Educational Support 199 1,417 Increased services by 1218 (612%) more OVCs from previous year 5,736 Increased services by 4319 (305%) more OVCs from previous year 2782% Household Economic Strengthening 76 81 Increased services by 5 (6.5%) more OVCs from previous year 1,191 Increased services by 1110 (1370%) more OVCs from previous year 1467% Psychosocial care (Psychological care after Oct'2009) 294 1,224 Increased services by 930 (316%) more OVCs from previous year 2,688 Increased services by 1464 (119%) more OVCs from previous year 814% Child protection 660 944 Increased services to 334 (43%) more OVCs from previous year 5 390 Increased services to 4396 (471%) more OVCs from previous year 716,6% General healthcare referral 1,021 2,382 Increased services to 1361 (33%) more OVCs from previous year 2,741 Increased services to 359 (15%) more OVCs from previous year 168% Clinical nutritional support _ 285 733 Increased services to 446 (157%) more OVCs from previous year 157% HIV prevention education 801 382 Decreased services by 419 (52%) less OVCs from previous year 1,199 Increased services by 817 (213%) more OVCs from previous year 49,6% Healthcare support for access to antiretroviral treatment _ - 4 - Food and/or food parcels 252 Government Agency (SASSA) discontinued food parcels Government Agency (SASSA) discontinued food parcels Government Agency (SASSA) discontinued food parcels Vocational training _ _ _ _ 25 | P a g e Notable Best Practices • The Education Support programme had the widest reach in all COS schools. Starting with just under 200 learners in its first year of operation, this unit was able to provide much needed homework support and school supplies to 5537 more OVCs by the final project year. Based on the target set by COS of assisting 26394 OVCs, over three years, this unit was able to assist 7352 (28%) OVCs to remain in and progress through the schooling system. Although a notable achievement, this indicates that the demand for COS homework support exceeds the current supply therefore, there is a space for other projects with a homework support focus to provide this service within this district. • Psycho social care is also a service offered at high levels within rural areas where access to this counseling is not easily obtained. COS has performed well to bring this service to outlying and under resourced areas and since the project began, this unit mobilized these services to 2394 more OVCs and their caregivers who required this intervention. • The Child Protection data shows that 2081 more OVCs were assisted with birth certificates, identity documents and benefited from Memory Work, since 2008. Setting up this service was so effective that schools and trained SBFs can continue without additional COS intervention. • General Health Care was very effective in rural schools and retired doctors and nurses were instrumental in providing these services to communities with little access to state health services. Since the project began, this unit was able to connect 1720 more OVCs and their families to local health professionals. • Clinical Nutrition was aimed at assisting OVCs with access to DoH programmes supplying multi-vitamins and other supplements to offset the effects of malnutrition in OVCs. According to the outgoing Project Manager, the need for Clinical Nutritional Support was not a big demand and all OVCs identified (733 OVCs) received it by the end of the final project year. 4.1.2 KEQ 1 Weaknesses and Threats 4.1.2.1 Incorrect understanding and expectations of the COS project Focus group participants at schools where the COS indicators were not comprehensively achieved, tended to understand and describe the COS project in terms of the services delivered to the intended beneficiaries and their families rather than a project that is meant to help relevant government, private sector and NGOs gain access to OVCs who can benefit from their services. Although an HDA respondent did intimate that “(COS) was not intended to be a service delivery program” this clear message was not translated adequately to all schools under study. SBF’s also noted that there was resistance to the COS project from teachers and school management because in cases when services requested were not received, SBF’s were accused of 26 | P a g e being “unprofessional and inept” this further reiterates the poor understanding of the facilitation role that SBFs play rather than that of actual service provider. This led to further estrangement of school stakeholders from the project. This attitude was also observed in schools during this evaluation where teachers and parent informants were resistant to participating in this evaluation and researchers had to extend the fieldwork by one additional week to encourage greater participation. This misunderstanding of the COS project objectives was also observed at Kweetsa focus groups where caregivers argued that the focus on children was undermining family structures as children ceased seeing their parents/ caregivers as role-models, and in some cases, “lost-respect”. These respondents argued that the project should rather focus on helping the parents/ caregivers and by extension the children will be assisted, instead of the inverse approach that currently holds. 4.1.2.2 Volunteer and stipend model not ideal for indigent communities The original COS proposal focused on “creating networks, building local capacity in a sustainable way and leaving behind a network of people that can ensure sustainability”, and the requirements by PEPFAR was for measurable service delivery to OVCs. The only way this vision could be achieved was through mobilising a mass volunteer presence at school level. Senior respondents from HDA cited that the original budget did not include allocations for volunteer stipends and the previous Project Manager added that volunteerism is difficult within indigent communities “as people always expect a permanent well-paying job at some point, and this has been a key challenge for COS in getting indigent youth to volunteer.” The effect that this has had on the successful implementation of the project is articulated clearly by one respondent “These people have to do the work as social workers, without the skills or qualifications to do so, and at less than R800 per month!” In order to manage the expectations of SBFs within a budget that made no allocation for stipends or salaries for them, COS re-prioritised their project objectives, scope and content in Year 3 of the project life cycle. The effect of this change was a reduction of the number of SBFs through a very effective Performance Management process administered by the project and contracts with poor performing SBFs were discontinued. Many high performing SBFs reported that they are constantly seeking permanent work offering salaries as opposed to a minimum stipend - this transient job nature of youth is understandable but this has resulted in service levels being compromised by the lack of staff continuity, delays in recruiting additional staff and the time required to train new staff to a suitable level – all not budgeted for in the initial funding proposal. In some cases, SBFs stated that they were impoverished by the project. SBFs were reportedly expected to cover the transport and telephone costs they accumulated during project implementation (they reportedly did not receive additional payment for this) and in some cases these costs were as high as the stipends they received. In one example showing how budgetary constraints affect output, an SBF stated that there is no budget to assist people who cannot afford to take the photos required for the Identification Document application, so she stated that “I failed to achieve my targets because of the lack of funds to assist people and not because I was not motivated or lacked the skill to assist them” 27 | P a g e 4.1.2.3 Community perception that COS does not respond to actual needs The inherent risk of any successful project is scope creep which entails the need to respond to every service requested from your target market. In focus group discussions in at least 5 schools, focus group participants lamented that COS does not focus on local challenges beyond the school such as unemployment of older youth and the ensuing problem associated with gangs and increased crime rates. Another area outside the scope of the COS project which respondents want included is assistance of abandoned, orphaned or vulnerable children from neighboring countries who are not attending schools because they have no valid South African identity documentation. The third area respondents want COS to deal with is the high level of Adult Illiteracy. Within a limited budget, COS is already challenged with providing its core services and is not able to take on additional work. A growing problem recognized by the DoE is the prevalence of teenage pregnancies, which is part of the definition of a vulnerable child according to the COS definition. One teacher articulated quite well that “we are not intervening to convince them to come back to school after having babies, and we are not convincing them that getting a boyfriend is not an alternative to education”. By including teenage pregnancy in the OVC definition, this support is an achievable function for all SBF’s as demonstrated in the Phomolong Case study attached. This unique case shows how an SBF who was alerted to a high pregnancy rate, was able to bring on board the DoE to secure the schooling of 300 pregnant teenagers and link these teenage girls to life saving pre and antenatal projects offered at clinics, thereby reducing the high infant mortality rate usually associated with teen births within this community. 4.1.2.4 Child participation in project life cycle Among senior project staff there was a clear recognition of the importance of ensuring child participation (at planning levels and not only at the level of implementation), and an awareness that this requires engagement with children that allows them to contribute meaningfully to the entire project process. One of the reasons cited during the in-depth interviews with representatives of COS and HDA for the reported lack of participation by children from the project planning stage (i.e. through a pre project implementation needs analysis), is that this project is modeled on similar USAID projects successfully conducted in other African countries so the initial project design did not require active consultation with children. In subsequent phases however there has been interaction and consultation with children and these engagements now form part of project reporting processes. Further, a study on the needs and participation levels of children was conducted by the HDA Monitoring and Evaluation Officer who confirmed that this has had a reported positive effect in a few schools – this was further evidenced in focus groups where child participation was conceptually understood and reportedly taking place. The quote below is from the Makeleketla focus group and illustrates the level of understanding and participation at the school. “We held child participation research with the assistance of the HDA research team, and we discovered how children would like to be treated and other things like that. When we were implementing the 28 | P a g e Hydroponic Gardens in different areas, we first consulted the children as beneficiaries and all things that are planted in our gardens were suggested by children themselves. So we involved them in some decision-making levels, but there is still room for improvement”. In other best practice examples, at Manyakeng, Makeleketla and Morobe it was reported that child participation in Project customisation has become so effective that “children in some cases take the first step” and approach Project representatives with new ideas and requests to participate in the Project changes. Although commendable, participants displayed through their answers a difficulty with how to practically include children in future project design and decision-making processes at the expense of the implementation phase deadlines. Recommendations 4.1.3 Recommendations for Key Evaluation Question 1  COS should implement a more inclusive Strategic Planning process going forward which allows for input from OVCs and their caregivers. This will clarify the COS project to beneficiaries and not raise untenable expectations from the community.  Regular performance management system should be put in place to monitor SBF skills gaps and ensure consistent service delivery in all schools. 4.2 Key Evaluation Question (KEQ) 2 How effective was the project in enhancing the capacity of schools based facilitators (SBFs) and SBSTs in identifying and addressing the needs of vulnerable children? Major Conclusion: Training accessed and conducted during the project was noted as positive, without exception1 amongst all staff and community members who were recipients of training. Training reports as well as oral submissions on this component indicated two important findings – firstly training was provided by a wide range of external service providers such as PACTSA, private training providers, government departments, the NPO sector and internal providers such as HDA’s M & E Officer and ‘trickle down’ training facilitated by capacitated COS staff themselves. Secondly, beneficiaries of training extended beyond project staff and included school and community stakeholders and even the OVCs targeted for support. This approach is valuable in that HDA/COS have the skills embedded within the organization to build capacity on a regular basis and by using external providers, the project is able to increase the span of skilled stakeholders required to run the project efficiently and enhance opportunities for handover to communities. The key weaknesses as identified by respondents related to not enough capacity building of the broader community to manage the project independently; inconsistent training of new SBFs resulted in poor data collection; major skills gaps between more experienced and less experienced SBFs, loss of institutional skills/ memory – all of which contribute to disruptions in service delivery levels. 29 | P a g e 4.2.1 KEQ 2 Strengths and Opportunities 4.2.1.1 Training content and recipients are relevant to the project The primary focus of training offered directly by PACTSA and HDA focused on generic project management skills (Project Cycle Management and Monitoring and Evaluation Methods respectively), however the COS program did leverage numerous specialised training courses from private and government sources. The impact of acquiring these skills was observed in the high quality services provided by COS staff in the Welkom and Johannesburg offices respectively. The table below shows the types of relevant training conducted and the number of staff who were up￾skilled during the project life cycle. From data gathered in the literature scan and from first-hand accounts in the field, it was clearly evident that the major beneficiaries of training were indeed those staff tasked with delivering the project objectives: The table below indicates that successes were registered in a few key areas. Table 6: Training Conducted over Project Cycle period Reported Training Received Total ever trained over 1-3 years Current staff with these skills (Skills Gaps) Assessment Comments Project Management and M&E Training Yr 1: No data 100% of current COS and school based teams Is the most critical skills needed for efficiency and project monitoring/reporting and this was achieved adequately Yr 2: 7 HDA/COS Yr 3: 20 Yr 4: Training for all new staff projected Training on COS Roles and Responsibilities Yr 1: No data 100% of current SBF’s and SBFT’s Trained on key tenets of the model, its aims, objectives and standards of service delivery. Inconsistencies in service levels at schools indicates that this training is not monitored for impact Yr 2: 74 Yr 3: 90 Yr 4: Training for all new appointees projected Training on Child and Youth Care Training Yr 1: No data 100% of those identified to participate in this training were trained Since this is a key performance area for SBF’s, the number of trained SBF’s is sufficient Yr 2: 109 (SBF’s) Yr 3: 31 Yr 4: Training for 30 more projected Counseling Training Yr 1: No data 22% of current SBF’s This is inadequate given that only 22% of schools are serviced by SBF’s that are trained in counseling Yr 2: 10 Yr 3: 20 Yr 4: Refresher training projected for all SBF’s still employed 30 | P a g e Reported Training Received Total ever trained over 1-3 years Current staff with these skills (Skills Gaps) Assessment Comments Hero Book Training (Hero Book) No data for previous years 11% of current SBF’s This is a good start; however the number of trainees must be increased so that all OVCs benefit from these skilled staff. Yr 3: 10 Yr 4: 15 Projected Memory Work Training (this was assessed very positively) No data for previous years 8 9% of current SBF’s This is inadequate for the number of learners who attend counseling support sessions needs to be extended Yr 3: 8 Yr 4: 8 more projected Health and Psycho￾Social training Yrs 1 and 2: No data 28 31% of current SBF’s Since this is a key performance area for SBF’s, the number of trained SBF’s is insufficient. None of the 28 trainees were deemed competent according to HWSETA assessments. Yr 3: 28 Yr 4: Projected that all 28 will be retrained HIV/AIDS Training Yr 1 : No data 50% of current COS, SBF’s and SBTF’s Since this is a key task for SBF’s, the number of trained SBF’s is insufficient. For Yr 4 COS is partnering with Bophelong Youth Skills Development Centre, Society for Family Health’s Subsidiary Youth AIDS and Matjhabeng HIV/AIDS Consortium who conduct widespread HIV/AIDS prevention education training. Partnerships with Lesedi Lechabile and CMT also contributed to training achievements in this study area. Yr 2 : 24 Yr 3: 45 Yr 4: Projections see Assessment Comments Child Care Yr 1: No data 31 34% of current SBF’s This certified training was conducted in partnership with NAACW. This is a fair number but ideally, all SBF’s working with children should acquire this training Yr 2: 30 Yr 3: 31 Yr 4: Projections for all SBF’s still on the project Care for Caregivers Training Yr 1: No data 22% of current SBF’s This is not a central focus of the COS project but does speak to the theory of change by empowering parents to provide community services Very good quality training was done by Thogomelo, however none of the trainees received competency ratings from the HWSETA. Yr 2: 11 Yr 3: 20 31 | P a g e 4.2.1.2 Monitoring and Evaluation Skills and Training improved project delivery The technical assistance provided by PACTSA and the ensuing training conducted by the HDA M & E Officer has been the most valuable contribution greatly improved to project efficiency and effectiveness. According to staff interviewed, the PACT MER advisors provided a range of technical assistance which enabled the COS program to improve data quality and use. One respondent stated that “they (Pact) ““have been always on the ground ready to assist”. Results of a recent and in the 3rd Quarter conducted a Routine Data Quality Assessment where it became evident that PACTSA training was highly effective. SBFs showed high capability in reporting on the various milestones achieved by the project. Another example of the results of Pact’s M&E capacity strengthening was the improved ability COS staff to PACTSA also capacitated the M&E incumbent to develop a query about children who had never received a service for this year or ever before, this skills transfer is reported by the ex-project manager as being very helpful to “identify children who have been lying idle in the database and are not getting a service.” The coordinators were able to focus on ensuring that these children got the services for which they had been identified. 4.2.1.3 Leveraging quality training from external providers to minimize cost of training The Department of Agriculture provided training on hydroponic gardens and animal husbandry and this is reported to have been particularly successful. According to COS staff, this training improved their HES strategies and helped COS forge strong relationships so that the department has agreed to train an additional 6 staff members in poultry management. The success in accessing government sponsored training has led to the respondents making recommendations to increase this component by accessing free courses such as the Community Care Worker course from the DoH; training on the Children’s Act and other relevant legislation offered by the Department of Justice. Other training envisioned for this final year is from the Community Media Trust (CMT) which produces educational series focusing on Health, HIV and AIDS, education, gender-based violence, topics of human rights and social development. CMT was sponsored by PEPFAR to develop an HIV prevention manual and they are currently in the process of seeking accreditation for this and which will be made available to COS for self study. Furthermore 2 COS staff are earmarked for the train-the-trainer course to acquire presentation skills which will be helpful for presenting the COS vision to new and existing stakeholders. 4.2.1.4 Training builds skills, instills confidence and creates employability opportunities in other sectors • The project has had some positive and measurable impact on the project staff and volunteers employed to implement it. In terms of the benefits of training, volunteers reported that their skills levels have improved particularly in their ability to accurately identify vulnerable children, as well as, the knowledge of how to assist these children. 32 | P a g e Training resulted in improved ability to implement the requirements of the project and consequently higher quality of services provided to intended beneficiaries of COS. • There was a noticeable improvement in service delivery since the project inception phases– especially in the ability of community members to access certain government services. Relationships with external stakeholders were strengthened which resulted in increased effectiveness in the referral component of COS. In particular, effective relationships were established with social workers and other social-development or social-support organizations, thereby, enhancing the overall effectiveness of COS and its integration into communities. Other benefits reported by recipients of training are: • The levels of self-esteem and sense of purpose among volunteers have reportedly increased. • Acquisition of new, relevant skills that they were able to apply to their work. • Improved ability to implement the requirements of the project and consequently higher quality of services provided to intended beneficiaries of COS. • Increased personal employability from full time employers • New job creation initiatives started within communities • Relationships with external stakeholders were strengthened which resulted in increased effectiveness in the referral component of COS. In particular, effective relationships were established with social workers and other social-development or social-support organizations, thereby, enhancing the overall effectiveness of COS and its integration into communities. • Noticeable improvement in service delivery since the project inception phases– especially in the ability of community members to access certain government services. • Collective training sessions resulted in improved relationships between communities and relevant government representatives. The COS Co-ordinator reported that although the project does not do staff exit-interviews nor keeps track of where SBF’s are employed after their tenure with the project, he asserted that most are offered full time positions in schools as clerks or grade R assistants. NGO’s are another source of employment for highly skilled SBF’s who have established relationships with such community based organizations so can network for full time jobs; very few SBF’s who seek work in Welkom or Bloemfontein are successful because of the high unemployment rate and low job vacancies in the province. 33 | P a g e 4.2.2 KEQ 2 Weaknesses and Threats The sustainability of organizations is dependent on highly skilled staff and in the case of the COS program, on skilled volunteers who are able to manage the project with limited assistance from COS. Although staff training conducted by COS was relevant and conducted consistently throughout the project term, two key weakness affecting sustainability was noted – firstly, community members were not effectively trained to manage the project in the event that COS were to discontinue and the high turnover of SBFs resulted in longer serving SBFs being highly skilled and effective whereas newer appointees receiving only basic training. This wide skills range is a direct result of the weakness of the COS volunteer model which is unable to retain highly experienced and skilled staff. 4.2.2.1 Training spend is not enough for consistent training of new recruits High SBF turnover and recruiting new SBFs from a community with a low skills base has affected the efficacy of the project - making training expensive in terms of financial costs and time taken off for training so targets were sometimes not met. This finding is confirmed by the outgoing Project Manager who reported that “new SBFs in the project usually have limited capacity, skills and knowledge and this has resulted in a perpetual need for the project to provide training and close monitoring. This has tended to be expensive for the project and has also at times delayed ‘reach’ of targets due to the extensive time taken to train and capacitate these SBFs.”23 In order to manage the training component of its work more efficiently, Equip for Life, a training and educational counseling organisation was contracted to assist in skills and knowledge development of SBFs. 4.2.2.2 Newly recruited SBFs lack basic workplace skills Analysis from the responses made by SBFs, indicate that because of the low stipends paid the COS program generally attracted SBF applicants with little or no work experience and therefore lacked basic workplace skills. SBFs with low levels of skills did compromise the project outputs “After three years, {SBFs} were still not completing forms or reporting correctly, which meant that the data generated was compromised and implementation was negatively affected.” Similarly, the referral system was initially problematic because SBFs “did not follow-up with the” intended beneficiaries, reportedly because “they were not properly trained, did not have the skills, did not understand what they were expected to do, or maybe they did do it, but they didn’t know how to report on it”. These capacity constraints can be attributed to inadequate supervision and insufficient on-the-job training. Under-skilled SBF’s also reported that they were overwhelmed by the monitoring and evaluation reporting requirements of the Project, and that this negatively impacted on their ability to provide services to the intended beneficiaries. 23 Bertha Magoge Handover Report, page 31 34 | P a g e Some reasons for poor performance cited by respondents were: • Inadequate on-the-job training, closer supervision from COS and no clear performance management processes • Unreasonable expectation of what SBFs are capable of doing • Stipend approach is not ideal for contexts of extreme poverty and unemployment • Absence of HDA M & E Officer in the field to conduct regular training and take immediate corrective action. • Project staff and HDA recognise that there are still not enough parents and teachers involved with and trained by the COS project. 4.2.2.3 Most training offered was not SAQA accredited The benefits of leveraging training accredited with SAQA are that it promotes professionalization of COS staff and their work; allows for Recognition of Prior Learning (RPL) and provides for access to formal higher education degrees in Community Development. Two SAQA accredited courses were facilitated by COS - firstly the NACCW course which is still underway with competency assessments and accreditation not completed so the impact of this training in terms of skills transfer and proficiency could not be determined at the time of this evaluation. Secondly, Thogomelo provided HWSETA accredited training in Psycho-Social Care, but it is reported by the outgoing Project Manager that “none of them met the HWSETA accreditation criteria and were therefore declared incompetent” 24 This means that psycho social services are provided by under￾qualified SBFs and the long term effects of this could negatively the psychological wellbeing of OVCs under their care. 4.2.2.4 Data Quality Issues Identified in RDQA process: The RDQA was an important process and shed light on the challenges associated with skills gaps, below are the key weaknesses (noted in the RDQA reports) threatening efficiencies and effectiveness and the remedial action taken by HDA/COS. At the time of this evaluation, the effectiveness of these remedies could not be measured because the key informant (PM) had resigned and was not accessible for interviews. 24 Bertha Magoge Handover Report, page 30. 35 | P a g e Table 7: Systemic Problems revealed through RDQA Process Problem Area Identified Remedial Action Planned/Implemented Weak file management system at the Welkom site office. HDA circulated the standard guidelines of file and data management to all staff. Remaining COS staff mentored on the job by M & E Officer. Low skills and capacity in data management among field and district office staff. This leads to under reporting of the number of children (OVC) served in the quarter (1st quarter). Conducting training needs assessment and plan to conduct short refresher MER courses for staff. Hosted a data quality management workshop for 13 cluster team leaders Limited quality controls for when data is transferred from paper based sources into computer. Continuing to monitor the quality of data and monitor the flow of error logs. Introduced carbonated data collection tools (COS1 & 2) to verify and control for errors. Setting up an online spreadsheet error log template which can only be accessed by HDA staff and this will speed up the process of rectifying errors on the forms. Lack of written guidelines on back up procedure for computerized information. A standardised weekly report from the M&E officer sent to the team on the reach of children, and any other issues which need attention to make the service delivery to the children effective and worthwhile. System failure to detect or pick out OVC lost to follow-up, dropped out of the project or those who have died as well as a lack of a standardized format for writing field visit reports by the project Coordinators. Streamlined the data flow so that there is less opportunity for missing data or late submission of data. Rapid data quality checks at the Welkom site M&E officer to do data audits on an ongoing basis and meeting with data capturers monthly this will continue improve data quality and mini reports about findings will be documented. 4.2.2.5 Inadequate Monitoring of Activities The COS project (like many similar projects) depend on funding and therefore must comply with donor requirements. Project implementation did to some extent become driven not by the COS ethos, but by the requirements of activity monitoring. The M & E Officer confirmed that at some stage “implementers started focusing on meeting the monitoring and evaluation requirements – bringing in the numbers and writing the reports – and no longer focused on the quality of the project or the sustainability of the interventions they were supposed to be making”. This is best 36 | P a g e illustrated by the discovery that some SBFs were reporting on services provided, which they had in fact not done – presumably to meet the reporting and target obligations. (This was exposed early on in the project and eliminated by means of the introduction of an improved monitoring and evaluation system). Furthermore an HDA manager also reported that they did not have enough monitoring and evaluation personnel specifically dedicated to the COS project. This meant that the number of site visits and training sessions that could take place were limited. The fact that monitoring and evaluation staff are situated in Johannesburg, while the project is being implemented in the Free State was also problematic and contributed to high costs and infrequency of visits. Training Community Forums to manage the project independently Transfer of ownership of the project to the community – Community respondents in the three best performing schools were confident that they can take over the project as volunteers. A respondent at Oziele Selele reaffirmed that “we know what the needs and problems are in the community, and have watched closely how COS does its work”. One of the key weaknesses of the COS model is therefore the non achievement of training community members in generic organizational management skills so that they are equipped to manage the project with limited involvement from COS. There is however a skills gap in the management capacity of these local communities, Nevertheless, with assistance from these communities, COS can focus its time and resources on poorer performing schools and in time also make them largely self-sufficient and requiring additional training support from COS so that the current levels of efficiency is maintained. 4.2.3 Recommendations for Key Evaluation Question 2  Data collection systems checking should not be regarded as a “once off” process and should therefore become part of Project Co-ordinators weekly routine and eventually the organisational culture of COS.  Sources of training with low (or no cost) must continue to be leveraged from external providers to decrease the demand on COS’s operational budget. The next COS budget must increase the training spend especially for community volunteers so that they are suitably skilled to manage the COS project independently. 37 | P a g e 4.3 Key Evaluation Question (KEQ) 3 In which areas of child wellbeing did the project make the biggest change? 4.3.1 KEQ 3 Strengths and Opportunities 4.3.1.1 Accessing community based circles of support The biggest lifestyle changes as reported by caregivers interviewed is that COS had excelled in its core focus to set up and grow community networks and multi-sector stakeholder involvement. SBF’s are generally reportedly to be most effective in facilitating access to child protection services. Many respondents were able to provide specific examples of assistance to abused children from police protection or in some cases, the removal of children to places of safety by Social Workers. In addition to the individual learners who received much needed protection, awareness of the rights of children has also increased within communities as a whole. This has resulted in caregivers being more knowledgeable about aid services within the community and having the confidence to approach them. The data presented in this section is in the chi square analysis format which is a formula measuring whether the expected results of the COS project matches the results that evaluators gathered from the field. The rule of a chi square analysis is that any result greater than 5% means that COS targets were indeed met or even exceeded. For the questions related to obtaining assistance from government, NGO and community based service providers, the results show a dramatic 265.02% increase in children and their caregivers accessing these services. Major Conclusion: The COS project objectives was to facilitate and monitor changes in the quality of life of OVCs and reflected the following internationally accepted indices - namely health, family life, community life, material well being, and access to quality education. The results show that the project registered success in the following areas:  Assisting children to access help and other services from community and government stakeholders  Improved the number of families who were able to access child support grants  Improved health and nutrition for the entire family  Helping children feel more secure in the family, in the community and at school  Access to school uniforms and other requirements Strengths centre on COS having a daily operational presence in schools and within households. This enabled SBF’s in particular to create important networks as need arose and to respond rapidly to individual need. Allocating an SBF per school is a highly effective model and is a best practice for OVC support in general. Weaknesses for this evaluation question derive from COS’s inability and lack of resources to retain SBF’s resulting in inconsistent service levels; increased training spend for as replacement staff and losing momentum with stakeholders because staff are constantly replaced. 38 | P a g e 4.3.1.2 Improved psychosocial wellbeing of OVCs and their families The second most remarkable change (> 219.32%) is closely related to the above. Vulnerable families feel less isolated and more integrated into the surrounding communities than they were before COS implementation took place. Most learner and caregiver responses note that their relationships within the community has improved as a direct consequence of the COS project and that they have established more amicable relationships between themselves and schools, with family relations living in and outside the home and within the community in general. Caregivers noted that participating children were benefited more because since the COS project began they have observed that their children are “less stressed”, having fewer “behavioral problems”, having “more self-confidence”, “playing more”, having “more friends”, displaying improved “emotional control”, behaving in more “self-disciplined ways” and participating in “sport” more frequently. Respondents attributed this positive trend to increased access to counseling and social services, which was facilitated exclusively by COS staff. 4.3.1.3 Improved Nutrition and health As a result of the upsurge in house visits, COS was able to identify household nutritional needs and respond to them effectively Data collected from caregivers and children indicate that more families are eating meals regularly(> 65.98%).. This is as a result of COS fast-tracking the DoA food garden project at homes and in schools, where they are needed most. A DoA responded noted that “although the DoA project was in place since 2000, it did not enjoy the success that it now does” and this is directly attributable to SBF’s linking an existing government project to homes and schools. As evidenced in the Handover Report of 2011, the project established hydroponic gardens in 7 schools, some of these gardens and two cattle were donated by DoA to one of the neediest schools. The food grown here is either cooked on the premises to feed learners or sold and money used to buy other staples such as maize meal and protein. As a result, diet has generally improved, with children and their families eating more fruit and vegetables (28.26%); higher consumption of proteins (20.59%) and more OVCs taking vitamins and supplements (16.85%). Some first-hand information from Caregivers/ Parents/ families in general are “We can obtain health services or medicines which we did not know was available for free at clinics and can now spend our money on other things needed at home” and “COS trained us on agriculture and food gardens and now we grow our own vegetables at home, so we have more to eat even though we spend less money on food” 4.3.1.4 Increased access to government grants These examples are laudable but even more exceptional is COS being instrumental in increasing the household income through their facilitation of access to child grants. The graph below shows that COS had been able to assist 38.9% more families to access social grants. In addition to those who already receive grants, the impact of this achievement is that nearly 60% of families are now able to buy good quality food, access health service providers and generally improve their quality of life. Two challenges remain; firstly COS must facilitate grant access to the 23.9% of families 39 | P a g e who are still threatened by the effects of chronic income poverty and secondly to obtain more data on the 16.2% who are unaware of their eligibility for social grants. Graph 3: COS Assistance in Accessing Grants 39% 24% 16% 21% Yes No Do I Qualify? Always Had 4.3.1.5 Improved Education Support One of the most frequently mentioned positive effects of the project has been on education according to the caregiver and teacher respondents. They reported improved literacy rates, improved pass rates, improved overall academic performance, improved attendance, and improved discipline among learners as a result primarily of the homework assistance provided to learners. Some respondents described the changes in educational status of the learners as “remarkable”. Also derived from field work data is that OVCs were given greater access to provisions of uniforms (> 43.42%), stationery (>20.77%) and school bags (>7.31%) which was an existing DoE project however COS was more successful than government in setting up the logistics for OVCs to receive these. 4.3.2 KEQ 3 Weaknesses and Threats • Household Economic Strengthening and Homework Support are still areas where the need is greater than the supply of services. Although COS has fared well in partnership with the Departments of Agriculture and Social Development in reducing income poverty and malnutrition, according to family respondents, there continues to be a strong need for interventions around food security (46.3% still required). This unit was able to provide grants and food gardens to 1391 more families since this project began. The results within semi urban and urban households25 were low and this is an area that would require more 25 Refer to Appendix 6: Service Delivery in 16 Sampled schools 40 | P a g e human and financial resources especially since these settlement types record high unemployment, food insecurity and income poverty rates. • HIV Education was provided to 398 more children and adults since the project began and given that the HIV/AIDS pandemic is high in this district, this result is insufficient for 3 years of operation. • Mobilising access to ARVs was very low with only 4 individuals being assisted by COS (over three years) to access this medication. • The database indicates that no Vocational Training was conducted to assist caregivers to obtain employability skills in order to decrease their dependency on state grants. • There is currently no government or community based organisation with the financial and human resources to absorb the COS project in its current form and scale. Unless HDA is able to source funding from an alternative donor, the COS model will have difficulty being sustained because a centralized co-ordinating body is still required to manage and monitor local service providers and community members. 4.3.3 Recommendation for Key Evaluation Question 3  The biggest threat to the continuance of these services is if SBFs are no longer active at school level and house visits/assessments are not done consistently – this would result in OVCs not being timeously identified for much needed circles of support. COS must put in place a staff retention strategy and increase its capacity and frequency of training to SBFs and community stakeholders. 41 | P a g e Section 5 : CONCLUSION COS has performed well in most cases and experienced some challenges in only a few areas. The indicators set by the COS project at its inception did not specify a percentage increase from its baseline however this evaluation can conclude the following: Indicator Rate of Achievement Concluding Comments Increased % of OVC who are of school￾going age and whose schooling is uninterrupted Achieved Parents data26 indicate marginal changes in OVC school attendance - 4% more OVCs attend school more often and 6% more children are no longer afraid to go to school. The data for Education Support indicates an impressive growth in the number of OVCs who seek assistance from COS homework support to ensure that their progression per standard is not interrupted: 199 learners in year 1 1417 learners in year 2 5736 learners in year 3 Increased % of OVC who are in the programme, who are eligible for grants and are receiving grants Partially Achieved The data analysed indicated a definite increase in the number of OVC receiving grants 76 households in year 1 81 households in year 2 1191 households in year 3 However, 23.9% of families who participated in this research still require grants and 16% are not sure of their eligibility status Increased % of OVCs who are not malnourished Partially Achieved This evaluation did not measure malnutrition medically; however it was observed that most schools had vegetable gardens, although many vulnerable homesteads still suffer from food insecurity. Appendix 8 shows that only 53.5% of the sample never skip a meal; 3.3% still skip a meal daily and 19.1% skip a meal more than once a week. OVC nutrition has improved at school but over weekends and school holidays, this is not guaranteed. Furthermore, SASA provided 252 food parcels which COS assisted with delivery in 2008/9 but government had discontinued this aid in years 2 and 3. Increased number of community / government-funded interventions established as a result of COS intervention Achieved DoE, DoA and DoHA now bring services to the level of the school and through satellite offices but departments of Health, Justice and the Local Municipality must be encouraged to work more closely with OVC’s 26 Refer to graphs in Appendix 7 – Absenteeism and Fear of going to School, page 65 of this report. 42 | P a g e Indicator Rate of Achievement Concluding Comments Increased # of OVCs receiving care and support Partially Achieved No SBF’s accredited on Thomogelo Psycho Social training. Of the 36 000 potential OVC’s in 60 school, the increases were marginal: Year 1 = 294 Year 2 = 1224 Year 3 = 2688 Increased # of OVCs served per service category Achieved According to database of services, all services areas showed increases except Vocational Training. Increased # of family members of OVC provided with care and support Partially Achieved ARV’s still not widely accessed (only 4 have benefitted in 3 years of the project), however referrals to health care has improved year on year: Year 1 = 1021 Year 2 = 2382 Year 3 = 2688 There is no data for parents who received psycho￾social treatment Increased % of schools based support teams Achieved Present in all schools. Effectiveness of SBST’s are inconsistent and not one is able to run the project independent of COS Increased % of established community forums. Whether formal or informal, to create dialogue between community members, leaders and government departments Achieved Present in all schools. Effectiveness of Community Forums are inconsistent and not one is able to run the project independent of COS Increased # of volunteers will be capacitated through workshops and meetings Partially Achieved The was clear evidence that the number of trained volunteers had increased however, high SBF turnover disrupts training and affects the provision of consistently high service levels 10 Indicators 5 Achievements 5 Partial Achievements 0 None Achievements This project is an example of a Best Practice. 43 | P a g e The COS recipe for success should be used as a best practice case for similar projects countrywide and four best practices can be replicated nationally - ffirstly, COS has an operational presence at every school where they operate and their monitoring tools are highly effective to collect information from each child/household. This made it possible to gather accurate information about the rate of vulnerability and thereby to put in place a package of services which addressed need at individual level. Secondly, COS has built strong relationships with community and government departments – 93 NGO’s and 18 representatives from government contributed to this evaluation and this was a clear indication that COS had indeed embedded their vision within this district very well that participation in any COS related processes are always widely supported. Thirdly, by including non COS schools in community events, the project understands very well that with a limited budget, they cannot assist all OVCs/their families and neither do they have the resources for ongoing marketing campaigns therefore they put in place annual jamborees where the entire district attend (irrespective of vulnerability) and learn about COS services. Finally COS has developed a good reputation due to service delivery within the community and within a relatively short period of time. This has two important implications - it creates an opportunity for COS to showcase their achievements to other donor agencies to consider funding the project beyond 2012 and with established community relationships, should COS register as an NPO then the transition will be easier because the COS brand is already known and highly valued. 44 | P a g e Section 6 : KEY LESSONS LEARNT AND OVERALL RECOMMENDATIONS Three key lessons were learnt over the COS project period: 6.1 Community wide interventions for OVCs work The COS model and approach towards assisting OVCs and their families, using schools as a base, is effective and it reflects best practice observed in the international OVC arena. Community events which included non COS schools, government, the NPO sector and the private sector has embedded the COS project within the community very well and this approach is a best practice for replication for similar projects operating throughout the country. Recommendations • COS should continue to host regular community events where community stakeholders are constantly made aware of the local OVC challenges and providing a central contact point for stakeholders to make contributions towards OVC care. • The data sheets which COS completes for OVCs at these community events provides an accurate picture of the actual numeric needs at local level. This information should be shared with government departments regularly so that national/provincial programmes and funding allocations are needs driven and based on actual numbers. 6.2 The Volunteer Model in indigent communities causes project instability A key weakness however is the dependence on volunteers from indigent communities to ensure a daily operational presence at schools. The volunteer model by nature does not encourage long term commitment to the project. Therefore the high transience of volunteers should be planned and budgeted for. Secondly, most highly skilled volunteers from indigent communities are more employable in urban nodes and are offered permanent and higher paying jobs away from the community, thus resulting in a high turnover rate of highly skilled SBFs. Those volunteers who have little or no workplace skills constitute the recruitment pool from which COS had to source replacement SBFs and this compromised the consistency of service levels in the schools under study. Recommendations • COS should allocate a larger budget to employ full time co-ordinators for each of the four local municipalities in this district. • Refresher training and sharing forums should be held monthly for all SBFs and community volunteers. This will ensure that service levels remain consistent at all schools and that community volunteers can manage the project with limited involvement from COS. 45 | P a g e • COS should also explore creative incentives to retain effective staff either through financing specialised training (career development); convening annual recognition awards ceremonies; providing once-off cash bonuses to high performers or brokering a full time employment arrangement with schools which the DOE and the private sector could jointly fund. 6.3 COS can be scaled down to focus on schools with greatest needs COS has succeeded in mobilising services offered by NGOs and government and in the three best practice schools under study (and others that COS should assess), COS should play a monitoring role rather than an implementing agency. These duties can be handed over to schools, community organisations and volunteers who – as a result of the COS project, now have the basic skills to continue providing support to their OVCs. The organisational management capacity of these schools and volunteers is however still limited for COS to exit immediately but through additional management training, COS will be able to reduce its management role and physical presence in better performing schools, and redirect their resources to poor performing schools. The indicators set by COS at project inception did not include actual numerical targets - the project aimed to achieve increases in all 10 Indicators. Community Management Forums who are ready to take over the COS project at their schools should be assisted by COS with developing specific numerical targets - for example “20% increase in access to grants per annum for X municipality” Recommendations • COS Welkom staff and the schools coordinators should assess each of their schools according to level of independence from COS. • Full time coordinators should be employed for each of the four municipalities in this district to manage the service levels of the schools in those areas. Having a municipal rather than a district focus will enable rapid responses to poor performing schools. • The M & E Officer should be based in Welkom and conduct SBF and volunteer training once a month in each of the four municipal areas. • Once COS assesses that a skilled community management forum is established in each municipality, COS should co-ordinate a strategic planning process for each forum where numeric targets are set for indicators. The future role of COS would then be to focus on monitoring and supervision only. 46 | P a g e Section 7 : ACKNOWLEDGEMENTS Umhlaba would like to commend the following persons who contributed to the success of this evaluation: • Kyle Orth of COS who contributed greatly to the finalisation of this Evaluation Report. • Myles Ritchie from HDA who encouraged the COS staff to work tirelessly to secure respondents. • Mpho Selebalo from COS who was so efficient and effective as an organiser that the schools she was tasked with, always registered the highest response rates. • Vusi Shongwe and Itumeleng Pitsi from COS who tried under very difficult circumstances and within a short space of time, to co-ordinate respondents at schools • Rita Sonko from PACTSA who provided valuable guidance throughout the project and more constructively during the reporting of these results • Addis Berhanu of PACTSA who made regular contact with the team leader to assess progress and offer support. With great appreciation, Umhlaba recognises the contributions by learners, caregivers, principals and teachers from whom valuable data and anecdotes were collected. Our research team also acknowledges the informants from government, the NPO and the private sector and for the ex Project Manager who availed herself for an interview. 47 | P a g e Section 8 : APPENDICES Appendix 1: Basic Situational Analysis of target schools and reduction criteria used Category and Name of School Type of Intervention Proposed Devolvement from high performing schools: Khotso Primary; Tlongkanyeni; Mamelang - Thuto Tataiso; Dr. Mngoma; Tjantjello; S.A. Mokhothu Six schools within the project are well resourced, performed well in internalizing COS activities within their day to day operations. The project does not need to continue investing its scarce resources. The schools are self sufficient; however COS Coordinators and provincial coordinator remain in touch with the Principal and SBST for minimal support monitoring visits. These schools will continue to be involved in the large scale service delivery events that are organized by COS for example Jamborees or PSS camps. These schools are ready for complete hand over to DOE and for the LSF to do day to day monitoring. Intensification of services to poorly performing schools: Akademia High; Matseripe Secondary; Monamodi Primary ; Matshidiso Secondary; J.C. Malotle Primary; Taiwe Comprehensive; Boliba Primary; Mantshebo primary; Nanabolela High; Tikwana high School; Nthutuzelo primary; Repholisitswe high; Matima lenyora primary; Diphetoho; Boase; Marematlou; Tikwe; Bovaal Twenty schools are poor performing. These schools have constituently registered a very low number of children (less than 30) despite being resourced with a SBF throughout the project cycle. This may be indicative of the fact that in certain areas where these schools are located there is simply not a high burden of orphans and vulnerable children that meet the criteria of support. As such the SBFs have failed to identify them in the required numbers to meet the set targets. From this point of view, it makes resourcing sense for the project to invest in schools where there are large numbers of OVCs. Another reason that has also been noted as contributing to the poor performance has been notable resistance on the part of some school authorities to prioritize the project. As such there is poor commitment towards meeting the targets set in the project. After three years of intervention, this situation is unlikely to change and therefore it is better to concentrate the scarce project resources in schools that see a value add through the circles of support. Too far from the project operation : T.S. Matlaletsa P/S; Boshof Hoer skool; Kagisano C/S; Kgololosego High school; Itokisetseng Due to this distance, these schools have not always received adequate support and monitoring due to restrictions in the budget. The result has been that due to this reduced level of support as compared to other schools within fair distance, the schools have performed poorly and are often not as clued up on the project in comparison to other schools which are closer. Schools that still benefit need of support and close monitoring: Hlaboloha; Ikemisetseng, Letlotlo; Mophate,Oziel Selele; Tshedisehang, Iphateleng; thabeleng, Katoloso; Letsibolo, Mmabana; Monyakeng, Dihwai; Tshireletso, Daluvuyo; Lehakwe, Setshabelo; Mokgwabong, Icoseng; Inpucuko, Iyumeleng; J.C Motumi, Khotsong; Marobe, Mohobo; Seqhobong, Thusanong; Malebaleba, Dieketseng; Phahamisanang, oiketong; Bahale, Kheleng These schools are deemed to be still in need of support and close monitoring. They are close enough to the project base and there are sufficient levels of enthusiasm and cooperation within the schools structures that will facilitate and support project implementation. The data shows that over the past three years these schools have continued to show notable progress 48 | P a g e Appendix 2: Roles and Responsibilities of the COS team MONITORING AND EVALUATION ROLES AND RESPONSIBILITIES Community Care Workers The identification and screening of children for eligibility in the community. Conduct a needs assessment with the child and parent/guardian. Information collected during the identification and needs assessment Make referrals to service providers. Complete referral form. Capturing of services received per child. Conduct home visits and complete the appropriate home visit form COS School Based Facilitators The identification and screening of children for eligibility in the school. Conduct a needs assessment with the child and parent/guardian. Information collected during the identification and needs assessment Make referrals to service providers ( Complete referral form) Submit record of all services received to Team leaders. Conduct home visits and complete the appropriate home visit form Cluster Team Leaders Collect data from the CCW and SBFs on a weekly basis Conduct 1st data verification of all forms submitted. Submit verified forms to Cluster Coordinators Cluster Coordinators Conduct 2nd data verification of all forms submitted; collate data and compile a monthly report Submit all forms to the Project Assistant. Compile success stories Project Assistant Enters data onto the database Files all COS Forms Project Associate Checks all data entries Manages Files and information retrieval Monitoring and Evaluation Officer Sets up and oversees the entire MER system of the project : consistently monitors the data collection tools for quality data collection, provides training on basic M&E and data collection tools, develops standard operating procedures to ensure data quality, verifies the data, analyses data and compiles reports. Conduct internal audits of the data quality on a quarterly basis. Conduct data quality assessment on a 6 months basis Analyses data and compiles reports. Assess implementation plan and benchmark calendar on a monthly basis Programme Manager Compiles reports. Assess implementation plan and benchmark calendar on a monthly basis. 49 | P a g e Appendix 3: Transcripts of Interviews 1. Transcript of Focus Group conducted at Tshomarelo Primary School, 22nd March 2012 2. Transcript of Focus Group conducted with representatives of the Community Care Forum and the South African Police Services in Dealesville, 23rd of arch 2012 3. Transcript of Focus Group conducted at Tlamanang Primary School in Hoopstad on 26th of March 2010 4. Transcript of Focus Group conducted at Senzile Combined School in Hertzogville on 19th of March 2012 5. Transcript of Focus Group conducted at Reseamohetse Public School of 22nd of March 2012. 6. Transcript of Focus Group conducted at Phomolong Intermediate School, 23rd of March 2012 7. Transcript of Focus Group conducted at Oziel Selele Comprehensive School in Khutsong, Bothaville, 23rd of March 2012 8. Transcript of Focus Group conducted at Monyakeng Primary School on 27th of March 2012 9. Transcript of Focus Group conducted at Morobe Primary School on the 28th of March 2012 10. Transcript of Focus Group conducted at Maremotlou School on the 20th of March 2012 11. Transcript of Focus Group conducted at Makeleketla Public School on the 22nd of March 2012 12. Transcript of Focus Group conducted at Magakajane Public School conducted on 20 March 2012 13. Transcript of Focus Group conducted at Kweetsa Primary School on the 20th of March 2012 14. Transcript of Focus Group conducted at Dihwai Public School in Allanridge on the 26th of March 2012 15. Transcript of Focus Group conducted at Dieketseng School on the 23rd of March 2012 16. Transcript of Focus Group conducted at Ntshireletse OVC drop-in Centre near Diketseng School on the 20th of March 2012 17. Transcript of Focus Group conducted at the Young Women’s Christian Association near Dieketseng School on the 20th of March 2012 18. Transcript of In-depth Interview conducted with representatives of the Department of Social Development in Welkom on 29th of March 2012 (Ms. C. Makhaotese) 19. Transcript of In-depth Interview conducted with a representative of the South African Police Services near Dieketseng School on the 20th of March 2012 (Captain Simon Blandile). 20. Transcript of In-depth Interview conducted with Adonio Simango, the HDA Monitoring and Evaluation Officer for the Circles of Support Programme on the 22 March 2012 21. Transcript of In-Depth Interview conducted with Shawn Malone PACTSA Programme Director on the 16th of March 2012 22. Transcript of In-depth Interview conducted with Saul Johnson Managing Director of HDA conducted on the 22nd of March 2012 23. Transcript of In-depth Interview conducted with Myles Richie a strategic consultant for the COS project and PM of the COS project in the Eastern Cape, conducted on the 22nd of March 2012. 24. Transcript of In-depth Interview conducted with Jennifer Baumann the Monitoring and Evaluation Manager for HDA National Systems on the 22nd of March 2012 25. Transcript of In-depth Interview with Addis Berhanu Monitoring and Evaluation Programme Manager for PACTSA on the 16th of March 2012 26. Transcript of In-depth Interview with HDA Project Manager/ Provincial Coordinator Vusi Shongwe on the 27th of March 2012 27. Transcript of In-depth Interview with HDA Project Manager/ Coordinator Mpho Selebalo on the 27th of March 2012 28. Transcript of In-depth Interview with HDA Project Manager/ Coordinator Itumeleng Pitsi on the 27th of March 2012. 29. Transcript of In-depth Interview with Social Worker within the Department of Education 30. Transcript of In-depth interview with Deputy Chief Education Specialist from the Department of Education. 31. Transcript of telephonic Interview with ex Project Manager Bertha Magoge. 50 | P a g e Appendix 4: Sampling Strategy Health Development Africa: Circle of Support (COS) Project Evaluation Sampling Strategy The sample was drawn using a randomized stratified multi-stage sampling with probability proportionate to size (PPS). The HDA COS project schools were stratified according to Urban/Rural/Semi-Urban settings and a two stage sampling technique for the selection of eligible children was utilized. I- Stratification of Schools as per the criteria The following criteria were used to stratify the schools (a) the duration the schools have been with the project(≥ 2 years); and (b) the urban/rural/semi-urban settings Table 4.1: HDA COS Project Schools; filtered by duration of stay with the program and Urban/Rural/Semi-Urban settings Health Development Africa Project Name Circle of Support (COS) Province / District Free State / Lejweleputswa School Name City Suburb Urban / Rural / Semi-Urban Primary / Secondary Annual OVC Served (Oct'10 - Sept'11) 1 Aramela Combined School Boshof Boshof Rural Combined 245 2 Boshoff Intermediate School Boshof Kareehof Rural Primary 88 3 Tshomorelo Primary School Dalesville Tswaranganang Rural Primary 294 4 Kegomoditswe Primary School Hertzogville Hertzogville Rural Primary 330 5 Senzile Combined School Hertzogville Hertzogville Rural Secondary 320 6 Thoriso Public School Hoopstad Tikwana Rural Primary 227 7 Tlamanang Public School Hoopstad Tikwana Rural Primary 226 8 Kagisano Combined School Soutpan Ikgomotseng Rural Combined 87 9 Verkeedevlei Verkeedevlei Verkeedevlei Rural Primary 240 10 Refihletse Combined School Verkeerdevlei Verkeerdevlei Rural Combined school 149 11 Matshediso Intermediate School Brandfort Brandfort Semi - urban Secondary 55 12 Monamodi Primary School Brandfort Brandfort Semi - urban Primary 73 13 Phomolong Intermediate School Hennanman Phomolong Semi - urban Primary 138 14 Kweetsa Intermideate School Hennenman Phomolong Semi - urban Primary 134 15 Moso Public School Hennenman Phomolong Semi - urban Primary 119 16 Reiketseditse Primary School Hennenman Phomolong Semi - urban Combined school 97 17 Concordia Theunissen Theunissen Semi - urban High 100 18 Leboneng Public School Theunissen Theunissen Semi - urban primary 81 19 Reseamohetse Public School Theunissen Theunissen Semi - urban Primary 86 20 Ecco Primary School Ventersburg Phahameng Semi - urban Primary 191 21 Kgauhelo Public School Ventersburg Ventersburg Semi - urban Primary 83 22 Makeleketla Primary School Winburg Makeleketla Semi - urban Primary 110 23 Naledi Ya Botjhabela Secondary school Winburg Makeleketla Semi - urban Secondary 68 24 Tlongkganyeng Primary School Winburg Makelketla Semi - urban Primary 67 25 Bahale S/S Hennenman Phomolong Semi - urban Secondary 134 26 Boitekong P/S Virginia Meloding Semi - urban Primary 158 27 Dihwai Public School Allanridge Allanridge Semi urban Primary 278 28 Tshireletso Public School Allanridge Nyakallong Semi urban Primary 195 29 Hlaboloha Public School Bothaville Bothaville Semi urban Primary 204 51 | P a g e School Name City Suburb Urban / Rural / Semi-Urban Primary / Secondary Annual OVC Served (Oct'10 - Sept'11) 30 Ikemisetseng Public School Bothaville Bothaville Semi urban Primary 53 31 Letlotlo Naledi Bothaville Kgotsong Semi urban Primary 180 32 Mophate School Bothaville Kgotsong Semi urban Secondary 193 33 Oziel Selele Comprehensive Bothaville Kgotsong Semi urban Secondary 153 34 Tsehdisehang Primary School Bothaville Bothaville Semi urban Primary 133 35 Ikgwentelle School Bultfontien Phahameng Semi urban Secondary 104 36 Magakajane Public School Bultfontien Bultfontien Semi urban Primary 106 37 Rainbow High Bultfontien Bultfontien Semi urban Secondary 25 38 Iphateleng Secondary School Wesselsbron Wesselsbron Semi urban Secondary 110 39 Ithabeleng Public School Wesselsbron Monyakeng Semi urban Secondary 208 40 Katoloso Primary School Wesselsbron Monyakeng Semi urban Primary 82 41 Letsibolo Primary School Wesselsbron Monyakeng Semi urban Primary 195 42 Mmabana Primary School Wesselsbron Monyakeng Semi urban Primary 170 43 Monyakeng High School Wesselsbron Monyakeng Semi urban Secondary 213 44 Ipopeng Secondary School Winburg Winburg Semi- urban Secondary 40 45 Icoseng Primary School Odendaalsrus Kutlwanong urban Primary 88 46 Impucuko Public School Odendaalsrus Kutlwanong urban Primary 55 47 Itumeleng Mabelle Primary School Odendaalsrus Kutlwanong urban Primary 97 48 Malebaleba Public Primary Odendaalsrus Odendaalsrus urban Primary 106 49 Marobe Primary School Odendaalsrus Kutlwanong urban Primary 113 50 Mohobo Public School Odendaalsrus Kutlwanong urban Primary 68 51 Khotsong Public School Ordendaalsrus Kutwanong urban primary 64 52 Lehakwe St Helan Park Motsethabong urban Primary 158 53 Boase School Virginia Eureka Park urban primary 151 54 Dieketseng Public School Virginia Meloding urban Primary 140 55 Marematlou Secondary School Virginia Virginia urban Secondary 148 56 Phahamisanang Primary School Virgnia Meloding urban Primary 145 57 Daluvuyo School Welkom Motse-Thabong urban Primary 100 58 Mokgwabong Primary School Welkom Thabong urban Primary 90 59 Nanbolela Secondary School Welkom Thabong urban Secondary 116 60 Setshabelo Primary school Welkom Welkom urban Primary 100 52 | P a g e II- Multi Stage Sampling Stage 1: Selection of intervention schools (a) Selection of PSU (Schools) in Rural Settings 1 st list Primary Sampling Units (schools) as per the Rural stratification above with corresponding approximate Measure of Size (MOS): total OVC served in the period Oct’10 – Sept’11 2 nd - Starting at the top of the list, calculate the cumulative measure of size 3 rd - Calculate the sampling interval 4 th - Select a random number (RS) between 1 and SI= 735 and compare the # with cumulated measure of size 5 th - Random # selected, RS = 702 Subsequent units are chosen Primary Sampling Units (PSU) (Intervention Schools) # of OVC served : Measures of Size (MoS) Cumulative Selection (*) Aramela Combined School 245 245 Kagisano Combined School 87 332 Refihletse Combined School 149 481 Boshoff Intermediate School 88 569 Tshomorelo Primary School 294 863 * Kegomoditswe Primary School 330 1 193 Thoriso Public School 227 1 420 Tlamanang Public School 226 1 646 * Verkeedevlei 240 1 886 Senzile Combined School 320 2 206 * Schools with * are selected (b) Selection of PSU (schools) in Semi-Urban Settings 1 st list Primary Sampling Units (schools) as per the Semi-Urban stratification above with corresponding approximate Measure of Size (MOS): total OVC served in the period Oct’10 – Sept’11 2 nd - Starting at the top of the list, calculate the cumulative measure of size 3 rd - Calculate the sampling interval 4 th - Select a random number (RS) between 1 and SI= 867 and compare the # with cumulated measure of size 5 th - Random # selected, RS = 835 Subsequent units are chosen SI = M/a Where, M - is the total cumulative measure = 2,206 a - is the # of PSU (schools) to be selected = 3 SI = 735 RS = 702 RS + SI = 1,437 RS + 2SI = 2,173 SI = M/a Where, M - is the total cumulative measure = 4,336 a - is the # of PSU (schools) to be selected = 5 SI = 867 RS = 835 RS + SI = 1,702 RS + 2SI = 2,569 RS + 3SI = 3,437 RS + 4SI = 4,304 53 | P a g e Primary Sampling Units (PSU) (Intervention Schools) # of OVC served : Measures of Size (MoS) Cumulative Selection (*) Reiketseditse Primary School 97 97 Concordia 100 197 Makeleketla Primary School 110 307 Tlongkganyeng Primary School 67 374 Boitekong P/S 158 532 Monamodi Primary School 73 605 Phomolong Intermediate School 138 743 Kweetsa Intermideate School 134 877 * Moso Public School 119 996 Leboneng Public School 81 1 077 Reseamohetse Public School 86 1 163 Ecco Primary School 191 1 354 Kgauhelo Public School 83 1 437 Dihwai Public School 278 1 715 * Tshireletso Public School 195 1 910 Hlaboloha Public School 204 2 114 Ikemisetseng Public School 53 2 167 Letlotlo Naledi 180 2 347 Tsehdisehang Primary School 133 2 480 Magakajane Public School 106 2 586 * Katoloso Primary School 82 2 668 Letsibolo Primary School 195 2 863 Mmabana Primary School 170 3 033 Naledi Ya Botjhabela Secondary school 68 3 101 Bahale S/S 134 3 235 Matshediso Intermediate School 55 3 290 Mophate School 193 3 483 * Oziel Selele Comprehensive 153 3 636 Ikgwentelle School 104 3 740 Rainbow High 25 3 765 Iphateleng Secondary School 110 3 875 Ithabeleng Public School 208 4 083 Monyakeng High School 213 4 296 * Ipopeng Secondary School 40 4 336 Schools with * are selected (c) Selection of PSU (schools) in Urban Settings 1 st list Primary Sampling Units (schools) as per the Urban stratification with corresponding approximate Measure of Size (MOS): total OVC served in the period Oct’10 – Sept’11 2 nd - Starting at the top of the list, calculate the cumulative measure of size 3 rd - Calculate the sampling interval 4 th - Select a random number (RS) between 1 and SI= 580 and compare the # with cumulated measure of size 5 th - Random # selected, RS = 410 SI = M/a Where, M - is the total cumulative measure = 1,739 a - is the # of PSU (schools) to be selected = 3 SI = 580 54 | P a g e Subsequent units are chosen Primary Sampling Units (PSU) (Intervention Schools) # of OVC served : Measures of Size (MoS) Cumulative Selection (*) Icoseng Primary School 88 88 Impucuko Public School 55 143 Itumeleng Mabelle Primary School 97 240 Malebaleba Public Primary 106 346 Marobe Primary School 113 459 * Mohobo Public School 68 527 Khotsong Public School 64 591 Lehakwe 158 749 Boase School 151 900 Dieketseng Public School 140 1 040 * Phahamisanang Primary School 145 1 185 Daluvuyo School 100 1 285 Mokgwabong Primary School 90 1 375 Setshabelo Primary school 100 1 475 Marematlou Secondary School 148 1 623 * Nanbolela Secondary School 116 1 739 Schools with * are selected Stage 2: Selection of Eligible Children from each School Eligible children were selected using HDA COS project OVC register for each selected schools. The OVC register generated from the database was used as sampling frame to undertake a random selection. RS = 410 RS + SI = 990 RS + 2SI = 1,569 55 | P a g e Appendix 5A: Absenteeism Rates: Parents Questionnaire Appendix 5B: Child Afraid to Go to School 56 | P a g e Appendix 6: Actual Services Provided to 16 sampled schools School Clinical Nutritional Support Household Economic Strengthening Child Protection General Health Care Psycho Social Care Education Support RURAL SCHOOLS Senzile 101 68 90 111 205 361 Tshomarelo 22 103 95 288 179 445 Tlamanang 107 10 37 168 133 154 Magakanjane 4 164 79 69 148 166 Makeleketla 2 61 67 77 209 157 SEMI URBAN SCHOOLS Ikemisetseng 9 13 29 62 66 43 Mophate 0 29 113 60 63 155 Oziel Selele 3 2 45 121 46 124 Raseamolong 1 7 30 66 15 103 Kweetsa 36 19 104 121 125 139 Monyakeng 39 41 54 155 67 327 URBAN SCHOOLS Diketseng 13 57 26 17 88 197 Marematlou 2 7 47 1 22 128 Dihwai 123 16 26 58 3 402 Phomolong 10 69 25 56 82 152 Morobe 7 1 23 26 31 176 57 | P a g e Appendix 7: Food Security at Home 58 | P a g e Section 9 : ATTACHMENTS List of In-depth Interviewees Name Organization Position Gender Mpho Selebalo HDA (COS) Co-ordinator F Itumeleng Pitsi HDA (COS) Co-ordinator M Vusi Shongwe HDA (COS) Provincial Co-ordinator M Adonia Simango HDA (COS) M & E Officer M Dr Saul Johnson HDA Managing Director M Nditsheni Mungoni HDA Director M Jennifer Baumann HDA M & E Manager F Myles Ritchie HDA Strategic Consultant M Addis Berhanu PACT SA MER Advisor M Angela Makgabo PACT SA Programme Manager F Limpho Mkhouane DoE Social Worker F Limpho Hlalele DoE Deputy Chief Education Specialist F TOTAL 12 5 F 7 M 59 | P a g e Number of Participating Community Stakeholders per School These findings are compiled from the data mined through qualitative means (focus groups and individual interviews) and from quantitative sources obtained from the questionnaires completed by learners and caregivers. The total number of respondents is tabulated below: No. School Name Learners Carer SBFs Community NGOs COS GOV TOTAL 1 Senzile 17 15 2 11 - 6 51 2 Tshomarelo 17 11 - 4 - 3 35 3 Tlamanang 15 12 1 10 - 2 40 4 Ikemisetseng (Bothaville) 19 25 1 - - - 45 5 Mophate Bothaville 12 14 - 3 - - 29 6 Oziel Selele (Bothaville) 16 17 - 6 - 2 41 7 Diketseng (Virginia) 12 28 - 10 - 1 51 8 Marematlou (Virginia) 16 24 4 - - - 44 9 Dihwai 15 19 1 10 1 1 49 10 Magakanjane 16 17 1 8 - - 42 11 Raseamohetse 16 16 - 3 1 1 38 12 Makeleketla (Winberg) 9 8 4 4 - - 25 13 Kweetsa 15 8 1 4 - - 28 14 Phomolong 13 8 1 3 - - 25 15 Monyakeng 16 10 4 4 1 1 36 16 Morobe (Odendalsrust) 21 21 7 10 - 1 60 TOTAL 245 253 27 90 3 18 636 60 | P a g e Types of Training Offered to COS HDA Training Team training is provided by HDA and this focused primarily on data collection, data management, reporting and general monitoring and evaluation activities. The current 2-day training sessions are said to be too short (and this was echoed in field too). More frequent (quarterly) and more extended training sessions were recommended by the COS team. It was recommended that HDA introduce additional training which focuses on implementation and quality service delivery, with the overall aim of improving service delivery to intended beneficiaries and enhancing the positive impact of the Project intervention. Further, a post-training evaluation conducted by HDA generated the following wish-list regarding training from participants (these findings were categorically corroborated during our in-field data collection activities); • The number of training sessions should increase • The length of the training sessions should increase • The number of subjects covered during training sessions should increase • Participants should receive certificates for the training sessions they attend. • Participants would like more role-play activities during training sessions PACTSA Training Respondents reported that extensive and effective training was provided on database management. However, the training which PACTSA aimed to provide, and for which extensive training material has been developed, is far broader in scope than what respondents actually received. According to the training scheduled presented to researchers by PACTSA, training courses are available in the following four areas; Project Capacity building, MERL Capacity Building, OD Capacity Building and Financial Capacity Building. Based on the in-depth-interviews conducted, it was confirmed that not all the necessary modules in these categories were conducted with the remaining COS implementers: • Project Capacity Building Support – HDA did not receive training in Technical Assistance on Household Economic Strengthening; Technical Assistance in Child Protection; Corrective action/ Project strengthening plans; Linkage to relevant tools, resources and contacts; marking/branding plans and compliance; sub agreement reviews and approval; procurement approvals; international travel approvals • MERL Capacity Building Support – HDA did not receive training in M & E Plan Revision; Tools development for data collection and collation; data verification for reporting; assistance with recruitment of M & E staff and External Data Quality Audit • OD Capacity Building Support – HDA did not receive training in any of the PACTSA courses cited on this appendix • Financial Capacity Building Support – HDA did not receive training in Review of capital equipment procurements and renovations; finance staff recruitment, selection and 61 | P a g e orientation; financial reporting and clearing of unsupported costs and Financial management for Non Financial Managers • Some participants also expressed dissatisfaction about the fact that the training was not accredited. The failure to do so resulted from financial and human capacity resource constraints. In addition to these training gaps, respondents from PACTSA and the COS team identified the following training needs which will strengthen the capacity of the current COS team: • Generic Project Management • Care and Support of Children – especially the procedures outlined in the Children’s Act concerning OVC referrals • Child and Youth Care Training • More mentoring and Coaching is needed by volunteers • How to conduct a Training and Development Needs Analysis (TDNA) – especially as it refers to building and strengthening the organisation with a large volunteer base. Much of the PACTSA training conducted to date was targeted at the three Circles of Support Coordinators and the Project Manager who resigned in December thus leaving a capacity gap. The PACTSA intention was that this team would train project staff and volunteers at school level. As will become evident from the discussion below on the reported training received in school, this skills “trickle-down” approach was not effective in transferring skills and building capacity where needed. Monitoring and Evaluation staff were in agreement that although the training had been positive and useful, that additional, expanded and consistent training sessions are required especially in this case where SBT’s and volunteers display a high turnover rate. 62 | P a g e Achievement of Output Indicators in Schools under study Service Number reached Target Set What is offered Education Support 5,736 3,500 66 Homework Support groups School materials Psychological Care 2,688 2,500 Therapy to 1100 Emotional Counseling Bereavement Marriage counseling Child Protection 5,390 3,300 Memory Work Introductory Home visits ID and birth certificates Clinical Nutritional Support 733 3,000 Nutritional Supplements Food Gardens Food Parcels Feeding Schemes Household Economic 1,191 150 Grants Household food gardens Free municipality electricity Food banks HIV Prevention 1,199 3,700 Training Workshops Primary and General Health Care 2,741 3,000 Check ups Immunization Referrals Local Clinic Care ARV’s 3 Not stated ART treatment 63 | P a g e CASE STUDY Phomolong Intermediate School is situated in the town of Henneman and depicts typical social deprivations associated with poor urban settlements within the Lejweleputswa District Municipality. Field researchers observed a high level of unemployment evidenced from the number of youth and adults loitering in the community. People in the surrounding the school looked very disillusioned and idle and conversations with children on the playground revealed that the children lacked role models within their community. There are no visible signs of facilities for social and extra mural activities except for one poorly tended soccer field. There was a discernible look of desperation in the eyes of the adult and children observed. In most poor communities, churches and schools play an important role in providing centralised support to the community but in this case some children travel great distances to get to this school which is a ramshackle and basic structure. The poverty of the community is reflected in this school that is generally dirty and unhygienic, staff members were unkempt and did not look enthusiastic; the school yard had no form of vegetation; the library is under equipped and children look generally unhappy. There are also large numbers of scholars who are too old to be attending an intermediate school (e.g. 18 year olds). The level of understanding and intellectual capacity was noted by teachers to be generally low for their grades in most age groups. The COS Project was faced with a school in deep distress however the highly effective SBF based at this school was able to coordinate children and parents/caregivers to participate in this research and this attested to the relationship she has with the community. Most importantly, because of her daily presence at this school, she was able to identify 300 pregnant teenagers who needed medical and psycho-social support. She was able to bring this distressing problem to the attention of the DoE Social Worker, which led to a Project put in place to assist these vulnerable teenagers. The DoE Social Worker indicated that the teen pregnancy problem would not have been brought to her attention because she is not able to visit schools regularly, teachers are reluctant to report this to the department (display an attitude that is it not their problem) and that these issues are generally hidden by the parent and the learner because of shame or fear of explusion. Now, these girls are assured of their continued education and have been referred to antenatal care, HIV screening and general sexual health advisory services. This SBF was also able to articulate to Social Workers the growing rate of gangsterism which is indicative of many communities rife with youth unemployment. Based on her experiences and observations, the DoE is also putting in place a peer Project to talk about the effects of gangsterism and to work with the local police to decrease violence against children in particular and the community crime rate in general. An important learning from this case study is that the SBF model is flexible enough for general societal problems to be identified and reported on even though they are not directly related to OVC support and that the budget does not accommodate for additional calls and meetings with relevant government officials. Another key learning is that if government agencies have accurate data, they are able to respond to specific challenges at school level thus widening the circle of support. Working in an integrated way with NGO’s/CBO’s is a useful method of obtaining ‘real life’ data on a regular basis and the 64 | P a g e DoE has begun to work very closely with COS in order to respond to challenges specific to the Henneman community. So the impact of this individual SBF will be felt immediately as well as in the long term. Firstly, 300 girl children will be able to continue their education hopefully until matric. Secondly, the babies born will be able to obtain birth certificates at birth, thirdly if child support grants are required, this will be fast-tracked, fourthly, the continued education of the teen mothers will provide opportunities for further study and professionalization; fifth these teen mothers will be identified for access to antenatal and post natal care, thus improving the child mortality and infant health rate as well as the health of the young mothers; sixth, 300 more young women will be screened for HIV/AIDS and counselling provided in sexual health. Finally, the risk of abortions and child abandonment will be greatly reduced through counselling and support. This particular circle of support offered to these young mothers will have a domino effect on the lives of their unborn babies who may have a better life experience than their parents have. This can only bode well for the next generation.