USAID/South Africa Umbrella Grants Management Project End of Project Partner Evaluation ANGLICAN AIDS AND HEALTHCARE TRUST (AAHT) VANA VETU PROGRAM June 2012 This publication was produced for the United States Agency for International Development. It was prepared by Wilma Wessels-Ziervogel, Adrian Di Lollo, and Dena Lomofsky of Southern Hemisphere. USAID/South Africa Umbrella Grants Management Project End of Project Partner Evaluation ANGLICAN AIDS AND HEALTHCARE TRUST (AAHT) VANA VETU PROGRAM This report is made possible by the generous support of the American people through the U. S. Agency for International Development (USAID), under the terms of APS 674-07-011, through United States Government’s President’s Emergency Plan for AIDS Relief, operated by FHI 360. The contents and opinions expressed herein do not necessarily reflect the views of USAID or the United States Government. AAHT UGM End of Project Evaluation i ACKNOWLEDGEMENTS Southern Hemisphere would like to thank the individuals and organizations that made this evaluation possible. We would like to thank FHI 360 for entrusting Southern Hemisphere with this task, and providing ongoing support during this evaluation process. We would also like to extend our thanks to AAHT, particularly the Vana Vetu Program staff at the headquarters and sample sites, for their openness to the evaluation process and commitment under tight time constraints. We express appreciation to all evaluation participants who shared their knowledge and experiences with us, and in so doing, contributed to generating knowledge about orphans and vulnerable children (OVC) programs to improve service delivery. Lastly, a special thanks to the evaluation team members, specifically Adrian Di Lollo, Victoria Tully and to the fieldwork team Noxolo Bill, Khanyisa Manzini, Helen Barnard and Limpho Klu for their dedication and hard work during this period. AAHT UGM End of Project Evaluation ii TABLE OF CONTENTS I. INTRODUCTION ...................................................................................................... 1 II. BACKGROUND ......................................................................................................... 2 III. METHODS .................................................................................................................. 4 IV. FINDINGS .................................................................................................................. 7 V. CONCLUSIONS ...................................................................................................... 29 VI. RECOMMENDATIONS .......................................................................................... 32 APPENDIX 1: PROGRAM FRAMEWORK FOR VANA VETU .................................. 36 APPENDIX 2: SCOPE OF WORK ................................................................................ 45 APPENDIX 3: SAMPLING FRAMEWORK .................................................................. 48 APPENDIX 4: PERSONS CONTACTED ..................................................................... 50 APPENDIX 5: ETHICAL GUIDELINES FOLLOWED ................................................ 52 APPENDIX 6: CONSENT FORM .................................................................................. 53 APPENDIX 7: DATA COLLECTION TOOLS ............................................................ 54 APPENDIX 8: LIST AND DATES OF SITE VISITS AND FIELDWORK ................. 98 APPENDIX 9: EVALUATION WORK PLAN .............................................................. 99 APPENDIX 10: COMPOSITION OF EVALUATION TEAM ................................... 101 APPENDIX 11: REFERENCES ..................................................................................... 102 AAHT UGM End of Project Evaluation iii ACRONYM LIST AAHT Anglican AIDS and Healthcare Trust ART Antiretroviral therapy CBO Community-based Organization CCCF Community Child Care Forums CCW Child Care Workers DOVC Diocesan Orphan and Vulnerable Children Coordinator FB False Bay FBO Faith-based Organization FG Focus Group HQ Headquarters NACCW National Association of Child Care Workers M&E Monitoring and Evaluation Mt Frere Mount Frere OVC Orphans and Vulnerable Children PE Port Elizabeth PEPFAR President’s Emergency Plan for AIDS Relief QRS Quest Research Services SAW Social Auxiliary Worker SHC Southern Hemisphere Consultancy UGM Umbrella Grants Management project USAID United States Agency for International Development VV Vana Vetu WC Western Cape AAHT UGM End of Project Evaluation iv EXECUTIVE SUMMARY INTRODUCTION UGM is a five year grants management program administered by FHI 360 with funding from the U.S. Agency for International Development (USAID). Through UGM, FHI 360 provides funding and technical assistance to USAID-selected NGO partners who provide HIV/AIDS services at local, provincial, and national levels in South Africa. FHI 360-UGM seeks to promote high quality service delivery in alignment with the priorities and goals of the South African government’s HIV/AIDS framework. FHI 360-UGM provides specialized capacity building and support services to build partners’ skills and competencies in program management, governance, human resource development, budgeting and finance, and monitoring, evaluation, and reporting. The FHI 360-UGM project objectives are to 1. Provide grants to USAID/PEPFAR partners that ensure an adequate resource flow to foster scale-up of activities 2. Implement effective monitoring, evaluation, and reporting systems to assess and document activities 3. Provide ongoing capacity building to support and enhance scale-up of activities, and sustainability of activities and partners Since 2007, FHI 360-UGM has supported thirteen South African NGOs including Anglican AIDS and Healthcare Trust (AAHT). The organization has received total funding through the UGM of R40,758,448. A summative evaluation of the Vana Vetu (VV) program was commissioned by FHI 360-UGM to evaluate AAHT’s progress toward achieving the program objectives, and to evaluate the key program outcomes and impacts related to improved health and well-being of orphans and vulnerable children (OVC). BACKGROUND The Anglican Church’s VV program for OVC uses a community-based model, and is implemented through parishes in 11 districts across four provinces in South Africa. The program develops the capacity of child care workers (CCWs) to provide direct services to OVC using a service delivery model that is center-based (usually at an Anglican parish or school) with an outreach component. The two main objectives of the program are to improve the social competence of OVC and build the capacity of communities, including CCWs and caregivers1, to respond to their own needs. In this way, VV mobilizes communities to commit to addressing the challenges of HIV/ AIDS, particularly related to children. This approach is well aligned to the National AIDS Plan to “mobilize and strengthen community-based responses for the care, support, and protection of OVC.2” METHODS Both qualitative and quantitative research approaches were used. A non-probability sampling method was used to select sites and participants; four sites (two rural and two urban) were selected from three of the four provinces where VV is based. The sample included youth between the ages of 12 and 20, primary caregivers, CCWs, community child care forums (CCCFs), team leaders (based at the sites), diocesan OVC coordinators (DOVCs), diocesan office representatives, partners, and VV program staff. A total of 89 survey interviews, 11 focus groups (FGs), and 19 semi-structured/in-depth interviews were conducted. FINDINGS                                                              1 Caregivers refer to those legally responsible for children, for example, parents, guardians, or grandparents 2 Department of Social Development National Strategy Plan, 2009-2012 AAHT UGM End of Project Evaluation v The findings are arranged according to the three main evaluation questions. I. To what extent was the organization able to meet the needs of children? The communities where VV operates are characterized by poverty; many households struggle to have their basic needs (food, clothing, and shelter) met. Other factors, such as poor service delivery and substance abuse, increase children’s vulnerability to child abuse, neglect, and exploitation. Children are also vulnerable to contracting HIV, due to the high prevalence among parents and the burden of disease in the country. These factors make it difficult for children to succeed in school, and are magnified for orphans and children in child-headed households. VV responds to these challenges by providing educational support, supervision, and care and support services for OVC, as well as facilitating access to social grants and collaborating with other service providers to respond to child protection issues. The findings highlight that VV’s educational support and provision for children’s basic needs are particularly valued across stakeholder groups, especially by children and caregivers. Gaps identified include the need to provide services to youth over 18 years and target caregivers and the community to enable more sustainable impacts. The results show that children feel positive (comfortable, safe, and assisted) in their interactions with CCWs. The VV program seeks to develop the knowledge of children regarding HIV prevention, care, and support, so that they are adequately equipped to protect themselves from HIV. The results show that 73% of children knew about prevention of HIV transmission. Children’s perceptions of safe sex also demonstrated positive results; between 49% and 79% believe that delaying sexual debut, abstinence, and consistent condom use are safer sexual choices. However, knowledge of the link between substance use and risky sexual behavior was low. While participants showed positive perceptions of others living with HIV, stigma is still prevalentchildren indicated they would be hesitant to disclose a family member’s HIV status because they would be ashamed. Urban areas show less positive results in their perceptions of safer sex, as being sexually active is related to social popularity. In addition, participants in urban areas had false perceptions of HIV treatment and misguided beliefs about curing HIV through having sex with a virgin. CCWs have shown an increased understanding of some aspects of HIV transmission and treatment, which has enabled them to fight existing misconceptions about HIV transmission that often lead to stigmatizing behaviors. This has also improved their competence to meet the needs of OVC. The VV program has enabled the development of networks to respond to OVC needs. CCCFs are established in three of the sample sites, and VV staff and CCWs play an active role in these groups . Generally, VV sites collaborate with government departments such as Education, Social Development, Health, Home Affairs, and the South African Police Services. Two of the rural sites interviewed show more widespread and strategic collaboration with other partners, such as the Traditional Council, the Department of Agriculture, and local businesses. II. What were the most significant changes brought about by the organization in improving the well-being of its beneficiaries? Children have shown that their social competence has been positively influenced by the VV program. Having basic needs met has been critical for the survival and well-being of children, especially in child￾headed households. Educational support has improved pupils’ interest in school, encouraged them to return to school, allowed them to better cope with school work, and has improved their academic performance. Changes in the self-esteem and levels of confidence of children are also evident. Children indicate feeling a sense of control over their own lives (82%) and feeling competent to help someone who is HIV positive (69%). The increased feelings of control and competency inform their improved ability to make responsible decisions and choices based on key values and their knowledge about HIV related risk factors. The findings highlight that the program successfully addresses the sense of isolation that is pervasive amongst many OVC. The children felt that VV has contributed to their sense of support, belonging, and social acceptance in many ways, including the relationships established with CCWs, who often play a parenting role, and by providing opportunities for OVC to have similar experiences as other children (such as traveling through outings or camps, and having school uniforms and afterschool care). AAHT UGM End of Project Evaluation vi Interacting with CCWs through the VV program has provided children with skills to cope with their circumstances and restored their sense of hope for the future. OVC also identified improved relationships with adults and peers as a significant change resulting from the program. The majority of participants sampled reported never having had sex, which implies that VV has been able to influence decisions regarding abstinence and delayed sexual debut. Over half of those who have had sex indicate that they used a condom, and have been tested for HIV. The program has only targeted primary caregivers directly in one of the sites included in this study. Some of the reported challenges with implementing this consistently are unrealistic planning by DOVCs and challenges with the participation of caregivers during the week (due to work commitments) and weekends (due to social commitments or substance abuse). The caregivers targeted indicate that participating in the VV program has improved their parenting and communication skills. The VV program provides an important support system for households in the community. Caregivers value the support provided, which allows them to engage in their household livelihood strategies while knowing their children are taken care of. Caregivers need to be encouraged and equipped to be responsible parents to ensure the sustainability of the outcomes achieved by VV. III. What were the key enablers and barriers in meeting project objectives? The leadership and management from the VV staff, as well as the capacity development provided to CCWs, have enabled CCWs to provide quality services to OVC. The use of CCWs also increases the acceptability of the program. VV staff has improved their supervision and coordination structures over time, including the recent introduction of two cluster coordinators into the organizational structure to provide supervisory support to DOVCs. The program now has a team of coordinators to supervise, mentor, and motivate CCWs, thereby increasing the strength of the program. Despite the fact that most of them have a grade 10 qualification, VV program staff has encountered challenges with CCWs’ educational limitations. The increased workload, monitoring and evaluation (M&E) tasks, and the reported inadequacy of stipends have caused a lack of motivation among CCWs and pose barriers to effective implementation. The ability of VV to ensure standardization of training is also impacted by the high turnover of CCWs. Some CCWs interviewed (in urban areas) indicate that they had not all undergone training, or that they had not all undergone the same training. Others who have been trained indicated that refresher training would be useful. As VV is a church-based program, buy-in from the parish clergy and parishioners is crucial. Often this is a barrier to implementation, particularly when the team leader is not Anglican. Community receptiveness is an additional challenge, particularly around caregivers’ reluctance to be involved in the program. CONCLUSION Vana Vetu’s services are clearly relevant to the communities they are serving, and the findings highlight the program’s ability to address OVCs’ needs. While direct services to children are important, further interventions are needed at a household and community level, as well as support for youth after they turn 18 years old, to enhance the sustainability of the outcomes. OVC show high levels of knowledge of HIV transmission and prevention, and have been equipped to protect themselves from HIV and make healthy sexual decisions. Lower levels of knowledge were evident regarding the perceived link between substance use and HIV, HIV vulnerability within monogamous relationships, and when having sex with someone who looks “healthy.” The results show that the VV program has contributed to a community that is more mindful of children’s rights, and that VV has developed CCWs’ capacity to better meet the needs of OVC. VV’s active participation on CCCFs and establishment of partnerships with other service providers has helped facilitate the community’s ability to meet the needs of OVC. AAHT UGM End of Project Evaluation vii It is clear that the VV program has improved the well-being of children on various levels. For children, the most significant changes have focused on educational outcomes, meeting basic needs, and improving social competence. The latter has been particularly evident in children’s’ improved self-esteem and confidence, sense of belonging and social acceptance, improved ability of OVC (especially those who have lost parents) to cope with circumstances, improved relationships with adults and peers, and ability to make positive, healthy decisions for their future. The most significant changes for caregivers are the support in their endeavor to ensure that their children are safe and protected and the provision of educational support and food. Engaging these caregivers in the program through a stronger outreach component could enhance the sustainability of the intervention outcomes. Having a team of coordinators to motivate the CCWs has been a strength of the VV program. Challenges to effective implementation of services by CCWs include the limited levels of education among CCWs, limited stipends, difficulties in collection of M&E data, uneven training received, and an ever increasing workload. Buy-in from parish leaders and parishioners is often a challenge to the successful implementation of the VV church-based model. Community receptiveness and the limited involvement of primary caregivers have been considered a barrier to further success. Sites operating in rural areas seem to be more effective than in urban areas, as children from rural areas consistently showed better knowledge, attitudes, and behaviors relating to HIV prevention, care, and support. In addition, quality of service provision is rated higher in rural areas, and rural sites show more initiative in the development of strategic partnerships. RECOMMENDATIONS Key recommendations are highlighted here; a more extensive list of recommendations is provided in the main report. Intervention Design The recommendations identified below address both the strengths and gaps in VV’s provision of holistic and relevant services that meet the needs of children.  The provision of food is currently not being supported through USAID/PEPFAR funding, and the provision of uniforms was supported until 2010. Sites sampled have, however, provided food and uniforms to some degree. Children’s participation in the program and in schooling is often hindered by the lack of food and uniforms. The South African Government’s National Guidelines on School Uniform encourages the wearing of school uniforms and the reduction in costs of school uniforms, but does not make it compulsory. While this removes a barrier to access schooling, it differentiates those children without uniforms and contributes to feelings of marginalization. It is recommended that food and clothes (specifically school uniforms) be provided by program sites as this is important for overcoming some of the barriers to participation; school uniforms also emerged as a key factor for increasing the feelings of social acceptance by OVC, which contributes to increased social competence.  The supervision and caregiving role of CCWs has been crucial in creating a sense of belonging for OVC and enhancing their ability to cope with challenges; this is a strength of the VV program and should be continued. The proactive outreach approach used is a critical enabler, as it improves access to OVC who are not in the school or religious system and should be encouraged in other center-based OVC programs.  While it is critical that children are directly targeted, it is also important that primary caregivers are engaged by the program to ensure sustainable outcomes. More intensive interventions are needed at a household level, which could be facilitated through additional community awareness raising activities, increased support groups for caregivers, and mentoring support from CCWs to caregivers at a household level. This would increase CCWs’ workload and the implications of this increase should be considered; the current ratio of CCWs to household/child (one CCW to 10 households, which can mean working with up to 40 children) is already felt to be too high. While the Children’s Act does not stipulate norms and standards for the ratio of CCW- to-child for outreach services, these could be explored with other organizations doing similar work. AAHT UGM End of Project Evaluation viii  Children’s perceived vulnerability to HIV infection through “risky” behaviors and situations is low. The HIV education curriculum for OVC needs to focus greater attention on topics such as substance use and its relationship to risky sexual behavior, HIV vulnerability within “monogamous” relationships, the dangers of concurrent sexual relationships, and dating when HIV positive.  Currently children exit the VV program once they turn 18 years old. An exit strategy should be developed for children after this point. The VV program could extend their services through providing support with career guidance and applications for further education or jobs. The VV program could continue to absorb children as CCWs, as is already being done, and consider extending the peer education program with a progression into the CCW position for outstanding peer educators. Partnerships and Networking Networks and relationship building are important for creating a community that is prepared to address challenges faced by OVC. VV can improve on their successful networks through the following recommendations.  Partnering with child protection service providers is important to address the vulnerability of OVC. It is recommended that inter-sectoral structures (such as CCCF) and child protection service providers (such as local NGOs, Departments of Social Development, Health, Home Affairs, and the South African Police Services) be utilized to facilitate access to services. This has been achieved by the VV program, and should continue to be strengthened through ongoing communication and collaboration.  Partnerships can be strengthened through building personal relationships within organizations like the Department of Social Development, to facilitate easier access. Program Model  The findings show that the VV program has a strong leadership and management team (including managers at headquarters, DOVCs, and team leaders at the parish level). Leadership has provided support with implementation and motivational challenges. The VV program leaders should continue to provide this support (or improve this where it is still a challenge) through - gaining an understanding of the community challenges through periodic field visits with CCWs - communicating their appreciation of the CCWs’ commitment to serving OVCs - maintain motivation by facilitating periodic review sessions to reflect and reconnect CCWs with their purpose - having weekly progress reporting, feedback, reflection, and planning meetings - having two-way, open communication  Buy-in at the church and parish level, from clergy and parishioners, has been a challenge under this model. The VV program should ensure that buy-in is facilitated at the diocesan and parish levels. Sensitization workshops can be conducted with the clergy to educate them on the challenges faced by OVC. When initiating discussions with parish leaders, a more consultative and participatory approach should be used to identify how the VV program can best respond to the needs of the church.  CCWs have found data collection for M&E to be a challenge, mostly due to their educational limitations and workload. It is recommended that data collection for M&E at the CCW level be as basic as possible. M&E support should continue through training initiatives to ensure hands-on data collection support is provided for CCWs. This could be provided by DOVC assistants who have a Grade 12 qualification and administrative and computer skills.  Stipends for CCWs are perceived as being inadequate, and leave CCWs unmotivated. Program sustainability and quality will be enhanced if CCWs are employed, and it is recommended that AAHT raise sufficient funding to employ CCWs.  A challenge in the VV program has been the CCWs’ basic levels of education, especially given their role in providing educational support to children. It is recommended that programs reliant on CCWs or similar volunteers be realistic about the expectations regarding educational support and AAHT UGM End of Project Evaluation ix provide continuous capacity development (through training and mentoring) to assist CCWs to provide educational support. Capacity development should take the form of training and follow-up mentoring support. The possibility of educators (at partnering schools) playing this mentoring role should be explored by VV.  Despite extensive training plans, training has not been consistently implemented for all CCWs. It is recommended that all CCWs be trained in the same core skills. AAHT UGM End of Project Evaluation 1 I. INTRODUCTION USAID/PEPFAR has funded AAHT’s Vana Vetu program since October 2007; FHI 360-UGM has administered this funding and provided technical assistance and capacity development to AAHT. In March 2012, USAID commissioned a summative evaluation of the FHI 360-UGM partners’ programs. FHI 360-UGM contracted Southern Hemisphere to conduct this evaluation for AAHT’s VV program. The evaluation was conducted from March through June 2012 in the three provinces where VV is implementing, and employed a mixed-method participatory evaluation methodology. The results of this evaluation will inform USAID’s future programming for OVC within South Africa. The purpose of this summative evaluation is to  determine whether the program objectives under each partner’s program were achieved  evaluate the key program outcomes and impacts related to improved health and well-being of the targeted beneficiaries The key evaluation questions and sub-questions are 1. To what extent has the VV program been able to meet the needs of its target populations?  Is the VV program meeting the needs of children and the community as they perceive them?  What are the most valuable contributions made by the VV program?  What are the strengths and challenges of the interventions used by VV to increase child well-being?  How do stakeholders perceive the VV program, in terms of quality of service provision and ease of access?  What are the immediate and intermediate outcomes in knowledge and attitudes regarding HIV prevention and child protection (intended and unintended outcomes) for the target groups and beneficiaries (CCWs, community partners, and children)?  To what extent has the VV program built a community that is competent to address issues faced by those affected by HIV and child neglect, abuse, and exploitation? 2. What are the most significant changes (intended and unintended) brought about by the VV program in improving the well-being of children on the following levels  changes in self-esteem, self-image, levels of confidence, communication skills, goal setting, ability to deal with challenges, and positive/healthy decision making  changes in academic progress and interest in school  changes in relationships and social security of the children’s households 3. What are the key enablers and barriers in meeting the project objectives?  What are the strengths and challenges of the model used by AAHT in terms of community receptiveness, capacity to implement, management and leadership of the program, and communication structure?  To what extent have the partnerships developed enabled or hindered service delivery?  Will CCWs, CCCFs, churches, and parishes be able to continue supporting OVC if VV no longer provided them with support? What other structures exist to enable continuity? The report contains a brief description of the VV program and the methodology used in this evaluation. The key findings are summarized according to the evaluation questions described above, and are followed by conclusions and recommendations for the implementation of future OVC programs. AAHT UGM End of Project Evaluation 2 II. BACKGROUND AAHT is a church-based organization that aims to be a holistic and engaged AIDS ministry accessible to all in Southern Africa. Their mission is to embody the unconditional love of Christ through their HIV/ AIDS programs. AAHT achieves this by providing holistic care and prevention services; promoting responsible sexual behavior; providing hope, acceptance, and love to vulnerable groups; and implementing collaborative interventions that are ecumenical, multi-faith, multisectoral and non￾sectarian. AAHT established the Vana Vetu (“our children”) program in October 2007, in response to the critical need to assist the growing numbers of children left orphaned or vulnerable due to the effects of the HIV/ AIDS pandemic in South Africa. The program is solely funded by USAID/PEPFAR, with a total project budget of R40,758,448 over a five year period. AAHT conducted a needs assessment in 2005 in targeted dioceses to determine the support they required to strengthen services to OVC (and CCWs). AAHT concluded that OVC services implemented by parishes were unstructured and uncoordinated, and that training for CCWs was needed to strengthen support to OVC. AAHT developed the VV program, a comprehensive approach built on the model of the Anglican Church of South Africa’s Isiseko Sokomeleza (Building a Foundation) program. Isiseko Sokomeleza was successfully piloted in three Eastern Cape dioceses in 2004 and 2005, in partnership with Heartbeat Centre for Community Development, the Barnabas Trust, and the Mother’s Union. Initially VV operated in the Southern Cape (part of the Western Cape Region); however due to losing volunteers (who did not want the administrative responsibilities of the program), unsupportive clergy, and negative dynamics between team members, sites closed down in this region. The VV program now operates in 10 urban and rural districts and sites/parishes in four provinces: Eastern Cape, Kwazulu –Natal, Western Cape, and Limpopo. The districts are geographically diverse but all have areas of deep poverty and high unemployment. Beneficiaries live in both informal settlements and government-subsidized houses in formal areas. AAHT works through the church networks to support OVC. The church is accessible to communities, and is well placed as a hub from which to roll out VV services. The program is managed from the AAHT head office in Cape Town; this office coordinates the activities of the six dioceses, each of which comprises a number of parishes/sites (3 to 4 parishes per diocese and 22 parishes in total). Two cluster coordinators play a critical bridging role between the VV program home office and DOVCs. DOVCs are full time employees responsible for supporting a number of team leaders (based at the parish level) and CCWs within a diocese. DOVCs are chosen through a thorough selection process facilitated by the diocese and VV home office. They are required to have a Grade 12 qualification or relevant tertiary qualification, driver’s license, report writing skills, communication skills, and time management skills. One team leader per site, selected from the parish or community, provides oversight to 102 trained CCWs and 2 Social Auxiliary Workers (SAW). AAHT’s model encourages community participation and supports traditional community life; as such, CCWs are recruited from local communities and are trained and mentored through the VV program to provide care and support for OVC, in collaboration with other community service providers. Training provided to CCWs includes  HIV education through an unaccredited training in HIV prevention, care, and support conducted by Siyafundisa. Topics covered include HIV prevention through abstinence, delaying sexual debut, being faithful and condom use, positive decision making, respect, peer pressure and gender equality, and stigma and HIV.  Child rights and protection through an accredited National Association of Child Care Workers (NACCW) training conducted by NACCW, for one week each month. Topics covered include the UN Convention and Children’s Charter, the Children’s Act, human rights, children’s rights AAHT UGM End of Project Evaluation 3 and responsibilities, code of ethics when working with children, working with households using a strengths-based approach, working holistically with children, dimensions of child development, considering the context of the child in interventions, and conducting observations and assessments.  Unaccredited trainings on working with OVC and community-based organizational capacity building. This is implemented over four to five days by Barnabas Trust and VV staff.  Parenting skills unaccredited training conducted over four days by Jonah Training and VV staff. This training covers topics such as communicating with children; healthy family relationships; emotional and safety needs of children; the role of grandparents, mothers, fathers, and siblings in the life of a child; health and nutrition; and grief and bereavement.  Empowering women leaders in the community, an accredited two week training course conducted by the University of Stellenbosch.  Accredited Tuberculosis Training conducted over five days by USAID and Pact. The team leaders and the CCWs are volunteers and receive stipends (CCWs receive ZAR 700 and team leaders receive ZAR 1,000 per month for approximately 20 hours of work per week). The target group of the VV program are OVC (girls and boys ages 0-18 years), families affected by HIV/ AIDS, women, primary caregivers, community and religious leaders, teachers, community-based organizations (CBOs), faith-based organizations (FBOs), and NGOs. Parishes and schools often serve as centers from which the VV program operates although the level of participation of parishes and schools in the implementation of the program varies. CCWs provide direct services to OVC through a home-based and center-based service delivery model, as illustrated in Figure 1 below. AAHT UGM End of Project Evaluation 1 Figure 1: Vana Vetu Community-Based Model VV works to support OVC by meeting basic and immediate needs while simultaneously building capacity in families, leaders, and communities to develop sustainable solutions to needs identified by children and their primary caregivers. They develop networks within the community and among service providers to build partnerships and ensure optimal availability, distribution, and coordination of OVC services across communities. The VV program develops the capacity of communities to respond to their own needs and mobilizes them to commit to addressing the challenges of HIV/AIDS, particularly as they affect children. The community-based approach used by VV is well aligned to the National AIDS Plan strategy to “mobilize and strengthen community-based responses for the care, support and protection of OVC.”3 The table below shows how VV services are strategically aligned with the National Strategic Plan (NSP) of the Department of Social Development (DSD) (2009).                                                              3 Department of Social Development National Strategic Plan, 2009-2012 AAHT UGM End of Project Evaluation 1 Table 1: VV strategies and activities to address DSD NSP DSD NSP VV activities include Strengthening and support the capacity of families to protect and care for OVC  Parenting support groups in the community  Home visits conducted by CCWs which allow them to monitor and intervene at this level  CCWs assist with parenting responsibilities (such as homework support) with which parents often need assistance Mobilize and strengthen community based responses for the care, support, and protection of OVC  Capacity building of CCWs (for example, 27 have undergone NACCW training and 7 have undergone Community AIDS Response training) to care for OVC  Participating in 11 CCCFs  Partnering with government, PEPFAR provincial liaisons, and NGOs to provide services and protection to OVC  Strengthening Anglican parishes based in communities to provide services to OVC Ensure access of OVC to essential services  Partnering with and referrals to government departments (such as Home Affairs, DSD, Health, access to education, etc.) to provide basic services and rights (including food, social security grants, identity documents, birth certificates) to OVC  An outreach component that allows for VV to identify out of school children  CCWs that live in the community are well placed to identify OVC that need essential services Raise awareness and advocate for the creation of a supportive environment for OVC  HIV education and support provided to OVC and CCWs  Awareness raising on child rights and child protection in communities Vena Vatu’s vision is that “0rphaned and vulnerable children receive appropriate support to grow and develop to their full potential.” The overall goal of the program is to contribute to improving the lives of OVC and their families so that they can become independent young adults, who are able to cope with their emotional, physical, and economic needs, and be part of a functional, healthy, and safe community. AAHT has two strategies—the children’s strategy focuses on providing direct services to OVC, and the community strategy focuses on developing the capacity of the community to care for OVC. The objective of the children’s strategy is to provide for the basic needs of OVC under the age of 18; OVC need a range of support services to meet their basic needs to enable them to complete high school. The short term outcome for OVC is improved knowledge of HIV prevention, care, and support, as well as child rights and protection. Access to basic services and the improved life skills and knowledge of children will lead to improved social competence4 , a key long term outcome of the VV program. There are four main components of the children’s strategy: HIV prevention and life skills program; child protection with a focus on social security and awareness raising for child abuse, exploitation, and neglect; educational support to facilitate the progression of OVC through school; and health and psychosocial support.                                                              4Social competence is defined in the VV Program Framework (September 2011) as: to be able to form positive relationships with adults and peers; to have emotional coping skills or resilience; to have positive self-esteem and good self￾confidence to be able to solve problems and make responsible decisions about their health and safety. AAHT UGM End of Project Evaluation 2 The objectives of the community strategy are to build the capacity of CCWs in communities affected by HIV/ AIDS to ensure Vana Vetu’s long-term sustainability, and to develop network linkages with CBOs to raise awareness of the situation among OVC and ensure that all children are safe and protected. The VV program is reliant on these partnerships with community service providers to ensure that children’s basic needs are met. The short-term outcomes for the community strategy are increased knowledge of and improved attitudes concerning HIV/AIDS prevention, care, and support, the rights of children, and protection of OVC. The expected long-term outcome of the community strategy is strengthened community care for OVC, ensuring that children are raised in a safe and protective environment. To achieve this, VV strengthens families and primary caregivers through parenting skills programs, home visits, and psychological support; increasing community capacity to care for OVC through building the capacity of CCWs and SAWs; and participating in and strengthening CCCFs. The final output entails conducting local campaigns to increase community awareness of HIV/AIDS prevention, stigma reduction, protection and care for children, and upholding children’s rights. These two strategies are illustrated in Table 2 below. Table 2: Summarized Program Framework for Vana Vetu5 Outcomes Objectives Activities Children’s Strategy The children participating in the Vana Vetu program have improved their knowledge of and attitudes towards HIV/ AIDS prevention, care ,and support; life skills; education; relationships with peers and adults; knowledge of children’s rights and of how to protect themselves from HIV and sexual abuse. Provide for basic needs of OVC (under 18 years) in 4 provinces (KZN, EC, WC, Limpopo) Educational support (afterschool and homework support 1-4 times per week), Psychological care (memory work and journals once per week, one-on-one support), Child protection discussions/activities, Health referral, HIV prevention education and life skills (10 hours), Household economic strengthening (through facilitating access to grants) The children participating in the Vana Vetu program have improved their social competence and are role modeling this behavior to their peers in the community. Community Strategy The communities in which Vana Vetu works, including the caregivers of OVC, are more knowledgeable of and have a more positive attitude towards HIV/ AIDS prevention, care, and support; and child rights and protection, with a focus on OVC. Build the capacity of communities affected by HIV/AIDS to develop sustainable solutions to identified OVC needs Training for CCWs including accredited NACCW training (one week per month, including grief and bereavement, psychosocial training); HIV prevention, care, and support training (3-4 days); Care for caregivers (3-4 days); Parenting skills (5 days); Accredited empowering women leaders in the community training (2 weeks); Accredited TB training (5 days); OVC training (4-5 days); CBO training (4-5 days) VV also conducts parenting support groups/interventions for parents and joint community awareness raising initiatives                                                              5 The table combines the objectives specified in the COP10 (see Objectives column) and the outcomes Program Framework developed in September 2011 (see outcomes column). Activities are also a combination of those specified in COP10 and the Program framework. AAHT UGM End of Project Evaluation 3 Strengthened community care for and support of OVC as service provision by respective stakeholders (Government Departments, NGOs, CBOs, and FBOs) is delivered in a more integrated manner. Develop network linkages within the community and other service providers Monthly stakeholder meetings, Partnership agreements, CCCFs established AAHT UGM End of Project Evaluation 4 III. METHODS A mixed method approach, combining qualitative and quantitative methodologies, was employed for this evaluation. The ‘most significant change’ methodology was used as an interviewing technique to understand the outcomes and impact for OVC as a result of the VV Program. IMPLEMENTATION PROCESS The process diagram below outlines key steps in the implementation of this evaluation. Figure 2: Evaluation Process Diagram An initiation workshop was facilitated by FHI 360-UGM for commissioned evaluation consultants to discuss the overall evaluation objectives, specific research questions, methodology, and logistics. This informed the development of an evaluation design and implementation strategy. Key VV program documents were reviewed and a total of 11 research instruments were designed, after which an evaluation planning meeting was conducted with the VV program staff to discuss the evaluation process. Fieldwork was conducted from May 2-11, 20126, and data was coded using NVIVO 9 for qualitative data analysis and SPSS for quantitative data analysis. A presentation of the preliminary findings was conducted for USAID, FHI 360-UGM, and AAHT staff; feedback on this presentation guided the report writing process. DATA COLLECTION METHODS Data was collected using key informant interviews, semi-structured interviews, in-depth interviews, structured self-completion surveys and focus group discussions (see Appendix 7: data collection Tools for instruments used). SAMPLING FRAME A non-probability sampling method was used for the selection of sites and participants for this evaluation. As a result, the statistical findings from this evaluation cannot, strictly speaking, be generalized to all VV program sites and beneficiaries. This sampling method nonetheless allowed for key sites to be selected based on their ability to best fulfill the information needs of evaluators, and for efficient data collection under time constraints.                                                              6Specific fieldwork dates are provided in Appendix 8: list and dates of site visits and fieldwork. AAHT UGM End of Project Evaluation 5 Sites where the VV program has been in existence for at least four years7 were selected for the sample. As illustrated in the table below, sites were selected to ensure coverage of three of the four provinces where VV operates to ensure different levels of performance (for example, in rural and urban areas) could be ascertained. Table 3: Selection of Sites Province Sites Period of existence Area Level of performance Rural Urban Good Average Poor Eastern Cape Port Elizabeth (PE) 6 years   Umzimvubu (UM) 6 years   Western Cape False Bay (FB) 4 years   Limpopo Sekhukhune (SK) 4 years   Totals 4 4-6 years 2 2 2 1 1 In total, 19 semi-structured/in-depth interviews, 89 survey interviews, and 11 focus groups were conducted. The detailed sampling frame can be found in AppendiX 3: SAMPLING framework. A summarized version is presented below: Table 4: Overview of Sampled Participants Method Interviewee Number One on one semi-structured interviews CEO and VV Program Director 2 Diocesan Office: 1 Bishop and 2 HIV Coordinators 3 DOVCs (one at each site) 4 Team Leaders (one at each site) 4 Partners (School representatives) 2 Focus groups VV staff at headquarters (3 participants) 1 CCWs and SAWs (between 5 – 10 participants) 2 CCCFs (between 5 – 10 participants) 2 Caregivers (between 5 – 10 participants) 3 Children in 3 sites (8-10 participants) between 12 – 18 years old 3 One on one in￾depth interviews Children: 3 Females and 1 male (One per site) between 16 – 18 years 4 Self-completion survey OVC: 60% female and 40% male; 45% from rural areas and 55% from urban areas; between 12 – 20 years old 89 The ethical standards adhered to for this evaluation can be found in appendix 5: ethical guidelines followed, and the consent form used in Appendix 6: consent form. LIMITATIONS The data limitations are  Due to the logistical challenges, the sample size of OVC in rural areas was smaller than in urban areas. Three of the four intended focus groups and 89 out of the intended 100 survey interviews with children were conducted.                                                              7The program leaders suggested that four years would be a long enough time frame to realize the intended outcomes. AAHT UGM End of Project Evaluation 6  Non-attendance (particularly in rural areas) of children and of one partner organization meant that substitute interviews had to be conducted. Substitute interviewees were not always the best match for the sampling criteria (see AppendiX 3: SAMPLING framework), which limited the depth of information collected.  Focus groups conducted with one CCCF had more CCWs present than CCCF members. This limited the data provided on the functioning of CCCFs within VV communities.  One focus group with caregivers (in a well performing rural site) failed to differentiate between the VV program and other OVC related programs being run from the same parish. In this instance, it was a challenge to attribute reported outcomes to the VV program in particular.  No baseline data for the survey are available, making longitudinal and attribution analysis impossible.  9% of children sampled for the survey reported that they did not receive assistance through the VV program8. However, when asked if participants were involved in the VV program, all had indicated involvement in the program at some point. This discrepancy in the data could be due to interviewee fatigue, as the question on the type of assistance provided by the CCW was close to the end of the questionnaire.  Time constraints for this evaluation process limited evaluators’ ability to be flexible and remedy some of the above mentioned limitations.                                                              8Questions asked included: Please tick how CCWs have helped you: answer options included: A CCW did not help me; A CCW visited my home; The CCW helped me to get a grant (e.g. child support grant); The CCW helped me to get a birth certificate/ ID; I attended a group (e.g. support group, memory box) run by the CCW; The CCW helped me with my school work; The CCW told me where to go for help with a problem; I spoke to the CCW about a problem I was having; The CCW helped me to get food/clothing/school uniform AAHT UGM End of Project Evaluation 7 IV. FINDINGS TO WHAT EXTENT WAS THE ORGANIZATION ABLE TO MEET THE NEEDS OF CHILDREN? This question assessed whether the Vana Vetu Program met the needs of beneficiaries through the services delivered, the quality of service provision, and the immediate/intermediate outcomes achieved in light of the key challenges identified in the community. Identified needs in the Vana Vetu Communities The table below summarizes the key problems in the communities as identified by the respondents. Table 5: Community problems identified by VV stakeholders (N9 = 25) Community problems Frequency of mention (total) Frequency of mention by children (n=7) Frequency of mention by caregivers (n=3) Frequency of mention by CCWs & staff (n=15) Educational needs (uniforms, bags, stationery, etc.) 14 7 (100%) 2 (67%) 4 (27%) Poverty, unmet basic needs 12 4 (57%) 2 (67%) 6 (40%) Child abuse 9 5 (71%) 2 (67%) 2 (13%) Children feeling not loved/cared for 8 3 (43%) 3 (100%) 2 (13%) Limited supervision at home 7 2 (29%) 3 (100%) 2 (13%) Substance abuse 6 2 (29%) 0 4 (27%) Children orphaned due to AIDS 2 1 (14%) 0 1 (7%) Difficulty accessing social grants 2 0 1 (33%) 1 (7%) Service providers not working together 1 0 0 1 (7%) Source: Qualitative interviews and focus groups with children, caregivers, staff, and CCWs The evaluation revealed a wide range of problems faced by OVC in the communities served by Vana Vetu. A number of these challenges are related to socioeconomic conditions and barriers to accessing to state-provided services, indicating that the services of the Vana Vetu program are correctly targeted. The key issues raised by respondents, including children, caregivers, CCWs, CCCFs, and staff, are briefly discussed below. Children’s access to school and motivation to continue with their education is inhibited by the affordability of school uniforms, school shoes, and other school equipment (such as school bags and stationery). This was recognized as a key problem by all children and most caregivers. Caregivers and children both identify having limited homework supervision as a factor that contributes to low levels of interest in school, and ultimately to poor performance. Many respondents across all stakeholder groups identified conditions of extreme poverty and high unemployment which leave OVC in a particularly vulnerable position. A few primary caregivers and staff indicated that while access to social grants is an important source of poverty alleviation,                                                              9 Number of respondents who answered the question AAHT UGM End of Project Evaluation 8 children and their families are often unable to access social grants because they lack the necessary documentation (including birth certificates, identity documents, and death certificates). Many children have been victims of physical or sexual abuse and do not have access to the psychosocial support to cope with this, frequently due to limited resources within the community for psychosocial support, and also leading to poor school performance. Some children feel unloved and uncared for due to abuse and neglect. Caregivers also expressed concern about their ability to show affection for their children, indicating that they are often overwhelmed with parenting responsibilities. Substance abuse (both drugs and alcohol) was identified by some children, staff, and CCWs as a problem. Children (particularly girls) tend to engage in unsafe sex when intoxicated and teenage pregnancies are common. Exposure to HIV/AIDS is a clear danger for such children. Few respondents (children and staff) indicated that child-headed households were a particular concern. Child-headed households are common and children in such households are particularly in need of both financial and psychosocial support. School drop-out rates are reportedly higher in such families. Vana Vetu’s response to the identified problems VV takes a holistic approach; this is both a strength of the program and a unique factor that distinguishes it from other services. Rather than simply intervening in one aspect of the lives of OVC, VV’s interventions target home, school, and community life— which all interlink to improve the life of the child. VV is prevention-oriented, seeking to protect OVC from the dangers and risks they face. The VV program offers services to meet the needs outlined in a variety of ways, including  Afterschool care services provided to OVC, where homework supervision takes place one to four times per week. Uniforms and school supplies were provided by the program between 2008 and 2010.  Assistance to OVC in acquiring birth certificates and identity documents in order to obtain social grants; this has provided households with some basic income to alleviate poverty, and is done through referrals to relevant departments. CCWs also refer OVC to public health care clinics for antiretroviral therapy (ART) and other health conditions. Although direct food aid is not provided through the VV program, when access to food is a problem, the CCWs have made referrals to appropriate agencies for food parcels. In rural areas, CCWs have encouraged and assisted households to undertake basic farming to supplement income as a poverty alleviation measure10.  Work with CCCFs, police, and health care services (among other stakeholders) to identify and respond to cases of child abuse. The VV program provides child protection education once a week through small group sessions (discussion, debates, role plays, and presentations by service providers). This helps children to understand their rights and support structures, and helps them to protect themselves.  Bereavement counseling and memory work provided by CCWs to OVC (particularly child headed households) once a week to help them deal with loss. Referrals are made to social workers if CCWs assess that children or families are in need of professional psychological support.  CCWs playing a parenting and supervisory role for children in child-headed households or in cases in which parenting support is inadequate. Through interventions and home visits, they may                                                              10 These aspects of food security were not evaluated. AAHT UGM End of Project Evaluation 9 form close bonds with children, making them feel loved and cared for. Recreational services (including camps and outings) also provide a sense of “special” treatment.  CCWs developing the capacity of caregivers to care for their children. This is conducted through targeting caregivers of OVC beneficiaries through parenting support groups and/or one￾on-one support provided during household visits by CCWs.  Each OVC participating in ten hours of HIV education workshop covering topics such as HIV prevention through abstinence, being faithful and condom use, positive decision making, respect, peer pressure, and gender equality. These take place at schools or church, and the training program is based on the Center for Disease Control and Prevention (CDC)’s Global AIDS Program South Africa and the Rutanang Peer Education and Life Skills program.  OVC participating in recreational activities such as outings to places of interest (museums, aquariums, zoo, planetariums), drama, and dance provided through the program. Most valued services Stakeholders were asked what they most valued about the services that VV offers. Educational support services were the most mentioned indicating that Vana Vetu is having a significant impact in terms of mitigating the factors (including extreme poverty and lack of parental support) which would prevent a child from remaining and performing in school. This view was strongly held by children and caregivers. Child protection services and recreational opportunities were also valued. The capacity building of CCWs was also appreciated by CCWs themselves. Specific mention was made of HIV education, child protection protocols and policies, life skills training, training in “Empowering Women Leaders in the Community,” gaining access to households and acceptance in the community, and networking. Table 6: Services most valued by VV stakeholders (N=25) Service Most valued (total) Children (N=7) Caregiver (N=3) Staff & CCWs (N=15) Educational/afterschool support 14 5 (71%) 3 (100%) 6 (40%) Child protection services 10 2 (29%) 2 (67%) 6 (40%) Recreation 6 3 (43%) 2 (67%) 1 (7%) Capacity building in the community 5 0 2 (67%) 3 (20%) HIV & AIDS education 5 3 (43%) 0 2 (13%) Services provided through home visits (assisting with cooking, household chores, assisting children when they are ill) 4 1 (14%) 1 (33%) 2 (13%) Child protection education 1 0 0 1 (7%) Source: Qualitative interviews and focus groups with children, caregivers, staff, and CCWs. Gaps in services The following are some of the gaps in services to OVC.  The limited access to qualified mental health professionals means that children, particularly those dealing with grief and bereavement, do not have sufficient access to psychosocial support. This was, however, only mentioned by program staff, and should be further explored.  Due to budgetary constraints, food is not provided in all afterschool programs. It was argued that attendance is highest when food is provided and that more children would attend if food was provided at all afterschool programs.  Children cease to qualify for services at 18 years of age11, despite their ongoing vulnerability. Accessibility and quality of service provision in relation to community needs                                                              11 According to the South African Children’s Act, people between the age of 0 – 18 are classified as children. AAHT UGM End of Project Evaluation 10 This section of the study sought to determine how stakeholders (children, caregivers, CCWs, and the Anglican Church) perceive the program in terms of access and quality. Access was assessed based on the equity and reach of the program; quality was assessed by exploring issues related to the child friendliness of the services. Accessibility: Reach of Vana Vetu services As a result of USAID/PEPFAR funding, the VV program has reached 42,222 OVC (an increase from 18,805 before October 2007). The table below provides a breakdown of OVC reached with specific VV services. The total targeted number of OVC was not reached due to dioceses closing down and a change in the definitions of indicators in 200912. Table 7: Number of OVC reached through VV services Service Target Actual Child protection interventions Not provided 13,666 General healthcare referrals Not provided 3,819 Healthcare support for access to anti￾retroviral therapy (ART) Not provided 985 HIV prevention education Not provided 11,811 Educational support services Not provided 35,010 Psychological care services Not provided 11,214 Total number of OVC reached 67,000 42,222 Source: Indicator Protocol Reference Sheets, November 2011 VV has both a center-based and an outreach component. The programs usually take place at venues close to where the children live within the community (either the parish, school, or other community facility). In most instances, venues are made easily accessible through Anglican churches and schools, but arrangements are also made for the CCWs to meet with children closer to where they live. Use of parish and school venues is free of charge. The proactive VV outreach approach is critical to ensure access to services. The home visitation service helps extend the reach of VV to vulnerable target groups (children and families) who may not be connected to institutions such as the government, church, or school. One caregiver in an urban focus group remarked: “Vana Vetu is doing good work. We did not yet approach them with a problem. They always come to us.” There is no evidence of HIV stigmatization attached to using VV services. Quality: Child friendly services Overall, the quantitative data shows positive responses from the vast majority of beneficiaries about their interactions with CCWs. There was generally more satisfaction among children in rural areas than in urban areas. This may be attributed to the fact that rural sites appear to be functioning better than urban sites (in terms of partnerships developed, capacity development of CCWs, parish support, and organizational dynamics). The difference between sites is expanded upon in later sections.                                                              12 Shelter, food, and psychosocial services were no longer counted which affected the target. HIV services after 2009 required 10 hours/sessions, which was more than previously required. AAHT UGM End of Project Evaluation 11 Table 8: Children’s perceptions of the quality of services provided by CCWs (N=89) Statement Agree Disagree Unsure/ No Answer Discrepancies between subgroups The CCW was easy to talk to 72%% 15% 13% 78% of males agreed while 68% of females agreed. 93% of those from rural areas agreed, while 55% from urban areas agreed. The CCW made me feel safe 80% 10% 10% 90% of participants from rural areas agreed, compared to 71% in urban areas. The CCW was helpful 79% 8% 14% Those from rural areas (88%) agreed more with this statement than those in urban areas (71%). I would tell my friends to go to the CCW if they needed help 84% 5% 11% Rural respondents agreed more (95%) than urban respondents (76%). Source: Quantitative survey with 89 children The qualitative interviews with children support the quantitative findings. Children also note that their opinions are respected and that all children are treated equally. Very few criticisms about the quality of services were evident in qualitative interviews. The only concerns included  The turnover of CCWs hindered children from being able to connect with new CCWs.  CCWs sometimes did not deliver on promises made (particularly for outings) due to poor planning by sites or lack of funding available for these activities. These criticisms allude to some of the key barriers to program implementation, including CCW turnover due to inadequacy of stipends and limited financial support for the program, leading to the decrease in certain services. It can be concluded that Vana Vetu’s services have been designed and operate in a child-friendly way. There is a clear sense in both the home-based and center-based programs that children feel respected, supported, safe, and valued. Knowledge and attitudes of children towards HIV prevention, care, and support, and child protection The intended outcome of this aspect of the program is that, “the children…have improved their knowledge of and attitudes towards HIV/AIDS prevention, care, and support; life skills; education13; children’s rights; and know how to protect themselves from vulnerability”14. These outcomes were evaluated through self-completed survey interviews. Knowledge of HIV prevention, testing, and treatment Children were asked about their knowledge of HIV transmission, and responses are shown in the graph below.                                                              13 Education is dealt with in the section on Most Significant Change. 14 VV Program Framework. September 2012 “I like how they treat all the children as equally important.” -Urban Female during in-depth interview “They treat us well with love and they always give us hope and courage.” -Urban Male Focus Group AAHT UGM End of Project Evaluation 12 Figure 3: Knowledge of HIV Transmission (N=89) 90 78 88 91 84 84 84 76 73 6 17 9 7 13 13 11 20 20 5 6 3 2 2 2 5 3 7 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Sex without condom Mother to child Sharing toilet Blood transfusion Sharing drug needles Kissing with bleeding mouth Sharing knives & forks Touching Sharing razor blade Correct Incorrect No response Source: Quantitative survey with 89 children Children show high levels of knowledge of HIV transmission through blood transfusion (91%), having sex without a condom (90%), sharing drug needles (84%), and “kissing when someone’s mouth is bleeding” (84%). Children furthermore understood that HIV could not be transmitted through sharing a toilet (88%) or sharing cutlery (84%). While results are still generally positive, children had slightly lower levels of knowledge about the following means of transmission  mother to child during birth and breastfeeding (78% knew this was possible)  touching (76% knew this was not possible)  sharing razor blades (73% knew this was possible) For mother to child transmission, children coming from urban areas were less knowledgeable about this mode of transmission (31% either answered incorrectly or did not know) compared to those from rural areas (13% either answered incorrectly or did not know). Regarding testing, 89% of children surveyed knew that it was possible to test for HIV, while 6% did not know and 5% were unsure. The majority of children thought that it was necessary to know your HIV status (87%), while 8% did not and 6% were unsure. Participants were less knowledgeable about the name of the treatment for those living with HIV, as only 54% indicated that it was called “antiretroviral” or “ARVs”, and the remaining respondents did not know or did not answer the question. Respondents from urban areas were less knowledgeable (39% knew the name) compared to those from rural areas (73% knew the name of the treatment). A common misconception about HIV is that it can be cured by having sex with a virgin. While the majority (75%) of children did not believe this to be true, 12% believed it was true and 12% did not know or did not answer. More respondents from urban areas responded incorrectly or did not respond at all (18% in urban areas compared to 5% in rural areas). AAHT UGM End of Project Evaluation 13 Knowledge of child rights and protection At one site, it was felt that the program had taught children how to protect themselves from the dangers and abuses that they are vulnerable to from adults. An urban DOVC noted that “children are more confident and more knowledgeable about how to protect themselves.” This feeling was echoed by children, like this urban girl who explained during her in-depth interview that “I feel safe, know how to protect myself.” However, this was the only site where there was a high level of increased knowledge on child protection. This suggests that in other sites, most changes in knowledge have related to HIV/AIDS knowledge. Attitudes towards sex and HIV 55% of children argued that being sexually active made one more popular. The remaining 27% indicated that this was not the case and 18% were unsure or did not answer. Subgroups showed that  Males (33%) were more inclined than females (23%) to think that being sexually active makes one more popular.  In rural areas, 60% of respondents indicated that having sex does not make one more popular, compared to 51% in urban areas. Children were asked the question, “What does safe sex mean?” to determine their perceptions about safe sexual behavior. The results are displayed in Figure 4. AAHT UGM End of Project Evaluation 14 Figure 4: Perceptions of Safe Sex among Children (N=89)15 Source: Quantitative survey with 89 children The results show  The majority felt that safe sex meant using a condom every time you had sex (79%), even though in a separate question, fewer (69% of all participants) believe that they would be able to protect themselves from HIV if they knew how to use a condom correctly. This suggests that even though they know that safe sex means using a condom every time in theory, respondents are uncertain that condoms will in fact protect them in practice.  Others felt that delaying sexual debut would keep one safe (49%). Females (20%) were more inclined to perceive this than males (14%). Even though these results are positive, there is still the perception among some children (between 20% and 33%) that safe sex means having sex with someone you love, having sex with someone who looks healthy, and keeping it a secret. This indicates that there is still a perceived invulnerability to HIV within “relationships,” that one’s health/HIV status can be known based on appearance, and that concealing sexual activity makes it safe. As indicated in the table below, children living in urban areas had mixed perceptions regarding safe sexual behavior. While urban respondents scored higher than rural respondents in their understanding that abstinence, consistent condom use, and delayed sexual debut enables safer sex, they were also more inclined to believe that safe sex means having sex with someone you love, someone who looks healthy, and keeping sex a secret.                                                              15 Multi-mention question AAHT UGM End of Project Evaluation 15 Table 9: Perceptions of safe sex among rural and urban children Safe sex means… Total No. of Children in agreement (N=89) % of Rural Children (N=40) % of Urban Children (N=49) …using a condom every time you have sex 70 (79%) 28 (70%) 42 (86%) …having no sex at all 56 (63%) 24 (60%) 32 (65%) …waiting until you are old enough to make an informed decision 44 (49%) 11 (28%) 33 (67%) …having sex with someone who looks healthy 29 (33%) 4 (10%) 25 (51%) …having sex with someone you love 27 (30%) 6 (15%) 21 (43%) …keeping sex a secret 18 (20%) 8 (44%) 10 (55%) Source: Quantitative survey with 89 children There was low awareness among participants that using substances leads to risky sexual behavior. Only 35% thought that “you are less likely to use a condom during sex if you or your partner has been drinking alcohol or taking drugs.” Most participants thought that one’s judgment and decision making would not be influenced (43%) by alcohol or drugs or were unsure (23%). This is of particular concern since one of the key problems faced by the communities sampled is substance abuse. Females were even less aware than males that using substances could lead to risky sexual behavior. HIV-related stigma Participants were surveyed on a number of questions to assess the extent to which they have negative associations with HIV infection. Table 10: HIV-related stigma among children (N=89) Question Do not stigmatize Stigmatize Unsure/ No response Differences between subgroups Do you think that if you get HIV it means you are going to die soon? 65% answered “no” 21% answered “yes” 14% 78% of rural and 55% of urban participants know HIV is not a death sentence Do you think people become HIV positive because they are bad? 67% answered “no” 17% answered “yes” 16% 23% of rural and 12% of urban participants believe that people become HIV positive because they are bad Do you think children who are HIV positive should still be allowed to go to school? 67% answered “yes” 19% answered “no” 14% More males (25%) than females (15%) indicate that children who are HIV positive should not go to school. Of those in rural areas, 85% believed that children should be allowed in school, while only 53% of urban respondents felt the same. If someone in your family was sick with HIV/AIDS would you keep it a secret because you would feel bad/ ashamed/ embarrassed? 40% answered “no” 46% answered “yes” 14% Females (19%) were more uncertain about their response than males (6%). Source: Quantitative survey with 89 children When asked more generalized questions on their perceptions of HIV, respondents show mostly positive attitudes. Approximately two-thirds of respondents believe that HIV is not a death sentence, AAHT UGM End of Project Evaluation 16 that people do not become HIV positive because they are bad, and that children living with HIV still have the right to be educated at a school. However, only 40% of respondents would not be ashamed or embarrassed to have an HIV positive family member. The discrepancy in responses suggests that while general perceptions of HIV-related stigma are positive, participants still carry negative associations of HIV when it is closer to home. When comparing responses of rural and urban participants, it is clear that there are higher levels of stigma in urban areas; one possible explanation is the rural sites’ higher levels of functioning and effectiveness in achieving program objectives. Differences between male and female participants include a more belief among males that children living with HIV should not be allowed to go to school, and less certainty among about whether or not they would be embarrassed to disclose a family members’ HIV status. Capacity of CCWs to provide HIV prevention, care, and support, and child protection services The VV program strengthens community care and support for OVC by developing CCWs that are knowledgeable about, and have a positive attitude towards, HIV/AIDS prevention, care, and support; child rights; and child protection. CCWs showed improvement in their knowledge of HIV prevention, care, and support through the VV program. Two of the four DOVCs interviewed and both CCW focus groups indicated that they learned about HIV transmission, testing, healthy living, and treatment. While the changes in knowledge of HIV were generally recognized by all CCWs, those in rural areas seemed more conscious of the knowledge gained on children’s rights and community work.  The knowledge gained on children’s rights was mentioned in both the rural sites sampled and one urban site. Knowledge included general awareness of children’s rights (such as the right to education), and also specifically of the rights of children in a service delivery environment (such as confidentiality).  CCWs in both rural areas found particular value in understanding how to approach households using a positive approach, that is build on their strengths as opposed to focusing on the challenges. CCWs from one site indicated that the NACCW training made possible by the VV program taught them about self-awareness and the holistic development of children. Translating knowledge into child care practice Interviewees (DOVCs and/or CCWs) indicated that CCWs were less stigmatizing towards those living with or affected by HIV/AIDS as a result of their participation in the program. Two DOVCs and one CCW group in three of the four sites sampled indicated that CCWs had improved their HIV education/intervention skills, in addition to their knowledge. A rural DOVC said of the CCWs, “they are able to narrate HIV/AIDS to the children, show them how it is transmitted (depending on the age of the child) and understand the need to keep information confidential.” Children interviewed in the survey also reflected positively on the support provided by, suggesting that CCWs have improved their level of compassion and care for OVC. Unintended outcomes Unintended positive changes have occurred for CCWs as a result of the VV program  The skills gained through the Vana Vetu program reportedly increase employability. “My knowledge has really changed about HIV since I got training. Now I know how to treat people with HIV and how to talk to children with HIV. I also know how children should be treated in society.” - Urban CCW Focus Group AAHT UGM End of Project Evaluation 17  One focus group with CCWs indicated that through their HIV training and involvement in the VV program, some CCWs had come to accept their own HIV positive status.  One CCW indicated improving her own parenting skills due to the parenting training received through the VV program: “It has really made a difference, even at home. I now have a certain way of talking to the children and I listen to them more.” Table 11: Outcomes for CCWs as a result of VV capacity development (N= 10) Outcomes Frequency of mention No of sites who indicate outcome (N=4) No of mentions in CCW focus groups (N=2) No. of mentions by staff (N=8) Discrepancies between subgroups Reduction in stigma towards those affected by HIV and increased ability to treat them with compassion 5 4 (100%) 100% (2) 37.5% (3) None Improvement in HIV knowledge 4 100% 100% (2) 25% (2) None Increased HIV education skills 4 75% 100% (2) 25% (2) None Improved knowledge on child rights 3 50% 50% (1) 0 Only in rural communities Improved understanding of community work 3 50% 50% (1) 0 Only in rural communities Other unintended changes 3 75% 100% (2) 12.5% (1) None Data Source: Qualitative semi-structured interviews with DOVCs, team leaders, and focus group interviews with CCWs Strengthening community care and support for OVC through integrated services The intention of the VV program was to strengthen community care through more integrated services, using CCCFs and partnerships with other community actors (departments, NGOs, CBOs, FBOs) as a platform. Child and Community Care Forums There are 11 CCCFs in the communities surrounding the parishes within which the VV program works. VV initiated seven CCCFs and revived one with DOVCs and team leaders. CCCFs are established in three of the four sampled sites; focus group interviews were conducted with two CCCFs (one urban16 and one rural). The primary role of the CCCFs is to facilitate the coordination of services amongst child protection stakeholders and to make referrals for services when needed. Partnerships and collaboration are crucial. VV’s partner CCCFs meet on a quarterly or monthly basis. In a well-performing rural site, the Traditional Council and the Council of Churches play a key role on the CCCF. VV team leaders or CCWs are also engaged in the CCCF in each of the sampled sites, and provided financial support for one site’s CCCF meetings. Even where VV has reportedly played a role in establishing or reviving CCCFs, there is a high level of community ownership. The members (including VV staff and CCWs) view the drive and interest of all the members as being critical for the successful functioning of the forum. Partnerships                                                              16 Although the CCCF in the urban area was meant to be conducted with the CCCF, it became apparent during the focus group that only 3 participants were in fact CCCF representatives. AAHT UGM End of Project Evaluation 18 The VV sites collaborate primarily with the government Departments of Education, Social Development, Health, Home Affairs, and the South African Police Services. Most partnerships are not based on a memorandum of understanding (MoU), as DOVCs often form relationships with individuals that are not always mandated to sign a MoU. Two sites in rural areas have developed partnerships with the Traditional Council, Council of Churches, Department of Agriculture, and local business (a Shoprite supermarket). This shows that while MoU facilitate easier access to services, the lack of a MoU is not necessarily an obstacle to cooperation. Partners generally collaborate to identify OVC, provide referrals to each others’ services, share resources, and combine community awareness raising efforts. Partnerships seem stronger in the rural areas; one participant at a rural CCCF focus group commented that “CBOs were working parallel instead of together – now they are not duplicating services and have identified areas where they can work together…We are also now able to refer to each other because we know each other’s areas of strengths.” Table 12 summarizes the key partners that VV collaborates with, and the nature of the partnerships. Table 12: Partnerships developed across sites sampled (N=4) Partner No. of sites (N=4) Nature of relationship Hospitals 3 Referral; Accompaniment for assessment and health/psychological assistance; Testing of children for illegal drugs Schools 4 Gaining access to children; Providing space for project implementation; Assistance with identifying, monitoring, and referring OVC Social Development 4 Positive and complementary relations; Referral for food parcels and grants; Information sharing at awareness raising sessions Police 3 Collaborating for following up cases of child protection; Information sharing at awareness raising sessions Home Affairs 2 Referrals for IDs and birth certificates (informal partnership – VV is reliant on Home Affairs providing this service) NGOs 2 Referral and sharing of work Agriculture 1 (rural) Provide seedlings and fencing for home gardens Traditional Council 1 (rural) Coordination of community role players Council of Churches 1 (rural) Assistance with identifying OVC Business (e.g., Shoprite) 1 (rural) Donation of food for feeding children Data Source: Qualitative Semi-structured interviews with DOVCs, Team Leaders and Focus Group Interviews with CCWs. WHAT WERE THE MOST SIGNIFICANT CHANGES BROUGHT ABOUT BY THE ORGANIZATION IN IMPROVING THE WELL-BEING OF ITS BENEFICIARIES? The following section highlights some of the most significant changes as a result of the Vana Vetu program. These changes have had a profound impact on the lives of many OVC and have helped fulfill some of the key outcomes of the program. AAHT UGM End of Project Evaluation 19 Table 13: Most significant changes for children (N=24) Outcomes Frequency of mention No./% of Children (N= 7) No./% of Adults17 verifying change in children (N=17) Differences across subgroups Improvement in educational outcomes (through one on one, homework help, etc.) 16 4 (57%) 12 (71%) Mentioned across all adult groups but particularly by caregivers and DOVCs. Responses were consistent across stakeholder groups. Children’s basic needs met (social grants, food, clothing, protection, etc.) 15 7 (100%) 8 (47%) More caregivers mentioned this as significant outcome than other adult groups. Caregivers have particular appreciation for social grants and protection of children, while children appreciated school uniforms as well as grants. Confidence and self-esteem 11 6 (86%) 5 (29%) More CCWs, team leaders, and DOVCs mentioned this (only one caregiver focus group mentioned change on this level). Sense of love, belonging, and acceptance 11 5 (71%) 6 (35%) Within the adult group, this was primarily mentioned by staff and volunteers. Ability to cope with challenges and have a restored sense of hope 8 4 (57%) 4 (24%) None Improvement in HIV and health knowledge 5 2 (29%) 8 (47%) Primarily mentioned by respondents in rural areas. Only one caregiver mentioned this change; otherwise broadly representative of other adults. Ability to set goals 6 4 (57%) 2 (12%) Mentioned by both adult respondents and staff members. Improved relationships 4 2 (29%) 0 None Data Source: Qualitative Interviews with children, caregivers CCWs, CCCFs, partners, DOVCs and team leaders Meeting basic needs Basic needs are defined as the support that children require to help them stay in and succeed at school, access social grants, deal with the psychological trauma they may have experienced in their lives, and provide for their health and protection needs. Meeting children’s basic needs was one of the most frequently mentioned significant changes for children. It was evident that Vana Vetu has been successful in meeting these needs (through direct service provision in addition to collaborations with police and health and welfare organizations), which has allowed children to exercise their rights to safety, security, and health. Helping children obtain birth certificates so they claim social grants, as well as ensuring children’s safety through the CCCFs and afterschool care were frequently noted across stakeholder groups as ways VV met the basic needs. One caregiver noted that had it not been for VV, “most of the children would be in the streets after school, thereby being exposed to hazards like drugs and human trafficking.” The following examples from rural sites are representative of the changes brought about by VV.                                                              17 These adults include caregivers, CCWs, CCCFs, partners, DOVCs, and team leaders. AAHT UGM End of Project Evaluation 20  In a child-headed household in which the mother had died of AIDS, the CCW ensured all children were tested. One of the children was HIV positive; the CCW helped ensure the child began ART. Through the program, the children also received school uniforms and were taken on outings.  A pair of orphaned 12 year old twins was struggling to survive and was not able to access social grants. They were referred to the program via the parish; now they have birth certificates and social grants, uniforms, and shoes. They have become involved with both educational and social events through the program. Many of the significant changes in personal development mentioned below are a result of the VV services provided to address children’s basic needs. Educational changes The evaluation examined the influence of the program in ensuring that children remain in school without interruption. Overall, the findings show that this was the most frequently mentioned significant change. The educational support provided to the children allows them to feel less alienated and marginalized in the education system, increasing the likelihood that they will remain in, return to, or progress at school. Evidence from a focus group of caregivers suggested that once children receive their basic provisions such as school bags, school uniforms, food, and clothing, they feel less alienated from other children and are better able to cope in school. Caregivers also showed appreciation for the provision of support, which they did not have the funds to provide. One caregiver noted that had it not been for Vana Vetu, “some of the children would not have gone to school as they (would not have had) uniforms or shoes.” The positive impact of the program on the ability of children to attend and cope with school, and to improve academically, was apparent. Homework help, literacy and numeracy teaching, and motivation provided by the CCWs have contributed to this. In the in-depth interviews, children commented on the help that the CCWs provided them in school.  “I used to care a lot about what people say about me, The CCWs encouraged me to focus on my goal and taught me how to handle criticism.”  “When I failed matric, the CCW was there for me and gave me courage to go back to school, and I passed my matric with great results.” Social Competence Social competence results from having positive self-esteem and self-confidence, having emotional coping skills or resilience, being able to solve problems and make responsible decisions, and being able to form positive relationships with adults and peers. As a result of the VV program, there has been a positive psychological impact for children, especially orphaned children. Attendance at holiday camps where children learn life skills, outings to places of interest, and one-on-one and group psychosocial support provided by CCWs have all contributed to these changes. Confidence and self-esteem The child-centered and -rights based approach of Vana Vetu has reinforced children’s sense that they matter and that their opinions are valued, boosting most children’s self-confidence and self-esteem. One boy commented during a focus group that “without the program I wouldn’t have the confidence to even talk to you because I don’t know what I think or say matters. I didn’t even know anything about children’s rights.” Participation in the program has also given some children the chance to discover and develop their leadership potential. During her in-depth interview, one child said “I discovered that other young people see me as a leader. I am a prefect in school now.” AAHT UGM End of Project Evaluation 21 One DOVC observed that before becoming involved in VV, vulnerable children had often become isolated by other children or their negative self-image and lack of confidence. Once involved in the program, children tended to become more confident, spoke in class, and mixed socially with other children. Sense of love, belonging, and social acceptance Although not specifically stated as an intended outcome, a sense of being loved, belonging, and social acceptance are attributes of resilience and are important prerequisites for reducing risk taking behavior among youth. The data shows that the program enhanced children’s sense of belonging within their communities through the CCWs’ treatment of OVC. The very positive role of the CCWs in reducing the sense of isolation and difference that OVC, especially orphans, feel is summed up by a child from Mt. Frere: “Both my parents died. Nomakhaya18 advised me and also gave me hope. She made me understand that I can have a parent in her. She gives me money and I have clothes like other children. I also have the chance to talk about a parent, as she is like (a) parent, I can be like the other children now who talk about their parents.” A team leader noted that the provision of essentials including food, school uniforms, and stationery not only encourages school attendance but also provides a sense of social acceptance for OVC. Children place great value on being ‘equal’ and are extremely sensitive to visible differences between those children who have parents and those who don’t. Even clothing can reveal this difference; one child said during her in-depth interview that “when you are in a track suit you look like the other children. There then is no difference between those that have parents and those who do not have parents.” By providing essential items, the VV program has also provided children with a sense of normality in their lives and improved their sense of equality and feeling of acceptance. Ability to cope with challenges and restore hope Another significant change has been children’s ability to cope with adversity and have restored hope. Children who have experienced adversity, especially the loss of parents, face psychological trauma and require additional support. In such cases, the VV program’s CCWs provide ongoing support through grief and bereavement counseling, memory box work, and playing a parenting role by providing supervision at home. CCWs also provide an alternative and more positive perspective, highlighting that orphans are not alone and can find love and support from others, such as the CCWs themselves, to give OVC a restored sense of hope for their future. Children participating in the program have also shown an improved ability to make positive decisions about their lives.                                                              18 Term that children use to refer to CCWs “We (the children on the program) go on holiday and are taken for outings or camps. For a change we can be the ones that boast about being taken out. Children with parents often boast to us about that – this time it’s our turn. When we all come back to school after holidays we all compete with stories of what we did and got up to. From all this we can then decide which group went to the better place and did better things.” -Rural Child during in-depth interview “My life was very hard before I met the Nomakhaya. You see, I don’t have parents. The Nomakhaya talked to me and worked with me. She also helped me to understand and be open about my life. She helped me see that I was not the only one without parents. I don’t think I would be where I am today; I wouldn’t be thinking the way I do. I would have no hope in life at all.” -OVC in the VV program AAHT UGM End of Project Evaluation 22 Problem solving and responsible decision making Self-efficacy, the belief in one’s ability to influence one’s life path, and feeling in control of one’s own life, is an important quality in support of positive and responsible decision making. VV addresses responsible decision making in its HIV education program. In the children’s survey, self-efficacy was measured by asking respondents whether they thought they could change the path of their lives by what they chose to do. The results are mainly positive.  Most children (82%) felt they could change their lives by the choices they made.  A smaller percentage (12.4%) felt unable to influence the course of their lives and 5.6% were unsure. Belief in one’s ability to help others is also an indicator of self-efficacy. Most participants (68.5%) felt they were able to help someone who is HIV positive, while 19.1% thought they could not help and the rest were unsure (12.4%). Females were less confident about their ability to help someone living with HIV (12.4% of females were unsure compared to 5.6% of males). Again, results in rural areas are more positive than in urban areas, with rural children (80%) showing more confidence than urban (59.2%) to help someone living with HIV. Qualitative data from interviews suggest that involvement in the program promotes healthy decision making in choosing friends and reducing substance abuse and unsafe sexual behavior. One child noted that “I know how to conduct myself and avoid mixing with the wrong company of children and avoid peer pressure.” The VV program has also provided children with an alternative to engaging in risky behaviors; one child at an urban focus group indicated that without the program, “me and my friends would be on drugs and would also not know about HIV/AIDS.” Of the children sampled in the quantitative survey, 11% reported having had sex19 and 89% reported not having had sex. Those who reported having had sex were mostly males (80%) from urban areas (70%). Of those who have had sex, 60% indicated using a condom the last time they had sex, 30% did not, and 10% could not remember. Of those who had not used a condom, most were males from urban areas. Knowing ones’ HIV status is also encouraged as part of responsible decision making. Most children had not yet been tested (60%). Among those who have already had sex, however, a far greater number had been tested for HIV (60%). Relationships Participation in the Vana Vetu program also led to improved relationships between OVC and their caregivers and friends. CCWs have enabled this through one on one interaction with OVC, as well as the “peer pressure” aspect of the HIV education program. The flexibility of the CCWs’ approach also provided an environment in which children could raise issues and concerns, such as challenges in relationships within the household. The CCW can then intervene with caregivers if necessary. Caregivers in one focus group indicated improved and increasingly respectful attitudes from their children which they specifically attributed to the VV program.                                                              19 This includes oral sex (mouth to penis, vagina, or anus), vaginal sex (penis to vagina) or anal sex. “It really helped me to have good relationships with my friends and especially my parents. I didn’t really have a relationship with my parents, I didn’t speak openly with my friends. After joining the program that changed and I started to communicate with my parents and friends and now I have a good relationship with them.” -Urban Male during in-depth interview AAHT UGM End of Project Evaluation 23 Improved parenting and household economic strengthening The household visits and support groups sponsored by the Vana Vetu program seek to raise awareness among primary caregivers of issues affecting OVC. These activities improve caregivers’ capacity to parent and protect their children; however, only one rural site indicated having conducted interventions directed specifically at caregivers. Challenges encountered with implementing these activities consistently are the unrealistic planning by DOVCs, as well as inconsistent participation of caregivers during the week (due to work commitments) and weekends (due to social commitments and substance abuse). Caregivers who have been part of workshops conducted through the VV program cited benefits including  having an improved ability to cope with their children  becoming aware of alternative ways of dealing with “disobedient” children  learning the importance of expressing their love for their children Caregivers reported using child care grants to meet children’s needs and strengthen the households’ ability to look after their children. However, it is evident that some guardians may be using money intended for their children for their own purposes. In cases of misuse, social workers and CCWs are called upon to monitor the situation at the household and intervene if necessary. A focus group of caregivers from a rural community noted the success of the gardening project initiated through the VV program. The gardening project was initially started for caregivers to tend vegetables to sell and better support their families; children without caregivers also joined the program. The caregivers noted that those children who are being fed vegetables grown through this project look well and are better able to concentrate in class. Unintended outcomes In three of the sampled sites, the capacity of caregivers to protect and look after their children themselves remains to be developed. Household heads and caregivers do receive support and assistance with some parenting responsibilities through the program and the CCWs. While this does not directly empower caregivers with additional parenting skills, it does address the needs of OVC. In some cases, this approach may be a better option as it  allows children to address some of the difficult questions/issues that they have, but don’t feel comfortable enough to speak about to their caregivers  provides educational support to OVC, which is sometimes challenging for caregivers who are not literate or educated  allows caregivers to engage in economic or other activity knowing that their children are in a safe environment. This was identified by caregivers as an important need to ensure that their children are not neglected or exposed to danger when there is no one to take care of them at home. Feedback about the support CCWs provided for caregivers was collected from interviews in the three sampled sites where no direct interventions with caregivers were implemented. It is clear that even without direct interventions, CCWs provide some relief for caregivers, filling gaps where parenting inadequacies exist, parents are both working, and children are orphaned. It also highlights the importance of having a combination of both center-based and outreach programs, as centers create safe spaces for children and a place where group activities can take place. “We are relieved knowing that our children have a safe environment to play in after school and that they are also fed…I don’t have to worry about what my child would eat after school as they do not even need supper by the time they get home. That make the little food available sustain us for longer periods.” -Rural Caregivers AAHT UGM End of Project Evaluation 24 Ideally, training and educational programs should also be provided for caregivers, with the aim of strengthening their parenting skills and reducing reliance on CCWs. Further interventions designed specifically to empower caregivers will help VV to develop a more sustainable impact. Households also benefit from the child protection support (particularly child support grants) facilitated through the VV program, as this often provides them with assistance in meeting their children’s basic needs. WHAT WERE THE KEY ENABLERS OF AND BARRIERS TO MEETING PROJECT OBJECTIVES? The table below summarizes the key enablers and barriers, both programmatic and contextual, influencing the success of the VV program. Table 14: Strengths and challenges of the Vana Vetu implementation model (N = 20 respondents) Level Aspect of Program Frequency of mention : Enablers Frequency of mention: Barriers Programmatic Leadership and management (VV) 10 2 Training (Capacity building) 7 7 Monitoring and evaluation 5 11 Communication 4 1 Workload 0 5 Financial support 1 11 Leadership from church 1 6 Contextual Community receptiveness 0 9 CCWs commitment and skills 8 7 Partnerships 1 3 Data Source: Qualitative interviews with VV staff and CCWs Programmatic enablers and barriers Enablers The management and leadership provided by VV staff at headquarters, as well as by DOVCs and team leaders at VV, were felt to be a significant enabler of the VV program. In particular, the oversight and motivation provided by DOVCs and team leaders to CCWs, who are sometimes despondent due to the challenges faced within the program, was considered a strength. The introduction of a “Cluster Coordinator” into the program has been recognized as a positive step towards increasing support to DOVCs. Leadership provided by headquarters is characterized as encouraging a culture of transparency and teamwork. Communication between site level staff and headquarters is strengthened through having proper equipment (including cell phones and laptops), which is particularly important given the challenges of communicating from remote areas. Network coverage in more remote areas remains a challenge for communication. CCWs rated the training they received highly; specific mention was made of the training on Empowering Women Leaders, gaining access and acceptance in the community, and networking (the latter two topics form part of the CBO training). CCWs’ increased capacity to provide services to children in the area of HIV suggests that the HIV training was also a key enabler. A positive culture has been created around M&E across the organization; support received in this area was acknowledged by a number of interviewees. M&E training was well received, as were the quarterly reviews that encouraged ongoing learning among staff and the M&E support provided to DOVCs. The two way feedback between headquarters and site level staff, which encourages the use of M&E for documenting achievements and learning, was also noted. AAHT UGM End of Project Evaluation 25 Barriers Limited financial support for the implementation of the VV program has been a challenge, particularly at the parish level, as this limits the capacity of the program to reach all the children in need. CCWs sometimes supplement the program funding from their own money to help keep children in the program and meet the basic needs of children and households (particularly in relation to food, which is not covered through VV program funding). Limited funding reportedly leads to low stipends for CCWs and team leaders, which contributes to low levels of motivation among CCWs and CCW turnover. Contrary to reports from CCWs and team leaders, VV reportedly has a 70% retention rate, which suggests that turnover is not problematic. VV program staff also indicates that stipends are competitive with similar organizations. The ability of VV to ensure standardization of training is negatively impacted by the high turnover of CCWs. Some CCWs interviewed in urban areas indicated that they have not undergone training at all, or that they had not all undergone the same training. Others who have been trained indicated that refresher training would be useful. The ratio of CCWs to children/households is considered too high. Some interviewees at the site level (CCWs, team leaders, and DOVCs) mentioned that the workload of CCWs hinders the quality of services. Despite much effort by VV to make data collection manageable (e.g., training in data collection, simplification of forms, translation of tools, technical assistance by FHI 360-UGM M&E team, and reports being written in home languages and translated by the Language Department at the University of Stellenbosch), CCWs still feel pressured by the required M&E data collection. This is said to be a challenge due to the already taxing workload, the reported low educational levels of CCWs, and changing M&E requirements for data collection, and can lead to CCWs not appropriately reporting on the work they have done. Leadership and buy-in to the VV program from the church has been noted in some sites as a key challenge. The successful implementation of the program is dependent on personal buy-in from the clergy based at the church, regardless of endorsement at the Diocesan level. In particular, non￾Anglican team leaders require endorsement and support, as they are sometimes met with reluctance by the parishioners because they are not Anglican. Another challenge mentioned is the distance between the AAHT coordinators at the headquarters and diocesan level, and the sites at parish level, which is felt to hinder understanding of the context within which CCWs are working. Characteristics of sites and the link with results achieved The data shows that rural areas20 have more positive results in terms of the levels of knowledge, attitudes, and practices among children, as well as in children’s own perceptions of the quality of services provided by CCWs. This sub-section looks at some of the characteristics that set rural and                                                              20 The original sample of sites indicated that one rural site was an “average” performing and an urban site was “well” performing. The findings however suggest that rural sites were stronger on performance. “CCWs are not taken care of. They receive very little stipend and they spend out of their pockets; they even support kids that are hungry.” -Rural CCW “There are not enough CCWs for the amount of child-related problems…as a result, not all cases can be taken/attended. Initially CCWs used to care for 10 families irrespective of the number of children…now CCWs can be looking after more than 40 children at a time. The workload is too great. They end up not being able…to give the quality needed.” -Rural DOVC AAHT UGM End of Project Evaluation 26 urban sites apart and may lead to this distinction in results. These are based on findings and fieldworkers’ observations. Table 15: Characteristics of rural and urban sites that enable or hinder implementation Characteristic Rural Sites Urban Sites Partnerships More strategic partnerships developed (Department of Agriculture, Traditional Council, Council of Churches, and local business) Partnerships seem restricted to essential stakeholders (Department of Home Affairs, Social Development, and SAPS) Capacity development Training was not mentioned as a challenge. Training unevenly implemented among CCWs (no training or different training provided among CCWs) Parish support At one site, support from parish leader was problematic Both sites mentioned not having adequate support from parish leader, and one site indicated that parishioners are not supportive Organizational dynamics Positive dynamics evident: good communication, team work, and understanding of roles between DOVC/team leaders and CCWs Negative dynamics noticed between DOVC/team leader and CCWs, in terms of communication, value placed on each others’ work, and mutual understanding Source: Interviews with DOVCs, team leaders, CCWs and fieldworker observations The table above shows that partnerships are more constructive and strategic in rural sites. Rural sites also showed no indication that training was unevenly implemented amongst CCWs, although this was mentioned in urban sites. While challenges with the support of the parish leader was mentioned in one rural site, this was mentioned in both urban sites, with one of these clearly indicating that parishioners treated them differently because they were not Anglican. Organizational dynamics observed by fieldworkers indicate that a more negative dynamic exists in urban sites; for instance, at urban sites greater discord between CCWs and their supervisors was in evidence. These are also evident through comments made by CCWs, team leaders, and DOVCs interviewed. Some CCWs felt that challenges they face were insufficiently understood by DOVCs and team leaders, and that favoritism towards certain CCWs was shown by team leaders; on the other hand, one DOVC felt that a lack of commitment from CCWs was evident. This shows that DOVC selection on the basis of their ability to lead and communicate effectively is important. The above mentioned issues may have contributed to the differences in outcomes and perceptions of the quality of services between urban and rural sites. This attribution should be treated with caution, as there are a number of extraneous variables that could be responsible for the results noted. Contextual enablers and barriers Enablers Interviewees spoke highly of the commitment and skills of CCWs, who are dedicated to alleviating the plight of OVC. The fact that CCWs often invest emotionally and financially in the lives of OVC emphasizes their level of commitment and passion for their work. CCWs’ instinctive understanding of childcare was also acknowledged. Although partnerships between VV and other stakeholders are only mentioned by one respondent as a “key enabler,” in our opinion, this has been a key strength in the success of this community-based model of care for OVC. Delivering comprehensive services to OVC would be a challenge if community stakeholders did not collaborate. While CCWs themselves play an integral role, they are limited in their specialized skills (including the provision of healthcare and psychosocial support) to deal with issues affecting OVC. As CCWs live within OVC communities and are concerned about “CCWs give attention to things that parents neglect to do…This is ‘heart work.’ CCWs give that.” -Urban Team Leader AAHT UGM End of Project Evaluation 27 childcare, they are well placed to identify OVC and provide immediate support and referral for further intervention. In this way, CCWs play a bridging role between OVC and more specialized service providers within the community. Some of the key enablers in developing partnerships have been  introducing the VV program through presentations to other stakeholders  the involvement of the Traditional Council, given the power they wield  developing personal connections within other organizations and departments, which make the service delivery process more efficient Barriers Community receptiveness is seen as a challenge to the successful implementation of the VV program. Caregiver apathy is cited by program staff, CCWs, and CCCFs as a major implementation barrier (seven interviewees make mention of this). Some of the examples provided were the disinterest shown by caregivers in the VV program, lack of cooperation (such as not providing access to information on the child or concealing abuse), and lack of reinforcing positive teaching within the household. CCWs argued that caregivers were more interested in receiving assistance with food, clothing, money, and supervision needs to relieve them of some of their duties than in participating fully in the program. This suggests that there is limited value placed by caregivers on the psychosocial nurturing that children need, but that VV plays an important supporting role in the community for caregivers. It also suggests that caregivers may be overburdened with the responsibilities towards their families which they often cannot fulfill. A focus group run with caregivers in a poorly performing site, however, highlighted that caregivers have only partial awareness about the VV program. This, coupled with the fact that caregivers had only had programmatic intervention with VV at one out of four sites, points to the limitations of the community and household level interventions of the program. Program implementation was initially met with resistance and distrust from the general community due to having been disappointed by previous organizations. Unrealistic expectations of the community in terms of the objectives and reach of VV’s work also hindered support of the program. These initial difficulties seem to have been overcome with time and the community’s positive experience of VV services. The low level of education of CCWs is a challenge for the educational support they can provide to children. This view was only held by adult respondents in this evaluation and was not shared by children interviewed. Despite the reported challenge in the educational level of CCWs, educational outcomes for children have been a strength of the VV program. CCWs are meant to complete Grade 10 (at least) to qualify as a VV volunteer; however, some CCWs do not meet this requirement, but are strong on other criteria (such as their passion for childcare) and are therefore admitted into the program. There is insufficient evidence in the data to specify which level of education is necessary to provide the needed educational support to OVC. Even where Grade 10 qualifications exist among CCWs, other factors such as a changing education curriculum, having completed schooling long ago, language barriers, or not having the skills to impart knowledge to children may hinder the quality of educational support provided to learners. Having teachers supporting CCWs in their understanding and delivery of the curriculum has helped to overcome some of these challenges. Some of the challenges in working with partners are  two way communication is not always optimal, especially in terms of keeping teachers abreast of children’s progress  the limited awareness of potential partners or service providers of the VV program results in their reluctance to collaborate, which is a challenge for efficient service delivery AAHT UGM End of Project Evaluation 28  bureaucracy exists in working with government departments, which makes relationship building a challenge; even where relationships have been developed, staff turnover within government makes maintaining these relationships a challenge Sustainability A number of efforts have been made by the VV program staff to prepare program implementers for the closure of the program in 2012. Some dioceses have been preparing their budgets to absorb some of the costs of the program. Approaches have also been made to organizations willing to help fund OVC programs, for example, tribal authorities, municipalities, Hope Africa, the KwaBaca Trust, the Nelson Mandela Children’s Fund, and the Department of Social Development. AAHT is in the process of developing proposals to consolidate the program, and submissions have been made to Lotto and Anglo American. Other program sites (nine in total) have registered as non-profit organizations with the help of the VV program and this will enable them to apply for funds individually. It is unlikely, however, that most parishes will be able to access the necessary funding to continue the work with the current levels of quality and reach. AAHT UGM End of Project Evaluation 29 V. CONCLUSIONS Findings highlight that extreme poverty and lack of access to basic social services characterize the communities in which VV operates. The most basic needs of OVC in this context are often not met. OVC are often out of school or left unsupervised after school, leaving them vulnerable to abuse and neglect. These challenges are amplified for child-headed households, in which older children must struggle to support younger siblings without adult support. Substance abuse is rife in these communities, leading, among other things, to unsafe sexual practices by children and exposure to violence, abuse, and exploitation. Poor understanding of HIV/AIDS issues was also cited as a problem. Findings show that the VV program is well designed to meet the needs of OVC, although there remain gaps in service provision and the program is already over-extended. The support provided to children to help them attend and complete school and the provision of child protection services were particularly valued. Both recreational opportunities and education, including HIV education, were also particularly valued by children. The inability of the program to provide meals at aftercare should be addressed as this hinders attendance by OVC. Services cease when children turn 18 years old, which could interfere with positive outcomes achieved by the program being sustained during the difficult transition to adulthood. While direct services to children are important in meeting their immediate needs, further assistance to caregivers themselves, as well as to other community organizations, is needed to sustain positive changes. The results show that children generally feel positive about their interaction with CCWs and report feeling safe, comfortable, and supported; this suggests that VV’s services are accessible and child friendly. Children have high levels of knowledge about HIV transmission and prevention, although rural children scored higher than urban children. The VV program has contributed to equipping children with the knowledge to be able to protect themselves and make healthy sexual decisions. An area which remains to be addressed is knowledge of the dangers of increased risk behavior due to substance abuse. Another challenge to be addressed is the distorted perceptions of children about what safe sex means (specifically, the belief that it means having sex with someone that you love, someone who looks healthy, or keeping it a secret); this may influence children’s perceived invulnerability to HIV infection, and could result in risky sexual behavior. This highlights the importance of addressing the issues of vulnerability to HIV within relationships and “risky” social behaviors in HIV education programs aimed at children/youth. The majority of participants sampled reported not yet having had sex and over half of those who have had sex indicate having used a condom, as well as having been tested for HIV. While this is an encouraging finding, it seems likely that reducing risky sexual behavior among sexually active OVC will require increased and ongoing life skills training. With regard to HIV related stigma, on the whole, participants seem to show positive attitudes towards those living with HIV, although many would be too ashamed or scared to disclose a family member’s status. This raises concern about whether children would in fact disclose and seek assistance should they, or a family member, be HIV positive. It was generally felt that CCWs have improved their ability to provide services to OVC. CCWs reported increased knowledge about HIV, especially regarding routes of transmission. This has improved CCWs’ competence to deal with OVC and educate the community to be mindful of children’s rights and to avoid stigmatizing HIV positive children. This supports the VV program theory that through developing the capacity of communities to care for children, children will be protected. AAHT UGM End of Project Evaluation 30 CCCFs were established or revived and supported by VV in three of the four sampled sites, and while VV staff and CCWs continue to play an active role in these structures, community ownership of and involvement in these structures was reported. Partnerships have played a key role in enabling service delivery to meet the needs of children, given the limitations of the VV program. Generally, VV sites collaborate mostly with government departments such as Education, Social Development, Health, Home Affairs, and the South African Police Services. Rural sites seem to have more widespread and strategic collaboration with a range of service providers. It is clear that the VV program has improved the well-being of children on various levels. The three most significant change areas identified by respondents were in relation to meeting basic needs, providing educational support, and facilitating the development of social competence. Having basic needs met, in conjunction with support with school work, has proven to be crucial in encouraging children to remain in or return to school. Both are critical to ensure improved school performance. In terms of social competence, the most significant changes identified for children were improved confidence and self-esteem, an increased sense of belonging and social acceptance, improved ability to cope with challenges, restored hope, improved problem solving, and more responsible decision making as well as enhanced relationships with adults and peers. Having a special bond and ongoing relationship with a CCW is an important cross-cutting factor in achieving these results. Children mention participation in afterschool activities, outings, and camps as helpful in reducing feelings of isolation, exclusion, and difference from those children with parents who can provide these kinds of opportunities for them. Only one site indicated having a program targeting caregivers within the home. Caregivers who have been supported by CCWs indicated improved parenting and communication skills. A positive unintended outcome is that CCWs often fulfill some of the parenting responsibilities for the children whom they support, and in this way address some of the pertinent challenges faced by caregivers (including level of education, parenting ability, unemployment, and livelihood strategies). While engaging caregivers in the program is critical to ensuring continuity and sustainability, the need for additional support for OVC by CCWs will always remain. There are a number of factors, both internal and external to the VV program, which enabled and hindered its effectiveness. Internally, it is clear that two key factors have enabled the positive outcomes of the program: the leadership and management provided by the VV staff and team leaders, and the capacity development for CCWs provided through the VV program. Training is also reported to be somewhat uneven among CCWs, particularly in urban areas. It is important to be mindful that CCWs are not paid and are often involved in programs such as VV because it is “heart work” (their calling and passion). An ever-increasing workload, time consuming and difficult M&E tasks, inadequate stipends which fail to cover costs, and the psychological strain inherent in such work all lead to burn out and low job satisfaction. Having a team of coordinators/leaders to drive CCWs and keep them motivated is key and has indeed been a strength of the VV program. Insufficient buy-in from the clergy and parishioners at the parish level has furthermore been a challenge for the implementation of the program. Of those sites studied, those operating in rural areas seem to be more effective than those operating in urban areas. Survey findings from children in rural areas consistently show better results in their knowledge, attitude, and practice around HIV prevention, care, and support, and also rate the quality of service provision higher. The initiative shown by VV in rural areas with regard to issues such as food security and development of strategic partnerships with business and Traditional Authorities indicates more strategic thinking and coordinated services in these sites. Factors in urban areas hindering success within the VV program are negative organizational dynamics and uneven training of CCWs. These may point to some of the programmatic weaknesses in these sites. In terms of external factors influencing the program, while CCWs play a central role in the identification, immediate support, and referral of OVC, VV staff and partners note challenges in the AAHT UGM End of Project Evaluation 31 CCWs’ ability to provide educational support to OVC. The results, however, show that children did not share this sentiment, as many valued the support provided by CCWs and linked significant changes in their educational achievements to this support. This evaluation confirms that meeting basic needs and providing educational support are important to improving school performance21. Providing HIV education, child protection education, afterschool educational support, and psychological support is essential for improving the social competence of children. Finally, competent community structures (including households) working together is necessary for protecting children and sustaining these outcomes.                                                              21 This is however only based on interviews conducted and not evidence of academic progress. AAHT UGM End of Project Evaluation 32 VI. RECOMMENDATIONS These recommendations take into account the key strengths and weaknesses of the program, as well the multiple challenges experienced. They are derived both from the stakeholders interviewed and observations made by the fieldworkers and evaluators. INTERVENTION DESIGN Holistic interventions addressing material needs, as well as educational and psychosocial support needs, are required for children to ensure their well-being and psychosocial development on various levels.  OVC living in VV communities are often faced with poverty and struggle with having their basic needs met. Basic services (provision of food and clothing/uniforms and facilitating access to social grants) are important for interventions with OVC, as these are fundamental challenges experienced. The provision of food is currently not being supported through USAID/PEPFAR funding, and the provision of uniforms were supported up to 2010. The South African government’s National Guidelines on School Uniform encourages the wearing of school uniforms and the reduction in costs of school uniforms, but does not made compulsory. While this allows access to school, it also undermines the reported sense of belonging and acceptance that is created through uniforms. Sites sampled have provided food and uniforms to some degree as children’s participation in the program and schooling is often hindered due to lack of food and uniforms. It is recommended that food and clothes (specifically uniforms) be provided by VV at all program sites as a method for overcoming barriers to children’s’ participation in the program and in school activities in general.  The challenges with poverty and having basic needs met indicate the need for more household economic strengthening interventions. Facilitating access to social grants as a way to alleviate poverty is a strength of the VV program and this service should continue to be developed. VV sites should also link up with existing government feeding schemes to improve food security. Other positive initiatives could, where appropriate, be replicated at all sites. For example, partnership with the Department of Agriculture at one site led to a gardening project, thereby improving food security for OVC. Saving schemes for caregivers could also be encouraged by CCWs, and these could provide opportunities for engaging caregivers on child care issues.  The supervision and caregiving role that CCWs play for OVC has been crucial in enhancing the ability of OVC to cope with the multiple challenges they face. The proactive outreach approach used by VV has proven to be critical in enabling this kind of support, especially in reaching the most vulnerable children who cannot be accessed through the school or religious system. This has been a strength of the VV program and should be continued. All center-based programs for OVC should incorporate an outreach component if they are to help out-of-school OVC.  The aspects of the outreach program that target parents at a household level can be strengthened. The findings show that while it is critical that children are directly targeted for support, it is equally important that their primary caregivers are engaged by the program for more sustainable outcomes to be achieved. Intensive interventions with caregivers should ideally be implemented by scaling up support groups for caregivers. At the household level, CCWs could also use a mentoring approach with caregivers, utilizing interaction time with children as an opportunity to nurture parenting skills.  While CCWs are playing an essential role in providing emotional support, psychological support for those OVC who have already lost or are faced with the loss of parents is identified as a challenge by program staff. Partnerships with social workers cannot always be facilitated to provide additional psychological support to OVC when required, as there is generally limited psychological support, specifically in rural areas. The extent of psychological support needed by children and their families should therefore be further explored, and the role that VV can AAHT UGM End of Project Evaluation 33 realistically play (given their capacity and community-based model) in responding to these needs should be considered.  While OVC’s general knowledge on HIV prevention is high, areas of weakness should be addressed. In particular, the HIV education curriculum needs to be more focused on topics such as the relationship of substance use to risky sexual behavior, HIV vulnerability within “monogamous” relationships, the dangers of concurrent sexual relationships in the context of HIV, and OVC dating when HIV positive.  The findings show that children particularly value opportunities for recreation. It was recommended that more opportunities be created by the VV program for children to participate in extracurricular sports, arts, and cultural activities.  Currently children exit the VV program once they turn 18 years old. An exit strategy should be developed for these children, and could include support with career guidance, applications for further education, or job applications. This will reinforce further positive decision making among youth. Absorbing children as CCWs is another good option for providing support after the cut￾off age. This is currently being facilitated through the VV program, but should be more formalized. VV should consider a further step or stage following peer education which could lead peer educators into a CCW position. PARTNERSHIPS AND NETWORKING Network development and relationship building are important for creating a community that is competent to address challenges faced by OVC. The following recommendations are based on the VV program’s successful network development and potential improvements to these networks in the future.  Partnering with child protection service providers is important for responding to the needs of children. It is recommended that intersectoral structures (CCCF) and child protection service providers (including local NGOs, Departments of Social Development, Health, Home Affairs, South African Police Services) be increasingly utilized, and where necessary, strengthened through ongoing communication and collaboration, to facilitate increased access to government services. While this is already being achieved by the VV program, this aspect of the program should continue to be strengthened.  While informal relationships and partnerships have been initiated and nurtured, there is little evidence to suggest formalized partnerships between VV program sites and other service providers. It is suggested that Memoranda of Understanding (MOUs) between future programs, such as VV, and key service providers (including Health, Social Development, Home Affairs, SASSA, etc.) be developed where possible to formalize relationships and ease service delivery. Making personal contacts within organizations also helps facilitate access to services.  As indicated by one program site, having traditional structures as part of CCCFs in rural communities seems to facilitate community participation, specifically through having a platform to speak at Imbizo’s. This should be encouraged in other rural VV sites. In some cases, traditional structures reinforce cultural practices (in particular with regard to the disregard for children’s rights) that may exacerbate the vulnerabilities OVC face; traditional structures should therefore be sensitized around child rights in a culturally acceptable and respectful manner. PROGRAM MODEL The following recommendations relate to AAHT’s model for implementing the VV program.  The findings show that the VV program has a strong leadership and management team which has provided support with implementation and motivational challenges experienced by CCWs. The VV program leaders should continue to provide this support (or improve this where it is still a challenge) through an increasing commitment to AAHT UGM End of Project Evaluation 34 - understanding the local challenges in the community through periodic field visits with CCWs - communicating their appreciation for the commitment of CCWs to serving children - facilitating periodic review sessions to reflect on work, and help reconnect CCWs with their purpose (as a way of ensuring that motivation levels of sustained) - having weekly progress reporting, feedback, reflection, and planning meetings - improving two way communication  CCWs have found the collection of M&E data to be a challenge, mostly due to their level of education and heavy workload. It is recommended that M&E at the CCW level be as basic as possible, to ensure that it doesn’t become an added burden. M&E support should continue through training initiatives, but further hands-on support is needed for data collection. DOVC assistants could provide this support, as they already assist DOVCs with collating M&E data. DOVC assistants have a Grade 12 qualification and administrative and computer skills; these are important for assisting with monitoring activities.  Improving outreach at household level interventions would require a lower CCW to household ratio than is currently in place, as CCWs already work with more children/households than is manageable. While the Children’s Act does not stipulate norms and standards for the ratio of CCW to child for outreach services, these could be explored with other organizations doing similar work. Also, since funding is an issue, VV must be careful to ensure that the quality and depth of service provision is not compromised by expanding the reach of the program.  Stipends for CCWs are perceived as inadequate by a number of CCWs, causing a decrease in motivation. It is firmly recommended that program sustainability and quality will be enhanced if CCWs who show capacity and commitment are employed. In the long run, raising sufficient funds to employ CCWs will aid the development of the program.  Buy-in at level of the church, particularly with clergy based at the parish and parishioners themselves, has been identified as a challenge for the VV model. The VV program should ensure that buy-in is not only facilitated at the diocesan level but also at a parish level. Sensitization workshops can be conducted with the clergy at parish level (and even parishioners) to assist with internalizing the challenges faced by OVC. A more consultative and participatory approach should furthermore be used to identify how the VV program can best accommodate the needs of the church. The VV program was initially met with distrust and/or unrealistic expectations by community members. It is important that the planning of OVC programs generally takes these “teething” problems into account. It is recommended that sufficient time be allowed when entering a “new” community to build rapport and understanding with key community leaders outside the church. An initial situation analysis would help to assess the level of buy-in and anticipated support by community members. A participatory community mobilization strategy could also be developed. However, all further initiatives which add an additional workload to an already overstretched workforce need to be balanced against the risk of undermining already existing initiatives. CAPACITY DEVELOPMENT The VV model is dependent on the capacity of CCWs to implement services; the recommendations below relate to capacity development needs.  CCWs’ basic level of education is sometimes inadequate and thus their capacity to provide, for example, educational support to children, is compromised. It is recommended that programs reliant on CCWs/similar volunteers should be realistic about the expectations regarding educational support, and continuous training for capacity development should be provided to assist CCWs to provide this support and to motivate them to remain in the program in the absence of remuneration. Capacity development should take the form of training and follow-up mentoring support. The possibility of educators playing this mentoring role to CCWs should be explored by VV. AAHT UGM End of Project Evaluation 35  Training in gaining access to households, HIV and stigma, and child rights were valued by CCWs, and should be replicated for all CCWs. Computer literacy, driving, and counseling training were further requested.  Training programs have not been consistently implemented across all sites and CCWs. A standardized training plan for CCWs should be developed to ensure that all CCWs are trained in the same core skills. This is essential to ensure quality services. There are a number of ways to ensure this, however, the pros and cons of each approach should be carefully considered. - Centralized training: periodic training is conducted, and facilitated by AAHT for all CCWs - Decentralized training: DOVCs are made responsible for developing the capacity of CCWs within their own area of responsibility - Training-of-trainers: Different CCWs are trained to specialize in various issues. For example, some CCWs would specialize in training methods for children, while a different set of CCWs could specialize in HIV education. These CCWs could then hold the core skills, and be responsible for developing the capacity of other CCWs. SUSTAINABILITY While certain efforts have been made to ensure the sustainability of the VV program, this has been a key challenge, and the sustainability of VV beyond the funding of USAID/PEPFAR is not guaranteed. The following recommendations were made to ensure the sustainability of VV program at parishes.  Parishes should be assisted by AAHT, in conjunction with dioceses, to ensure VV program sustainability. AAHT should not focus exclusively on developing the capacity of CCWs to provide services to children, but also develop the organizational capacity of parishes to sustain the program. Structured training and support should thus be provided to parishes, dioceses, and CCWs on topics including fundraising, management, basic finance, governance, planning, and M&E. AAHT should assist local parishes to register themselves as CBOs.  Dioceses need to take greater ownership of the VV program. This could be facilitated through building relationships with the Diocesan HIV/Social Development Coordinators and ensuring ongoing collaboration on similar tasks.  Local resource mobilization strategies should be developed by VV program sites. This should include forming stronger linkages with the local business communities to provide the necessary support to the program (such as food, clothing, and donations). AAHT UGM End of Project Evaluation 36 APPENDIX 1: PROGRAM FRAMEWORK FOR VANA VETU Narrative Indicators Target (over 3 years) Means of verification Assumptions Goal: The lives of OVC and their families are improved. Financial independence of adults who participated in the Vana Vetu program as children 1. At least 50% of the graduates should be financially independent in that they do not rely entirely on social grants to support themselves and their families 1. Impact evaluation report (survey with Vana Vetu graduates) From outcomes to goal: Functioning local economy provides opportunities for income generation in the community Opportunities for Vana Vetu graduates to access further education and training exists Vana Vetu graduates are able to access financial support to continue with their education following high school Health care systems support healthy life and continues to provide antiretrovirals to those who need them Legislation and policy that supports and sustains care for children is in place and being implemented effectively Progression of Vana Vetu graduates to further education and training (FET) 2. 50% of Vana Vetu graduates progress to FET having completed high school at Grade 12 2. Impact evaluation report (survey with Vana Vetu graduates) Rights of children are understood and incidence of child neglect, abuse, and harmful punishment in the community 3. Trends reflect a reduction in child neglect, abuse and punishment over time; and the communities understand children’s rights 3.1 SAPS records, impact evaluation report (possibly through a survey with Vana Vetu graduates), Vana Vetu documents 3.2 Possibly include baseline studies on child care and support trends, including upholding of children’s rights Health and social commitment of children who graduated from Vana Vetu 4.1 The prevalence rate of Vana Vetu participants who received the HIV/AIDS awareness component is 10% less than that of the estimated prevalence rate for the same age group in the South African population; Vana Vetu graduates who were already HIV positive are living positively 4.1 Impact evaluation report (possibly through a survey with past graduate, noting that this will probably rely on self-disclosure which may be unreliable) 4.2 Vana Vetu graduates are perceived by community members as displaying constructive social behaviors and are not in conflict with the law, and 4.2 Impact evaluation report AAHT UGM End of Project Evaluation 37 Narrative Indicators Target (over 3 years) Means of verification Assumptions participate in ensuring that children in the community are protected from harm Purpose: The two project purposes each relate to a different strategy—services to children (Strategy 1) and community strengthening (Strategy 2) 1.1 Long term: The children participating in the Vana Vetu program have improved their social competence and are role modeling this behavior to their peers in the community. 1.1.1 Influence of the program in ensuring that children remain in school without interruption 1.1.1 Participation in the program has motivated children to remain in school (attitude to school and education is positive) 1.1.1 Evaluation Report From outputs to outcomes: Cooperation of other stakeholders in the network of support to OVC at community level 1.1.2 Children are able to progress to the next grade 1.1.2 65% of children are able to progress to the next grade 1.1.2 Evaluation Report, AAHT monitoring data 1.1.3 Children's behaviors in terms of social competence 1.1.3 65% of children are viewed by themselves and by the majority of key stakeholders as having improved their social competence 1.1.3 Evaluation Report 1.1.4 Teachers, guardians and other key stakeholders’ perceptions of children's role modeling behavior 1.1.4 On the whole, teachers, guardians, and other key stakeholders have positive perceptions of children's role modeling behavior 1.1.4 Evaluation Report (qualitative methods) 1.1.5 Children's decisions about safe sex behavior (12 – 17 year olds) 1.1.5 7,760 (100%) of the children who have been through the Vana Vetu program practice safe sex 1.1.5 Evaluation Report (survey with Vana Vetu graduates) 1.2 Short term: The children participating in the Vana Vetu program have improved their knowledge of and 1.2.I Knowledge of HIV prevention, care, and support (based on the curriculum) with a focus on abstinence and delayed sexual debut 1.2.1 65% of children know about HIV transmission prevention, care, and support based on the curriculum; the program has helped children to make a decision about abstinence and delayed sexual debut 1.2 .1Evaluation Report AAHT UGM End of Project Evaluation 38 Narrative Indicators Target (over 3 years) Means of verification Assumptions attitudes towards HIV/AIDS prevention, care, and support; life skills; education; relationships with peers and adults; Children’s Rights; and know how to protect themselves from vulnerability. 1.2.2 Percentage of children who have knowledge of organizations that deal with abuse and prevention 1.2.2 6,720 children know how to protect themselves from abuse and neglect (know where to go for help and the contact details of organizations) 1.2.2 Evaluation Report 1.2.3 Children’s beliefs about whether they are able to protect themselves from abuse and HIV/AIDS 1.2.3 Children believe that they are able to protect themselves from abuse and HIV/AIDS 1.2.3 Evaluation Report 1.2.4 Percentage of children in the Vana Vetu program that are motivated to succeed in their education 1.2.4 80% of the Vana Vetu program participants want to remain in school 1.2.4 Evaluation and Annual Reports 1.2.5 Attitudes towards education 1.2.5 The majority of children in the Vana Vetu program perceive education to be important for enhancing their life chances 1.2.5 Evaluation Report 1.2.6 Percentage of children who have a sense of purpose and hope for the future  percentage who say that they have increased their level of confidence  percentage that understand positive relationships with peers and with adults  percentage who say that their problem solving has improved 1.2.6 All or a realistic percentage of children have understood and absorbed the life skills lessons (50% and above on all the elements of social competence) (12 – 17 year olds) 1.2.6 Evaluation Report AAHT UGM End of Project Evaluation 39 Narrative Indicators Target (over 3 years) Means of verification Assumptions  percentage who say that they are now better to able to make good decisions 2.1 Long term: Strengthened community care for and support of OVC as services to children are provided by the respective stakeholders (government departments, NGOs, CBOs, FBOs) in an integrated manner. 2.1.1 Role of CCCFs in the child protection system in the communities 2.1.1 50% of CCCFs are playing a positive role in child protection in the communities, are networking with child protection stakeholders, and are acting on child protection concerns 2.1.1 Evaluation Report 2.1.2 CCCFs interventions are community driven, responding to community needs in relation to caring for vulnerable children 2.1.2 Communities, with the assistance of the CCCFs , take responsibility for their needs in relation to caring for vulnerable children 2.1.2 Evaluation and Annual Reports 2.1.3 Nature of the relationships between government departments, NGOs, CBOs, FBOs and communities 2.1.3 and 2.1.4 Key stakeholders are working together to provide child protection services, including prevention and early intervention, in an integrated manner 2.1.3 and 2.1.4 Evaluation Report and minutes of CCCF meetings 2.1.4 The extent to which service delivery for children is delivered in an integrated manner 2.2 Short term: The communities in which Vana Vetu works, including the caregivers of OVC, are knowledgeable 2.2.1 Extent to which community members are mindful of children’s rights and are working together to care for the vulnerable children in their communities 2.2.1Community members are mindful of children’s rights and are working together to care for the vulnerable children in their communities 2.2.1 Evaluation and Annual Reports AAHT UGM End of Project Evaluation 40 Narrative Indicators Target (over 3 years) Means of verification Assumptions of and have a positive attitude towards HIV/AIDS prevention, care, and support, child rights and child protection, with a focus on OVC. 2.2.2 Community awareness of HIV/AIDS prevention and stigma, child rights, and child protection 2.2.2 Communities are aware of HIV/AIDS prevention, rights of OVC, and child protection campaigns and their key messages 2.2.2 Evaluation and Annual Reports 2.2.3 Children's perceptions of safety/ vulnerability in the community and at home 2.2.3 Children feel safe and protected in the community and at home 2.2.3 Evaluation Report (qualitative or quantitative inquiry); Baseline study could be considered in the future 2.2.4 Awareness of the rights of families and care givers of OVC 2.2.4 Families and caregivers of OVC are aware of their rights 2.2.4 Evaluation Reports 2.2.5 The extent to which parents/ caregivers of Vana Vetu program participants have better parenting skills and are able to protect the children from vulnerability, including teenage mothers who have been in the Vana Vetu program. 2.2.5 Parents / caregivers have better parenting skills and knowledge about how to protect their children from vulnerability, as do the teenage mothers who have been in the Vana Vetu program. 2.2.5 Evaluation Reports Outputs: Services to Children (Strategy 1) 1. HIV/AIDS prevention education and life skills program for OVC 1.1 Number of children participating in 10 hours of HIV/AIDS prevention 1.1 7,760 children received HIV/AIDS prevention education 1.1 Annual Reports 1.2 Children’s satisfaction with the HIV/AIDS prevention and life skills program (materials, content, facilitation) 1.2 Children who participated in the program are generally satisfied with the programs 1.2 Session evaluation forms AAHT UGM End of Project Evaluation 41 Narrative Indicators Target (over 3 years) Means of verification Assumptions 2. Child protection services which predominantly focus on ensuring that the children identified as vulnerable in the communities by the volunteers and their care givers are receiving the necessary social grants, and that children know their rights and how to protect themselves from abuse 2.1 Percentage of children who involved with Vana Vetu who received a social grant 2.1 Children who need social grants are identified and all of these children receive their grants with the support of Vana Vetu 2.1 Annual Reports 2.2 Value of social grants in improving the lives of the children 2.2 Children view the social grants as being used to enable their access to food, clothes, and education. *Note: This indicator changed in COP 2010. Prior to this, food, shelter, and personal care were counted as services. The target for food was 1,500. 2.2 Evaluation Reports 2.3 Type of information that children receive about how to protect themselves from abuse and neglect 2.3 Children receive information about where to report abuse and neglect and who to go to for help 2.3 Monitoring data and Evaluation Report 3. OVC are supported with regard to continuing their basic education support 3.1 Children's views on the value of homework supervision and other forms of educational support 3.1 Children perceive the homework supervision as being useful and enabling them to succeed at school 3.1 Evaluation Report 3.2 Number of children supported through the education support program 3.2 6,800 children supported 3.2 Evaluation Report 3.3 Children's views of the life skills program 3.3 Children are positive about the content, materials, and facilitation of the life skills program 3.3 Evaluation Report (M&E still to develop internal form for this) AAHT UGM End of Project Evaluation 42 Narrative Indicators Target (over 3 years) Means of verification Assumptions 4. OVC are receiving physical and psychological health care and support 4.1 Number of OVC who gained access to the health care system 4.1 2,400 children gained improved access to the health care services (ART target 1,220) 4.1 Monitoring data (Form 15) and Annual Reports 4.2 Extent to which psychological support services as envisaged by the program are being delivered 4.2 Children receive psychological support from social workers at parish or office of social worker 4.2 Monitoring data (attendance register and program) and Annual Reports 4.3 Number of children that receive psychological support from professionals 4.3. Children receive psychological support by professional social worker at the parish or office of social worker 4.3 Monitoring data (attendance register and program) and Annual Reports 4.4 Children receive psychological support from CCWs 4.4 4,900 children receive psychosocial support from CCWs at the parish 4.4 Monitoring data (attendance register and program) and Annual Reports Outputs: Community Strengthening (Strategy 2) 1. Families and primary caregivers are strengthened to take better care of the OVC 1.1 Awareness of the Vana Vetu program and its aims and services amongst caregivers 1.1 Caregivers are aware of the program 1.1 Evaluation Reports and Monitoring Form 23 1.2 The role that Vana Vetu plays in strengthening the family in terms of prevention and early intervention 1.2 Families are strengthened as a result of the Vana Vetu program 1.2 Evaluation Reports 2. The technical capacities of community members (CCWs and SAW) is enhanced to be able 2.1 Kind of training attended by CCW’s 2.1 Training is appropriate for the services that the CCWs provide 2.1 Evaluation and Annual Reports 2.2 Ability to use their new skills in the program to care for OVC 2.2 The CCWs are enabled by the program to apply their new skills to the benefit of the OVC in their care 2.2 Evaluation Reports AAHT UGM End of Project Evaluation 43 Narrative Indicators Target (over 3 years) Means of verification Assumptions to provide services to OVC 2.3 Number of participants enrolled in the CCW and SAW courses 2.3 17 CCWs attended the training 2.3 CARE Report, registration forms, attendance registers, Annual Report 2.4 Percentage of participants who graduate from the courses 2.4 50% of CCWs graduated/completed the course 2.4 CARE Report, attendance register, organization timesheets 2.5 Reasons for drop out/non￾completion of the courses 2.5 Understanding the reasons why people drop out is used to improve the program 2.5 CARE report, AAHT staff feedback, Annual Report, attendance register, organization timesheets 3. CCCFs are established 3.1 Success of the program in establishing CCCFs 3.1 The CCCFs are established/set up in every community (25CCCFs to be established, one for each parish) 3.1 Evaluation, District, and Annual Reports 3.2 Lessons learned about establishing a CCCF in communities where there are few networks or resources 3.2 Vana Vetu assists communities with little or no networks and resources to establish a CCCF for the benefit of vulnerable children and their families 3.2 Evaluation Reports 3.3 The designation of the stakeholders on the CCCFs 3.3 Stakeholders who participate in the CCCFs comprise key local role players as identified with the local Vana Vetu program implementers 3.3 Evaluation and Annual Reports 4. Community awareness on HIV/AIDS prevention and stigma, and on the rights of children to be protected from 4.1 Community awareness of the Vana Vetu program and its purpose 4.1 Communities are aware of the Vana Vetu program and its purpose; communities are aware that Vana Vetu and the Anglican Church parish are available to support OVC 4.1 Evaluation and Annual Reports 4.2 Number and nature of 4.2 Participation of Vana Vetu program in 4.2 Annual Reports AAHT UGM End of Project Evaluation 44 Narrative Indicators Target (over 3 years) Means of verification Assumptions neglect and harm, is raised through campaigns community awareness campaigns/interventions major national days including World AIDS Day (December), Youth Day (June), 16 days of activism for No Violence Against Women and Children, Heritage Day (September), Children’s Day (November) AAHT UGM End of Project Evaluation 45 APPENDIX 2: SCOPE OF WORK Work Scope Southern Hemisphere Consultants 3828-01 Period of performance: April 9, 2012 – July 20, 2012 Detailed Task Description: Now in the fifth and final year of the project, FHI 360-UGM, at the request of USAID, is commissioning an external evaluation of our grantees. Partner organizations are non-governmental organizations (NGOs) working at national, provincial and local levels in South Africa, primarily implementing services related to services for orphans and vulnerable children (OVC), HIV care and support, HIV counseling and testing, and HIV prevention. These partners have received funding for a period of three to five years under PEFPAR, as well as both organizational and technical capacity building support. Southern Hemisphere Consultants is being contracted to execute the evaluation for one UGM Partner: AAHT. The focus of the each partner evaluation will be to:  Determine whether the program objectives under each partner’s program were achieved  Evaluate the key program outcomes and impacts related to improved health and well-being of the targeted beneficiaries Most specifically, Southern Hemisphere Consultants will seek to answer the following key evaluation questions for AAHT, utilizing tools, methods, and sub-questions approved reviewed and approved by FHI 360:  What were the most significant changes brought about by AAHT improving the well-being of OVC in targeted communities?  To what extent was AAHT able to address the needs of children within the community?  How do stakeholders (children, care givers, DoSD, community representatives, the Anglican Church) perceive the program; in terms of quality and ease of access? The focus of the evaluation is to assess effectiveness of the partner organization in addressing the needs of beneficiaries in targeted communities. The evaluators will be required to carefully consider the suitability and feasibility of design options that are likely to offer the best chance of establishing the value of the program in responding to the needs of targeted beneficiaries and communities. Both qualitative and quantitative data collection techniques should be employed. Data will be collected from various sources using appropriate data collection methods and tools for any given evaluation question. The final design to be employed will be determined after the contractor has had a chance to undertake a front-end analysis and is therefore able to select the best design option that specifies; which people or units will be studied; how they will be selected and the kinds of comparison that should be made. Data will be collected from various program sites for the partner. The evaluation will be undertaken in two stages and with expected outcomes for each stage as expressed below: Stage 1: Finalization of Evaluation Protocol Contractor will refine an evaluation protocol which demonstrates: AAHT UGM End of Project Evaluation 46  Understanding the relationship between program stages and the proposed broad evaluation question  Understanding the context for program delivery and key factors that influence program implementation  Understanding the existing theoretical and empirical knowledge about the program and examining program theory  A comprehensive stakeholder analysis and determination of roles of key stakeholders in the evaluation  Balancing costs and benefits of the evaluation and advising on the most strategic questions to include in the evaluation  Developing the Finalized Implementation Strategy and Methodology Report Stage 2: Implementation of the Evaluation Contractor will implement the partner evaluation following submission and approval of the Implementation Strategy and Methodology Report:  Pre-test instruments  Train data collectors  Undertake the evaluation data gathering process  Prepare data for analysis  Clean data  Enter data into electronic data analysis systems  Undertake comprehensive data analysis  Formulate the findings MILESTONES DELIVERABLES DUE DATES PAYMENT Data collection (April 9 – May 11) 1. Finalized Implementation Strategy and Methodology Report submitted to FHI 360 April 13 15% 2. Evaluation Work Plan, including key activities and timeframes submitted to FHI 360 April 13 10% 3. Data Analysis Plan, including dummy table/graphs for presenting data submitted to FHI 360 May 4 15% Data analysis and development of PowerPoint Presentation, including summary of evaluation process and results (May 14 – June 1) 4. Oral and PowerPoint Presentation (half-day) of preliminary findings to USAID, FHI 360, and partner (May 28- June 1, 2012) Presentations completed by June 1 15% Development of final written report, including an executive summary with highlights of the evaluation and key findings (June 1 5. Draft written report submitted to FHI 360 June 15 15% 6. Final report submitted to FHI July 20 15% AAHT UGM End of Project Evaluation 47 –July 20) 360 Development of brief paper (two￾pager) for the partner, targeting community audiences on key findings from the evaluation (June 1 –July 20) 7. Two-page paper submitted to FHI 360 July 20 15% AAHT UGM End of Project Evaluation 48 APPENDIX 3: SAMPLING FRAMEWORK Interviewee Guidelines for recruiting Instrument Rationale Total # of interviews Location AAHT key informants (HQ Senior Management) N/A Semi-structured one on one To provide insight into the strategy and intention of VV 2 Western Cape VV HQ staff Those working on the VV program Focus group (FG) To provide insight into the strategy and intention of VV as well as the enablers and challenges 1 (6-10 people) Western Cape Diocesan Office (such as, Bishop/HIV Coordinator) N/A Semi￾structured one on one To provide insight into the perceived relevance and buy￾in from implementers, determine sustainability outside of VV support 1 Bishop 2 HIV Coordinator False Bay and Port Elizabeth DOVCs N/A Semi￾structured one on one To provide in-depth information on how VV is implemented and the observed outcomes 4 (one for each site) False Bay, Port Elizabeth, Umzimvubu, and Sekhukhune Parish/Community Leaders Needs to be the person heading up the VV for the Parish/Community Semi￾structured one on one To provide in-depth information on how VV is implemented and the observed outcomes 4 (one per site) False Bay, Port Elizabeth, Umzimvubu, and Sekhukhune Volunteers (CCWs and SAWs) Needs to be a combination of those committed and those not committed Focus group They are closest to the community and so will provide useful insight into the outcomes on a community level and for children 2 Port Elizabeth and Sekhukhune CCCFs The following should be represented: Counselor, school, CPF, grandmothers Focus group They are closest to the community and so will provide useful insight into the outcomes on a community level and for children 2 Umzimvubu and False Bay Caregivers Household heads, guardians attending support groups Focus group They are closest to the children and so will provide useful insight into and verification of the outcomes on a household level and for children 3 (one per site) Sekhukhune, Umzimvubu, and False Bay Partners (such as school teachers and departments) Should be partners/other service providers who VV work closely with Semi￾structured interviews one on one To provide insight into the perceived relevance of the VV and to verify outcomes 2 Port Elizabeth and False Bay Children PE: Males FB: Females Focus group To get an overview of quality, access, relevance of services as well as the most significant 4 (one per site) False Bay, Port Elizabeth, Umzimvubu, and AAHT UGM End of Project Evaluation 49 Umzimvubu: Females Sekhukhune: Males 16– 18 years Max 10 participants changes Sekhukhune Males and females 16 – 18 years In-depth one on one To provide in-depth stories of changes facilitated through the program 4 (one per site) False Bay, Port Elizabeth, Umzimvubu, and Sekhukhune 12 – 20 years Must have participated in HIV prevention education program 50/50 males/females Self-completion survey facilitated by fieldworker in groups of 10-15 To provide an indication of changes in knowledge, attitude, and reported behavior, and of the perception of service 100 (25 at each site) False Bay, Port Elizabeth, Umzimvubu, and Sekhukhune AAHT UGM End of Project Evaluation 50 APPENDIX 4: PERSONS CONTACTED Anglican AIDS Healthcare Trust (AAHT) Headquarters (Western Cape) Thabisa Vili, Chief Executive Officer Rozette Jeptha, Vana Vetu Program Director Charmaine Liddle, Vana Vetu Program Officer Babalwa Mgayi, Development Officer Bukelwa Booysen, Personal Assistant Diocese of Umzimvubu, St. George’s Parish, Mount Frere (Eastern Cape) Saraphina Hlongana, CCCF Representative Portia Mbunna, CCCF Representative Akhona Kweyamk, CCCF Representative Nonjongo Sincwahk, CCCF Representative Voyokazi Mabongo, CCCF Representative Nomaza Buso, DOVC Rev Vuyan Buso, Team Leader Diocese of Port Elizabeth, James Calata Parish, Goven Mbeki Township (Eastern Cape) H F Scharman, CCW BJ Bashman, CCW T. Takana, CCW V K Mohala, CCW Bothna Whana, CCW B.V Sidumo, CCW T. Snhman, CCW N Mzuzu, CCW N Sokomi, CCW N Gobingla, CCW N Marti, CCW T Palamente, CCW Vuyani Mlwazi, CCW N Mauata, CCW S Nam, CCW M Hliso, CCW A N Tukula, CCW Rev Ludidi, HIV Coordinator Anneline Naidoo, DOVC AAHT UGM End of Project Evaluation 51 Nonceba Menze, Team Leader Mr. Mzuzu, School Principal Diocese of False Bay, St. Paul’s, Faure (Western Cape) LL Rooi, CCW T Adendert, CCW LC Geneke, CCW Sophia Jantjies, CCW Herwell Bunton, CCW Caroline Hoko, CCW Jonathon Maart, CCW Mari Garder, Team Leader C F Paulsen, DOVC Merwyn Castle, Bishop of False Bay Diocese Mark Williams, School Principal Patrina Pokoe, Social Welfare Coordinator for Diocese of False Bay Diocese of St. Mark the Evangelist, Jane Furse, Sekhukhune (Limpopo) Angelina Mbinane, CCW Ntori Mathabathe, CCW Prescillam Molagane, CCW Norah Kgoloko, CCW Canon Moses Nemakhavhani, Team Leader Ellen Makgane, DOVC AAHT UGM End of Project Evaluation 52 APPENDIX 5: ETHICAL GUIDELINES FOLLOWED  Discussions on religion, organizational, and country politics will not be encouraged.  Interviewees will be interviewed in the language that they are comfortable in. Self-completion quantitative interviews for children will be translated into the relevant languages.  Separate interviews/focus groups will be conducted with management and junior level staff.  Evaluators will remain neutral and refrain from discussing donor funding approaches. They will, however, acknowledge the anxiety related to the end of PEPFAR funding and reiterate that the results will not be used to determine further funding.  Fieldworkers will honor dates and times for interviews, and ensure that interviews are focused and reasonable in terms of time.  Fieldworkers will be from the provinces and therefore have good insight into cultural norms which they will adhere to.  All participants will be informed about the purpose and use of the evaluation; the right to participate or withdraw; and their right to privacy and anonymity.  Consent forms (attached) will be signed by participants (under 18 years) and their guardians.  Follow up counseling support will be organized between Southern Hemisphere and AAHT in the case of sensitive information being shared and to ensure that no harm is done to children in the interviewing process. AAHT UGM End of Project Evaluation 53 APPENDIX 6: CONSENT FORM Dear Parents/Guardians/Caregivers Southern Hemisphere Consultants is conducting an evaluation on behalf of the Anglican AIDS and Healthcare Trust (AAHT) and the Vana Vetu program. The aim of the evaluation is to determine how well the program has been implemented and what change has come about as a result of the project. Because your child has participated in the program, we would like to invite him/her to participate in a discussion where he/she will be required to tell us about their experience of the project. Any information that your child gives will remain confidential and will not be discussed with anyone. No children’s names will be used in reports that are written. Southern Hemisphere fieldworkers are qualified to do research work and will ensure that a “safe” space will be created for your child to share his/her experiences and opinions. Should any sensitive information be shared, counseling support will be provided by Southern Hemisphere/Vana Vetu for your child. The information emerging from the research will be used to further develop AAHT and the Vana Vetu OVC project and similar projects so that it can better serve the needs of the communities that they work in. Please complete the section below as a sign of your consent to allow your child to participate in this research. Photographs may be taken during the discussion for illustrative purposes, please let us know if you do not wish for your child to be photographed. Name of Parent: Name of Child: Date: Signature of Parent: Consent for Photos Yes No If you have any questions about this research, please feel free to contact us. Wilma Wessels or Victoria Tully Southern Hemisphere Consultants Tel: 021 4220205 Email: wilma@southernhemisphere.co.za or vicki@southernhemisphere.co.za AAHT UGM End of Project Evaluation 54 APPENDIX 7: DATA COLLECTION TOOLS PROJECT: Vana Vetu Program (Anglican AIDS and Healthcare Trust – AAHT) Key informant interviews (HQ Staff and Bishops) Interviewer instructions: This questionnaire should be used as a discussion guide, not as a strict interview schedule. You may find questions and answers repetitive if you stick to it, so be flexible. If you think a question has already been answered, move on. Use the questionnaire as a checklist to see that you have covered all the issues. Take your time. DO NOT LEAVE POINTS HANGING IN THE AIR. PROBE and ASK WHY WHY WHY! Relax and listen actively. If the person does not have anything to say to a question, do not leave it blank. Write N/A for no answer. Respondent’s name: Role of the respondent: Respondent’s telephone no.: Date and venue of interview: Interviewer’s name:  Southern Hemisphere Consultants has been commissioned by USAID through FHI 360 (Umbrella Grants Management project) to conduct a Summative Assessment of the Vana Vetu (VV) program.  The purpose of this evaluation is to understand the outcomes and impact of the program for children and community members. This means that we will explore the changes they have experienced as a result of the Vana Vetu program.  The results of this evaluation will not determine future funding. The intention is to gather information to generate an understanding of what works and does not work in terms of achieving the expected outcomes related to the well-being of children within a specific context. This information will be shared with government departments.  For this reason, we are interested in getting information from project stakeholders who are willing to share openly and honestly about the strengths and achievements but also the challenges faced.  While information will remain anonymous, it will be shared as part of the consolidated report. No names will be mentioned in the report.  The interview will take approximately 1 hour. Are you willing to proceed? Introduction 1. Tell me about the VV program: key objectives, how it is being rolled out, key role players Relevance of the Program AAHT UGM End of Project Evaluation 55 Key Evaluation Question: To what extent was the AAHT able to address the needs of children within the community? 2. Why was the VV program initiated in communities? What were the problems that VV attempted to address? How were these needs were established (e.g. needs assessment)? 3. How were services designed in a way to address the needs of the community? Is the program strategy sufficient to meet the needs in the community in terms of child protection and HIV prevention? 4. How well aligned is the VV program in terms of government agenda (OVC Policy and Children’s Act)? 5. In your opinion, what is VVs unique offering in child protection and HIV prevention? What were the most valuable contributions/services/products that VV have contributed to the community and children? Quality of services and access Key Evaluation Question: How do stakeholders (children, caregivers, DoSD, community representative and the Anglican church) perceive the program in terms of quality and ease of access? 6. How has the VV program been designed in a way that ensures accessibility (affordability, adequacy, availability, attainability) and quality service provision? How has it taken into account issues around associated stigma to its services? 7. How have services been designed in a child-friendly way (respect for children’s opinions, open environment, etc.)? 8. At what level do children participate in decision making at a Program level? Effectiveness Key Evaluation Question: What were the key enablers and barriers in meeting project objectives and outcomes? 9. In terms of AAHTs model, what were the strengths and challenges experienced in reaching objectives and outcomes? Think about the following: a. Community response b. Staff (Volunteers and CCFs) skills c. Management of the Program d. Monitoring, evaluation and reflective processes e. Communication and structure (e.g. between HQ, diocese, parishes, CCFs, Volunteers) f. Leadership (from HQ and church) g. Other 10. What are the gaps in being able to provide holistic services (prevention, responsive, basic needs, psychosocial support) to support the well-being of children? 11. What effort has been made in terms of ensuring that services can be sustained at community/parish level? We are about to end the interview. Recommendations 12. Going forward, what do you think are the key things that need to change, or be reinforced, to ensure that the well-being of children is improved? 13. Is there anything else you would like to add? Do you have any questions? AAHT UGM End of Project Evaluation 56 PROJECT: Vana Vetu Program (Anglican AIDS and Healthcare Trust – AAHT) Focus Group with VV HQ Staff Interviewer instructions: This questionnaire should be used as a discussion guide, not as a strict interview schedule. You may find questions and answers repetitive if you stick to it, so be flexible. If you think a question has already been answered, move on. Use the questionnaire as a checklist to see that you have covered all the issues. Take your time. DO NOT LEAVE POINTS HANGING IN THE AIR. PROBE and ASK WHY WHY WHY! Relax and listen actively. If the person does not have anything to say to a question, do not leave it blank. Write N/A for no answer. No. of participants: Positions of staff members: Date and Venue of focus group: Moderators Name: Instructions to facilitator  This is a focus group / workshop with VV staff.  You will need khokies and flipchart paper  SHC/site will have organized refreshments. As the facilitator, you should decide when these should be served. If you feel the energy is getting low, it is a good idea to have a food break.  Remember, you cannot paraphrase what people are saying enough. Paraphrasing means summarizing what people are saying in your own words. You should constantly be checking your understanding, by saying things like "so what you are saying is …," or "If I understand you correctly, you are suggesting that..." This is especially useful if there is a difference of opinion, then you can say “I see there are two views here, the one is that…and the other is that...Do you agree with me?” then you can move on.  The time frames are very tight. Focus on getting key information and keeping strictly to the times allocated per question.  Use the flipchart paper if you want to capture what people are saying.  Most important of all, relax and enjoy the discussions. Welcome and overview Instructions to facilitator Time: 5 mins Method: Plenary Purpose: To introduce participants to the session and clarify what the discussion AAHT UGM End of Project Evaluation 57 with be about  Southern Hemisphere Consultants has been commissioned by USAID through FHI 360 (Umbrella Grants Management project) to conduct a Summative Assessment of the Vana Vetu (VV) program.  The purpose of this evaluation is to understand the outcomes and impact of the program for children and community members. This means that we will explore the changes they have experienced as a result of the Vana Vetu program.  The results of this evaluation will not determine future funding. The intention is to gather information to generate an understanding of what works and does not work in terms of achieving the expected outcomes related to the well-being of children within a specific context. This information will be shared with government departments.  For this reason, we are interested in getting information from project stakeholders who are willing to share openly and honestly about the strengths and achievements but also the challenges faced.  While information will remain anonymous, it will be shared as part of the consolidated report. No names will be mentioned in the report.  The interview will take approximately 2 hours. Are you willing to proceed? Introduction of participants Instructions to facilitator Time: 5 mins Method: Game Purpose: To allow participants to get to know each other and relax in their environment. Instruction: Play a game with participants to get them to relax and know each other (2 truths one lie). Relevance Instructions to facilitator Time: 30 mins Method: Plenary Evaluation Question: How do stakeholders (children, caregivers, DoSD, community representative and the Anglican church) perceive the Program in terms of quality and ease of access? To what extent was the AAHT able to address the needs of children within the community? Instruction: AAHT UGM End of Project Evaluation 58 Brainstorm and capture using spider diagram: What are the key problems that children/community face, that VV is trying to address? Probe:  How well do you think VV responds to these needs that you have identified?  What are the strengths in VV program design?  What are the challenges in the design?  What do you think makes VV unique as an organisation? What is their unique offering/value that they add in the community? Where do they have the most impact?  How has the program been designed in a way that takes into account access (affordability, adequacy, availability, attainable, stigma, child friendly) and quality of services?  At what level do children participate in decision making at a program level? Effectiveness Instructions to facilitator Time: 30 mins Method: Group work and feedback in plenary Evaluation Question: What were the key enablers and barriers in meeting project objectives? Instruction  Divide the group into 2 and allocate the criteria according to the table below.  Provide each group with flipchart and khokies and ask them to: “Identify how (category) has facilitated/hindered the outcomes for the communities/children? (NB: each category does not have to be covered. Only relevant categories to be covered)” Divide flipchart as follows: Organizational capacity area Strengths Challenges Recommendations Leadership (from HQ and church) Communication and structure (e.g. between HQ, diocese, parishes, CCFs, Volunteers) Staff (Volunteers and CCFs) skills Community response Monitoring, evaluation and reflective processes Management of the program Other AAHT UGM End of Project Evaluation 59 Impact Instructions to facilitator Time: 40 mins Method: Fairground technique (each flipchart has a different dimension of change) Purpose: To explore the most significant changes brought about by AAHT improving the lives of OVC in targeted communities? (Planned, unplanned, etc.). Based on the feedback received from your parishes and diocese, what have been the most significant changes in:  The lives of children (e.g. self esteem/image, confidence, resilience, etc.)  Households (e.g. parenting skills, HES, relationships between children and parents)  Community actors in CP (CCWs, SAW, CCFs) – have community networks been facilitated and do they work collectively?  Community at large (e.g. changes in awareness, response, advocacy for child protection, etc.) Sustainability and Recommendations Instructions to facilitator Time: 15 mins Method: Plenary Purpose: To what extent has the results been sustained?  What effort has been made in terms of ensuring that services can be sustained at community/parish level (funds, ownership, etc.?  Going forward, what do you think are the key things that need to change, or be reinforced, to ensure that the well-being of children is improved?  Is there anything else you would like to add? Do you have any questions? Fieldworker Observations: Thank you for taking the time to share this information with me! AAHT UGM End of Project Evaluation 60 PROJECT: Vana Vetu Program (Anglican AIDS and Healthcare Trust – AAHT) Focus Group with Household Heads/Care givers Interviewer instructions: This questionnaire should be used as a discussion guide, not as a strict interview schedule. You may find questions and answers repetitive if you stick to it, so be flexible. If you think a question has already been answered, move on. Use the questionnaire as a checklist to see that you have covered all the issues. Take your time. DO NOT LEAVE POINTS HANGING IN THE AIR. PROBE and ASK WHY WHY WHY! Relax and listen actively. If the person does not have anything to say to a question, do not leave it blank. Write N/A for no answer. Province: Site: No. of participants: Date and Venue of focus group: Moderators Name: Instructions to facilitator  This is a focus group / workshop with Volunteers implementing the VV program in the community.  You will need: khokies and flipchart paper  SHC/site will have organized refreshments. As the facilitator, you should decide when these should be served. If you feel the energy is getting low, it is a good idea to have a food break.  Remember, you cannot paraphrase what people are saying enough. Paraphrasing means summarizing what people are saying in your own words. You should constantly be checking your understanding, by saying things like "so what you are saying is …," or "If I understand you correctly, you are suggesting that..." This is especially useful if there is a difference of opinion, then you can say “I see there are two views here, the one is that…and the other is that...Do you agree with me?” then you can move on.  The time frames are very tight. Focus on getting key information and keeping strictly to the times allocated per question.  Use the flipchart paper if you want to capture what people are saying.  Most important of all, relax and enjoy the discussions. Welcome and overview Instructions to facilitator Time: 5 mins Method: Plenary AAHT UGM End of Project Evaluation 61 Purpose: To introduce participants to the session and clarify what the discussion with be about  Southern Hemisphere Consultants has been commissioned by USAID through FHI 360 (Umbrella Grants Management project) to conduct a Summative Assessment of the Vana Vetu (VV) program.  The purpose of this evaluation is to understand the outcomes and impact of the program for children and community members. This means that we will explore the changes they have experienced as a result of the Vana Vetu program.  The intention is to gather information to generate an understanding of what works and does not work in terms of achieving the expected outcomes related to the well-being of children within a specific context. This will be used to share models of how OVC Programs could work within a South African context. This information will be shared with government departments.  For this reason, we are interested in getting information from project stakeholders who are willing to share openly and honestly about the strengths and achievements but also the challenges faced.  While information will remain anonymous, it will be shared as part of the consolidated report. No names will be mentioned in the report.  The interview will take approximately 1.5 hours. Are you willing to proceed? Introduction of participants Instructions to facilitator Time: 5 mins Method: Game and plenary discussion Purpose: To allow participants to get to know each other and relax in their environment. 1. Play a game with participants to get them to relax and know each other (2 truths one lie). Relevance Instructions to facilitator Time: 15 mins Method: Game and plenary discussion Evaluation question: How do stakeholders (volunteers, children, parents and the community) perceive the Program in terms of quality and ease of access? To what extent was the AAHT able to address the needs of children within the community? Instruction: Brainstorm and capture using spider diagram: What are the key problems that children/community face in terms of their safety? AAHT UGM End of Project Evaluation 62 Probe: 1. Who are the key role players when it comes to child protection and looking after the needs of children affected by HIV? 2. Have you heard about VV (note: community members may not know Program name, so ask about CCF, VV, Aux SW and CCWs)? What do they do?  What do you think about the work they do? Is it necessary/a priority? Why/why not?  What do you like about VV/other? What value are they adding in the community?  What do you not like about VV/other? What do you think they should change about their work? 3. When you hear/see VV or their staff – do they want to use their services? Do you mind if people see you talking to them?  How does VV/team treat the community and its children? How did they treat you when you interacted with them?  If you had a problem, would you go to them? Why/Why not? Impact Instructions to facilitator Time: 45 mins Method: Plenary discussion Purpose: To explore the most significant changes brought about by AAHT improving the lives of OVC in targeted communities? (Planned, unplanned, etc.). 4. Thinking back to before VV/Volunteer assisted you/your child, what was your/your child’s life like? What were the challenges, how did you/your child feel? 5. Looking at where you are now, what you think are the most significant changes that you experienced as a result of VV work? Why was this a big change for you/your child? How have the lives of children and families changed? Note to facilitator (the categories you should probe into are):  The lives of children (e.g. self esteem/image, confidence, resilience, etc.). When did you start seeing these changes?  Households (e.g. parenting skills, HES). How did the changes in the child affect the family?  Community at large (e.g. changes in their own and other community members awareness, response to child rights, advocacy for child protection, etc.) 6. How do you think your/your child’s life would have been different had VV not helped you? What would it have looked like? Sustainability and Recommendations Instructions to facilitator Time: 15 mins Method: Plenary Purpose: To what extent has the results been sustained? 7. In terms of the changes you have explained above, have these changes lasted over time, or have they faded? AAHT UGM End of Project Evaluation 63 8. If you could change one thing about the VV so that children can benefit from their services, what would it be? Why? Fieldworker notes and observations: AAHT UGM End of Project Evaluation 64 PROJECT: Vana Vetu Program (Anglican AIDS and Healthcare Trust – AAHT) Focus Group with Children Interviewer instructions: This questionnaire should be used as a discussion guide, not as a strict interview schedule. You may find questions and answers repetitive if you stick to it, so be flexible. If you think a question has already been answered, move on. Use the questionnaire as a checklist to see that you have covered all the issues. Take your time. DO NOT LEAVE POINTS HANGING IN THE AIR. PROBE and ASK WHY WHY WHY! Relax and listen actively. If the person does not have anything to say to a question, do not leave it blank. Write N/A for no answer. Province: Site: No. of participants: Male/Female: Date and Venue of focus group: Moderators Name: Instructions to facilitator  This is a focus group / workshop with children and young people.  You will need: a ball, khokies/crayons, flipchart paper, pages for children  SHC/site will have organized refreshments. As the facilitator, you should decide when these should be served. If you feel the energy is getting low, it is a good idea to have a food break.  Remember, you cannot paraphrase what children are saying enough. Paraphrasing means summarizing what people are saying in your own words. You should constantly be checking your understanding, by saying things like "so what you are saying is …," or "If I understand you correctly, you are suggesting that..." This is especially useful if there is a difference of opinion, then you can say “I see there are two views here, the one is that…and the other is that…Do you agree with me?” then you can move on.  Some sensitive issues may come up in this session so be sensitive to how participants react and create a safe space for participants to share.  If any serious issues come up for participants that you think we should follow up on, please make a note of this and inform the parish contact person and Southern Hemisphere immediately after the workshop.  The time frames are very tight. Focus on getting key information and keeping strictly to the times allocated per question.  Use the flipchart paper if you want to capture what people are saying.  Most important of all, relax and enjoy the discussions.  If you notice the energy levels being low, do an energizer with children. AAHT UGM End of Project Evaluation 65  Before the focus group, make sure you establish (with the Parish Contact Person) what the Vana Vetu (VV) Program, Child Care Workers, Auxiliary Social Workers, and CCCFs are known as in the community, so that you can use the same language.  Remember to collect the consent forms Welcome and overview Instructions to facilitator Time: 5 mins Method: Plenary Purpose: To introduce participants to the session and clarify what the discussion with be about  Hi, my name is ____________________ and I’m from a company called Southern Hemisphere. We would like to find out about how children/young people live in this community.  We need to understand how children/young people are or are not being protected and supported in the community, and what’s contributing to you feeling safe to grow into healthy, happy adults. We also want to understand what role the Vana Vetu program has played in your community.  We want to use this information so that we can learn about what works well but also what does not work well. So, it is important that you share openly and honestly how you feel as this will help us to improve services to children.  There are no right or wrong answers here. This is not a test.  As young people, your experiences, views, knowledge and opinions are important in this process and we would therefore like you to share this with us.  Do you have any questions before we start?  Would you still like to participate in this discussion? Introduction of participants Instructions to facilitator Time: 5 mins Method: Game Purpose: To allow children to get to know each other and relax in their environment. Instruction: Play a game with children to get them to relax and know each other. This should be physical but also offer them the space to say their name. Below is an example of a game, but you can also use your own, as long as it creates a relaxed environment where children can also play. Pattern ball game: Ask the group to stand in a circle. Toss a ball around the circle. The person to receive the ball should say their name (and possibly one thing they like to do for fun). Each person should receive the ball only once. Once everyone has received the ball, pass it to the person who started out with the ball. They now need to pass the ball around again, however the person throwing the ball needs to remember who they threw the ball in the first, and also the name of the person. They then call the out the name of the person as the ball gets passed around. Develop a pattern with the ball in this way. You may do this more than once to get children warmed up. AAHT UGM End of Project Evaluation 66 Ground Rules Instructions to facilitator Time: 5 mins Method: Plenary Purpose: To establish group contract and create a safe environment for young people. To cover ethical issues around child participation.  Before we start we need to think about some rules that we want to stick to while we are in this session. We need these rules so that everyone feels comfortable to share how they feel. What do you think some of those rules should be?  The following rules should be covered:  Feel free to express yourselves, this is a safe space – you will not be judged for your opinion. Can we agree on that?  Confidentiality between each other.  Explain anonymity: Even though we need to write a report based on all the information we get in our interviews (mention who else you will interview), we will not say who said what. You name will therefore be protected.  Right to withdraw: If at any point you feel that you do not want to answer a question or that you do not want to participate in the focus group, it is your right to leave. (Moderator will need to make sure that an adult is there to fetch them).  If you feel sad after our discussions here, or you would like to talk about it some more, come and speak to me (or the VV contact) after the session.  Have fun! This is not school, there are no right or wrong answers, just be honest about what you think…good and bad!  The Focus Group will last about 1.5 hrs. Are you ok with those times?  Explain that there will be food and drink and when they will be served.  Does anyone have any questions before we start? Introduction Instructions to facilitator Time: 10 mins Method: Plenary roundabout (Note this can be done as part of the introduction of participants) Purpose: To determine when and how participants were involved in the Vana Vetu Program. 1. Please explain when (and how long) you were involved with CCWs/Volunteers and how you were involved (ie what services they provided to you/how they helped you).  Are you still currently involved in the Program? If so how? Relevance Instructions to facilitator AAHT UGM End of Project Evaluation 67 Time: 45 mins Method: Plenary and Group work Purpose: To determine the extent to which AAHT were able to address the needs of children within the community and to understand the enablers and barriers in meeting project objectives. 1. What are the kinds of problems that children face in the community? What makes them feel unsafe/unhappy or not cared for? (Plenary) 2. Mapping exercise: thinking about the problems you spoke about above (specifically in relation to child protection, OVC, HIV) who are the people/organizations that play a role in caring/protecting/looking after children? Draw this on a community map (group work, presentation and debrief).  Probe for role of VV/volunteers, CCWs and CCCFs (if they did not mention this, make a note that it was not mentioned spontaneously)  Who would you feel most comfortable going to? Why?  In your interaction with VV/Volunteers, CCWs and CCCFs, how did they treat you or make you feel? (Note to fieldworker: Remind young people that it is important to speak about the good and bad so we can learn from both)  In terms of VV/Volunteers, CCWs and CCCFs, what do you like about their work, what do you think they should do better?  Remember the problems that you said children face in the community. Do VV/Volunteers, CCWs and CCCFs play a role in addressing these problems? How? Impact Instructions to facilitator Time: 45 mins Method: Plenary Purpose: To explore the most significant changes brought about by AAHT improving the lives of OVC in targeted communities? (Planned, unplanned, etc.). Method: Picture drawing exercise Instruction: On your own, draw a picture of your life after you were assisted by VV/Volunteers, CCWs and CCCFs. What were the biggest changes for you after they assisted you? Some of the things you could think about are (you don’t have to cover all of them, and if there was no change that is also ok):  The way you feel about yourself (e.g. self esteem/image, confidence, communication skills)  The way that you address problems in your life (e.g. resilience, etc.) – give examples of these  What are your goals and do you think you can achieve this?  What changed in your house/with your parents (e.g. parenting skills, relationship with parents, ability for parents to provide food/clothes/schooling, etc.)  Community – how safe do you feel in your community and why? Allow individuals to share their pictures, and probe for the following: AAHT UGM End of Project Evaluation 68 3. How did VV help you? 4. What was your life like before VV helped you? 5. How do you think your life would have been different had VV not helped you? What would it have looked like? Where would you be? We are now approaching the end of the session. 6. If you could think the VV program for one thing, what would it be? 7. If you could change on thing about VV, what would it be? Wrap up and debrief Instructions to facilitator Time: 5 mins Method: Plenary Purpose: To wrap up discussion, debrief participants and bring them back to reality. 8. How do you feel now after this session? Do you have any questions for me? What are you going to do after the session? 9. Fieldworker Observations: Thank you for taking the time to share this information with me! AAHT UGM End of Project Evaluation 69 PROJECT: Vana Vetu Program (Anglican AIDS and Healthcare Trust – AAHT) In-depth Interview with Children Interviewer instructions: This is an interview with a child/young person who has either been involved in the VV for a very long time, or is exiting the Program. The purpose is to get an in-depth understanding of the most significant changes brought about by AAHT improving the life of the child. This questionnaire should be used as a discussion guide, not as a strict interview schedule. You may find questions and answers repetitive if you stick to it, so be flexible. If you think a question has already been answered, move on. Use the questionnaire as a checklist to see that you have covered all the issues. Take your time. DO NOT LEAVE POINTS HANGING IN THE AIR. PROBE and ASK WHY WHY WHY! Relax and listen actively. If the person does not have anything to say to a question, do not leave it blank. Write N/A for no answer. Before the focus group, make sure you establish (with the Parish Contact Person) what the Vana Vetu (VV) Program, Child Care Workers, Auxiliary Social Workers and CCCFs are known as in the community, so that you can use the same language. Remember to collect the consent form (if under 18 years) Province: Site: Name of respondent: Male/Female: Age of respondent: Date and Venue of interview: Interviewers Name:  Hi, my name is ____________________ and I’m from a company called Southern Hemisphere. We would like to find out about how children/young people live in this community.  We need to understand how children/young people are or are not being protected and supported in the community, and what’s contributing to you feeling safe to grow into healthy, happy adults.  We want to use this information so that we can learn about what works well but also what does not work well. So, it is important that you share openly and honestly how you feel as this will help us to improve services to children. We also want to understand what role the Vana Vetu Program has played in your community.  There are no right or wrong answers here. This is not a test.  As young people, your experiences, views, knowledge and opinions are important in this process and we would therefore like you to share this with us.  Do you have any questions before we start? AAHT UGM End of Project Evaluation 70  Would you still like to participate in this discussion? Questions: 1. Tell me about some of the problems that children and you have faced in your community. 2. Who helps to address these problems? Who helped you specifically? (If VV is not mentioned, ask if they played any role) a. How did they make you feel while they were helping you? b. What did you like, what didn’t you like about them? c. Did you feel safe and cared for by them? d. If you could change one thing about how they helped you, what would you change? 3. Please explain when (and how long) you were involved with CCWs/Volunteers and how you were involved (ie what services they provided to you/how they helped you).  Are you still currently involved in the program? If so how? 4. Thinking back to before the CCW assisted you/your family, what was your life like? What were the challenges, how did you feel? What was making you sad? 5. Think about where you are now, what has changed in your life after you received support from VV? What changed in terms of (the significant change may be identified beforehand through key informant interviews): a. The way you feel about yourself (e.g. self esteem/image, confidence, communication skills) b. The way that you address problems in your life (e.g. resilience, etc.) – give examples of these c. What are your goals and do you think you can achieve this? d. What changed in your house/with your parents (e.g. parenting skills, relationships with family members, ability for parents to provide food/clothes/schooling, etc.) e. How do you feel about your safety? 6. How do you think your life would have been different had VV not helped you? What would it have looked like? Where would you be? 7. Did the changes you felt last or did they fade? 8. If you could thank the VV Program for one thing, what would it be? 9. If you could change on thing about VV, what would it be? 10. How do you feel now after this session? Do you have any questions for me? What are you going to do after the session? Observations of Interviewer: Thank you for taking the time to share with me! AAHT UGM End of Project Evaluation 71 Children Survey: Final: English/Afrikaans This is a questionnaire about what you think and know about sex, HIV/AIDS. Hierdie is ‘n vrae lys oor wat U dink en weet of ken van Seks, HIV Vigs en AIDS. The purpose is to help us understand what the Vana Vetu Program has taught you, and to get your opinion about the Vana Vetu Program. The information will help us to improve services for children. Die doel van dit is om ons te help verstaan wat U geleer het van die Vana Vetu Program af, en om U opinies te kry oor die spesifieke Program.Hierdie inligting sal ons help om die dienste vir Kinders te verbeter.  We do NOT want to know your name. Ons wil nie weet wat U naam is nie.  The questionnaire is NOT a test. Die vrae lys is nie ‘n toets nie.  There are NO wrong or right answers! Daar is geen regte of verkeerde antwoord nie.  Thank you for helping us by answering the questions honestly! Dankie dat U ons help om die antwoorde eerlik te beantwoord! Instructions / Instruksies 1. Please do NOT write your name on the questionnaire. Moet asseblief nie U naam op die vrae lys skryf nie. 2. Please make sure you answer ALL questions. Maak asseblief seker om AL die antwoorde te beantwoord. 3. If you do not understand a question, please raise your hand and someone will come to help you. As U nie ‘n vraag verstaan nie, steek asseblief U hand op sodat iemand U kan help. Definitions / Definiesies 1. When you see the word ‘Sex’ in this questionnaire it means when partners agree to have oral sex (mouth to penis, vagina, or anus), vaginal sex (penis to vagina) or anal sex (penis to anus). AAHT UGM End of Project Evaluation 72 2. In this questionnaire ‘protected sex’ means sex with a condom, and ‘unprotected sex’ means sex without a condom. Before we start the questionnaire please complete the following Practice questions: 1. What is your favorite TV Program? (please tick)  Wat is U gunsteling program? (Merk asseblief)  Generations  Isidingo  Backstage  I Don’t Know  Ek weet nie 1. What is your favorite subject at school? (Please write down the answer if you know it OR tick the box if you don’t know) ___________________________________________________ I Don’t Know  Ek weet nie How old are you? _______________ years old. Are you a boy or a girl? (Please tick)  Wanneer U die woord ‘Seks’ in die vrae lys sien beteken dit wanneer ‘n paartjie saam stem om mondelike seks te he(mond na penis of vagina),vaginale seks (penis na vagina)of anale seks In die vrae lys beteken ‘veilige seks’ seks met ‘n kondoom, en ‘onveilige seks’ seks sonder ‘n kondoom . Voor ons begin met die vrae lys kan U asseblief die volgende oefen vrae voltooi: Wat is U gunsteling vak op skool? (Skryf asseblief die antwoord neer as U weet of merk as U nie weet nie) Hoe oud is U? AAHT UGM End of Project Evaluation 73 Is U manlik of vroulik? (Merk asseblief)  Boy/Manlik   Girl/Vroulik  What is the name of the area in which you live? Wat is die naam van die area waar U woon? __________________________________ What grade are you in? In watter graad/standard is U tans? __________________________________ When did you attend the HIV training? (please tick) Wanneer het U die HIV sesies bygewoon? (Merk asseblief) This year/Hierdie jaar (2012) Last year/ Verlede jaar (2011) 2010/2010 Before/Voor 2010 1. Do you think that you can change your life by what you choose to do? Dink jy dat jy jou lewe kan verander deur middel van wat jy kies om te doen? Yes I can/Ja ek kan  No I can’t/Nee ek kan nie  I don’t know/ Ek weet nie  2. How can you get HIV from another person who has HIV? (please answer all questions) Hoe kan U HIV vigs kry van ‘n ander persoon af wat HIV vigs het? (beantwoord asseblief alle vrae) From having sex without a condom Deur middel van omgang/seks te het sonder ‘n kondoom Yes  No  Don’t Know  Ewe Hayi Andiyazi AAHT UGM End of Project Evaluation 74 Ja Nee Weet nie From a mother to her child during birth and breastfeeding Deur ‘n ma gedurende geboorte en borsvoeding Yes  No  Don’t Know  Ja Nee Weet nie From sharing a toilet Deur middel van ‘n bad kamer/ toilet saam gebruik Yes  No  Don’t Know  Ja Nee Weet nie From getting a blood transfusion (blood donation) Deur middel van bloed oor tappings (bloed skenking) Yes  No  Don’t Know  Ja Nee Weet nie From sharing drug needles Dwelm naalde te deel Yes  No  Don’t Know  Ja Nee Weet nie From kissing when someone’s mouth is bleeding Deur iemand te soen wie se mond bloei Yes  No  Don’t Know  Ja Nee Weet nie From sharing knives and forks Om messe en vurke te deel Yes  No  Don’t Know  Ja Nee Weet nie From touching Aanraaking Yes  No  Don’t Know  Ja Nee Weet nie From sharing razor blades Om skeer lemme te deel Yes  No  Don’t Know  Ja Nee Weet nie 3. What does ‘safe sex’ mean? (please tick two answers only)  Wat beteken ‘veilige seks/omgang’? (Merk slegs twee antwoorde)  AAHT UGM End of Project Evaluation 75 Safe sex means having sex with someone you love Veilige seks/omgang beteken om seks/omgang te het met iemand wat U lief het Safe sex means having sex with someone who looks healthy Veilige seks/omgang beteken om seks/omgang te het met iemand wat gesond lyk Having no sex at all Om glad nie seks/omgang te het nie Safe sex means using a condom every time you have sex Veilige seks/omgang beteken om ‘n kondoom te gebruik elke keer wanneer U seks/omgang het Safe sex means waiting until you are old enough to make an informed decision Veilige seks/omgang beteken om te wag totdat U groot genoeg is om ‘n vaste besluit te kan neem Safe sex means keeping sex a secret. Veilige seks/omgang beteken om seksueele omgang ‘n geheim te hou 4. Do you think that if you get HIV/AIDS it means you are going to die soon? Dink U dat wanneer U HIV/VIGS kry ,dit beteken dat U gou sal dood gaan? Yes/Ja  No/ Nee  I Don’t Know/ Ek weet nie  5. What is the name of the treatment (medicines) for people who are sick with AIDS? Wat is die naam van die medikasie wat mense gebruik wat siek is met HIV Vigs? (Please write down the answer if you know it OR tick the box if you don’t know) (Skryf asseblief die antwoord neer as U weet of merk as U nie weet nie) _________________________________________________________ I Don’t Know Ek weet nie  6. Is there a test to find out if you have HIV? Is daar ‘n toets wat gedoen kan word om uit te vind of U HIV vigs het? Yes/Ja  No/ Nee  I Don’t Know/ Ek weet nie  7. Do you think it is important to find out if you are HIV positive? AAHT UGM End of Project Evaluation 76 Dink U dat dit belangrik is om uit te vind of U HIV Positief is? Yes/ Ja  No/ Nee  I Don’t Know/ Ek weet nie  8. Do you think people who have HIV/AIDS can be cured by having sex with a virgin (someone who has never had sex)? Dink U dat mense gesond sal word deur te seks/omgang te het met ‘n maagd(iemand wat nog nooit seks/omgang gehad het nie) Yes/Ja  No/Nee  I Don’t Know/ Ek weet nie  9. Who is responsible for taking care of children affected by HIV in your community? Wie is verantwoordelik vir die sorg van kinders wat geaffekteer is duer HIV Vigs in U gemeenskap? (Skryf asseblief die antwoord neer as U weet of merk as U nie weet nie) (Please write down the answer if you know it OR tick the box if you don’t know) _________________________________________________ I Don’t Know/Ek weet nie  10. Who would you talk to if you wanted to get help if you with an HIV related issue? Met wie sal U praat as U ‘n help nodig het met ‘n HIV vigs verwante saak? (You can tick more than one) (U kan meer as een merk)  I would not talk to anyone Ek sal met niemand praat nie I would talk to my parents or adults that I live with Ek sal met my ouers of ouer mense praat met wie ek bly I would talk to my friend Ek sal met my vriend/in praat I would talk to my teacher Ek sal met my onderwyser praat I would talk to someone at the clinic Ek sal met iemand by die kliniek praat I would talk to someone at Church Ek sal met iemand by die kerk praat AAHT UGM End of Project Evaluation 77 I would talk to the CCCF Ek sal met die CCCF praat I would talk to the Child Care Worker Ek sal met die Kinder Welsyn/Maatskaplike werkster praat I Don’t know who I would talk to Ek weet nie met wie ek sal praat nie 11. Do you think people your age are more popular if they are having sex? Dink U dat kinders U ouderdom meer popular is wanneer hulle seks/omgang het? Yes/Ja  No/ Nee  I Don’t Know/ Ek weet nie  12. Have you ever had sex? Het U al ooit seks/omgang gehad? Yes/Ja  No/Nee  13. Did you use a condom the last time you had sex? Het U ‘n kondoom gebruik die laaste keer U seks/omgang gehad het? Yes/Ja  No/ Nee  I have never had sex/Ek het nog nooit seks/omgang gehad nie  14. Do you know your HIV status at this point in time (through an HIV test)? Ken U op die oomblik wat U HIV status is? Yes/ Ja  No/ Nee  15. Is it true that you are less likely to use a condom during sex if you or your partner has been drinking alcohol or taking drugs? Is dit waar, dat U heel waarskynlik nie ‘n kondoom sal gebruik wanneer U of U eggenoot seks/omgang het gedurende U onder die invloed van alcohol of dwelms is? Yes/ Ja  No/ Nee  I Don’t Know/ Ek weet nie  16. Do you think you can protect yourself from HIV if you know how to use a condom correctly? Dink U dat U Uself kan beskerm van HIV vigs af as U weet hoe om ‘n kondoom korrek te gebruik? Yes/Ja  No/ Nee  I Don’t Know/ Ek weet nie  AAHT UGM End of Project Evaluation 78 17. If someone in your family was sick with HIV/AIDS would you keep it a secret because you would feel bad/ashamed/embarrassed? As iemand in U familie siek is met HIV/VIGS, sal U dit ‘n geheim hou want U voel sleg, verneederd of skaam omtrent dit? Yes/ Ja  No/ Nee  I Don’t Know/ Ek weet nie  18. Do you think children who are HIV positive should still be allowed to go to school? Dink U dat kinders wat HIV/ VIGS het nog steeds toe gelaat mag word om skool by te kan woon? Yes/Ja  No/ Nee  I Don’t Know/ Ek weet nie  19. Do you think people become HIV positive because they are bad people? Dink U dat mense HIV positief raak omdat hulle slegte mense is? Yes/ Ja  No/ Nee  I Don’t Know/ Ek weet nie  20. Do you think there is something YOU can do to help people who are HIV positive? Dink U daar is iets wat U kan doen om mense te help wat HIV positief is? Yes/ Ja  No/ Nee  I Don’t Know/ Ek weet nie  21. Who do you live with? Met wie bly U? (You can tick more than one)(Kan meer as een merk)  Mother / Moeder Father / Vader Grandmother/Ouma Aunt or Uncle/ Oom of Tante Sister or brother who is younger than 25 years old/ Broer of suster wat jonger as 25 jaar oud is Sister or brother who is older than 25 years old/ Broer of suster wat ouer as 25 jaar oud is AAHT UGM End of Project Evaluation 79 Neighbour/ Bure I live alone/ Ek bly alleen Other :/ Ander: 22. Please tick how CCWs have helped you: Merk asseblief hoe CCW’s U gehelp het: (You can tick more than one) (Kan meer as een merk)  A CCW did not help me ‘N CCW het my nie gehelp nie A CCW visited my home ‘N CCW het my huis besoek The CCW helped me to get a grant (e.g. child support grant) Die CCW het my gehelp om ‘n toelae te kry(bv kinde toelag) The CCW helped me to get a birth certificate/ ID Die CCW het my gehelp om ‘n geboorte sertifikaat/ ID te kry I attended a group (e.g. support group, memory box) run by the CCW Ek’t ‘n groep by gewoon(hulp groep) wat deur die CCW gereel was AAHT UGM End of Project Evaluation 80 The CCW helped me with my school work Die CCW het my gehalp met my skool werk The CCW told me where to go for help with a problem Die CCW het my gese waarheen ek kan gaanom hulp te kry met ‘n probleem I spoke to the CCW about a problem I was having Ek’t met die CCW gepraat omtrent ‘n problem wat ek gehad het The CCW helped me to get food / clothing / school uniform Die CCW het my gehelp om kos/ klere/ skool uniform te kry Other Ander 23. How did you feel about the CCW that helped you? (please answer all questions) Hoe het U gevoel teenoor die CCW wat U gehelp het?(antwoord asseblief alle vrae) The CCW were easy to talk to Die CCW was maklik om mee te praat Yes/ Ja  No/ Nee  Don’t Know/ Ek weet nie  The CCW made me feel safe Die CCW het my laat veilig voel Yes/ Ja  No/ Nee  Don’t Know/ Ek weet nie  AAHT UGM End of Project Evaluation 81 The CCW was helpful Die CCW was hulpvaardig Yes/ Ja  No/ Nee  Don’t Know/ Ek weet nie  I will tell my friends to go to CCW if they need help Ek sal my vriende se om na CCW te gaan as hulle hulp nodig het Yes/ Ja  No/ Nee  Don’t Know/ Ek weet nie  Thank You/Baie Dankie Vir U tyd en Saamewerking AAHT UGM End of Project Evaluation 82 PROJECT: Vana Vetu Program (Anglican AIDS and Healthcare Trust – AAHT) Bishops/HIV Coordinators Interviewer instructions: This questionnaire should be used as a discussion guide, not as a strict interview schedule. You may find questions and answers repetitive if you stick to it, so be flexible. If you think a question has already been answered, move on. Use the questionnaire as a checklist to see that you have covered all the issues. Take your time. DO NOT LEAVE POINTS HANGING IN THE AIR. PROBE and ASK WHY WHY WHY! Relax and listen actively. If the person does not have anything to say to a question, do not leave it blank. Write N/A for no answer. Respondent’s name: Role of the respondent: Respondent’s telephone no.: Date and Venue of interview: Interviewer’s Name:  Southern Hemisphere Consultants has been commissioned by USAID through FHI 360 (Umbrella Grants Management project to conduct a Summative Assessment of the Vana Vetu (VV) program.  The purpose of this evaluation is to understand the outcomes and impact of the program for children and community members. This means that we will explore the changes they have experienced as a result of the Vana Vetu program.  The results of this evaluation will not determine future funding. The intention is to gather information to generate an understanding of what works and does not work in terms of achieving the expected outcomes related to the well-being of children within a specific context. This information will be shared with government departments.  For this reason, we are interested in getting information from project stakeholders who are willing to share openly and honestly about the strengths and achievements but also the challenges faced.  While information will remain anonymous, it will be shared as part of the consolidated report. No names will be mentioned in the report.  The interview will take approximately 1 hour. Are you willing to proceed? Introduction 1. Tell me about the VV program: key objectives, how it is being rolled out, key role players Relevance of the Program Key Evaluation Question: To what extent was the AAHT able to address the needs of children within the community? AAHT UGM End of Project Evaluation 83 2. Why was the VV Program initiated in communities? What were the problems that VV attempted to address? How were these needs were established (e.g. needs assessment)? 3. How were services designed in a way to address the needs of the community? Is the program strategy sufficient to meet the needs in the community in terms of child protection and HIV prevention? 4. In your opinion, what is VVs unique offering in child protection and HIV prevention? What were the most valuable contributions/services/products that VV have contributed to the community and children? Effectiveness Key evaluation question: What were the key enablers and barriers in meeting project objectives and outcomes? 5. In terms of AAHTs model, what were the strengths and challenges experienced in reaching objectives and outcomes? Think about the following:  Community response  Staff (Volunteers and CCFs) skills  Management of the Program  Communication and structure (e.g. between HQ, diocese, parishes, CCFs, Volunteers)  Leadership (from HQ and church)  Other 6. What effort has been made in terms of ensuring that services can be sustained at community/ parish level? We are about to end the interview. Recommendations: 7. Going forward, what do you think are the key things that need to change, or be reinforced, to ensure that the well-being of children is improved? 8. Is there anything else you would like to add? Do you have any questions? AAHT UGM End of Project Evaluation 84 PROJECT: Vana Vetu Program (Anglican AIDS and Healthcare Trust – AAHT) DOVC Interviewer instructions: This questionnaire should be used as a discussion guide, not as a strict interview schedule. You may find questions and answers repetitive if you stick to it, so be flexible. If you think a question has already been answered, move on. Use the questionnaire as a checklist to see that you have covered all the issues. Take your time. DO NOT LEAVE POINTS HANGING IN THE AIR. PROBE and ASK WHY WHY WHY! Relax and listen actively. If the person does not have anything to say to a question, do not leave it blank. Write N/A for no answer. Province: Site: Respondent’s name: Role of the respondent: Respondent’s telephone no.: Date and Venue of interview: Interviewer’s Name:  Southern Hemisphere Consultants has been commissioned by USAID through FHI 360 (Umbrella Grants Management project) to conduct a Summative Assessment of the Vana Vetu (VV) program.  The purpose of this evaluation is to understand the outcomes and impact of the program for children and community members. This means that we will explore the changes they have experienced as a result of the Vana Vetu program.  The results of this evaluation will not determine future funding. The intention is to gather information to generate an understanding of what works and does not work in terms of achieving the expected outcomes related to the well-being of children within a South African context. This information will be shared with government departments.  For this reason, we are interested in getting information from project stakeholders who are willing to share openly and honestly about the strengths and achievements but also the challenges faced.  While information will remain anonymous, it will be shared as part of the consolidated report. No names will be mentioned in the report.  The interview will take approximately 1.5 hours. Are you willing to proceed? Introduction AAHT UGM End of Project Evaluation 85 1. Tell me about the VV program: key objectives, how it is being rolled out in your cluster/diocese, key activities and key role players. Relevance Key Evaluation Question: To what extent was the AAHT able to address the needs of children within the community? 2. Why was the VV program initiated in communities? What were the problems that VV attempts to address? How did you know that these were problems within the community? 3. How do the services offered by VV address the needs of the community? 4. What are some of the gaps in VV services that hinder the achievement of improving children’s lives (in terms of financial security, education, child protection and health)? 5. What makes VVs different/unique as a service provider working towards improving children’s lives? Impact Key Evaluation Question: What were the most significant changes brought about by AAHT improving the lives of OVC in targeted communities? What have been the more immediate/intermediate outcomes for the groups that were targeted? 6. What were some of the key changes observed/reported for the children targeted by VV? Think about their:  Protection  psychosocial development (i.e. feelings about themselves, ability to cope with challenges, relationships with those around them, ability to make good decisions)  education  HIV prevention and care  financial security, etc. 7. To what extent has VV built a community (parents, church/parish, team leaders, care workers, DOVC, CCCFs) that is competent/able to address issues faced by OVC?  How has Volunteers knowledge on HIV and Child Rights changed?  Volunteers Attitudes/Beliefs/Stigma towards those affected by HIV/AIDS has changed?  Beliefs about child rights?  What kind of capacity has been developed within the community around child protection and HIV?  How has sharing of knowledge of services for referrals been facilitated through the Program?  What role did the VV program play in enabling the above? 8. Tell me about the partners/service providers you work with. Who are they? How do you work together? What the strengths and challenges of your relationship?  How coordinated is your response to HIV and Child Protection issues? What coordination mechanisms are in place?  What role do CCCFs play in protecting children?  What role did the VV play in setting up the above? Effectiveness Key Evaluation Question: What were the key enablers and barriers in meeting project objectives and outcomes? AAHT UGM End of Project Evaluation 86 9. Thinking about the changes reported/observed, what were the strengths in achieving these? What were the challenges that prevented you from achieving even further change? Probe for the following categories:  Community response  Staff (Volunteers and CCFs) skills  Management of the Program  Monitoring, evaluation and reflective processes  Communication and structure (e.g. between HQ, diocese, parishes, CCFs, Volunteers)  Leadership (from HQ and church)  Other 10. What are the gaps in being able to provide holistic services (prevention, responsive) to support the improvement of children’s lives? Sustainability Key Evaluation Question: To what extent has the results been sustained? 11. Will CCWs, CCCFs, church, parish be able to continue supporting OVC if VV was over? What structures exist to enable continuity (e.g. fundraising, proposal writing, community ownership, etc.)? What prevents this from happening? 12. In terms of the changes you have heard about/observed for children and the community, have these changes lasted over time, or have they faded? We are about to end the interview. Recommendations: 13. Going forward, what do you think are the key things that need to change, or be reinforced, to ensure that the well-being of children is improved? 14. Is there anything else you would like to add? Do you have any questions? AAHT UGM End of Project Evaluation 87 PROJECT: Vana Vetu Program (Anglican AIDS and Healthcare Trust – AAHT) Focus Group with Volunteers (Also known as: Auxiliary Social Workers, Peer Educators, CCWs, CCCFs) Interviewer instructions: This questionnaire should be used as a discussion guide, not as a strict interview schedule. You may find questions and answers repetitive if you stick to it, so be flexible. If you think a question has already been answered, move on. Use the questionnaire as a checklist to see that you have covered all the issues. Take your time. DO NOT LEAVE POINTS HANGING IN THE AIR. PROBE and ASK WHY WHY WHY! Relax and listen actively. If the person does not have anything to say to a question, do not leave it blank. Write N/A for no answer. Province: Site: No. of participants: Type of volunteers (aux social worker, CCCF, CCW): Date and Venue of focus group: Moderators Name: Instructions to facilitator  This is a focus group / workshop with Volunteers implementing the VV program in the community.  You will need: khokies and flipchart paper  SHC/site will have organized refreshments. As the facilitator, you should decide when these should be served. If you feel the energy is getting low, it is a good idea to have a food break.  Remember, you cannot paraphrase what people are saying enough. Paraphrasing means summarizing what people are saying in your own words. You should constantly be checking your understanding, by saying things like "so what you are saying is …," or "If I understand you correctly, you are suggesting that..." This is especially useful if there is a difference of opinion, then you can say “I see there are two views here, the one is that…and the other is that…Do you agree with me?” then you can move on.  The time frames are very tight. Focus on getting key information and keeping strictly to the times allocated per question.  Use the flipchart paper if you want to capture what people are saying.  Most important of all, relax and enjoy the discussions.  Please ensure that the register in signed. AAHT UGM End of Project Evaluation 88 Welcome and overview Instructions to facilitator Time: 5 mins Method: Plenary Purpose: To introduce participants to the session and clarify what the discussion with be about  Southern Hemisphere Consultants has been commissioned by USAID through FHI 360 (Umbrella Grants Management project) to conduct a Summative Assessment of the Vana Vetu (VV) program.  The purpose of this evaluation is to understand the outcomes and impact of the program for children and community members. This means that we will explore the changes they have experienced as a result of the Vana Vetu program.  The results of this evaluation will not determine future funding. The intention is to gather information to generate an understanding of what works and does not work in terms of achieving the expected outcomes related to the well-being of children within a specific context. This information will be shared with government departments.  For this reason, we are interested in getting information from project stakeholders who are willing to share openly and honestly about the strengths and achievements but also the challenges faced.  While information will remain anonymous, it will be shared as part of the consolidated report. No names will be mentioned in the report.  The interview will take approximately 2 hours. Are you willing to proceed? Introduction of participants Instructions to facilitator Time: 15 mins Method: Game and plenary discussion Purpose: To allow participants to get to know each other and relax in their environment. To get an overview of how VV is implemented within the community. 1. Play a game with participants to get them to relax and know each other (2 truths one lie). 2. Ask the participants to tell you about the history of their Child Care Forum/Voluntary work. When did it start, who initiated it and how was it initiated? 3. What do they aim to achieve through their voluntary work, through their CCF or through the VV Program? AAHT UGM End of Project Evaluation 89 4. What is the Committee/Volunteer/VV known for in their particular area of operation? Relevance Instructions to facilitator Time: 15 mins Method: Game and plenary discussion Evaluation question: How do stakeholders (volunteers, children, parents and the community) perceive the program in terms of quality and ease of access? To what extent was the AAHT able to address the needs of children within the community? Instruction: Brainstorm and capture using spider diagram: What are the key problems that children/community face, that VV is trying to address? Probe:  What services do you provide for children in your community?  How well do the services that you provide respond to these needs that you have identified in the community?  What are the challenges to providing services in relation to the need in the community?  What is their most valuable offering to the community? Which services enable them to address key needs/where do they have the most impact?  When community members hear VV – do they want to use your services? What do they think of the work that you do? (Probe: Is there positive or negative association with VV name – HIV stigma; Do they see you as the “go to” people in terms of OVC? Effectiveness Instructions to facilitator Time: 30 mins Method: Group work and feedback in plenary Evaluation Question: What were the key enablers and barriers in meeting project objectives? Instruction  Divide the group into 2 and allocate the criteria according to the table below.  Provide each group with flipchart and khokies and ask them to: “Identify how (category) has facilitated/hindered the outcomes for the communities/children? (NB: each category does not have to be covered. Only relevant categories to be covered)” Divide flipchart as follows: AAHT UGM End of Project Evaluation 90 Organizational capacity area Strengths Challenges Recommendations Volunteer skills Probe: What role has VV played in your skill development? Management of volunteers (by Parish and diocese) Communication Partnerships Other Impact Instructions to facilitator Time: 45 mins Method: Fairground technique (each flipchart has a different dimension of change) or Plenary discussion Purpose: To explore the most significant changes brought about by AAHT improving the:  levels of knowledge and attitudes of volunteers towards HIV and Child Rights  lives of OVC in targeted communities? (Planned, unplanned, etc.). 5. What have been the most significant changes you have experienced personally through being involved in the Vana Vetu Program? Think about the following: a. How your knowledge on HIV and Child Rights has changed? b. Attitudes/Beliefs/Stigma towards those affected by HIV/AIDS has changed? c. Beliefs about child rights? d. Other changes How did VV enable the above mentioned changes? 6. Based on what you see in the community, what have been the most significant changes that you see as a result of your work? How have the lives of children and families change? What changes do you see in the community? Think of the following categories: a. The lives of children (e.g. self esteem/image, confidence, resilience, education, HIV prevention, etc.). To what extent are children acting as “role models” to other children? b. Households (e.g. parenting skills, HES) c. Community actors in CP (CCWs, SAW, CCCFs) – have community networks been facilitated and do they work collectively to address issues for children? How do you work together? What is in place to ensure that you work together (probe: referral pathways, platforms for discussions issues affecting children, combined projects, etc.) d. Community at large (e.g. changes in awareness, response, advocacy for child protection, etc.) 7. What role do you think you played in assisting children to experience the above mentioned changes? How do you think their lives (children and their families) would have been different had it not been for your assistance? 8. What do you think enabled you to support these changes for children? AAHT UGM End of Project Evaluation 91 9. What about the children who have not changed? Why has the VV program not been able to change their lives? What hindered you from supporting changes for these children? Sustainability and Recommendations Instructions to facilitator Time: 15 mins Method: Plenary Purpose: To what extent has the results been sustained? 10. Going forward, what do you think are the key things that need to change, or be in place, to ensure that the well-being of children is improved? 11. In terms of the changes you heard about/observed for children and in terms of the community capacity, have these changes lasted over time, or have they faded? 12. Going forward, what do you think are the key things that need to change, or be in place, to ensure that the lives of children is improved? 13. Is there anything else you would like to add? Do you have any questions? AAHT UGM End of Project Evaluation 92 PROJECT: Vana Vetu Program (Anglican AIDS and Healthcare Trust – AAHT) Partners Interviewer instructions: This questionnaire should be used as a discussion guide, not as a strict interview schedule. You may find questions and answers repetitive if you stick to it, so be flexible. If you think a question has already been answered, move on. Use the questionnaire as a checklist to see that you have covered all the issues. Take your time. DO NOT LEAVE POINTS HANGING IN THE AIR. PROBE and ASK WHY WHY WHY! Relax and listen actively. If the person does not have anything to say to a question, do not leave it blank. Write N/A for no answer. Province: Site: Respondent’s name: Role of the respondent: Respondent’s telephone no.: Date and Venue of interview: Interviewer’s Name:  Southern Hemisphere Consultants has been commissioned by USAID through FHI 360 (Umbrella Grants Management project) to conduct a Summative Assessment of the Vana Vetu (VV) program.  The purpose of this evaluation is to understand the outcomes and impact of the program for children and community members. This means that we will explore the changes they have experienced as a result of the Vana Vetu program.  The results of this evaluation will not determine future funding. The intention is to gather information to generate an understanding of what works and does not work in terms of achieving the expected outcomes related to the well-being of children within a specific context. This information will be shared with government departments.  For this reason, we are interested in getting information from project stakeholders who are willing to share openly and honestly about the strengths and achievements but also the challenges faced.  While information will remain anonymous, it will be shared as part of the consolidated report. No names will be mentioned in the report.  The interview will take approximately 1.5 hours. Are you willing to proceed? Introduction AAHT UGM End of Project Evaluation 93 1. Tell me what you know about the VV program in your community: key objectives, key activities, how it is being rolled out? Relevance Key Evaluation Question: To what extent was the AAHT able to address the needs of children within the community? 2. Why was the VV Program initiated in communities? What were the problems that VV attempts to address? How did you know that these were problems within the community? 3. How do the services offered by VV address the needs of the community? 4. What are some of the gaps in VV services that hinder the achievement of improving children’s lives (in terms of financial security, education, child protection and health)? 5. What makes VVs different/unique as a service provider working towards improving children’s lives? Impact Key Evaluation Question: What were the most significant changes brought about by AAHT improving the lives of OVC in targeted communities? What have been the more immediate/intermediate outcomes for the groups that were targeted? 6. What were some of the key changes observed/reported as a result of the VV Program? Think about the following levels and mention some examples of the changes you have seen:  Children targeted by VV (self esteem, self-image of children in community, confidence, communication skills, goal setting, presentation skills, ability to deal with challenges, positive decision making, ability to overcome challenges, health living, education)  Household level: guardians responsibility in child care, Household Economic Strengthening – how benefitted child, etc.  School (academic progress of children)  Community (do children feel safe and protected in the community – particularly looking at the contributors/actors in the AAHT program e.g. CCFs) 7. To what extent has VV built a community (parents, church/parish, team leaders, care workers, DOVC, CCCFs) that is competent/able to address issues faced by OVC?  What kind of capacity has been developed within the community around child protection and HIV?  How has sharing of knowledge of services for referrals been facilitated through the program? 8. Tell me about the partners/service providers VV works with. Who are they? How do you work together? What the strengths and challenges of your relationship?  How co-ordinated is their response? What co-ordination mechanisms are in place?  What role do CCCFs play in protecting children?  What role did the VV play in setting up the above? Recommendations We are about to end the interview. AAHT UGM End of Project Evaluation 94 9. Going forward, what do you think are the key things that need to change, or be reinforced, to ensure that the well-being of children is improved? 10. Is there anything else you would like to add? Do you have any questions? Fieldworker Observations: Thank you for taking the time to share this information with me! AAHT UGM End of Project Evaluation 95 PROJECT: Vana Vetu Program (Anglican AIDS and Healthcare Trust – AAHT) Team Leaders (this could be referred to Parish/Community Leaders also) Interviewer instructions: This questionnaire should be used as a discussion guide, not as a strict interview schedule. You may find questions and answers repetitive if you stick to it, so be flexible. If you think a question has already been answered, move on. Use the questionnaire as a checklist to see that you have covered all the issues. Take your time. DO NOT LEAVE POINTS HANGING IN THE AIR. PROBE and ASK WHY WHY WHY! Relax and listen actively. If the person does not have anything to say to a question, do not leave it blank. Write N/A for no answer. Province: Site: Respondent’s name: Role of the respondent: Respondent’s telephone no.: Date and Venue of interview: Interviewer’s Name:  Southern Hemisphere Consultants has been commissioned by USAID through FHI 360 (Umbrella Grants Management project) to conduct a Summative Assessment of the Vana Vetu (VV) program.  The purpose of this evaluation is to understand the outcomes and impact of the program for children and community members. This means that we will explore the changes they have experienced as a result of the Vana Vetu program.  The results of this evaluation will not determine future funding. The intention is to gather information to generate an understanding of what works and does not work in terms of achieving the expected outcomes related to the well-being of children within a specific context. This information will be shared with government departments.  For this reason, we are interested in getting information from project stakeholders who are willing to share openly and honestly about the strengths and achievements but also the challenges faced.  While information will remain anonymous, it will be shared as part of the consolidated report. No names will be mentioned in the report.  The interview will take approximately 1.5 hours. Are you willing to proceed? AAHT UGM End of Project Evaluation 96 Introduction 1. Tell me about the VV program in your community: key objectives, how it is being rolled out in your parish/community, key activities, key role players Relevance Key Evaluation Question: To what extent was the AAHT able to address the needs of children within the community? 2. Why was the VV Program initiated in communities? What were the problems that VV attempts to address? How did you know that these were problems within the community? 3. How do the services offered by VV address the needs of the community? 4. What are some of the gaps in VV services that hinder the achievement of improving children’s lives (in terms of financial security, education, child protection and health)? 5. What makes VVs different/unique as a service provider working towards improving children’s lives? Impact Key Evaluation Question: What were the most significant changes brought about by AAHT improving the lives of OVC in targeted communities? What have been the more immediate/intermediate outcomes for the groups that were targeted? 6. What were some of the key changes observed/reported as a result of the VV Program? Think about the following levels and mention some examples of the changes you have seen:  Children targeted by VV (self esteem, self-image of children in community, confidence, communication skills, goal setting, presentation skills, ability to deal with challenges, positive decision making, ability to overcome challenges, health living, education)  Household level: guardians responsibility in child care, Household Economic Strengthening – how benefitted child, etc.  School (academic progress of children)  Community (do children feel safe and protected in the community – particularly looking at the contributors/actors in the AAHT Program e.g. CCFs) (Note to fieldworker: Please take note of specific significant stories of change for children that you could possibly use for your in-depth interview…check whether an interview with this child will be possible) 7. For those children who did not get the intended benefits of the VV program, what were the challenges/barriers? 8. To what extent has VV built a community (parents, church/parish, team leaders, care workers, DOVC, CCCFs) that is competent/able to address issues faced by Orphans and vulnerable children?  What kind of capacity has been developed within the community around child protection and HIV?  How has sharing of knowledge of services for referrals been facilitated through the program? 9. Tell me about the partners/service providers you work with. Who are they? How do you work together? What the strengths and challenges of your relationship? AAHT UGM End of Project Evaluation 97  How co-ordinated is your response? What coordination mechanisms are in place?  What role do CCCFs play in protecting children?  What role did the VV play in setting up the above? Effectiveness Key Evaluation Question: What were the key enablers and barriers in meeting project objectives and outcomes? 10. Thinking about the changes reported/observed, what were the strengths in achieving these? What were the challenges that prevented you from achieving even further change? Probe for the following categories  Community response  Staff (Volunteers and CCFs) skills  Management of the Program  Monitoring, evaluation and reflective processes  Communication and structure (e.g. between HQ, diocese, parishes, CCFs, Volunteers)  Leadership (from HQ and church) 11. What are the gaps in being able to provide holistic services (prevention, responsive) to support the well-being of children? Sustainability and Recommendations Key Evaluation Question: To what extent has the results been sustained? 12. Will CCWs, CCCFs, church, parish be able to continue supporting OVC if VV was over? What structures exist to enable continuity (e.g. fundraising, proposal writing, community ownership, etc.)? What prevents this from happening? 13. In terms of the changes you have heard about/observed for children and the community, have these changes lasted over time, or have they faded? We are about to end the interview. Recommendations: 14. Going forward, what do you think are the key things that need to change, or be reinforced, to ensure that the well-being of children is improved? 15. Is there anything else you would like to add? Do you have any questions? Fieldworker Observations: Thank you for taking the time to share this information with me! AAHT UGM End of Project Evaluation 98 APPENDIX 8: LIST AND DATES OF SITE VISITS AND FIELDWORK Province Site Date Interviews conducted Western Cape Anglican AIDS and Healthcare Trust Headquarters, Braehead House, Kenilworth, Cape Town April 17, 2012 (site visit only) April 24, 2012 Interviews: CEO, Program Manager Focus Group: Program staff Western Cape Diocese of False Bay, St. Paul’s, Faure-Urban May 2-4, 2012 Interviews: School Principal, Bishop, HIV coordinator, Team Leader, DOVC, Child Focus Groups: Caregivers, CCWs, Children Survey: Children Eastern Cape Diocese of Umzimvubu, St Georges Parish, Mt Frere (Rural) May 2-4, 2012 Interviews: DOVC, Team Leader, Child Focus Groups: Community members, CCCFs, Children Survey: Children Eastern Cape Diocese of Port Elizabeth, James Calata Parish, Goven Mbeki Township (Urban) May 2-4, 2012 Interviews: HIV coordinator, DOVC, Partner, Child Focus Groups: CCCF, Children Survey: Children Limpopo Diocese of St Mark The Evangelist, Jane Furse, Sekhukhune - Rural May 7-9, 2012 Interviews: DOVC, Team leader, Child Focus Groups: Caregivers, CCWs Survey: Children AAHT UGM End of Project Evaluation 99 APPENDIX 9: EVALUATION WORK PLAN                                                              22 Data capturing can happen over such a short period only if sample remains small Output Activity Date Person Document review FHI to send all documents to SHC April 2 Alison Review documents April 3 – 12 (ongoing until report) Wilma and Adrian Front-end analysis Conduct initial telephonic meeting with Client April 11 SHC and FHI Front-end analysis report sent to client: Implementation strategy and methodology, evaluation work plan, sample and method, roles/responsibilities, key outcomes and questions and tools April 13 Wilma & Vicki Invoice 1 submitted Submit to Alison (Deliverable 1 and 2: Implementation Strategy and Evaluation Work plan) April 13 Loretta Evaluation planning workshop Evaluation initiation workshop with AAHT April 17 Wilma & Vicki Amendments to work plan, instruments, etc. sent to UGM April 20 Wilma Instrument design First draft evaluation tools April 2 Wilma Second draft evaluation tools April 5 Wilma Final evaluation tools April 13 Wilma Send to/Share with AAHT and FHI 360 for approval April 17 Wilma Amend instruments based on feedback April 20 Wilma Translation of quantitative instrument April 23-24 QRS Fieldwork co￾ordination Database sent to SHC for sampling and contact details for contacting sites selected April 18 AAHT Letter drafted for sites and approved and signed by AAHT on letterhead April 18 (sent) Vicki Roll out plan for f/w developed (draft): Which sites, when, timeframes, requirements, etc. April 18 Vicki Sites contacted (phone and fax letter with times and requirements for fieldwork) April 16 -25 Vicki Set up fieldwork with key informants April 16 – 18 Vicki Prep fieldworker brief April 20 Vicki and Wilma Training of fieldworkers Briefing of fieldworkers: overview of evaluation, ethics, instruments, sample, roll out plan, contracts, fieldworker report, etc. April 23 Wilma & Vicki Fieldwork: Conduct key informant interviews April 24 Wilma & Adrian Conduct interviews (SS, FG and survey) in sites, transcribe and record May 2-11 Fieldworkers (including Adrian & Wilma) Quality checking by SHC (and filing) and feedback to fieldworkers May 9 Wilma & Vicki Send/post quant interviews to SHC/QRS (transcripts and recordings) (Overnight) May 11 Fieldworkers and QRS Invoice 2 Submit to Alison (together with deliverable 3: Data Analysis Plan) May 4 Loretta Data analysis Quant data captured and analyzed according to tabs (including correlations, graphs, significance tests) May 14-16 QRS Design and Set up code sheet for qualitative data analysis and development of analysis plan April 25-26 Wilma & Vicki Send data analysis plan, dummy tables and graphs to FHI 360 April 27 Wilma Code qualitative data in NVIVO22 May 14-18 Adrian & Wilma Formulate key findings May 17-25 Vicki, Adrian and AAHT UGM End of Project Evaluation 100 Wilma Conduct presentation Conduct feedback presentation to UGM May 28 – June 1 Wilma/Dena Invoice 4 Submit to Alison (with deliverable 4: Presentation) June 1 Loretta Report writing Determine AAHT and USAID requirements for report writing structure: who writing for, approximate length of long and short reports April 30 Wilma Report writing and edit (quant, qual, graphs, photos, summary diagrams) May 28 – June 15 Wilma and Adrian Invoice 5 Submit to Alison (with deliverable 5: Draft report) June 15 Loretta Finalize report Final report and 2 page community friendly report June 29 Wilma and Adrian Closeout letter submitted to client and report printed June 29 Vicki & Wilma Invoice 6+7 Submit to Alison (deliverable 6 and 7: Final report and 2 page summary) June 29 Loretta AAHT UGM End of Project Evaluation 101 APPENDIX 10: COMPOSITION OF EVALUATION TEAM Role Person Responsibilities Project Manager Wilma Wessels￾Ziervogel Management of the evaluation (client liaison, finance management, contract management, fieldwork management, management of work plan), quality control, instrument design, fieldwork, data analysis and report writing Project Leader Dena Lomofsky Technical assistance and oversight Project Coordinator Vicki Tully Management of fieldwork process and project support Senior Consultant Adrian Di Lollo Data analysis and report writing Quantitative data analysts QRS (Cletus Dube) Capturing, cleaning and analyzing quantitative interviews, translation of quantitative interview schedules Fieldworker Limpho Klu Fieldwork in Limpopo Fieldworker Khanyisa Manzini Fieldwork Eastern Cape (rural) Fieldworker Noxolo Bill Fieldwork Eastern Cape (urban) Fieldworker Helen Barnard Fieldwork Western Cape AAHT UGM End of Project Evaluation 102 APPENDIX 11: REFERENCES  Anglican AIDS and Healthcare Trust. Annual Progress Report. 2009.  Anglican AIDS and Healthcare Trust. Annual Progress Report. 2010.  Anglican AIDS and Healthcare Trust. COP10 Partner Project Plan: AED UGM Partner Profile.  Colvin, Mark. “End of Term Review of the NSP 2007-2011: Final Report.” November 2011.  Department of Social Development. NAP Indicator Reporting Tool: Final.  Greater Capital. AAHT Mid-Term Evaluation Final Report. January 2011.  Lomofsky, Dena and Vicky Tully. Program Framework, Program Theory and Results Chain for Vana Vetu Program of Anglican AIDS Healthcare Trust. September 2011.  Mahery, P., L. Jamieson, and K. Scott. Children’s Act Guide For Child and Youth Care Workers. Cape Town, South Africa: Children’s Institute (University of Cape Town) and National Association of Child and Youth Care Workers (NACCW), January 2011.  South African National AIDS Council (SANAC). National Strategic Plan on HIV, STIs and TB, 2012- 2016.  The United States President’s Emergency Plan for AIDS Relief, South Africa Strategic Information (SASI) Manual: For Implementing Partners in South Africa. April 2011.