USAID/South Africa Umbrella Grants Management Project End of Project Partner Evaluation WOZ’OBONA/SEKHUKHUNE EDUCARE PROJECT July 2012 This publication was produced for the United States Agency for International Development. It was prepared by Andries Mangokwana, Patricia Martin, and Oscar Musandu-Nyamaharo of Break the Chains Development Services. USAID/South Africa Umbrella Grants Management Project End of Project Partner Evaluation WOZ’OBONA/SEKHUKHUNE EDUCARE PROJECT This report is made possible by the generous support of the American people through the U. S. Agency for International Development (USAID), under the terms of APS 674- 07-011, through United States Government’s President’s Emergency Plan for AIDS Relief, operated by FHI 360. The contents and opinions expressed herein do not necessarily reflect the views of USAID or the United States Government. Woz’obona UGM End of Project Evaluation Report i ACKNOWLEDGEMENTS We would like to thank FHI 360 for appointing Break the Chains Development Services (BTC) to conduct this evaluation. It has been a learning experience for all parties involved and has given Break the Chains Development Services invaluable insight into the realities and challenges faced by rural communities in dealing with issues affecting orphans and vulnerable children in South Africa. We would also like to thank Woz’obona/Sekhukhune Educare Project (SEP) for assisting us with fieldwork logistics and identifying government stakeholders and community members to be interviewed and we would like to thank them for taking the time to meet with us for interviews. Thank you to the evaluation team comprising of Andries Mangokwana as the Team leader, Patricia Martin as the Children’s Expert, and Oscar Musandu-Nyamaharo, Limpho Klu, Gemi Malau and Kashfi Mahmud as consultants and Hlamalani Rikhotso as the Project Coordinator. Furthermore, we would like to thank all stakeholders who participated in the evaluation and who gave us valuable insight into what the program has achieved, its challenges, and lessons learned. Lastly and most importantly, we would like to thank the children who participated actively in the focus group discussions and provided us with valuable insight into their experiences with regard to issues that affect them in their communities.    TABLE OF CONTENTS I. INTRODUCTION ..................................................................................................................... 1 II. BACKGROUND ........................................................................................................................ 2 III. METHODS ................................................................................................................................... 6 IV. FINDINGS ................................................................................................................................. 11 V. CONCLUSIONS ...................................................................................................................... 32 VI. RECOMMENDATIONS ......................................................................................................... 33 VII. APPENDICES ............................................................................................................................ 36 APPENDIX I: PERSONS CONTACTED .................................................................................... 36 APPENDIX II: SCOPE OF WORK ............................................................................................... 37 APPENDIX III: EVALUATION TEAM AND STAKEHOLDER INTERVIEWS ................... 39 APPENDIX IV: INSTRUMENTS DESIGNED ............................................................................. 41 APPENDIX V: WORK PLAN ........................................................................................................ 63 APPENDIX VI: REFERENCES ........................................................................................................ 64 Woz’obona UGM End of Project Evaluation Report iii ACRONYM LIST ABET Adult Basic Education and Training BTC Break the Chains Development Services CCW Child Care Worker DOA Department of Agriculture DOE Department of Education DOH Department of Health DoHA Department of Home Affairs DSD Department of Social Development ECD Early Childhood Development HES Household Economic Strengthening M&E Monitoring and Evaluation MoU Memorandum of Understanding NSP National Strategic Plan OVC Orphans and Vulnerable Children PEPFAR President’s Emergency Plan for AIDS Relief SAG South African Government SAPS South African Police Service SASSA South Africa Social Security Agency SEP Sekhukhune Educare Project STATS SA Statistics South Africa UGM Umbrella Grants Management USAID United States Agency for International Development    EXECUTIVE SUMMARY INTRODUCTION UGM is a five year grants management program administered by FHI 360 with funding from the U.S. Agency for International Development (USAID). Through UGM, FHI 360 provides funding and technical assistance to USAID-selected NGO partners who provide HIV/AIDS services at local, provincial, and national levels in South Africa. FHI 360-UGM seeks to promote high quality service delivery in alignment with the priorities and goals of the South African government’s HIV/AIDS framework. FHI 360-UGM provides specialized capacity building and support services to build partners’ skills and competencies in program management, governance, human resource development, budgeting and finance, and monitoring, evaluation, and reporting. The FHI 360-UGM project objectives are to 1. Provide grants to USAID/PEPFAR partners that ensure an adequate resource flow to foster scale-up of activities 2. Implement effective monitoring, evaluation, and reporting systems to assess and document activities 3. Provide ongoing capacity building to support and enhance scale-up of activities, and sustainability of activities and partners Since 2007, FHI 360-UGM has supported thirteen South African NGOs including Woz’obona. The organization has received total funding through the UGM of R9,643,138. Woz’obona (also known as the Sekhukhune Educare Project [SEP]) is a NGO which provides a range of services designed to improve the well-being of orphans and vulnerable children (OVC) in the rural Makhuduthamaga Municipality of Limpopo Province. The organization uses an integrated approach to development working with the child, the household, the community, and government institutions to ensure that the overall environment is beneficial to OVC. This report contains an evaluation of Woz’obona/SEP’s work under its five year grant through the FHI 360 Umbrella Grants Management (UGM) project. In response to one of the most severe national HIV epidemics, the South African Government (SAG) has developed a comprehensive HIV/AIDS plan in close coordination with USAID and President’s Emergency Plan for AIDS Relief (PEPFAR). USAID/PEPFAR, through the FHI 360-UGM project, supported Woz’obona between 2007 and 2012 in the implementation of services to OVC, their households, and communities. The project sought to improve the well-being of OVC in the area through ensuring that they had greater access to essential services. The Woz’obona program strategy is rights-based and focuses on the development of innovative models and the facilitation of sustainable projects through collaboration with like￾minded institutions. The following table shows Woz’obona’s overall goal and objectives. Overall Goal Program Objectives Create an enabling environment in which children are stimulated to develop and learn while receiving proper care and protection Build resilience amongst OVC through provision of essential services Strengthen the capacity of households and communities to respond to the need of OVC Improve the livelihoods of poor families through increased food security and enhanced income generating opportunities Woz’obona UGM End of Project Evaluation Report v Strengthen the capacity of Woz’obona to sustain its work Strengthen the collaboration with other role players involved in child care and development EVALUATION METHODS A participatory qualitative research methodology was chosen and non-probability sampling was used. The evaluation methodology included three core activities  a document review  interviews with key informants, including both former and current beneficiaries, household members, and representatives of community and government agencies working with the program  focus groups with beneficiaries, Woz’obona/SEP staff, community caregivers, and household members FINDINGS The evaluation revealed that the services and support that Woz’obona/SEP sought to provide were well-suited to meet the specific needs and challenges of OVC living in remote, traditional communities, which are marked by high rates of poverty, unemployment, and HIV prevalence. Table 1 illustrates the number of children reached against targets set annually. Table 1: Overall Target versus Performance Results Year Target Performance Performance % % Deviation 2008 2,800 3,006 107% 7% 2009 4,500 4,563 101% 1% 2010 3,500 3,871 111% 11% 2011 3,500 3,713 106% 6% 20121 3,500 1,747 50% -50% Total 17,800 16,900 95% -5% Overall, the findings show that Woz’obona/SEP has been able to respond to the needs of children around access to health, psychological support, and education with the sole intention to build the resilience of OVC. There were, however, needs that emerged during implementation where no interventions were provided. These gaps include children feeling unsafe both in the community and at school, widespread bullying in schools, high rates of teenage pregnancies, and the provision of information on tuberculosis (TB) prevention and treatment in line with South Africa’s 2012 HIV/ AIDS and TB National Strategic Plan (NSP). The most significant changes which have been effected through Woz’obona’s intervention include  Improvement, expansion, and adoption of an effective model of Early Childhood Development (ECD) Provision: The program has demonstrated a model of home-based ECD provision which intensifies the ability to reach children in remote and impoverished areas by training community members to provide services in their homes.                                                                   1 This is for semi-annual performance.  Access to psychosocial support: Lay counselors and community child care workers (CCWs) were able to reach a large number of OVC and listen to their concerns. Prior to this project, children were often isolated and unable to voice their concerns to a caring adult. Through Woz’obona’s intervention, children became better able to cope with the trauma of losing one or both parents.  Improving school performance and self-esteem: The extra classes in math, science, and English, combined with the provision of homework support, have had positive changes among beneficiaries.  Improving access to healthcare: Access to immunization has improved, and there is positive impact on awareness, monitoring, and responses to children’s health needs and nutritional requirements.  The needs of households were met through the Household Economic Strengthening (HES) program which included assistance in accessing grants, asset growth and protection, and income growth activities (through poultry raising and knitting projects). The HES program has improved economic opportunities for 194 participating households by empowering beneficiaries with a range of skills and options to improve their household income, reduce debt, and pay for their children’s schooling. Evidence suggests that the practices developed in the program may be sustained.  Community: The program has had a substantial impact in raising community awareness about the situation of OVC and their needs. CONCLUSIONS The evaluation findings show that the Woz’obona program was relatively successful in achieving its overarching goal of creating an enabling environment in which children are stimulated to develop and learn while receiving proper care and protection. The Woz’obona/SEP initiative has applied a model of development that engages a wide range of service providers. In so doing, the organization has ensured that state resources are accessible to OVC and Woz’obona has mobilized existing community capacity to support their program interventions. The program has reflected the core values of the organization—to place the needs of the child at the heart of all activities. The community education and awareness programs, strengthening of household economies, and engagement of people close to the child have played an important role in creating an enabling environment for OVC. There were a number of challenges to the design, development, and implementation of the program. These included the lack of baseline and contextual information against which to assess needs, develop interventions, and measure progress; traditional attitudes and practices; informal and flexible working partnerships; and insufficient resources. Despite these challenges, this evaluation shows that the program has been successful in realizing its key goal and strategic objectives. Its success provides a foundation upon which Woz’obona should build the next stage of its program, which should be more strategic and sustainable as measured against the prevailing policy and resource environment. Moreover, its experiences can be shared with other organizations, both government and NGOs, to learn strategies for designing and implementing effective rural community-based programs to address the needs of vulnerable children. RECOMMENDATIONS  Advocacy and Engagement The next stage of the program should look at moving towards a stronger focus on sustainable beneficiary-driven advocacy. This will include building capacity of children, parents, and volunteers to tackle the underlying causes of vulnerability that exist in the targeted communities. Woz’obona UGM End of Project Evaluation Report vii In addition, this requires a stronger emphasis on empowering children and their caregivers with knowledge of their rights and remedies for redress in current national policies and programs, rather than direct provision of material support and psychosocial services.  Strengthen and Formalize Partnerships Through Consultative Processes – Convene a results-sharing workshop with stakeholders and agree on the way forward. – Explore the possibility of convening quarterly meetings with key stakeholders to coordinate and collaborate on program activities. – Institutionalize partnerships through Memoranda of Understanding (MoU) with relevant government departments and ensure that resources are leveraged to avoid duplication.  Strategic Shaping of Interventions to Address Drivers of Risk and Vulnerability – It is recommended that Woz’obona implement a regular review processes to assess the emerging needs of children in the community, as measured against existing policies and service delivery, and realign its program focus and intervention design to ensure the best fit. – The choice and design of interventions related to HIV prevention education should be guided by national, provincial, and district level information about trends in sexual behavior, age of sexual debut, and the gender dimensions informing relationships in rural communities.  Opportunities to Scale-up Program Delivery There would be value in – developing a standard costing model for the different services the program has provided – defining the essential job profiles to implement the program to the required standards – identifying different agencies which may be interested in supporting the continuation and expansion of the program  Address Sustainability of the Program – Diversify funding sources – Woz’obona should review its programs to ensure maximum alignment with the policy and programmatic priorities of the South African government as a means of ensuring that investments in interventions will continue to yield returns through complementary state interventions. In addition, close alignment with state objectives creates greater opportunities to secure funding from departments and development partners. – The piloting of the ECD model by the Limpopo Department of Social Development offers another opportunity for program sustainability and funding. ECD and Home-Based Community Care are part of the Expanded Public Works Program under the social cluster. If the pilot proves successful, Woz’obona is well-placed to be selected as an implementing agent. – Woz’obona’s greenhouse was built with the hope of implementing agricultural activities through the use of greenhouse technology; this intervention has the potential to strengthen the economy of local households. Construction has taken longer than projected due to bureaucratic requirements related to building such a structure. During the evaluation, a representative from the Department of Agriculture noted that through Woz’obona/SEP training, “parents now have knowledge and skills for water-efficient vegetable home gardens, income generation achieved with home gardens, nutrition improved, and malnutrition and vitamin deficiency reduced.”  Monitoring and Evaluation It is recommended that the organization should – Institutionalize monitoring and evaluation (M&E) processes across all its programs, and move towards developing and implementing a fully systematized M&E plan. – Ensure the collection of baseline and routine data in relation to service provision and program objectives to measure progress and review program efficiency and effectiveness. – Update the project’s database of beneficiaries and make this available for use by other institutions working to serve OVC. – Clearly document the model and practices used in the program and share with relevant stakeholders. Woz’obona UGM End of Project Evaluation Report 1 I. INTRODUCTION Woz’obona (also known as the Sekhukhune Educare Project) is an NGO which provides a range of services including material, psychosocial, health, and educational support designed to improve the well-being of OVC in the Makhuduthamaga Municipality of Limpopo Province. Since 2007, Woz’obona has received PEPFAR funding through FHI 360-UGM to implement a program that aims “to create an enabling environment in which children are stimulated to develop and learn while receiving proper care and protection.” The project has sought, during this period, to impact positively on the following  Resilience of OVC through the provision of essential services  Strengthening the capacity of households and communities to respond to the needs of OVC  The livelihoods of poor families caring for OVC through increased food security and enhanced income generating opportunities  The capacity of Woz’obona to sustain its work  Collaboration between Woz’obona and like-minded players to collectively promote the care and development of OVC in the targeted communities USAID has requested FHI 360-UGM to undertake a summative evaluation of Woz’obona’s OVC program funded by PEPFAR. Break The Chains Development Services (BTC) was contracted to undertake the evaluation of Woz’obona/SEP. The purpose of the evaluation was to  determine whether the program objectives were achieved  evaluate the key program outcomes and impact related to improving the health and wellbeing of the targeted beneficiaries  generate knowledge by identifying the program enablers and barriers Three evaluation questions established by FHI 360-UGM were used as the parameters for the evaluation. 1. To what extent was Woz’obona able to meet the needs of target populations within the community? 2. What were the most significant changes brought about by Woz’obona improving the well-being of OVC in targeted communities? 3. What were the key enablers and barriers in meeting project objectives? This report provides an overview of the program’s intent, purpose, and objectives as well as a summary of the interventions initiated by Woz’obona/SEP over the past five years. Furthermore, the findings of the evaluation with regards to impact and sustainability are presented. The report concludes with a summary of the key findings and a number of recommendations to further build upon the foundation laid by Woz’obona during the project period through replication of best practices and the further development of Woz’obona as it continues work in targeted communities.    II. BACKGROUND THE UMBRELLA GRANTS MANAGEMENT PROJECT UGM is a five year grants management program administered by FHI 360 with funding from the U.S. Agency for International Development (USAID). Through UGM, FHI 360 provides funding and technical assistance to USAID-selected NGO partners who provide HIV/AIDS services at local, provincial, and national levels in South Africa. FHI 360-UGM seeks to promote high quality service delivery in alignment with the priorities and goals of the South African government’s HIV/AIDS framework. FHI 360-UGM provides specialized capacity building and support services to build partners’ skills and competencies in program management, governance, human resource development, budgeting and finance, and monitoring, evaluation, and reporting. The FHI 360-UGM project objectives are to 4. Provide grants to USAID/PEPFAR partners that ensure an adequate resource flow to foster scale-up of activities 5. Implement effective monitoring, evaluation, and reporting systems to assess and document activities 6. Provide ongoing capacity building to support and enhance scale-up of activities, and sustainability of activities and partners Since 2007, FHI 360-UGM has supported thirteen South African NGOs including Woz’obona. The organization has received total funding through the UGM of R9,643,138. BACKGROUND ON WOZ'OBONA/SEP Organizational Overview Woz’obona is a national organization established in 1988 to provide early childhood development services and support in remote areas lacking access to essential early learning and stimulation services. In 1994, the program commenced work in Jane Furse in the Limpopo Province. In recognition of the need to respect the language and culture of the area (Sepedi), the Woz’obona operations in this area were known as the Sekhukhune Educare Project. The vision of the organization is a child-friendly society. Its mission is to develop innovative models and facilitate sustainable programs through collaboration with like-minded institutions and enable communities to improve the lives of children. The three primary developmental strategies employed by the organization are:  enabling and mobilizing childcare resources within communities to meet the needs of children  offering nationally recognized skills development and training  ensuring and facilitating a proper development foundation for children Program Goal, Objectives, and Key Activities Between 2007 and 2012, the program sought to achieve the overarching goal of reducing the impact of HIV/AIDS among OVC in the Makhuduthamaga Local Municipal by creating an enabling environment in which children are stimulated to develop and learn while receiving proper care and protection. The realization of this goal was furthered through key activities designed around five strategic objectives (listed in the table below) which were shaped by the needs of OVC in the targeted communities. Table 1: Strategic Objectives Woz’obona UGM End of Project Evaluation Report 3 Woz’obona Program Objective and Activities Program Objectives Key Activities Build Resilience Amongst OVC Through Provision of Essential Services  Facilitate family math and science sessions especially for OVC not enrolled in preschool  Facilitate extra classes in math, science, English, and accounting and provide homework assistance  Facilitate HIV-prevention education  Organize regular immunization campaigns  Facilitate appropriate child feeding workshops  Facilitate referrals to health facilities  Raise awareness on child illnesses  Facilitate memory work sessions with children and their households  Identify and refer children with trauma for professional counseling Strengthen the Capacity of Households and Communities to Respond to OVC Needs  Conduct children’s rights awareness workshops during community sharing sessions  Conduct participatory theaters in communities on children’s rights  Conduct community dialogue sessions on children’s rights. Improve the Livelihoods of Poor Families Through Increased Food Security and Enhanced Income Generating Opportunities  Assist households to participate in voluntary savings and loans initiatives  Assist households to start income generating projects and small businesses  Facilitate household budget workshops  Conduct referrals and form linkages with relevant stakeholders  Conduct small business training Strengthen the Capacity of Woz’obona to Sustain its Work  Support the design of innovative and unique programs  Train staff in proper accounting and reporting practices  Develop good financial and M&E reporting systems  Develop good fundraising proposals  Develop sustainability plans Strengthen Collaboration with Other Players Involved in Child Care and Development  Conduct partnership recruitment and marketing campaigns (making marketing and awareness presentations to potential partners) Project Model The project has a dual character. It is a rights-based model premised on the promotion, realization, and respect for children’s rights. It is also a developmental model, with a focus on building sustainable capacity for the care and resilience of OVC. Woz’obona’s model is led by the phrase “Follow the child”, a philosophy which guided Maria Montessori in the development of responsive preschool methodologies. Woz’obona’s program has “followed the child” in the development of its interventions. It has followed the children in order to learn what is happening in their lives, to learn what their needs are, and to respond to those needs through the program design and methodology. The child is at the center of the program’s model of family- and community-driven care and support. Woz’obona recruited community volunteers, family members, and children, and capacitated and supported them to work together at a family, community, and institutional level to provide care and support to the targeted children. The immediate objective of the project is to identify children who are living in vulnerable situations, assess their needs, and provide appropriate care, protection, development, and stimulation services to improve their well-being. Woz’obona operates under the philosophy that a holistic, environmental approach to service delivery is essential for the sustained improvement of OVC’s well-being. This intervention is contextualized within this approach and, the program promotes the strengthening of the broader care environment through the following interventions:  direct interventions to improve household economies and financial capacity such as the promotion of savings/loans clubs  mobilizing and capacitating community members and organizations to play an active and ongoing role in monitoring and improving children’s well-being through training on the identification of vulnerable children and the provision of appropriate support  assessing the sufficiency of State services to meet the needs of vulnerable children, facilitate children’s access to the available services, and advocate for the improved availability and delivery of services Management Structure The National Director of Woz’obona is based in Pretoria. The daily activities of Woz’obona/SEP are managed by a Regional Manager and a Program Coordinator, both of whom work from the Jane Furse offices. In addition, they are supported by the Finance Manager and Monitoring, Evaluation and Reporting Officer. Organizational oversight is undertaken by a Board comprising representatives from government, civil society, and the community. Scope and Budget This program dates back to 2002, when Woz’obona/SEP initiated the Safety Nets Project as a focused response to the growing needs of OVC in the Sekhukhune community. The project began with funding from the Nelson Mandela Children’s Fund and quickly expanded into 13 villages. USAID/PEPFAR support was provided through FHI 360-UGM from 2007–2012. Project Context The targeted communities are found within the Makhuduthamaga Local Municipality, which is comprised of 146 rural settlements with a population of just over 300,000 people. The social and economic profile of the area is marked by a substantial youth population and high levels of poverty, unemployment, and HIV prevalence. When the program commenced in 2007  The proportion of the population under the age of 20 was 53%.  The unemployment rate was approximately 75%.  Over half (55%) of households indicated they had no source of income at all  Households that did not receive an income depended almost entirely on government social security grants  Over half (56%) of the population had no formal education  The HIV prevalence was 16%.2 The area was previously part of the former South African homeland system. As such, it is marked by historical legacy of poor infrastructure and huge backlogs in the delivery of basic services such as water and sanitation. Water insecurity inhibits the agricultural activity in the area; as a result, food security poses an ongoing challenge to many residents. Adults regularly migrate out of the area to urban centers such as Polokwane, Johannesburg, and Pretoria. Of those with jobs, the government was the primary employer, followed by limited retail and trading services. The area is culturally traditional, and this posed various challenges to the promotion of the rights of women and children. The lower status of women and children inhibits their ability to express concerns, needs, and rights. The program designers were constantly challenged to mobilize                                                                   2Community Survey, Statistics South Africa, 2007 Woz’obona UGM End of Project Evaluation Report 5 community members in promoting and protecting children’s rights in a culturally appropriate manner. III. METHODS EVALUATION METHODOLOGY BTC recruited an evaluation team with substantial experience in evaluation methodologies, social impact studies, and working with vulnerable children and youth. The evaluation team members are listed in Appendix1. The evaluation methodology included three core activities:  a document review  interviews with key informants, including both former and current beneficiaries, household members, and representatives of community and government agencies  focus groups with beneficiaries, Woz’obona/SEP staff, community caregivers, and household members Sampling Given the limited scope of the evaluation, a choice was made to employ non-probability sampling.3 Sampling choices were determined by the purpose of the evaluation, convenience, and the need for simplicity and cost-effectiveness. The sample was drawn from Woz’obona management, staff, caregivers, current children beneficiaries, adult beneficiaries (both current and former), community representatives and government department representatives. Document Review BTC reviewed a number of project and broader contextual documents to gain an understanding of the context, purpose and progress made in achieving the project goals. Project documentation reviewed included  Woz’obona’s Logical Framework  annual work plans from 2007 to 2011/2012  annual reports for 2008/2009 and 2010/2011  program framework, theory, and results chain The purpose of the review was to gain insights into the rationale, logic, interventions, intended outcomes, and indicators of success, as well as to measure progress made against program targets. While the program’s documents were useful in this regard, the absence of a document describing Woz’obona’s implementation model created a gap in the overall picture painted through this stage of the evaluation. Broader contextual documents reviewed included  National and district level statistical reports generated by Statistics SA, including the report of the 2007 and 2010 General Household Survey and the 2007 Community Survey  National policy documents governing the care, protection, and support of orphans and other vulnerable children in South Africa o National Strategic Plan 2007 – 2011 o 2008 South African National HIV Prevalence, Incidence, Behavior, and Communication Survey o Amended National Norms and Standards for School Funding, 2008 o National Department of Health’s National Comprehensive HIV/ AIDS Plan statistics, 2008 o South African Schools Act                                                                   3 Bailey K, Methods of Social research, Second Edition, 1982 Woz’obona UGM End of Project Evaluation Report 7 o National Guidelines on School Uniforms o National OVC Policy Framework, 2005 The broader contextual documents were reviewed to understand the prevailing socioeconomic conditions in areas such as Jane Furse at the commencement of the project. Additionally, the review sought to assess government policy priorities with respect to meeting the rights and needs of children affected by HIV/ AIDS as well as the alignment of this project’s objectives and design with furthering SAG’s goals. This alignment was important for two reasons. First, it is an implicit requirement of the project, given that part of the FHI 360-UGM project rationale is to contribute to the realization of the SAG goals through supporting organizations in South Africa already working in HIV education, prevention, care, and treatment. Second, alignment with national priorities indicates significant responsiveness to the most pressing needs of children affected by HIV/AIDS, as indicated by their inclusion in national OVC policies and programs. Interviews and Focus Groups Preparation for the Interviews and Focus Groups In preparation for the interviews and focus groups, BTC engaged with the leadership of Woz’obona/SEP. Meetings were held with the National Director, as well as the Regional Manager and the Program Coordinator. These initial discussions helped BTC to develop a detailed list of stakeholders that should be engaged in the evaluation and to agree on an approach to ensure a cooperative, positive evaluation process. Engaging with Woz’obona’s leadership enabled BTC to gain a detailed understanding of the organization’s values and approach to working with children and to understand the design of the program. These briefings were particularly helpful in preparing for the evaluation given the intuitive way in which many of the project activities have grown since the program began and the paucity of documentation in tracking these developments. Instrument Design A number of interview and focus group guidelines were developed to ensure that the core evaluation questions were covered and that the beneficiaries had the opportunity and guidance necessary to share, to the maximum extent possible, the diversity, richness, and complexity of their program knowledge and experience as well as its benefits and challenges. A list of all the instruments developed is attached as Appendix 2. Upon the development and client approval of the instrument, a detailed work plan was developed to ensure the timely progress of the fieldwork. The work plan is attached as Appendix 3. Interviews and Focus Groups with Key Informants and Stakeholders Interviews and focus groups were undertaken with the key informants listed in the tables below. A detailed list of stakeholders who were interviewed is attached as Appendix 4 Table 2: Individuals Interviews Held Interviews Stakeholder Group Number of Individuals Woz’obona management 3 Government representatives 8 Community representatives 14 Total number of interviews 25 Table 3: Beneficiary Focus Groups Conducted Focus Groups Stakeholder Number of Focus Groups Number of Beneficiaries Caregivers/volunteers 2 21 Woz’obona staff 1 8 Former beneficiaries 1 11 Adult beneficiaries 2 26 Children- boys (two age-groups4 ) 2 29 Children- girls(two age-groups) 2 28 Total number of people reached 10 123 Interviews with Key Informants Semi-structured purposive interviews were conducted with key informants. The interviews with the Woz’obona management focused on the background and history of the program in addition to program design, strategy, relevance, effectiveness, efficiency, impact, and recommendations. Interviews with the government and community representatives focused on their involvement with the program; their experience of the program’s effectiveness, relevance, and impact; and their overall impressions of the program. Focus Groups with Woz’obona Staff, Caregivers, Former Child, and Adult Beneficiaries Focus groups were held with staff from Woz’obona/SEP’s Jane Furse office, caregivers, and former child beneficiaries to gain insight into the effectiveness and impact of the program. The staff focus group dealt with the project’s background as well as its implementation, effectiveness and impact. A focus group was held with caregivers to determine their involvement in the project and their experience of the effectiveness, relevance, and impact of program. The focus group with former child beneficiaries (who are now over 18 and have graduated from the program) focused largely on the outcomes and impact that the program had on their lives. The focus group with the adult beneficiaries focused on their involvement in the HES component of the program. Participants discussed the successes of the program and the positive impacts on their economic well-being as well as the challenges and lessons learned. Focus Groups with Current Child Beneficiaries Four focus groups were held with boys aged between 12 and 14 years and girls aged between14 and 18 years. Discussions focused on establishing their needs, the responsiveness of the program in recognizing and meeting those needs, and the impact and difference that the program interventions had on their well-being. The Rationale and Ethics of the Focus Groups with Children The inclusion of children in the evaluation was founded on their rights to be heard and to participate in matters that affect them. The process was designed around respecting children’s rights to a protected and safe environment, to protection of their dignity, not to be discriminated against and to be treated fairly. In order to realize these objectives, the design and facilitation process were guided by the following principles and practices:  The focus groups sessions were divided into single-sex sessions in order to respect, protect, and promote their rights to privacy and dignity. This created an environment where the children felt safe and free to share fairly sensitive information and personal experiences.                                                                   4 We had 2 groups of boys and girls of between the ages of 12-14 and 15-18 years, respectively Woz’obona UGM End of Project Evaluation Report 9  Children were provided with the opportunity and adequate information to give their informed written consent before participating in the session, as were their caregivers. The process and meaning of informed consent was explained to the children at the beginning of each session to ensure they understood the consent form they had signed.  At the beginning of the session, children were briefed on the purpose of the session and how the information they provided will be used in the evaluation and thereafter. They were reassured that their identities would be protected and the information provided would be treated respectfully and confidentially.  Inasmuch as children have a right to speak, so too do they have the right to be silent. The children were advised that they were not obliged to participate in any discussion if they did not wish to do so.  Children were encouraged to ask questions about the process and the evaluation to enable their informed and meaningful participation in the focus groups.  During the session, the facilitators did not use technical language, jargon or acronyms, or any language that could have made the children feel uncomfortable or stigmatized. Data Analysis The data analysis process involved the following steps: 1. Writing-up of interviews and focus groups The fieldworkers wrote up their interviews and focus-group materials and translated them into English, where applicable. This process was standardized among the different fieldworkers through the use of instruments such as interview report-templates with standardized headings. The transcripts were submitted to the team coordinator to be checked for purposes of data quality assurance. 2. Reading and reviewing the data The information provided in the interview and focus group transcripts was initially reviewed for clarity and coherence and to gain a holistic picture of the data. Where information was not clear, the assigned fieldworker was asked to clarify it. 3. Coding and Analysis The data from the transcripts were then coded analyzed against the core evaluation questions. Upon completion of all transcriptions, they were taken for coding using NVIVO. A code sheet was drawn up and the findings were analyzed according to the three evaluation questions. The evaluation team undertook the thematic analysis. PRESENTATION OF PRELIMINARY RESULTS TO FHI 360 AND WOZ’OBONA/SEP The preliminary results of the data analysis were presented at a meeting with key stakeholders, USAID, PEPFAR, and Woz’obona/SEP, convened by FHI 360 on June 8, 2012. Comments and feedback were received after the presentation. Some of the feedback included concerns that the presentation did not provide sufficient substantive information relevant to the key evaluation questions, that the findings lacked supporting evidence, and that the conclusions and recommendations lacked insight into Woz'obona's model. Furthermore, the presentation did not provide sufficient added value in determining whether the project objectives and key program outcomes and impacts were achieved. These comments informed the development of a draft report which sought to integrate and address the concerns which had been raised. LIMITATIONS OF THE EVALUATION Two significant challenges were experienced in undertaking the evaluation:  a lack of baseline information about the situation and needs of children and households in the targeted communities  no written documentation describing Woz’obona’s project model and focus areas and the rationale linking these to the needs of OVC in Jane Furse at the commencement of the project in 2007 or how these needs were progressively met through implementation of the model within the various focus areas during the life-span of the program Project documentation indicates that the program was preceded by a needs assessment of OVC in the targeted communities; however, this assessment appears to have been almost entirely based on inferences drawn from broader district-level data from the national Community Survey about adult demographic, educational, unemployment, and household grant dependency ratios in the municipal district within which Jane Furse falls. It was not based on child-specific data collected through the national Community Survey or collected by the program coordinators. Consequently, there was no documentation recording the situation or baseline data about the situation and needs of children (or their households) in the targeted communities (an exception is ECD, where some baseline data as to ratio of qualified practitioners was collected and documented). As a result, the findings about the needs of children and their caregivers, as well as the analyses and findings, were based on the recollections and perceptions of those who participated in the interview process and focus groups rather than on comparative baseline data. This created a number of difficulties for the analysis of information. The evaluation team sought to fill this gap in documentation by referring to national and district level data sources and policies documenting and responding to the situation and needs of OVC and their households from similar rural socioeconomic backgrounds in 2007. Consequently, there was a strong reliance on verifiable national and district level contextual documents to inform the analysis and conclusions drawn on the appropriateness and adequacy of the program response to OVC needs at program start, as well as the impact analysis. The project staff attributed the lack of data and documentation to the overriding pressure on service delivery. This resulted in the prioritization of resources to undertake substantive programmatic work rather than a focus on the development and documentation of processes, implementation, and best practices and lessons learned. Woz’obona UGM End of Project Evaluation Report 11 IV. FINDINGS OVERALL PROJECT PERFORMANCE 2007-2012 In 2007, Woz’obona conducted an assessment of the situation and needs of children in Makhuduthamaga by reviewing district level demographic, poverty, educational, unemployment, HIV/AIDS, and migration data collected and published annually by Statistics South Africa. Based on the low levels of education amongst the adult population in the district (56% formal education rate, 18.1% primary education rate, 19.5% secondary education rate, and 0.4% tertiary education rate), the 60-75% unemployment rate, the near-universal dependency on state grants for household income, and the 16% HIV prevalence rate in 2007, Woz’obona concluded that that there was a high number of vulnerable children in the area. It further concluded that, an increasing number of children faced a serious threat to their growth and development, a breakdown in their family structure, and poverty. Woz’obona is an early childhood and youth development organization which provides services designed to respond to the needs of vulnerable children and youth. Based on the preceding assessment and situational analysis, Woz’obona determined that vulnerable children and youth, as well as their households, were exposed to the following risks and gaps in protective services and support5  poor quality of early childhood education services and low enrolment in ECD centers due to parental ignorance of ECD value  exposure to child abuse and neglect with little access to protection services and psychosocial support  illness and lack of access to appropriate health services, including ARVs  lack of stimulation and play time in addition to a lack of available recreational facilities  stigma and discrimination directed at orphans due to the HIV status of their deceased parents with little access to psychosocial support and life skills education to remedy the situation  trauma associated with grief and having to nurse sick parents with no recourse to psychosocial support  high levels of dropout resulting in low levels of literacy and education  risky sexual behavior, especially amongst teenagers, with little access to appropriate life skills education  generally low access to state- and community-based support and services due to a lack of focused targeting towards OVC by state service providers and/or community stakeholders  low levels of household income, high levels of unemployment, and high levels of food insecurity leading to high levels of hunger and an inability to afford the cost of school fees and uniforms Woz’obona developed the project to respond to these needs of OVC and their households through the development of a holistic, community- and family-based, child-centered program of support to provide safety nets OVC. The program sought to create an enabling environment in which children are stimulated to develop and learn while receiving proper care and protections by  building the resilience of OVC through the provision of essential psychosocial, educational, recreational, and nutritional support and services, as well as increasing their access to grants and other social services  strengthening the capacity of households and communities to respond to the needs of OVC  improving the livelihoods of poor families through increased food security and enhanced income generating opportunities                                                                   5 Details obtained from the COP11 Partner Project Plan  strengthening the collaboration with other state and non-government role players involved in child care and protection to increase the protective network for OVCs  strengthening Woz’obona’s capacity to sustain its work The methodology and mechanisms to realize the objectives were designed, not only to respond to the needs of the OVC, but also to build a sustainable network of caregivers within the child’s family and community. To achieve this strategic objective, Woz’obona developed a community- and rights-based approach to the provision of services targeted at OVC. In addition to building the capacity of OVC’s families to provide the necessary care and support, the program built and capacitated a cadre of community volunteers to identify vulnerable children, assess their needs, and provide a range of services in homes, schools, drop in centers, and child support groups. Volunteer caregivers receive training to identify and provide the following material, psychosocial, and educational support:  clinical nutritional support  child protection interventions  health care referral  health-care support to access ARVs  HIV prevention education  psychological care  educational support related to school attendance and school advancements  HES Overall Performance Measured Against Targets With three-quarters of the 2012 year still remaining, Woz’obona has come very close to reaching its overall target of providing care and support to 17,800 OVC between 2008 and 2012. As of April 2012, the organization has reached 16,900 OVC. The following graph shows the number of children reached against the target set for each year. Figure 1: Overall Target Versus Performance Results Woz’obona UGM End of Project Evaluation Report 13 The care and support provided between 2007 and 2012 is shown in the table below. Table 5: Achievements in Terms of Key Reporting Indicators Achievements (2007-2012) Indicators Achievement Clinical nutritional support6 315 Child protection interventions 2,747 General health care referral 3,322 HIV prevention education 840 Psychological care 4,761 Educational support 3,950 Household economic strengthening 1,553 Health care support for access to antiretroviral treatment 415 In addition, the project has reached 1,553 (64%) of the vulnerable households identified by volunteer caregivers in the targeted communities. This is a testimony to the effective way in which the program has worked with community stakeholders to ensure its maximum reach into some of the poorest households, It is worth noting the significant number of people who benefited from the psychosocial care provided by the project. This has been provided by training a significant number of volunteer community child care workers including  30 trained in lay counseling skills  29 trained in memory work  100 attended the psychosocial support training workshop  15 trained in play therapy  30 trained in child illnesses  24 attended home-based care workshop over a period of 56 days DID THE PROGRAM MEET THE NEEDS OF THE TARGETED BENEFICIARIES? The preceding section clearly illustrates that the program targets were met. This section reviews whether the targets were met concerning the following needs of the targeted beneficiaries (and in so doing, whether they advanced the HIV/AIDS and OVC objectives of the state):  the needs of OVC in the targeted communities  the needs of the households supporting OVC The Program Model and the Needs of OVC Through the combination of volunteer caregivers trained to identify and provide services and support, families were strengthened to meet the needs of OVC. Through an expanded referral network of community-based state and NGO service providers sharing a focus on OVC, the program sought to meet the following needs of OVC through the provision of relevant support and services7  nutritional needs through clinical nutritional support                                                                   6 This intervention was stopped in 2008 after the organization realized that malnutrition is not a problem in the area 7 Details obtained from the COP 11 Partner Project Plan  child protection needs through the provision of counseling, memory games, access to social grants and legal documents, the temporary placement of displaced children, and anti-child abuse campaigns  general health needs through the referrals to health care facilities and immunization campaigns at local preschools and schools in collaboration with local health clinics  treatment needs of HIV-positive children through the facilitation of access to ART, the provision of adherence support, the facilitation of access to food for HIV-positive children on treatment, and the facilitation of treatment literacy sessions to families with HIV-positive children;  HIV prevention knowledge through peer education and HIV/ AIDS awareness sessions administered in child support groups  psychological care through the referral of children for counseling and the provision of home￾based counseling for children by trained lay counselors under the direction of a psychologist  educational support at early childhood level through the provision of training for ECD practitioners; facilitating family literacy at preschools, child support groups and households through trained FMSL facilitators; and child development monitoring in preschools  educational support at the primary school level through the provision of children’s education clubs to improve performance at school using practical projects; the provision of homework supervision; the provision of school uniforms; and the negotiation of school fee exemptions  educational support at the high school level throughthe provision of after-school lessons in key subjects (math, science, English, and commercial subjects like accounting) by qualified teachers; the provision of school uniforms; and negotiating school fee exemptions As will be shown in the next section of this report, Woz’obona’s chosen model of care and support, has, in the main, proven to be appropriate and well-chosen to effectively meet the needs and circumstances of children in the targeted community and promote the realization of national HIV/AIDS and OVC principles and objectives. The lack of baseline information on OVC needs at the beginning of the project and justification for the model chosen was a challenge in the evaluation of the program design, methodology, and focus areas. To overcome this, the evaluation referred to national sources of data and information on the situation and needs of children in similar communities. The project was preceded by an assessment of the needs of OVC in the targeted communities, based on inferences drawn from district-level data from the national Community Survey about adults in the municipal district within which Jane Furse falls; it was not based on child-specific data collected through this survey or by the program coordinators. On the basis of the high rates of adult poverty, unemployment, migration, low levels of education, and high HIV prevalence, the program concluded that there were equally high numbers of children made vulnerable as a result of orphanhood and the breakdown of the family structure and poverty. The program assumed that this negatively affected children’s education, development, health, nutrition, psychosocial well-being, and protection from abuse and neglect. International and national research, data, and national OVC policy directives and priorities confirm the validity of the conclusions drawn by Woz’obona about children’s needs, and its choice to address these needs through a family- and community-based model of care and protection. International and national research recognizes that the arrival of HIV/AIDS in communities and families aggravates already high levels of poverty and social upheaval, to the detriment of children. The national OVC Policy Framework (2005) recognized that families were increasingly, as a result of the co-occurring sets of circumstances, unable to provide for children’s basic rights and needs such as shelter, food, medical care, education, and psychosocial support. The Framework identified and prioritized the most effective strategies for securing an enabling environment for more effective delivery of the services and support necessary for the well-being of OVCs. Woz’obona’s methodology reflects five of these six key strategies. As such, its chosen service delivery model is responsive and appropriate to addressing the challenges faced by OVCs in the communities of Jane Furse. Woz’obona UGM End of Project Evaluation Report 15 The program sought to  strengthen and support the capacity of families to protect and care for children in their communities and households  mobilize and strengthen community-based responses for the care, support, and protection of orphans and other children made vulnerable by HIV/ AIDS  assure access for orphans and children made vulnerable by HIV/AIDS to essential services  raise awareness and advocate for the creation of a supportive environment for OVC  engage civil society in playing an active role to support the plight of orphans and children made vulnerable by HIV/ AIDS The services and support that the program sought to provide were equally well suited to meet the needs of OVC living in rural traditional communities, where there were high levels of poverty and unemployment and high HIV prevalence. The focus on improving access to social security, education, psychosocial support, healthcare, and HIV prevention education was strategic and appropriate to leverage the maximum benefits for children and their resilience to the impact of HIV/ AIDS. From reviewing international and national data on children’s experiences in similar communities, Woz’obona’s program focus, offering a package of care and support which sought to improve access to social grants, advocate for school fee exemptions, provide assistance for school uniforms, provide support with school work and homework, provide psychosocial support, facilitate access to healthcare, and provide HIV treatment and prevention education, was well targeted to meet the needs of children affected by HIV/AIDS in rural Jane Furse. The appropriateness of the program design and focus areas was confirmed by the children participating in the evaluation focus groups. Children were asked to share their concerns, anxieties, and problems they experienced at home, in the community, and at school with which they need support. The children in all of the focus groups raised the following needs, which are in line with the needs that the project sought to address. Educational, Income, and Recreational Needs A substantial number of the children raised difficulties with their education as a key issue, indicating that the program focus on education recognizes and responds to a dominant need among children. The range of educational needs raised by the children included the lack of access to resources such as school uniforms and pocket money, and difficulty in learning and passing math and science. Other difficulties included not having enough teachers, unlawful teacher behavior and abuse, and absent teachers due to strike action. The poor quality of teaching was raised by one participant who observed that “at school, they do not teach us very well.” Lack of school uniforms was highlighted by a number of children as both a cost issue and a source of discrimination. Children without uniforms can be sent home from school until they are able to return with a uniform; this has negative consequences for the child’s learning as it results in a loss of school time. It can also hurt a student’s dignity and self-esteem, since other children tend to witness this humiliation and exclusion. The concern expressed by one of the participants that “I do not have enough help at home with my homework” was common across all focus groups. Almost all of the participating children raised the need for help with their homework, given that their parents are unable and/or unwilling to provide this support. The children, especially boys, shared their experiences and concerns about abuse by teachers and the routine use of corporal punishment. One of the older boys told the facilitators that ‘’the teachers swear at us and hit us with sticks.’’ Frequent concerns were mentioned about the high levels of bullying that all the children experienced at school, which affects children’s safety and emotional well-being. Lack of transportation to school and safety was a common concern, especially amongst the girls. Safety traveling to and from school was a significant issue for children, due to the thick bushes and dangerous people along the road. A number of them said they feel anxious about being kidnapped, or even worse, at any time. One of the children said, “It is not safe for us when we walk to school and back as there is a man who kills children and sells their parts.” A number of the children said that school would make them feel safer and it would mean they would not have to worry “about who can kidnap us on our way to school.” The poor state of school infrastructure and the lack of school and recreational facilities were mentioned in at least two of the focus groups. One boy asked why his school was in a poor state of repair and did not look like other schools in the country; one girl mentioned the lack of sports facilities as a problem. The facilitators observed the lack of these facilities, as two of the focus groups were hosted at a local school which was in a state of poor repair and had no sports field or playground for the children. Protection Needs In addition to the protection needs mentioned regarding educational experiences, a number of children, especially in the older girls’ focus group, expressed concerns about their safety in the community. This concern was linked to the high levels of alcohol and drug abuse in their communities. Life Skills Needs Almost all of the children that participated in the focus groups indicated a need for life skills support around HIV/AIDS; children were specifically concerned about peer pressure to engage in risky behaviors such as early and unprotected sex and using drugs and alcohol. Almost all of the children had experienced peer pressure; given the sensitivity of the subject matter, children did not indicate which of them had succumbed. Other Needs While educational and protection needs were common amongst the focus group participants, none of the participating children raised concerns or needs associated with the core areas of the program (health, dealing with grief or trauma, and HIV/ AIDS). This does not mean that the core program areas did not have an impact on the children served; instead, it is reasonable that the focus group responses are indicative of the selection criteria used to identify and include children in the program activities, and the sample of children selected for participation in the focus groups. It was not practical or ethical to question the children about their specific vulnerabilities in a focus group environment, but they were asked how they came to participate in the program as a way of assessing whether they were selected due to a potential vulnerability or another factor. In three of the four focus group sessions, most of the children were told by other children or volunteers about the program, indicating a degree of randomness in their selection. There was little evidence of mechanisms for identifying and targeting the most vulnerable children by, for example, working with schools to identify children in need of extra support or relying on referrals from social workers. From two out of the four focus groups, it appeared that many of the children in the focus groups were not the most vulnerable in the community. The last focus group (boys ages 12–14) at first presented a significant mismatch between their needs and the program; it seemed as if many of the participants had never participated in a program and did not know of the program. The evaluation facilitators were advised that, despite the briefing of the child care worker volunteer about careful selection of the children for the focus groups based on the program criteria and participation, she had brought along children to the focus group that knew little of the program and had not participated in it. Did the Program Meet the Targeted Needs of OVC? There was substantial evidence from program data and the focus group discussions that the Woz’obona program has consistently and effectively met many of the identified and stated needs of OVC in the targeted communities. Woz’obona UGM End of Project Evaluation Report 17 The program provided children with the space and opportunities for recreation, sport, and leisure that they had not previously enjoyed. This was identified as one of the notable program strengths by a number of the children who participated in the focus groups. SEP provided a number of children with access to nutrition, healthcare, and child protection services and support. During the life of the program  315 children received clinical nutritional support  2,747 children received child protection interventions  3,322 children received general healthcare referrals  415 children received access to antiretroviral treatment One of the children participating in the focus group testified that she has “been assisted to apply for a grant and…now [is] able to buy food.” The program increased children’s access to quality early childhood education services and met the diverse education needs of many children.  109 children were reached through home-based ECD programs  90% of households/parents now see the value and need for early childhood education and stimulation  80% of practitioners are now trained to provide quality early childhood education services; this marks a 40–50% increase from the original 30–40%  3,950 children received support that improved their educational outcomes. For many of the children that participated in the focus groups, the space provided and the support given in completing their homework filled a gap created by the low literacy levels of their parents and the lack of physical space for working at home. One of the boys spoke highly of the program, because, in his words, “I did not have enough help at home with my homework and through SEP, I am getting the necessary assistance.” Another boy said that, “it has helped with the homework and filled the gap of no support at home.” In addition to the support with their homework, a number of children participated in the extra math and science classes offered through the program. For some, this support was instrumental in allowing them to pass these subjects. Children that participated in the evaluation and had attended the extra math, science, and accounting lessons confirmed that whereas they struggled with these subjects before, after the lessons, it was much easier for them. For some, the lessons resulted in numeracy skills that were previously lacking, and for others, in an improved ability in math and science, results which allowed them to progress academically. One child said, “I can even count money now unlike before I joined SEP.” For another child, the program “allowed me to be promoted to the next grade.” Another boy told facilitators that “I have been having serious challenges with math and science and now [since his participation in extra lessons] have started enjoying the subjects.” In addition, they felt that participation had increased their levels of English￾language proficiency. One of the children proudly stated, “I am now confident to speak in English.” A result of the educational support is that, in the words of one participant, “I am now more focused with my education and eager to follow my dreams.” A total of 840 children received HIV prevention education though the Woz’obona program. The life skills and HIV prevention interventions contributed to the levels of knowledge about HIV prevention among the children; they learned the primary causes of HIV transmission and about the behavior that should be adopted to avoid infection. When asked whether they knew how people become HIV positive, they listed the correct responses including unprotected sex, blood-to-blood transfusion, and mother-to-child transmission. In addition, many of them, especially the older children, recognized that having multiple partners, unprotected sex, sleeping with people with HIV, drinking alcohol, prostitution, and getting involved with “sugar daddies” exposed them to higher risks of infection. The life skills education appears to have motivated children to avoid risky behaviors. The girls in the focus groups said they had modified their behavior as a result of the program and were now more cautious after learning prevention information from Woz’obona. One girl said, “My behavior has changed through SEP and I have now adopted a positive life style.” Another indicated that “the program has raised my knowledge and made me more careful.” While the evidence shows improvement in HIV prevention knowledge and the behavior of children in the targeted communities, attributing this change to Woz’obona is complicated by the fact that children received HIV and life skills education from two (or perhaps more) different sources: school and Woz’obona. When the children were asked, they were clear that information came from two sources, but indicated that the Woz’obona’s information was clear and more in-depth than the school-based education. The Woz’obona program appears to have augmented the school information and was easy to understand. Woz’obona provided psychological care to 4,761 children through participation in the Memory Game. This is administered to all children who participate in the program, by volunteers trained in the provision of psychosocial support, as a coping mechanism to aid children to deal with grief, trauma, and stress. Children who participated in the focus groups were unable to explain why they participated in the memory box game, but many said that they felt better after they had because it made them feel “lighter.” One of the children observed that the memory box game has “helped me to deal with stress related issues that has been caused by family conflicts.” At least two children in the focus groups had been referred to professional counseling and said it helped them deal with issues they were facing. The facilitators did not probe the issue of psychosocial support too deeply in the focus groups due to the sensitivity of the issue, the right of children to privacy, and the need to avoid causing further trauma to children who had experienced grief. Unmet Needs Data collection revealed that the program failed to recognize or meet a number of needs common to OVC, and that some of the support and services provided were not aligned with the scope and nature of the needs on the ground.  Protection from corporal punishment: Despite the fact that corporal punishment in schools has been outlawed for many years, children in the focus groups spoke of it being practiced as a matter of routine in the local schools. Children specifically noted this as a need that they would like to see the Woz’obona program address as part of its child protection interventions.  Protection against bullying: A number of children indicated that bullying is rife in the schools and they believe that an institution such as Woz’obona/SEP could and should play a more proactive role in addressing it.  Transport and safety: Respondents indicated that they fear for their safety in the community, especially on their walk to school through thick bushes. Generally, the children were concerned that the local chief and other stakeholders were not doing enough to address the safety in the community, including to and from school. The children stated that Woz’obona should engage with other stakeholders to encourage solutions through the provision of, for example, transport to school.  Teen pregnancies: Teen pregnancies are of growing national concern. Woz’obona’s HIV prevention module includes information on sexual and reproductive health. The older girls that participated in the focus groups indicated that they had benefited from the education and had a clear understanding of the issue and their reproductive rights. They knew how to avoid unplanned pregnancies and that they have the right of access to contraceptives and to termination of pregnancy services. However, education on sexual and reproductive health and rights is taught to only the older girls and boys. The younger girls and boys that participated in focus groups had little, if any knowledge on pregnancy and how to avoid it. Woz’obona’s decision to educate only older children is a missed opportunity for addressing this issue among vulnerable children. The older and younger children indicated that their peers were sexually active from a young age, often as a result of peer pressure on girls. In 2008, 5.9% of girls (compared to 5.3% in 2002) had Woz’obona UGM End of Project Evaluation Report 19 engaged in sexual activities before the age of 15,8 a statistic which provides strong grounds for early and meaningful education for girls and boys.  Gender education: The communities served by the project are deeply traditional, and gender plays a significant role in successful prevention interventions. Nevertheless, the inputs from the children indicated that the education provided did not cover this issue or address harmful customs or stereotypes common in customary communities.  TB prevention education: While the participating children exhibited high levels of knowledge on HIV prevention, there was little knowledge on TB prevention and treatment. South Africa’s 2012 HIV/ AIDS and TB National Strategic Plan has deliberately incorporated TB within its purview, given the high prevalence of co-occurring HIV and TB. It is advisable that HIV prevention programs align with the NSP and teach prevention in relation to both diseases so as to secure national prevention and cure targets. Services Provided by Woz’obona that did not align with Pressing OVC Needs The majority of children who participated in the focus groups did not note an inability to access grants as a problem, and none mentioned school fees or hunger as pressing needs in their lives. This is indicative of a mismatch between the program’s focus (initially) on clinical nutrition, the promotion of access to social grants, and the negotiation of school fee exemptions and the needs of OVC in the targeted communities. While it may be argued that the focus group sample was small and children with these needs did not participate, it is also likely that the absence of these needs amongst the focus group participants is representative of the broader OVC community. The partial misalignment of needs and responses is attributable to a number of factors, including the inadequacy of the initial situational and needs assessment of children, and a shifting national policy and service delivery landscape during the duration of the program. Within a year after the program began, Woz’obona determined that there was little need for the nutritional support intervention, given the low levels of malnutrition in the targeted communities.9 It is likely that a more stringent situational assessment before the initiation of the project would have provided an accurate indication of needs on the ground and enabled resources to have been directed more effectively to meet the most pressing needs from the outset. Since 2008, the rate and ease of access to documents necessary for grant applications and the grant access rate have improved substantially, especially in the Limpopo province. Access to the CSG in the province is now nearly universal amongst eligible households, reducing the scope of the need to negligible proportions (Statistics South Africa, 2012). The introduction and expansion of the no-fee school policy in 2008 resulted in all schools in Jane Furse being declared no-fee schools. As a result, there is no need for school fee exemption applications or negotiations; thus support has not been needed in this area since soon after the start of the program. The issue of school uniforms was raised by a number of children, both the need for support the uniform cost, but also the need for support to prevent the discriminatory punitive measures implemented by the schools when children could not afford a uniform. Children spoke of being sent home when they were not attired in their uniforms, this is expressly outlawed by the South African Schools Act as well as the National Guidelines on School Uniforms. While there is a close alignment between children’s needs and the program’s focus on uniforms, the program has thus far focused on material support for school uniforms. There is clearly a need for awareness-raising and advocacy among teachers, children, and caregivers of the need to combat discrimination against not wearing a uniform, and for the development and implementation of appropriate supportive school uniform and discrimination policies. DID THE PROGRAM MEET THE NEEDS OF HOUSEHOLDS TO STRENGTHEN THEIR CAPACITY AND ECONOMIC STRENTH TO LOOK AFTER THE OVC?                                                                   8HSRC, South African National HIV Prevalence, Incidence, Behavior and Communication Survey, 2008 9 Written communication by Woz’obona representative dated 9 June 21012. The program identified a number of challenges of households looking after OVC, including unemployment, difficulty in accessing grants, food security, poor budgeting skills, a lack of parental involvement in and supervision of their children’s education, and the provision of poor quality early childhood development learning and stimulation. The program sought to meet the economic needs of households through the HES program. The aim of the HES program is to provide households with sustainable strategies and interventions to reduce their economic vulnerability. HES interventions implemented are listed below.  Social assistance: Households were supported to access to child support grants, foster care grants, and other social grants  Asset growth and protection: Households were supported to save (through credit associations, rotating savings, and accumulation associations), to access credit, and to engage in agriculture including food gardening  Income growth: income generating activities, such as poultry and knitting As is shown below, there was substantial evidence from program data and the focus groups discussions that the Woz’obona program has met the identified needs of households caring for OVC in the targeted communities. The program has  aided households in generating income through eight active savings and loan scheme groups  assisted households to manage household income through budget workshops  facilitated access to social grants for household members  catalyzed income generation through various initiatives, including assisting families to access social grants, start income generating projects and small businesses, and through small business training  Facilitated improved food security through the promotion of agricultural activities and the provision of agricultural materials and support in collaboration with the Department of Agriculture. In addition, the program has laid a foundation for improving food security through the greenhouse project (currently under development), with the aim of fostering agriculture through the use of greenhouse technology. While the project has commenced, it has taken longer than projected because of the bureaucratic requirements involved in building such a structure. Household members that participated in focus groups confirmed that the program had met their needs. They indicated that the program had assisted them in ways listed below.  obtaining the documents necessary to access social grants from the South African government  The income obtained from grants allowed households to begin participating in the savings and loans clubs. There are 194 households participating actively in savings and loan clubs throughout the 8 villages.  Additional income secured because of participation in the clubs further enabled households to – Eliminate debts to external agencies – Improve the quality of life by being able to purchase items such as refrigerators, water tanks, furniture – Pay for children to complete school, and in some cases enroll in tertiary education opportunities – Initiate small business activities such as vending or trading o Access to agricultural inputs and training meant households could establish food gardens. The production of food has allowed additional income from grants and savings and loan activities to be used to buy equipment. In addition, because households spent less on the purchase of food, they could use the additional money for other activities such as Woz’obona UGM End of Project Evaluation Report 21 transport, school, and income generation. The greenhouse project that is being built will also make a meaningful contribution in food security. o Participants in the scheme indicated that the financial management and budgeting skills they were taught were very useful, and that they shared these skills with others of their acquaintance. The HES initiative has value because it works with a wide range of stakeholders to leverage multiple resources and build capacity among beneficiaries to maximize the use of surplus household resources. The savings and loans clubs were felt to be particularly successful because they were managed by the members. Members set the interest and repayment terms; if payments are not made, the matter is reported to the police. Generally, the clubs received a high rate of return on their contributions and the annual payment of this profit to members had a substantial impact on their economic well-being. The success of the integrated, escalating, and self-sustaining principals of the HES program in meeting the economic needs of households is clearly illustrated by the following case study. The way in which Ms. Elisa Marumo (not her real name) has experienced the Woz’obona/SEP interventions illustrates both the integrated nature and the value of the program design. A resident in the Jane Furse Area, Ms. Marumo is directly responsible for the care of three children. She was trained by the program in ECD provision and works as an assistant at the Drop In Center. She has received a stipend as well as some play and sports equipment from the program so that she can implement home-based ECD activities. She also works part time as a volunteer with Love Life. Having received training from Woz’obona/SEP, Ms. Marumo was able to prepare a CV profiling her skills and experience, which enabled her to receive an increased stipend. Ms. Marumo is a member of a savings and loans club and was initially able to participate because of the stipend she received from the program, which supplemented the pension she receives from the government. The HES has assisted Ms. Marumo to  participate in two savings societies  eliminate her debt  purchase infrastructure to harvest rainwater  start using her bank account to save money rather than just as a conduit for her pension  participate in a club where they give the members groceries for big occasions or to supplement the household  join a funeral society where members contribute R20 per month if there are already funds in the coffers or R50 if the funds are depleted; this means the burden of a funeral is less likely to exhaust all the resources of a member  start a small business as a vendor within the community  pay for her son’s studies at the University of Limpopo  initiate a new savings society and encourage members to join Ms. Marumo’s experience is not unique. All the savings and loans clubs initiated through the HES report that members’ financial security and the diversity of their income has increased because of their participation. WHAT WERE THE MOST SIGNIFICANT CHANGES BROUGHT ABOUT BY THE ORGANIZATION IN IMPROVING THE WELL-BEING OF ITS BENEFICIARIES? The information provided through statistical data, focus groups, and interviews provide evidence that the Woz’obona project has met its four primary objectives, and in so doing has succeeded in creating a more enabling environment in the target communities in which children are stimulated to develop and learn while receiving proper care and protection. This section of the report documents the most significant changes brought about by the project that have contributed to the significant gains made in  building the resilience of OVC through the provision of essential services  strengthening the capacity of households and communities to respond to the needs of OVC  improving the livelihoods of poor families through increased food security and enhanced income generating opportunities  strengthened collaboration with other role players involved at a community-level in child care and development  strengthening the capacity of Woz’obona to sustain its work Significant Changes that Built the Resilience of OVC through Access to Essential Services Improved ECD Provision and Quality All stakeholders involved in the evaluation agreed that Woz’obona/SEP played a central role in promoting the expansion and improvement of the quality of ECD services in the targeted communities. ECD provision has improved extensively in the area due to the intervention of Woz’obona/SEP, including improved access through the promotion and development of home-based ECD interventions in the targeted rural communities. It is significant to note that improving access to ECD services for children aged 0–4 years in under-serviced rural areas is a national priority, and the promotion of home and community-based ECD programs to realize these goals is the nationally sanctioned strategy of choice10. Woz’obona has furthered a key national ECD priority specifically for vulnerable children, and laid the foundation for the scaling-up of its interventions that will result in the expansion of the strategy to the benefit of all children in the targeted communities, beyond the boundaries of Woz’obona’s operations. A Department of Social Development official noted that “the methodology used by the organization of training community members to run a home-based ECD intervention provided a new model of delivery which they would now pilot for application across the province.” In addition to expanding access to ECD services, Woz’obona has substantially improved the quality of ECD services in the community. This is particularly significant given the consensus among researchers and practitioners that ECD services have the most significant potential of all social development intervention to equalize the development and learning opportunities for children living poverty and under other constrained circumstances, but only if the services provided are of an adequately high quality11.  The quality of services has been improved as a result of Woz’obona interventions securing access to state funding for ECD sites, improved management skills at ECD sites, and the training of ECD practitioners.  At least 165 ECD practitioners have received training and qualified as practitioners through the work of the program. This has had a substantial impact on provision within the municipality, both in increasing the number of children who have access to ECD and the quality of the ECD services provided.  The interventions and their impact were explained as follows by one of the interview respondents: “Now we have best practice in crèches, because Woz’obona has trained us on                                                                   10 The National Integrated Plan for Early Childhood Development in South Africa, 2005 – 2010 prioritizes the provision of ECD services to children aged 0-4 in rural areas of South Africa through community and home-based ECD programs. 11 The World Bank (2010) Investing in Young Children: An Early Childhood Development Guide for Policy Dialogue and Project Preparation & Center on Developing Child, Harvard University (2007) Framework for Early Childhood Policy: Using Evidence to Improve Outcomes in Learning, Behaviors and Health for Vulnerable Children http://www.developingchild.harvard.edu,   Woz’obona UGM End of Project Evaluation Report 23 how to manage the crèche and deal with the children…Before people wanted to just do crèches because it was quick money, now after the training, those people were properly trained with ECD.”  Another respondent added that as a result of the project interventions, “Educators in drop-in centers know how to engage children in interesting activities that also assist in developing the children’s skills.”  For another respondent, the training had the following impact on improved services: “The accredited ECD training that is offered...is based on Montessori theory and it is practical and applicable. SEP has kept in contact with the people that have been trained, share materials and any other relevant information. For example, I was made aware of the S.A ECD Awards funded by ABSA by SEP.” Community perceptions of the value and need for ECD, both for OVC and children more generally, shifted positively over the course of the program. In addition, the capacity of parents to provide early learning, development, and stimulation at home improved, ensuring sustainable access to these services for the majority of children in the area. This was illustrated by the following comment made by one of the interview respondents: “Parents have gained knowledge and skills and are now able to teach basic numeracy and literacy at home and use knowledge to be resourceful to create teaching and instruction materials with resources available. This is sustainable capacity building.” Improved Educational Opportunities and Outcomes The space and support provided by community volunteers to learners, through the after care and children’s support groups, to enable them to complete their homework and school projects was perceived as extremely valuable and had an impact on their school performance. This is illustrated by the following quotes.  A community member indicated that “Learners appear to be behaving different when they go to SEP extra classes than when they are in the normal classroom. They are well behaved and more committed, and as a result they concentrate.”  A former beneficiary who passed Grade 12 and was admitted to an Institution of Higher Education attributed her success to the weekend classes for grade 12 learners, the provision of photocopies of old question papers, and the assistance with the revision of school work provided through the Woz’obona program. She stated that “these interventions were key to her passing and acquiring her national senior certificate with sufficiently high marks to move onto tertiary education.”  The support provided by the program extends beyond support to learners. It included building the capacity of parents and caregivers to support children at home, through the family literacy and adult education components of the program. As a result, parents have gained knowledge and skills and are able to teach basic numeracy and literacy at home. They can use their knowledge to be resourceful, and create teaching and instruction materials with the resources available to them. This has had a positive impact on sustainably improving children’s learning opportunities and outcomes, which had been impeded by the low literacy levels of parents and their inability to support children with their school work. A respondent from the Department of Social Development indicated that “parents had no knowledge and skills. Since being trained in care and Family, Math, Science, and literacy, Woz’obona/SEP has plugged the gap in knowledge and care.” Improved Attitudes, Prospects, and Social Behavior Children that have participated in the educational support, recreational and sport, and HIV life skills programs, as well as adults and service providers who care for them, confirmed that these interventions have improved educational outcomes and the commitment and attitude of children to their school and work, their prospect, and their social behavior.  Children that have participated in the program have become more hopeful and ambitious as a result of improved school outcomes, exposure to possible career choices, and their improved self-esteem. This is illustrated by the comment made by one of the key informants that “I see that the children are able to form their own dreams and see a possibility in the future.”  The benefit of the program for children who have participated in the extracurricular homework support, sport, and HIV prevention activities goes beyond their school performance to include an improved attitude and commitment to school.  A number of respondents confirmed that children who participated in the program were more positive than their peers before they participated, and were less inclined to engage in negative social activities such as petty crime or substance abuse. The sport and cultural activities provided as part of the afterschool activities were seen as particularly useful in assisting them to resist the “do nothing” culture prevalent among young people in their communities. This is summed up in the following statement made by a child beneficiary: “Having participated in the Chance to play; now I want to go to school to study sports management, because I see how it changed my life.” Service providers, caregivers, and community members have observed changes as described below.  “I see a lot of changes—no street loitering by children. I have not heard of cases of rape, late coming, or children who have disappeared.”  “At individual level, SEP’s activities are motivational by design and therefore children get motivated to continue with positive lifestyle and stay away from drugs. Soccer by its nature encourages teamwork and children learn how to work with others, develop communication and listening skills.”   Improved HIV Prevention Knowledge and Engagement in Less Risky Behaviors The HIV prevention interventions made a significant contribution to the levels of knowledge among the children about HIV prevention. Children were well-versed in the primary causes of HIV transmission, as well as on the behavior that should be adopted to avoid infection. This is a notable achievement give that national statistics have shown declining levels of knowledge, especially among girls in rural areas12. Moreover, numerous children indicated that the information provided through the program encouraged them to change their previously risky behavior. Children also spoke positively about the relevance and accessibility of the HIV prevention information. Please refer to the earlier discussion on the extent to which the program met the life skills needs of OVC in the targeted communities for evidence of this impact. Improved Access to Healthcare Woz’obona has worked in collaboration with the Department of Health (DOH) to raise awareness and support the implementation of health programs such as immunizations for children under the 5, the developmental assessment of children in Grades R and 1, and parental education in the identification and assessment of diseases and improved food and nutritional practices. Collaboration has taken the form of training provided by the DOH to Woz’obona/SEP caregivers and the referral of children and their caregivers to health services. The DOH is of the opinion that the program activities have had a positive impact on improving awareness, monitoring, and responses to children’s health needs and nutritional requirements. The Assistant Manager for Primary Health Care confirmed that although no baselines were established and no system implemented to monitor impact, the general perception and assessment is that “before the Woz’obona program, parents and households did not have the knowledge and skills to identify diseases or to create teaching and instruction materials with local resources available, including the creation of safe home-based play activities for children and OVC. Parents did not take it as their responsibility to acquire awareness and knowledge as well as assessment of basic disease, numeracy and literacy, and now they do. They have changed from being uninformed and passive to being very much aware, interested, proactive, and active in matters of child ECD, disease identification, and OVC support.”                                                                   12 HSRC, South African National HIV Prevalence, Incidence, Behavior and Communication Survey, 2008 Woz’obona UGM End of Project Evaluation Report 25 A Department of Health representative interviewed stated that "the capacity building that has been created is one which is sustainable although Woz’obona/SEP could strengthen it with a memorandum of agreement for partnerships with the department.” Improved Psychosocial Well-being of OVC Woz’obona/SEP trained lay counselors to monitor the psychosocial health of OVC and to refer children experiencing severe trauma to qualified psychologists and social workers. Caregivers were also trained to provide memory workshops, which encouraged participants to create Memory Boxes. Partnerships established with professional groups such as social workers and psychologists meant that people were confident they could refer issues beyond their scope. Professionals working with the program attest to the remarkable engagement with OVC achieved by the project. OVC who had participated in the Memory Game activities found it useful. They indicated that it had allowed them deal with painful feelings and, as one young man expressed it, “we are now more able to open up even in front of friends and this helped me to heal.” Significant Changes that Improved the Capacity and Livelihoods of Families and Communities to Meet Needs and Provide Care and Support to OVC Improved Household Capacity and Income to Meet the Needs of OVC The economic health of households participating in the HES has improved, both substantively and sustainability. The scope and nature of the impact is discussed in more detail under the previous heading related to the extent to which the program addressed the needs of households. The program has improved the levels of income by diversifying income streams, improving access to social security grants, increasing savings and access to loans, which has in turn resulted in improved infrastructure and access to services in the household and the commencement of additional income generating activities. Significantly, it has catalyzed and empowered the use of minimal available resources—whether in the form of grant income, business income, or income from savings and loan clubs—to achieve maximum returns through wise budgeting and investment choices. The level and self-sustaining nature of the support provided is summed up in the following comment made by one of the participating household members: “I was resurrected from the dead and through the facilitation of the grant, I am now able to contribute towards savings and loans and pay my debts.” An immediate impact that enhanced family income was the improved access to social grants and resultant improved ability of parents to meet their children’s essential needs. In the words of one key informant, before the program, “children did not get grant and SEP has assisted in getting birth certificates and registering them with the South African Social Security Agency (SASSA). The grant assists parents to buy whatever the child needs and they are able to be free to work with other children.” However, access to the grants was for many a stepping stone to improved and sustainable economic well-being, through the smart investment of grant income. One of the household beneficiaries interviewed explained the dynamic nature of the support provided by the program. She initially depended on children’s grants, and spent all the money on household expenses and had no other source of income. However, through the HES program, she has been enabled to use some of the grant money to engage in entrepreneurial activities to generate more money. One type of income generating activities that have been supported by the program includes agricultural initiatives. A representative from the Department of Agriculture appreciated the fact that through Woz’obona/SEP training, “parents now have knowledge and skills for water-efficient vegetable home gardens, income generation achieved with home gardens, nutrition improved and malnutrition and vitamin deficiency reduced.” In addition, parents and caregiver’s literacy and numeracy (as well as their appreciation of the value and need for ECD) has improved through the program interventions. This has resulted in more parents including their children in ECD programs, the development and provision of early childhood stimulation and development activities, using local available resources at home, and the greater involvement of parents in supporting their children with the school and homework. This impact is discussed in more detail under the improved ECD provision and quality and improved educational opportunities and outcomes for OVC sections. Improved Community Capacity to Provide Care and Support for OVC The program has had a substantial impact in raising community awareness about the situation of OVC and their needs. The primary interventions that have raised community awareness are community-based activities such as community theater and the use and training of community-based volunteers as child protection workers. The organization uses theater address the plight of OVC, which dramatizes developmental and OVC issues and engages communities in discussions around possible solutions. Theater activities include cultural drama, dance, poetry, and storytelling. The awareness raised through this program has assisted in mitigating the stigma associated with HIV/AIDS, and mobilized a large number of individuals and community-based organizations to improve their care and support for OVC and monitor their well-being. In addition, the community-based child care workers raise awareness of children’s rights and services through their home visits, community workshops, and interactions and partnerships with the school and local government structures; they also promote the recognition and appropriate responses by community stakeholders of the rights and needs of OVC in the targeted communities. As a result of the promotion of the involvement of women community volunteers in the program, and their successive child protection interventions among caregivers and households, women have become particularly confident and more willing and able to advocate against child abuse and neglect. This was well articulated by a respondent who said, “women in the community have become confident. I was so impressed that women and community now have a direction. Abuse in families can no longer be a hidden and private thing. Women have become involved and are standing up.” The broader communities have also been empowered through the various awareness-raising campaigns and household interventions to claim their children’s rights and secure services to meet their needs. In the words of one community member, “the ccommunity is able to identify where they could get help like birth certificates, bursary forms, etc. They are able to apply for grants and food parcels.” Significant Changes that Strengthened Collaboration with Other Stakeholders Involved at a Community Level in Child Care and Development One of the key concerns which the program sought to address was the absence of a common and special focus, among all service providers within the community with a child care and protection mandate, on the needs and services required by OVC. The program succeeded in addressing this by raising awareness of the need for specific responses to OVC’s needs among state and NGO service providers, and through the development of various partnerships and referral networks among the relevant stakeholders. It must be noted that a number of state partners, notably the health department, indicated that the flexibility and informality of the partnership relationships created implementation difficulties. Woz’obona was urged to develop formal memoranda of understanding with its partners in the future. Stronger OVC Focused Partnerships with Government Departments All government stakeholders that participated in the evaluation which have worked with Woz’obona/SEP over the past five years confirmed that they have a positive relationship with Woz’obona. They further confirmed that the relationship has assisted them in acquiring a better understanding of the needs of OVC and has prompted them to review their service provision to ensure they are better able to respond to those needs. ECD provisions for OVC by the Department of Social Development (DSD) have been strengthened, and the reach and quality of the service expanded, because of the relationship between DSD and Woz’obona UGM End of Project Evaluation Report 27 Woz’obona. As the respondent put it, Woz’obona “has made a huge difference to the situation that prevailed before.” The relationship has resulted in the decision by the DSD to pilot the program’s home-based ECD model, which offers enormous opportunity to scale-up the model to the entire province. The Department of Education (DOE) expressed satisfaction with the program and their participation in it. The DOE representative that participated in the evaluation confirmed that Woz’obona has collaborated with the DOE and contributed to the development of OVC-responsive ECD interventions, including the development of a Critical Phase Curriculum and Practitioner Guide; the payment of monthly allowances/stipends of ZAR 2,000 to practitioners, which is providing jobs and income to practitioners and reducing poverty; developing human capital through the provision of accrediting training to ECD practitioners who did not have prior training; and the development of guidelines for an integrated monitoring and management system of OVC and ECD centers and programs. Woz’obona has a working partnership with the Department of Agriculture (DOA). The DOA views Woz’obona as a key implementing partner in national agricultural and food security initiatives; as a result of the partnership, there have been positive changes in household food security and support to OVC. The departmental representative interviewed spoke specifically of the collaboration with Woz’obona in respect of awareness campaigns, supporting the setting up of vegetable gardens, the provision of inputs support and production extension services, and the development of the greenhouse project as making a positive difference to the alleviation of household food insecurity and vulnerability. Although the department has not kept records of Woz’obona/SEP beneficiaries and production figures, the department has observed how households have acquired the knowledge to plant and produce food with limited water resources/water-efficient methods, and produced enough for household consumption and surplus for the market, thereby supplementing household income. Significant Changes that Strengthened the Capacity of Woz’obona to Sustain its Work Woz’obona community caregivers have received extensive training in a wide range of courses including  30 trained in lay counseling skills  29 trained in memory work  100 attended psychosocial support training workshop  15 trained in play therapy  30 trained in child illnesses  24 attended home-based care workshop over a period of 56 days Staff members have done courses in management, bookkeeping, database administration, human resources management, monitoring and evaluation, and people-centered community development. The impact of this training has contributed to improved knowledge and the implementation of an organizational and monitoring system. As one respondent stated, “The program has had positive results, on our staff, the CCW; working with an American funder requires a lot of things, so we built up and developed dramatically. Systems were tight, so you’re pushed to learn. Now we can handle any funder.” Another staff member added that he “learned a lot in monitoring and evaluation. At one stage, I was up to give in but I was encouraged by FHI 360 staff to soldier on and I am happy I did. I now know what is required from the program to show its results.” WHAT WERE THE KEY ENABLERS AND BARRIERS IN MEETING THE PROJECT OBJECTIVES? Programmatic Enablers The Model Itself Woz’obona/SEP’s model of development has proved to be an effective tool in changing the environment for OVC in Makhuduthamaga Municipality and, more generally, in the Limpopo Province. The organization locates the child at the center of their development intervention but does this with the understanding that if a child’s resilience is to be achieved, the environment in which s/he lives must also be transformed. The benefits of the model can be summed up as follows  The breadth and contextual nature of the engagement means the program is widely known. In addition, in most communities people see the value of participating in its activities, while state agencies recognize the value of working in partnership with it.  Woz’obona/SEP has worked to ensure the program is sensitive to the culture of the area. Great care has been taken by the representatives of the organization to develop relationships of trust with communities and with government institutions, and this has enabled the organization to operate even when there have been challenges.  Finally, an important element of the program’s success has been its ability to generate energy and capacity for support to OVC at a very local level. The program has worked with parents and neighbors to become the providers and advocates of children’s welfare. This has enabled the program to have an immediate and textured understanding of the environment, and to build an appreciation for its work because many people know others who have participated in it.  The combination of home-based and center-based ECD has enabled the organization to reach out to most children who would have otherwise not been able to access ECD. Alignment of the Program with National Priorities The close alignment of the program with national OVC, HIV/AIDS, ECD, and rural development policies and programs has enabled the program to work in partnership with state agencies, leverage resources for their beneficiaries, and simultaneously promote national objectives and interventions. In addition, the close alignment provides a sound foundation for ensuring the sustainability of interventions. Initiatives that mirror and/or compliment national priorities and interventions are more likely to find traction and support and to have a more substantial and sustained impact than interventions run in isolation of the broader policy context. As a result of the close proximity with, for example, the national ECD policy priorities to secure ECD services for children aged 0-4 in rural, under-serviced areas through center-, home-, and community-based programs (as articulated in the National Integrated Plan for ECD 2005–2010 and the Children’s Act No 38 of 2005, as amended), Woz’obona was able to secure a productive partnership with the DSD, advocate for the successful expansion of the ECD program through the DSD adopting its model as an operational model, and secure state funding, resources, and training for community-based ECD sites and practitioners. A further example of the program’s successful leverage of resources and impact through its close alignment with national policies is the alignment between the programs at school level providing HIV prevention education. The Woz’obona program has built on, and complemented, the school-based life skills education rather than operating in isolation. Working together, the programs have had a more substantial and sustained impact, helping to realize national policy objectives. This evaluation has established a close alignment between the program and the following national policies, which has contributed to the more effective realization of South African government goals and objectives related to the care and support of OVC, their families, and communities; the food security and economic development of rural families; and the prevention of HIV among the youth of South Africa.  The HIV/ AIDS and STI National Strategic Plan, 2007–2011  The National Policy Framework on Orphans and Other Children Made Vulnerable by HIV/AIDS (2005) and the accompanying National Action Plan for Orphans and Other Children made Vulnerable by HIV/AIDS, 2009-2012 Woz’obona UGM End of Project Evaluation Report 29  The National Integrated Plan for Early Childhood Development in South Africa, 2005-2010  The Children’s Act No 38 of 2005, as amended  The National Policy on HIV/AIDS for Learners, Educators in Public Schools, and Students in Further Education and Training Institutions, 1999  The Integrated Food Security Strategy for South Africa, 2002 Community-Based Recruitment of Volunteers The program has recruited volunteers from the same villages where services are being rendered, and all those who volunteer are screened before they are taken onboard. They are required to be highly committed, motivated, and literate so that they are able to fill in forms and write reports. The volunteers have in-depth knowledge of the community, cultural practices, and norms. The area is a close-knit community and the volunteers have established relationships with professional persons, making it easy to refer clients where necessary. Self-Sustaining Strategies Maximize Existing Resources The organization has been able to be implement activities that complement and yield positive returns out of the services available to meet the needs of OVC and their households; some of these activities include the use of the CSG as a platform for entrepreneurial activities, and the use of local, inexpensive resources to develop early childhood education and stimulation activities. Contextual Enablers Community Interest and Motivation to Participate Within the local community there are established authority structures—both through the state institutions and through traditional leadership— as well as a number of existing civil society institutions working on issues that impact on OVC’s well-being. Each of the stakeholders has a vested interest in improving the quality of life of vulnerable members of the community. Woz’obona’s/SEP’s careful and continuous engagement with these stakeholders to develop a network of care and support has allowed them to mobilize community-wide commitment, interest, and support for their work and to ensure a high degree of community-ownership of their activities. Woz’obona/SEP has capitalized on the desire of community members to both care for children and improve their economic condition by employing and training people within communities. There has therefore been a sense that the program is “of the people—for the people.” Commitment of Stakeholders A strength of Woz’obona’s intervention is the recognition that the multiplicity of problems confronting the children requires the collective response by a broad range of stakeholders in the design and delivery of adequate solutions. This has led the organization to forge strong links with external stakeholders such as Department of Home Affairs (for legal documents), the Department of Social Development (for grant processing), the Department of Health (for clinic referrals), traditional leaders (for gaining entry into the various communities), the South African Police Service, and educators (for child care, protection, and education). Program Barriers Lack of Parental Support The high levels of illiteracy and low levels of formal education in the targeted communities means that many parents remain unaware of the value and importance of early childhood education and basic education. Economic and cultural pressures also contribute to parents prioritizing children’s employment over education and the right to play. In the words of one respondent, “parents want their children to do household chores.” Consequently, low parental support for the organization’s programs has, at times, been a challenge. To overcome this resistance, the organization has made use of village sharing sessions, theaters in schools, and psychosocial support groups to encourage parents to let their children play and to change the societal attitudes towards children. Resources The organization has become almost entirely dependent on PEPFAR funding, which is a challenge for the sustainability of the program. This was well articulated by one management respondent who stated that “now that we’re coming to an end with USAID, we have to find a replacement for funding.” The following comment made at the focus group held with staff summarizes the overriding concerns about sustainability: “Everyone is very worried about the ending of funding, come September 2012.” The resource requirements of the organization are limited in relation to the scope of need. This has impacted the organization’s ability to deliver all interventions and realize related objectives. The organization has very few staff members and only three vehicles, and the area that must be covered is vast. At the small offices where they operate, there was no space to establish a library to facilitate book clubs. The ICT infrastructure is also minimal, with the result that the server is frequently not working and one cannot access the database as and when needed. Furthermore, in the focus group held with staff, the concern was raised that there had not been any salary increases for the last four years due to shortage of funding, and that this has had an impact on staff morale. The lack of funding threatens the sustainability of the program, specifically the retention of child care workers and staff. This could lead to a loss of the expertise and experience that has been built up over many years. Partnerships While the importance of working with a wide range of organizations to deal with the variety of problems that affect the beneficiary group is central to the program methodology, the flexibility and informality of many of the partnerships at times undermined the realization of this principle. Partnership and stakeholder involvement could have been more structured. Some stakeholders such as Department of Health indicated that “they had resources but without an MoU, it’s difficult for SEP to access and benefit from the Departmental programs and resources including materials and training programs.” The organization has been in the process of entering into an MoU with the department, but it has been delayed due to bureaucratic requirements. The other respondent explained, “two-way partnership is desired, one-sided partnership is not sustainable. Woz’obona brought us close to their programs, but did not come close to our programs of similar campaigns to avoid duplication with two-way partnerships.” Service Delivery Emphasis Versus Advocacy Approach The program’s interventions to secure access to the services and protection necessary for the rights of OVC has assumed a strong service-delivery orientation, rather than a longer-term more sustainable advocacy focus. In comparison to the household economic strengthening interventions, which have focused on empowering caregivers to take ownership of available resources and assert their rights to ensure their own development, the child-focused interventions either provide a service or engage with other service providers on behalf of the child to ensure that they provide the relevant service. There is room for more advocacy-focused interventions to empower caregivers and children to become agents of change. This can be done by empowering children and caregivers to become stronger advocates for the changes that are needed to respond to children’s needs. Woz’obona could take a stronger facilitative role in linking children, caregivers, service providers, and decision makers to identify and address rights-abuses such as corporal punishment and safe routes to schools. At present, the network appears to be a strongly service-delivery focused network, rather than focusing on empowering children and caregivers to exercise their rights. The latter approach is inherently more sustainable than a stronger service-delivery approach to the creation of an enabling environment for OVC and to building the resilience of OVC. Contextual Barriers Corporal Punishment The focus groups conducted with children revealed high levels of abuse by teachers and continued use of corporal punishment, despite the legal prohibition and criminalization of abuse and corporal Woz’obona UGM End of Project Evaluation Report 31 punishment in terms of the South African Schools Act, 1996 and the Children’s Act No. 38 of 2005. The children indicated that abuse levels are high and that they have no method or means of reporting corporal punishment, and where it has been reported, nothing has been done about it. The issue of corporal punishment is especially relevant given that the most recent Stats South Africa General Household Survey (2011) has shown an increase in corporal punishment at schools from 16.8% in 2009 to 17.2% in 2011. Increases in Limpopo are higher than the average, jumping from 14.6% to 19.3%. The prolific use of corporal punishment and its apparent acceptance in the target communities is indicative of harmful cultural attitudes to child protection and the use of violence against children. This requires any child protection program to focus on addressing underlying attitudes and tackling clear manifestations of harmful practices associated with these attitudes. The evidence suggests that, while the program worked to ensure cultural sensitivity, there was insufficient focus on advocacy to change harmful attitudes and practices. Cultural Beliefs It has been observed that men’s role in the organization’s programs, including the volunteers, is very minimal. This has not been a deliberate strategy but reflects the traditional attitude that taking care of children is a woman’s role, and has made it difficult for the organization to recruit men as volunteers and for men to participate effectively in children’s programs. As one respondent noted, “we have never consciously sought to leave men out, but it so happens that women respond [more] easily to volunteer work than men. We once had some men but they never numbered more than five at any given time. I guess it is just in line with the allocation of duties by society wherein child￾caring and child-rearing are mostly seen as the role reserved for women, but we do not have a policy as an organization of targeting only women.” In the targeted communities, there is a strong belief in witchcraft and this has impacted on the success of program activities. As one person noted, “It is difficult for some parents to even give Woz’obona/SEP’s books for fear of being bewitched. We are working hand in hand with teachers to assist them with homework.” The underlying traditional gender stereotypes that influence the lack of recruitment of male volunteers effects more than the involvement of men in the care of children. It is also at the heart of various harmful and violent practices against women and children which drive the high rate of HIV infection among girls in rural South Africa. The HIV/ AIDS National Strategic Plan recognizes the link between gender-based violence (GBV) and risk of HIV infection, and calls for HIV prevention measures to address harmful gender stereotypes and GBV. When the issue was explored with children during the focus groups, it became apparent that the HIV prevention interventions have not been designed to address the issue of gender or to educate children on their gender rights and promote changed attitudes among boys and girls. There is space to introduce a gender component into all elements of the program in moving forward, including in HIV prevention training, so as to better equip boys and girls to address the drivers of inequality and HIV transmission among young girls in traditional communities. Stigma Attached to HIV The children’s enthusiasm for participating in the session dealing with HIV prevention and the depth and detail of their answers indicated that they have good knowledge and confidence in the area of HIV prevention. However, among the adults there is still much work to do. Stigma associated with HIV/ AIDS has continued to impede the realization of program objectives. For example, it has negatively impacted the ARV referral component of the health program as, in the words of one respondent, stigma associated with HIV means that “parents are not willing to have referral letter signed and returned to SEP as a way of verifying that they have indeed been referred to the clinic for medical assistance.” V. CONCLUSIONS In conclusion, the evaluation found that the Woz’obona program was relatively successful in achieving its overarching goal of creating an enabling environment in which children are stimulated to develop and learn while receiving proper care and protection. Overall, the evaluation has found that the health, psychological support, and educational needs of OVC have been met. The organization has consistently exceeded its annual targets for securing services for OVC. Moreover, the program has understood the needs of the constituency it serves. While these needs are complex, interrelated, and at times overwhelming, Woz’obona has implemented a program which has identified and prioritized interventions sensitive to the community and local OVC context. The program has enabled significantly more children to access ECD services and improved the quality of ECD provision through recruiting community volunteers to provide the training. , ensuring that they complete ECD training programs, providing accredited training to caregivers and monitoring their competence, and establishing a home-based ECD approach. The program has demonstrated that it is possible to train members of the community to be competent and accredited professionals, to engage large numbers of children in ECD care and to overcome the community perception that such a service is unnecessary. The Limpopo Department of Social Development’s pilot of home-based ECD demonstrates the significant change that the program has had and the benefits that these activities may offer. The program has contributed to increasing educational opportunities and the improvement of educational outcomes for OVC by providing services which supplement the school curriculum and support OVC with homework. The program has trained community members and institutions to better understand HIV/AIDS and preventing transmission. Workshops have been held in participating schools and the levels of knowledge are significantly improved. The program has had significant impact on ensuring that children have access to both health and psychological services, thereby contributing towards building their resilience. There were, however, needs which have not been recognized and addressed in the program design. These include children feeling unsafe, both in the community and at school; bullying in schools; the high rate of teenage pregnancies; and the lack of information on gender, harmful stereotypes, and core issues linked to HIV/ AIDS such as the co-occurrence of TB. The HES program has transformed economic opportunities for participating households. It has empowered beneficiaries with a range of skills and options to improve their household income, and evidence suggests that the practices developed in the program will be sustained. The close alignment with national policies and programs has enabled the program to maximize limited resources and allowed the forging of close working partnerships with state agencies. The capacity of both the staff and child caregivers, as well as the institution has been positively impacted upon. The organization has now essential elements such as an M&E Plan, tools, and capacity in the development and implementation of M&E systems. Woz’obona UGM End of Project Evaluation Report 33 VI. RECOMMENDATIONS The work undertaken by Woz’obona/SEP over the past five years has had a positive impact on the communities in the Makhuduthamaga Municipality. As the program draws to a close in September 2012, these recommendations will focus on building on the strengths and learnings of the past five years so as to maximise the program impact in the future. A Stronger Advocacy Orientation The next stage of the program should look at a reorientation away from the strong focus on service delivery to a stronger focus on sustainable beneficiary-driven advocacy. An advocacy orientation requires a stronger emphasis on empowering children and their caregivers with knowledge of their rights and remedies for redress in terms of current national policies and programs, rather than on the provision of material support and direct provision of services such as psycho-social support. Parents and children should be empowered to know and claim their rights in conflict situations or in situations where they are denied access to services. This should be complemented with awareness-raising amongst the relevant service providers and the provision of support to children to hold service providers to account for failure to respect, protect and promote their rights. For example, in the case of school uniforms, children, caregivers and teachers must be educated about the rights of children not to be discriminated against. In the case of corporal punishment, children, caregivers and educators should be educated on the prohibition against corporal punishment and on alternative positive forms of discipline and the advantages of the employing these. An advocacy orientation also requires a focus on building the capacity of not only children, but also parents and volunteers to engage in advocacy to tackle the underlying causes of vulnerability – for example, to lobby the local government, local police/ community policing forums to create safe routes to school by cutting down the long grass and providing regular presence at the routes in question in the morning and afternoon; lobby schools and encourage parental participation in school governing bodies to improve the school space, to develop a school code of conduct and child￾friendly reporting mechanism for children to report violence and corporal punishment; train volunteers on corporal punishment and workshops with schools on positive alternative forms of discipline and make a clear reporting line available through Woz’obona systems. Likewise, an advocacy orientation requires the promotion and facilitation of opportunities for children to be drivers of change and express their concerns and needs with traditional leaders and local government service providers directly so as to encourage children’s participation and to encourage accountability for absent services. Given the level of respect government agencies and other stakeholders have for the program there would be value in the organization actively bringing together interested parties to make the program experience visible and to explore ways in which the work can continue. Strengthen and Formalize Partnerships Through Consultative Processes Government agencies have supported or incorporated several activities of the project, but it is unlikely they really understand the role Woz’obona/SEP has played in facilitating community participation and ensuring activities to continue on a daily basis. It is therefore suggested that in the next three months the organization should:  Convene a results sharing workshop with stakeholders and agree on the way forward  Explore the possibility to convene quarterly meetings with key stakeholders to coordinate and collaborate on program activities;  Institutionalize the partnership by entering into Memorandum of Understanding with relevant government departments and ensure that resources are leveraged and duplication is avoided The Strategic Shaping of Interventions to Address Drivers of Risk and Vulnerability The evaluation has shown that in some areas of program delivery not all the beneficiaries’ needs are being adequately met, and in some cases, the shape of the interventions have not been designed to adequately address the specific risks and challenges faced by children in the context of specific needs. Issues such as school safety, child and adolescent sexuality, gender and traditional attitudes to the role and place of children could all be more robustly addressed through a more strategic review and development of interventions aggressively tackled by the project. In moving this recommendation forward, Woz’obona should  Implement a regular review processes to assess the emerging needs of children in the community as measured against the prevailing policy and service delivery backdrop and realign its program focus and intervention design to ensure a close fit between the two.  The choice and design of interventions related to HIV-prevention education should be guided by national, provincial and district level information about trends in sexual behavior, age of sexual debut as well as the gender dimensions informing relationships in rural communities. Opportunities to Scale-up Program Delivery The program model used by Woz’obona/SEP however is not one which is likely to be applied to state agencies. The importance of building relationships, focusing on the child’s needs, capacitating the members of OVC households to become service providers and monitoring a set of interrelated contexts and needs is not an activity which can be transferred to any one agency. As the program documents its practice and refines its learning’s with other stakeholders and interested parties it will be essential that questions of scale are addressed. There would be value in:  Developing a standard costing model for the different services the program has provided  Defining the essential job profiles to implement the program to the required standards  Identifying different agencies which may be interested in supporting the continuance and expansion of the program. Address Sustainability of the Program Sustainability of the program should be addressed through the following interventions:  Through the diversification of funding sources and the development of a resource mobilization strategy.  The program should review its programs to ensure maximum alignment with policy and programmatic priorities of the government of South Africa as a means of ensuring that investments in interventions will continue to yield returns through complementary state interventions. This requires regular review of the program to adjust it to the dynamic and ever￾evolving policy environment. In addition, close alignment with state objectives creates greater opportunities for securing funding from departments and development partners.  The piloting of the ECD model by the Limpopo Department of Social Development offers another window of opportunity for program sustainability and funding. ECD and Home-Based Community Care are part of the Expanded Public Works Program under the social cluster. If the pilot proves successful, the organization is well-placed to becoming an implementing agent, which will attract funding from government and therefore the organization will retain its expertise and experience built up over the years.  The greenhouse is a project that was build with the hope of implementing agriculture through the use of greenhouse technology; this intervention would go a long way to strengthen the economy of the households (HES). It has taken longer than the projected time because of the red-tape involved in building such a structure. A representative from the Department of Agriculture appreciated the fact that through Woz’obona/SEP training “parents now have Woz’obona UGM End of Project Evaluation Report 35 knowledge and skills for water efficient vegetable home gardens, income generation achieved with home gardens, nutrition improved and malnutrition and vitamin deficiency reduced”. Monitoring and Evaluation Through the technical support and mentoring from FHI 360-UGM, the organization has now a Monitoring and Evaluation Plan together with the necessary data collection tools and increased capacity to monitor, evaluate and report. It is recommended that the organization should:  Institutionalize the monitoring and evaluation process across all its programs and move towards developing a fully systematized Monitoring and Evaluation framework.  Ensure the collection of baseline statistics in relation to the services to be provided as well as the objectives sought to be realized so as to be able to measure progress and review program efficiency and effectiveness at regular intervals.  The database of beneficiaries should be updated and made available for use by other institutions working to serve OVC.  The model and practices used in the program should be clearly documented and shared with relevant stakeholders. VII. APPENDICES APPENDIX I: PERSONS CONTACTED Woz’obona/SEP (Management)  Lucy Thornton, National Director  Peter Sekgobela, Regional Director  Shadrack Mphela, Program Manager  Ephenia Matebane, M&E Manager  Dennis Gorivoto, Finance Manager Woz’obona/SEP (Staff)  Charity Bakwa  Letty Kgoale  Alice Magatikele  Selina Ranala  Lauda Mashabela  Thelma Mokwala  Anna Ratau Woz’obona UGM End of Project Evaluation Report 37 APPENDIX II: SCOPE OF WORK Now in the fifth and final year of the project, FHI 360-UGM, at the request of USAID, is commissioning an external evaluation of our grantees. Partner organizations are non-governmental organizations (NGOs) working at national, provincial and local levels in South Africa, primarily implementing services related to services for orphans and vulnerable children (OVC), HIV care and support, HIV counseling and testing, and HIV prevention. These partners have received funding for a period of three to five years under PEFPAR, as well as both organizational and technical capacity building support. BtC Break the Chains is being contracted to execute the evaluation for one UGM Partner: Woz’obona. The focus of the each partner evaluation will be to:  Determine whether the program objectives under each partner’s program were achieved  Evaluate the key program outcomes and impacts related to improved health and wellbeing of the targeted beneficiaries Most specifically, BtC Break the Chains will seek to answer the following key evaluation questions for Woz’obona, utilizing tools, methods, and sub-questions approved reviewed and approved by FHI 360:  What were the most significant changes brought about by Wozo improving the well-being of OVC in targeted communities?  To what extent was Wozo able to address the needs of children within the community?  How do stakeholders (children, care givers, DoSD, community representatives) perceive the program; in terms of quality and ease of access? The focus of the evaluation is to assess effectiveness of the partner organization in addressing the needs of beneficiaries in targeted communities. The evaluators will be required to carefully consider the suitability and feasibility of design options that are likely to offer the best chance of establishing the value of the program in responding to the needs of targeted beneficiaries and communities. Both qualitative and quantitative data collection techniques should be employed. Data will be collected from various sources using appropriate data collection methods and tools for any given evaluation question. The final design to be employed will be determined after the contractor has had a chance to undertake a front-end analysis and is therefore able to select the best design option that specifies; which people or units will be studied; how they will be selected and the kinds of comparison that should be made. Data will be collected from various program sites for the partner. The evaluation will be undertaken in two stages and with expected outcomes for each stage as expressed below: Stage 1: Finalization of Evaluation Protocol Contractor will refine an evaluation protocol which demonstrates:  Understanding the relationship between program stages and the proposed broad evaluation question  Understanding the context for program delivery and key factors that influence program implementation  Understanding the existing theoretical and empirical knowledge about the program and examining program theory  A comprehensive stakeholder analysis and determination of roles of key stakeholders in the evaluation  Balancing costs and benefits of the evaluation and advising on the most strategic questions to include in the evaluation  Developing the Finalized Implementation Strategy and Methodology Report Stage 2: Implementation of the Evaluation Contractor will implement the partner evaluation following submission and approval of the Implementation Strategy and Methodology Report:  Pre-test instruments  Train data collectors  Undertake the evaluation data gathering process  Prepare data for analysis  Clean data  Enter data into electronic data analysis systems  Undertake comprehensive data analysis  Formulate the findings During the period of performance of April 16, 2012 –July 20, 2012, payment to BtC Break the Chains will be fixed price based on the payment schedule determined by the deliverables below: MILESTONES DELIVERABLES DUE DATE AYMENT Data collection (April 16 – May 11) 1. Finalized Implementation Strategy and Methodology Report submitted to FHI 360 April 20 15% 2. Evaluation Work Plan, including key activities and timeframes submitted to FHI 360 April 20 10% 3. Data Analysis Plan, including dummy table/graphs for presenting data submitted to FHI 360 May 4 15% Data analysis and development of PowerPoint Presentation, including summary of evaluation process and results (May 14 – June 1) 4. Oral and PowerPoint Presentation (half-day) of preliminary findings to USAID, FHI 360, and partner (May 28-June 1, 2012) Presentations completed by June 1 15% Development of final written report, including an executive summary with highlights of the evaluation and key findings (June 1 –July 20) 5. Draft written report submitted to FHI 360 June 15 15% 6. Final report submitted to FHI 360 July 20 15% Development of brief paper (two-pager) for the partner, targeting community audiences on key findings from the evaluation (June 1 –July 20) 7. Two-page paper submitted to FHI 360 July 20 15% Woz’obona UGM End of Project Evaluation Report 39 APPENDIX III: EVALUATION TEAM AND STAKEHOLDER INTERVIEWS Evaluation Team Name Qualifications Expertise Role Andries Mangokwana Master’s degree- Social Planning and Administration Project Management, Impact assessment and M & E systems Team Leader to Project Manage and liaison with the client Patricia Martin Master’s of Law Children’s Rights Expert Children’s Rights Expert Oscar Musandu-Nyamayaro Master of Science Degree in Urban and Rural Development Planning Knowledgeable about government systems and quantitative data Senior Consultant Kashfi Mahmud Master’s degree in Philosophy ECD expertise and evaluation Consultant Gemi Malau B.Soc Sc ( Hons) Experience in social facilitation, youth programs and evaluation Consultant Limpho Klu Master’s Degree in Adult Basic Education Impact Assessment and M & E Consultant Stakeholder Interviews Community Representatives Name Organization/Position Mpogeng M Regional Program leader-Love Life Mohlala V Co-ordinator-Under 15 Zonal league Chabedi P Branch Executive committee Member-ANC Mphelane M Retired Teacher Sithole S Co-ordinator-Itireleng/Zamane Crèche Maphanga J.N Clinic Matenche GM Community Development Worker Kgoloko N Former Interim Local Chief Bapela C Youth Madihlaba JM Educator at Moretsele Primary School Kgopane R Ward Committee (Steering Committee) Member Sekwati A Pedi Mamone Tribal Council Mampuru N.S Tribal Headman Makwatanyane J Drop in Center (support Group) Facilitator Nkadimeng S.M MK Umbrella Government Stakeholders Stakeholder Group Name Organization Education Mr Aphane Moraka Johannes Department of Education, Lebowakgomo Offices Social Services Vhaloyi Xikombisa Yolanda Department of Social Welfare, Jane Furse Local Government Queen Makhabela Makhuduthamaga Local Municipality, Ex Mayor Agriculture Tshwane Phatudi-Mpahelele Department of Agriculture, Nebo Regional Office Home Affairs Seraki MS Department of Home Affairs, Jane Furse Regional Office Health Mphelane MC Department of Health, Madibong Clinic Local Government/ Community Phetla RE Makhuduthamaga Municipality, ex￾volunteer Security Lesufi TS SAPS Jane Furse Woz’obona UGM End of Project Evaluation Report 41 APPENDIX IV: INSTRUMENTS DESIGNED Instrument 1: Management PROGRAM EVALUATIONS FOR HIV PROGRAM PARTNERS- UMBRELLA GRANTS MANAGEMENT PROJECT FHI 360 I. Interview Schedule for Woz’obona Management Staff Questionnaire No: Interviewer: Date of Interview: Interviewee Details: Name of Interviewee : Designation : Physical address : __________________________________________________________ Land line : __________________________________________________________ Mobile no : __________________________________________________________ E-mail address : __________________________________________________________ 1. Background and History of the Project 1.1 Tell me about the Woz’obona project. Probe: how did it start? Why was it initiated? 1.2 What is your role in the Woz’obona project? 2. Program Design/ Strategy 2.1 Tell me how the program was designed/conceptualised? 2.2 To what extent is the overall design of the project relevant? 2.3 What are the strengths of the strategy? 3. Effectiveness 3.1 To what extent are the project objectives being achieved or are expected to be achieved? ( The interviewer should probe for all the key objectives as outlined in the Project Plan) 3.2 How has implementation taken place? What went according to plan and what did not? 3.3 What has enabled you to achieve the objectives? 3.4 What were the barriers that made it difficult to achieve your objectives? 3.5 Stakeholder Involvement What is the nature of the involvement of stakeholders and how effective is their involvement? 4 Efficiency 4.1 Would you say that resources for delivering this project have been well managed and utilised? Explain. 4.2 What could have been done to make implementation more efficient / cost friendly (time labour) and achieved same results? 5 Relevance To what extent was Woz’obona able to address the needs of children within the community? 6 Impact 6.1 What were the most significant changes brought about by Woz’obona in improving the well￾being of OVC in targeted communities? Woz’obona UGM End of Project Evaluation Report 43 6.2 Are there signs of foreseen or unforeseen positive or negative impacts? Explain 7. Overall Impressions 7.1 What would you regard as the main successes? 7.2 What would you regard as the main challenges? 7.3 What are the main lessons learned?/ What would you change the program and why? 8. Recommendations What recommendations would you make? Specific Comments (Description of interviewer’s observations, insights, reflections) Thank you very much for your time and effort! Instrument 2: Focus Group Guideline-Woz'obona Staff - PROGRAM EVALUATIONS FOR HIV PROGRAM PARTNERS- UMBRELLA GRANTS MANAGEMENT PROJECT FHI 360 - Guideline: Focus group discussion with Woz’obona Project Staff - TOTAL TIME NEEDED: 3 Hrs Instructions to facilitator  This is a focus group / workshop with Woz’obona Project Staff  Remember, you cannot paraphrase what people are saying enough. Paraphrasing means summarising what people are saying in your own words. You should constantly be checking your understanding, by saying things like "so what you are saying is …", or "If I understand you correctly, you are suggesting that....". This is especially useful if there is a difference of opinion, then you can say “I see there are two views here, the one is that … and the other is that ….. Do you agree with me?” Then you can move on.  The time frames are very tight. Focus on getting key information and keeping strictly to the times allocated per question.  Use the flipchart paper if you want to capture what people are saying.  Remember to establish ground rules before you start!!!  Most important of all, relax and enjoy the discussions. 1. Welcome and overview  Time: 10 minutes Hello, my name is------------------------------------------------------------Thank you very much for agreeing to this focus group discussion. I am doing a program evaluation for the Woz’obona. The evaluation is being commissioned to obtain an independent perspective on the following: 1. Determine whether the program objectives were achieved; 2. Evaluate the key program outcomes and impacts related to improved health and well being of the targeted beneficiaries. This is so that we can all be objective about the project and how it has been implemented. Much of the focus of this discussion will be on the success, challenges, lessons learned and recommendations.  We want to understand what your experiences have been.  There are no right or wrong answers here; in fact, I am hoping that you give me the information that we do not know. The information will be used to improve the project in the future.  I am sure that not everyone will agree with everything that is said. Please feel free to disagree with others opinions, we want to know all the different opinions around the table. But let us disagree in a respectful way. Woz’obona UGM End of Project Evaluation Report 45  Feel free to express yourselves, this is a safe space – you will not be judged for your opinion. Can we agree on that?  I hope that you will find it interesting to share information with each other.  Does anyone have any questions before we start?  Explain the time table and ask if anyone has any time concerns.  The Focus Group will last about 3 hours.  Complete the attendance register  Do a ground rules or rules for sharing exercise before you begin. Ask people to raise 6 rules for the group behavior e.g. respect each others opinions, share openly, go to the toilet when you need to, switch cell phones off. Write these on the flip chart paper (10 minutes) 2. Introductions  Outcomes  People feel relaxed, an open atmosphere is created  Create a positive frame of mind and identify the highlights of the project.  Time: 15 minutes 3. Background and History of the project  Outcomes: Establish understanding of the background and history of the project.  Time: 15 minutes 3.1 Tell me about Woz’obona project and its history. Probe: how did it start? Why was it initiated? 3.2 How has the project grown and transformed since inception up to now? Probe for changes in:  Overall objectives  Types of intervention  Relationships with internal stakeholders  Relationships with external stakeholders  Why have these changes occurred? ESTABLISH: What the background and history of the project is. 4. Project implementation  Outcomes: Establish how the project has been implemented in relation to effectiveness; efficiency; relevance, impact and lessons learned.  Time: 1 and 30 minutes 4.1 Effectiveness  To what extent are the project objectives being achieved?  How has implementation taken place? What went according to plan and what did not?  Is the target group/beneficiary group receiving services as intended? 4.2 Efficiency  Would you say that resources for delivering this project have been well managed and utilised? Explain. 4.3 Relevance  To what extent was Woz’obona able to address the needs of children within the community? 4.4 Impact  What were the most significant changes brought about by Woz’obona in improving the well￾being of OVC in targeted communities? Probe for impact at Individual level, household and community  Are there signs of foreseen or unforeseen positive or negative impacts? ESTABLISH: How the project has been implemented in relation to effectiveness; efficiency; relevance and impact. 5. Successes and challenges  Outcomes: Identify the successes and challenges of the project to date.  Time: 30 minutes.  Divide the flipchart into two; write successes in the one column and challenges in the other  Ask the participants to brainstorm the successes and challenges thus far.  Once the brainstorm is complete, ask if anyone wants to disagree with anything that is on the board, ESTABLISH: Successes and Challenges of the Project. 6. Lessons learned  Time: 15 min  Outcomes: What have been the lessons learned in the implementation of the project?  Identify what has been the lessons learned from the project implementation. ESTABLISH: Lessons learned 7. Closing off  Time: 5 minutes  We have come to the end of the discussion. Is there anything that anyone wants to say to close off.  Thank you very much for your time and for sharing your views with me. Woz’obona UGM End of Project Evaluation Report 47 Instrument 3: Focus Group Guideline – Woz'obona Caregivers - PROGRAM EVALUATIONS FOR HIV PROGRAM PARTNERS- UMBRELLA GRANTS MANAGEMENT PROJECT FHI 360 - Guideline: Focus group discussion with Caregivers - TOTAL TIME NEEDED: 2 Hrs Instructions to facilitator  This is a focus group / workshop with Caregivers  Remember, you cannot paraphrase what people are saying enough. Paraphrasing means summarising what people are saying in your own words. You should constantly be checking your understanding, by saying things like "so what you are saying is …", or "If I understand you correctly, you are suggesting that....". This is especially useful if there is a difference of opinion, then you can say “I see there are two views here, the one is that … and the other is that ….. Do you agree with me?” Then you can move on.  The time frames are very tight. Focus on getting key information and keeping strictly to the times allocated per question.  Use the flipchart paper if you want to capture what people are saying.  Remember to establish ground rules before you start!!!  Most important of all, relax and enjoy the discussions. 1. Welcome and overview  Time: 5 minutes Hello, my name is------------------------------------------------------------Thank you very much for agreeing to this focus group discussion. I am doing a program evaluation for the Woz’obona. The evaluation is being commissioned to obtain an independent perspective on the following: 3. Determine whether the program objectives were achieved; 4. Evaluate the key program outcomes and impacts related to improved health and well being of the targeted beneficiaries. This is so that we can all be objective about the project and how it has been implemented. Much of the focus of this discussion will be on the success, challenges, lessons learned and recommendations.  We want to understand what your experiences have been.  There are no right or wrong answers here; in fact, I am hoping that you give me the information that we do not know. The information will be used to improve the project in the future.  I am sure that not everyone will agree with everything that is said. Please feel free to disagree with others opinions, we want to know all the different opinions around the table. But let us disagree in a respectful way.  Feel free to express yourselves, this is a safe space – you will not be judged for your opinion. Can we agree on that?  I hope that you will find it interesting to share information with each other.  Does anyone have any questions before we start?  Explain the time table and ask if anyone has any time concerns.  The Focus Group will last about 2 hours.  Complete the attendance register  Do a ground rules or rules for sharing exercise before you begin. Ask people to raise 6 rules for the group behavior e.g. respect each others opinions, share openly, go to the toilet when you need to, switch cell phones off. Write these on the flip chart paper (10 minutes) 2. Introductions  Outcomes  People feel relaxed, an open atmosphere is created  Create a positive frame of mind and identify the highlights of the project.  Time: 15 minutes 3. Involvement of caregivers in the project  Outcomes: Establish how the caregivers were involved.  Time: 10 minutes 3.1 Tell me how did you get involved in Woz’obona project ? ESTABLISH: Involvement of the project. 4. Project implementation  Outcomes: Establish how the project has been implemented in relation to effectiveness; efficiency; relevance, impact and lessons learned.  Time: 45 minutes 4.1 Effectiveness  To what extent are the project objectives being achieved?  How has implementation taken place? What went according to plan and what did not?  Is the target group/beneficiary group receiving services as intended? 4.2 Relevance  To what extent was Woz’obona able to address the needs of children within the community? 4.3 Impact  What were the most significant changes brought about by Woz’obona in improving the well￾being of OVC in targeted communities? Probe for impact at Woz’obona UGM End of Project Evaluation Report 49 Individual, household and community level. Please give concrete examples of the changes that has occurred  Are there signs of foreseen or unforeseen positive or negative impacts? ESTABLISH: How the project has been implemented in relation to effectiveness; relevance and impact. 5. Successes and challenges  Outcomes: Identify the successes and challenges of the project to date.  Time: 20 minutes.  Divide the flipchart into two; write successes in the one column and challenges in the other  Ask the participants to brainstorm the successes and challenges thus far.  Once the brainstorm is complete, ask if anyone wants to disagree with anything that is on the board, ESTABLISH: Successes and Challenges of the Project. 6. Lessons learned  Time: 10 min  Outcomes: What have been the lessons learned in the implementation of the project?  Identify what has been the lessons learned from the project implementation. ESTABLISH: Lessons learned 7. Closing off  Time: 5 minutes  We have come to the end of the discussion. Is there anything that anyone wants to say to close off.  Thank you very much for your time and for sharing your views with me. Instrument 4: Focus Group Guideline - Woz'obona Beneficiaries PROGRAM EVALUATIONS FOR HIV PROGRAM PARTNERS- UMBRELLA GRANTS MANAGEMENT PROJECT FHI 360 Guideline: Focus group discussion with beneficiaries (Age group-15-17) TOTAL TIME NEEDED: 1.5 Hrs Instructions to facilitator  This is a focus group / workshop with beneficiaries  Remember, you cannot paraphrase what people are saying enough. Paraphrasing means summarising what people are saying in your own words. You should constantly be checking your understanding, by saying things like "so what you are saying is …", or "If I understand you correctly, you are suggesting that....". This is especially useful if there is a difference of opinion, then you can say “I see there are two views here, the one is that … and the other is that ….. Do you agree with me?” Then you can move on.  The time frames are very tight. Focus on getting key information and keeping strictly to the times allocated per question.  Use the flipchart paper if you want to capture what people are saying.  Remember to establish ground rules before you start!!!  Most important of all, relax and enjoy the discussions. Number of children: Site/Village: Venue: Date: Facilitator: Language : 1. Welcome and overview Ice- breaker 2. Introductions Beneficiaries feel relaxed, an open atmosphere is created Woz’obona UGM End of Project Evaluation Report 51 3. Involvement of beneficiaries in the project 3.1 When and how did you become part of the Woz’obona project? 3.2 What has been the highlight moment for you since your participation in the project? 4. Needs 4.1 What are your needs? 4.2 Which of those needs are being addressed by the Woz’obona? 5. Effectiveness 5.1How often do you get this help? 5.2 How easy is it to use these services? 5.3 Which of the services did you like most and why? 5.4 Which of the services did you dislike and why? 6. Impact 6.1 Since the participation in Woz’obona, how has your life changed and that of your family? 6.2 Which one (significant changes) for you stands out? 7. Closing off  Time: 5 minutes  We have come to the end of the discussion. Is there anything that anyone wants to say to close off?  Thank you very much for your time and for sharing your views with me. Instrument 5: Semi Structured Interview-Stakeholders PROGRAM EVALUATIONS FOR HIV PROGRAM PARTNERS- UMBRELLA GRANTS MANAGEMENT PROJECT FHI 360 II. Interview Schedule for Stakeholders (Dept of Social Development, Education, Health, Agriculture, Home Affairs, Safety and Security, District and Local Municipality) Questionnaire No: Interviewer: Date of Interview: Interviewee Details: Name of Interviewee : Designation : Physical address __________________________________________________________ Land line __________________________________________________________ Mobile no __________________________________________________________ E-mail address __________________________________________________________ 4. Background and Context of the Project 1.1 What is your position and role within your organization? 1.2 What has been your involvement with Woz’obona project? Probe for the support provided by the different stakeholders 5. Effectiveness of the Program 2.1 Has the program managed to provide services for the intended beneficiaries/target group? 2.2 How well do you think the program has been successful in addressing the needs of the children? 2.3 What will you say has been your organization’s contribution in the attainment of Woz’ona’s objectives? Woz’obona UGM End of Project Evaluation Report 53 3 Relevance To what extent was Woz’obona able to address the needs of children within the community? 4 Impact 4.1 What were the most significant changes brought about by Woz’obona in improving the well￾being of OVC in targeted communities? 5 Overall Impressions 5.1 What would you regard as the main successes? 5.2 What would you regard as the main challenges? 5.3 What are the main lessons learned? / What would you change the program and why? 6. Recommendations What recommendations would you make? Specific Comments (Description of interviewer’s observations, insights, reflections) Thank you very much for your time and effort! Instrument 6: Semi Structured Interview – Community Members PROGRAM EVALUATIONS FOR HIV PROGRAM PARTNERS- UMBRELLA GRANTS MANAGEMENT PROJECT FHI 360 III. Interview Schedule for Community Members (To be determined but could include Makgosi, church leaders, SGB members, ward committee members, health workers, CSO representatives) Questionnaire No: Interviewer: Date of Interview: Interviewee Details: Name of Interviewee : Designation : Physical address : __________________________________________________________ Land line : __________________________________________________________ Mobile no : __________________________________________________________ E-mail address : __________________________________________________________ 1. Background and Context of the Project 1.1 What is your position and role within the community? 1.2 What is it that you know about the Woz’obona project? 2 Relevance 2.1 What are the biggest challenges faced by children in the community? 2.2 To what extent was Woz’obona able to address the needs of children within the community? Woz’obona UGM End of Project Evaluation Report 55 3 Impact 3.1 What were the most significant changes brought about by Woz’obona in improving the well￾being of OVC in targeted communities? 4 Overall Impressions 4.1 What would you regard as the main successes of the project? 4.2 What would you regard as the main challenges of the project? 5. Recommendations What recommendations would you make? Specific Comments (Description of interviewer’s observations, insights, reflections) Thank you very much for your time and effort! Instrument 7: Focus Group Household Beneficiaries PROGRAM EVALUATIONS FOR HIV PROGRAM PARTNERS- UMBRELLA GRANTS MANAGEMENT PROJECT FHI 360 - Guideline: Focus group discussion with Household Beneficiaries - TOTAL TIME NEEDED: 2 Hrs Instructions to facilitator  This is a focus group / workshop with Household Beneficiaries  Remember, you cannot paraphrase what people are saying enough. Paraphrasing means summarising what people are saying in your own words. You should constantly be checking your understanding, by saying things like "so what you are saying is …", or "If I understand you correctly, you are suggesting that....". This is especially useful if there is a difference of opinion, then you can say “I see there are two views here, the one is that … and the other is that ….. Do you agree with me?” Then you can move on.  The time frames are very tight. Focus on getting key information and keeping strictly to the times allocated per question.  Use the flipchart paper if you want to capture what people are saying.  Remember to establish ground rules before you start!!!  Most important of all, relax and enjoy the discussions. 1. Welcome and overview  Time: 5 minutes Hello, my name is----------------------------------------------------------- from Break the Chains Development Services. Thank you very much for agreeing to this focus group discussion. I am doing a program evaluation for the SEP. The evaluation is being commissioned to obtain an independent perspective on the following: 1. Determine whether the program objectives were achieved; 2. Evaluate the key program outcomes and impacts related to improved health and well being of the targeted beneficiaries; 3. This is a knowledge generating evaluation commissioned by USAID; and 4. The evaluation will not be used to determine future funding. This is so that we can all be objective about the project and how it has been implemented. Much of the focus of this discussion will be on the success, challenges, lessons learned and recommendations. . We want to understand what your experiences have been.  There are no right or wrong answers here; in fact, I am hoping that you give me the information that we do not know. The information will be used to improve the project in the future.  I am sure that not everyone will agree with everything that is said. Please feel free to disagree with others opinions, we want to know all the different opinions around the table. But let us disagree in a respectful way. Woz’obona UGM End of Project Evaluation Report 57  Feel free to express yourselves, this is a safe space – you will not be judged for your opinion. Can we agree on that?  I hope that you will find it interesting to share information with each other.  The Focus Group will last about 2 hours.  Complete the attendance register 2. Introductions  Outcomes  People feel relaxed, an open atmosphere is created  Create a positive frame of mind and identify the highlights of the project.  Time: 15 minutes 2.1 Can you please tell us your name, how many people there are in your household and (ask for some other trivial data, like favourite color/animal/household product, to get group at ease). 3. Involvement of beneficiaries in the Household Economic Services (HES) program  Outcomes: Establish how the household beneficiaries were assessed and involved in the program  Time: 20 minutes 3.1 Tell me, have you ever been visited by any of the SEP staff? How was the visit and what did they say/ask you? 3.2 Have you ever done a cross visit, or did someone visit you on a cross visit? What is a special moment from the visit? 3.3 Have you been to any of the SEP workshops? What was your highlight moment? ESTABLISH: Involvement of household beneficiaries’ feedback in the program. 4. Project implementation  Outcomes: Establish how the project has been implemented in relation to effectiveness, relevance, and impact.  Time: 45 minutes 4.1 Effectiveness To what extent are the project objectives being achieved? (Note to facilitator: Probe for each specific objectives as follows) 1. To start and continue a savings program, eliminate debt 2. That cross-learning with regards to skills, finances took place 3. Started and continued small businesses (self-income generation)  Is the target beneficiary receiving focused services as intended? 4.2 Relevance  Did the SEP programs able help you in making your financial situation better? How did it help you? Please explain. 4.3 Impact  What were the most significant changes brought about by SEP in improving the financial well-being of the beneficiaries? Probe for impact on: (Please give concrete examples of the changes that has occurred)  Household budget  Debt elimination  Savings programs  Income generating activities (small businesses) have signs of sustainability  Access to opportunities/materials not available before  Changes in the overall well-being of the household  Immunization campaigns  How have the children’s behavior changed  Has the OVC academic performance improved  Has your child been referred for health services?  Regular awareness on child health?  Are there signs of foreseen or unforeseen positive or negative impacts? ESTABLISH: How the project has been implemented in relation to effectiveness; relevance and impact. 5. Successes and challenges  Outcomes: Identify the successes and challenges of the project to date.  Time: 20 minutes.  Divide the flipchart into two; write successes in the one column and challenges in the other  Ask the participants to brainstorm the successes and challenges thus far.  Once the brainstorm is complete, ask if anyone wants to disagree with anything that is on the board, ESTABLISH: Successes and Challenges of the Project. 7. Data Sheet  Time: 10 min  Outcomes: Collection of individual household data and identification by facilitator of candidates for further individual interview 7.1 Ask each participant to collect a data sheet (as attached). Explain each field, and help them to fill in the details. ESTABLISH: Collection of data 8. Lessons learned  Time: 10 min  Outcomes: What have been the lessons learned from the program?  Identify what has been the lessons learned from the program implementation. ESTABLISH: Lessons learned 7. Closing off  Time: 5 minutes  We have come to the end of the discussion. Is there anything that anyone wants to say to close off.  Thank you very much for your time and for sharing your views with me. Woz’obona UGM End of Project Evaluation Report 59 Instrument 8: Focus Group Former Child Beneficiaries - PROGRAM EVALUATIONS FOR HIV PROGRAM PARTNERS- UMBRELLA GRANTS MANAGEMENT PROJECT FHI 360 - Guideline: Focus group discussion with Former Child Beneficiaries - TOTAL TIME NEEDED: 2Hrs Instructions to facilitator  This is a focus group / workshop with Former Child Beneficiaries  Remember, you cannot paraphrase what people are saying enough. Paraphrasing means summarising what people are saying in your own words. You should constantly be checking your understanding, by saying things like "so what you are saying is …", or "If I understand you correctly, you are suggesting that....". This is especially useful if there is a difference of opinion, then you can say “I see there are two views here, the one is that … and the other is that ….. Do you agree with me?” Then you can move on.  The time frames are very tight. Focus on getting key information and keeping strictly to the times allocated per question.  Use the flipchart paper if you want to capture what people are saying.  Remember to establish ground rules before you start!!!  Most important of all, relax and enjoy the discussions. 1. Welcome and overview  Time: 5 minutes Hello, my name is----------------------------------------------------------- from Break the Chains Development Services. Thank you very much for agreeing to this focus group discussion. I am doing a program evaluation for the Woz’obona. The evaluation is being commissioned to obtain an independent perspective on the following: 5. Determine whether the program objectives were achieved; 6. Evaluate the key program outcomes and impacts related to improved health and well being of the targeted beneficiaries; 7. This is a knowledge generating evaluation commissioned by USAID; and 8. The evaluation will not be used to determine future funding. This is so that we can all be objective about the project and how it has been implemented. Much of the focus of this discussion will be on the success, challenges, lessons learned and recommendations. We want to understand what your experiences have been.  There are no right or wrong answers here; in fact, I am hoping that you give me the information that we do not know. The information will be used to improve the project in the future.  I am sure that not everyone will agree with everything that is said. Please feel free to disagree with others opinions, we want to know all the different opinions around the table. But let us disagree in a respectful way.  Feel free to express yourselves, this is a safe space – you will not be judged for your opinion. Can we agree on that?  I hope that you will find it interesting to share information with each other.  The Focus Group will last about 2 hours.  Complete the attendance register 2. Introductions  Outcomes  People feel relaxed, an open atmosphere is created  Create a positive frame of mind and identify the highlights of the project.  Time: 10 minutes 2.1 Can you please tell us your name, which area/village you are from, how many people there are in your household and (ask for some other trivial data, like favourite color/animal/household product, to get group at ease). 3. Initiation of beneficiaries into SEP  Outcomes: Establish how the child beneficiaries were assessed and initiated in the program  Time: 15 minutes 3.1 Tell me, when was the first time you heard of SEP? Who did you hear from and what did they tell you? 3.2 How did you decide to be part of SEP? What was your highlight moment? ESTABLISH: Initiation of former child beneficiaries in the program. 4. Project implementation  Outcomes: Establish how the project has been implemented in relation to effectiveness, relevance, and impact.  Time: 45 minutes 4.1 Effectiveness To what extent are the project objectives being achieved? ( Probe for each specific objectives as follows: 1. Did they help with school? a. Were you part of the book club? b. Did you take extra classes? c. What else did SEP do to help you w/ school? (encourage attendance, help w/ homework, create motivating community?) 2. Do you remember the HIV prevention education? (classes + peer) 3. Did SEP help you access health facilities? a. Did you receive immunization through them? Woz’obona UGM End of Project Evaluation Report 61 b. Were you reffered to a doctor by them? c. Did they train people around you (family, etc) to help w/ your treatment? d. Were you part of a child feeding program? 4. Did the program help your overall psychological well-being? a. Did they refer you to a professional counsellor? b. Did you do memory work sessions with them? 5. Was there follow up after you first entered the program? (end of the year, etc? How was follow up conducted, by whom?) 6. Did any of your families receive help from SEP w/ regards to economic strengthening? (workshops on budgeting, starting savings plans, starting small businesses) 4.2 Relevance  Did the SEP programs help you in making your situation better? With school? With family? With community? With health? With understanding HIV? 4.3 Impact  What were the most significant changes brought about by Woz’obona in improving the well￾being of the former beneficiaries? Probe for impact on: (Please give concrete examples of the changes that has occurred)  School performance (grades, homework, reading ability, etc.)  Understanding about HIV and prevention  Access to health facilities  Psychological well-being  Economic strengthening in their families  Are there signs of foreseen or unforeseen positive or negative impacts? ESTABLISH: How the project has been implemented in relation to effectiveness; relevance and impact. 5. Successes and challenges  Outcomes: Identify the successes and challenges of the project to date.  Time: 15 minutes.  Divide the flipchart into two; write successes in the one column and challenges in the other  Ask the participants to brainstorm the successes and challenges thus far.  Once the brainstorm is complete, ask if anyone wants to disagree with anything that is on the board, ESTABLISH: Successes and Challenges of the Project. 7. Data Sheet  Time: 15 min  Outcomes: Collection of individual household data and identification by facilitator of candidates for further individual interview 7.1 Ask each participant to collect a data sheet (as attached). Explain each field, and help them to fill in the details. ESTABLISH: Collection of data 8. Lessons learned  Time: 10 min  Outcomes: What have been the lessons learned from the program?  Identify what has been the lessons learned from the program implementation. ESTABLISH: Lessons learned 7. Closing off  Time: 5 minutes  We have come to the end of the discussion. Is there anything that anyone wants to say to close off.  Thank you very much for your time and for sharing your views with me. Woz’obona UGM End of Project Evaluation Report 63 APPENDIX V: WORK PLAN Date Time Stakeholder Method Team member 23/04/2012 10h00 Caregivers Focus Group Limpho 14h00 Caregivers Focus Group Limpho Any time Management Interview Kashfi Management Interview Kashfi 24/04/2012 10h00 Staff Focus Group Kashfi 10h00 Household Beneficiary Focus Group Limpho 14h00 Household beneficiary Focus Group Limpho Management ( National Director) Interview Andries 25/04/2012 09h00 Community Reps Interviews Andries/Kashfi/Limph o 15h00 Former Beneficiary Focus Group Limpho 26/04/2012 Community Reps Interviews Andries/Kashfi/Limph o 30/04/2012 10h00 Children Focus Group Gemi/Patricia 14h00 Children Focus Group Gemi/Patricia 01/05/2012 10h00 Children Focus Group Gemi/Patricia 14h00 Children Focus Group Gemi/Patricia 02/04-05/2012 Govt Stakeholders ( 9 ) Interviews Oscar Community Reps Interviews Oscar/Gemi    APPENDIX VI: REFERENCES Bailey K, Methods if Social Research, Second Edition, 1992 Sekgobela P, Creating a child Friendly Society, a paper presented at a National ECD Conference, 2012 Community Survey, Statistics S.A 2007 Adopted Integrated Development Plan, Makhuduthamaga Local Municipality, 2012 Peter Njaramba, Maleemisa Ntsala and Shanya Pillay, A case study-Sekhukhune Educare Project-A study of the Safety Nets Program,2008 South African National HIV Prevalence, Incidence Behavior and Communication Survey, HSRC, 2008 Statistics South Africa, 2012 Gregory Dees, Beth Battle Anderson, and Jane Wei-skillern, Scaling Social Impact Strategies for spreading social innovations,2004