USAID/South Africa Umbrella Grants Management Project End of Project Partner Evaluation HANDS AT WORK    July 2012 This publication was produced for the United States Agency for International Development. It was prepared by Nirvana Pillay and Nur Hidayati, of Results in Health, The Netherlands, and Health and Development Africa, South Africa. USAID/South Africa Umbrella Grants Management Project End of Project Partner Evaluation HANDS AT WORK This report is made possible by the generous support of the American people through the U. S. Agency for International Development (USAID), under the terms of APS 674-07-011, through United States Government’s President’s Emergency Plan for AIDS Relief, operated by FHI 360. The contents and opinions expressed herein do not necessarily reflect the views of USAID or the United States Government. Hands at Work UGM End of Project Evaluation ii ACKNOWLEDGEMENTS Results in Health and Health and Development Africa would like to extend their gratitude to the staff at Hands at Work, and all participating community-based organizations, for the full support they gave to this evaluation process. The evaluators greatly appreciated their availability to assist with all fieldwork logistics and respond to requests. Similarly, gratitude must be extended to all stakeholders and program beneficiaries who participated so actively in this assessment. This evaluation was implemented by the following people  Dr. Hilbrand Haak and Dr. Saul Johnson, Senior Technical Advisors  Ms. Nirvana Pillay, Senior Principal Researcher  Ms. Nur Hidayati, Senior Research Coordinator  Ms. María Belizán, Research Analyst  Ms. Priscilla Morley, Research Analyst    Hands at Work UGM End of Project Evaluation iii TABLE OF CONTENTS I. INTRODUCTION ..................................................................................................... 1 II. BACKGROUND ........................................................................................................ 3 III. METHODS ............................................................................................................... 10 IV. FINDINGS ................................................................................................................ 13 V. CONCLUSIONS ..................................................................................................... 33 VI. RECOMMENDATIONS ......................................................................................... 35 APPENDIX 1: COMPOSITION OF THE EVALUATION TEAM ............................ 39 APPENDIX 2: SAMPLING FRAMEWORK ................................................................ 40 APPENDIX 3: DATA COLLECTION TOOLS USED ............................................... 41 APPENDIX 4: LIST OF SITES AND DATES VISITED ............................................. 74 APPENDIX 5: PERSONS CONTACTED ................................................................... 75 APPENDIX 6: REFERENCES ....................................................................................... 77 APPENDIX 7: SCOPE OF WORK ............................................................................... 78 Hands at Work UGM End of Project Evaluation iv ACRONYM LIST ART Antiretroviral Therapy COP Country Operational Plan CBO Community-based Organization DoA Department of Agriculture DoH Department of Health DHA Department of Home Affairs DOTS Directly Observed Treatment Support for TB DSD Department of Social Development ECD Early Childhood Development FBO Faith-based Organization FGD Focus Group Discussion GRIP Greater Nelspruit Rape Intervention Programme HBC Home-based Care HDA Health and Development Africa HES Household Economic Strengthening KII Key Informant Interview MER Monitoring, Evaluation, and Reporting MSC Most Significant Change OVC Orphans and Vulnerable Children PCG Primary Caregiver PEPFAR President’s Emergency Plan for AIDS Relief RDP Reconstruction and Development Program RiH Results in Health, The Netherlands RST Regional Support Team SASSA South African Social Security Agency UGM Umbrella Grants Management Project Hands at Work UGM End of Project Evaluation v EXECUTIVE SUMMARY INTRODUCTION UGM is a five year grants management program administered by FHI 360 with funding from the U.S. Agency for International Development (USAID). Through UGM, FHI 360 provides funding and technical assistance to USAID-selected NGO partners who provide HIV/AIDS services at local, provincial, and national levels in South Africa. FHI 360-UGM seeks to promote high quality service delivery in alignment with the priorities and goals of the South African government’s HIV/AIDS framework. FHI 360-UGM provides specialized capacity building and support services to build partners’ skills and competencies in program management, governance, human resource development, budgeting and finance, and monitoring, evaluation, and reporting. The FHI 360-UGM project objectives are to 1. Provide grants to USAID/PEPFAR partners that ensure an adequate resource flow to foster scale-up of activities 2. Implement effective monitoring, evaluation, and reporting systems to assess and document activities 3. Provide ongoing capacity building to support and enhance scale-up of activities, and sustainability of activities and partners Since 2007, FHI 360-UGM has supported thirteen South African NGOs including Hands at Work. The organization has received total funding through the UGM of R12,335,933. The purpose of this evaluation was to assess whether the Hands at Work program achieved its objectives under the PEPFAR-funded program managed by the FHI 360-UGM. Results in Health and Health and Development Africa evaluated key program outcomes and impacts related to improved health and well￾being of orphans and vulnerable children (OVC), which was the focus of this grant. The evaluation sought to answer three main questions. 1. To what extent was Hands at Work able to meet the needs of OVC? 2. What were the most significant changes brought about by Hands at Work and its community-based organization (CBO) sub-partners in improving the well-being of OVC in targeted communities? 3. What were the key enablers and barriers in meeting project objectives? BACKGROUND ON HANDS AT WORK Hands at Work, Africa is a faith-based organization (FBO) established in 2002 to provide care and support to vulnerable children and families affected by HIV/ AIDS. Hands at Work is present in eight African countries and has a head office, or Hub, in White River, Mpumalanga, South Africa. Hands at Work South Africa supports CBOs in the Bushbuckridge and Clau Clau areas of Ehlanzeni, Mpumalanga; PEPFAR funding supported Hands at Work’s South Africa projects. The Hands at Work model is based on the premise that by building the capacity of local CBOs to care for and support vulnerable children, you can increase service provision for vulnerable children and their families. They believe that it is more sustainable to build and support local CBOs than to provide direct services to children. The objectives of Hands at Work under the PEPFAR grant were to Hands at Work UGM End of Project Evaluation vi 1. build the capacity of caregivers to care for orphans and vulnerable children and the sick by the end of December 2011 2. provide a holistic package of basic services to OVC through partner CBOs 3. build the capacity of local home-based care organizations (HBC) and CBOs in organizational functioning Hands at Work supports the initiation of CBOs by identifying local champions and leaders in the community and using them to develop a foundation for creating a registered legal entity. These CBOs then recruit care workers who can run and support the program. Care workers identify children and families in need of care through home visits and other community networks, and then address these needs through referrals, networks, and direct service provision. There are three steps to the Hands at Work model. 1. Identify vulnerable communities, find a leader or champion to drive the formation of a CBO, establish the CBO as legal entity, and continuously provide support to build the capacity of the CBO. 2. Increase the capacity of care workers to identify vulnerable children and families and to support access to services (health, education, psychosocial, and physical) through home visits and at the CBO care centers. 3. Build the capacity of primary caregivers (those who live with and care for children and who may be children themselves, in the case of child-headed households) to adequately care for vulnerable children. Hands at Work considers itself an enabling organization, as it builds the capacity of CBO staff and volunteers and supports them in their work. METHODS The evaluation was conducted between March and June 2012, with fieldwork occurring over two weeks in April and May. The data collection tools were developed using key evaluation questions identified by FHI 360 and Hands at Work. A qualitative data collection methodology was used. Quantitative data were examined through a desk review of reports made available to the evaluation team; these were limited in number and content. The qualitative methods used were key informant interviews, focus group discussions (FGDs), the Most Significant Change method, and field observations. All focus groups and interviews were conducted onsite at the CBOs in and around White River. Of the nine CBOs that are currently being supported by Hands at Work, four were included in the sample. A fifth CBO that had “graduated” from Hands at Work and no longer received support from the organization was also included in this evaluation. FINDINGS Question 1: To What Extent was Hands at Work Able to Meet the Needs of its Target Population? Most respondents in either focus groups or interviews felt that the essential elements of the Hands at Work model were successful and that capacity building of CBOs resulted in service delivery to children. CBOs had undergone Hands at Work capacity building activities in the form of training and on the job support and mentorship. Care workers at CBOs ensured access to services for vulnerable children across the areas of food provision, education, psychosocial support, and health. Despite these successes, respondents also reported challenges with service provision including difficulties in physically accessing two of the care centers, as well as difficulties accessing legal documents and social Hands at Work UGM End of Project Evaluation vii grants. The inconsistent supply of food parcels from the Department of Social Development (DSD) was raised as a major problem for many beneficiaries. Due to a lack of detailed quantitative data, the evaluation is not able to report on the extent or scale of services provided to children, nor were the evaluators able to assess the quality of the services provided. Available data suggest that at a quantitative level, Hands at Work did not meet its target of reaching 10,000 children between 2007 and 2012. As of September 2011, Hands at Work reported that 6,172 children had been reached. Question 2: What Were the Most Significant Changes Brought About by Hands at Work and its CBO Sub-partners in Improving the Well-being of OVC in Targeted Communities? Different groups of beneficiaries had varying perceptions of the most significant change brought about by Hands at Work. Primary caregivers reported food provision as the most significant change. The provision of these food parcels was not supported through the PEPFAR grant, but by the South African DSD. However, access to these food parcels was facilitated by Hands at Work, and its prominence in the discussions with caregivers indicates the importance of meeting the basic food security needs of vulnerable children. The most significant change for youth and young mothers was a sense of identity and future after exposure to Youth Camps and the Young Mums program. Youth also reported becoming agents of change or role models for other vulnerable children. The most significant change for care workers as a result of Hands at Work capacity building was their improved knowledge and skills to deal with vulnerable children and to help meet their needs. This in turn resulted in recognition and appreciation by the community for their services to vulnerable children. Question 3: What Were the Key Enablers and Barriers in Meeting Project Objectives? The Hands at Work capacity building model embeds its work in local communities. Support by the local community was found to be both a key programmatic and contextual enabler for Hands at Work. The commitment of champions in the local communities and the commitment of care workers facilitated the organizational development brought about through the PEPFAR grant and international support. Contextual enablers included the South African government’s HBC program, which laid a solid foundation in the community for care work, and government stakeholders who ensure that children are able to access to services. The most significant programmatic barriers for Hands at Work included lack of human resources and capacity (both at Hands at Work and at CBO level). This was exacerbated by rapid turnover of staff and volunteers. Hands at Work staff felt that the differences in programmatic focus between Hands at Work and PEPFAR (capacity building of CBOs by Hands at Work versus actual service delivery to vulnerable children) hindered the program at times. Contextual barriers included low socioeconomic conditions in communities served by Hands at Work, which resulted in high levels of community expectations that Hands at Work was not always able to meet. CONCLUSION Overall, respondents felt that the Hands at Work program had been effective in developing increased capacity to care for vulnerable children in severely deprived communities. However, lack of verifiable data prevented the evaluators from assessing the scale and quality of the services that were provided to OVC. In addition, the evaluators have doubts as to the ongoing sustainability of the CBOs established and supported by Hands at Work, and they are likely to require ongoing support for some time. Hands at Work UGM End of Project Evaluation viii In assessing the Hands at Work Objective 1, the evaluation found that primary caregivers reported increased capacity to care for children, and that this was a result of the capacity building interventions from Hands at Work and the CBOs. In relation to Objective 2, the evaluation found that services were being provided to vulnerable children including access to education (fees, uniforms, afterschool care) and healthcare. The CBOs did provide support for the physical needs of OVC (food, clothes, home repairs, social grants). The provision of psychosocial support through Youth Camps and the Young Mum programs seemed to be a particular success, but the capacity to run these activities and the scale was limited. Supporting data to explore the extent of provision of services was either not available or incomplete. The lack of available reported data for each of the specific services provided to children over the entire life of the grant makes it is difficult to assess whether the project met Objective 2 in terms of the numbers of children being reached. The evaluation found that Objective 3 had been partly achieved. CBOs reported varying levels of organizational functioning. Their ability to fundraise, access government services, and engage and network with other stakeholders varied greatly; none seemed likely to be able to function without considerable support. The evaluation found some tension between the Hands at Work capacity building model and PEPFAR’s intense focus on service provision to vulnerable children. While building community capacity to support OVC may be a longer term solution in these communities in need, it does not lend itself to the type of service-driven reporting often required by PEPFAR, causing some frustration from all sides. RECOMMENDATIONS For Hands at Work Hands at Work clearly provides some niche areas, particularly psychosocial support, early childhood development (ECD), and building the capacity of CBOs. They need to strengthen these areas while forming partnerships to ensure that they can facilitate improved access to critical services for vulnerable children. Some specific suggestions are listed below.  Improve networking with government. Given the type of model used by Hands at Work, it is essential that CBOs have good networks in their communities. This could include linking care workers to government-driven initiatives like the child and youth care worker program (through the DSD), care and support activities at schools (through the Department of Basic Education), and the community health outreach program (through the Department of Health). The Department of Agriculture (DoA) could play an important role in the push for food security through the establishment of food gardens at care centers and in the communities. The CBOs could initiate joint efforts with the Department of Home Affairs (DHA) through one day or one week events (jamborees) to access legal documents. The South African Social Security Agency (SASSA) could be invited to these events to facilitate access to grants.  Improve partnerships and networks with other local organizations, NGOs, and community structures. Hands at Work and the CBOs could conduct mapping exercises to assess which organizations are active in communities, and how these groups could improve access to services for vulnerable children and their families. Possible organizations in the areas could include Greater Nelspruit Rape Intervention Programme (GRIP), Childline, and Child Welfare.  Sustainability needs to be addressed more fully across all CBOs supported by Hands at Work; this is currently one of the major weaknesses of the model. Hands at Work could offer training and onsite support to ensure that sustainability plans are in place, and sustainability plans should be built into workplans from inception. Hands at Work UGM End of Project Evaluation ix  Continue to build and strengthen Youth Camps and the Young Mums programs. These programs are well regarded in the community and could be expanded. Hands at Work should explore ways in which these programs could be offered more at a local level in communities, and secure funding to ensure their continued existence. Both programs fill a critical psychosocial function for young people. The children of the Young Mums program could also benefit from improved support.  Continue to build and strengthen ECD skills of care workers. ECD was cited by many beneficiaries as an important service provided by Hands at Work, and should be another area where they continue to expand.  Continue to build capacity of CBO managers in leadership skills and financial management.  Increase focus on income generation as a strategy for Household Economic Strengthening (HES). Hands at Work reported that this area of activity was not strong in South Africa because of the availability of social grants. There should be renewed focus on this, and encouragement for households to look beyond social grants to strengthen their economic viability. Hands at Work reported successes with income generation in other African countries, and models from other countries should be adapted and replicated in South Africa. For USAID  The Hands at Work model illustrates the complexities of capacity building as a vehicle for service provision to vulnerable children. It would be useful for USAID to draw together organizations involved in similar activities in this area to document promising practices and lessons for building sustainable community organizations.  Further evaluation could be conducted using a cohort study to explore the outcomes for vulnerable children over an extended period, and assess which mechanisms have the most impact at ensuring these children are able to reach their potential. For the South African Government  Government departments need to increase their efforts to strengthen and build networks with organizations that offer services to communities. They should also increase collaboration among the government sectors to work better together to provide service packages, especially in communities where there are multiple needs.  There needs to be more debate and thought on the best way to build capacity to provide services in vulnerable communities. A continued focus exclusively on accredited training inhibits a more flexible approach to improving capacity in this area, and is out of the reach of many CBOs. Hands at Work UGM End of Project Evaluation 1 I. INTRODUCTION PURPOSE OF THE EVALUATION The purpose of this evaluation was to  determine whether Hands at Work achieved its program objectives in the period of the PEPFAR grant (2007-2012)  evaluate key program outcomes and impacts related to the improved health and well-being of target beneficiaries The evaluation measured the grant’s impact on Hands at Work South Africa, focusing specifically on beneficiaries at the community level, and looked at Hands at Work’s partner CBOs, who were supported with PEPFAR funding over the past five years. The evaluation attempted to identify what activities and interventions worked as intended, and document some of the key challenges. The evaluation focused on answering the following three key questions 1. To what extent was Hands at Work able to meet the needs of its target population?  To what extent was Hands at Work successful in enhancing their CBO sub-partners’ capacity to identify and successfully respond to the needs of OVC? o What kind of training was offered? What was the content? To who was training offered? What were the levels of attendance and participation, and benefits to participants? o Did training result in enhanced capacity to deliver services? o What other capacity building strategies were employed? Did they result in enhanced capacity to deliver services?  How do stakeholders (children, caregivers, community representatives) perceive the program in terms of quality and ease of access? o Did the CBO support vulnerable children with their physical needs, access to education, access to health, and access to psychosocial support? o What was the impact of the Youth Camps on vulnerable children (especially youth)? 2. What were the most significant changes brought about by Hands at Work and its CBO sub-partners in improving the well-being of OVC in targeted communities? 3. What were the key enablers and barriers in meeting project objectives? KEY AUDIENCES OF THIS REPORT Key audiences of this evaluation report include Hands at Work, USAID, FHI 360, South African Government Departments (including Health, Basic Education, Social Development, and Home Affairs), and South African NGOs and research organizations working on similar issues. KEY COMPONENTS OF THE REPORT  The Background section describes the legal and policy environment for South African OVC programs, the UGM project, and Hands at Work’s model and core focus areas. It also summarizes Hands at Work’s objectives and their alignment with PEPFAR indicators.  The Methods section presents the evaluation design, data collection methods, sampling frame, and the study limitations. Hands at Work UGM End of Project Evaluation 2  The Findings section is organized around the three guiding evaluation questions, and assesses the program’s achievements across activities and different beneficiary groups.  The Conclusion section assesses Hands at Work’s achievements against each of the project objectives.  Recommendations, based on the findings and conclusions, have been provided for each of the different audiences of this report.  The Appendices include information on the composition of the evaluation team, the sampling framework, data collection tools used, list and dates of sites visited, Scope of Work, and references. Hands at Work UGM End of Project Evaluation 3 II. BACKGROUND THE SOUTH AFRICAN OVC POLICY FRAMEWORK OVC programs in South Africa must be aligned with the government’s policy framework for the care of OVC, as set out in the DSD’s Policy Framework for Orphans and Other Children Made Vulnerable by HIV and AIDS South Africa, released in July 2005. This framework identifies six key strategies to guide the development of comprehensive, integrated, and quality programmatic responses for OVC. 1. Strengthen and support the capacity of families to protect and care. 2. Mobilize and strengthen community-based responses for the care, support, and protection of orphans and other children made vulnerable by HIV/ AIDS. 3. Ensure that legislation, policy, strategies, and programs are in place to protect the most vulnerable children. 4. Assure access for orphans and children made vulnerable by HIV /AIDS to essential services. 5. Raise awareness and advocate for the creation of a supportive environment for OVC. 6. Engage the civil society sector and business community in playing an active role to support the plight of orphans and children made vulnerable by HIV/AIDS. The sixth strategy is crosscutting, in that it supports the implementation of the other five strategies. Implementing these strategies is key to ensuring South Africa’s achievement of the Millennium Development Goals and the United Nations General Assembly Special Session Declaration of Commitment on HIV/ AIDS1. THE UMBRELLA GRANTS MANAGEMENT PROJECT UGM is a five year grants management program administered by FHI 360 with funding from the U.S. Agency for International Development (USAID). Through UGM, FHI 360 provides funding and technical assistance to USAID-selected NGO partners who provide HIV/AIDS services at local, provincial, and national levels in South Africa. FHI 360-UGM seeks to promote high quality service delivery in alignment with the priorities and goals of the South African government’s HIV/AIDS framework. FHI 360-UGM provides specialized capacity building and support services to build partners’ skills and competencies in program management, governance, human resource development, budgeting and finance, and monitoring, evaluation, and reporting. The FHI 360-UGM project objectives are to 1. Provide grants to USAID/PEPFAR partners that ensure an adequate resource flow to foster scale-up of activities 2. Implement effective monitoring, evaluation, and reporting systems to assess and document activities 3. Provide ongoing capacity building to support and enhance scale-up of activities, and sustainability of activities and partners Since 2007, FHI 360-UGM has supported thirteen South African NGOs including Hands at Work. The organization has received total funding through the UGM of R12,335,933.                                                              1 Grady et al., July 2008 Hands at Work UGM End of Project Evaluation 4 HANDS AT WORK Hands at Work, Africa is a FBO operating in eight countries in sub-Saharan Africa. Figure 1 below describes the organizational structure of Hands at Work Africa in the eight countries in which it operates, and the relationships with its partners across the globe. The overseas arms of Hands at Work serve to network international churches and donors with Hands at Work, Africa, which has a head office (or hub) in White River, Mpumalanga, South Africa. Hands at Work, Africa supports 80 CBOs in Nigeria, Zimbabwe, Mozambique, Swaziland, South Africa, Zambia, the Democratic Republic of the Congo, and Malawi. Operations in each country are managed by Regional Support Teams. In South Africa, Hands at Work is currently active in the Clau Clau and Bushbuckridge areas of Ehlanzeni, Mpumalanga, and has service centers in both areas. Volunteers from all over the world are networked into the Hands at Work structures across Africa. Figure 1: Organizational Structure of Hands at Work, Africa This evaluation focuses on Hands at Work South Africa (hereafter referred to as Hands at Work). Hands at Work was founded in the late 1980s when George and Carolyn Snyman began ministering to the poor and sick through an organization then known as Masoyi home-based care. Gradually, their focus shifted to the plight of children affected by HIV/AIDS, and expanded as more and more volunteers came forward to support their work. Hands at Work was formally registered as a not for profit organization in 2002. During the life of the PEPFAR grant, Hands at Work had sub-partners that included Southern Cross Mission in the Northern Cape and the Tsibogang Christian Action Group in the Northwest. However, for time and logistical reasons, they did not take part in this evaluation. The Vision and Mission of Hands at Work Hands at Work describes its vision and mission as “We envision the local Church in Africa effectively caring for the dying, orphans, and widows, and unified in this mission with the Church outside Africa. Our mission is to, through relationships with the local Church in Africa, challenge, encourage, develop, and support the ministry of servanthood among those in need in their community through the replication of the Hands at Work community intervention model. We believe the biblical mandate to care for the dying, widows, and orphans is not only for the Church in Africa, but also Hands at Work UGM End of Project Evaluation 5 elsewhere, and Hands at Work will be a prophetic voice to the Churches outside Africa, challenging them to fulfill their mandate.” As reflected in its vision and mission, Hands at Work has a unique community intervention model that is replicated in the countries in which it works. Its work is supported by partners who are members of the Church outside of Africa. The Hands at Work Model In South Africa, the Hands at Work program has a strong focus on children who are orphaned and vulnerable. The HIV epidemic has been a catalyst for most OVC policy formulation and improvements (such as the DSD policy framework) and is a significant factor in making communities, families, and children vulnerable. However, Hands at Work (in line with most other OVC programs) recognizes that many factors contribute to circumstances of vulnerability and does not only target those affected by HIV/AIDS. OVC programs focus on interventions at many levels, including policy, community, household, and individual, to ensure care and support of vulnerable children. Different programs work at the levels at which they are best suited. The Hands at Work model is community-based and community-driven, and intervenes at the household/family and community levels. The Hands at Work model operates on the premise that effective capacity building of local entities such as CBOs results in enhanced community-driven responses to care for and support vulnerable children. This, in turn, facilitates improved service provision to vulnerable children and their families. Hands at Work does not directly provide physical services to children, although it does provide psychosocial support. Hands at Work’s process for establishing relationships with community CBOs is detailed below. 1. Identify CBOs whose area of work is similar to that of Hands at Work 2. Hands at Work approaches these CBOs and requests that they work together 3. If they agree, Hands at Work offers them capacity building and development programs consisting of formal training, day-to-day mentorship, and exchange visits 4. These activities are intended to improve the functioning of the CBOs and translate into more effective and efficient service delivery to OVCs. Hands at Work has previously worked with 17 CBOs; however, 6of the CBOs no longer need assistance and have chosen to function independently. They now work with 11 CBOs, as shown in Figure 1 above. Hands at Work’s activities are aligned with strategies 1, 2, and 4 of the aforementioned DSD OVC framework. The context in which Hands at Work operates The Ehlanzeni Municipality in Mpumalanga is mainly rural and semi-urban with low or poor levels of infrastructure and high levels of unemployment. The unemployment rate in Ehlanzeni is 36.1% overall, although higher among women. The priority issues in the municipality in terms of basic services include a lack of the provision of clean portable water and poor access to sanitation and electricity. The priority issues for social development are high levels of HIV/AIDS, limited access to healthcare, and difficulties with accessing social grants. 73.7% of households in Bushbuck Ridge do not have access to clean water, while 45.4% of households do not have access to adequate sanitation. Economic development in the areas comes from tourism (from the Kruger National Park), agriculture, mining, and forestry. Ehlanzeni is moving towards the mature phase of the HIV/AIDS epidemic, which means that there are increasing numbers of AIDS deaths, AIDS orphans, and AIDS-related illnesses. As a result, palliative and home-based care, treatment, and prevention issues are a priority for the district. -Ehlanzeni Integrated Development Plan Hands at Work UGM End of Project Evaluation 6  Strengthen and support the capacity of families to protect and care (Straegy1)  Mobilize and strengthen community-based responses for the care, support, and protection of orphans and other children made vulnerable by HIV/AIDS (Strategy 2)  Assure access for orphans and children made vulnerable by HIV/AIDS to essential services (Strategy 4) Hands at Work identifies underserved communities in which many children are vulnerable and mobilizes faith-based mechanisms to care for and provide services to the most vulnerable children. Their mission is to “support the poorest of the poor” and to help children reach their development potential. There are three essential steps to the Hands at Work model. 1. Identify vulnerable communities and find a leader or champion (referred to as the “Man of Peace” or “Mother Teresa” by Hands at Work). These people help to drive the formation of a CBO and establish a legal entity; Hands at Work then provides support to build the capacity of CBOs. The CBO should establish a community care center where care workers and beneficiaries have a dedicated space to meet and provide services. 2. Improve the capacity of care workers at the CBO to identify vulnerable children and families and to facilitate access to services for these children (health, education, psychosocial, and physical) 3. Build the capacity of the primary caregivers (those who live with and care for children and who may be children themselves, in the case of child-headed households) to adequately care for children The three steps of the Hands at Work model are closely aligned to the program objectives. Figure 2 below highlights the key elements of the Hands at Work model in relation to capacity building activities. Figure 2. Hands at Work Model Capacity Building Activities CBOs’ care workers facilitate service provision for OVCs through the establishment of care centers, and by networking and supporting primary caregivers through community-based structures. The model’s underlying assumption is that a sustainable way to provide services to vulnerable children is through working with communities, in order to  build on what is working well (local champions and leaders who are already doing work in their communities) “Children in Africa are not on the ladder. They have no choices…Hands at Work…help[s] kids get on the ladder.” -Key informant interview Hands at Work UGM End of Project Evaluation 7  work with existing structures (such as churches and schools)  advocate and network for services for children A related assumption is that supporting and building primary caregivers’ (PCGs) capacity to care for children results in improved outcomes for OVC. Hands at Work Strategic Objectives under the UGM project The strategic objectives under the UGM project are aligned to the Hands at Work model, which seeks to build capacity of PCGs (Objective 1) and CBOs (Objective 3), and provide access to services for OVC (Objective 2).  UGM Objective 1: Build capacity of caregivers in caring for OVC and the sick by the end of December 2011  UGM Objective 2: Provide a holistic package of basic services to OVC through partner CBOs  UGM Objective 3: Build the capacity of local HBC/CBOs in organizational functioning To meet the three objectives mentioned above, Hands at Work has delivered various capacity building activities, as described below. Building capacity (through training and mentoring) of caregivers Foundation Training Program Training for Care helps to reorient care workers to focus on identifying vulnerable children and delivering services to OVC. Home visits are a key part of this training. Hands at Work provides ongoing mentorship (through service centers supported by people in the Regional Support Team) and exchange programs to build the knowledge and skills of care workers. Home-Based Palliative Care Training This training improves the knowledge and skills of care workers to enable them to provide home-based care for patients and their families across the continuum of illness. Walking with Wounded Children Training ‘Walking with Wounded Children’ is a 10 day course provided for care workers; the objective of this training is to equip care workers/volunteers to build a relationship with children by joining them in their play and storytelling (biblical, folk tales, true life, and therapeutic/healing stories). The care workers learn to apply various tools relevant to the various age groups with which they work and to manage behavioral issues among these children. The steps involved include 1. Connecting with children, a 10 day course based on a needs assessment 2. Three to six month practical training with a group of identified children; outcomes are set with the trainers and mentors during this time 3. Out of the trained group of care workers, a few are identified based on their competence and are trained/equipped to train others. This training is conducted over 18 days under the supervision of a mentor; care workers are then given an opportunity to plan and present a 5 day course to fellow care workers, again under the guidance of a trainer and a mentor. Finally, they commence training in their own areas and in teams, with support from Hands at Work staff. 4. Walking with Wounded Children uses same process as the first three steps, but incorporates new tools. Topics covered include a healing community, active listening, signs and symptoms of trauma and the causes of trauma, using participatory methods to work with children, sexual abuse, and sexuality education. This is followed by three to six month practical with one child using the knowledge and tools acquired in the course. Hands at Work UGM End of Project Evaluation 8 5. All the four steps are accompanied by ongoing mentorship provided by Hands at Work. Training for teachers A specific training on ECD has been developed to provide a comprehensive approach to children from birth to nine years of age, with the active participation of their parents and caregivers. Its purpose is to encourage children to develop their full cognitive, emotional, social and physical potential. Care centers Care centers are open five days a week; they offer ECD services for children under six and afterschool care for children in school. All children receive a meal at the care centers, and afterschool activities include homework support, recreational activities, and play. All CBOs have care workers who work at the care center and are responsible for implementing activities with the children, families, and community. Youth Camps The youth camps provide psychosocial support and life skills training to children ages 12 to 18. The camps are between three and five days long, and typically include children from two CBOs at a time. Youth camps are regularly scheduled during the end-of-year school holidays. They are not scheduled regularly throughout the year, but appear to be organized by Hands at Work when requested by CBOs. Young Mums Program This program supports young mothers to access services like education, food, clothes, legal documents, and social grants. It also offers psychosocial support to young women who are experiencing the double burden of their own vulnerability, compounded by their role as new and young mothers. The program’s goals are aligned to the PEPFAR indicators of providing education, social grants, and improved health status. Training and mentorship of primary caregivers (done by Hands at Work) PCGs are directly responsible for a children’s care, but most often are not the biological parent. Hands at Work delivers formal training on how to support children for PCGs, and provides them with ongoing mentorship. Care workers are expected to provide home visits to the homes of OVC and the PCG. PEPFAR Indicators through the Life of the Project The PEPFAR indicators reached by Hands at Work’s program include the number of OVC who received the following services  Clinical nutritional support  Child protection interventions  Psychological care  Effect of educational support on school attendance and school advancement (ECD, primary school, high school)  General healthcare referral  Healthcare support for access to antiretrovirals  HIV prevention education  Household economic strengthening Hands at Work was required to report on the services above, which were introduced in 2010 and marked a shift in focus from capacity building and palliative care to the delivery of services to children. Hands at Work UGM End of Project Evaluation 9 The evaluation was not able to access detailed reports for the five year period of the grant; however, the above indicators were used to guide the qualitative assessment of services provided to OVC. Service provision was divided into four main areas: education, health, psychosocial support, and physical support.2. The targets and achievements of Hands at Work are represented in the table below. The decline in the number of OVC receiving services in 2010 and 2011 was as a result in the reduction of the number of sub-partners receiving support from Hands at Work. Table 1. Hands at Work Targets and Total Number of OVC Served Targets and Total Number of OVC Served Targets Actual OVC Served Year 1: October 2007–September 2008 0 9,115 Year 2: October 2008–September 2009 9,000 8,966 Year 3: October 2009–September 2010 9,500 4,088 Year 4: October 2010–September 2011 8,000 4,035 Year 5: October 2011–September 2012 0 0 Geographic location The geographic locations of the Hands at Work program funded by PEPFAR included rural and semi￾urban areas in the Clau Clau and Bushbuckridge areas of Ehlanzeni, Mpumalanga, as well as in other provinces in South Africa. Table 2. Geographic Coverage Hands at Work Geographic Coverage Province District Site (Geographical areas) Mpumalanga Ehlanzeni Mbombela—sub-district Bushbuckridge Northwest Ngaka Modiri Molema Ratlou—sub-district Bojanala Mafikeng—sub-district Ramotshere—sub-district Ditsobotla—sub-district Dr Ruth Segomotsi Mompati District Naledi—sub-district Northern Cape J.T. Gaetsewe Ga-Segonyana–local municipality Gamagara—local municipality Moshaweng—local municipality                                                              2 “Physical support” in the evaluation questions referred to all services that respond to the physical needs of vulnerable children: food (outside of clinical nutritional support), clothes, home repairs, and access to grants. Hands at Work UGM End of Project Evaluation 10 III. METHODS EVALUATION DESIGN AND METHODOLOGY The evaluation team used a mixed methods systems approach to assess Hands at Work’s effectiveness in addressing the needs of vulnerable children. In developing the data collection tools (see Appendix III), the evaluation team sought to understand the context for program delivery, key factors influencing program implementation, and existing knowledge about the program. Qualitative data collection approaches were employed and included key informant interviews (KII), focus group discussions, and the Most Significant Change technique. Where possible, the qualitative findings were used to validate the quantitative statistics obtained from Hands at Work. All focus groups and interviews were conducted onsite at the CBOs in and around White River and Bushbuckridge, Mpumalanga. The sampling framework can be found in Appendix II. Data collection methods  Desktop Review: The team consulted and analyzed reports, a case study of Hands at Work, country operational plans (COP), and other documents related to the organization and OVC policy and programs  Semi-structured Interviews: KIIs were done individually or conducted in pairs; rarely, these were conducted in groups of up to four when it was necessary to interview a range of people together  Focus Group Discussions: FGDs were held with different respondent groups of four to twelve respondents  The Most Significant Change (MSC) Methodology: Six stories were collected from a range of beneficiaries and Hands at Work staff Most Significant Change The MSC technique is a means of “monitoring without indicators, ” a form of participatory monitoring and evaluation that measures change through the stories of who did what, when, and why, and highlighting reasons why changes happen. The process involves the collection of significant change stories from the field and the systematic selection of the most significant of these stories by groups of designated stakeholders or staff. The MSC technique is relevant for assessing the most significant changes brought about by Hands at Work and its CBO sub-partners in improving the well-being of OVC in targeted communities. Using the MSC method, respondents were asked three questions to identify changes in different domains. 1. What is the best (most significant) change in children’s (quality of) life/well-being since the Hands at Work program has been implemented? 2. What is the best (most significant) change in your community since the start of the Hands at Work program for vulnerable children? 3. What is the best (most significant) change in organizational capacity as a result of the Hands at Work program? Findings from the MSC method were cross-checked and matched (triangulated) with the findings from the KIIs and FGDs. In order to support the evaluation findings, quotes taken from the MSC stories are included in this report; however, due to limited time for data collection, the stories themselves are not included. Elements of the stories are used to support and confirm other findings. Hands at Work UGM End of Project Evaluation 11 Site Selection Five CBOs in two sites were selected: Senzokuhle and Joy in the Clau Clau area and Welverdiend, Belfast, and Ndzalama in the Bushbuckridge area. Sites were selected to include a CBO that had graduated (no longer receiving support from Hands at Work), CBOs that were performing well (but not yet ready to graduate), and CBOs that still required significant support from Hands at Work. Selection of Respondents  KIIs were conducted with all levels of staff from Hands at Work, including program leaders, coordinators, and managers from the Hub, Regional Service Team (RST), and Service Centers.  KIIs were conducted with CBO coordinators and managers.  FGDs were conducted with care workers, primary caregivers, youth, and young mothers.  KIIs were conducted with community members. Tables 3 and 4 below provide a summary of the respondent categories, numbers and sampling methods. Table 3. Summary of Sampling Method and Respondents Summary of Sampling Method and Respondents Data collection methods Respondents Number of respondents Male Female Key Informant Interview (including MSC) 31 9 22 Focus Group Discussion 140 11 129 Most Significant Change only 1 0 1 Total 172 20 152 Table 4. Summary of Respondents per grouping, data collection method, and number Summary of Respondents per grouping, data collection method, and number Type of respondents Data collection method and number Number of respondents Leadership 1 KII 2 Program Management 1 KII 1 Field coordinators and Service Center Staff 5 KIIs 13 CBO coordinators 5 KIIs 10 Care workers 5 FGDs 51 Young Mothers 2 FGDs 14 Primary Caregivers 5 FGDs 57 Youth 3 FGDs 20 Community members 3 KIIs 3 Not provided 1 MSC 1 Total 172 DATA ANALYSIS A preliminary content analysis was conducted by analyzing the basic themes of transcripts to uncover common themes emerging from the interviews and focus group discussions. These themes were then categorized, organized, and analyzed using Atlas.ti software. Triangulation was used to ensure the maximum use of data collected and to cross check results from various data collection methods and respondents (beneficiaries, staff, and volunteers) in answering the key research questions. Responses were further analyzed according to the project objective, and are presented and analyzed according to the key evaluation questions. Hands at Work UGM End of Project Evaluation 12 ETHICAL CONSIDERATIONS The following measures were taken to uphold ethical standards.  Informed consent was sought from all study participants to record interviews and focus groups, and to take photographs were applicable.  Children under 16 were not included in data collection, as there was insufficient time to obtain informed consent from their caregivers.  Facilitators/interviewers were trained to ensure they were able to respond appropriately to questions and concerns around the evaluation. LIMITATIONS OF THE EVALUATION  No baseline data. There is no baseline data from, or prior to, the start of the grant in 2007 against which figures can be compared. It is not possible to compare and assess whether there has been an increase in the numbers of children receiving services, of CBOs formed, or of care workers and primary caregivers trained. Similarly, there is no record of the number of youth camps held over the five year period. This means that the evaluation is unable to measure whether there have been changes in significant outcome indicators over the life of the grant.  Lack of quantitative information. There are gaps in reports and a lack of verified data, so it was not possible to make comparisons across periods of time. Where data are available, it is largely for the 2010–2011 reporting period and there are still gaps. The evaluation team did not receive any data on training. Although respondents mentioned training and capacity building activities and children accessing services, there was insufficient quantitative evidence to support or refute their perceptions. Where data are available, this has been used to substantiate some of the qualitative findings.  No children under 16 were included in the evaluation. The evaluation did not hold KIIs or FGDs with children under the age of 16, as obtaining consent would have taken too long. Information about children receiving services was provided by their primary caregivers, youth, and care workers.  The evaluation only explored the program as it exists currently. Over the past few years, the Hands at Work program included working with sub-partners in Northern Cape and Northwest. These partners were not part of this evaluation.  Limited data collection time. The Hands at Work program ended in March 2012, and was granted a one-month cost extension to support the evaluation process. At the end of April 2012, many of the key Hands at Work and CBO staff attended a conference in Zambia and did not return until the second week of May. People employed under the PEPFAR grant also left the project at the end of April. Both these events compromised the data collection time (third week of April to the second week in May). The evaluation team collected data from five CBOs and Hands at Work over a period of eight days.  Limitations to the MSC method. The story selection process was compromised by insufficient time for collecting and verifying stories. These issues were compounded by the fact that the MSC expert was not available during the story selection process; as a result, the evaluation is unable to draw on the MSC results as the method ideally should be used. However, the stories did provide useful insights and depth about the program.    Hands at Work UGM End of Project Evaluation 13 IV. FINDINGS TO WHAT EXTENT WAS HANDS AT WORK ABLE TO MEET THE NEEDS OF ITS TARGET POPULATION? Hands at Work’s target population includes communities, CBOs, care workers, and primary caregivers, all of whom deliver services to children; the program’s ultimate beneficiaries are OVC. The Hands at Work model reaches OVC through targeted interventions at the community level (through CBOs and care workers) and the household level (through primary caregivers). While they are not direct beneficiaries of services, primary caregivers are an important part of the target population as they are directly in contact with OVC and ensure that care, support, and services are provided to children at the household level. The assumption is that supporting and building the capacity of primary caregivers to care for children results in improved outcomes for OVC. Primary caregivers are also responsible for the household’s social and economic activities, and the intention is that services provided to OVC can have a positive impact on the capacity of the primary caregiver to manage and care for the household. This evaluation question is analyzed and presented in two parts.  To what extent did Hands at Work succeed in enhancing their CBO sub-partners’ capacity to identify and successfully respond to the needs of OVC? (Objectives 1 and 3)  To what extent did Hands at Work provide services to beneficiaries (children and youth) through CBOs? (Objective 2) To what extent did Hands at Work succeed in enhancing their CBO sub-partners’ capacity to identify and successfully respond to the needs of OVC? Capacity Building Activities for CBOs and Care Workers Hands at Work builds capacity within communities by supporting the establishment of CBOs and providing ongoing support to enable CBOs to respond to the needs of vulnerable children. The table below provides an overview of the two main levels at which capacity development activities are provided, and the range of activities at these levels. Table 4: Hands at Work Capacity Building Activities for CBOs and Care Workers Capacity Building Activities for CBOs and Care Workers Target Group Capacity Building Activities Provided CBOs  Consult with community to identify children’s champions (“Mother Teresa/Man of Peace”)  Support the formation of a CBO with champions and relevant stakeholders  Set up a Board of Directors  Train managers and coordinators (monitoring and evaluation, report writing, data collection, fundraising, proposal development)  Recruit and train care workers  Provide mentorship in networking with community resources and other stakeholders  Set up exchange visits with other CBOs to encourage sharing of lessons learned Care Workers  Train care workers to conduct home visits, identify and refer children, and provide services to children (Hands at Work foundational training, HIV/AIDS, palliative care, Walking with Wounded Children)  Provide ongoing support and mentorship—“walk with care workers” Source: Key informant interviews and focus group discussions Hands at Work UGM End of Project Evaluation 14 Reported Outcomes of Capacity Building Interventions—Perceptions of the Extent to Which Hands at Work Enhanced CBO and Care Workers’ Capacity to Identify and Respond to OVC Registration and establishment of CBOs The aim of establishing and registering CBOs is to legalize their status so they are eligible to receive funding from government and other donors. All five CBOs in the evaluation sampled are registered as non-profit organizations with the DSD as a result of the Hands at Work intervention. All CBOs have functional and hands-on Boards of Directors who have been trained in organizational governance. CBO leaders and field coordinators reported receiving training in administration; report writing; conducting monitoring, evaluation, and reporting (MER), data collection; and fundraising. Hands at Work staff also offered CBOs onsite support in management, administration, data collection, and reporting. All CBOs in the sample reported that they had received training and onsite capacity building interventions by Hands at Work. Hands at Work reported that it takes about three years of working with CBOs before they see a change, such as becoming more confident about paperwork and workplans, and having a better understanding of finance, donor relationships, and partnerships. A presentation to FHI 360-UGM in December 2011 reported that Hands at Work had reached its target of building the capacity of 52 organizations between 2007 and 2012; 18 were reached through Hands at Work, 33 through the Southern Cross Mission, and 1 through the Tsibogang Christian Action Group. Establishment of Care Centers Hands at Work assisted CBOs to set up care centers where community-based services are accessible to children. These centers provide three essential services: basic health, food security, and education. At the care centers, care workers provide ECD, afterschool care, homework support, meals, and facilitate access to services. All CBOs in the sample had care centers providing meals, ECD, and afterschool care. Measuring the capacity of CBOs to determine their capacity levels Hands at Work monitors the capacity of each CBO as part of their capacity building approach. The CBOs’ capacity is measured by ranking them, from 1 (the lowest score) to 5 (the graduate level). Hands at Work assesses how well CBOs work with the community, perform their roles and responsibilities, and the quality of their relationships with stake holders. This ranking is also based on how well the Board functions. Based on these considerations, only one CBO (Joy) was ranked at ‘graduate’ level; it was able to secure additional funding from the European Union and as a result Hands at Work decided to no longer provide it with funding. However, the decision to graduate Joy did not seem to be based on the standard process of measuring and assessing improved capacity of CBOs, which calls into question the value of this monitoring process. Care workers trained to identify OVC and deliver services to vulnerable children As a result of the focus on accredited training, Hands at Work and the CBOs had to select care workers for training based on their literacy levels and proficiency in English; as a result, not all of the care workers who required training received it. Attendance at training was also limited due to a lack of funding. There was a greater flexibility when Hands at Work was able to provide non-accredited training, which provides an opportunity to train greater numbers of workers and provide them key skills for less money than the accredited training. “Hands at Work is helping us a lot because long ago we didn’t have a place where we could meet with all the children and we used to go to the clinic. Hands at Work managed to build a room where we can manage to meet with them almost every day. In this building, the kids are receiving the food every day.” -FGD, Care workers Hands at Work UGM End of Project Evaluation 15 Care workers are appointed by CBOs on a voluntary basis, or receive stipends in order to support the work of the CBO. As with many similar organizations across South Africa, CBOs find it difficult to retain good care workers, who are quick to leave their position if a salaried job becomes available. Home visits are a central part of the Hands at Work model. They allow care workers to visit families and identify service needs, as well as provide counseling to children and primary caregivers. It is also an opportunity to refer children and families to suitable services. Care workers are required to collect information during home visits and report on services offered to the children. Hands at Work staff are also required to “walk with care workers,” meaning that they accompany care workers on home visits to assist and offer on the job mentorship. Hands at Work’s capacity building activities improved care workers’ ability to support and care for children. Care workers felt that they were better able to understand the needs of OVC and establish better relationships with them. The training also helped care workers build relationships between children and their primary caregivers. These findings were reported across all beneficiary groups, including youth and primary caregivers. Primary caregivers reported feeling supported by care workers, who often help bridge some of the emotional difficulties caregivers experience, such as integrating a child into a new home after a parent has died. In one focus group, a grandmother told the story of a child in her care who always cried about her late mother and did not want to help with housework or do her own laundry. She described how things improved after the care worker spoke to the child and visited her regularly. A primary caregiver discussed the importance of the care workers during a FGD, saying “They personally speak to the children. Sometimes as parents, we give up.” The primary caregivers interviewed appreciated that care workers are able to talk to the children and help with roles, discipline, and behavioral difficulties. These issues are often complex and challenging for primary caregivers to deal with, since they have to take over parental responsibility under difficult and emotional circumstances. Reported Challenges of Capacity Building for CBOs and Care Workers The low level of managerial capacity of field coordinators (CBOs’ key management staff) and Boards is a major challenge to CBOs’ effective functioning. The coordinators are expected to coordinate the projects, visit different CBOs, provide assistance and guidance to CBOs and care workers, and assist in the identification of vulnerable children. The Boards are supposed to oversee the management and overall program of CBOs; however, few of the Board members actually have skills in program management, administration, reporting, accounting, and monitoring. PEPFAR provided program management training as Hands at Work did not have capacity to deliver this training, while Hands at Work coordinated the logistical arrangements. Training programs alone cannot develop the level of skills needed for effective organizational functioning—these capabilities are built through ongoing support and mentoring by Hands at Work staff, and this investment is costly and time consuming. One key informant interview respondent noted the challenges in financial management and “Before I didn’t have any friendship with the children, now they call me sisi [sister].” “Before I was afraid to go to a house where there was an orphan, now I freely go.” “We get the knowledge to understand the situation of the child if they have got a problem. Because we find that when a kid comes with a problem we used to ignore them or chase them away, but at this moment, after the workshop, we take consideration of each child if they come with a problem.” -Care workers in FGDs “They also come straight to our homes and call all our kids and sit down with them and tell them what to do and what not to do. To clean the house, to go to the school and in the mornings to take some rubbish to the rubbish bin.” - Primary Caregiver FGD Hands at Work UGM End of Project Evaluation 16 monitoring, saying “They [CBOs] needed to show the receipts for the money they received and they needed a lot of paper work. You find a lot of CBOs staff that are willing to do things but they don’t know what and how to do it.” Furthermore, the fact that much care work is volunteer-based often results in a high turnover of care workers, who leave the CBOs to take up paid employment. Skills are built and people leave, which in turn leaves a gap in the organizations. This lost investment in human resources leads to a requirement for more recruitment and training; it also leads to an increased workload for the remaining care workers. Another reported challenge was in managing community expectations around the role of Hands at Work and the level at which it is able to provide services directly. The needs of vulnerable children and families in the communities are overwhelming, and there is an ever-present pressure to deliver services. The demand placed on care workers to meet community expectations and needs means that they are often not able to practice what they learn in training or to improve their skills. In handling complicated cases, such as sexual assaults or domestic violence, some care workers have referred people to GRIP; however, this relationship was not universal among the CBOs. To assist OVC to receive identity documents, children need to be referred to the relevant government institutions. However, only a few CBOs have managed to build successful networking and partnerships with other organizations/institutions including with the private sector for awareness campaigns, donations of food and clothing, and other support. Data from the organization was unavailable to quantify the extent of training that was provided, and it is not possible to comment on the extent of the coverage that was achieved under this program. Although coordinators and caregivers reported positively on training and capacity building activities, it did not happen as regularly as they required. One criticism that was raised by CBOs is that Hands at Work did not spend enough time conducting a needs assessment to determine what training was required, by whom, and to what extent. There was also a need to take a more holistic view of organizational capacity building, since the training of individuals can be ineffective if they go back and work in an organization that is not supportive towards them. An area of concern already raised previously is the lack of systematic networking among the CBOs. There are several important institutions which address OVC’s needs, such as churches, schools, clinics, community work program, and other NGOs operating in the province (including the South African National Council on Alcoholism and Drug Dependence, GRIP, and Childline). However, the findings indicate limited evidence of sufficient networking and partnerships between CBOs and other organizations in the areas in which they operate. A key informant noted the importance of building partnerships with other organizations, commenting that they “…would have loved to see more linkages with other NGOs in the area because there is quite a lot going on. We don’t talk to Childline and they are working with the most vulnerable. We don’t speak to GRIP, who are just down the road and is also a FHI 360 partner.” “The challenge is because we are not working [referring to care worker as not a formal job] when somebody find a job we have to replace with another, who does not know what to do and what is going on. We have to constantly replace with people who have no skills. We need more training. We need training regularly.” -KII with CBO leadership “The CBOs have such an amount of work, and have people every day saying, for example that this child has nothing to eat. So you take them for a day for training and when they arrive back to the community they forget what they learnt and don’t have time.” -KII with Hands at Work Hands at Work UGM End of Project Evaluation 17 In summary, Hands at Work has had mixed results with efforts to build the capacity of CBOs. While the fact that all the CBOs are registered and have functioning boards is a great success, it is quite clear that they are certainly not self-sufficient, which is not surprising in this environment. In addition, these organizations still require support to strengthen their management structures and systems. The focus on building the capacity of care workers is appropriate, and seems to have been largely successful. However, as mentioned previously, the evaluators were unable to determine the scale of the training provided to the care workers. The focus on providing accredited training inhibits the possibilities of capacity building as an overall approach, but this is an overall comment, as many organizations in South Africa are facing similar constraints. It would certainly be beneficial for Hands at Work to have a more formalized needs assessment process when engaging with CBOs, and to monitor and evaluate their organizational and staff capacity building programs. The establishment of care centers is a great success, and is a model worthy of replication. To What Extent Did Hands at Work Provide Services to Beneficiaries (Children and Youth)? This question looked at the services delivered to beneficiaries, including children and youth. Although primary caregivers are not direct beneficiaries of Hands at Work (they are not targeted for service provision), this section also looks at the impact of building the capacity of primary caregivers, as this is an activity in which Hands at Work and CBOs invest. As primary caregivers are responsible for the well￾being of children and families, it is important to explore the impact of activities aimed at building their capacity to cope with the challenges of raising children and of meeting their physical, social, emotional, educational, and health needs. Service Provision Activities for Children, Youth, and Primary Caregivers Table 5 below summarizes the services available at each of the CBOs in the sample. The data sources for this summary are the FGDs and KIIs with CBO coordinators and managers, care workers, primary caregivers, and youth. Hands at Work UGM End of Project Evaluation 18 Table 5: General Summary of service provision activities at CBOs Reported Outcomes of Services Provided – Perceptions of Extent to Which Hands at Work Provided Services to Children, Youth and Primary Caregivers Perceptions of Services Provided to Children Education All CBOs were supported children’s education through services including advocacy at schools to reduce/eliminate fees and the provision of school uniforms, shoes, stationery, and bags. CBOs assist children with their homework through their afterschool programs, and assist children who require protection. All care centers in the sample undertake ECD activities for young children, and all care centers have care workers trained in ECD. Older children also attend the care centers after school, where they are able to access support and supervision for homework. Summary of Service Provision Activities at CBOs CBO Service Provision Activities (as reported by all respondent groups) Education Psychosocial Support Physical Health Ndzalama  Provide school supplies: uniforms, school bags, stationary, toiletries  Provide after care and support with homework  Conduct home visits by care workers  Facilitate support groups with children and PCGs  Run youth camps  Offer PCG training  Provide meals to children at center  Provide money for transport to access documents and grants  Refer to Social Workers  Support with securing housing and furniture  Refer sick children to clinic  Accompany children to clinic  Visits by Clinic Sister to the CBO to provide information Pfunani/ Welverdiend  Provide school supplies: uniforms, school bags  Conduct home visits by care workers  Facilitate support groups with children and PCGs  Run youth camps  Offer PCG training  Provide meals to children at center  Provide money for application fees for social grants and documentation  Refer sick children to clinic Senzokuhle  Provide school supplies: uniforms  Initiate contact and follow up with schools  Conduct home visits by care workers  Run youth camps  Offer PCG training  Provide meals to children at center  Provide transport to DHA for documents for social grants  Refer sick children to clinic  Accompany children to clinic Belfast  Provide school supplies: uniforms, school bags  Provide after care and support with homework  Conduct home visits by care workers  Run youth camps  Offer PCG training  Provide meals to children at center  Provide money for transport to access documents and grants  Provide toys and clothes  Refer sick children to clinic  Accompany children to clinic Joy (graduated CBO)  Provide school supplies: uniforms  Provide after care and support with homework  Initiate contact with schools to follow up children  Conduct home visits by care workers  Facilitate support groups with children and PCGs  Run youth camps  Offer PCG training  Provide meals to children at center  Provide food parcels  Refer to DHA for documents for social grants  Refer sick children to clinic  Accompany children to clinic Hands at Work UGM End of Project Evaluation 19 The Hands at Work 2010 Quarter 4 report indicated that the majority of children were progressing to the next grade at school. This was the only evidence available in relation to school progress, as Hands at Work has not documented the progress of OVC’s performance at school as part of the MER system. Most of the education findings related to improved access, and very little was reported in terms of retention and progression. Health The majority of respondents from the primary caregiver and youth focus groups reported access to healthcare from health clinics, if and when needed. Primary caregivers and youth reported that care workers assist children and other members of their families by visiting them at home and assisting with medical reminders, transport to health facilities, and with referrals to other health facilities (such as, clinic to hospital) if necessary. Care workers also go to the clinics with the children and clinics often prioritize children accompanied by care workers, thereby avoiding long queues. Focus group participants did not report any direct services in HIV testing, access to ART, or direct HIV/AIDS-related services (as per PEPFAR indicators). However, the 2010 and 2011 quarterly reports indicated that children did receive services through referrals in these areas. The reason these services were not mentioned in focus group discussions could be because beneficiaries were not comfortable discussing this information in groups. Care workers did report stigma around HIV in the communities, and noted that some patients are reluctant to go to clinics. Care workers explained that rather than force patients to attend clinics, they try to build up a relationship in order to accompany them to a health facility. Hands at Work reported assisting malnourished children and running both Vitamin A campaigns and de￾worming drives. Regular de-worming of children was done by qualified professionals (nurses) from local clinics mobilized by Hands at Work during special days. This was highlighted by some respondents as an important intervention for vulnerable children, as worms can deplete the body of essential nutrients and increase the risk of malnutrition. Physical Physical support refers to services that meet the physical needs of children such as clothing, food, and shelter. Youth and primary caregivers reported receiving support with household maintenance and repair. A report from the first quarter of 2011 indicates that 12 families received RDP (Reconstruction and Development Program) homes as a result of CBO intervention and support. Primary caregivers, youth, and care workers reported food assistance to be one of the most important services received by children, especially the daily meal provided to children afterschool at the care centers. One primary caregiver observed that “…when the kids arrive from the HBC [care center] they keep on smiling and they say that they have enough food. Because of this they are always happy.” Primary caregivers reported that DSD provided food parcels, but concerns were raised that their delivery was erratic from month to month. DSD food parcels include 12.5 kilograms (kg) of maize meal, 5 kg sugar, 10 kg rice, fish, beef, 2 liters of fish oil, eggs, 6 cakes of bathing soap, polish, and washing powder. At the care centers, Hands at Work’s CBOs provide one meal per day for OVCs. Respondents across all focus groups mentioned food frequently, in particular primary caregivers, who expressed that food support was a critical element in caring for children. It was not always clear whether “I wake up at the morning then I go to school and then I come here for food and we do the homework here and then we play games.” -FGD with Youth “Community used to say we will never get an education and we have shown them that we have. Most of the girls have passed Grade 12.” -FGD with Young Mothers “They also write letters to the clinic or the hospital to say what the problem is so that we can receive assistance.” -FGD with Primary Caregivers Hands at Work UGM End of Project Evaluation 20 primary caregivers were referring directly to the food received by children at care centers or through DSD. Physical support also includes supporting children and primary caregivers to access the legal documents needed to apply for social grants. While many respondents reported receiving this service, access to these documents was reported to be an ongoing challenge for vulnerable children. Psychosocial support Respondents reported receiving various forms of psychosocial support, which took place at Youth Camps, during home visits, and at ECD centers. A common finding among youth respondents was that it is important for their well-being that there is someone (a care worker) who knows them by name, has an ongoing relationship with them and “keeps an eye on them.” During focus groups, youth commented on the importance of the care workers’ support.  “They teach us who you are and the value of the person who you are. Now we know how important we are.”  “To me they are my friends, everything I want to tell them I am free speaking. I am free with them. Anything I want to tell them I tell them. They tell me when I am wrong. They are not rubbishing me. They call me and tell me to sit down and they are talking to me. They are respecting me and I am respecting them too.” Perceptions of Services Provided to Youth Two Hands at Work programs for youth, the Youth Camps and the Young Mums program, play an important role in ensuring the psychosocial well-being of OVC. These are the only services that Hands at Work offers directly to beneficiaries. Young Mums is not a stand-alone psychosocial intervention, but helps young mothers and their children to access services. Youth Camps Youth camps were frequently mentioned across all respondent groups. The camps appear to fundamentally affect the lives and experiences of youth and other children. Camps were discussed in all focus group discussions with primary caregivers, and participants mentioned that children in their care had been to the camps. There were interactive and enthusiastic discussions about the camps in the three youth focus groups, indicating that young people appreciated and enjoyed the camps. Hands at Work runs youth camps at the White River hub during the December holidays, and on request at other times. There was no available data to indicate the number or frequency of these camps during the life of the program; however, data for the period of July to December 2011 indicate that which 798 youth ages 10–18 attended youth camps and activities. The role of youth camps is to create the opportunity and space for children to learn life skills and increase their knowledge about HIV/ AIDS. Youth are also exposed to biblical teaching. They are introduced to the network of support around them and learn how to access it. Respondents noted the many things children learned at the camp; highlights from the focus group discussions are listed below.  “Hands at Work teaches the kids. When they arrive [back home from Youth Camp] they tell their grannies what they learned. They take the children to see places. They didn’t have that opportunity before, last year they went to a game reserve. They also went to White River.” (Primary caregiver)  “The kids are taught to clean the houses and make the house to be nicer.” (Primary Caregiver)  “We learn a lot. They help and teach on our behavior. To live better life. Not to do anything that is not for our age.”(Youth)  “There was a Bible club—taught us not to forget God wherever we go. God is next to me everywhere I go. He knows my future and he is next to me.” (Youth) Hands at Work UGM End of Project Evaluation 21  “How to reach our goals and dreams and not to give up, focus on school, pass Matric, and get bursaries or further studies in order to reach goals.”(Youth)  “They receive information about HIV/AIDS. They teach others about it. Knowing about these diseases they are able to take care of themselves.” (Care worker) Care workers reported seeing a difference in children’s attitudes after they had been on camps. A few care workers have attended the camps and found that it helped build their relationships with the children. One care worker gave an example of one boy who wasn’t going to school; “after coming from the camp he did not only start the school himself and enjoys it, but also tells other kids to go to school.” Many of the focus group responses related to children feeling cared for, supported, and visible through these camps; thus the camps serve as a critical space for psychosocial support to young people. Highlights from the focus group discussions with youth about the psychosocial impact of camps are listed below.  “The youth camp is the place where we found a solution or breakthrough. We have many challenges in life; we don’t have support at home because we lost our families. Without Hands at Work, I don’t think I was going to make it, but now I have a future.”  “They tell us that if we live with an uncle or a guardian that doesn’t treat you good you come straight over here and they will help you.”  “They teach us who you are and the value of the person who you are. Now we know how important we are.” The camps also support leadership development and increase the capacity of young people to care for others in their community. One care worker explained, “The camps help them to choose their career, how to accept what they are, in order to get a better education.” Children are also able to share what they have learned with the adults in their lives. Young Mums Young Mums is a program for teenage and young mothers, developed by a member of the Hands at Work team who, as a young mother herself, recognized the vulnerability of young pregnant girls and mothers in the community. The program helps address the needs of young pregnant girls and mothers. During the MSC interview, the Hands at Work staff member discussed her motivation for starting this program: “When I was at school I happened to have a baby and I know what it means to have a baby as a 19 year old girl in grade 12, and not knowing what to do and you haven’t planned. Looking back at my experience, how to help these girls then became my focus…” Many of these young women are destined not to complete their schooling, so the program focuses on encouraging young mothers to continue education. In October 2010, the Young Mums program was selected as a best practice case study for the Early Intervention and Prevention Program in South Africa by UNICEF and DSD. The program educates girls about HIV/ AIDS and making safe and healthy choices. It also has a skills building component where the groups meet weekly to learn a craft like sewing, so they can support themselves. The two stories below from a young mothers’ focus group reflect the nature of the program.  “I was living with my grandmother and sister, and there were other girls who were treating me badly in the family. They were telling me to go and said I should go and get my mom from the grave. I then left the house and went to live with my boyfriend. I felt like nothing and had a baby. I was called to the [Young Mums] group and told them what my problems are. I was encouraged to go back to school. I then got sick, and the coordinator took me to the doctor and went with me to the doctor. They also bought clothes for the child as I had nothing at that stage.” Hands at Work UGM End of Project Evaluation 22  “I went to the clinic when I was pregnant and saw X3 sitting under the tree. She called me over and spoke to me. I took her phone number and called her to come and speak about what they could help with. She told me to go back to school and get an education. Young Mums went and spoke to the school and bought uniform for me. At school when there were parents meetings, X used to go on my behalf as I am an orphan. I had given up on life.” Unfortunately this program is not widespread across the sample sites, as it is currently only available at Masoyi HBC. The program has been affected by the end of the PEPFAR grant. Perceptions of Services Provided to Primary Caregivers Primary caregivers from all the sampled CBOs had undergone some basic Hands at Work training focused on building relationships with children, learning how to talk to children, and understanding the plight of children who have lost parents and siblings. Between July and December 2011, 80 primary caregivers received PCG training to sensitize them to the possible feelings and emotions of OVC in their care, and help them to deal with vulnerable children. The following quotes from primary caregivers FGDs exemplify their opinions on the services provided.  “They advise on how to improve the relationship with the kids and help them. For example, for a kid that likes to wear clothes as her mother used to wear, allow her to do it because that reminds her of her mother.”  “They give advice to give the orphans the same as for their other children or even give them first to make them feel well.”  “We learned that we have to protect the children and we know now that when they are sad [it] is because they are remembering their parents.”  “I want to talk about HAW [Hands at Work]—they help us on how to communicate with these children even from 0 years upwards. How to make that child yours, things like this. The communication between that child and the parent, now we know what to do, what not to do; to think before saying something to that child. Because we sometimes hurt that child without knowing what she feels, what she thinks, what she wants. Hands at Work helps a lot through the trainings, the trainings are very good, are helping a lot. It is painful that they are not going to continue.”  “When I took my sister home when my mother died…my sister used to look at my husband in a hard way and my husband is a hard man. She used to say that she hated him. Then when I learned something from Hands at Work, I went and taught my husband so he tried to do that. Now they are close, they are like daughter and father.” Quality Assessment of Services Provided In analyzing quality of the services provided by CBOs or Hands at Work based on the findings, a quality assessment is used as seen in the table below.                                                              3 Name removed to protect the identity of the person referred to. Hands at Work UGM End of Project Evaluation 23 Table 6. Quality Assessment of Services Provided Quality Assessment of Services Provided Quality of Services Education Psychosocial Support Physical Health Physical access4 Care Centers have basic service facilities (such as pit/long drop toilet, water) and three of five care centers are accessible (within walking distance). A referral network to local health clinics exists. Local health clinics are located close to most CBOs; some are within walking distance and free of charge. Availability5 Most OVC receive sufficient school materials, supplies, and uniforms to encourage their school retention; however, the distribution is erratic. An afterschool program to help OVCs with their homework is available; however, progress monitoring of performance at school is not done. Regular contacts with schools are not well established in most centers (only two out five centers have initiated contact with schools to follow-up on OVCs). Safe environment is available for OVC; however, materials (toys/books) are not available. Care workers provide home visits to OVC; almost all OVC visited regularly. Support groups for OVC and PCGs organized by care workers. Youth camps is run by Hands at Work for youth and covered all CBOs; Young Mums program is run by Hands at Work, but is only available in Masoyi HBC. Training for PCGs provided to cover all CBOs. Access to IDs and birth certificates is facilitated; access to DHA if still difficult particularly for migrant OVC. Care workers are always available to assist OVC five days a week. One meal per day is available in all care centers for OVC; some also provide money for transport to the centers. No information available regarding the availability of services at health clinics as interviews with health workers were not included in the evaluation plan. 4 Access refers to whether the service can be easily accessed by beneficiaries 5 Availability of service refers to whether the services are available once OVCs and PCGs reached the centers Hands at Work UGM End of Project Evaluation 24 Quality Assessment of Services Provided Quality of Services Education Psychosocial Support Physical Health Appropriateness6 Responsive to OVC educational needs; however, it is reported that the distribution of the school supplies is not regular. Training materials developed were responsive to PCG and OVC needs Responsive to OVC’s physical need especially food food (in addition to one meal each day, DSD provided food packages for OVC through PCGs, although the distribution is erratic) No information available regarding the appropriateness of services at health clinics as interviews with health workers were not included in the evaluation plan Sustainability7 School supply provision is dependent on availability of donors, which may not be sustainable Psychosocial activities are dependent on availability of donors, which is not sustainable. Advocacy efforts are in place to integrate psychosocial activities within school and community activities, but this is not yet established. Formal referral linkages between CBOs and other service providers (NGOs, schools, etc.) are not yet established. One meal per day and DSD food provision are dependent on availability of donors, which is not sustainable. The home garden activity is not strongly promoted and advocated among beneficiaries. Formal referral linkages between CBOs with health clinics are not yet established. Source: Findings analysis using ‘Standard Service Delivery Guidelines for OVC Care and Support Program’ (2010) 6 Appropriateness refers to the adaptation of services and overall care to need 7 Sustainability means the service is designed in a way that it could be maintained at the community level, in terms of direction and management as well as procuring resources, in the foreseeable future Hands at Work UGM End of Project Evaluation 25 Challenges to service provision Many challenges were reported in relation to service provision, both in terms of access and quality as mentioned in Table 6 above. Some of the specific challenges are detailed below. Location of Care Centers Two of the five CBOs are not located in communities and are not within walking distance of schools and homes. The distance to the centers, and the transport costs, are challenges for participants. Home Visits Some of the home visits by CBO staff were reported to be erratic, especially in the last project year. Home visits are seen as important because they give care workers the opportunity to address the PCG’s needs as well as children’s, and they are able to monitor vulnerable children and their homes. Primary caregivers at one of the CBOs reported that they have not seen the care workers at their homes since the beginning of 2012. One primary caregiver commented, “the HBC needs to visit us more often. They didn’t come to my house. I see them on the road always but didn’t come to teach me something that I don’t know; or to teach my child.” Access to Education A few challenges were reported regarding access to education. The provision of uniforms and clothing can be erratic and inconsistent (one caregiver indicated that the uniform has not been received for the current year), and children are sometimes provided with incorrectly sized uniforms. One caregiver explained, “I think they look at those children who are the poorest and start providing them with uniforms first. That means not all of us can receive.” There were also reports that one uniform was not enough for vulnerable children, as it meant they had to wear it every single day. As mentioned in the Table 6, there seems to be insufficient attention to school performance of vulnerable children, and there seemed to be no mechanism to monitor school performance. These challenges negatively impact the quality of educational and school related services provided to OVC. IDs and Birth Certificate The need for constant support in obtaining legal documents in order to apply for social grants (particularly for migrant OVCs) was often mentioned as a challenge. The issue seems to lie more with the South African government’s bureaucratic system, often making it difficult for illiterate beneficiaries to obtain legal documents on their own. Social Grants Although beneficiaries reported receiving help to access social grants, care workers felt that supporting primary caregivers to budget and use their money appropriately was challenging, and something that they were unable to help with. Regular Supply of Food from DSD Respondents frequently reported that the supply of food from DSD was not consistent. It was not possible to assess whether this related to the DSD food provision processes or whether it was due to CBOs’ capacity to access food from DSD for families; however, given the overall lack of a responsive bureaucracy in the area, it is likely that it is the former. Ongoing Support for Youth Over 18 A challenge for the program was that youth over the age of 18 are no longer eligible for support. This is a problem as many youth in this category are still at school, have no jobs, and are sometimes heads of households. One care worker expressed concern that after they stop receiving support, “they then finish using drugs or not doing nothing. The care workers consider that something might be done for children above 18.” Stigma in the Community Hands at Work UGM End of Project Evaluation 26 Stigma leads many people to turn down CBO and care worker support. Some families do not want people to know that they are struggling economically or dealing with HIV. Many patients do not want to be touched by care workers, as they are so used to being rejected by their families. It takes care workers a significant amount of time to build relationships with these families and to start offering support. Key Evaluation Question 1 Summary Hands at Work has an ambitious program of building community capacity to respond to the needs of OVC. They have had clear successes, such as formalizing their CBO partners, and in providing considerable training to CBO managers, care workers, and primary caregivers. They have established care centers, which have been an important safe space for OVC support. Some of their greatest success has been in providing psychosocial support to beneficiaries. However, Hands at Work not focused enough attention on responding to the basic needs of children, especially ensuring that they have enough food. The quality of the services provided is reasonably good in regards to physical access, availability, and appropriateness for each service. It has been difficult to assess the scale and coverage of their services, mainly as a result of fairly weak MER systems. This is an area they need to improve; to make the intervention sustainable, Hands at Work will need to demonstrate to funders that they can make a difference to children’s lives in the long term. WHAT WERE THE MOST SIGNIFICANT CHANGES BROUGHT ABOUT BY HANDS AT WORK AND ITS CBO SUB-PARTNERS IN IMPROVING THE WELL￾BEING OF OVC IN TARGETED COMMUNITIES? Respondents’ perceptions of the most valued services and the most significant changes varied greatly across the beneficiary groups. The most valuable services and most significant changes were identified and analyzed by triangulating the findings from various respondents and data collection methods, as summarized in Table 7 below. The Most Valuable Services Triangulation of findings revealed that meals provided at care centers8 was perceived strongly as the most valued service for primary caregivers, apart from access to identity documents and provision of ECD services. This finding was also supported by care workers during the focus group discussions and interview sessions. In interviews, focus group discussions, and MSC stories with youth and young mothers, camps were frequently mentioned as the most valuable service for youth, while for young mothers it was the benefits they obtained from training and support group meetings. Changes That Can Be Attributed to Hands at Work/CBO Interventions Primary caregivers experienced many significant changes in handling and treating children after attending the series of Hands at Work trainings, which led to improved quality of care and treatment of, and improved relationships with, the children. Caregivers noted specific changes they made including speaking nicely to children, providing children with food, and playing with children.                                                              8 Respondents also mentioned DSD food parcels as the most valued service but it is not part of the Hands at Work intervention “They took the children to a camp. When the children came back, we asked them how they have been helped and how their lives have changed. They usually say they feel light after they have come from the camp.” -KII with a community member “The youth camp is the place where we found a solution or breakthrough. We have many challenges in life; we don’t have support at home because we lost our families. Without Hands, I don’t think I was going to make it, but now I have a future.” -FGD with youth Hands at Work UGM End of Project Evaluation 27 The series of capacity building activities for care workers helped establish and improve relationships with children, as well as increasing their knowledge and skills in dealing with children’s challenges and finding solutions. Care workers learned how to treat children, and identify and work with them when they have problems. The youth camps and the Young Mums program have provided many benefits to youth and young mothers. Respondents reported that the camps improved their knowledge and life skills9; shaped their attitudes, confidence, and habits; and encouraged them to seek a better future. The Most Significant Changes Although there were challenges with the MSC methodology, the stories collected were analyzed to help identify the most significant changes. Primary caregivers reported an increase in food provision and food security for children as the most significant change; this was also perceived to be the most valuable service. This finding reinforces the idea that food is one of the most important basic needs for vulnerable children in vulnerable communities, and food provision significantly changes the conditions of vulnerable children. The most significant change for youth and young mothers was a sense of identity and optimism after exposure to the programs. Apart from personal changes, youth also reported becoming agents of change for others. They are seen as role models for other vulnerable children in the community. Young mothers felt that they had been empowered by the training they received. In the group discussion, they were able to share their life stories and experiences, as well as share ideas for income generating activities. The most significant change for care workers was their communities’ recognition of the services they provide to vulnerable children. The knowledge and skills obtained during Hands at Work training and mentoring gave care workers more confidence in dealing with the issues of vulnerable children in their community. It also guided them on how to optimize their help to vulnerable children and primary caregivers, and to fulfill the needs of vulnerable children. The commitment and work of the care workers has been highly appreciated by the community. Table 7. Thematic analysis of most valued services and most significant changes for beneficiary groups Findings across themes, respondents, and methods Themes Respondents Primary Caregivers Youth and Young Mothers Care workers Most valuable services  Food  Access to ID for purposes of grants and HES  ECD  Youth Camps  Care Centers for children after school  Food Changes that can be attributed to Hands at Work/CBO  Better knowledge, attitudes, and skills in treating and handling children  Children accept their circumstances and are more confident  Children are equipped with skills on how to deal with their problems  Better relationships with children  Improved knowledge and skills                                                              9 Defined as psychosocial abilities for adaptive and positive behavior that enable individuals to deal effectively with the demands and challenges of everyday life. These are loosely grouped into three broad categories of skills: cognitive skills for analyzing and using information, personal skills for developing personal agency and managing oneself, and interpersonal skills for communicating and interacting effectively with others. (UNICEF, 2003) “For me the most important change since I have joined [the] Hands program is that I am able to take responsibility on my own; it is a big achievement in my life.” -MSC Interview “Young Mums taught me to stand up to my problems. I learned this during training.” -FGD with Young Mothers Hands at Work UGM End of Project Evaluation 28 Findings across themes, respondents, and methods Themes Respondents Primary Caregivers Youth and Young Mothers Care workers intervention  Ability to handle children’s problems and challenges  Better relationships with children  Children are more open to their guardians, and experience less stress and psychosomatic disorders (such as headache, stomachache)  Improved attitudes after Youth Camps: respect, healthier habits, better self-management  Better (higher) education level, particularly for young mothers in identifying and handling children’s problems Most Significant Changes  Availability of food for children through provision of food (DSD parcels and one meal a day), as compared to before receiving food from the CBO or DSD  A sense of identity for children/youth  A sense of future  Feeling useful and proud of themselves, able to help OVC  Children receive holistic support  Proud to be recognized and respected by the community Key Evaluation Question 2 Summary Respondents have different perceptions of the most significant change that Hands at Work has brought about in the lives of vulnerable children. Primary caregivers reported food provision as the most significant change, while youth and young mothers felt it was a sense of identity and optimism after exposure to youth camps and the young mums program. For care workers, community recognition of their services to vulnerable children was the most significant change as a result of their work with CBOs. WHAT WERE THE KEY ENABLERS AND BARRIERS IN MEETING PROJECT OBJECTIVES? The evaluation addressed enablers and barriers at two levels: program level (internal factors) and contextual level (external factors). Programmatic Enablers and Barriers Programmatic Enablers The Local Community The Hands at Work capacity building model embeds the work in local communities. As an organization, Hands at Work provides very few direct services and is entirely dependent on building CBOs’ capacity to support vulnerable children and families to access the services they need. This model ensures that communities own and sustain the process, and all levels of the community have a role to play. In addition to the CBO coordinators and care workers, other members of the local community provide support on an ad hoc basis. These members are usually from churches, schools, private businesses, local farms, and HBC organizations. The Hands at Work model ensures that the skills and knowledge it helps to build up (among CBOs, leaders and coordinators, care workers, and primary caregivers) remain in the community and creates social capital and informal networks that will continue to exist should Hands at Work or the CBOs cease to operate. Hands at Work UGM End of Project Evaluation 29 Historically in South Africa, and in Africa more generally, communities have always cared for vulnerable children; the strength of the Hands at Work model is that it builds on this community capacity. However, the approach also has weaknesses, especially in relation to the delivery of services, which is explored later under programmatic barriers. Care worker Commitment The commitment of care workers is an enabler of the program. Many care workers continue to work in spite of the limited availability and uncertainty of stipends or other forms of remuneration. Their work is emotionally and often physically challenging, sometimes requiring travel over vast distances. Although care worker turnover presents a challenge as described later, they remain the backbone of the program to deliver services to vulnerable children. Not only is this true for Hands at Work and the CBOs they support, but there is also a strong body of literature to indicate that this is true of other CBOs. The story of the building of the Senzokuhle CBO provides a clear example of the importance of strong, committed, and dedicated care workers and community leadership. A group of dedicated women (care workers) bought the building and land where Senzokuhle is currently located, and over a few years, the women had saved the money they received for lunches and for transport to attend meetings in order to purchase the land. Organizational Development Hands at Work reported that the PEPFAR grant has contributed to its strong organizational model and structure, and its sound financial structure. This is not only true for Hands at Work South Africa, but for all of Hands at Work—the entire organization has benefited from organizational development as a result of the grant. At the start of the grant, Hands at Work did not have this organizational structure in place, which proved a barrier to managing the PEPFAR grant. The organization has developed administrative, reporting, and financial mechanisms which have contributed to a solid organizational foundation. Hands at Work staff have been trained in MER, administration, HR, finance, and reporting, which has enabled them to train, support, and mentor the CBOs they work with to improve their organizational capacity. It also means that Hands at Work has increased capacity to work with donors and to fundraise, and has been able to build capacity of Hands at Work offices in other countries. International Support Hands at Work is largely supported by international churches, donors and volunteers. This support takes the form of funding or voluntary works. The international branches of Hands at Work serve to network churches in developed countries with Hands at Work across Africa, and volunteers from all over the world offer weeks or months of service across Hands at Work’s eight African sites. In South Africa, these volunteers provide organizational support at the Hub (administration, reports, data management, filing), work in communities in the CBO projects (building, repairs, or maintenance to homes and to CBO care centers), and help develop CBO capacity (on the job support and mentoring). Programmatic Barriers   Difference in Programmatic Focus between Hands at Work and PEPFAR The focus of this PEPFAR grant was to ensure that services were delivered to children, and grant recipients were required to demonstrate that these services had been provided. This required an intense focus on data collection, capturing, and reporting. While Hands at Work’s overall goal is to deliver “If all this has to shut down, there are still incredible people whose lives have been changed and they will continue in whatever way, to serve.” -KII with Hands at Work leadership Hands at Work UGM End of Project Evaluation 30 services to children, their approach is not one of direct service delivery. The model focuses on capacity building, and service delivery is viewed as a consequence of this capacity development. Access to services happens primarily at the CBOs (through care workers and primary caregivers), which presents a number of challenges in capturing data, in particular the problem of attribution. However, based on the findings, Hands at Work will still need to strengthen their MER system, particularly to measure the outcome and impacts of building capacity of CBOs to deliver services to beneficiaries. Throughout the life of the grant, Hands at Work reported difficulties aligning with PEPFAR requirements, and managing the balance between building capacity, delivering services, and monitoring access to services. While these program components are not mutually exclusive and in principle represent a good service delivery model, the reality is that they require different levels of focus especially in relation to MER. Staff at Hands at Work felt that these differences in program emphasis were a hindrance to implementing their program. Human Resources Although capacity building is a core function of Hands at Work, capacity gaps are often too wide to enable a focused program of service delivery. This is compounded by high turnover of Hands at Work staff, care workers, and volunteers. One reason for this turnover lies in the socioeconomic context of Bushbuckridge, where high levels of unemployment and few job opportunities mean that people will take whatever job opportunities arise, and when better paying opportunities are available, they leave their volunteer positions for these jobs. Across South Africa, care work is plagued by high turnover, as care workers seek better and more regularly paid employment opportunities rather than only being volunteers. The lack of resolution around consistent government payment of care worker stipends, limited career opportunities, and a difficult and challenging work environment make this an unstable field of work. This staff turnover leaves both Hands at Work and the CBOs with limited capacity to provide services to children as well as to collect, monitor, and report program information. Furthermore, training must be continuous to realize a sufficient level of organizational competence in all areas. Insufficient Capacity and Skills for Monitoring, Evaluation, and Reporting The Hands at Work program, especially at the beginning of the grant, did not have the organizational and technical capacity to handle PEPFAR’s reporting requirements. Although capacity grew over the life of the project, it remains a challenge for Hands at Work and was particularly hard to develop among the CBOs, who still reported limited capacity for MER. This limited project management (planning, implementation, MER) capacity could be one of the reasons why the evaluation found it difficult to access quantitative data. During a KII, Hands at Work staff commented on the difficulty of complying with the strict funder stipulations based on target numbers, and to help the funder understand the organization’s objectives in order to be flexible with the program. These limitations had significant repercussions for the PEPFAR grant, which required regular reporting. The gap between targets and reported data over the period of the PEPFAR grant may be illustrative that Hands at Work and the CBOs were not able to adequately capture the services delivered to children and the lack of internal capacity to conduct regular process monitoring. Change of the Umbrella Grants Manager over the Life of the PEPFAR Grant Over the course of the PEPFAR grant, Hands at Word had three separate UGM managers. The prime UGM changed from Pact/Starfish to FHI and then to AED/FHI 360. This was unsettling for the Hands at Work program, as each organization had different priority and management approaches. It took time to build relationships with new UGM staff and to explain the Hands at Work model, core values and principles. Hands at Work staff commented during a KII that during the life of the PEPFAR program, “the Umbrella Grant Manager changed almost every year, where they had to recast vision and build relationships.” Focus on Accredited Training Hands at Work UGM End of Project Evaluation 31 Accredited training is at the center of the South African government’s capacity building agenda, and is supported by PEPFAR. Consequently, Hands at Work training programs offered during the PEPFAR grant had to begin an accreditation process, which had huge financial, time, and capacity implications. Hands at Work had limited capacity to get all their trainings accredited, although some have now been successfully been through this process. This focus on accredited training can mean missed opportunities to deliver unaccredited training to those who require it. In addition, capacity is built in numerous ways, such as on the job mentoring and walking with care workers, and not just through formal training. Thus, overinvestment in training is seen as a misdirected capacity building opportunity. Contextual Enablers and Barriers Contextual Enablers The contextual enablers include all the factors external to the organization that contributed to its successes. The Local Community The local community is a both a programmatic and contextual enabler. The community-based Hands at Work model depends on the support and commitment of the community to drive the program and enable or support it in its local context. Hands at Work staff and CBO leaders and managers also reported that local support from churches, traditional leaders, farms, and businesses played an enabling role. At some of the CBOs, coordinators and care workers actively seek this support. However, much of this support occurs more organically as part of embedded community practices to support those in need. South African Government HBC Program The South African government’s HBC program provided the context and framework for the start up of many of the CBOs supported by Hands at Work. Many care workers started off as Directly Observed Treatment Support for Tuberculosis (DOTS) supporters or as community and home-based care workers, and were available and trained by the government to support the initiation and ongoing work of CBOs. The initial focus of the PEPFAR grant was on palliative care, which is still at the core of the Hands at Work vision and mission; this created a solid platform for their activities through the grant. Government Stakeholders Many government stakeholders support the CBOs’ work, including DSD (provision of food parcels and social grants), the Department of Health (access to free healthcare), the Department of Basic Education (access to free education), and DHA (provision of legal documents). The combined efforts of these government departments, mediated by CBOs’ support and referrals, enables children to access services in vulnerable communities. Contextual Barriers Social and Economic Conditions of the Region Hands at Work implements its programs in the most vulnerable communities, and socioeconomic conditions in these areas are substantially worse than in other parts of the country. These areas typically have low levels of infrastructure and poor provision of basic services. Providing children with services like legal documents, shelter, and clothes often takes a long time, and mechanisms of referral and networking do not happen easily. Government departments and other stakeholders have capacity challenges which weaken and slow down the general levels of service delivery. Recent civic unrest and protest in Mpumalanga points to community dissatisfaction with lack of service delivery from government departments. High Levels of Community Needs Working in these poor areas means that needs are overwhelming and makes it very difficult to reach everyone. Due to Hands at Work’s focus on the most vulnerable, many are excluded from the program even though they are vulnerable and may need support. Hands at Work UGM End of Project Evaluation 32 Community Beliefs and Cultural Barriers Many of the communities in which Hands at Work operates are still affected by HIV stigma and discrimination. Many families in need of support do not want to be associated with Hands at Work for fear of being stigmatized as “HIV people.” It takes time for CBOs and care workers to win these families over, though this time lag can have dire consequences for those that are already very ill and have few options in terms of treatment. Lack of Support of Local Churches Hands at Work is a FBO and encourages partnerships and linkages with local churches. Although a few CBOs mentioned some of the local churches, this was not a common finding. A few respondents even reported that there are many local churches that do not collaborate with Hands at Work. One Hands at Work staff member noted during a KII that “the churches are not involved with the work Hands does. I got into the work with Hands through the church, but the church is not involved. It would be important to have their support, to be part of the overseeing, part of the involvement.” Key Evaluation Question 3 Summary The Hands at Work model of building communities’ capacity to care for vulnerable children means that their work is embedded in local communities. The local community was found to be both a key programmatic and contextual enabler for Hands at Work. The most significant barriers to Hands at Work meeting its objectives included lack of human resources and capacity, rapid turnover of staff and volunteers, and differences in program approaches between Hands at Work and PEPFAR. Hands at Work UGM End of Project Evaluation 33 V. CONCLUSIONS The Hands at Work program has initiated and supported a number of important community projects aimed to supporting vulnerable children in poor communities. Building community capacity is not a straightforward process, and is unlikely to be linear; setbacks and challenges are to be expected. However, this approach can take longer to address the priority needs of children. While the Hands at Work program has had clear successes, it does not seem to have achieved the coverage of services that would improve the lives of vulnerable children. BUILDING CAPACITY OF CAREGIVERS TO CARE FOR OVC AND THE SICK Overall, primary caregivers reported increased capacity to care for children as a result of the capacity building interventions of Hands at Work and the CBOs. Primary caregivers are trained to care for and support children and other family members who are ill or dying. Caregivers are often raising children who are not their own and are left in their care when family members die. Many of these primary caregivers are grandmothers, who need considerable support to care for children. They reported that PCG trainings helped them with parenting skills and with understanding children’s emotional experiences. The coverage of this training was unclear. Regular care worker visits to primary caregivers’ homes is felt to be a further level of support for primary caregivers, and also serve as a form of psychosocial support. Where home visits were not regular, primary caregivers said they felt let down by the CBOs. SERVICE PROVISION OF A HOLISTIC PACKAGE OF SERVICES TO OVC Focus group findings suggest that services were being provided to vulnerable children, including access to education (fees, uniforms, afterschool care), access to healthcare, support to meet physical needs (food, clothes, home repair, and social grants), and psychosocial support. Care centers are a very important form of service provision for children at a range of levels. They give younger children access to ECD activities and care. For school age children, the centers provide a safe space after school to be given a meal, socialize with other children and adult care workers, and receive homework support. The most notable service challenges raised related to food and access to identity documents. Reliance on food from DSD resulted in many frustrations expressed across the CBOs. BUILDING CAPACITY OF CBOS IN ORGANIZATIONAL FUNCTIONING The findings suggest that each of the CBOs has a different level of organizational functioning. The CBOs demonstrated varying degrees of maturity and independence, capacity to fundraise or access government services, and ability to engage and network with other stakeholders. There was very little evidence to suggest that the CBOs were able to fundraise to ensure their long-term sustainability. Even the graduated CBO, which had reached a level of organizational independence, questioned whether it would be sustainable in the long term. Evidence of sufficient networking with other stakeholders in the communities was missing—there was no discussion of relationships built with other NGOs in the areas. One interview mentioned Childline in relation to training, but there was no sense that Hands at Work or the CBOs were actively involved in networking and building relationships. There appears to be an overreliance on government services with insufficient evidence of networking. For example, all the CBOs had negotiated fee exemptions with local schools and provide uniforms; however, the relationship appeared to stop there. CBOs did not liaise with schools to ensure that children on their Hands at Work UGM End of Project Evaluation 34 register are not only receiving education, but reaching their education potential. One school principal indicated that while the school knew of the CBO and their work, and a CBO member sat on the school governing body, the principal knew little about Hands at Work’s wider activities such as youth camps and afterschool activities at care centers. There were also many frustrations mentioned with DSD and the provision of food parcels, but there was little mention of what CBOs could do to increase food security in homes, such as the development of food gardens. The CBOs interviewed showed little evidence of thinking through sustainability issues or of having developed long-term strategies or workplans. Given that some of the organizations had low levels of capacity and skills, this is perhaps an unrealistic expectation. Basic organizational skills are the focus of Hands at Work’s CBO capacity building and, from the beginning, there has been insufficient emphasis on sustainable thinking. There is a definite need for Hands at Work to encourage CBO leaders and managers to always think of sustainability and funding, from as early as possible in their support. OVERALL CONCLUSION The Hands at Work model is premised on the belief that building community capacity, through CBOs and care workers, will result in service delivery to vulnerable children and their families. The PEPFAR service provision model focuses on granting children direct access to a range of services to help them meet their development potential and overcome their vulnerabilities. Other than youth camps and training, Hands at Work does not provide services directly, but works through CBOs to provide services. The lack of available data for each of the specific services provided to children over the life of the grant make it difficult to assess whether the project objectives have been met. The available quantitative data suggest that the targets set by Hands at Work were not achieved. Provision of services directly to children by outside agencies is not sustainable, but focusing too heavily on capacity building may leave poor children vulnerable for long periods. Hands at Work needs to combine their focus on capacity building, their clear niche in psychosocial support, and a commitment to meeting children’s basic needs. Hands at Work UGM End of Project Evaluation 35 VI. RECOMMENDATIONS FOR HANDS AT WORK The Hands at Work capacity building model has strengths and the organization should continue to build on these. Strengthening the areas described below would extend their reach in vulnerable communities. Improve Quality Assurance As the program grows, the need for technical assistance, supportive supervision, and mentoring to enhance capacity for quality programming and interventions will also grow. Providing standard operating protocols and other guidelines will become increasingly crucial as the number of partners and the scope of implementation widens, such as using national guidelines on OVCs and other relevant guidelines and protocols available in South Africa. It is also important to develop and use standardized/accredited OVC training manuals, including manuals for training trainers, volunteers/care workers, primary caregivers, and the children themselves, and offer refresher training on the care and support of OVCs. In addition, a strong MER system is needed to measure the outcome and impact of the capacity building provided to CBOs. The care center model has shown its effectiveness in reaching OVCs; however, the combination of a care center and an outreach activity performed by care workers (home visits) is still appropriate and recommended, supported by a strong MER system. Improve Networking with Government Improved networking with government could include linking care workers to other initiatives led by DSD, the Department of Basic Education, and the Department of Health (DoH), such as the Child and Youth Care worker programs, school care and support activities, and community health outreach programs. Hands at Work should explore these programs and help CBOs link with them to access trainings (usually already accredited) and provide possible career opportunities for care workers. Food security was a critical issue for all beneficiaries, and although food gardens were mentioned occasionally (especially in relation to households), there did not appear to be a significant drive to establish gardens in care centers. The DoA could be an important resource for the establishment of food gardens. This is an effective food security strategy used in many South African OVC projects and offers a space for collaboration and networking with the Department of Agriculture. Hands at Work should make more arrangements with DHA for one day or week long events or jamborees10 that enable beneficiaries to access identity documents, passports, and birth and death certificates. SASSA should be invited to these events to facilitate access to grants at the same time. Events held previously by Hands at Work had positive results; for example, in the third quarter of 2010, an event led to 198 people accessing documents and 28 orphans applying for grants. These activities are therefore an effective way of reaching large numbers of people. Improve Partnerships and Networks with Other Local Organizations, Non￾governmental Stakeholders, Community Structures, and Private Sectors While partnerships and relationships were mentioned by respondents, this did not appear to be a particular strength for Hands at Work or the CBOs they support. Structures such as Child Protection Forums and NGOs like GRIP, Childline, and Child Welfare could be useful partners for Hands at Work. A mapping exercise could uncover a range of potential networks and partners to support their work.                                                              10Many OVC programs hold daily or weekly events (often called jamborees) which bring various government departments to the community, especially DoH, DHA, and SASSA. Hands at Work UGM End of Project Evaluation 36 Grant managers, who support a range of community-based projects that deal with vulnerable populations, could facilitate this exercise. The issue of stipends for care workers (as volunteers) needs to be addressed, as it is the main cause for high turnover of care workers in CBOs. Efforts could be made by Hands at Work to establish partnerships with private sectors for funding for care worker stipends or looking for additional funding opportunities. Sustainability Sustainability issues need to be addressed more fully across all the CBOs supported by Hands at Work. This should include fundraising, which needs to be a constant focus for the organization. CBOs in the sample were concerned about sustainability and some were still very reliant on Hands at Work support. Hands at Work could offer training and onsite support to ensure that sustainability plans are in place. Another effort to make the program sustainable is through community (schools, churches, and other key stakeholders) involvement and participation in the life cycle of the program. Ensuring adequate community participation in the identification and selection of children to benefit from the program is very important. The participatory process is a valuable opportunity for communities to appreciate the magnitude of the challenge and the number of children in need. It helps build community support for the program, ensures that benefits reach the right children, and produces community pledges and the mobilization of additional resources for the support of the children and their families, even when the program ends. Another challenge that needs to be addressed, as it will (indirectly) ensure sustainability, is the low literacy level of beneficiaries, particularly primary caregivers. Hands at Work needs to design activities to improve literacy levels of beneficiaries, including OVC (which can also be done through schools) and primary caregivers (which can be done through support group meetings at care centers). Continue to Build Capacity of CBOs to Run Youth Camps and the Young Mums Program Both the youth camps and the Young Mums program fill a critical psychosocial function for young people. Hands at Work should explore ways in which these programs could be offered at a local level, and secure funding to ensure the continuation of the program. There could also be improved support for the children of young mothers. The evaluation found that Hands at Work helps young mothers continue with schooling and to access services like healthcare and grants. There is the potential to improve support for the care of their children, for example through crèches. Hands at Work needs to improve the capacity of CBOs to take over both of these programs when they are ready. Ongoing capacity building, including trainings and mentoring, are needed to prepare the CBOs to implement youth camps and Young Mums independently. Since Young Mums is an innovative program, it may benefit from further evaluation to determine the best way to extend the program. Continue to Build and Strengthen the ECD Skills of Care Workers Many beneficiaries cited ECD as an important Hands at Work service. This should be another ongoing and key focus area for Hands at Work. ECD facilities allow for the support, care, protection, and educational development for pre-school children. It also offers primary caregivers additional support with young children in their care. Increase Focus on Income Generation as a Strategy for Household Economic Strengthening Income generation is cited as a Hands at Work activity; however, it did not emerge as an area for discussion in the focus groups with beneficiaries or CBOs. Vulnerable communities, such as those Hands at Work UGM End of Project Evaluation 37 supported by Hands at Work, need to focus on income generation as a way of supplementing the social grants they receive. There are many local and Africa-wide examples and models of best practice in income generation. In other African countries, Hands at Work has promising practices with regard to income generation, and this should be shared and absorbed in South Africa. Hands at Work reported that this area of activity was not strong in South Africa due to the availability of social grants; however, households need to look beyond social grants, as these are insufficient to ensure basic needs of their members. FOR USAID Develop Robust Tools to Measure Capacity Building Programs A number of international initiatives, some funded by USAID, are currently looking at how to best measure capacity building initiatives. While service delivery indicators for OVC programs are well established, capacity building indicators, and the tools for measuring these, are not well developed. This has implications for a program like Hands at Work, with its strong capacity building focus. Developing well established methods for measuring capacity building will allow USAID and others to determine whether these programs are successful in achieving their objectives. Commission Research into OVC Programs One possible way to measure capacity development is to commission research that includes longitudinal studies of OVC programs looking at outcomes for children over a substantial period of time. Such a study could evaluate a range of programs that include various models of care and support, and could use a MSC methodology, collecting stories over the life of a program to illustrate the key factors that make a difference to the life of a vulnerable child, and the model that is best placed to deliver these services. FOR THE SOUTH AFRICAN GOVERNMENT Capacity Building for CBOs and NGOs The community and non-governmental sector is the backbone of service delivery and support to vulnerable children and families in South Africa. Continued support to build the capacity of this sector is critical in the context of a country that simply does not have the quantity and quality of social workers, health workers, and educators necessary to improve the development outcomes of families, and in particular children. Strengthened Linkages between CBOs and Government Ongoing efforts need to be made across government departments to strengthen and build networks with organizations offering services to communities. There should also be efforts to increase collaboration among different government sectors to provide a more complete service packages in communities where there are multiple needs. Move the Capacity Building Agenda beyond Accredited Training The South African capacity building agenda focuses on accredited training through the relevant Sector Education Training Authorities. Organizations are under pressure to organize and deliver accredited training, and other forms of non-accredited training are discouraged. The process for accrediting training programs currently presents many challenges, and needs to be simplified and made more accessible for CBOs such as Hands at Work. These organizations have limited resources and capacity to accredit their training programs, but have the potential to reach large numbers of people with relevant training. This finding is not unique to this evaluation and is true for many smaller CBOs and NGOs in South Africa. Hands at Work UGM End of Project Evaluation 38 There is a need for robust debate and thinking on how best to build capacity to care for vulnerable children in South Africa—to move beyond just training. Hands at Work UGM End of Project Evaluation 39 APPENDIX 1: COMPOSITION OF THE EVALUATION TEAM The Hands at Work program was evaluated by a team composed of one Senior Researcher, two Research Analysts, and a Team Leader. One of the Research Analysts is fluent in the local languages of the program areas, and conducted interviews in the vernacular where appropriate. The overall process was coordinated and managed by an experienced project Team Leader and Senior Researcher, with Senior Technical Advisors providing input and backstopping as needed. Hands at Work Evaluation Team: Role and Responsibilities Role Name(s) Responsibilities Senior Technical Advisors Saul Johnson and Hilbrand Haak Oversee entire project and assist in study design; manage the technical team; provide input to data collection tools/processes; meet with client Senior Researchers Nirvana Pillay and Nur Hidayati Oversee field data collection, Senior Interviewers, provide backstopping and technical assistance to research analysts where needed; data analysis; assist in conducting field preparation workshop train fieldworkers; draft reports and case studies Research Analysts Priscilla Morley and María Belizán Develop data collection tools and guides; collect qualitative data; assist with qualitative data analysis, interpretation, and report writing; develop project database Hands at Work UGM End of Project Evaluation 40 APPENDIX 2: SAMPLING FRAMEWORK The respondents and data collection methods being utilized with each respondent/respondent group are outlined in the table below. The evaluation took place at the Hands at Work Hub in White River and a selection of CBOs was sampled in the Bushbuckridge and Clau Clau areas, where there has been significant engagement with Hands at Work. Hands at Work advised that there was no activity currently in the Northern Cape or the Northwest provinces, and so these were not part of the sample. These were sub-partners that closed out during the course of 2011. Sampling strategy Target of respondent/s Organization Data collection Method Hands at Work leadership Hands at Work Individual or group face-to￾face interviews Hands at Work program management Hands at Work Individual or group face-to￾face interviews Field Coordinators and Service Center staff Individual or group face-to￾face interviews. CBO leaders and staff At least four of the nine currently active CBOs in the Bushbuckridge and Clau Clau areas One graduated CBO in the Clau Clau area Individual or group face-to￾face interviews Community members N/A Individual or group face-to￾face interviews Primary Caregivers CBOs Focus Group Discussions Care workers CBOs Focus Group Discussions Youth CBO Focus Group Discussions Young Mothers CBO Focus Group Discussions There are currently nine CBOs receiving support from Hands at Work— five in the Clau Clau area and four in the Bushbuckridge areas. Three CBOs in the Clau Clau area and two CBOs from the Bushbuckridge area were included in the sample. One “graduated” CBO that no longer receives support from Hands at Work was also included, for a total of five CBOs evaluated. Criteria to select the CBOs was guided by Hands at Work, but was also independently determined after an initial period in the field. Hands at Work UGM End of Project Evaluation 41 APPENDIX 3: DATA COLLECTION TOOLS USED Instrument 1. Hands at Work Project Leadership (HUB) – In-depth Interview INTRODUCTION I am ……………………………… and ……………………….. from Health and Development Africa/Results in Health. We are a consulting company conducting an evaluation to determine if and to what extent Hands at Work, and the CBOs they support, have improved the well-being of vulnerable children in the districts where they operate. The aim of this interview is to find out more about your thoughts, perspectives and feelings about the work done by Hands and the CBOs they work. The main questions that we are going to ask are:  To what extent did Hands succeed in enhancing their CBO sub-partner’s capacity to identify and successfully respond to the needs of vulnerable children?  What were the key enablers and barriers in meeting project objectives? This interview is part of an evaluation of Hands at Work. This means that we are looking back at the work of Hands over the past 5 years and finding out as much as we can (knowledge generation) about the successes and challenges Hands experienced in providing care and support to vulnerable children through the support of CBOs and other programs. This evaluation is not to provide more funding or take away funding. It is to help improve the care and support services provided to vulnerable children through this program and through other programs. We will be using recorder to audiotape the interview, to help us analyze the interview. Please sign the attendance register for our record-we will not use your name when writing the report. If you agree to this conversation being recorded and to be interviewed, please sign the consent form and hand it back to us. If at any time you feel that there is a question you are not comfortable answering, or that you would not like to continue being interviewed, please let us know, and we will stop. We will be speaking to you for about an hour. BIOLOGICAL AND BACKGROUND INFORMATION 1 Date of interview 2 Time interview began 3 Time interview ended 4 Time in minutes 5 Province 6 Area/Town 7 Interviewee name/s 8 Interviewers name 9 Organization 10 Position/Role 11 For how long have you been involved in Hands at Work Hands at Work UGM End of Project Evaluation 42 12 Gender of respondent/s (tick one) Male Female 13 Interviewer additional observations and remarks BACKGROUND TO HANDS AT WORK Q1. What is the model that Hands at Work uses? Q2. Can you please name all the CBOs that Hands supports and tell us a bit about them? Q3. What is Hands role in identifying and responding to the needs of vulnerable children? Q4. Can you please tell us about all the programs you offer? Probe: Youth camps, Young Mums, Vocational skills, PCG training, etc. Q5. Do you work with other stakeholders and organizations in the communities (other than the CBOs)? Please tell us about this. Probe: traditional leaders, government partners, other NGOs and partners, community structures like schools and clinics CAPACITY DEVELOPMENT OF CBOs Q6. What kinds of capacity development does Hands offer to CBOs? Q7. Does Hands offer training to CBOs? What are these? Probe: How many? How often? Who attends? Q8. What is the content of the training programs? Q9. How did the training build capacity in CBOs to identify and respond to the needs vulnerable children? Q10. Were there any other capacity building strategies used (other than training)? What were these? Could you kindly tell us more about them. SERVICE PROVISION FOR VULNERABLE CHILDREN Q11. What kind of services do children receive from the CBOs. For each category, please give us details.  Access to education?  Access to social grants and other legal documentation? Hands at Work UGM End of Project Evaluation 43  Access to healthcare/services ? Probe: When respondent lists services, ask for details about each service and how it is accessed. Q12. Please tell us how Hands works directly with children? Please describe these. Please tell us about the impact they have on children. Probe: Ask about :  Youth Camps?  Young Mums Program?  Any others mentioned Q13. What were the challenges Hands experienced in the roll out of programs? How have you dealt with these? KEY ENABLERS AND BARRIERS IN MEETING PROJECT OBJECTIVES Q14. What worked really well (in relation to the work of the project/program)? What are you proud of? Q15. Were there any specific people or groups that supported the project/program? How? What did they do? Q16. What could have been done better? Why do you say this? What were some of the challenges you experienced in meeting your objectives? Were those challenges addressed and how? CLOSING COMMENTS Is there anything else you would like to share with us? Is there anything you would like to know from us? Hands at Work UGM End of Project Evaluation 44 Instrument 2. Hands at Work Regional Support Team - In-depth Interview (Group or individual) Interviewer notes: This tool is used to interview a range of program coordinators in the RST that each has different roles and responsibilities. Please probe as appropriate in relation to their specific program area. INTRODUCTION I am ……………………………… and ……………………….. from Health and Development Africa/Results in Health. We are a consulting company conducting an evaluation to determine if and to what extent Hands at Work, and the CBOs they support, have improved the well-being of vulnerable children in the districts where they operate. The aim of this interview is to find out more about your thoughts, perspectives and feelings about the work done by Hands and the CBOs they work within the community. The main questions that we are going to ask are:  To what extent did Hands succeed in enhancing their CBO sub-partner’s capacity to identify and successfully respond to the needs of vulnerable children?  What were the key enablers and barriers in meeting project objectives? This interview is part of an evaluation of Hands at Work. This means that we are looking back at the work of Hands over the past 5 years and finding out as much as we can (knowledge generation) about the successes and challenges Hands experienced in providing care and support to vulnerable children through the support of CBOs and other programs. This evaluation is not to provide more funding or take away funding. It is to help improve the care and support services provided to vulnerable children through this program and through other programs. We will be using recorder to audiotape the interview, to help us analyze the interview. Please sign the attendance register for our record-we will not use your name when writing the report. If you agree to this conversation being recorded and to be interviewed, please sign the consent form and hand it back to us. If at any time you feel that there is a question you are not comfortable answering, or that you would not like to continue being interviewed, please let us know, and we will stop. We will be speaking to you for about an hour. BIOLOGICAL AND BACKGROUND INFORMATION 1 Date of interview 2 Time interview began 3 Time interview ended 4 Time in minutes 5 Province 6 Area/Town 7 Interviewee name/s 8 Interviewers name 9 Organization 10 Position/Role 11 For how long have you been involved in Hands at Work Hands at Work UGM End of Project Evaluation 45 12 Gender of respondent/s (tick one) Male Female 13 Interviewer additional observations and remarks ROLE OF THE RST COORDINATORS AND MANAGERS Q1. What is your role in the RST? Q2. What is your relationship with the HUB? Q3. What is your relationship with the Service Centers? Q4. What is your relationship/involvement with CBOs? Q5. What programs do the RST offer? Probe: different levels of details for specific programs managed by respondent? Q6. Which program/s of these are you directly involved with? How? Q7. Impact of these programs on vulnerable children? Q8. What were the challenges that have been experienced in the roll out of programs? How have you dealt with these? Q9. Are there other stakeholders and organizations in the communities (other than the CBOs) that the RST is involved with? Please tell us about this. Probe: traditional leaders, government partners, other NGOs and partners, community structures like schools and clinics KEY ENABLERS AND BARRIERS IN MEETING PROJECT OBJECTIVES Q10. In relation to your work/role, what worked really well (in relation to the work of the project/program)? What are you proud of? Q11. In relation to your work/role what could have been done better? Why do you say this? CLOSING COMMENTS Is there anything else you would like to share with us? Is there anything you would like to know from us? Hands at Work UGM End of Project Evaluation 46 Instrument 3. Hands at Work Service Center Team - In-depth Interview (Individual or Group) INTRODUCTION I am ……………………………… and ……………………….. from Health and Development Africa/Results in Health. We are a consulting company conducting an evaluation to determine if and to what extent Hands at Work, and the CBOs they support, have improved the well-being of vulnerable children in the districts where they operate. The aim of this interview is to find out more about your thoughts, perspectives and feelings about the work done by Hands and the CBOs they work with in your community. The main questions that we are going to ask are:  To what extent did Hands succeed in enhancing their CBO sub-partner’s capacity to identify and successfully respond to the needs of vulnerable children?  What were the key enablers and barriers in meeting project objectives? This interview is part of an evaluation of Hands at Work. This means that we are looking back at the work of Hands over the past 5 years and finding out as much as we can (knowledge generation) about the successes and challenges Hands experienced in providing care and support to vulnerable children through the support of CBOs and other programs. This evaluation is not to provide more funding or take away funding. It is to help improve the care and support services provided to vulnerable children through this program and through other programs. We will be using recorder to audiotape the interview, to help us analyze the interview. Please sign the attendance register for our record-we will not use your name when writing the report. If you agree to this conversation being recorded and to be interviewed, please sign the consent form and hand it back to us. If at any time you feel that there is a question you are not comfortable answering, or that you would not like to continue being interviewed, please let us know, and we will stop. We will be speaking to you for about an hour. BIOLOGICAL AND BACKGROUND INFORMATION 1 Date of interview 2 Time interview began 3 Time interview ended 4 Time in minutes 5 Province 6 Area/Town 7 Interviewee name/s 8 Interviewers name 9 Organization 10 Position/Role 11 For how long have you been involved in Hands at Work 12 Gender of respondent/s (tick one) Male Female 13 Interviewer additional observations and remarks Hands at Work UGM End of Project Evaluation 47 BACKGROUND TO HANDS AT WORK SERVICE CENTER Q1. What is the role of the Service Center? Q2. What is your role in the Service Center? Q3. What is your relationship to the Hub and the RST? Q4. Can you please name all the CBOs that Hands supports and tell us a bit about them? The name? The community they serve? How long Hands has supported them? Rate them from 1-5 (1 inception and need a lot of support and 5 being CBOs ready to graduate) and why you rank them like this Q5. What is Hands role in identifying and responding to the needs of vulnerable children? How do they do this? Q6. Can you please tell us about all the programs you offer? Probe: Youth camps, Young Mums, Vocational skills, PCG training etc Q7. Do you work with other stakeholders and organizations in the communities (other than the CBOs)? Please tell us about this. Probe: traditional leaders, government partners, other NGOs and partners, community structures like schools and clinics CAPACITY DEVELOPMENT OF CBOs Q8. What kinds of capacity development does Hands offer to CBOs? Q9. Does Hands offer training to CBOs? What are these? Probe: How many? How often? Who attends? Is it schedules? Who facilitates? Q10. How do you determine the CBO’s capacity building/training needs? Q11. What is the content of the training programs? Q12. What was the benefit of the training for CBOs? Q13. Were there any other capacity building strategies used (other than training)? What were these? Could you kindly tell us more about them? SERVICE PROVISION FOR VULNERABLE CHILDREN Q14. What kind of services do children receive from the CBOs? For each category, please give us details. Access to education? Access to social grants and other legal documentation? Access to healthcare/services? Probe: When respondent lists services, ask for details about each service and how it is accessed. Q15. Please tell us how Hands works directly with children? Please describe these. What is the impact of these on children? Hands at Work UGM End of Project Evaluation 48 Probe: Ask about: Youth Camps? Young Mums Program? Any others mentioned Q16. Are there other organizations, individuals or stakeholders that help CBOs with capacity building? Q17. What were the challenges Hands experienced in the roll out of programs? How have you dealt with these? KEY ENABLERS AND BARRIERS IN MEETING PROJECT OBJECTIVES Q18. What works really well (in relation to the work of the service center)? What are you proud of? Q19. Are there any specific people or groups that supported the project/program work of the CBO? How? What do they do? Q20. What could the service center do better? Why do you say this? What are some of the challenges you experienced in meeting your objectives? Were those challenges addressed and how? CLOSING COMMENTS Is there anything else you would like to share with us? Is there anything you would like to know from us? Hands at Work UGM End of Project Evaluation 49 Instrument 4. CBO Leaders/Coordinators – In-depth Interview (Individual or Group) INTRODUCTION I am ……………………………… and ……………………….. from Health and Development Africa/Results in Health. We are a consulting company conducting an evaluation to determine if and to what extent Hands at Work, and the CBOs they support, have improved the well-being of vulnerable children in the districts where they operate. The aim of this interview is to find out more about your thoughts, perspectives and feelings about the work done by Hands and the CBOs they work with in your community. The main questions that we are going to ask are:  To what extent did Hands succeed in enhancing the CBO sub-partner’s capacity to identify and successfully respond to the needs of vulnerable children?  What were the key enablers and barriers in meeting project objectives? This interview is part of an evaluation of Hands at Work. This means that we are looking back at the work of Hands over the past 5 years and finding out as much as we can (knowledge generation) about the successes and challenges Hands experienced in providing care and support to vulnerable children through the support of CBOs and other programs. This evaluation is not to provide more funding or take away funding. It is to help improve the care and support services provided to vulnerable children through this program and to other programs. We will be using recorder to audiotape the interview, to help us analyze the interview. Please sign the attendance register for our record-we will not use your name when writing the report. If you agree to this conversation being recorded and to be interviewed, please sign the consent form and hand it back to us. If at any time you feel that there is a question you are not comfortable answering or that you would not like to continue being interviewed, please let us know, and we will stop. We will be speaking to you for about an hour. BIOLOGICAL AND BACKGROUND INFORMATION 1 Date of interview 2 Time interview began 3 Time interview ended 4 Time in minutes 5 Province 6 Area/Town 7 Interviewee name 8 Interpreter name (where relevant) 9 Organization 10 Position 11 For how long have you been involved in Hands at Work Hands at Work UGM End of Project Evaluation 50 12 Gender of respondent (tick one) Male Female 13 Interviewer additional observations and remarks CAPACITY DEVELOPMENT OF CBOs Q1. Please tell us about the role of the CBO within the Hands at Work initiative. Q2. Has the CBO received any training/capacity development from Hands? Q3. Did the training improve your ability to identify vulnerable children? If yes, how? Q4. Did the training improve the manner in which you provided care and support to vulnerable children? Please explain? Q5. What were the highlights of the training? What stands out for you? Q6. Did you experience any challenges with regards to the training or implementing what you had learnt? Q7. Were there any other capacity building strategies that Hands used to increase your skills in identifying and responding to the needs to vulnerable children? Probe: mentoring or on the job coaching. Please tell me more about what was done and the benefits and challenges thereof. SERVICE PROVISION FOR VULNERABLE CHILDREN Q8. What is the CBOS’s role and experience in working with vulnerable children in the community? Please tell us about it? PROBE: What kind of service do you provide to vulnerable children in your community? Q9. How does the CBO support vulnerable children access physical needs for example clothes, shelter, and food parcels? PROBE: What services do you provide to vulnerable children to meet their physical needs? What are some of the barriers or challenges you experience in providing these services? Q10. How does the CBO support or refer vulnerable children to meet their emotional (psychosocial) needs? PROBE: Describe the services that you provide or refer vulnerable children for? What are some of the barriers or challenges you experience in meeting the emotional needs of vulnerable children? Q11. How does the CBO help vulnerable children access health services (either directly or through a referral system)? Probe: Name the kind of health services that you helped provide. What enabled you to do this? What were some of the barriers or challenges you experienced in meeting the health needs of vulnerable children? Q12. How does the CBO help vulnerable children access educational services? Hands at Work UGM End of Project Evaluation 51 PROBE: What enabled you to meet the educational needs of the vulnerable children? What were some of the barriers or challenges experienced in meeting the educational needs? Q13. How does the CBO help vulnerable children access social grants and legal documentation? PROBE: What enabled you to access social grants and/or legal documents? What were some of the barriers or challenges in accessing social grants and/or legal documentation? Q14. In your knowledge what has been some of the key successes in identifying and providing services to vulnerable children in your community? Do you have any stories or examples you would like to share with us? Q15. In your knowledge what has been some of the challenges you have experienced in identifying and providing services vulnerable children in your community? Please tell us about this. Q16. What was the impact of the Youth HIV prevention camps? Please describe. NETWORNING AND COMMUNITY MOBILIZATION Q17. After being involved with Hands at Work were you able to network with other stakeholders in the community to meet the needs of vulnerable children? Q18. Are there other organizations providing services to vulnerable children? Tell us more about them. Do you have a networking/supportive relationship with them? What kind of services do you source from these organizations? Q19. What are some of the successes you have achieved in working with other stakeholders? What are some of the challenges that you have experienced? How did you manage the challenges? Did you receive assistance from Hands? GENERAL EVALUATION OF THE PROJECT From having been involved with Hands at Work/CBO and looking back/reflecting on your work: Q20. What worked really well (in relation to the work of the project/program)? What are you proud of? Q21. What could have been done better? Why do you say this? What were some of the challenges you experienced in meeting your objectives? Were those challenges addressed and if so, how? CLOSING Is there anything else you would like to share with us? Is there anything you would like to know from us? Hands at Work UGM End of Project Evaluation 52 Instrument 5. Key Community Members (School)11 – In-depth Interview INTRODUCTION I am ……………………………… and ……………………….. from Health and Development Africa/Results in Health. We are a consulting company conducting an evaluation to determine if and to what extent Hands at Work, and the CBOs they support, have improved the well-being of vulnerable children in the districts where they operate. The aim of this interview is to find out more about your thoughts, perspectives and feelings about the work done by Hands and the CBOs they work with in your community. This interview is part of an evaluation of Hands at Work. This means that we are looking back at the work of Hands over the past 5 years and finding out as much as we can (knowledge generation) about the successes and challenges Hands experienced in providing care and support to vulnerable children through the support of CBOs and other programs. This evaluation is not to provide more funding or take away funding. It is to help improve the care and support services provided to vulnerable children through this program and to other programs. We will be using recorder to audiotape the interview, to help us analyze the interview. Please sign the attendance register for our record-we will not use your name when writing the report. If you agree to this conversation being recorded and to be interviewed, please sign the consent form and hand it back to us. If at any time you feel that there is a question you are not comfortable answering or that you would not like to continue being interviewed, please let us know, and we will stop. We will be speaking to you for about an hour. BIOLOGICAL AND BACKGROUND INFORMATION 1 Date of interview 2 Time interview began 3 Time interview ended 4 Time in minutes 5 Province 6 Area/Town 7 Interviewee name 8 Interpreter name (where relevant) 9 Organization 10 Position 11 For how long have you been involved with Hands at Work/CBO?                                                              11 A similar tool will be used for other key community members. We will refer to the name of the specific CBO in the community once we have mapped the community and sampled the specific CBO. Hands at Work UGM End of Project Evaluation 53 12 Gender of respondent (tick one) Male Female 13 Interviewer additional observations and remarks SERVICE PROVISION FOR VULNERABLE CHILDREN Q1. Tell me of your experience and understanding of the role of Hands/CBO with regards to vulnerable children in your school and community. What kind of services do they provide to vulnerable children in the school or community? Q2. What kind of services do children receive from the CBOs? For each category, please give us details.  Access to education?  Access to social grants and other legal documentation?  Access to healthcare/services? Probe: When respondent lists services, ask for details about each service and how it is accessed. Q3. Do you know about the HIV prevention Camp and the Young Mum’s program? If so, tell us about it? PROBE: What was the impact of these programs on young people? What are your observations of the impact? Q4. In your opinion, what has been some of the key successes in working with Hands/CBO in identifying and providing services to vulnerable children? Q5. In your knowledge what has been some of the challenges that have been experienced in the identification and service provision to vulnerable children? PARTNERSHIPS AND RELATIONSHIP WITH Hands at Work/CBO Q6. What kind of relationships do you have with Hands/CBO? Please explain? Q7. How does Hands/CBO strengthen and maintain relationship with you? Q8. How often did you meet to discuss issues? Q9. Did you receive feedback on cases that you referred? Q10. What were the key successes in your partnership with Hands/CBO? Q11. What has been some of challenges that you experienced with that relationship? Q12. What could have been done differently in terms of strengthening the relationship with you? COMMUNITY PERCEPTION OF HANDS AND THEIR SERVICES Q13. In your opinion does the community know about Hands/CBO? Please explain Q14. What kind of relationship does Hands/CBO have with the community? Please explain Hands at Work UGM End of Project Evaluation 54 Q15. How are the CBO volunteers (care workers) received by the community? Q16. What have been some of challenges that they have experienced in the communities? GENERAL EVALUATION OF THE PROJECT From having been involved with Hands at Work/CBO and looking back/reflecting on your work: Q17. What worked really well (in relation to the work of the project/program)? What are you proud of? Q18. What could have been done better? Why do you say this? What were some of the challenges you experienced in meeting your objectives? Were those challenges addressed and if so, how? CLOSING COMMENTS Is there anything else you would like to share with us? Is there anything you would like to know from us? Hands at Work UGM End of Project Evaluation 55 Instrument 6. Care worker - Focus Group Discussion INTRODUCTION I am ……………………………… and ……………………….. from Health and Development Africa/Results in Health. We are working to find out more about how Hands at Work and the CBO (use name) have improved the well-being of children, youth and communities through the care and support services offered. The aim of this focus group discussion/interview is to find out more about your thoughts, perspectives and feelings about the work done by the CBO in your community. There is no right or wrong answer. We are more interested in hearing about your experiences and how they have made a difference in your life. This focus group discussion/interview is part of an evaluation of Hands at Work. This means that we are looking back at the work of Hands over the past 5 years and finding out as much as we can (knowledge generation) about the successes and challenges Hands experienced in providing care and support to vulnerable children through the support of CBOs and other programs. This evaluation is not to provide more funding or take away funding. It is to help improve the care and support services provided to vulnerable children through this program and to other programs. We will be using this recorder to record the discussion. This is just going to help us to go back after the discussion and hear exactly what you said. If you agree to our discussion being recorded and to form part of the discussion, and to take some pictures of you during the evaluation process, please sign the consent form and hand it back to us. Please sign the attendance register for our records; we will not use your name when writing the report. If at any time you feel that there is a question you are not comfortable answering or that you would not like to continue being part of the discussion, please also feel free to let us know. We will be speaking to you for about an hour. Participatory tools for facilitators to use to probe and encourage respondents A. Use cards to draw a happy/smiley face, neutral face and sad face, and use the cards to probe how respondents felt about the services/events/activities. Use the cards to ask questions regarding impact and outcomes, perception of quality of the service provided and to learn about changes that happened in their life (for MSC) and capacity development. E.g. What make you happy/smile – related to event/activity you are involved? – Facilitator shows the smiling face while asking the question and then probe further. E.g. What make you sad/crying – related to event/activity you are involved? – Facilitator shows the sad face while asking the question and then probe further. B. Draw a timeline on a chart/card to compare before and after respondents joined the programs – and facilitates how respondents felt before their involvement and during/after their involvement in the program C. Use a diagram of domain (with colorful paper) to show the type of changes that happen in respondents’ life – the domains are divided to 3 categories: knowledge and skills, change in condition of children - children’s well being, and other. Asking them what is the most important changes from all the changes mentioned, if from the discussion there is already a pattern it means the change which mentioned most often is the most important one for the group    Hands at Work UGM End of Project Evaluation 56 Biographical Information (researcher to complete this section) 1 Date of interview 2 Time interview began 3 Time interview ended 4 Time in minutes 5 Province 6 Area/Town/Community 7 Interviewer name 8 Note taker name 9 Translator Name 10 Number in FGD 11 Gender of focus group respondents (how many males; how many females?) Male Female 12 Interviewer additional observations and remarks BACKGROUND - CBO Q1. Can you tell us a little bit about the CBO? Probe: core function? How long it’s been in existence? Q2. How long have you worked with this CBO as a care worker? Probe: Get a sense of the range of moths/years worked by the Care workers for the CBO. Q3. What is the CBO’s role in identifying and responding to the needs of vulnerable children? Q4. Can you please tell us about all the programs the CBO offer? Probe: Youth programs, Young Mums, Vocational skills etc CAPACITY DEVELOPMENT OF CAREWORKERS Q5. Have you received any training to help you to do your work in supporting vulnerable children? Please describe. Probe: What kinds of training? Provided by whom? How often? Q6. What did you learn in the training? Probe: Ask for details. Q7. How did the training help you with your care work in identifying and responding to the needs vulnerable children? How do you know this (how do you measure this)? Hands at Work UGM End of Project Evaluation 57 Probe: ask for specific examples if they have any – relating to their capacity before and after training Q8. Were you offered any other support (other than training) to develop your skills and knowledge? What were these? Could you kindly tell us more about them? SERVICES OFFERED BY CBOs THROUGH CAREWORKERS Q9. Do you work with vulnerable children in accessing social grants or other legal documentation? How? Q10. Do you work with and support vulnerable children access education? How? Q11. Do you work with and support vulnerable children access health services? How? Q12. Do you work with any other family members of the vulnerable children you have identified? Please tell us more about this. What have you noticed about other family members since you have been involved with supporting the family? Q13. Do you think that the Kids Camps have an impact on youth? Please elaborate Q14. Do you think that the Young Mums Program has an impact on young women? Please describe. Q15. What are the challenges you/the CBO experiences in the delivery of services to vulnerable children? How have you dealt with these? KEY ENABLERS AND BARRIERS IN MEETING PROJECT OBJECTIVES Q16. What has worked really well (in relation to the work of the project/program)? What are you proud of? Q17. Are there any specific people or groups that supported the project/program/your work? How? What did they do? Q18. What could you do better? Why do you say this? MOST SIGNIFICANT CHANGE Q19. What is the best (most significant) change in children’s (quality of) life/well-being since the CBO Program has been implemented? Could you tell us 1 or 2 stories Probe: What happened? When did it happen? How did the change happen? Why is it the most significant change to you? Why do you think the change happen? Q20. What is the best (most significant) change in your community since the existence of CBO? Probe: What happened? When did it happen? How did the change happen? Why is it the most significant change to you? Why do you think the change happen? Q21. What is the best (most significant) change in YOUR capacity to provide services to vulnerable children? Probe: What happened? When did it happen? How did the change happen? Why is it the most significant change to you? Why do you think the change happen? CLOSING COMMENTS Is there anything else you would like to share with us? Is there anything you would like to know from us? Hands at Work UGM End of Project Evaluation 58 Instrument 7: Primary Caregivers – Focus Group Discussion INTRODUCTION I am ……………………………… and ……………………….. from Health and Development Africa/Results in Health. We are working to find out more about how Hands at Work and the CBO (use name) have improved the well-being of children, youth and communities through the care and support services offered. The aim of this focus group discussion/interview is to find out more about your thoughts, perspectives and feelings about the work done by the CBO in your community. There is no right or wrong answer. We are more interested in hearing about your experiences and how they have made a difference in your life. This focus group discussion/interview is part of an evaluation of Hands at Work. This means that we are looking back at the work of Hands over the past 5 years and finding out as much as we can (knowledge generation) about the successes and challenges Hands experienced in providing care and support to vulnerable children through the support of CBOs and other programs. This evaluation is not to provide more funding or take away funding. It is to help improve the care and support services provided to vulnerable children through this program and to other programs. We will be using this recorder to record the discussion. This is just going to help us to go back after the discussion and hear exactly what you said. If you agree to our discussion being recorded and to form part of the discussion, and to take some pictures of you during the evaluation process, please sign the consent form and hand it back to us. Please sign the attendance register for our records; we will not use your name when writing the report. If at any time you feel that there is a question you are not comfortable answering or that you would not like to continue being part of the discussion, please also feel free to let us know. We will be speaking to you for about an hour. Participatory tools for facilitators to use to probe and encourage respondents A. Use cards to draw a happy/smiley face, neutral face and sad face, and use the cards to probe how respondents felt about the services/events/activities. Use the cards to ask questions regarding impact and outcomes, perception of quality of the service provided and to learn about changes that happened in their life (for MSC) and capacity development. E.g. What make you happy/smile – related to event/activity you are involved? – Facilitator shows the smiling face while asking the question and then probe further. E.g. What make you sad/crying – related to event/activity you are involved? – Facilitator shows the sad face while asking the question and then probe further. B. Draw a timeline on a chart/card to compare before and after respondents joined the programs – and facilitate how respondents felt before their involvement and during/after their involvement in the program Biographical Information (researcher to complete this section) 1 Date of interview 2 Time interview began Hands at Work UGM End of Project Evaluation 59 3 Time interview ended 4 Time in minutes 5 Province 6 Area/Town/Community 7 Interviewer name 8 Note taker name 9 Translator Name 10 Number in FGD 11 Gender of focus group respondents (how many males; how many females?) Male Female 1 12 Interviewer additional observations and remarks RELATIONSHIP TO THE CBO Q1. What do you know about the CBO? Have you received any support from them? Q2. How did you find out about the CBO and the program for vulnerable children? Tell us about how you and your child/ren were registered for the program. Was it easy to gain access to the program? Please tell us more about this. Q3. Please tell us a bit about your experience/relationship with the CBO. Probe: How often do you meet with the CBO/Care workers? What happens in these meetings/visits? SERVICE DELIVERY TO VULNERABLE CHILDREN Q4. What kind of services has your child/ren received from the CBO? For each category, please give us details.  Access to education?  Access to social grants and other legal documentation?  Access to healthcare/services when you need them? Probe: When respondents list services, ask for details about each service and how it was accessed? Hands at Work UGM End of Project Evaluation 60 Q5. Has the CBO provided your child/ren with the opportunity to talk about any personal problems he/she is experiencing? What kind of support was provided to your child to address his/her problems? Q6. Has your child/ren been involved in any of the youth camps? Tell us about this. How has it impacted on/affected your child? Probe: Which camps? What were the impacts? IMPACTS AND OUTCOMES FOR PRIMARY CAREGIVERS Q7. Did you attend the training for Primary Caregivers or any other training? Probes: Ask about training: was the respondent trained so they can support children? What was the nature of the training? Q8. How did the training contribute to your knowledge and behavior in providing support to your child/ren? Q9. Have you experienced any challenges with the CBO program for vulnerable children? Tell us more. PERCEPTION OF QUALITY AND EASE OF ACCESS TO SERVICES BY OTHER STAKEHOLDERS Q10. Apart from Hands and CBO staff, are there any other organizations or people that assist you/your child in accessing services? Please describe this for us. Q11. What kind of support do you receive from them? MOST SIGNIFICANT CHANGES Q12. What is the best (most significant) change in your child/ren’s (quality of) life/well-being since you have been involved with the CBO program? Probes: What happened? When did it happen? Why is this the most significant change to you? Why do you think the change happened? Q13. What is the best (most significant) change in your community since the existence of the CBO program for vulnerable children? Probes: What happened? When did it happen? Why is this the most significant change to you? Why do you think the change happened? CLOSING COMMENTS Is there anything else you would like to share with us? Is there anything you would like to ask us? Hands at Work UGM End of Project Evaluation 61 Instrument 8: Youth (16–20 Years) –Focus Group Discussion INTRODUCTION I am ……………………………… and ……………………….. from Health and Development Africa/Results in Health. We are working to find out more about how Hands at Work and the CBO (use name) have improved the well-being of children, youth and communities through the care and support services offered. The aim of this focus group discussion/interview is to find out more about your thoughts, perspectives and feelings about the work done by the CBO in your community. There is no right or wrong answer. We are more interested in hearing about your experiences and how they have made a difference in your life. This focus group discussion/interview is part of an evaluation of Hands at Work. This means that we are looking back at the work of Hands over the past 5 years and finding out as much as we can (knowledge generation) about the successes and challenges Hands experienced in providing care and support to vulnerable children through the support of CBOs and other programs. This evaluation is not to provide more funding or take away funding. It is to help improve the care and support services provided to vulnerable children through this program and to other programs. We will be using this recorder to record the discussion. This is just going to help us to go back after the discussion and hear exactly what you said. If you agree to our discussion being recorded and to form part of the discussion, and to take some pictures of you during the evaluation process, please sign the consent form and hand it back to us. Please sign the attendance register for our records; we will not use your name when writing the report. If at any time you feel that there is a question you are not comfortable answering or that you would not like to continue being part of the discussion, please also feel free to let us know. We will be speaking to you for about an hour. Participatory tools for facilitators to use to probe and encourage respondents A. Use cards to draw a happy/smiley face, neutral face and sad face, and use the cards to probe how respondents felt about the services/events/activities. Use the cards to ask questions regarding impact and outcomes, perception of quality of the service provided and to learn about changes that happened in their life (for MSC) and capacity development. E.g. What make you happy/smile – related to event/activity you are involved? – Facilitator shows the smiling face while asking the question and then probe further. E.g. What make you sad/crying – related to event/activity you are involved? – Facilitator shows the sad face while asking the question and then probe further. B. Draw a timeline on a chart/card to compare before and after respondents joined the programs – and facilitate how respondents felt before their involvement and during/after their involvement in the program    Hands at Work UGM End of Project Evaluation 62 Biographical Information (researcher to complete this section) 1 Date of interview 2 Time interview began 3 Time interview ended 4 Time in minutes 5 Province 6 Area/Town/Community 7 Interviewer name 8 Note taker name 9 Translator Name 10 Number in FGD 11 Gender of focus group respondents (how many males; how many females?) Male Female 1 12 Interviewer additional observations and remarks SERVICE DELIVERY TO YOUTH Q1. What do you know about Hands at Work? Please tell us a bit about your experience with Hands. What kind of support have you received from Hands? Q2. What do you know about the CBO? How did you find out about the CBO’s program with vulnerable children? Please tell us a bit about your experience/relationship with the CBO. Q3. How were you identified to become part of the program? What kind of support have you received from the CBO? Q4. What kind of services have you received from the CBO? For each category, please give us details.  Access to education?  Access to social grants and other legal documentation?  Access to healthcare/services when you need them? Probe: When respondents list services, ask for details about each service. Q5. Does the CBO support anyone else in your family? (siblings, parents, grandparents, primary caregivers) Hands at Work UGM End of Project Evaluation 63 Q6. Has the CBO provided you with the opportunity to talk about any personal problems you may have experienced? With whom? What kind of support was provided to you to address your problems? Probe: Allow respondents to share their personal stories IMPACTS AND OUTCOMES OF YOUTH CAMPS Q7. Which Hands at Work camps have you been involved with? How many? How often? Q8. What did you learn from them? Q9. What other activities were you involved with at the camps? What did you enjoy about these activities? Why? Q10. How have the camps impacted on or changed your life? Please describe. Q11. Did you experience any challenges while you were at the camps? Tell us more. PERCEPTION OF QUALITY AND EASE OF ACCESS TO SERVICES BY OTHER STAKEHOLDERS OR COMMUNITY MEMBERS Q12. Apart from Hands and CBO staff, are there any other organizations or people that assisted you in accessing services? Please describe this for us. Probe: ask about other members in the community that help and support children (neighbors, teachers, friends etc) Q13. What kind of support do you receive from them? MOST SIGNIFICANT CHANGE Q14. What is the best (most significant) change in your (quality of) life/well-being since you have been involved with the Hands/CBO program and/or participated in the Youth camps? Probes: What happened? When did it happen? Why is this the most significant change to you? Why do you think the change happened? Q15. What is the best (most significant) change in your community since the existence of the Hands/CBO program for vulnerable children? Probes: What happened? When did it happen? Why is this the most significant change to you? Why do you think the change happened? CLOSING COMMENTS Is there anything else you would like to share with us? Is there anything you would like to ask us? Hands at Work UGM End of Project Evaluation 64 Instrument 9: Young Mums (Over 16 Years) – Focus Group Discussion INTRODUCTION I am ……………………………… and ……………………….. from Health and Development Africa/Results in Health. We are working to find out more about how Hands at Work and the CBO (use name) have improved the well-being of children, youth and communities through the care and support services offered. The aim of this focus group discussion/interview is to find out more about your thoughts, perspectives and feelings about the work done by the CBO in your community. There is no right or wrong answer. We are more interested in hearing about your experiences and how they have made a difference in your life. This focus group discussion/interview is part of an evaluation of Hands at Work. This means that we are looking back at the work of Hands over the past 5 years and finding out as much as we can (knowledge generation) about the successes and challenges Hands experienced in providing care and support to vulnerable children through the support of CBOs and other programs. This evaluation is not to provide more funding or take away funding. It is to help improve the care and support services provided to vulnerable children through this program and to other programs. We will be using this recorder to record the discussion. This is just going to help us to go back after the discussion and hear exactly what you said. If you agree to our discussion being recorded and to form part of the discussion, and to take some pictures of you during the evaluation process, please sign the consent form and hand it back to us. Please sign the attendance register for our records; we will not use your name when writing the report. If at any time you feel that there is a question you are not comfortable answering or that you would not like to continue being part of the discussion, please also feel free to let us know. We will be speaking to you for about an hour. Participatory tools for facilitators to use to probe and encourage respondents A. Use cards to draw a happy/smiley face, neutral face and sad face, and use the cards to probe how respondents felt about the services/events/activities. Use the cards to ask questions regarding impact and outcomes, perception of quality of the service provided and to learn about changes that happened in their life (for MSC) and capacity development. E.g. What make you happy/smile – related to event/activity you are involved? – Facilitator shows the smiling face while asking the question and then probe further. E.g. What make you sad/crying – related to event/activity you are involved? – Facilitator shows the sad face while asking the question and then probe further. B. Draw a timeline on a chart/card to compare before and after respondents joined the programs – and facilitate how respondents felt before their involvement and during/after their involvement in the program    Hands at Work UGM End of Project Evaluation 65 Biographical Information (researcher to complete this section) 1 Date of interview 2 Time interview began 3 Time interview ended 4 Time in minutes 5 Province 6 Area/Town/Community 7 Interviewer name 8 Note taker name 9 Translator Name 10 Number in FGD 11 Interviewer additional observations and remarks SERVICE DELIVERY TO YOUNG MUMS Q1. What do you know about Hands at Work? Please tell us a bit about your experience with Hands. What kind of support have you received from Hands? Q2. Is there a CBO that offers you support? What do you know about the CBO? Q3. How did you find out about the program for Young Mums? Please tell us a bit about this. Q4. How were you identified to become part of the program? Probe: What are the conditions to be part of the program? Q5. Are you still a part of the program? If not, why not. Q6. Please tell us about the program for Young Mums. Probe: what is the content? Is there training? How often do they meet? Is the support continuous? Q7. What kind of support and services have you received FOR YOURSELF from the CBO/Hands/Young Mums program? For each category, please give us details.  Access to education?  Access to social grants and other legal documentation? Hands at Work UGM End of Project Evaluation 66  Access to healthcare/services when you need them? Probe: When respondents list services, ask for details about each service. Q8. What kind of support and services have you received FOR YOUR CHILD from the CBO/Hands/Young Mums program?  Access to education?  Access to social grants and other legal documentation?  Access to healthcare/services when you need them? Probe: When respondents list services, ask for details about each service. Q9. Does the CBO/Hands support anyone else in your family? (siblings, parents, grandparents, primary caregivers) Q10. Has the CBO/Hands provided you with the opportunity to talk about any personal problems you may have experienced? With whom? What kind of support was provided to you to address your problems? Probe: Allow respondents to share their personal stories PERCEPTION OF QUALITY AND EASE OF ACCESS TO SERVICES BY OTHER STAKEHOLDERS OR COMMUNITY MEMBERS Q11. Apart from Hands and CBO staff, are there any other organizations or people that assist you in accessing services for yourself or your child? Probe: ask about other members in the community that help and support children (neighbors, teachers, friends etc) Q12. What kind of support do you receive from them? Please describe this for us. MOST SIGNIFICANT CHANGE Q13. What is the best (most significant) change in your (quality of) life/well-being since you have been involved with the Young Mums program? Probes: What happened? When did it happen? Why is this the most significant change to you? Why do you think the change happened? Q14. What is the best (most significant) change in your community since the existence of the Young Mums program? Probes: What happened? When did it happen? Why is this the most significant change to you? Why do you think the change happened? CLOSING COMMENTS Is there anything else you would like to share with us? Is there anything you would like to ask us? Hands at Work UGM End of Project Evaluation 67 Instrument 10. Most Significant Change Tool INTRODUCTION I am Nur Hidayati from Results in Health. We are a consulting company working with Health and Development Africa, to conduct an evaluation to determine if and to what extent Hands at Work has improved the well-being of vulnerable children in the districts where they operate The aim of this interview is to collect stories through an interview to find out your thoughts, perspectives and feelings about the work done by Hands at Work in your community. There is no right or wrong answer. We are more interested in hearing about your experiences and how they have made a difference in your life. We will be using this recorder to record the discussion. This is just going to help us to go back after the discussion and hear exactly what you said. If you agree to our discussion being recorded and to form part of the discussion, and to take some pictures of you during the evaluation process, please sign the consent form and hand it back to us. Please sign the attendance register for our records; we will not use your name when writing the report. If at any time you feel that there is a question you are not comfortable answering or that you would not like to continue being part of the discussion, please also feel free to let us know. We will be speaking to you for about an hour. Story Collection Form Name of Interviewer Date and time of Interview Identity of Respondent Code …. Age Village Questions 4. What is the best (most significant) change in children’s (quality of) life/well-being since Hands at Work Program has been implemented? a. What happened? b. When did the change happen? c. How did the change happen? d. Why is it the most significant change to you? e. Why do you think the change happened? 5. What is the best (most significant) change in your community since the existence of Hands at Work program for vulnerable Hands at Work UGM End of Project Evaluation 68 children? a. What happened? b. When did the change happen? c. How did the change happen? d. Why is it the most significant change to you? e. Why do you think the change happened? 6. What is the best (most significant) change in your capacity related to organizational and program management? a. What happened? b. When did the change happen? c. How did the change happen? d. Why is it the most significant change to you? e. Why do you think the change happened? Hands at Work UGM End of Project Evaluation 69 Most Significant Change Story Selection Tool Story Number Group Domain Why does this group think the change is important? Organization’s reason for selection Consortium’s reason for selection Hands at Work UGM End of Project Evaluation 70 Informed consent form                 INFORMED CONSENT I,______________________________________ hereby confirm that:  I have been informed about the objectives of the evaluation; and the reason, benefits and risk of my participation in this interview.  I am aware that the information I provide in the interview will be anonymously processed into the evaluation report. In view of the requirements of the evaluation, I agree that the data collected during this evaluation can be processed by the evaluator.  I may, at any stage, without prejudice, withdraw my consent and participation in the evaluation.  I have had sufficient opportunity to ask questions and (of my own free will) declare myself prepared to participate in the interview.  In view of the requirements of evaluation, I agree that: o a tape recorder can be used for the purpose of data collection and analysis during this evaluation o That all of the above has been explained to me in a language that I am comfortable with Full name of Participants: Signature / Thumbprint: Date: _________________ Full Name of the Interviewer: Signature Date:_________________ For the interviewer Please specify: This Informed Consent belongs to a Participant of Focus Group / In- depth Interview. Target Group: _____________________________ Hands at Work UGM End of Project Evaluation 71 Attendance Register Forms for Focus Groups ATTENDANCE REGISTER FOR FOCUS GROUP WITH PRIMARY CAREGIVERS # First name Surname Age in years Gender (male or female) Educational level Relationship to child (mother, father, aunt, etc) # of children in the Hands at Work/CBO programs 1 2 3 4 5 6 7 8 9 10 11 12 13 Hands at Work UGM End of Project Evaluation 72 ATTENDANCE REGISTER FOR FOCUS GROUP WITH HANDS AT WORK YOUTH/YOUNG MUMS # First name Surname Age in years Gender (male or female) Educational level Hands at Work programs attended (HIV prevention camps, Young Mums programs, etc) 1 2 3 4 5 6 7 8 9 10 11    Hands at Work UGM End of Project Evaluation 73 ATTENDANCE REGISTER FOR FOCUS CAREWORKERS # First name Surname Age in years Gender (male or female) Educational level 1 2 3 4 5 6 7 8 9 10 11 12 13 Hands at Work UGM End of Project Evaluation 74 APPENDIX 4: LIST OF SITES AND DATES VISITED List of Sites Visited and Dates of Visits Date Area Site April 16, 2012 White River Hands at Work Hub April 17, 2012 Clau Clau Senzokuhle HBC April 18, 2012 White River Hands Hub April 19, 2012 Belfast Belfast Hbc Masoyi Masoyi HBC April 20, 2012 Welverdiend Pfunani HBC White River Hands at Work Hub May 9, 2012 Clau Clau Joy Hbc May 10, 2012 Cork Ndzalama Hbc May 11, 2012 White River Hands at Work Hub (MSC And Young Mums) Hands at Work UGM End of Project Evaluation 75 APPENDIX 5: PERSONS CONTACTED Hands at Work Leadership George Snyman, Chief Executive Officer Lynn Chotowetz, Operations Director and Program Manager—PEPFAR Hands at Work RST members Wedzerai Chiyoka, Program Manager–PEPFAR Busisiwe Sityata Jones, Service Center Manager—Clau Clau Simon Mgwenya, Service Center Manager— Bushbuckridge Vusi Mabuza, PCG Trainer Audrey Sibiya, PCG Trainer Sindiswe Lubisi, Monitoring and Evaluation Officer Emily, Capacity Building, Walking with Wounded Children Service Center Leadership Fortunate Kunene, Field Coordinator— Clau Clau Sindiswe Lubisi, Finance Officer— Clau Clau Audrey Sibiya, Service Center Coordinator, Bushbuckridge Tinyiko Mabunda, Field Coordinator, Bushbuckridge Eulanda Mathebula Field Coordinator, Bushbuckridge CBO Leadership Sarah Shabangu, Coordinator, Senzokuhle Dudu Vilakazi, ECD Coordinator, Senzokuhle Sibongile Nkosi, Community care worker, Senzokuhle Pumzile Singwane, Data capturer, Senzokuhle Phumzile Shabangu, Project manager, Senzokuhle Thembi Nyathi, Coordinator, Pfunani Gladys Masayo, HBC Coordinator, Belfast Doris Mtilabini, Treasurer, Belfast Fortunate Kunene, Former Project Manager, Joy James Ngwenya, Current Project Manager, Joy Agnes Nkuna, ECD Teacher, Ndzalama Pinky Nyathi, Administrator, Ndzalama Florah Khumalo, Coordinator, Ndzalama Hands at Work UGM End of Project Evaluation 76 University Students Elvis Mahlanya Fortunate Maile Hands at Work UGM End of Project Evaluation 77 APPENDIX 6: REFERENCES  AED and Centers for Disease Control and Prevention. HIV Prevention Capacity Building: A Framework for Strengthening And Sustaining HIV Prevention Program. 2005. Viewed on July 14, 2012 at http://coach.fhi360.org/Libraries/Capacity_Building/HIV_Prevention_Capacity_Building.Sflb.Ashx  Department of Social Development, South Africa. Policy Framework for Orphans and other children made vulnerable by HIV and AIDS South Africa. Pretoria, South Africa: July 2005. Viewed on June 6, 2012 at http://www.cindi.org.za/files/Policy-Framework-for-OVC-Final.pdf  Ehlanzeni District Municipality Integrated Development Plan, 2008–2009. Viewed on June 25, 2012 at http://cgta.mpg.gov.za/documents/idp_ehdm.pdf  Federal Democratic Republic Of Ethiopia. Standard Service Delivery Guidelines for Orphans and Vulnerable Children’s Care and Support Programs. 2010. Viewed on July 15, 2012 at http://Www.Hciproject.Org/Sites/Default/Files/Collaboratives/Ethiopia%20OVC%20Quality%20Standa rds_Feb2010.Pdf  FHI 360. “An overview of FHI 360’s Umbrella Grants Management (UGM) Project.” Inception Workshop, FHI 360 Office, Pretoria, March 2012.  Hands at Work in Africa. AED Project Presentation. December 12, 2011.  Hands at Work Annual Reports  Hands at Work Country Operational Plans for COP 11 (Project Partner Plan), COP 10 (Profile 01- 09-10), and COP 9 (Profile 01-09-09).  Hands at Work Project Presentation. December 12, 2011.  Hands at Work website. Viewed on May 29, 2012 at http://www.handsatwork.org/vision-mission-values/  Hands at Work. Quarterly Reports 2–4. 2010.  Hands at Work. Quarterly Reports 1–4. 2011.  NSW Health Department. A Framework for Building Capacity to Improve Health. 2001. Viewed on July 15, 2012 at http://Www.Health.Nsw.Gov.Au/Pubs/2001/Pdf/Framework_Improve.Pdf  Pursell, R. and Mangxaba, J.W. A Case Study : Hands @Work Masoyi Home-Based Care OVC Program. Khulisa Management Services, July 2008. Hands at Work UGM End of Project Evaluation 78 APPENDIX 7: SCOPE OF WORK Now in the fifth and final year of the project, FHI 360-UGM, at the request of USAID, is commissioning an external evaluation of our grantees. Partner organizations are non-governmental organizations (NGOs) working at national, provincial and local levels in South Africa, primarily implementing services related to services for orphans and vulnerable children (OVC), HIV care and support, HIV counseling and testing, and HIV prevention. These partners have received funding for a period of three to five years under PEFPAR, as well as both organizational and technical capacity building support. RIH/HDA is being contracted to execute evaluations for two of the UGM Partners: GRIP and Hands at Work. The focus of the each partner evaluation will be to:  Determine whether the program objectives under each partner’s program were achieved  Evaluate the key program outcomes and impacts related to improved health and well-being of the targeted beneficiaries Most specifically, RIH/HDA will seek to answer the following key evaluation questions for Hands at Work (HAW), utilizing tools, methods, and sub-questions approved reviewed and approved by FHI 360:  To what extent did HAW succeed in enhancing their CBO sub-partner’s capacity to identify and successfully respond to the needs of OVC?  What were the most significant changes brought about by HAW and its CBO sub-partner’s in improving the well-being of OVC in targeted communities?  How do stakeholders (children, care givers, DoSD, community representatives) perceive the program; in terms of quality and ease of access? The focus of the evaluation is to assess effectiveness of the partner organizations in addressing the needs of beneficiaries in targeted communities. The evaluators will be required to carefully consider the suitability and feasibility of design options that are likely to offer the best chance of establishing the value of the program in responding to the needs of targeted beneficiaries and communities. Both qualitative and quantitative data collection techniques should be employed. Data will be collected from various sources using appropriate data collection methods and tools for any given evaluation question. The final design to be employed will be determined after the contractor has had a chance to undertake a front-end analysis and is therefore able to select the best design option that specifies; which people or units will be studied; how they will be selected and the kinds of comparison that should be made. Data will be collected from various program sites for each partner. Evaluations will be undertaken in two stages and with expected outcomes for each stage as expressed below: Stage 1: Finalization of Evaluation Protocol Contractor will refine an evaluation protocol which demonstrates:  Understanding the relationship between program stages and the proposed broad evaluation question  Understanding the context for program delivery and key factors that influence program implementation Hands at Work UGM End of Project Evaluation 79  Understanding the existing theoretical and empirical knowledge about the program and examining program theory  A comprehensive stakeholder analysis and determination of roles of key stakeholders in the evaluation  Balancing costs and benefits of the evaluation and advising on the most strategic questions to include in the evaluation  Developing the Finalized Implementation Strategy and Methodology Report Stage 2: Implementation of the Evaluation Contractor will implement the partner evaluation following submission and approval of the Implementation Strategy and Methodology Report:  Pre-test instruments  Train data collectors  Undertake the evaluation data gathering process  Prepare data for analysis  Clean data  Enter data into electronic data analysis systems  Undertake comprehensive data analysis  Formulate the findings During the period of performance of April 9, 2012 –July 20, 2012, payment to RIH/HAD will be fixed price based on the payment schedule determined by the deliverables below: MILESTONE DELIVERABLE DUE DATE Data collection (April 9 – May 11) 1. Finalized Implementation Strategy and Methodology Report submitted to FHI 360 (Hands at Work) April 13 2. Evaluation Workplan, including key activities and timeframes submitted to FHI 360 (Hands at Work) April 13 3. Data Analysis Plan, including dummy table/graphs for presenting data submitted to FHI 360 (Hands at Work) May 4 Data analysis and development of PowerPoint Presentation, including summary of evaluation process and results (May 14 – June 1) 1. Oral and PowerPoint Presentation (half- day)of preliminary findings to USAID, FHI 360, and partner (May 28- June 1, 2012) (Hands at Work) Presentations completed by June 1 Development of final 2. Draft written report submitted to FHI 360 (Hands at Work) June 15 Hands at Work UGM End of Project Evaluation 80 written report, including an executive summary with highlights of the evaluation and key findings (June 1 – July 20) 3. Final report submitted to FHI 360 (Hands at Work) July 20 Development of brief paper (two-pager) for each partner, targeting community audiences on key findings from the evaluation (June 1 – July 20) 4. Two-page papers submitted to FHI 360 (Hands at Work) July 20