USAID/South Africa Umbrella Grants Management Project End of Project Partner Evaluation HEARTBEAT CENTRE FOR COMMUNITY DEVELOPMENT June 2012 This publication was produced for the United States Agency for International Development. It was prepared by Elizabeth Zishiri of Feedback Research and Analytics. USAID/South Africa Umbrella Grants Management Project End of Project Partner Evaluation HEARTBEAT CENTRE FOR COMMUNITY DEVELOPMENT This report is made possible by the generous support of the American people through the U. S. Agency for International Development (USAID), under the terms of APS 674-07-011, through United States Government’s President’s Emergency Plan for AIDS Relief, operated by FHI 360. The contents and opinions expressed herein do not necessarily reflect the views of USAID or the United States Government. Heartbeat UGM End of Project Evaluation 1 ACKNOWLEDGEMENTS  I would like to extend sincere gratitude to Heartbeat for assisting us in setting up interactions with the beneficiaries and for the input they provided over the duration of the project.  I would also like to thank the research team Fazeela Hoosen, Lufuno Maitakhole, Lindiwe Mdhuli, and Mike Mashiyane for all their hard work during the data collection process.  A special thank you to Donna Podems, Rebekah King, Terence Beney, Fazeela Hoosen, Mokete Mokena, Daleen Botha, and Marlene Sinclair for your guidance, clarifications, and assistance in producing this report.  To the respondents, Heartbeat staff, community stakeholders, primary caregivers, and children, this report would not have been possible without your valuable contributions. TABLE OF CONTENTS I. INTRODUCTION ....................................................................................... 10 II. BACKGROUND .......................................................................................... 12 III. METHODS ................................................................................................... 23 IV. FINDINGS .................................................................................................... 27 V. CONCLUSIONS ......................................................................................... 43 VI. RECOMMENDATIONS ............................................................................. 45 APPENDIX I: COMPOSITION OF THE EVALUATION TEAM .................... 47 APPENDIX II: DATA COLLECTION TOOLS USED ...................................... 48 APPENDIX III: LIST OF SITES VISITED AND DATES OF VISITS .............. 60 APPENDIX IV: SCOPE OF WORK .................................................................... 61 APPENDIX V: REFERENCES ............................................................................. 63 Heartbeat UGM End of Project Evaluation 3 ACRONYM LIST ARV Antiretroviral ASC Afterschool Center ASW Auxiliary Social Worker CCW Child Care Worker CDF Community Development Facilitator DSD Department of Social Development ECD Early Childhood Development HES Household Economic Strengthening ID Identity Document M&E Monitoring and Evaluation OVC Orphans and Vulnerable Children PEPFAR President’s Emergency Plan for AIDS Relief RFP Request for Proposal SAO Site Administration Officer SASSA South African Social Security Agency SW Social Worker UGM Umbrella Grants Management Project USAID United States Agency for International Development EXECUTIVE SUMMARY INTRODUCTION UGM is a five year grants management program administered by FHI 360 with funding from the U.S. Agency for International Development (USAID). Through UGM, FHI 360 provides funding and technical assistance to USAID-selected NGO partners who provide HIV/AIDS services at local, provincial, and national levels in South Africa. FHI 360-UGM seeks to promote high quality service delivery in alignment with the priorities and goals of the South African government’s HIV/AIDS framework. FHI 360-UGM provides specialized capacity building and support services to build partners’ skills and competencies in program management, governance, human resource development, budgeting and finance, and monitoring, evaluation, and reporting. The FHI 360-UGM project objectives are to 1. Provide grants to USAID/PEPFAR partners that ensure an adequate resource flow to foster scale-up of activities 2. Implement effective monitoring, evaluation, and reporting systems to assess and document activities 3. Provide ongoing capacity building to support and enhance scale-up of activities, and sustainability of activities and partners Since 2007, FHI 360-UGM has supported thirteen South African NGOs including Heartbeat. The organization has received total funding through the UGM of R14,180,058. This report discusses the processes followed and the findings established in the Umbrella Grants Management (UGM) project end of project evaluation of the Heartbeat Centre for Community Development, conducted by Feedback Research and Analytics (Feedback RA). The purpose of the evaluation is to determine the extent to which Heartbeat’s objectives under the United States Agency for International Development (USAID) President’s Emergency Plan for AIDS Relief (PEPFAR)-funded project related to care and support of orphans and vulnerable children (OVC) have been achieved; identify program outcomes related to the well-being of OVC, families, and their communities; and generate knowledge on the program strategies that are most effective or ineffective in improving the lives of OVC. The evaluation set out to 1. assess the extent to which Heartbeat was able to address the needs of children within their target communities 2. document stakeholder perceptions of program quality and ease of access to program services 3. identify the most significant changes affected by Heartbeat in a. improving the well-being of OVC in targeted communities b. increasing the capacity of families and communities to care for their OVC 4. discuss key enablers of and barriers to meeting project objectives The primary audiences for this evaluation report are USAID, FHI 360, Heartbeat and their Afterschool Center (ASC) staff, and community stakeholders and beneficiaries. Specific information outputs based on this report have been prepared for ASC staff, stakeholders, and beneficiaries. The report consists of a background section, the evaluation methodology, findings arranged by the key evaluation questions, conclusions, and recommendations. BACKGROUND ON HEARTBEAT Heartbeat is a NGO established in 2000 by Dr. Sunette Pienaar-Steyn in response to the increase in the number of orphans resulting from HIV-related deaths in South Africa. The overarching goal of Heartbeat’s activities is captured in its mission statement and is concerned with empowering OVC to Heartbeat UGM End of Project Evaluation 5 ‘reach their full potential’ and achieve such outcomes as being self reliant, finding employment, matriculating, acquiring skills, obtaining a degree or post-matric qualification, becoming entrepreneurial, behaving responsibly, and pursuing goals (Southern Hemisphere Consultants 2011). Heartbeat’s stated goal is supported by the objectives of  increasing provision for the protection of the rights of OVC  increasing community capacity to care for, and protect the rights of, OVC Operating from fourteen centers in local communities across six provinces, Heartbeat offers a comprehensive package of services that address OVCs rights to material provision, education, and physical and psychosocial well-being. Beneficiaries are selected using criteria that stipulate that the OVC should be from a child-headed household, youth-headed household, relative-headed household or be a vulnerable child who is still in school and below the age of 18. Services at site or center level are provided through an ASC and through home visits conducted by child care workers (CCWs). In addition to PEPFAR-funded programming, Heartbeat also provides material support (meals and food parcels, school uniforms, stationery, and clothing), which are not funded by PEPFAR. Through PEPFAR funding, Heartbeat strengthened a number of program areas including healthcare referrals and HIV prevention education. METHODS The evaluation was conducted using a mixed methods approach that relied primarily on an analysis of qualitative data obtained from headquarters and center staff including CCWs, community stakeholders, and beneficiaries including caregivers and OVC. It was supported by both quantitative and qualitative analyses of secondary data submitted by Heartbeat, including monitoring data and annual reports. The sampling of centers for inclusion in the evaluation was driven by an emphasis on cases that revealed the factors and conditions influencing program efficacy. The sampling criteria applied in selection were  provincial distribution  center performance  degree of urbanization  services offered  stage of project implementation Four centers, out of fourteen, were included based on the sampling criteria. In order to ensure that the evaluation was conducted in alignment with ethical standards, the following principles were adopted  age of participation for children was restricted to the age range in which children can legally provide informed consent  evaluation team followed Heartbeat’s protocols and allowed them to select the children to participate  informed consent was obtained from children; however, the approval of their guardian was also required for them to participate  evaluation did not require the collection of potentially sensitive data from children; however, provision would be made for dealing with sensitive matters should they arise FINDINGS The presentation of findings is guided by the key evaluation questions that were arrived at collaboratively with input from FHI 360 and Heartbeat. 1. To what extent was Heartbeat able to address the needs of children within the community? The consensus among stakeholders is that Heartbeat programs respond directly to their needs, a finding confirmed by the alignment of the intervention model and programming with reported needs. Services were valued differently across different stakeholder groups.  Material support emerged overwhelmingly as the most valued service among beneficiaries, as demonstrated in the responses of OVC and their caregivers. The responses of center staff and CCWs confirmed this finding.  In addition to material support, caregivers included facilitating access to mandatory documents and registration for accessing social grants as a most valued service.  Educational support is highly valued by OVC and improved school performance is the effect most frequently observed by OVC, caregivers, center staff, and CCWs.  Psychosocial support allowed for addressing emotional vulnerabilities and led to an improvement in life skills, according to OVC. Caregivers, center staff, and community stakeholders also noted the value of life skills for OVC. However, factors undermining the effectiveness of program implementation in the three key areas of education, psychosocial and material support were also observed.  CCWs are often not equipped to provide homework assistance, especially for children in higher grades, and the inconsistent implementation of activities such as the payment of crèche fees for early childhood development (ECD) and the extra lessons undermined both outcomes and relationships with beneficiaries.  Gains made by OVC in terms of their psychological well-being are vulnerable to deleterious conditions in the household and community. Programming to strengthen the capacity of households and the community to provide psychosocial support for OVC is absent, despite strengthening the capacity of households and community to take care of OVC being an explicit objective of Heartbeat’s interventions.  Material support, including the direct provision of food and other resources that Heartbeat provides through funding other than PEPFAR, emerged as critical service delivery areas. Inconsistent delivery or withdrawal of this service negatively affected uptake of and participation in other Heartbeat programs. The only need that consistently emerged as not being addressed by Heartbeat was providing for the tertiary education, skills development, and employment opportunities of OVC. This is a key concern of all stakeholders, who demonstrate an awareness of the limited prospects of OVC and their increased vulnerability once they age out of social protection and Heartbeat programming. Support for tertiary education, an activity that was planned to be introduced on 2012, was not implemented. 1.1. How do stakeholders (children, primary caregivers, care workers, and community representatives) perceive the program in terms of quality and ease of access? Quality and ease of access were assessed in terms of visibility, affordability, proximity, and age￾appropriateness or child-centeredness.  Heartbeat centers benefit from high visibility in communities due to efforts at engagement, especially during the establishment phase of a center in a community, and a deliberate communication strategy.  Services are offered at no cost to the beneficiary; however, affordability is influenced by the proximity of beneficiaries to Heartbeat centers. Geographically dispersed communities are serviced by single sites and transport to access those services becomes prohibitive for a substantial proportion of beneficiaries. Cost of transport also detrimentally affects the delivery of outreach services, with transport costs being a key concern mentioned by CCWs at every site in the sample.  Centers are not optimally resourced; there is limited outdoor space, and the range of possible outdoor activities is severely limited. Even if playground equipment were provided, it is unlikely that three of the four sites would be able to utilize it. Heartbeat UGM End of Project Evaluation 7  Indoor space is also limited, with the result that psychosocial activities and counseling services are not provided with dedicated space.  Center staff and CCWs observed that centers were not adequately equipped to care for children younger than 8 years of age, both in terms of general material resources such as age￾appropriate books and arts and crafts materials, and the lack of staff expertise in working with children younger than 8. 2. What were the most significant changes brought about by Heartbeat in improving the well-being of OVC in targeted communities? Significant changes were observed at OVC, family, and community levels. There was substantial consensus across all stakeholder groups regarding the most significant changes observed in the lives of OVC.  Improved performance of OVC at school emerged as the most significant change, according to observations made by all stakeholder groups.  The change of OVC’s circumstances due to the facilitation of access to resources, either through direct material support or entry to the social protection system, was similarly emphasized by stakeholders. Assistance with obtaining documentation for social protection was notably less of an emphasis in Vosloorus, where it would appear social protection services are easily accessed.  In addition, changes in the psychological, and to a lesser extent the physical, well-being of OVC were consistently noted.  For Heartbeat staff, progress after completing school represents the most significant change. Admission to tertiary education institutions and gainful employment indicate the fulfillment of Heartbeat’s mission to empower OVC to reach their potential. The limited achievement in this area is attributable to a programming gap. There are no consistent mechanisms for achieving this outcome, while the support for tertiary education activity scheduled for 2012 has not been implemented. Observations of significant change at family and community level were not as readily volunteered by respondents. Nevertheless the evidence allows for the following findings to be presented  For families, the most significant change is easing the burden of care through direct material support and through access to the social protection system facilitated by Heartbeat.  The most significant change at community level is the establishment of the Heartbeat centers, which offer a mechanism through which the community can deliberately care for its OVC, through referrals, donations, volunteer activities, and general support. 3. What were the key enablers of and barriers to meeting project objectives? The key enablers to achieving project objectives were  eliminating obstructions to access, specifically the cost of services and the stigma associated with accessing OVC services  embedding Heartbeat and its services in target communities by early and continued community engagement, and the hiring of local community members as center staff (although these activities were not always optimally executed)  mobilizing resources for effective program implementation, through training of center staff and providing organizational development support to centers (although these activities were not always optimally executed)  enhancing the efficacy of Heartbeat services through networking with and referrals to other organizations and services The barriers undermining program effectiveness are either inherent to the context in which the programs are being implemented, have emerged from the manner in which the programs are being implemented, or are due to Heartbeat’s organizational structure and processes.  The most important contextual barriers undermining program efficacy are the wide geographic dispersion of communities in which Heartbeat centers are located, which results in high and at times prohibitive costs for beneficiaries to access services, as well as for CCWs to deliver services through home visits.  The most important programmatic barriers undermining program efficacy are insufficient material and infrastructural resources at centers, and inconsistent implementation of program activities. This includes irregular material support, especially with regards to food provision; the discontinuation of ECD support through payment of crèche fees; the delayed implementation of the extra lessons program; and the weak implementation of activities under Household Economic Strengthening.  The crucial organizational barriers to program efficacy are the inefficiencies observed in the flat management structure at center level, which results in uncertain reporting lines and accountability, and opaque processes linking headquarters to center operations, with the result that communication on program delays and discontinuations from headquarters are inadequately communicated to centers. The reasons for inconsistencies in program implementation require additional clarification, but based on the current organizational arrangements, are the responsibility of Heartbeat headquarters. CONCLUSION Heartbeat was able to respond to the needs of the OVC in terms of providing education and psychosocial support, as well as access to social protection services, the relevance of each endorsed by stakeholders. It is also clear that each program area is highly valued by stakeholders, but that intervention in terms of material resources is the most critical need confronting OVC and a major challenge faced by households and communities. While direct material support is not a sustainable solution to resolving the deficits in provision for the physical needs of OVC, it remains a critical element of the service package offered by Heartbeat. To adequately meet the material support needs sustainably, this programming area must be augmented with household economic strengthening (HES) interventions. HES also addresses the most important programming gap identified by stakeholders, which was improving the future prospects of OVC as they age out of social protection and Heartbeat by providing them with tertiary education, skills development, and employment opportunities. With the inclusion of the planned HES activities, Heartbeat’s intervention model appears to be comprehensive. The value of five years of implementation as a test of the model has not been realized, for two reasons. Some activities appear to have been sub-optimally implemented on a consistent basis. For example, the homework support provided by CCWs appears to have been inadequate for older children especially, due to the CCWs’ limited familiarity with the syllabus content. The second is the lack of program fidelity, for example, the interrupted implementation of material support activities and the delayed implementation of HES interventions. Lapses in program fidelity are of particular concern, imply setbacks in partner performance, and can only be satisfactorily explained by Heartbeat headquarters staff. Despite these performance issues, Heartbeat interventions unquestionably effected significant change in the lives of OVC, facilitating access to social protection and thus alleviating the severe deficiencies in material provision to some extent; contributing to improved school performance; providing a means to address their substantial emotional health needs, and consequently improving the character of their relationships with others; and in some cases supporting children to embark on a path that may allow them to achieve their full potential. RECOMMENDATIONS Organizational Structures and Processes  Review causes of under-performance at Heartbeat headquarters that undermine consistent program implementation Heartbeat UGM End of Project Evaluation 9  Improve communication from Heartbeat headquarters to the centers, specifically with regards to program implementation decisions. Center staff feels excluded from the decision making process concerning operations and unprepared to manage the impact of decisions on beneficiary relationships.  Reform the organizational structures and processes at centers to ensure accountability for performance at center level. The flat management structure is not supporting operational efficacy, nor does it appear to be enhancing staff relations at centers.  Reform organizational structures and processes between headquarters and centers to resolve the inefficiencies introduced by multiple reporting lines. While engagement on matters that support implementation can continue to include multiple people, accountability needs to be streamlined with a single channel of accountability and priority communication to someone in a center manager role. Program design  Strengthen the consistency of food provision and material support activities. The critical need addressed and the incentive it represents make the strengthening and stabilizing of this program activity imperative for Heartbeat.  Address the causes of implementation delays and strengthen capacity to implement HES activities. These activities address the lack of sustainability inherent in material support, as well as the crucial programming gap—improving the future prospects of OVC that age out of social protection and Heartbeat.  Introduce program activities that directly target families. In order to sustain the gains realized with OVC, their systemic context needs to be addressed. It became apparent during the evaluation that the need for psychosocial support for the OVC’s household members—who are also affected by HIV/AIDS— is urgently required.  Introduce program activities that promote community involvement in Heartbeat centers, to maximize improvement in capacity at the community level.  Review the selection criteria and adjust to accommodate the realities of child vulnerability. In particular, the aging out of OVC from social protection and Heartbeat needs to be done more smoothly and include vulnerable children whose parents are present in the household.  Enhance center capacity to cater to children less than 8 years of age. Implementation strategies  Introduce a mechanism for communicating significant changes to services and programming with beneficiaries and stakeholders.  Increase awareness of Heartbeat services. A communication strategy that prioritizes community outreach would improve uptake of services and community capacity to care for its OVC.  Improve accessibility of Heartbeat services by augmenting outreach activities to beneficiary households.  The capacity of CCWs would have to be reviewed and the staff contingent for that function realistically increased in order to effectively meet outreach targets.  A solution for CCW transport that works consistently needs to identified and implemented.      I. INTRODUCTION This report discusses the processes followed and the findings established in the UGM end of project evaluation of the Heartbeat Centre for Community Development, conducted by Feedback RA. PURPOSE OF THE EVALUATION The purpose of the evaluation is to  determine the extent to which Heartbeat’s objectives under the UGM grant to provide care and support for OVC have been achieved  identify program outcomes related to the well-being of OVC, their families, and their communities  generate knowledge on the program strategies that are most effective or ineffective in improving the lives of OVC EVALUATION QUESTIONS In order to fulfill the evaluation purpose, the following evaluation questions were addressed. These questions incorporate key UGM questions from the request for proposals (RFP), as well as additional evaluation questions proposed by Heartbeat. 1. To what extent was Heartbeat able to address the needs of children within the community? a. Is the program meeting the needs of communities and the children as perceived by key stakeholders? b. Which program services were most valuable to OVC, their families, and the community? c. Are there additional areas/services that the key stakeholders would like Heartbeat to provide? 1.1. How do stakeholders (children, primary caregivers, carers, community representatives) perceive the program in terms of ease of access and quality? a. To what extent were services accessible to OVC in terms of cost, location, availability, and appropriateness (child-centeredness)? b. Is there any stigma associated with accessing and using Heartbeat services? 2. What were the most significant changes brought about by Heartbeat in improving the well-being of OVC in targeted communities? (planned or unplanned) a. What were the most significant changes observed within families and in the community? b. How did Heartbeat change the anticipated life path of children through their intervention? 3. What were the key enablers and barriers in meeting project objectives? a. What were the strengths and challenges in Heartbeat’s organizational structure that affected program implementation and the achievement of outcomes? b. To what extent have partnerships enabled or hindered service delivery? c. What were the strengths and weaknesses of Heartbeat’s program design? KEY AUDIENCES The key audiences for this report are USAID, FHI 360, Heartbeat, and future funders or implementers of OVC care and support interventions. In addition, Heartbeat’s ASC staff, who are largely drawn from the local community, as well as community stakeholders and beneficiaries, have an interest in the evaluation findings. For the latter audiences, specific information outputs have been prepared—a summary of the key content of this evaluation report and an accompanying infographic. KEY COMPONENTS OF THE REPORT Heartbeat UGM End of Project Evaluation 11 This report begins with a description of Heartbeat and its PEPFAR funded activities, then details the methodology employed in conducting the evaluation. The findings are arranged by the key evaluation questions, which address the extent to which the needs of the OVC were met, the most significant changes to beneficiaries as a result of Heartbeat interventions, and the key enablers of and barriers to meeting project objectives. The report concludes with a summary analysis and recommendations towards improving the achievement of outcomes in the future. II. BACKGROUND THE UMBRELLA GRANTS MANAGEMENT PROJECT UGM is a five year grants management program administered by FHI 360 with funding from the U.S. Agency for International Development (USAID). Through UGM, FHI 360 provides funding and technical assistance to USAID-selected NGO partners who provide HIV/AIDS services at local, provincial, and national levels in South Africa. FHI 360-UGM seeks to promote high quality service delivery in alignment with the priorities and goals of the South African government’s HIV/AIDS framework. FHI 360-UGM provides specialized capacity building and support services to build partners’ skills and competencies in program management, governance, human resource development, budgeting and finance, and monitoring, evaluation, and reporting. The FHI 360-UGM project objectives are to 1. Provide grants to USAID/PEPFAR partners that ensure an adequate resource flow to foster scale-up of activities 2. Implement effective monitoring, evaluation, and reporting systems to assess and document activities 3. Provide ongoing capacity building to support and enhance scale-up of activities, and sustainability of activities and partners Since 2007, FHI 360-UGM has supported thirteen South African NGOs including Heartbeat. The organization has received total funding through the UGM of R14,180,058. HEARTBEAT CENTRE FOR COMMUNITY DEVELOPMENT History, Vision and Mission Heartbeat is a NGO established in 2000 by Dr. Sunette Pienaar-Steyn in response to the increase in the number of orphans resulting from HIV-related deaths in South Africa. The overarching goal of Heartbeat’s activities is captured in its mission statement and is concerned with empowering OVC to reach their full potential. A child achieving their full potential may manifest in a variety of outcomes across individual OVC, including being self reliant, finding employment, matriculating, acquiring skills, obtaining a degree or post-matric qualification, becoming entrepreneurial, behaving responsibly, and pursuing goals (Southern Hemisphere Consultants 2011). The stated goal is supported by the objectives of  increasing provision for the protection of the rights of OVC  increasing community capacity to care for, and protect the rights of, OVC The spirit informing Heartbeat’s efforts are captured in their vision, mission, and guiding principles as presented in Box 1. Box 1: Heartbeat's Vision, Mission, and Guiding Principles Heartbeat's Vision, Mission, and Guiding Principles Vision: Heartbeat dreams of all children in Africa being happy and having opportunities to fulfill their dreams. Mission: To empower orphaned and vulnerable children to reach their full potential through quality service provision, development, and capacity building. Guiding Principles: Children’s Rights, Community Participation, Sustainable Development, Partnerships The Heartbeat Intervention Model Heartbeat adopts a rights-based model of intervention in alignment with the South African Constitution. Operating from fourteen centers in local communities across six provinces, Heartbeat offers a Heartbeat UGM End of Project Evaluation 13 comprehensive package of services that addresses OVC’s rights to material provision, education, and physical and psychosocial well-being. In so doing, Heartbeat envisages influencing the life path of the OVC towards the realization of positive outcomes, which they would otherwise be prevented from attaining due to their detrimental life circumstances. Heartbeat’s intervention model is constituted by three key components: the package of services offered by Heartbeat, the modalities of program delivery it employs, and the organizational arrangements instituted to facilitate service delivery, especially at center level. The description of the intervention model also indicates the scope of PEPFAR funding and its contribution to the implementation of the comprehensive service package. Identifying the scope of PEPFAR funding is important, in light of evaluation findings demonstrating the interventions most valued by stakeholders and upon which the evidence suggests the efficacy of Heartbeat substantially depends. Table 1: Geographic Reach of Heartbeat Interventions in South Africa Province Location of Centers Gauteng Vosloorus, Mohlakeng, Nellmapius, Atteridgeville, Katlehong, Tembisa, and Khutsong Limpopo Backenburg Free State Botshabelo and Pieter Swartz Mpumalanga Emthonjeni North West Wedela and Kokosi Kwa-Zulu Natal KwaJobe The Heartbeat Package of Services Heartbeat’s package of services consists of programs aligned directly to three of its six organizational objectives. The three program-aligned objectives address specific OVC needs directly, while the remaining three objectives refer to either more generalized outcomes representing the accumulated impact of Heartbeat’s interventions or to organizational performance. Heartbeat’s programs, associated activities, and their alignment to organizational objectives are all reflected in Table 2. Heartbeat UGM End of Project Evaluation 15 Table 2: Heartbeat Objectives, Programs and Activities Objectives Programs Activities OVC Specific Objectives To ensure access to or to directly provide for the physical rights of OVC as described in the South Africa Constitution Access to social protection  Assisting OVC to acquire birth certificates and identity documents (IDs)  Grant application assistance  Referrals to relevant services  Assistance for household to access to indigent programs  Ensuring access to basic healthcare Material support  Providing food parcels to OVC households and ASC meals to OVC  Facilitating the establishment of food gardens  Providing clothing, blankets, and toiletries to OVC To provide for the psychosocial rights of OVC Children’s empowerment  Assisting OVC with household chores  Monitoring adult presence in households through home visits  Providing counseling and support group interventions  Hosting youth camps  Topical workshops on issues including child abuse, HIV prevention  HIV prevention education  Foster care placements  Referrals to relevant services To provide for the intellectual rights of OVC Education  Supporting ECD through paying of crèche fees  Providing homework assistance  Providing academic assistance (extra lessons)  Providing school uniforms and stationery  Facilitating school fees exemptions  Conducting the post￾secondary program to secure tertiary education/employment placements General Objectives To develop a culture of support for OVC within communities  Through following the guiding principles (see Box 1) To ensure quality project management and maximum service impact  Through staff development To improve the quality of life of OVC  Through all the programs/activities Source: Compiled from secondary data submitted by Heartbeat Important to note is that two of Heartbeat’s four core program areas address the physical rights of OVC and include activities that directly address the material deficits that characterize the circumstances of OVC. A number of the activities in these two core programs fall outside of PEPFAR funding parameters. In the package of services, the substantial majority of activities are focused on OVC as beneficiaries, while very few would directly contribute to enhancing the capacity of families and communities to take care of OVC. This observation becomes pertinent when considering limited findings of most significant change at family and community level. In addition, the manner in which OVC are included in or excluded from Heartbeat programs proves important to findings. Beneficiaries are selected using criteria that stipulates that the OVC should be from a child-, youth-, or relative-headed household or should be a potential orphan who is still in school and below the age of 18. Modalities of Delivery All services supported by Heartbeat are delivered either at a local community based facility—frequently refered to as the Heartbeat ASC—or from the center through outreach efforts to households in the community served by the center. The two primary modalities of delivery are 1. The ASC: Activities at the ASC are mainly OVC focused and include providing meals to attending OVC; offering educational activities such as homework support, academic support, and a toy library to ECD-aged OVC; and offering psychosocial support activities such as group and individual counseling, workshops, and other life skills related activities such as drama, dance, and sports. 2. Home Visits: Heartbeat’s CCWs carry out home visits where they assist with homework and household chores, identify OVC needs, and refer children to other services, such as health and social services, as necessary. The two modalities of service delivery are particularly pertinent when reviewing findings on the perceived quality of service delivery (see findings on Key Evaluation Question 1) and the factors that impede or enable program efficacy (see findings on Key Evaluation Question 3). Organizational Arrangements for Service Delivery The third component of the Heartbeat intervention model is the organizational structure, processes, and development efforts. Organizational arrangements should prepare and deploy adequate human and supporting resources to efficiently deliver services and optimize the effectiveness of outcomes. Figure 1 Error! Reference source not found.illustrates the perceived organizational structure as communicated in staff interviews and focus groups, and Heartbeat UGM End of Project Evaluation 17 Table 3 provides a summary of the roles and responsibilities of center staff. Although the Community Development Facilitator (CDF) acts as the center manager to which all other center functions are required to report, s/he is perceived to be at the same management level as the team leader, social worker, Choza, and site administration officer (SAO). Additionally, each staff member at the center level has a manager they report to at Head Office. Interviews with center staff suggest that the combination of dual reporting lines and the perception of a flat management structure at center level blur reporting lines, undermine center-level accountability, and ultimately undermine operational effectiveness. Figure 1: Heartbeat Centre Organizational Structure Source: Adapted from Heartbeat Staff Interviews Child Care Workers/Volunteers Choza (Educationa l support) Team Leader Social Worker /Auxiliary Social Worker (ASW) Site Administration Officer (SAO) Community Development Facilitator (CDF) Education Manager (at HQ) Children’s Empowerment Manager (at HQ) Access Manager (at HQ) Financial Manager (at HQ) Table 3: Heartbeat Staff Roles and Functions at Center Level Designation Role & Function Community Development Facilitator (CDF) Project manager who oversees all projects at site level; liaise with different stakeholders at site level on behalf of Heartbeat. Responsible for access support (application for legal documents, lobbying for school fees exemptions, applying for registration on the indigent’s database). Social Worker (SW) / Auxiliary Social Worker (ASW) Responsible for psychosocial support at the ASC and during home visits (activities include counseling, support groups, workshops, memory work, referrals). The Social Worker is also responsible for intakes/removals of OVC. Team Leader Supervising CCWs, doing spot check for CCWs, conducting fortnightly debriefing meetings with CCWs, referring CCW issues to SW/CDF/Choza/SAO and to relevant departments or NGOs. The Team Leader also assists with cleaning and preparing meals. Site Administration Officer (SAO) Responsible for distributing food parcels and presents Choza Responsible for ASC activities (homework assistance, academic assistance, toy library, drama, dance, peer education, extra lessons), assist with preparing ASC meals and work closely with the Social Worker in support groups and workshops. SCOPE OF HEARTBEAT’S PEPFAR FUNDED ACTIVITIES Alignment of Program Areas with PEPFAR Priorities The scope of Heartbeat’s PEPFAR-funded activities does not deviate significantly from Heartbeat’s typical programming. Heartbeat UGM End of Project Evaluation 19 Table 4 shows the programs supported by PEPFAR and the activities implemented under those programs, matching them to Heartbeat’s programming and intervention model. Two important observations that have bearing on the subsequent evaluation findings are 1. PEPFAR funding does not cover direct material support, a crucial program area for Heartbeat in support of its objective to ensure access to or to directly provide for the physical rights of OVC as described in the SA Constitution. 2. PEPFAR funding parameters prompted the introduction of additional program areas and activities, and the strengthening of others that had enjoyed less emphasis prior to Heartbeat receiving PEPFAR funding. Table 4: Alignment of PEPFAR Programs and Heartbeat Program Areas PEPFAR-Funded Programs Activities of PEPFAR-Funded Programs Heartbeat Program Areas Psychological care Conduct home visits, identify vulnerable children, and conduct therapeutic intervention for OVC Children’s empowerment Educational support Provide ECD school fees, network with ECD schools, monitor school performance, facilitate school fees exemptions, provide extra lessons (English and Math) Education Child protection ID drive, assist with birth registrations, awareness raising events Access to social protection Prevention education Workshops on HIV prevention, peer education Children’s empowerment Capacity building Identify and implement accredited training Staff development Health referrals Referrals, follow-ups, and monitoring adherence Access to social protection Household Economic Strengthening Support for tertiary students, grant application assistance, individual savings Related but not equivalent to material support Source: Adapted from PEPFAR and Heartbeat secondary data Funding Heartbeat is funded by PEPFAR and other funders. Table 5 shows the total donations that Heartbeat received in each financial year and the proportion of the budget attributable to PEPFAR funding. These funding patterns are relevant to subsequent findings in two respects: they have an influence on the determination of sustainability, and they enable Heartbeat to continue to implement programs that fall outside of PEPFAR funding parameters. Table 5: Heartbeat Funding and Proportion Attributable to PEPFAR Period Total Funding PEPFAR Proportion October 2009–September 2010 R 13,306,770.64 42.04% October 2010–September 2011 R 25,665,426.39 23.56% October 2011–March 2012 R 10,147,095.74 20.18% October 2009–March 2012 R 49,119,292.77 27.87% SUMMARY OF OBSERVATIONS RELEVANT TO SUBSEQUENT FINDINGS The exposition of the UGM project and partner background not only offers a general understanding of context vital to the plausible interpretation of evaluation findings, but also serves to draw attention to specific observations that will prove pertinent to findings, conclusions drawn, and subsequent recommendations to be presented in the remainder of the evaluation report. These key observations are summarized here as a conclusion to this section of the evaluation report. One of Heartbeat’s four core program areas, material support, addresses the physical rights of OVC and includes activities that mitigate the material deficits characterizing the circumstances of OVC. However, the activities in this program fall outside of PEPFAR funding parameters (see Heartbeat UGM End of Project Evaluation 21 1. Table 2). With the exception of activities facilitating household access to social protection and the intended HES activities, which the evaluation will show were not implemented as planned, the substantial majority of programming executed by Heartbeat is focused on OVC as beneficiaries (see 2. Table 2). Very few activities would directly contribute to enhancing the capacity of families and communities to take care of OVC. 3. Heartbeat restricts its beneficiary pool by employing selection criteria stipulating that OVC should be from a child-, youth-, or relative-headed household or should be a potential orphan who is still in school and below the age of 18. While these appear to be broad parameters, they exclude vulnerable children residing in households that include their biological parents, which the evaluation later suggests is not appropriate to the context. 4. Heartbeat utilizes two main modalities of service delivery—After School Centers and home visits— each with implications in terms of the perceived quality of service delivery and the factors that bar or enable program efficacy. The evaluation will specifically highlight problems of access to Heartbeat centers and the limitations that inadequate resources impose on home visits. 5. The evaluation shows that the Heartbeat’s organizational structure is characterized by inefficient dual reporting lines at the center level and between center and headquarters, and undermines center-level accountability because of a flat management structure. 6. PEPFAR funding parameters prompted the introduction of additional program areas and activities, and the strengthening of others that had enjoyed less emphasis prior to this funding. The evaluation shows that the results of these donor requirements is both a strengthening of Heartbeat programming and a strain on program performance. 7. Heartbeat receives funding from sources other than PEPFAR, which allows it some flexibility to include activities that fall outside of PEPFAR funding parameters and prompts certain conclusions in the remainder of the evaluation concerning sustainability. Heartbeat UGM End of Project Evaluation 23 III. METHODS EVALUATION DESIGN AND METHODOLOGY The evaluation was conducted using a mixed methods approach that relied primarily on an analysis of qualitative data obtained from staff, community stakeholders, and beneficiaries, supported by both quantitative and qualitative analysis of existing data. Table 6 outlines the evaluation design that was utilized. The design shows the key evaluation questions that were addressed, the key variables under each evaluation question, the data collection and analysis method that was used to respond to each question, and the units used for making comparisons across the findings. Table 6: Evaluation Design Evaluation Design Evaluation Questions Variables Data Collection Method Analysis Method and Procedure Comparisons To what extent was Heartbeat able to address the needs of children within the community? OVC needs Alignment of services with OVC needs Interviews Focus groups Document review Thematic analysis Frequently reported responses by theme Heartbeat sites Participant type Geographical location How do stakeholders perceive the program in terms of quality and ease of access? Accessibility Affordability Child-centeredness Stigma-free environment Interviews Focus groups Thematic analysis Frequently reported responses by theme Heartbeat sites Participant type Geographic location What were the key enablers and barriers in meeting project objectives? Organizational structure Program Model Community buy-in Mentoring, capacity building, and support Partnerships Funding Sources Interviews Focus groups Thematic analysis Frequently reported responses by theme Heartbeat sites Participant type Geographic location What were the most significant changes brought about by Heartbeat in improving the well￾being of OVC in targeted communities? Psychological and emotional well-being Access to legal documents and social grants Educational readiness, access, and performance Health Economic strengthening Interviews Focus groups Document review Thematic analysis Frequently reported responses by theme Heartbeat sites Participant type Geographic location Range of services DATA COLLECTION METHODS The evaluation relied primarily on qualitative data collected in interviews with key informants employed at Heartbeat headquarters, community stakeholders, and Heartbeat center staff, as well as focus group discussions with CCWs, caregivers, and OVC. Structured interview and focus group discussion schedules were developed and employed to conduct data collection activities. The interviews and focus groups were audio recorded for reference and assurance purposes, while detailed interview notes were used as the primary documents for analysis (see Appendix II for interview schedules and informed consent forms for each of the categories engaged). In addition to the interview and focus group data, secondary data from monitoring reports and other statistical information was submitted by Heartbeat in Microsoft Excel or Word formats, and was used to inform sampling and supplement the interview and focus group data. Secondary data also provided corroborating evidence for triangulation on some of the evaluation focus areas. SAMPLING Sampling of Sites Given the evaluation’s focus on knowledge generation for learning to inform future program development and implementation, the sampling of centers for inclusion was driven by an emphasis on rich cases that reveal the factors and conditions influencing program efficacy. The selection of four sites from the sampling frame of fourteen was conducted collaboratively, with inputs from the evaluation team, FHI 360, and Heartbeat. This sample allows for comparisons of outcomes and impact based on relevant site characteristics, such as degree of urbanization, and levels of performance as determined by monitoring data. However, the sampling approach does not allow findings to be generalized for any externally valid evaluative statement on Heartbeat’s performance under the UGM. Instead, the findings reflect on the efficacy of the services provided by Heartbeat under a variety of typical circumstances and provide indicative rather than representative findings on performance. The sampling criteria applied in selection were provincial distribution, center performance, degree of urbanization, services offered, and stage of project implementation. Table 9 provides a summary of the centers (out of 14) that were included based on the sampling criteria. Table 7: Centers included in the evaluation sample Centers in Sample Criteria Site Key Selection Criteria KwaJobe (KwaZulu-Natal) Rural, early implementation Botshabelo (Free State) Peri-urban, located in the Free State (which has multiple sites) Nellmapius (Gauteng) Urban, located in Gauteng (which has multiple sites), challenges in performance Vosloorus (Gauteng) Urban, located in Gauteng (which has multiple sites), strong performance Sampling of Interview and Focus Group Respondents The sampling of key informants, community stakeholders, and beneficiaries was purposive and managed in collaboration with Heartbeat. All program staff and the CCWs at the centers were interviewed, with few exceptions. In each case where particular center staff members were not available, the interview sample was supplemented by alternates. All the additional key informant interviews planned with headquarters staff took place (Founder, Programs Manager, Monitoring and Evaluation (M&E) Manager and three middle managers). All deviations from the proposed sample are shown in Table 6. Two sample deviations require some comment. 1. Including multiple focus groups of the same respondent category in the data set represents better practice; it allows for some smoothing of the dynamics of this technique that skew data. Focus Heartbeat UGM End of Project Evaluation 25 groups are inherently non-comprehensive with respect to the topics under discussion and may emphasize discussion points as a function of the prevailing group dynamics rather than the realities being discussed. Unfortunately, multiple focus groups of primary caregivers and OVC could not be secured at each site as intended. 2. The intended sample of community stakeholders was already limited in the planning. Both the reduced number of community members interviewed, as well as the fact that each contributes a narrowed perspective on community affairs, must be considered a qualification of the community stakeholder data set. Table 6: Data Collection Methods and the Proposed Sample against the Actual Sample Data Collection Methods and Sample Data Collection methods Proposed sample Actual Sample KwaJobe Botshabelo Nellmapius Vosloorus Program staff interviews at site level 4 site staff (1 CDF, 1 Choza, 1 SW or ASW, 1 team leader) As proposed but 2 ASW were interviewed As proposed, additionally SAO was interviewed The team leader was away on training; additionally a SAO was interviewed The Choza was on maternity leave; additionally an M&E officer was interviewed and a SW and ASW were interviewed CCWs focus groups 1 group of 8-10 1 group of 13 1 group of 7 1 group of 7 CCWs and 4 volunteers 2 CCWs (other CCWs were on training) Community stakeholders interviews 4 community stakeholders 1 principal, 1 community leader 2 teachers, 1 municipal officer 1 teacher, 1 NGO, 1 community leader 2 teachers, 1 NGO Primary caregivers focus groups 2 groups of 8-10 1 group of 12 1 group of 14 1 group of 9 2 groups of 10 OVC 12-18 years old focus groups 2 groups of 8-10 1 group of 10 1 group of 12 2 groups of 15 and 16 respectively 2 groups of 10 INFORMED CONSENT Informed consent was obtained from all participants and local legal guidance regarding the participation of children was adhered to. In order to ensure that the evaluation was conducted in alignment with ethical standards, the following principles were adopted  The age of participation for children was restricted to the age range in which children can legally provide informed consent  The evaluation team followed Heartbeat’s protocols and allowed them to select the children to participate  Informed consent was obtained from children; however, the approval of their guardian was also required for them to participate  The evaluation did not require the collection of potentially sensitive data from children and the instruments were prepared accordingly; however, provision would be made for dealing with sensitive matters should they arise and the legal obligation of reporting knowledge of any harm done to children would be strictly adhered to. A consent form detailing the purpose of the evaluation, risks, benefits, and conditions of participation was provided to each participant. The form was explained in English or the local vernacular language as required. After explanation of consent, it was re-emphasized to participants that their participation was voluntary and that they could choose not to participate or stop participating at anytime. Proceeding to data collection with each participant was dependent on their explicit consent. In the case of adult participants, this included the signing of the consent form. In cases where the participant was unable to sign due to illiteracy, a member of the evaluation team or center staff signed on their behalf after verbal consent was given. LIMITATIONS OF THE EVALUATION  The restricted time frame allowed for conducting the evaluation dictated the evaluation design and methods, introducing a number of limitations.  The purposive selection of less than one third of Heartbeat sites allows for documenting learning that might be relevant to all of Heartbeat and other OVC focused programs, but it does not allow general statements on Heartbeat’s performance to be validly made.  Primary data for analysis was collected through key informant interviews and focus groups. Consequently, the data are perception-based which provides certain limitations in terms of the confidence with which it can be used to draw conclusions. For example, while caregivers can easily express an opinion on the program activities they value most, they may not be able to accurately identify those program activities that are most effective in realizing outcomes for OVC.  Observations of change in OVC cannot be disaggregated by age or gender. OVC beneficiary data was collected in mixed focus groups and any attempt to disaggregate responses delivered in a focus group setting is methodologically unsound. Heartbeat UGM End of Project Evaluation 27 IV. FINDINGS The findings of this evaluation will be arranged by the three key evaluation questions, presented according to different stakeholder perspectives (Heartbeat staff, community stakeholders, primary caregivers, and OVC), and compared across sites and site characteristics. KEY EVALUATION QUESTION 1: TO WHAT EXTENT WAS HEARTBEAT ABLE TO MEET THE NEEDS OF ITS TARGET POPULATION(S)? Challenges faced by OVC In order to assess the extent to which Heartbeat met the needs of OVC, different stakeholders (OVC, primary caregivers, and community stakeholders) were asked to state the challenges faced by OVC in their respective communities. Table 7 and Table 8 show the challenges that were reported, per stakeholder and per site, respectively. The order in which the challenges have been presented in the tables represents the frequency with which the challenges were reported, the first challenge being the most frequently reported. While the content of the tables reflect the most persistent themes, they are not exhaustive; additional challenges reported are listed separately. Table 7: Challenges faced by OVC per stakeholder (multiple responses possible) Challenges Faced by OVC, per stakeholder OVC N=73 (Individual OVC) Primary Caregivers N=4 (Focus Groups) Community Stakeholders N=11(Individual members)  Lack of school uniforms and stationery (26%)  Poor school performance (21%)  Lack of food/hunger (18%)  Lack of parental care/support (10%)  Conflict in intergenerational relationships (4%)  Lack of food/hunger (100%)  Conflict in inter-generational relationships (75%)  Difficulties in accessing social grants (50%)  Poor health (50%)  Lack of clothing (50%)  Lack of parental care/support (73%)  Substance abuse (36%)  Low self esteem (27%) Source: Community stakeholder interviews and OVC and primary caregiver focus groups While the OVC perspective reflects immediate felt needs, the primary caregivers report on the severe difficulties they experience in their efforts to provide for the OVC in their care. It is interesting to note that the areas of prioritized need are mirrored across both groups, the emphasis falling on physical and material needs, followed by conflict in the household. Of the two groups, OVC emphasize education. In addition to raising broader social issues such as the ubiquity of substance abuse in impoverished communities, a persistent theme in community stakeholder interviews was the unwillingness or inability of primary caregivers to effectively care for OVC. Such a perspective may influence community members’ assessment of the worth of OVC-targeted interventions. Programs that focus on family preservation may be viewed as inadequate or unequal to addressing the needs of OVC. Table 8: Challenges faced by OVC per site Challenges Faced by OVC, per site KwaJobe Botshabelo Nellmapius Vosloorus  Difficulties in accessing IDS and birth certificates  Poor school performance  Lack of parental care/support  Difficulties in accessing grants  Lack of food/hunger  Lack of food/hunger  Lack of school uniforms and stationery  Sexual abuse  Lack of parental care/support  Poor school performance  Poor school performance  Lack of school uniforms and stationery  Lack of food/hunger  Lack of parental care/support  Difficulties in accessing grants  Poor school performance  Lack of school uniforms and stationery  Lack of parental care/support  Lack of food/hunger  Conflict in intergenerational relationships Source: Program staff interviews, CCW focus groups, community stakeholder interviews, and OVC and primary caregiver focus groups The challenges across sites echo the findings per stakeholder, emphasizing physical and material needs, conflict in the household, and education-related issues. However, differences are observed in some areas. Difficulties in accessing IDs, birth certificates, and social grants are frequently reported in KwaJobe, reflecting a common service delivery deficit shared by rural communities in South Africa. Sexual abuse was frequently reported by respondents in Botshabelo, and is documented here. The evaluation team was constrained by its explicit ethical framework from exploring the issue beyond this initial level of observation. Other challenges not listed in the tables above that emerged less frequently include  additional psychosocial challenges such as low self esteem, emotional and physical abuse, risk-taking behavior that manifests as substance abuse, teenage pregnancies, and peer pressure to engage in these patterns of behavior  additional material challenges related to the impoverished conditions of households and communities accessing Heartbeat services, including no money to pay for school fees, participate in school field trips, or purchase basic necessities such as clothing and toiletries  persistent poor health of the OVC, their parents, and caregivers  education-related challenges such as poor school attendance  risks to personal security 1A. Is the program meeting the needs of the communities and the children as perceived by key stakeholders? The consensus among stakeholders is that Heartbeat programs respond directly to their needs. The alignment between programming and reported needs confirms this consensus and is reflected in Table 11. However, stakeholders did make critical observations regarding the effectiveness of programming in education, psychosocial, and material support. Heartbeat UGM End of Project Evaluation 29 Table 9: How Heartbeat met the needs of the community Heartbeat’s Activities and Community Needs Most Reported Challenges/Needs Activities/services offered by Heartbeat to meet the needs PEPFAR Funded? Comments Lack of school uniforms and stationery Provision of school uniforms and stationery N Heartbeat covers this with other funds but not sufficiently Lack of food/hunger Provision of ASC meals and food parcels N Heartbeat covers this with other funds but not sufficiently Poor school performance Assistance with academic performance through homework support and extra lessons Y Need for more experienced staff to assist with homework. Extra lessons (Math and English) start too late in the year. School readiness assistance was stopped. Lack of parental care/support Counseling at the ASC and during home visits, support groups, workshops, and memory work Y Disrespectful youths Counseling at the ASC and during home visits, support groups, and workshops Y Difficulties in accessing social grants Assistance with applying for social grants Y Hinged on the availability of IDs and birth certificates Difficulties in accessing IDs and birth certificates Assistance in applying for IDs and birth certificates; ID drives Y Reports of long waits Sexual abuse Counseling at the ASC and during home visits, support groups, workshops, and referrals to relevant departments Y Substance abuse Counseling at the ASC and during home visits, support groups, workshops, and referrals to relevant departments Y Low self esteem Counseling at the ASC and during home visits, support groups, workshops, referrals to relevant departments, and access to basic needs like food and uniforms Y Lack of clothing Provision of clothing N Heartbeat covers this with other funds but not sufficiently Poor health Referrals to clinics/hospitals Y Source: Secondary data, program staff interviews, CCW focus groups, community stakeholder interviews, and OVC and primary caregiver focus groups Educational support While stakeholders consistently attributed improved school performance among OVC to the education programming offered by Heartbeat, a number of critical observations were made.  OVC and community stakeholders indicated that the CCWs were in many instances not able to provide the homework assistance due to their own deficits in content knowledge, particularly when it came to content in the secondary school syllabus. This claim was to some extent confirmed by CCWs themselves.  Center staff reported that the extra lesson project in the sites under investigation was not implemented successfully, with lessons for 2012 not having commenced at the time of the evaluation. Furthermore, there was a lack of clarity as to what may be expected in terms of proceeding with the activity, with program staff at site level waiting for direction from the headquarters office. It appears that this activity was delayed rather than discontinued. As a centrally managed intervention, the reasons for delay must be clarified with Heartbeat headquarters.  Similarly, the payment of crèche fees under ECD was also welcome but center staff reported that it was stopped at the end of 2011, with no clear indication of whether it was going to resume. It would appear that the discontinuation was partly a funding issue, but also a tactic to encourage caregivers to assume responsibility for the educational well-being of OVC in their care. The rationale for assuming the latter would be effective is unclear. Center staff interviewed reported that, in addition to undermining outcomes, inconsistent implementation of program activities has resulted in strained relations between centers and their beneficiaries. The situation is exacerbated by the apparent ineffective communication with center staff from headquarters, with the result that Heartbeat is not in a position to credibly manage expectations regarding discontinued or delayed program activities. Evidence of stained relationships could also be observed in data obtained from primary caregivers. Psychosocial support While psychosocial support activities were unanimously considered invaluable, stakeholders indicated unintended negative consequences that undermine outcomes. It was reported that progress made by OVC has led to exacerbating conflict in the household. This is in part due to the fact that the rights￾based approach to empowering OVC can lead to conflicting perspectives between OVC and their caregivers on issues such as corporal punishment, and an assertive stance on the part of OVC that is not understood or well received by others in the home. The solution suggested by stakeholders is that psychosocial support interventions be extended to include the household or family. Gains made by OVC in their psychological well-being are vulnerable to deleterious conditions in the household and community. There is little evidence of programming to strengthen the capacity of households and the community to provide psychosocial support for OVC, despite the latter being an explicit objective of Heartbeat’s interventions. Material support and access to social protection services The vigor with which critique of material support activities was voiced by stakeholders is indicative of the value assigned to this area of programming by Heartbeat staff and especially its beneficiaries. PEPFAR funding does not support the provision of food, school uniforms, stationery, and clothing. Although Heartbeat covered this area with other means it was not implemented optimally, primarily due to the inconsistent flow of funding for this programming area. In addition to inconsistencies in the provision of food parcels, the criterion that disqualifies beneficiaries from receiving such assistance once social grant payouts commence is unpopular. The economic and emotional burden material provision imposes on caregivers was starkly apparent during fieldwork. Beneficiaries are overwhelmingly in favor of direct material support to households, especially in terms of food provision. PEPFAR’s preference for HES interventions over material support is intended to secure sustainable outcomes and ultimately eliminate the necessity for relief interventions to meet the physical needs of beneficiaries. However, although the PEPFAR funding supported an HES program, it was not successfully implemented through the Heartbeat centers. The two specific interventions under HES—support for tertiary students and the individual savings intervention—were either not executed or carried out selectively. It should be noted that HES is a programming area that Heartbeat was required to adopt and strengthen as a result of receiving PEPFAR funds. Nevertheless, it is clear that implementation of the two intended HES activities has proven weak, and the explanation for this deficit in program fidelity needs to be sought from Heartbeat headquarters. This finding echoes a fairly consistent thread emerging throughout the evaluation that points to inefficacies in organizational management, manifest as programming delays and limited guidance from headquarters for Heartbeat center staff. In addition, the reorientation of emphasis from direct material support to HES and the resulting decrease in consistent funding for material support activities had an important unintended consequence. When inconsistencies in the material support program emerged, the uptake of beneficiaries in other Heartbeat interventions was negatively affected. The provision of food proved to be the most significant Heartbeat UGM End of Project Evaluation 31 incentive for attending the ASC and participating in supporting programs. Once the incentive was removed, participation and consistent attendance at the ASC declined. While stakeholders recognize that government processes are protracted, there is a perception that Heartbeat is not doing enough to accelerate access to social protection services. Stakeholders also reported difficulties in maintaining access to social protection, evidenced by reports of registration being suspended then reinstated after some months. The issues with social protection reported by primary caregivers and CCWs are not uncommon and, according to the Department of Social Development (DSD) and the Auditor General, are attributable in part to either flaws in the implementation of social protection system or fraudulent activity. 1B. Which program services were most valuable to OVC, their families, and the community? Stakeholders were asked to indicate the most valuable services provided by Heartbeat in an open￾ended discussion. The services emphasized in those discussions were noted and the results are shown in Table 10. The evidence in response to this evaluation question clearly shows that material support is the most valued service among all stakeholders and across all sites. Stakeholder responses are corroborated by the fact ASC attendance is negatively affected once beneficiaries begin to access social grants and Heartbeat ceases to supply them with food parcels. Material support, especially the provision of food parcels and meals, is a significant inducement to participate in other Heartbeat programs. Table 10: Most Valued Services Most Valuable Program Services Stakeholders Material Psychosocial Education Social Protection OVC KwaJobe X X OVC Botshabelo X X X OVC Nellmapius 1 X OVC Nellmapius 2 X OVC Vosloorus 1 X OVC Vosloorus 2 X Heartbeat staff and community stakeholders X X Source: Program staff interviews, community stakeholder interviews, and OVC 1C. Are there additional areas/services that the key stakeholders would like Heartbeat to provide? When asked to suggest additional ways in which Heartbeat can assist OVC in their communities, stakeholders across all sites repeatedly mentioned the strengthening of material support (food, school uniforms, and clothing), confirming the priority of this program area for all beneficiaries. The strengthening of certain educational and psychosocial support activities was also proposed.  Program staff, community stakeholders, and CCWs in Botshabelo and Nellmapius saw the need to assist OVC post-matric with scholarships, skills development, and employment opportunities. The recognition of the necessity to cultivate work readiness was echoed by OVC in Nellmapius and Vosloorus, who requested computer lessons at the ASCs.  Beneficiaries and community stakeholders (most of whom are teachers) in Botshabelo, Nellmapius, and Vosloorus suggested that more experienced staff and more resources need to be channeled into the extra lessons project if quality results are to be realized.  Family, caregiver, and community support groups were suggested as ways of strengthening psychosocial support programs that currently focus almost exclusively on OVC. While the rationale for these proposals was frequently rooted in concrete examples illustrating the need to address household conflict or the desire to benefit from the interventions that produced observed changes in OVC, stakeholders demonstrated an intuitive recognition that to sustain the gains achieved through psychosocial support, changes need to be made in household and community contexts. In addition, stakeholders suggested general improvements to Heartbeat’s service provision.  It was suggested that the graduation age for OVC from Heartbeat be 19 instead of 18 years of age. Currently, the graduation age does not mirror that of graduation from school, where a child begins formal education at 7 years old and finishes at 19 years of age.  A substantial proportion of its OVC beneficiaries experience difficulty in accessing Heartbeat services, specifically services which are center-based (see 1.1A below for a discussion of this challenge). In suggesting additional services Heartbeat might offer, the access issue was reiterated, with suggestions for additional centers, a more extensive outreach program, and a transport service for community members to and from the center. An unexpectedly persistent response to the discussion on potential additional services was the insistence on the establishment of orphanages in communities; all stakeholders except for primary caregivers in KwaJobe, Botshabelo, and Nellmapius made this proposal. The stakeholders were motivated by their observations of the severe vulnerability of OVC to neglect, abuse, and exclusion from the social protection system. From their perspective, and by implication, the principle of family preservation governing OVC policy nationally does not account for many circumstances of OVC including those whose family environment is particularly dysfunctional, where households of their extended family are acutely impoverished and incapable of assuming the burden of additional household members, and those living in communities where the current constellation of services available to OVC, or offered by government or NGOs, are simply unequal to the problems confronting OVC. For community stakeholders sensitized to the life threatening circumstances of these OVC, the introduction of institutions in communities appears to offer a solution—the assumption by the state of full responsibility for every aspect of the OVC’s life. For Heartbeat, the critical and chronic risk to personal safety of a proportion of their OVC beneficiaries is highlighted by this dialogue, and may need to be considered in future programming. KEY EVALUATION QUESTION 1.1: HOW DO STAKEHOLDERS (CHILDREN, PRIMARY CAREGIVERS, CARE WORKERS AND COMMUNITY REPRESENTATIVES) PERCEIVE THE PROGRAM IN TERMS OF QUALITY AND EASE OF ACCESS? This section assesses the extent to which Heartbeat services are accessible to OVC in terms of visibility, affordability, proximity, and age-appropriateness or child-centeredness. In addition, it considers whether OVC are stigmatized when accessing Heartbeat services. Together, these criteria constitute the quality of Heartbeat services for the purposes of this evaluation. Table 11 summarizes the stakeholder perceptions of program quality and ease of access across sites. Table 11: Stakeholder Perceptions of Program Quality and Ease of Access Stakeholder Perceptions of Program Quality and Ease of Access KwaJobe Botshabelo Nellmapius Vosloorus Heartbeat is visible in the community Yes Yes Partially Partially Services are easily accessible No No No Partially Services are affordable Yes Yes Yes Yes Services offered are age appropriate Yes Yes Yes Yes The ASC is child-centered No No No No Services are free of stigma Yes Yes Partially Yes Source: Program staff interviews, CCW focus groups, community stakeholder interviews, and OVC and primary caregiver focus groups Heartbeat UGM End of Project Evaluation 33 1.1A. To what extent were services accessible to OVC in terms of visibility, proximity, affordability, age- appropriateness, and child-centeredness? Visibility Heartbeat centers tend to benefit from highly visible locations within their communities, as well as high levels of community awareness. However, in Nellmapius and Vosloorus it was reported that community members are less familiar with Heartbeat and its services, evidenced by incorrect referrals from the community, community members mistaking Heartbeat for a government agency, and the overwhelming association of Heartbeat with the provision of food relief. Proximity and Affordability Heartbeat establishes a single center in each community it serves, which means that inevitably the location may not be convenient for some of its beneficiaries. Problems of proximity to and accessibility of services is exacerbated by the size of communities serviced in urban areas and the geographic spread of beneficiary households in rural areas. Issues of proximity also affect delivery of outreach services, with CCWs required to travel extensively and cover substantial distances between beneficiary households. While Heartbeat provides its services free of charge to beneficiaries, affordability remains an issue due to travel costs to the centers. It is clear from the interview and focus group data that the cost of travel to centers for beneficiaries to access services is almost prohibitive. Age-appropriateness of services and child-centeredness of ASCs The only issues raised by respondents in terms of the age-appropriateness of services and child￾centeredness of ASC environments were with reference to toddlers and pre-schoolers. The general view was that these age categories of OVC are not adequately catered to by Heartbeat. Deficiencies included the inexperience and lack of specific expertise of Heartbeat staff to provide care for these groups, and a shortage of resources (educational materials and toys, appropriately sized furniture, catering equipment, cots and mattresses) required to serve these groups. With the exception of the Nellmapius center, the restricted outdoor space occupied by ASCs allows for little in the way of playgrounds and the installing of sports equipment. The limited indoor space also precludes privacy and there are no dedicated areas for counseling and psychosocial support activities. In KwaJobe, there is no proper fencing around the center, compromising the safety of the OVC. 1.1B. Is there any stigma associated with accessing Heartbeat services? Heartbeat takes considerable care to ensure that services are not perceived as being exclusively intended for a disadvantaged subset of the community and that centers are a community resource. Respondents at all sites confirm that efforts to avoid stigma have been largely successful. However, a few respondents in Nellmapius indicate that the collection of food parcels or ASC meals sometimes attracts ridicule, especially among adolescents. KEY EVALUATION QUESTION 2: WHAT WERE THE MOST SIGNIFICANT CHANGES BROUGHT ABOUT BY THE ORGANIZATION IN IMPROVING THE WELL-BEING OF ITS BENEFICIARIES? This section highlights the significant changes realized in OVC, their families, and in the community as a result of the services offered by Heartbeat. To address this question, Heartbeat staff, CCWs, community stakeholders, primary caregivers, and OVC were asked to describe changes in the OVC, the family, and the community attributable to Heartbeat’s activities. Responses from the different stakeholders were collated and are presented in Table 12 and Table 13 in order of the most frequently reported changes for each stakeholder or site. The changes recorded in the table are not an exhaustive list, and additional, less frequent observations are listed separately. Changes observed in OVC beneficiaries According to its objectives, Heartbeat aspires to provide for the psychosocial, intellectual, and physical rights of OVC and to improve the quality of life of OVC. The changes presented in this section relate to psychosocial and intellectual rights and also show an improvement in the quality of life, especially in terms of improved health, access to social protection, admission to tertiary education, and employment. Table 12: Changes observed in OVC per stakeholder Changes observed in OVC, per stakeholder OVC N=6 (Focus groups) Primary Caregivers N=4 (Focus groups) CCWs N=4 (Focus groups) Community Stakeholders N=11 (Individuals) Heartbeat Staff N=19 (Individuals)  Better school performance (100%)  Positive outlook on future (83%)  Increased self confidence (67%)  Improved health (67%)  Better school performance (100%)  Access to IDs (100%)  Increased self confidence (100%)  Improved health (75%)  Better school performance (75%)  Decrease in risk behavior (75%)  Access to social grants (50%)  Access to IDs (50%)  Better school performance (70%)  Access to social grants (40%)  Access to IDs (40%)  Tertiary admission/ employment (59%)  Better school performance (53%)  Access to social grants (35%)  Improved interpersonal, communication skills (35%) Source: Program staff interviews, CCW focus groups, community stakeholder interviews, and OVC and primary caregiver focus groups Respondents referred to four areas of significant change affected by Heartbeat activities.  There appears to be a consensus across all stakeholders (community stakeholders, key informants at Heartbeat headquarters, center staff including community care workers, caregivers, and OVC) that better school performance is the most significant change in OVC. In addition, education support is the programming area most commonly associated with Heartbeat. When stakeholders were asked what activities Heartbeat provides, the majority of responses related to educational activities.  The next most significant change noted by all stakeholders, with the exception of OVC but emphasized particularly by caregivers and CCWs, is access to social protection. Specifically discussed was the process of obtaining the documents required to access grants (which caregivers found exceptionally challenging) and ultimately accessing grants. Gains in terms of physical rights, including material support, and the means to provide for material support, emerge as the central concern for all stakeholders including OVC. Throughout the evaluation, this was regarded as a primary need, most valued service, and a significant change to the life circumstances of OVC.  Psychosocial changes that include increased self confidence and improved interpersonal and communication skills were also observed, and considered significant across different stakeholders. These changes were most often noted by OVC, their caregivers, and Heartbeat center staff.  Improved health and increased knowledge concerning HIV prevention are changes that were also noted by stakeholders across all sites, reported most frequently by caregivers. This result reflects positively on the referrals system and prevention education, both of which were intensified with PEPFAR funding. Although significant changes were observed in OVC that can be attributed to discrete programming areas (education, access to social protection, and psychosocial support), it is important to reflect on the integrated way in which change is affected. For example, OVC report having a positive outlook on their future not as an exclusive result of psychosocial support, but as a cumulative result of having the burdens of material support lifted, having recourse to address the education related challenges, and having Heartbeat role models who have graduated into opportunities that promise an exit from the cycle of poverty. This example illustrates the case for recognizing that while certain programs or Heartbeat UGM End of Project Evaluation 35 activities are measurably more effective than others, Heartbeat’s impact is the cumulative result of a comprehensive package of different but complementary services. Table 13: Changes observed in OVC per site Changes observed in OVC, per site KwaJobe Botshabelo Nellmapius Vosloorus  Access to social grants  Access to IDs and birth certificates  Better school performance  Improved health  Increased knowledge of HIV/AIDS prevention  Improved interpersonal, communication skills  Tertiary admission/employment  Better school performance  Access to social grants  Increased knowledge of HIV/AIDS prevention  Increased self confidence  Increase in discipline among youths  Better school performance  Tertiary admission/employment  Improved health  Improved interpersonal, communication skills  Access to social grants  Increased knowledge of HIV/AIDS prevention  Better school performance  Improved health  Increased self confidence  Increased knowledge of HIV/AIDS prevention  Access to IDs  Sense of belonging Source: Program staff interviews, CCW focus groups, community stakeholder interviews, and OVC and primary caregiver focus groups For Heartbeat staff, it is progress after school that represents the most significant change. Admission to tertiary education institutions and gainful employment indicate the fulfillment of Heartbeat’s mission to empower OVC to reach their potential. Botshabelo and Nellmapius, for example, have supported a number of Heartbeat graduates who have either enrolled in tertiary institutions or have been gainfully employed, some of them at these two centers. While the majority of Heartbeat program staff reported such changes, the examples related were exceptions rather than the norm. The limited achievement in this area is partly attributable to the inconsistent implementation of deliberate mechanisms to secure these outcomes. This is at least true for the period under review, where planning reflects the intention to support tertiary education under HES but activities were not executed. As discussed earlier, the reasons for delayed implementation appear to be on the part of Heartbeat headquarters. Other changes mentioned across sites and by a variety of stakeholders, but not at the frequency of those discussed thus far, include psychosocial-related effects such as happiness, exposure to pursuits that broaden life experience such as participation in outdoor activities, and perceptions of improved personal safety. Table 16 presents the emphasized changes from the perspective of OVC themselves, based on the disaggregated data on most significant changes observed in OVC. While changes attributed to education interventions (improved school performance) and physical rights and material support are also emphasized, the means to secure material support (gaining access to social protection) is not. This variation on the material support and physical rights theme is plausibly a function of the fact that securing the means to material ends remains the concern of primary caregivers, that the social protection system distributes grants due to children to primary caregivers, and that grants are employed by households for general benefit. OVC are seldom concerned with these means, but their experience of the effects that these means bring about is keenly felt. “Heartbeat graduates, some are nurses, soldiers, police and they see it and say ‘Yoh! If it was not for you guys’…” -Site Staff, Botshabelo Table 14: Most Significant Changes to OVC according to OVC by site OVC-reported Most Significant Changes, by site Focus Group Access to food Counseling, support and care Improved school performance Access to legal documents and grants Access to school uniforms, stationary, clothes Post￾matric assistance (career guidance) KwaJobe X X Botshabelo X X X Nellmapius 1 X Nellmapius 2 X X Vosloorus 1 X Vosloorus 2 X Changes observed in the family Generally, Heartbeat provides services that offer care directly to OVC. Very few sites reported implementing interventions that improve the capacity of families to care for OVC. Consequently, respondents offered little data in response to questions on the most significant changes to families affected by Heartbeat interventions. The paucity of responses was exacerbated by the fact that the indirect effects of Heartbeat interventions in this regard are not always obvious to stakeholders. In terms of family specific interventions, Heartbeat staff and caregivers at KwaJobe and Botshabelo mentioned workshops that were held for OVC family and guardians. In KwaJobe, guardians were invited to a workshop on how to prepare a will (the intervention was described as educating caregivers on succession planning) and in Botshabelo, guardians were advised on how to budget their social grant money and save for the future. In addition to these discrete events, counseling is provided for caregivers and families on an as-needed basis as issues relating to the individual OVC emerge. In response to reports of the ill-treatment of OVC in a household, CCWs highlighted instances in which they proactively engaged relevant household members in a counseling process. The extent to which these direct interventions facilitate change at family level cannot be determined from the evidence provided by respondents. One particular intervention was consistently characterized by caregivers, community stakeholders, and CCWs as strengthening the capacity of households to care for OVC. Although intended for the benefit of OVC specifically, Heartbeat’s efforts at facilitating access to social protection substantially improve a family’s capacity to provide care for its OVC. Grants represent a reliable and consistent resource on which the provision of care depends. In addition, grants are typically used to the more general benefit of the entire household, improving the circumstances, even if only marginally, of the systemic context in which the OVC is raised. Social grants are a critical if insufficient basis for care in families with few income alternatives. While it may be argued that social protection perpetuates dependency, this assistance makes families independent of Heartbeat for basic material support, and in this way acts as a strategy for sustainability. Adequately strengthening families’ capacity to care for their OVC in terms of material support would necessitate more robust HES; while such activities are reflected in Heartbeat’s planning, these were not effectively implemented and respondents did not refer to any such activities as interventions that led to significant changes in families. Respondents did offer explanations of how interventions for OVC resulted indirectly in positive changes at the family level.  Material support, such as the provision of school uniforms, funds for school fees, and food parcels, alleviated the burden of provision on the household and primary caregiver, augmenting the existing means of the household to provide care to their OVC. Heartbeat UGM End of Project Evaluation 37  The improved health of OVC, attributed to Heartbeat interventions, was also considered to alleviate the burden of care on the family.  The psychosocial support offered to OVC and the changes this affected led to improved relations within the family, between the guardians and the OVC, and between OVC and household members generally. It might be argued that these changes improve an orientation towards the care of OVC in the family, which enhances the capacity of the family unit to care for its OVC. Changes observed in the community Respondents offered little data in response to questions about observed significant changes at the community level. The responses suggested some gains in terms of Heartbeat’s objective to develop a culture of support for OVC within communities. Across sites, the common message was that the community displays more support towards OVC, as evidenced by referrals of children to Heartbeat and donations made to OVC through Heartbeat. This change is paired in stakeholder responses with the feeling that OVC are less of a burden to the community as a result of Heartbeat’s presence, and less of a social risk as the potential for OVC involvement in crime is reduced. Together, these two reported changes make it apparent that communities require a mechanism through which their capacity to care for OVC is increased. Heartbeat offers such a mechanism through the Heartbeat centers, which provide communities with a channel to mobilize resources, volunteer their time, or refer OVC to a place where they can be cared for. Without such a center, a community— especially a disadvantaged community—has no means for caring for its OVC, OVC are perceived as an unmanageable burden, and community members become frustrated with the obligation to care and ultimately dismissive of the needs of their OVC. Based on this interpretation of the data, it is reasonable to conclude that the presence of Heartbeat in a community represents a significant change at community level. 2B. How did Heartbeat change the path of children through their interventions? Based on the evidence presented in this evaluation, there is no doubt that the efforts of Heartbeat have resulted in substantial positive impact on the prevailing circumstances of OVC in each of the areas of identified need. In addition, there is compelling evidence to demonstrate changes at family level regarding the household’s capacity for taking care of its OVC, although such change is modest. At household level, capacity to care is dependent on access to resources, a need directly addressed through Heartbeat’s material support programming and, more sustainably, through facilitation of access to the social protection system. At community level the evidence, though insufficient for conclusive findings, appears to support the argument that Heartbeat’s presence in communities offers an indispensable mechanism that increases the community capacity to care for its OVC. Together these results positively influence the present and immediate future of OVC. However, this conclusion must be qualified by acknowledging that the circumstances of OVC are overwhelming. The extent to which the cumulative benefit of the intervention can change the immediate future of OVC beneficiaries, under a variety of particular circumstances, and the way in which the changes are affected, remains unclear. There is also evidence confirming long-term changes for some Heartbeat beneficiaries that have improved their future prospects as a result of Heartbeat interventions. In these instances, beneficiaries have acquired skills, qualifications, and have obtained employment—the means by which they are empowered to extricate themselves from the poverty and dependency that typified their lives as children. However, evidence indicates that these are exceptional cases, rather than the norm. Although the paths of OVC beneficiaries have been changed, some more significantly than others, the focus of programming is on the immediate circumstances of OVC. The rationale for this focus is sound, as the resources required to implement interventions that secure more promising future outcomes are severely limited. In addition, it is critical to acknowledge that the detrimental circumstances faced by OVC in South Africa are systemically pervasive, and not able to be addressed by a single organization. KEY EVALUATION QUESTION 3: WHAT WERE THE KEY ENABLERS AND BARRIERS IN MEETING PROJECT OBJECTIVES? This section discusses the programmatic and contextual enablers and barriers that have been instrumental in achieving or hindering achievement of project objectives. Enablers and barriers discussed in this section are distinct from the critical program design features identified and discussed in the assessment of the relevance, quality, and comprehensiveness of Heartbeat’s intervention model. Key Enablers The key enablers in achieving project objectives are summarized in Table 15, categorized and arranged as prioritized by respondents. These enablers were instrumental in lowering barriers to access of services being offered, embedding Heartbeat and its programs in the community, ensuring that resources for intervention were mobilized, and that the effectiveness of the intervention model was optimized. Table 15: Key Enablers in achieving project objectives Key Enablers to Achieving Project Objectives Enablers Effect of Enablers  Providing services free of charge to beneficiaries  Ensuring that no stigma is associated with accessing services Lowering barriers to access  Following local protocols and engaging community structures when establishing a center  A sound messaging strategy  Sourcing center staff locally Embedding Heartbeat and its programs in the community  Training of center staff by Head Office  Organizational development support by Head Office  Funding  Staff's passion for working with children Mobilizing resources for effective implementation  Partnerships and referrals Enhancing the efficacy of the intervention model Lowering Barriers to Access The enabler most noted by respondents is Heartbeat services being offered free of charge. It would appear that cost of access remains the key consideration for caregivers and OVC beneficiaries, echoing the prominence of material needs as a concern of OVC, households, and community stakeholders. While cost represents a material prohibitive factor, the stigma of vulnerability and poverty acts as a forceful social prohibitive factor to accessing services. Heartbeat takes care to position itself as a service provider to the entire community, and in doing so, has managed to dilute the potential stigma associated with its interventions. This effort is noted and considered an important enabler by respondents. Embedding Heartbeat and its Programs in the Community In the process of establishing a center, Heartbeat takes care to enter the community observing the protocols expected by the local authority structures and engaging the community. The process consistently includes cultivating partnerships with key community institutions, including schools, police, and any relevant community forums. Early efforts to secure buy-in from the community are identified by Heartbeat center staff and community stakeholders as a crucial factor in both obtaining the necessary support from community members and as the mechanism by which the people are sensitized to the presence of the center, its role as a community resource, and the services it offers. Heartbeat continues to maintain these relationships, utilizing them as channels to repeatedly convey the appropriate messages to elevate visibility and accessibility in the community. In addition to the process it follows in establishing a center and the communication strategy it implements, Heartbeat embeds itself in the target community by recruiting center staff locally. In this Heartbeat UGM End of Project Evaluation 39 way, Heartbeat adopts a local identity while being in a position to capitalize on local intelligence when reaching out and recruiting OVC beneficiaries and their families. The systematic approach to embedding Heartbeat in the community lays a foundation that contributes to the realization of program objectives. Mobilizing Resources for Effective Implementation An enabler consistently mentioned by respondents is the passion of center staff for working with children. While no evidence was apparent that this characteristic served as a deliberate criterion for recruitment, training of staff by the Heartbeat head office is deliberate. Center staff recognizes the importance of the training in equipping them to effectively deliver services. In addition to training, Heartbeat headquarters supports organizational development through management support as required. Community stakeholders also recognized the pivotal importance of funding introduced into the community by virtue of the establishment of and program implementation by the local Heartbeat center. The importance of funding, and the consistent flow of funding, was emphasized by center staff especially. Respondents also criticized the inconsistency of funding experienced in particular instances, as documented later in the section on barriers to achieving program objectives. Enhancing the Efficacy of the Intervention Model Respondents mentioned various aspects of the Heartbeat intervention model when identifying enablers in achieving various objectives. The effectiveness of the intervention model has already been assessed in this evaluation against criteria of relevance, quality, and comprehensiveness. An important observation made by respondents and not considered in the evaluation of the model is the extent to which Heartbeat facilitates access to a broader service network. Heartbeat successfully engages with governmental departments such as Home Affairs, South African Social Services Agency (SASSA), DSD, and the Department of Health in securing access to services for its beneficiaries. It is also successful in securing resources from private sector sponsors, such as Tiger Brands, which provides Heartbeat with food parcels for the OVC. Heartbeat also refers beneficiaries to NGOs that provide services such as home-based care. Key Barriers Table 16 presents the key barriers that hinder attainment of project objectives. These barriers are challenges are either inherent to the context in which the programs are being implemented, related to the manner in which programs are implemented, or attributable to Heartbeat’s organizational structure and processes. Many of the barriers identified correspond to an enabler identified in the preceding section, implying that to overcome a number of the barriers, all that is required is a strengthening of current arrangements or activities of the intervention model and programs. A number of the barriers, however, do not correspond to enablers and will require additional efforts to mitigate. Table 16: Key barriers to achieving project objectives Key Barriers to Achieving Project Objectives Organizational Programmatic Contextual  Inefficiencies in organizational structure and processes  Inadequate training and support in some areas  Inconsistent implementation of communication strategy  Insufficient infrastructural and material resources  Capacity burdens of funding prerequisites  Inconsistent program implementation  Inappropriate criteria for selection of beneficiaries  Lack of age-appropriateness in service environments and delivery  Wide geographical dispersion of OVC and households  Rivalry in local NGO community  Stigma associated with being labeled an orphan or impoverished child  Community norms in conflict with Heartbeat’s human rights based approach  Lack of security at Heartbeat centers Organizational barriers Inefficiencies in organizational structure and processes Inefficiencies in the current organizational structure and communication processes were consistently raised as barriers to efficacy across all sites. Poor communication was reported between Head Office staff and site staff; between middle managers, CDFs and the rest of the site staff; and between CCWs and ASC staff. Moreover, site staff said they felt excluded from decision making by Head Office and CDFs. Compounding this is the fact that reporting lines are unclear; it is unclear to many whether site staff should report to or get direction from the CDF or middle managers. Staff shortages at center level were also reported, particularly in terms of social workers and the shortage in CCWs. Inadequate training and support in some areas Related to challenges in the organizational structure is the need for more training and support at site level in order to achieve objectives. There are reports that the training provided is not sufficient; for example, not all CCWs are trained to be care workers, CCWs are not trained in HIV/AIDS and antiretroviral (ARV) treatment (despite dealing with health referrals and follow-ups), and administration staff require computer literacy training. It was also reported that training provided is not always relevant; for example, the training on succession planning in one site was done by a SAO who does not engage with the beneficiaries as much as a Choza or CCW would. There were reports that the training agenda is not always clearly communicated to site staff, making selection of the person most suited to attend the training difficult. Staff that engage with the beneficiaries, especially the CCWs, requested debriefing sessions and counseling with a social worker external to Heartbeat in order to maintain confidentiality. Inconsistent implementation of communication strategy Although the section on key enablers showed that Heartbeat has a sound communication strategy that contributes to the visibility of Heartbeat, there are still some community members who are not clear what Heartbeat is and what it does. Some respondents asserted that Heartbeat is known only for its food, and other services that they provide are not well known. There were reports of erroneous referrals to Heartbeat centers and mistaken identity of Heartbeat as a government organization. Programmatic barriers Inconsistent program implementation It is not always clear when funding for certain activities will stop. At the end of 2011, payment of crèche fees for ECD was suspended in all sites and extra lessons for high school learners for 2012 had not begun at any sites. The future of these services was not clear to staff at the sites. Other services that were suspended include transportation to health clinics in Nellmapius and payment of excursions in Vosloorus. In addition, the individual savings intervention and the support for tertiary students, both activities under the HES program area, were either selectively or not implemented. Heartbeat UGM End of Project Evaluation 41 Unfortunately, the evaluation has not been able to clearly determine the reasons for these deficiencies in execution. At site level, activities are not conducted because resources and support are not provided from headquarters, while reasons for delays are not clearly communicated to center staff. Heartbeat’s headquarters must provide clarification on the reasons for implementation inconsistencies. It was reported by center staff, CCWs, and caregivers that discontinuation of activities and delays in programming have resulted in strained relationships between Heartbeat and beneficiaries. Insufficient infrastructural and material resources The most frequently reported programmatic barrier across all sites is the insufficiency of infrastructural and material resources. A lack of space at Heartbeat centers hinders the implementation of program activities (for example, counseling activities at centers without dedicated, private space). Space limitations at the KwaJobe and Vosloorus centers undermines their capacity to offer age-appropriate activities for younger OVC because there can be no dedicated, equipped playground areas. At KwaJobe, the lack of water and electricity made it difficult for staff to carry out routine tasks such as preparing meals and writing reports. Resource and materials shortages at all sites curtail the provision of services for all age groups. Children below 8 years old are able to receive services, as there are no materials for activities suitable to the age group and no staff trained specifically to manage their care. Shortages of food parcels were reported in KwaJobe and Nellmapius. As has already been documented, the inconsistent supply or withdrawal of food parcels has deleterious consequences as it leads to beneficiaries dropping out of other Heartbeat programs. Material support interventions, especially the provision of food, incentivize beneficiary participation in Heartbeat activities. CCWs reported a lack of resources necessary to efficiently conduct home visits, such as transport and protective clothing when visiting households with severely ill household members. They also reported that the stipends they receive are insufficient. Capacity burdens of funding prerequisites Another barrier repeatedly raised, especially by key informants as well as staff at Botshabelo and Vosloorus, is that sometimes funding necessitates substantial changes in the organization as a result of donor service delivery requirements. Services such as healthcare referrals and prevention education had to be scaled-up to meet the UGM objectives. While these demands may prompt the strengthening of organizational capacity, it frequently undermines capacity as staff members are subject to greater demands without their capacity to deliver, as individuals or an organization, being commensurately enhanced. Another funding requirement that was questioned at center level was the standardization of activities across sites. It was asserted that services that are valuable in one site may not be equally valuable in another site. In Vosloorus for example, the site staff explained that ID drives and healthcare referrals are not as valuable, since beneficiaries had easy access to government departments and clinics. Additionally, funding only provides for specific services and not for any additional needs that may arise. Criteria for selection of beneficiaries Heartbeat has criteria for OVC intakes; although the criteria set parameters within which Heartbeat operates, it also manifests as a barrier. OVC that do not meet the criteria cannot be registered with Heartbeat. Some community members consider the criteria discriminating as some referrals they make do not fit the criteria. Heartbeat staff reported that they cannot assist people who do not fit the criteria, even if they deem them to be in need. Ultimately, this dissatisfaction may risk compromising community buy-in. Three issues in particular were consistently raised. 1. The graduating age of Heartbeat is 18 years old and OVC who have reached 18 are taken off the program, even if there are not yet self sufficient. It has been suggested that an additional year would smooth the aging out process. 2. The Heartbeat criteria does not, when strictly applied, allow for services to be provided to children living in households where their biological parents also reside. This condition is based on a limited definition of vulnerability that perhaps no longer reflects the reality confronting children in the communities being serviced. 3. OVC have to attend ASC activities a certain number of times per month in order to access material support, such as food and stationary. Sometimes the ASC is too far and OVC only arrive when the center is about to close. If an OVC is awarded a social grant, he is no longer entitled to a food parcel, which has resulted in the loss of beneficiaries across all sites as a lot of value is placed on the food parcels. Contextual barriers Geographical dispersion of OVC beneficiaries and households The most reported contextual barrier is the fact that the geographic areas covered by each center are substantial, beyond the ability of the centers to adequately service given their current capacity and resources. The extent of the areas to be covered results in difficulties for eligible OVC to attend the ASC and access or qualify for access to services; it also makes it exceptionally difficult for CCWs to fulfill their outreach and follow-up obligations effectively. Rivalry and services duplication in the local NGO community There were reports of rivalry with other NGOs who are involved in similar work. Civil society responds to current donor priorities in order to secure funding, which may result in competing for beneficiaries in order to make up the numbers monitored by the donor and lead to duplications in service delivery. Stigma associated with being labeled an orphan or impoverished child Although not widely reported, some beneficiaries are still stigmatized as orphans or impoverished children, specifically adolescents receiving ASC meals and food parcels. The extent to which this deters beneficiaries from accessing services is unclear but not significant it would appear. However, the objective of developing a supportive culture in the community is undermined. Community norms in conflict with Heartbeat’s human rights based approach Heartbeat's rights-based approach may be in conflict with accepted community practices. In KwaJobe for example, corporal punishment is widely accepted. This conflicts with Heartbeat’s education of its OVC beneficiaries about their rights; when OVC assert these rights in the home, it can lead to discord within the household. This consequence is an unintended but significant result of implementation not thoroughly considered and focused exclusively at OVC, rather than inclusive of the family and community. Lack of security at Heartbeat center sites Another contextual barrier is the insecure environments that some sites operate in. A number of break-ins have been reported in Botshabelo and Nellmapius, disrupting the services as essential resources such as food and educational equipment are stolen. Heartbeat UGM End of Project Evaluation 43 V. CONCLUSIONS This section summarizes the evaluation findings relating them to the purpose of the evaluation, and highlights implications for future interventions. THE EXTENT TO WHICH HEARTBEAT’S OBJECTIVES HAVE BEEN ACHIEVED  Heartbeat, with PEPFAR funding, was able to meet the organizational objectives of providing and protecting the rights of the OVC in psychosocial, intellectual, and access areas. The psychosocial, educational, and access to rights-based services activities contributed to the achievement of Heartbeat’s objective of improving the quality of life of OVC.  The PEPFAR-funded activities did not support the Heartbeat objective of providing for physical rights, although these are indicated to be the prioritized needs of the OVC (with regards to school uniforms, stationery, and food). Although Heartbeat provided for physical rights with other funds, the findings suggest that it was not sufficient as physical needs remain the most reported need among the OVC.  Heartbeat’s objective of developing a culture of support for OVC within communities was partially met through the incidental effects of benefits resulting from services offered to OVC. However, Heartbeat was not intentional in providing services to the family or the community that would have been instrumental in achieving these objectives, such as family support groups and community programs.  The objective of ensuring quality project management and maximum service impact was obstructed by the ambiguous organizational structure of Heartbeat, which has resulted in confusions and delays in implementation processes.  A more rounded approach that incorporates all the rights of the children, community participation, and quality project management is necessary if all of Heartbeat’s objectives are to be achieved. PROGRAM OUTCOMES RELATED TO THE WELL-BEING OF OVC  Various outcomes related to the well-being of OVC were realized under the PEPFAR funding and the interventions providing educational, psychosocial, and access to rights-based services support. Examples include better school performance, improved life skills, access to social grants and legal documents, increased knowledge of HIV/AIDS prevention, and improved health.  The most significant change observed in the OVC is the improvement in school performance.  The most valued activities that Heartbeat provides were reported to be provision of food and psychosocial support.  The PEPFAR-funded activities also contributed to some changes at family and community levels, the major ones being that both the family and the community understand the OVC and care for them better. PROGRAM ENABLERS AND BARRIERS  The key enablers to achieving objectives were reported to be free services, community involvement, a stigma-free environment, working with partners, and a sound marketing strategy. If the Heartbeat program is to continue successfully and if similar services are to be duplicated, such key enablers should be sustained regardless of type of area or geographic location, as they were reported to have worked in all the sites.  The major barriers were reported to be the difficulty of traveling to the ASC and of CCWs going into households because of the long distances, and the problems with the organizational structure to include communication problems, unclear reporting lines, and staff shortages. Successful continuation and duplication of services relies on addressing these key barriers in all environments. However, in rural areas such as KwaJobe, cultural values and practices should also be acknowledged and negotiated if impact is to be realized.  An effective strategy that can be observed is providing PEPFAR-funded programs that are closely aligned to those of Heartbeat. Heartbeat UGM End of Project Evaluation 45 VI. RECOMMENDATIONS This section collates recommendations directed at organizational structure, program design, and implementation strategies based on the findings of this evaluation. ORGANIZATIONAL STRUCTURES AND PROCESSES  Review causes of under-performance at Heartbeat headquarters undermining consistent program implementation. It is apparent from the findings that the delayed implementation of programming is a significant challenge, the causes of which did not emerge clearly from the evaluation. In order to resolve these challenges, the causes need to be identified.  Improve communication from Heartbeat headquarters to Heartbeat centers, specifically with regards to program implementation decisions. Center staff consistently report dissatisfaction with communication. Heartbeat centers are unclear on the reasons for delays in execution of planned activities and the discontinuation of others. They also criticize what is perceived to be exclusion from the process of decision making concerning operations and the belated communication of instructions that are implemented with immediate effect, without allowing time for preparing to manage the impact of decisions on beneficiary relationships.  Reform the organizational structures and processes at centers to ensure accountability for performance at center level. The flat management structure is not supporting operational efficacy, nor does it appear to be enhancing staff relations at center level. This change will prove difficult to execute because the perceptions of managers as equals is now so embedded. Without center-based reporting lines and accountability, performance will continue to be haphazard and relationships between managers will continue to be unnecessarily sensitive on operational matters.  Reform organizational structures and processes between headquarters and centers to resolve the inefficiencies introduced by multiple reporting lines. One center manager should be ultimately accountable to headquarters and a single, prioritized reporting process instituted. A center manager should also be the primary engagement and communication point for the headquarters–center interface. While engagement on matters that support implementation can continue to be multiple, accountability needs to be streamlined. PROGRAM DESIGN  Strengthen the consistency of food provision and material support activities. While dependency on funding and its parameters are recognized, it is apparent that food provision and material support are the activities that meet beneficiaries’ most urgently perceived needs. Furthermore, this activity functions as the entry point for recruiting OVC into programs, targeting more sustainable outcomes and the incentive for continuous participation by beneficiaries. It is indisputable that the strengthening and stabilizing of this program activity is an imperative for Heartbeat.  Address the causes of implementation delays and strengthen capacity to implement household economic strengthening activities. These activities address both the lack of sustainability inherent in material support and food provision activities, as well as the most frequently observed gap in programming—improving the future prospects of OVC that age out of social protection and Heartbeat. The causes for delays did not emerge clearly from the evaluation, thus this recommendation cannot be made more specific.  Introduce program activities that directly target families. While facilitating access to social protection directly benefits families, efforts at deliberate family programming have been very limited. In order to sustain the gains realized with OVC, their systemic context needs to be addressed. Family￾directed activities can sensitize caregivers to the outcomes OVC programs are attempting to achieve, and equip caregivers to execute supporting activities in the household. Family programs would also address potential conflicts in the household that emerge from a clash between children’s rights and cultural norms. It became apparent during the evaluation that the need for psychosocial support for the OVC’s household members—who are also affected by HIV/ AIDS—is urgently required.  Introduce program activities that promote community involvement in Heartbeat centers. The evaluation indicates that the establishment of a center provides a community with a mechanism for caring for their OVC, and in this way improves community members’ capacity to do so. Encouraging broader community involvement with the center, perhaps by hosting community events at the center, is required to maximize this improvement in capacity at community level. The objective of community targeting activities would be to augment the efforts to raise awareness of Heartbeat, to create opportunities for community members to participate in Heartbeat events, and to enhance Heartbeat’s status as a community institution that participates in community life. The CDF would take charge of these program activities.  Review the selection criteria and adjust to accommodate the realities of child vulnerability. Two realities specifically need to be accommodated.  The process of OVC aging out social protection and Heartbeat needs to be smoother; this can be accomplished by implementing the support for the tertiary education activity under HES and implementing similar activities, and raising the age limitation in alignment with these new program areas.  Vulnerability is not only found in children whose parents are absent from the household, thus this should not be a criteria for inclusion in Heartbeat programming. A more appropriate definition of vulnerable needs to be adopted that acknowledges this reality.  Enhance center capacity to cater to children in age groups under 8 years. Findings have revealed that Heartbeat services and activities across all sites do not cater for toddlers and pre-scholars. It is important for Heartbeat to develop activities suited for this age group and to source resources and staff that can be instrumental in implementing these activities. IMPLEMENTATION STRATEGIES  Introduce a mechanism for communicating significant changes to services and programming with beneficiaries and stakeholders. While Heartbeat prides itself on early and consistent communication with community stakeholders, significant changes are introduced promptly and without the pacing required to effectively manage the impact of changes on stakeholder relationships.  Increase awareness of Heartbeat services. Although Heartbeat has a communication strategy, it focuses on cultivating the referral system with partnering organizations. A communication strategy that also prioritizes community outreach would improve awareness of Heartbeat and uptake of services, as well as improve community capacity to care for its OVC (see recommendation in previous section entitled Introduce program activities that promote community involvement in Heartbeat centers).  Improve accessibility of Heartbeat services. This would be accomplished by augmenting outreach activities to beneficiary households, and two issues would have to be addressed.  The capacity of CCWs would have to be reviewed and the staff contingent for that function realistically increased in order to effectively meet outreach targets.  A solution for CCW transport that works consistently needs to identified and implemented. Options may include purchasing a center vehicle or contracting a service that drives a daily route for delivering and picking up CCWs, providing additional money for CCWs to use public transport, or providing CCWs with bicycles. Heartbeat UGM End of Project Evaluation 47 APPENDIX I: COMPOSITION OF THE EVALUATION TEAM Heartbeat Evaluation Team Composition Team Member Role Terence Beney Senior Evaluator, Technical Lead Elizabeth Zishiri Project Manager Fazeela Hoosen Evaluation Assistant Rebekah King Evaluation Assistant Mokete Mokone Field Worker Coordinator Mike Mashiyane Field Worker Lufuno Maitakhole Field Worker Lindiwe Mdhuli Field Worker Daleen Botha Contracts Management    APPENDIX II: DATA COLLECTION TOOLS USED Key Informant Interview Guide Background Information  Name of person(s) being interviewed:  Designation (current occupational role):  Role in Heartbeat:  Length of involvement in the Heartbeat program (yr): Goals and Objectives 1. What is the overall goal of the programs/services funded by PEPFAR under the FHI360-UGM project? 2. How do the goals of the FHI360-UGM project fit into the overall organizational vision/mission? 3. What are the key program activities and services offered for OVC under the UGM? 4. What is the rationale for providing these services for OVC? Program Design 5. How are the services structured to ensure the following: a. Access: b. Affordability: c. Child-centered (e.g. age appropriate): d. Stigma-free environment: Program Implementation 6. What factors have enabled/hindered the program to meet its objectives? (Probe on the following areas: Organizational structure, Program design, Implementation challenges, Partnerships, Community buy-in/ownership, Training, Resources) Program Outcomes and Impact 7. What changes in the lives of OVC have you observed in the following areas as a result of the program? 8. What other changes have been brought about by the program for OVC, families, and communities? 9. Did the changes brought about by the intervention last over time? 10. What structures and systems exist to ensure long term continuity? Other Comments or Questions 11. Do you have any other comments that you believe we should be aware of when conducting this evaluation? THANK YOU FOR TAKING THE TIME TO PROVIDE YOUR VALUABLE INPUTS Heartbeat UGM End of Project Evaluation 49 Program Staff Interview Guide Background Information  Name of person(s) being interviewed:  Designation (current occupational role):  Role in Heartbeat:  Length of involvement in the Heartbeat program (yr): Goals and Objectives 1. When did you become involved with caring for OVC? 2. Tell me the story of how the center started? 3. What is the overall goal of the programs/services funded by PEPFAR under the FHI360-UGM project? 4. How do the goals of the FHI360-UGM project fit into the overall organizational vision/mission? 5. What are the objectives of the FHI360-UGM project? Program Design 6. What are the key activities offered to OVC and their families? 7. What is the rationale for providing these services for OVC? 8. Which of these activities do you believe has the most influence on OVC well-being? Why do you say so? 9. How are the services structured to ensure the following: a. Access: b. Affordability: c. Child-centered (e.g. age appropriate): d. Stigma-free environment: Support from Heartbeat 10. What training or support have you received from Heartbeat? 11. What are you personally able to do now that you could not do before the training and support from Heartbeat? 12. Have you increased your knowledge or skills? If yes, in what? 13. How would you describe your confidence in your ability to take care of OVC in your community since receiving training? 14. How would you rate the quality of the training and support provided by Heartbeat? Program Objective Achievement 15. What progress has been made in achieving the objectives of FHI 360-UGM project? Probe for each objective under the UGM. 16. What factors have enabled/hindered the program to meet its objectives? (Probe on the following areas: Organizational structure, Program design, Implementation challenges, Partnerships, Community buy-in/ownership, Training, Resources) Program Outcomes and Impact 17. What changes have been brought about by the program for OVC, families, and communities? 18. Did the changes brought about by the intervention last over time? 19. What role did the Child Care Workers play in achieving these changes in OVC well-being?  PROBE: How much of the changes observed in OVC, families, and the community can be attributed to Child Care Workers and other Heartbeat staff?  PROBE: How much of the changes observed in OVC, families, and the community can be attributed to the activities that take place at the After School Center? 20. What structures and systems exist to ensure long term continuity? Other Comments or Questions 21. What else should be done in future to improve in the well-being of OVC and their families? 22. Do you have any other comments? THANK YOU FOR TAKING THE TIME TO PROVIDE YOUR VALUABLE INPUTS Heartbeat UGM End of Project Evaluation 51 Community Stakeholders Interview Guide Background Information  Name of person(s) being interviewed:  Designation (current occupational role): Background 1. Describe the challenges faced by OVC in this community? 2. What role does the Heartbeat’s center (ASC) play in this community? Activities 3. What activities does the Heartbeat provide to OVC and their families? 4. Do you think these activities are meeting the needs of OVC in the community? 5. Which activities do you believe have the most influence on OVC well-being? 6. Do you know how Heartbeat’s activities are structured to ensure: a. Access: b. Affordability: c. Child-centered (e.g. age appropriate): d. Stigma-free environment: Program Implementation 7. What factors have enabled/hindered the program to meet its objectives? (Probe on the following areas: Organizational structure, Program design, Implementation challenges, Partnerships, Community buy-in/ownership, Training, Resources) Program Outcomes and Impact 8. What changes have been brought about by the program for OVC, families, and communities? 9. Did the changes brought about by the intervention last over time? 10. How important were child care workers in achieving these changes? PROBE: Would these changes have occurred without the services and support that child care workers’ provide to OVC and their families? 11. What changes have you observed in the community’s ability to care for OVC? 12. To what do you attribute these changes? PROBE: Has Heartbeat had any impact on these changes? 13. What has helped Heartbeat to make these changes in the lives of OVC? 14. What challenges has Heartbeat faced in addressing the needs of OVC? Other Comments or Questions 15. What else should be done in future to improve in the well-being of OVC and their families? 16. Do you have any other comments that you believe we should be aware of when conducting this evaluation? THANK YOU FOR TAKING THE TIME TO PROVIDE YOUR VALUABLE INPUTS Child Care Workers Focus Group Guide Background 1. When did you first become involved in caring for OVC? 2. What is the mission of Heartbeat? PROBE: Was it before or after you became involved with Heartbeat? Activities 3. What activities do you provide for OVC and their families? 4. Why does Heartbeat provide these activities? 5. How do you ensure that the services you provide are: a. Access: b. Affordability: c. Child-centered (e.g. age appropriate): d. Stigma-free environment: Support from Heartbeat 6. What training or support have you received from Heartbeat? 7. What are you personally able to do now that you could not do before the training, support from Heartbeat? 8. Have you increased your knowledge or skills? If yes, in what? 9. How would you describe your confidence in your ability to take care of OVC in your community since receiving training? 10. How would you rate the quality of the training and support provided by Heartbeat? Program Outcomes and Impact 11. What changes have been brought about by the program for OVC, families, and communities? 12. Where the changes immediate or did they take time? 13. What has helped you to make these changes in OVC and their families? 14. Did the changes brought about by the intervention last over time? 15. What challenges have you faced in improving the well-being of OVC? Other Comments or Questions 16. What else should be done in future to improve in the well-being of OVC and their families? 17. Do you have any other comments that you believe we should be aware of when conducting this evaluation? THANK YOU FOR TAKING THE TIME TO PROVIDE YOUR VALUABLE INPUTS Heartbeat UGM End of Project Evaluation 53 Primary Caregiver Focus Group Guide 1. Tell me what you know about the activities at this center? 2. How did you find out about this center? What made you get involved? 3. What activities does your child participate in through this center? FACILITATOR: Take a count for each activity/service. 4. Before your child was involved with the center, what concerned you most about taking care of them? 5. How has the center helped you to deal with these concerns? 6. How is your child’s life different since coming to the center? 7. Where the changes immediate or did it take time? 8. How much did the work of the child care worker contribute to these changes? PROBE: Would these changes have occurred without the services and support provided by the child care worker? 9. How much did the work of the staff at Heartbeat (at the center) contribute to these changes? PROBE: Would these changes have occurred without the services and support provided by the child care worker? 10. Have these changes lasted over time? Why or Why not? PROBE for each area of change. 11. How has your child’s involvement at the center impacted the rest of the family? PROBE: Positive and negative impacts 12. Does your child like coming to the center or participating in activities? Why or why not? PROBE: What do you think they enjoy most? 13. How do the staff and volunteers interact with the children? 14. Are the activities offered appropriate for children? 15. Do you have any trouble accessing the activities? PROBE for issues with location, transport 16. What else should be done in future to improve the well-being of OVC and their families in your community? 17. Are there any comments that you would like to make? OVC Focus Group Guide 1. Tell me of the activities offered at this center? 2. How did you find out about this center? What made you get involved? 3. What activities do you participate in though this center? FACILITATOR: Take a count for each activity/service. 4. Worries and concerns activity Part 1. Now, we will do an activity that will help us to understand more about your concerns and worries. We have passed out two different colors of Post-Its or Sticky Notes. Take a few of each color. On the YELLOW Post-Its, write the answer to the following question. Before you were involved in activities at the center, what were some of the challenges in your life? Write only ONE worry or concern per Post-It/Sticky Note. When you are finished put all of your Post-Its on the big paper marked CHALLENGE. Follow-up Questions Part 2. Please think about your life now, after you’ve been participating in the activities as this center. On the PINK Post-Its or Sticky Notes, please write the answer to the following question. When you are finished put all of your Post-Its on the flip chart paper marked HOW THE CENTER HELPED. How has the center helped you to deal with these challenges/problems? [FACILITATOR]: Ask some participants to share what they wrote down. [FACILITATOR]: Summarize the challenges noted on the flip chart papers. 5. How is your life different since you became involved with the center? 6. Of the changes you’ve mentioned, which do you think are the greatest or most significant? 7. Greatest change activity This activity will help us to understand which activities had the greatest impact on the changes in your life. Part 1. We are passing out a packet of cards. Each card is labeled with a program or activity. As group, identify select the cards for the programs, activities you have participated in through the center. [FACILITATOR] Check in with group after 2 minutes. Part 2. Now, using the cards that are left, rank the programs and services in order based on how much they influenced the changes in your life since coming to the center. The activity that had the greatest impact on the changes in your life should be ranked first, while the programs or activities that had the least impact should be ranked last. D7.1. Tell me about how you ranked the activities. D7.2. Why were these activities more important to the changes in your life? [FACILITATOR]: Probe participation for consensus and agreement. 8. Have these changes lasted over time? Why or Why not? PROBE for each area of change. 9. How has your involvement at the center impacted the rest of your family? PROBE: Positive and negative impacts 10. Do you like coming to the center or participating in activities? Why or why not? What do you enjoy most? 11. How do the staff and volunteers at the center interact with you? PROBE: Do you think you are treated with care and respect? 12. Are the programs and services offered suitable for your age group? Heartbeat UGM End of Project Evaluation 55 13. Do you have any trouble accessing the programs and services? PROBE for issues with location, transport 14. What else should be done in future to improve the lives of young people in your community? 15. Are there any comments that you would like to make that we didn’t touch on during the discussion? CONSENT FORMS HEARTBEAT STAFF CONSENT FORM Project: FHI 360 Umbrella Grants Management (UGM) Partner Evaluation for Heartbeat Researcher(s): Elizabeth Zishiri, Fazeela Hoosen Dear Heartbeat Staff: Feedback Research and Analytics (FeedbackRA), a Pretoria based research company, has been contracted by FHI 360 to conduct an evaluation of Heartbeat’s programs and services under the Umbrella Grants Management project (UGM). As part of this evaluation, FeedbackRA will conduct interviews with Heartbeat staff to learn how programs and services are delivered and their benefit to orphaned and vulnerable children (OVC), their families, and communities. You have been identified to participate in a 60-75 minute in-depth interview. The interview will be audio-recorded. You will be asked about the needs of OVC, Heartbeat service delivery, program outcomes and impact, the sustainability of services, and lessons learnt. The interview will be led by a researcher from Feedback RA and will be held at your office. The information shared will help in determining the impact of PEPFAR funding for OVC services and identify the best ways to care for and support OVC in future. The knowledge generated will be utilized by USAID, the South African government and Heartbeat. Please note that the outcomes of the evaluation will not be utilized to determine future funding. Your participation in the interview is completely voluntary. You can decide not to participate at any time without any reason or explanation. Your participation in the interview will have no effect on your relationship with other organizations or people involved in the UGM project. All of the information you share during the interview will be kept confidential. Your name will not be linked to any information you provide or used in any reports. No one outside of the evaluation team at Feedback RA will have access to the information you share. There are no direct benefits to your participation in the in-depth interview, but the information provided may be helpful to others in future. There are no known risks involved with your participation in the interview. If you have any questions or concerns about the evaluation or the interview, please feel free to contact the Project Manager, Elizabeth Zishiri, at 012-430-2009 or at ezishiri@feedbackra.co.za . If you understand these conditions and agree to participate, complete the section below. I _______________________________________________________ hereby give consent to participate in an audio recorded interview for the FHI 360 UGM Partner Evaluation of Heartbeat. _______________________________________________________________________ (SIGNATURE) Signed at _________________________________ on this __________ day of _____________ 2012. Heartbeat UGM End of Project Evaluation 57 COMMUNITY STAKEHOLDER CONSENT FORM Project: FHI 360 Umbrella Grants Management (UGM) Partner Evaluation for Heartbeat Researcher(s): Elizabeth Zishiri, Fazeela Hoosen Dear Community Stakeholder: Heartbeat is a national organization that provides care and support services to Orphaned and Vulnerable Children (OVC) and their families in your community. The program is funded by USAID through the FHI 360 Umbrella Grants Management Project (UGM). USAID and FHI 360 would like know the impact of these program and services on OVC and their families in your community. Feedback Research and Analytics, a research company, has been asked by FHI 360 to conduct an evaluation of Heartbeat’s programs and services. As part of this evaluation, Feedback RA will conduct interviews with community stakeholders to learn more about how programs and services are delivered and their benefit to OVC, their families, and communities. The knowledge generated will be utilized by USAID, the South African government and Heartbeat. Please note that the outcomes of the evaluation will not be utilized to determine future funding. You have been identified to participate in a 60–90 minute, in-depth or group interview. The interview will be audio-taped. You will be asked about needs of OVC, services, and the impact of their on OVC well-being and the sustainability of services provided by Heartbeat. The interview will be led by a researcher from Feedback RA and will be held at your office or a nearby venue. The information shared will help to identify the best ways to care for and support OVC. The knowledge generated will be utilized by USAID, the South African government and Heartbeat. Please note that the outcomes of the evaluation will not be utilized to determine future funding. Your participation in the interview is completely voluntary. You can decide not to participate at any time without any reason or explanation. Your participation in the interview will have no effect on your relationship with the other organizations or people involved in the program. All of the information you share during the discussion will be kept confidential. Your name will not be linked to any information you provide or used in any reports. No one outside of the research team at Feedback RA will have access to the information you share. There are no direct benefits to your participation in the focus group discussion, but the information gathered during the interview may be helpful to others in future. There are no risks involved with your participation in the interview. If you have any questions or concerns about the focus group, please feel free to contact the Project Manager, Elizabeth Zishiri, at 012-430-2009 or at ezishiri@feedbackra.co.za. If you understand these conditions and agree to participate, complete the section below. I _______________________________________________________ hereby give consent to participate in an audio recorded interview for the FHI 360 UGM Partner Evaluation of Heartbeat. _______________________________________________________________________ (SIGNATURE) Signed at _________________________________ on this __________ day of _____________ 2012. PRIMARY CAREGIVERS CONSENT FORM Project: FHI 360 Umbrella Grants Management (UGM) Partner Evaluation for Heartbeat Researcher(s): Elizabeth Zishiri, Fazeela Hoosen Dear Caregiver: Heartbeat is a national organization that provides care and support services to Orphaned and Vulnerable Children (OVC) and their families in your community through a local center. The program is funded by USAID through the FHI 360 Umbrella Grants Management Project (UGM). USAID and FHI 360 would like know the impact of these program and services on OVC and their families in your community. Feedback Research and Analytics, a research company, has been asked to conduct an evaluation of Heartbeat’s programs and services. As part of this evaluation, Feedback RA will hold focus group discussions with other caregivers to learn more about how programs and services offered by Heartbeat have benefited OVC, their families, and communities. You have been identified to participate in a 90 minute focus group discussion. The focus group will be audio-taped. You will be asked about your child’s experience with Heartbeat programs and services. The focus group discussion will be led by a facilitator from Feedback RA and will be held at a venue in your community. The information shared will help to identify the best ways to care for and support OVC. Your participation in the focus group is completely voluntary. You can decide not to participate in the focus group at any time without any reason or explanation. Your participation in the focus group will have no effect on your relationship with the organizations or people involved. All of the information you share during the discussion will be kept confidential. Your name will not be linked to any information you provide or used in any reports. No one outside of the research team at Feedback RA will have access to the information you share. There are no direct benefits to your participation in the focus group discussion, but the information gathered during the focus group may be helpful to others in future. There are no risks involved with your participation in the focus group. If you have any questions or concerns about the focus group, please feel free to contact the Project Manager, Elizabeth Zishiri, at 012-430-2009 or at ezishiri@feedbacra.co.za. If you understand these conditions and agree to participate, complete the section below. Age of your child: __________ Gender (circle correct answer): M / F Time on the program: ________ I_________________________________________________ hereby give consent to participate in an audio recorded focus group discussion for the FHI 360 UGM Partner Evaluation of Heartbeat. _______________________________________________________________________ (SIGNATURE) Signed at _________________________________ on this _________ day of _____________ 2012. Heartbeat UGM End of Project Evaluation 59 YOUTH FOCUS GROUP CONSENT FORM Project: FHI 360 Umbrella Grants Management (UGM) Partner Evaluation for Heartbeat Researcher(s): Elizabeth Zishiri, Fazeela Hoosen Dear Youth: Heartbeat is an organization that helps youth around South Africa who are Orphaned or Vulnerable. A company that gives Heartbeat money would like to know if what they do is helping to make young people’s lives better. Feedback Research and Analytics, a research company, has been asked to study Heartbeat. As part of this study, Feedback RA will hold focus group discussions with young people involved with Heartbeat to find out if they have benefited from the programs and activities offered. You have been selected to participate in a 90 minute focus group discussion with other young people. The discussion will be tape recorded. You will be asked about your experience with Heartbeat programs and activities. The focus group discussion will be led by someone from Feedback RA and will be held in your community. The information shared during the discussion will help to identify the best ways to help youth who are Orphaned or Vulnerable. Your participation in the focus group is completely voluntary. You can decide not to participate at any time and you do not have to give a reason. Your participation in the focus group will have nothing do with your relationship with Heartbeat or other people involved in the programs and activities. What you say during the discussion will be kept private. Your name will not be linked to anything you say or used in any reports. No one outside of Feedback RA will know what you said. There are no benefits to your participation in the focus group discussion, but the information from the focus group may be helpful to others in future. There are no risks involved with you taking part the focus group. If you have any questions about the focus group, you can call the Project Manager, Elizabeth Zishiri, at 012-430-2009 or at ezishiri@feedbackra.co.za If you understand these conditions and agree to participate, complete the section below. Age: __________ Gender (circle correct answer): M /F Time on the program: _________ _____________________________________________________________ will take part in a 90 minute focus group discussion about my involvement with Heartbeat in my community. . _______________________________________________________________________ (SIGNATURE) Signed at _________________________________ on this __________ day of _____________ 2012. APPENDIX III: LIST OF SITES VISITED AND DATES OF VISITS Dates and Sites visited Date Activity April 23 Travel to KwaJobe (KwaZulu-Natal) April 24–25 Field work KwaJobe (KwaZulu-Natal) May 2 Travel to Botshabelo (Free State) May 3–4 Field work Botshabelo (Free State) May 7–8 Field work Nellmapius (Gauteng May 9–10 Field work Vosloorus (Gauteng) Fieldwork Schedule DAY 1 Team: Lead 1, Lead 2, Fieldworker 1, Fieldworker 2 8:00 – 9:00 Travel to Site 9:00 – 9:30 Logistics, Set up 9:30 – 11:00 Group Interview – Care Workers (1 Facilitator, 1 Assistant) Interview with Social Worker (1Facilitator, 1Assistant/Note taker) 11:00 – 11:30 Open for Logistics 11:30 – 1:00 Focus Group with Primary Caregivers/Guardians (OVC ages 0 – 11) (1 Facilitator, 1 Co-facilitator, 2 Note takers/ Assistants) 1:15 – 2:45 Focus Group with Primary Caregivers/Guardians (1 Facilitator, 1 Co-facilitator, 1 Assistant) Interview with CDF (1 Facilitator) 3:00 – 4:30 Focus Group with OVC 12-18 (1 Facilitator, 1 Co-facilitator, 2 Assistants) 4:30– 5:00 Team Debrief and Wrap up DAY 2 9:00 -10:30 Interview with Choza (1 Facilitator, 1Assistant/Note taker) Interview with Team leader (1Facilitator, 1 Assistant/Note taker) 10:30 – 11:00 Open for Logistics 11:00 – 12:00 Community Stakeholder Interview # 1 (1 Lead, 1 Fieldworker) Community Stakeholder Interview # 2 (1 Lead, 1 Fieldworker) 12:00 -1:00 LUNCH (can be used for data collection if necessary) 1:00 – 2:00 Program Staff In -depth Interview (1 Lead, 1 Fieldworker) Program Staff In-depth Interview (1 Lead, 1 Fieldworker) if necessary 2:00 – 3:00 Community Stakeholder Interview # 3 (1 Lead, 1 Fieldworker) Community Stakeholder Interview # 4 (1 Lead, 1 Fieldworker) 3:00 – 4:30 Focus Group with OVC ages 12-18 (1Facilitator, 1 Co-facilitator, 2 Note takers/ Assistants) 4:30 – 5:00 Team Debrief and Wrap up Heartbeat UGM End of Project Evaluation 61 APPENDIX IV: SCOPE OF WORK Now in the fifth and final year of the project, FHI 360-UGM, at the request of USAID, is commissioning an external evaluation of our grantees. Partner organizations are non-governmental organizations (NGOs) working at national, provincial and local levels in South Africa, primarily implementing services related to services for orphans and vulnerable children (OVC), HIV care and support, HIV counseling and testing, and HIV prevention. These partners have received funding for a period of three to five years under PEFPAR, as well as both organizational and technical capacity building support. Feedback Research and Analytics is being contracted to execute evaluations for two of the UGM Partners: Heartbeat and Noah. The focus of the each partner evaluation will be to:  Determine whether the program objectives under each partner’s program were achieved  Evaluate the key program outcomes and impacts related to improved health and wellbeing of the targeted beneficiaries Most specifically, Feedback Research and Analytics will seek to answer the following key evaluation questions for Heartbeat, utilizing tools, methods, and sub-questions approved reviewed and approved by FHI 360:  What were the most significant changes brought about by Heartbeat improving the well-being of OVC in targeted communities?  To what extent was Heartbeat able to address the needs of children within the community?  How do stakeholders (children, care givers, DoSD, community representatives) perceive the program; in terms of quality and ease of access? The focus of the evaluation is to assess effectiveness of the partner organizations in addressing the needs of beneficiaries in targeted communities. The evaluators will be required to carefully consider the suitability and feasibility of design options that are likely to offer the best chance of establishing the value of the program in responding to the needs of targeted beneficiaries and communities. Both qualitative and quantitative data collection techniques should be employed. Data will be collected from various sources using appropriate data collection methods and tools for any given evaluation question. The final design to be employed will be determined after the contractor has had a chance to undertake a front-end analysis and is therefore able to select the best design option that specifies; which people or units will be studied; how they will be selected and the kinds of comparison that should be made. Data will be collected from various program sites for each partner. Evaluations will be undertaken in two stages and with expected outcomes for each stage as expressed below: Stage 1: Finalization of Evaluation Protocol Contractor will refine an evaluation protocol which demonstrates:  Understanding the relationship between program stages and the proposed broad evaluation question  Understanding the context for program delivery and key factors that influence program implementation  Understanding the existing theoretical and empirical knowledge about the program and examining program theory  A comprehensive stakeholder analysis and determination of roles of key stakeholders in the evaluation  Balancing costs and benefits of the evaluation and advising on the most strategic questions to include in the evaluation  Developing the Finalized Implementation Strategy and Methodology Report Stage 2: Implementation of the Evaluation Contractor will implement the partner evaluation following submission and approval of the Implementation Strategy and Methodology Report:  Pre-test instruments  Train data collectors  Undertake the evaluation data gathering process  Prepare data for analysis  Clean data  Enter data into electronic data analysis systems  Undertake comprehensive data analysis  Formulate the findings During the period of performance of April 9, 2012 – July 31, 2012, payment to Feedback Research & Analytics will be fixed price based on the payment schedule determined by the deliverables below: MILESTONES DELIVERABLES DUE DATE Data collection (April 9 – May 11) 1. Finalized Implementation Strategy and Methodology Report submitted to FHI 360 (Heartbeat) April 13 2. Evaluation Work Plan, including key activities and timeframes submitted to FHI 360 (Heartbeat) April 13 3. Data Analysis Plan, including dummy table/graphs for presenting data submitted to FHI 360 (Heartbeat) May 4 Data analysis and development of PowerPoint Presentation, including summary of evaluation process and results (May 14 – June 1) 4. Oral and PowerPoint Presentation (half-day) of preliminary findings to USAID, FHI 360, and partner (May 28- June 1, 2012) (Heartbeat) Presentations completed by June 1 Development of final written report, including an executive summary with highlights of the evaluation and key findings (June 1 – July 31) 5. Draft written report submitted to FHI 360 (Heartbeat) June 15 6. Final report submitted to FHI 360 (Heartbeat) July 31 Development of brief paper (two-pager) for each partner, targeting community audiences on key findings from the evaluation (June 1 – July 31) 7. Two-page papers submitted to FHI 360 (Heartbeat) July 31 Heartbeat UGM End of Project Evaluation 63 APPENDIX V: REFERENCES Chene, Swart. Impact Analysis: Heartbeat Model of Care. Murray and Roberts, Rockefeller Brothers Fund, 2007. FHI360. Where we work: FHI360. http://www.fhi360.org/en/CountryProfiles/SouthAfrica/res_SA_UGM.htm (accessed June 12, 2012). Heartbeat. About Us: Heartbeat. http://www.heartbeat.org.za (accessed June 12, 2012). Heartbeat. Heartbeat Draft Final Report. 2010. Heartbeat. Heartbeat Draft Final Report. 2011. Khulisa Management Services. "A Case Study: Heartbeat." Southern Hemisphere Consultants. Program framework, program theory and results chain for Heartbeat Program. 2011. Tulane University. A Case Study: Tswelopele Partnerships for Reaching Vulnerable Children in South Africa with the Heartbeat Model of Care. 2011.