USAID/South Africa Umbrella Grants Management Project End of Project Partner Evaluation NOAH (NURTURING ORPHANS OF AIDS FOR HUMANITY) June 2012 This publication was produced for the United States Agency for International Development. It was prepared by Rebekah King of Feedback Research and Analytics. USAID/South Africa Umbrella Grants Management Project End of Project Partner Evaluation Noah (Nurturing Orphans of AIDS for Humanity) This report is made possible by the generous support of the American people through the U. S. Agency for International Development (USAID), under the terms of APS 674-07-011, through United States Government’s President’s Emergency Plan for AIDS Relief, operated by FHI 360. The contents and opinions expressed herein do not necessarily reflect the views of USAID or the United States Government. Noah UGM End of Project Evaluation i ACKNOWLEDGEMENTS  Feedback RA would like to thank the partner organization, Noah, for their contributions to the implementation of this evaluation including their assistance with sampling, access to secondary data, planning and logistics of field work, and participation as key informants.  The evaluation team would also like to thank the Ark staff, community stakeholders, and beneficiaries for participating in the evaluation and providing valuable feedback on their experiences at Arks.  Finally, the author would like to acknowledge the members of the evaluation team for the assistance with the design, implementation, and management of the evaluation and development of the draft report. The evaluation team included Terence Beney, Mokete Mokone, Washington Dudu, Pule Goqo, Elizabeth Zishiri, Fazeela Hoosen, and Daleen Botha. Noah UGM End of Project Evaluation ii TABLE OF CONTENTS I. INTRODUCTION ....................................................................................... 1 II. BACKGROUND .......................................................................................... 3 III. METHODS .................................................................................................... 6 IV. FINDINGS .................................................................................................. 12 V. CONCLUSIONS ........................................................................................ 30 VI. RECOMMENDATIONS ............................................................................ 32 APPENDIX I: EVALUATION SCOPE OF WORK ........................................... 34 APPENDIX II: DATA COLLECTIONS TOOLS ............................................... 36 APPENDIX III: EVALUATION WORK PLAN ................................................. 49 APPENDIX IV: SAMPLING FRAMEWORK ...................................................... 50 APPENDIX V: COMPOSITION OF EVALUATION TEAM ........................... 51 APPENDIX VI: NOAH ORGANOGRAM .......................................................... 52 APPENDIX VII: TYPICAL ARK STRUCTURE ................................................. 53 APPENDIX VIII: DEMOGRAPHIC PROFILE OF BENEFICIARIES FOCUS GROUP PARTICIPANTS ..................................................................... 54 APPENDIX IX: SUMMARY OF DATA ANALYSIS APPROACH .................. 55 APPENDIX X: OVC SELF REPORT OF PARTICIPATION IN ARK ACTIVITIES AND SERVICES .................................................................... 56 APPENDIX XI: PERSONS CONTACTED ........................................................ 57 REFERENCES ........................................................................................................ 58 Noah UGM End of Project Evaluation iii ACRONYM LIST ADO Ark Development Officer BBI Bulk Buying Initiative CAC Child Activity Coordinator CBO Community-based Organization COP Country Operational Plan DSD Department of Social Development, Republic of South Africa ECD Early Childhood Development ID South African Identity Document KZN Kwa-Zulu Natal NGO Non-governmental Organization Noah Nurturing Orphans of AIDS for Humanity OVC Orphans and Vulnerable Children PEPFAR President’s Emergency Plan for AIDS Relief RFP Request for Proposals SAG South African Government SAPS South African Police Service SOW Scope of Work UGM Umbrella Grants Management project USAID United States Agency for International Development Noah UGM End of Project Evaluation iv EXECUTIVE SUMMARY INTRODUCTION UGM is a five year grants management program administered by FHI 360 with funding from the U.S. Agency for International Development (USAID). Through UGM, FHI 360 provides funding and technical assistance to USAID-selected NGO partners who provide HIV/AIDS services at local, provincial, and national levels in South Africa. FHI 360-UGM seeks to promote high quality service delivery in alignment with the priorities and goals of the South African government’s HIV/AIDS framework. FHI 360-UGM provides specialized capacity building and support services to build partners’ skills and competencies in program management, governance, human resource development, budgeting and finance, and monitoring, evaluation, and reporting. The FHI 360-UGM project objectives are to 1. Provide grants to USAID/PEPFAR partners that ensure an adequate resource flow to foster scale-up of activities 2. Implement effective monitoring, evaluation, and reporting systems to assess and document activities 3. Provide ongoing capacity building to support and enhance scale-up of activities, and sustainability of activities and partners Since 2007, FHI 360-UGM has supported thirteen South African NGOs including Nurturing Orphans of AIDS for Humanity (Noah). The organization has received total funding through the UGM of R43,466,191. Feedback Research and Analytics (FeedbackRA) was contracted by FHI 360 to conduct an end of project evaluation of the Umbrella Grants Management project (UGM) partner Noah for the project period of 2007 to 2012. The purpose of the evaluation was to determine the extent to which the program objectives have been achieved; identify key program outcomes and impact related to the well￾being of orphans and vulnerable children (OVC), their families, and communities; and generate knowledge about strategies that are effective or ineffective in improving the lives of OVC. In support of this purpose, the evaluation sought to answer the following key questions  What were the most significant changes brought about by Noah improving the well-being of OVC in targeted communities?  To what extent was Noah able to address the needs of children within the community?  How do stakeholders (children, care givers, DoSD, community representatives) perceive the program; in terms of quality and ease of access? The key audiences for this report are USAID, FHI 360, Noah, and future funders or implementers of OVC care and support interventions. BACKGROUND Noah Noah is a not-for-profit organization established in 2000 to empower communities to take care of their orphans and vulnerable children. Noah works to realize this vision by partnering with communities to establish and successfully run community-based organizations (CBOs), also known as Arks, which support the well-being of OVC. Noah’s organizational objectives are to  establish and build the capacity of CBOs and committee members, staff, and volunteers to support and protect vulnerable children and their families in their communities across Gauteng and Kwa￾Zulu Natal (KZN) Noah UGM End of Project Evaluation v  provide direct services to children through the provision of a social franchise model of care for vulnerable children  maintain a core of dependent CBOs which provide good services to children but have not been able to fundraise due to marginalization of the communities in which they reside Noah is led by a CEO and eight senior staff responsible for the areas of finance; program development and implementation; monitoring, evaluation, and reporting; human resources; and relationships and communication. The organization is governed by a Board of Directors consisting of professionals from various sectors of the business community. Noah Program Model The Noah model includes two focus areas—establishing and building the capacity of local service centers known as Arks, and providing care and support services for OVC. Ultimately, by establishing and mentoring Arks through to organizational maturity, Noah intends to fulfil its vision of empowering communities to take care of their vulnerable children. Activities to mentor and build the capacity of Arks include organizational development and training, mentorship, and supervision of staff. Care and support services for OVC are delivered at Arks and include early childhood development (ECD), psychosocial support, home visits, child protection interventions (such as assistance with obtaining South African Identity Documents (IDs) and birth certificates, and assistance applying for social grants), health assistance activities, HIV/AIDS prevention education, educational support (including homework supervision, numeracy, and literacy classes) and aftercare. Although not covered under PEPFAR funding, Noah also implements a food security program which includes daily meals at Arks and the provision of food parcels. PEPFAR funds supported 42 Arks in Gauteng, Kwa-Zulu Natal and Northwest provinces. Noah’s PEPFAR-supported project objectives are outlined in Table 1. Table 1: Noah PEPFAR-Supported Objectives Noah PEPFAR-Supported Project Objectives To provide psychological care to OVC through bereavement counseling. To provide child protection services to OVC through assistance in obtaining birth certificates, ID documents, and access to child protection services. To conduct home visits to OVC and their families. To provide educational assistance to OVC in aftercare through implementation of a literacy program (Gauteng). To provide early childhood development programs. METHODS The evaluation was conducted using a mixed methods approach that relied primarily on an analysis of qualitative data obtained from staff, community stakeholders, and beneficiaries, and was supported by both quantitative and qualitative analysis of existing data. To ensure that a rich data set for identifying most significant changes was obtained and that the diversity of Arks was represented, six Arks were sampled. Sampling criteria included geographic location, degree of urbanization, Stage of Progression (a measure of an Ark’s capacity to sustain services independent of Noah, based on measures of performance), and type of services offered. Key components of the evaluation included partner engagement and document review; key informant interviews with five head office staff and one Board Member; in-depth interviews with 45 Ark staff, volunteer staff, and community stakeholders1; focus group discussions with 84 caregivers (guardians) of OVC ages 2–18; focus group discussions with 149 OVC ages 12–18; and review and analysis of secondary data. Informed consent was obtained from all participants. Structured interview guides were used to collect data on OVC needs and challenges, service delivery, quality of services, most significant                                                              1 Ark Managers were interviewed individually. Child activity coordinators, volunteers, and other staff were interviewed in groups, as it was suggested by Noah that these staff would be more comfortable talking in groups. Noah UGM End of Project Evaluation vi changes, most valuable activities, program barriers and enablers, sustainability, and lessons learned. Data were analyzed qualitatively using ATLAS.ti software. Demographic data from beneficiaries and monitoring data was captured and analyzed using descriptive statistical techniques in Microsoft Excel. FINDINGS To what extent was Noah able to address the needs of children within the community? Data gathered from key informants and beneficiaries confirms that OVC in target communities are overwhelmed by poverty and a lack of resources to meet their basic needs, emotional burdens that undermine their psychosocial well-being, difficulties with school performance, and vulnerability to exploitation and abuse provoked by the absence of robust familial care. It is further evident that for many beneficiaries, their generalized vulnerability is rooted in or exacerbated by the effects of HIV/AIDS in their families and communities. With PEPFAR support, Noah successfully implements crucial interventions in psychosocial support, educational support, and social protection that respond to these major challenges confronting OVC. Furthermore, Noah continues to directly address food insecurity through the provision of daily meals and food parcels, despite support for these interventions being discontinued by PEPFAR in 2010. Key informants and beneficiaries are insistent that not addressing unemployment and its implications for the future prospects of OVC represents a programming gap in Noah’s service offering. How do stakeholders perceive the program in terms of quality and ease of access? The assessment of the quality of Noah’s program was based upon beneficiary and key informant perceptions of the extent to which services were affordable, age appropriate, aligned with best practices, delivered in a stigma-free environment, and satisfactory to beneficiaries. With respect to affordability, age appropriateness, and alignment with best practice (such as those implied by the Children’s Act), the general consensus is that all Arks in the sample are providing high quality activities and services to vulnerable children and their families. Across all sites, caregivers and OVC were largely satisfied with Ark activities. The majority of OVC indicated that they liked coming to the center and enjoyed positive interactions with staff characterized by support, mutual trust, love, and care. Two issues undermining the perceived quality of the program were raised: accessibility remains a challenge for beneficiaries who live far from the Arks and for whom travel is problematic, often due to cost, and some respondents reported being victims of discrimination because they were observed attending the Ark and accessing services. It is apparent that the stigma associated with being impoverished, an orphan, and affected by HIV /AIDS in some manner persists to some extent in communities serviced by Arks. What were the most significant changes brought about by Noah in improving the well-being of OVC in targeted communities? The evaluation attempted to identify the most significant changes experienced at individual, family, and community level as a result of Noah’s interventions. The most significant perceived changes reported for OVC include better school performance, increased food security, greater knowledge of HIV/AIDS prevention, improved psychosocial well-being (including increased self confidence and the ability to better communicate and express personal feelings), improved life skills (such as discipline, responsibility, and decision making), and a greater sense of personal safety. As a result of Ark-sponsored food parcels, families of OVC benefit from improved food security. Other changes reported within families include increased happiness, better family relations (particularly between OVC and their siblings), and more cooperative children, which caregivers attribute to OVC applying at home the life skills they learned at the Ark. It is important to note that respondents did not consistently report any family level changes that improved capacity to care for OVC. At the community level, Ark staff, community stakeholders, and beneficiaries reported greater community awareness of the plight of OVC and the community’s active involvement in caring for their Noah UGM End of Project Evaluation vii OVC through the local Ark. Both results—increased awareness and a mechanism (the Ark) for delivering services to OVC—indicate an increased capacity to care for OVC at the community level. The activities and services respondents consider most valuable in achieving change include meals and food parcels, help with applying for ID or birth certificate, assistance with social grants, home visits, homework supervision, and counseling/play therapy. What were the key enablers and barriers in meeting project objectives? Evidence from the evaluation suggests that six factors contribute most significantly to the successful achievement of project objectives.  efforts undertaken to involve the community in both the establishment of a local Ark and maximizing the relevance of the Arks programs locally  evidence-based approach to designing and implementing programs and services, which not only includes adjusting programming based on community inputs on local needs, but adopting a foundation for program design based on childhood development theory  Noah’s investment in training Ark staff and developing the organizational capacity of local Arks  Provision of services free of charge at Arks  establishment of a vehicle – the Ark - through which communities are enabled to take care of their OVC  dedication and commitment of staff and their passion for children Further, there is evidence that these factors represent effective strategies for improving the well-being of OVC in wide range of communities (including peri-urban, informal settlement, township, and rural) with scarce resources. Evidence from the evaluation suggests that four factors most significantly undermine the successful achievement of project objectives. 1. While the importance of providing a distinct vehicle (the Ark) through which the community can take care of its OVC is confirmed in the evaluation, there are a number of challenges that emerge as a direct result of the center-based model that undermine effective delivery of services to beneficiaries. a. inadequacy of some facilities in terms of security, basic utilities, and amenities affects the safety, comfort, and predisposition towards interventions of the OVC being serviced b. inaccessibility of Arks for a portion of the OVC beneficiaries in a community because of their distant location from the center; inconsistent exposure to interventions and services effects the ability to achieve outcomes for these OVC 2. Persistent staff shortages and the Ark’s reliance on volunteers to support staff in service delivery should supplement staff shortages; however, because volunteers are hard to retain, staffing issues are compounded. 3. The key intended outcomes for Noah are to increase the capacity of both families and communities to take care of their OVC. There is clear evidence that the necessary prerequisite for improving community capacity is the establishment of a vehicle through which OVC can be cared for, which is at the heart of the Noah model (the Ark). Other than the Arks, the development of family and community capacity to care for OVC is not incorporated into Noah’s programming. 4. The Noah model includes a deliberate mechanism to support the maturation of individual Arks through stages towards a self-sustaining, autonomous organization (Stage 4). However, the final transition to a Stage 4 organization has proven to be consistently and inordinately difficult to achieve, largely as a result of the limited success of individual Arks to fundraise and the failure of some Arks to meet Department of Social Development (DSD) facility standards, which would qualify them for DSD subsidies. CONCLUSION Noah UGM End of Project Evaluation viii Findings indicate that Noah has met its project objectives by providing high quality essential care and support services that are responsive to the needs and challenges of beneficiary OVC. The PEPFAR￾funded services have contributed to significant changes being perceived by respondents in the well-being of OVC, most notably improved school performance, improved psychosocial well-being (such as, increased confidence and improved communication skills), improved life skills, and increased knowledge of HIV/AIDS prevention. The provision of meals and food parcels implemented by Noah through other funding has resulted in improved food security among beneficiary OVC. Very few respondents identified significant changes at the family and community level as a result of Ark activities; however, evidence suggests that the prerequisite for increasing community capacity to care for OVC is to offer the community a vehicle for the provision of such care, which Noah provides through the local Arks. Although not reported by participants, the establishment of the Ark, employment of Ark staff, and appointment of volunteers from the community represent some level of increased capacity within the community. An Ark provides a facility where services can be rendered by community members who have received extensive training and ongoing support from Noah on how to care for OVC. While the evaluation provides no evidence for the increased capacity of families to care for their OVC, there is evidence to demonstrate the positive effects of Ark interventions for OVCs’ families. Those responding indicated that OVC participation in Ark activities has improved the well￾being of families through access to food and improved family relations within the OVC’s household. Participants identified a range of critical factors that helped to facilitate the realization of these outcomes, including extensive efforts to secure community buy-in, staff training and organizational development support from Noah, adopting an evidence-based program design approach, and providing services on a no fee basis. Although not explicitly mentioned, it is apparent that establishing an Ark in a community is central to empowering the community to mobilize in the service of its OVC, offering a mechanism for sourcing and channelling resources, establishing partnerships, and recruiting dedicated and committed individuals from the community. There are key programmatic and contextual barriers that threaten the achievement of sustainable outcomes for OVC, including the poor amenities and inaccessibility of some Ark facilities, the persistent staff shortages and dependency of Arks on volunteers, the lack of program activities dedicated directly to improving the capacity of families to take care of their OVC, and the persistent difficulty in nurturing the maturation of Arks to a Stage 4 organization. RECOMMENDATIONS Based on the evaluation findings, the following are recommendations for Noah, Ark staff, and other implementers to improve existing OVC care and support programs, as well as enhance their scale-up and replication. Program Design  Intervene at the family level. Noah’s interventions directly develop community capacity to care for their OVC by establishing a local center or Ark for delivering services to OVC; however, the current model does not directly develop the capacity of families to care for their OVC in a systematic manner. As a result, no significant changes were reported at the family level that reflects increased capacity to care for OVC. To improve impact at the family level and ensure changes realized in OVC are reinforced and sustained, further deliberate intervention is required. As Noah implements its new Realignment Strategy, it should introduce a focus on programs and services to preserve and strengthen families directly. Interventions for the family would fit into Arks’ existing center-based and home visit service delivery mechanisms, both of which are presently focussed on delivering support to OVC directly and do not consistently include interventions targeting the family as beneficiary.  Analyze the hindrances to achieving Stage 4 classification for Arks and adjust programming accordingly. It is apparent from the evaluation data that while the staged progression of Arks is intended to embed sustainability into the Noah model, it is consistently problematic for Arks to achieve that level of autonomy. The evaluation was not able to identify all the specific factors that undermine the Noah UGM End of Project Evaluation ix achievement of Stage 4; a specific diagnostic evaluation is required to inform the necessary adjustments in program design that will more effectively facilitate the realization of this key program objective.  Augment outreach services as a strategy to extend service provision and mitigate the issues of accessibility. While there are clear advantages to a center-based model, it presents challenges with accessibility for some children, especially in rural and peri-urban communities. An assessment to determine the scope and nature of inaccessibility at these sites would help Arks to plan and deliver essential services in children’s homes. As the findings indicate, the services deemed most important by beneficiaries (help with accessing identification documents and social grants, home visits, homework supervision, and counseling) could be delivered in a home setting. This approach coincides with Noah’s planned strategic shift towards programming aimed at family preservation and household economic strengthening. In places where accessibility is an issue, services to OVC could be coupled with services to the family. Organizational Development  Investigate volunteer behavior to inform the development of a volunteer retention strategy. It is recommended that Noah investigate why their volunteers stop volunteering and explore the types of incentives that would motivate them to stay. Based on feedback from volunteers, Noah can identify low or no cost incentives for volunteers or determine the feasibility of providing remuneration for volunteers. The importance of the human resource challenge is accentuated in light of the intent of the Realignment Strategy to enhance the outreach service delivery mechanism, which implies the need to extend the human resource base as well.  Provide Arks with more intensive capacity building and support around resource mobilization. Despite training and support from an Ark Development Officer (ADO), Ark staff and committee members report continuing challenges with securing funding to sustain services for OVC over time. The ability to source funding is the key prerequisite for, and appears to be the primary hindrance to, achieving Stage 4 in the Ark progression model. Given these barriers, it is recommended that Noah incorporate a more intensive focus on resource mobilization into the revised Stage Progression process, provide supplementing training and mentoring in this regard as necessary, and provide Arks with opportunities to incrementally apply resource mobilization skills and competencies throughout the stage progression process. Program Implementation  Resolve the issues undermining the efficacy of nutrition interventions. Given the critical importance of food provision to outcomes and OVC participation in other Ark activities (demonstrated by the findings presented in this evaluation), the weaknesses of the Bulk Buying Initiative (BBI) and the revised program implemented in July 2011 should be urgently addressed. It is clear from nutrition evaluation reports that Noah is keenly aware of the urgency of the matter and is actively pursuing solutions to these challenges. The current efforts being pursued by Noah should remain cognizant of the need to determine a long term solution that adequately responds to nutritional needs of OVC, reflects the recommendations of Arks (including children), promotes empowerment, and is cost effective.  Adapt the current program model to improve accessibility in some rural and peri-urban sites. As noted earlier, the center-based model to provide services to OVC presents accessibility challenges for some children in rural and peri-urban communities. An assessment to determine the scope of inaccessibility would help the Arks to plan and deliver essential services to children in a home setting, if they are not able to attend the Arks.  Adopt a strategy to address the risks threatening the security of Arks and their beneficiaries. To help in addressing security concerns, Noah and the Arks should develop and implement a multi-pronged strategy that includes low cost measures, such as building relationships with relevant community structures (local police presence and community policing forums) and consistently including security concerns in the messaging directed at community members through the Arks’ participation in Noah UGM End of Project Evaluation x community forums. In addition, the security strategy should include planning for security expenditures such as hiring security staff, erecting fencing, and installing security systems.  Improve Ark facilities to meet eligibility criteria for DSD funding. Noah should work with the Arks to improve facilities and address security issues to satisfy DSD minimum standards and subsidy eligibility requirements. For existing Arks, this may include allocation of resources to make improvements to existing structures and the advocacy/engagement of government and relevant service providers to access basic services (such as water, sanitation, and electricity). For future Arks, DSD requirements should be used as guidelines for selecting or constructing facilities. Meeting DSD minimum standards has a direct bearing on Ark sustainability, as it makes the facility eligible for DSD subsidies. Noah UGM End of Project Evaluation 1 I. INTRODUCTION PURPOSE OF THE EVALUATION FeedbackRA was contracted by FHI 360-UGM to conduct an end of project evaluation of Noah for the period of 2007 through 2012. This report details the design, implementation, and findings of the evaluation of Noah’s care and support services to OVC and CBO development funded by USAID/PEPFAR. The purpose of the evaluation was to determine the extent to which the program objectives have been achieved and to identify key program outcomes related to the well-being of OVC, their families, and communities. The evaluation will also generate knowledge regarding program strategies that are most effective or ineffective in achieving desired outcomes. EVALUATION QUESTIONS The evaluation was guided by a series of questions identified by FHI 360 and Noah as relevant to assessing program performance. The key and sub-evaluation questions2 include 1. To what extent was Noah able to address the needs of children within the community? 2. How do stakeholders perceive the program in terms of quality and ease of access? 3. What were the most significant changes brought about by Noah in improving the well-being of OVC in targeted communities? 3.1 Which program services were most valuable to OVC, their families, and the community? 3.2 How did Noah and the Arks change the path of OVC through the interventions? 3.3 What role did the Arks play in achieving these outcomes/changes? What role did Noah play in mobilizing Arks to fulfill this role? 4. What were the key enablers and barriers to meeting project objectives? 5. Will Arks be able to continue supporting OVC in future? 5.1 What structures and systems exist to enable continued service provision to OVC at Arks (sustainability)? KEY AUDIENCES The key audiences for this report are USAID, FHI 360, Noah, and future funders or implementers of OVC care and support interventions. In addition, Ark staff, who are largely drawn from the Ark’s local community, as well as community stakeholders and beneficiaries, all have a legitimate interest in the evaluation findings. Specific information outputs have been prepared for the latter audiences, namely a summary of the key content of this evaluation report and an accompanying infographic. COMPONENTS OF THE REPORT This evaluation report is organized as follows  Section II provides background on UGM and Noah, including an overview of the Noah program model and relevant contextual factors  Section III includes a discussion of the methodology and approach used to conduct to the evaluation  Section IV outlines the findings of the evaluation, organized according to evaluation question  Sections V and VI provide the conclusion and recommendations for Noah                                                              2 The list of sub-evaluation questions included in the evaluation protocol was revised to address redundancy and overlap with the key evaluation questions. All questions referred to in this section will be addressed in the findings. Noah UGM End of Project Evaluation 2  The Appendices include the evaluation Scope of Work (SOW), data collection tools, workplan with timeframes for data collection, sampling framework, composition of the evaluation team, and references Noah UGM End of Project Evaluation 3 II. BACKGROUND THE UMBRELLA GRANTS MANAGEMENT PROJECT UGM is a five year grants management program administered by FHI 360 with funding from the U.S. Agency for International Development (USAID). Through UGM, FHI 360 provides funding and technical assistance to USAID-selected NGO partners who provide HIV/AIDS services at local, provincial, and national levels in South Africa. FHI 360-UGM seeks to promote high quality service delivery in alignment with the priorities and goals of the South African government’s HIV/AIDS framework. FHI 360-UGM provides specialized capacity building and support services to build partners’ skills and competencies in program management, governance, human resource development, budgeting and finance, and monitoring, evaluation, and reporting. The FHI 360-UGM project objectives are to 1. Provide grants to USAID/PEPFAR partners that ensure an adequate resource flow to foster scale-up of activities 2. Implement effective monitoring, evaluation, and reporting systems to assess and document activities 3. Provide ongoing capacity building to support and enhance scale-up of activities, and sustainability of activities and partners NURTURING ORPHANS OF AIDS FOR HUMANITY (NOAH) Organizational Background Noah is a registered not-for-profit Section 21 Company that partners with communities to assist them in taking care of OVC in the community. The organization was first conceptualized in 2000 by the founder, Dr. Gregory Ash, in an effort to address the orphan crisis brought on by the HIV/AIDS epidemic. Noah’s vision is to empower communities to take care of their vulnerable children. This vision is embodied in their mission of empowering communities with the knowledge, skills, strategies, and self confidence to successfully run their own community-based organizations supporting the well￾being of vulnerable children. Noah’s work in communities is guided by their core values of empowerment, integrity, passion, resourcefulness, and ubuntu3. According to the 2011 Monitoring, Evaluation, and Reporting Framework, the organization’s broad objectives (or purposes) are to  establish and build the capacity of CBO and committee members, staff, and volunteers to support and protect vulnerable children and their families in their communities across Gauteng and KZN  provide direct services to children through the provision of a social franchise model of care for vulnerable children  maintain a core of dependent CBOs which provide good services to children but have not been able to fundraise due to marginalization of the communities in which they reside Noah is led by a Chief Executive Officer (CEO), and senior management team consisting of a Programs Director, Chief Financial Officer, Program Implementation Manager, Programs Development Manager, Monitoring and Evaluation (M&E) Manager, Human Resources Generalist, Financial Manager, and Relationship and Communication Manager. A Board of Directors provides general oversight with Audit and Risk and Remuneration and Nomination Committees overseeing these respective functions. Noah’s organizational chart can be found in Appendix VI. Noah’s approved general operating budget for financial year 2013 is R39,510,856.27, 42% of which is funded by PEPFAR. Although Noah does not receive any other U.S. government funding, they receive a variety of grants from corporations and foundations. Table 2 shows the total funding annual income and                                                              3 Ubuntu is the African humanist ethic that emphasizes the importance of human relationships to individual and communal well￾being, often illustrated by the African saying “I am what I am because of who we all are.” Noah UGM End of Project Evaluation 4 percentage of PEPFAR support for each financial year since the beginning of the UGM project. Over the life of the project, PEPFAR funding has represented 30-40% of Noah’s total funding. Table 2: Noah Funding and Percentage from PEPFAR Noah Funding and Percentage from PEPFAR Financial Year Total Donations Received % PEPFAR 2007 (Mar 06 –Feb 07) R 25,038,133.00 32 2008 (Mar 07 – Feb 08) R 35,213,450.00 34 2009 (Mar 08 – Feb 09) R 42,713,360.00 41 2010 (Mar 09 – Feb 10) R 41,046,953.00 37 2011 (Mar 10 – Feb 11) R 32,787,537.00 41 2012 (Mar 11 – Feb 12) R 37,362,537.75 39 Source: Noah Finance Department Noah Program Model In accordance with the organizational mission and objectives, the Noah program model has two primary areas of focus—mentoring and capacity building of CBOs (Arks), and providing care and support services for OVC with the goal of empowering and capacitating communities to address the needs of their vulnerable children. The theory of change underpinning Noah’s program model is depicted below. Figure 1: Noah Theory of Change Source: Program framework, program theory and results chain for Noah, 2012 Services targeted at CBOs/Arks include training, organizational development, mentorship, and staff supervision. Care and support services for OVC are delivered at resource centers and include ECD, psychosocial support, home visits, child protection interventions (such as IDs and birth certificates, assistance with applying for social grants), health assistance activities, HIV/AIDS prevention education, educational support (such as homework supervision, numeracy, and literacy classes) and aftercare. Although not covered under PEPFAR funding, Noah also implements a nutrition program which includes meals and food parcels. While the primary focus of direct services is OVC, the community’s capacity to care for its OVC is improved by the establishment of an Ark, which is the mechanism through which care can be provided. Furthermore, some interventions also indirectly benefit families. Evidence discussed in the evaluation findings indicates that the provision of food parcels alleviates hunger in the household and the burden of provision from the caregiver, while psychosocial support to OVC is credited with leading to improved relationships in OVC households. Noah is currently planning to expand non-center activities to more explicitly target families. Noah UGM End of Project Evaluation 5 OVC activities are implemented by Ark staff which includes an Ark Manager, Child Activity Coordinators (CAC), volunteers, and other staff. The Ark Manager is responsible for management of the Ark and service delivery, staff supervision, financial management, project management, reporting to Noah’s head office, and liaising with the Ark Committee, a community-based governing body that provides oversight for the Ark. CACs implement all center programs and activities with support from volunteers. Most Arks also have a cook for meal preparation. Some Arks have supplemental staff such as a security guard, gardener, or general assistant. A typical Ark structure is included in Appendix VII. PEPFAR funds support 42 Arks in Gauteng, Kwa-Zulu Natal, and Northwest provinces, serving a range of communities including peri-urban, informal settlements, townships, and deep rural areas. In addition, the communities served by Arks are characterized by poverty and dependency on social grants, unemployment, and high HIV prevalence. Noah’s strategic objectives, as articulated in therequest for proposals (RFP) for this evaluation, are outlined in Table 3. The table also includes the relevant indicators reported to USAID/PEPFAR by Noah to monitor progress toward achievement of these objectives. Table 3: Noah PEPFAR Project Objectives and Corresponding Indicators Noah PEPFAR-Supported Objectives and Corresponding Indicators Objective Indicator To provide psychological care to OVC through bereavement counseling Number OVC receiving psychosocial support interventions (i.e., bereavement counseling/play therapy) To provide child protection services to OVC through birth certificates, ID documents, and other child protection interventions Number of OVC receiving child protection interventions Conduct home visits to OVC and their families Not reported for UGM project To provide educational assistance to OVC in aftercare through implementation of a literacy program (Gauteng) Not reported for UGM project To provide ECD programs Number of OVC participating in ECD programs Source: FHI 360 Request for Proposals: Program Evaluations for HIV/AIDS Program Partners Umbrella Grants Management, South Africa, COP11Partner Project Plan Over the life of the UGM project, Noah has provided services to thousands of OVC in a wide range of communities across Gauteng, Kwa-Zulu Natal, and Northwest. As shown in Table 4, Noah consistently exceeded their targets for service delivery over the life of the project (2007 to 2012), with the exception of Year 3. Achievements for this year (2009 to 2010) do not reflect OVC reach for Quarter 1. The current USAID information system was being implemented in Quarter I of that year, however reach for the omitted quarter was included in the reporting submitted for Quarter II. Table 4: PEPFAR Targets and Achievements for OVC Reached over the Life of the Project PEPFAR Targets and Achievements for OVC Reached over the Life of the Project Time Frame Targets Achievements October 2007 – September 2008 14,000 15,678 October 2008 – September 2009 10,000 10,029 September 2009 – October 2010 8,500 7,583 October 2010 – September 2011 5,050 6,472 October 2011 – April 2012 4,500 7,099 Total 42,500 46,861 Source: Noah Financials Noah UGM End of Project Evaluation 6 III. METHODS OVERVIEW OF METHODOLOGY AND APPROACH The evaluation was conducted using a mixed methods approach that relied primarily on an analysis of qualitative data obtained from staff, community stakeholders, and beneficiaries, as well as both quantitative and qualitative analysis of existing data. Secondary monitoring and evaluation data were used to complement qualitative findings and to help assess program objective achievement and the impact of service delivery on OVC, families, and communities. This mixed methods approach allows for triangulation and verification of findings across data sources. The evaluation also draws on elements of participatory evaluation, as Noah’s inputs are reflected in the planning and implementation of the evaluation including the evaluation questions, sampling strategy, and selection of secondary data for analysis. Noah was also engaged at the end of the evaluation to assist in verification of findings. To ensure that a rich data set for identifying most significant changes was obtained, and that the diversity of Arks was represented, six Arks were sampled. Sampling criteria included geographic location, degree of urbanization, Stage of Progression (a measure of an Ark’s capacity to sustain services independent of Noah, based on measures of performance) and type of services offered. Key components of the evaluation included partner engagement and document review; key informant interviews with five head office staff and one Board Member; in-depth interviews with 45 Ark staff, volunteers, and community stakeholders4; focus group discussions with 84 caregivers (guardians) of OVC ages 2–18; focus group discussions with 149 OVC ages 12–18; and review and analysis of secondary data. Informed consent was obtained from all interview and focus group participants. Structured interview guides were used to collect data on OVC needs and challenges, service delivery, quality of services, most significant changes, most valuable activities, program barriers and enablers, sustainability, and lessons learned. Data were analyzed qualitatively using ATLAS.ti software. Demographic data from beneficiaries and monitoring data were captured and analyzed using descriptive statistical techniques in Microsoft Excel. EVALUATION DESIGN Overview of Data Collection Data collection activities were conducted from April 19–May 10, 2012, within the three week time frame designated by FHI 360. Following key informant interviews at Noah headquarters, site visits were made to six Noah sites (Arks) for in-depth interviews and focus groups with Ark staff, volunteers, community stakeholders, and beneficiaries. Figure 2 provides a summary of data collection activities for the evaluation. A demographic profile of the beneficiaries engaged for the evaluation can be found in Appendix VIII. Sampling Frame Six sites were selected for primary data collection from the 42 PEPFAR-funded Arks. These sites were selected based on the following criteria  province  performance based on the Ark’s stage of progression or organizational development  type of area (urban, peri-urban or rural)  daycare versus aftercare sites                                                              4 Ark Managers were interviewed individually. CACs, volunteers, and other staff were interviewed in groups as it was suggested by Noah that these staff would be more comfortable talking in groups. Noah UGM End of Project Evaluation 7  range of services offered The sites or Arks selected include Kliptown, Swaneville, Melusi, Nkobongo, Zamimpilo, and Hlangabezu. Details on the characteristics of the Ark sample are included in Appendix IV. Given the focus on knowledge generation for this evaluation, this sampling frame was chosen to ensure the sites selected would provide rich information on the factors and conditions that influence program efficacy. This sampling frame will also allow for comparisons of outcomes and impact based on relevant site characteristics. Key informants, Ark staff, and volunteers were sampled based on their role/position in the organization, knowledge of program implementation and outcomes, and availability to participate. Caregiver and OVC focus group participants, as well as community stakeholders, were selected based upon availability of respondents during the site visit. The selection of the latter sample was mediated by the local Ark. Figure 2: Summary of Data Collection Activities and Participant Sample Informed Consent Informed consent was obtained from all participants in accordance with U.S. government ethical guidelines for research with human subjects. This implies that local legal guidance regarding the participation of children should be adhered to. Currently, there are no laws in South Africa regulating the rights of research participants (Strode, Grant, Slack & Mushartwa, 2005). However, there are a number of provisions in various acts that provide a legal framework for the ethical participation of children in research. In order to ensure that the evaluation was conducted in alignment with the highest ethical standards the following principles were adopted  age of participation for children was restricted to the age range in which there is a legal basis for informed consent  selection of participating children was mediated through Noah  while informed consent would be obtained from children, participation would still require assent by the children’s guardians, which would be obtained via the partner mediated selection  evaluation did not require the collection of potentially sensitive data from children and the instruments were prepared accordingly A consent form detailing the purpose of the evaluation, risks, benefits, and conditions of participation was provided to each participant. The form was explained in English or the local vernacular language as required. After explanation of consent, it was reemphasized to participants that their participation was voluntary and that they could choose not to participate or stop participating at anytime. Proceeding to data collection with each participant was dependent on their explicit consent. In the case of adult Noah UGM End of Project Evaluation 8 participants, this included the signing of the consent form. In cases where the participant was unable to sign due to illiteracy, a member of the evaluation team or Ark staff signed on their behalf after verbal consent was given. Evaluation Components Partner Engagement With the assistance of FHI 360, an initial teleconference was held with Noah on April 12, 2012 to formally introduce the FeedbackRA evaluation team, provide an overview of the evaluation scope and focus, and schedule a face-to-face meeting to plan for data collection. On April 13, the evaluation team met with the CEO and M&E Manager to review the evaluation methodology, finalize the sampling approach, and plan data collection activities. The evaluation team liaised with the Programs Implementation Assistant throughout the data collection phase for logistical support. The M&E Manager and Assistant M&E Manager were also engaged during analysis and report development processes to assist with accessing necessary data and information. Document Review A range of program documents from FHI 360 (Country Operational Plan (COP)10 and COP11 Partner Project Plans, Noah program framework, program theory and results chain, 2009-2010 Annual Reports, research and evaluation reports, strategy documents, etc.) were reviewed for information on program design, implementation, indicators of objective achievement, and contextual factors that may influence outcomes and impact. The document review informed the development of data collection instruments, the analysis of the Noah program theory of change, data analysis, and the development of recommendations. All documents utilized in the development of the report are included in the References section. Key Informant Interviews Key informant interviews were conducted with five Noah senior staff and a board member at Noah offices in Johannesburg and Umhlanga, Durban, KZN. One key informant interview was conducted via telephone due to scheduling constraints. Key informants included the Programs Implementation Manager, Programs Development Manager, M&E Manager, two Ark Development Officers, and the founder of the organization, who is currently the Vice Chairman of the Board of Directors. A structured interview guide was used to obtain perceptions of program objectives, alignment with organizational objectives, services offered, the quality of service delivery, progress on objective achievement, changes in OVC well-being, and enablers and barriers for program success and sustainability. Ark Staff, Volunteers, and Community Stakeholder Interviews In-depth interviews were conducted with staff, volunteers, and community stakeholders at the Ark level in individual or groups settings. While actual numbers of staff vary by Ark, four to eight Ark level staff or volunteers were interviewed in each site. One to two stakeholders were selected from within the Ark Committee for in-depth interviews. In some sites general community members (such as a neighbor of the center, teacher, traditional leader, or pastor) were also interviewed to obtain community perspectives of those not formally affiliated with Noah. Ark staff, volunteers, and community stakeholders were selected based on their role/position at the Ark or in the community, knowledge of program implementation and outcomes, and their availability to participate. A structured interview guide was used to obtain perceptions of services offered, the quality of service delivery, changes in OVC well-being, and enablers and barriers for program success. All questions were asked in an open-ended manner with probing and follow-up questions used as needed to elicit responses. Focus Group Discussions with Caregivers of OVC Seven focus group discussions were conducted with caregivers of OVC ages 2–7 and 12–18 across sites. In five sites, one focus group discussion was conducted, while two focus groups discussions were held with caregivers in Nkobongo. Caregivers with children within these age ranges were chosen to Noah UGM End of Project Evaluation 9 capture the full range of OVC served by daycare and aftercare Arks. Caregivers of children ages 2–7 provided opinions on the impact of daycare services to younger children who could not be interviewed for the evaluation. Caregivers of older OVC (ages 12–18) provided another assessment of the impact of services on OVC that could be used to triangulate data collected directly from older OVC. There were some deviations from the proposed sample in the evaluation design. Although not identified as part of the sampling frame, some caregivers with children ages 8–11 were recruited by Ark staff for focus group discussions. Furthermore, the proposed data collection procedure stipulated that two caregiver focus groups be conducted at each site, but this was only implemented in one instance (Nkobongo). At three sites there were sufficient respondents to conduct two focus groups, but Ark staff and respondents expressed a preference for a single group discussion, to which the evaluation team deferred. Ark staff at the remaining two sites indicated that it would difficult to recruit enough caregivers for two focus groups. Table 5 provides a summary of the number of caregiver focus groups conducted, number of participants, and the age of their children per site. Table 5: Proposed versus Actual Sample for Focus Group Discussions with OVC Caregivers Proposed versus Actual Sample for Focus Group Discussions with OVC Caregivers Ark Proposed Sample Actual Sample Age Range of Caregivers’ Children Number of Groups Number of Participants Number of Groups Number of Participants Kliptown 2 10-12 per group 1 5 Data not collected5 Swaneville 2 10-12 per group 1 14 1 – 16 Melusi 2 10-12 per group 1 24 5 – 17 Nkobongo 2 10-12 per group 2 Group 1: 10 4 – 16 Group 2: 6 2 – 18 Zamimpilo 2 10-12 per group 1 8 2 – 14 Hlangabezu 2 10-12 per group 1 15 7 – 15 A structured focus group guide was used to gather data on perceptions of Noah’s programs and services and the most significant changes observed in their child’s life since enrolling at the Ark. All questions were asked in an open-ended manner, while the facilitator probed for further information and clarity. Focus group interview guides are included in Appendix II. Focus Group Discussions with OVC 12–18 Nine focus group discussions were conducted with older OVC, ages 12–18. The OVC focus groups constitute a non-probability sample with the recruitment of participants mediated by the local Ark. The proposed data collection process indicated that two focus groups would be conducted at each site; however this procedure was implemented at only three of the sites (Melusi, Nkobongo, Zamimpilo). A single OVC focus group was conducted at each of the remaining three sites (Kliptown, Swaneville, and Hlangabezu). While enough respondents were recruited at each site to conduct two focus groups, at the latter three Ark staff and respondents expressed a preference for a single group discussion, to which the evaluation team deferred. The preference was motivated by the logistical practicality of ensuring that respondents were allowed sufficient time to make their way to and from the center on the data collection day. Details of the proposed and actual sample of OVC 12–18 are included in Table 6.                                                              5 As noted in the limitations section, this data was not collected in this site due to an error in the consent form. Noah UGM End of Project Evaluation 10 Table 6: Proposed and Actual Sample of OVC 12-18 for Focus Group Discussions Proposed and Actual Sample of OVC 12–18 for Focus Group Discussions Ark Proposed Sample Actual Sample Number of Groups Number of Participants Number of Groups Number of Participants Kliptown 2 10-12 per group 1 15 Swaneville 2 10-12 per group 1 17 Melusi 2 10-12 per group 2 Group 1: 13 Group 2: 18 Nkobongo 2 10-12 per group 2 Group 1: 15 Group 2: 18 Zamimpilo 2 10-12 per group 2 Group 1: 15 Group 2: 19 Hlangabezu 2 10-12 per group 1 19 A structured focus group guide was used to gather data on OVC perceptions of Noah’s programs and services and the most significant changes in their life since enrolling at the Ark. All questions were asked in an open-ended manner, while the facilitator probed for further information and clarity. In addition, OVC focus groups included two interactive activities designed to collect information on OVC needs before enrolling at the Ark, how those needs were subsequently addressed, and the most valuable activities and services offered at the Ark, based on the significance of the change those activities made in the life of the child. Focus group schedules and interactive activity descriptions are included in Appendix II. Secondary Data Review FeedbackRA worked with the Noah M&E Manager to identify existing data that would assist in assessing objective achievement and the impact of service delivery on OVC well-being. Monitoring data relating to service delivery was reviewed to clarify program implementation mechanisms and identify trends in program performance. In addition, findings from previous evaluations were reviewed for relevant data on context, outcomes, and impact in selected sites. Data Analysis Data were analyzed using both qualitative and quantitative methods. Interview and focus group data were coded and analyzed by evaluation question using ATLAS.ti. Key themes were identified based on the frequency of responses. Where data were collected in group settings, frequencies represent the number of “groups” in which the theme emerged. For all evaluation questions, findings were compared across participant types and sites (Arks). Where appropriate, comparisons were drawn based on the Ark performance as measured by the Stage of Progression. Basic demographic information collected from caregivers and youth/OVC including number of children at the Ark, age, gender, and the year of enrollment, were captured and analyzed for descriptive statistics (counts and frequencies) using Microsoft Excel. In addition, secondary data collected by Noah for monitoring and evaluation purposes was reviewed and analyzed in the same manner. A summary of the data analysis approach is included in Appendix IX. LIMITATIONS OF THE EVALUATION  Time to conduct the evaluation. The data collection phase of the evaluation had to be planned and implemented within a three week time frame. The tight timeline presented some logistical challenges that affected recruitment of beneficiaries and the representiveness of the final sample.  Variance in the number and size of focus groups across sites Focus groups with caregivers and OVC varied in number and size based upon recruitment processes at sites and the availability of beneficiaries. While the original sampling called for two groups of caregivers and OVC per site, this was not always feasible. Multiple focus groups allows for some control over the bias inherent in Noah UGM End of Project Evaluation 11 focus group data—the emphasis of discussion topics as a function of immediate group dynamics rather than as an indicative reflection of actual experience.  Difficulty in disaggregating findings by age or gender. Focus group discussions with OVC were mixed by age and gender (included both boys and girls of any age between 12 and18). Given that focus groups findings are reported as themes based upon group consensus, it is not possible to disaggregate findings by age or gender. Instead, findings are generally attributed to OVC (both girls and boys) between the ages of 12 and 18. Information on age and gender was collected on consent forms for caregivers and OVC focus group participants and can be found in Appendix VIII.  Non probability sampling. This sampling approach limits the extent to which findings can be generalized for an externally valid evaluative statement on Noah’s performance under UGM. The findings will reflect the efficacy of the Noah model of intervention under a variety of typical circumstances and provide externally indicative findings on performance.  Missing data on demographic variables for some participants and their children. Some caregivers, primarily grandmothers, or older relatives were unable to provide the age of their children or the year they enrolled at the Ark. Additionally, demographic information was not collected during the youth focus group at the pilot site due to an error in the consent form. This was corrected and the data was collected where possible at all remaining sites. Noah UGM End of Project Evaluation 12 IV. FINDINGS TO WHAT EXTENT WAS NOAH ABLE TO ADDRESS THE NEEDS OF CHILDREN WITHIN THE COMMUNITY? In order to determine the extent to which Noah has met the needs of the children they serve, it is necessary to understand the challenges OVC face in the selected communities and the extent to which the services offered by Arks reflect these needs. This section outlines the findings on needs and challenges and alignment with service delivery at Ark level. Needs and Challenges Facing OVC Community stakeholders and beneficiaries were asked about the challenges faced by OVC in their communities. Table 7 presents the most consistently reported challenges, in order of the frequency with which the challenge was noted. Frequency is reflected in percentages, which represent the proportion of total respondents or focus groups that raised the issue. The table is not an exhaustive presentation of challenges mentioned; the additional challenges that were mentioned more than once are subsequently listed. Among community stakeholders, the most frequently reported challenges faced by OVC in the community are hunger, crime (such as gangs and theft), HIV/AIDS, poverty, unemployment, substance abuse, the lack of basic material necessities, and teenage pregnancy. These themes were echoed by the other respondent groups, with the noteworthy exception of HIV/AIDS. Table 7: Perceptions of Challenges Facing OVC in the Community Perceptions of Challenges Facing OVC in the Community Community Stakeholders (n= 13 individuals) Caregivers of OVC 2-18 (n= 7 Focus Groups) OVC 12-18 (n= 149 individuals)6 Hunger/ lack of food (68%) Hunger/lack of food (71%) Hunger, lack of food (37%) Poverty (46%) Coping with the death of parents (71%) Coping with the death of parents (9%) Crime (46%) Poverty (43%) Poverty (8%) HIV/AIDS (38%) Behavior problems (22%) Abuse and mistreatment (5%) Substance use (smoking, alcohol, drugs) (38%) Disinterest in school (22%) Challenge could not be read/not understandable (6%) Unemployment (38%) Poor communication and interpersonal skills (22%) Fighting or bullying (5%) Teenage pregnancy (23%) Poor school performance (22%) Lack of basic necessities (clothing, school uniforms, shoes, etc) (23%) Source: Interviews with community stakeholders, caregiver focus groups, OVC focus groups When asked about what concerned them in taking care of their children, caregivers identified not having food, poverty, coping with parents’ death, behavior problems, and disinterest in school as major concerns. During focus group discussions, OVC were asked to write down a challenge or problem they faced before coming to the center. In addition to food shortages, poverty, and dealing with the death of parents, OVC also wrote about challenges with abuse and mistreatment, and fighting or bullying. Other themes that emerged from discussions with OVC about challenges experienced before coming to the center included not having friends, respecting others, and paying school fees. A small percentage of OVC indicated that they did not have any problems before coming to Noah. Across all participants, the following challenges were also identified:                                                              6 Findings presented in this column are based on OVC self report of challenges faced before coming to Noah. Each focus group participant wrote their challenges on a post-it note which was collect and analyzed to produce counts and frequencies of challenges across all 149 OVC focus group participants. Noah UGM End of Project Evaluation 11  Living alone or not having a caregiver (such as, child-headed households)  No ID document or birth certificate  Being subject to stigma  Mental or physical health issues  Family troubles  Inadequate guidance/supervision  No shelter, living on the street  Not being able to play with friends  Low self esteem/self worth  Prevalence of transactional sex in community  Rape and sexual abuse  Dropping out of school  Teenage pregnancy  Attitude problems (such as anger)  Parents left or moved away  Low self confidence When considered across Arks, the data reveals a similar prioritizing of themes (Table 8 presents challenges in order of frequency reported across sites). Hunger or lack of food emerged as the dominant theme in all sites except for Kliptown. This suggests that while access to food is a major concern for OVC and their families in Swaneville, Melusi, Nkobongo, Zamimpilo, and Hlangabezu, it is less of an issue for beneficiaries in Kliptown. Although the community served by the Kliptown Ark is an informal settlement, there are services in the Kliptown area. It is presumed that residents can access food resulting in less dependency on the Ark. Instead, participants were more concerned about bad behavior among OVC, especially fighting and bullying. Aside from food, poverty, and the death of parents, other challenges raised were generally unique to sites. In Hlangabezu (and to a lesser extent in Swaneville), crime was reportedly a major concern including gangsters, break-ins, and theft. Hlangabezu community stakeholders suggested that crime is an ongoing issue that affects the safety of children attending the Ark. In Melusi, additional challenges reported include OVC inability to open up/express feelings and attitudes, problems which are likely directly related to coping with the death of a parent/relative. Zamimpilo was the only site in which abuse or mistreatment emerged as a key theme. Similarly, community stakeholders and beneficiaries in Swaneville highlighted disinterest in school and low self confidence as challenges facing their OVC. Table 8: All Participants’ Perceptions of Challenges Faced by OVC by Site (Ark) Themes on Challenges Facing OVC by Site (Ark) Among all Participants Kliptown Swaneville Melusi Nkobongo Zamimpilo Hlangabezu Behavior Problems Hunger/lack of food Hunger/lack of food Hunger/lack of food Hunger/lack of food Hunger/lack of food Fighting Poverty Poverty Death of parent, coping with loss Death of parent, coping with loss Crime Bullying Lack of interest in school Unable to open up, express feelings Poverty Abuse or mistreatment Death of parent, coping with loss Low self confidence Death of parent, coping with loss Paying school fees Unemployment in family Crime Attitude problems (such as anger) Neglect Source: Interviews with Ark staff, CAC/volunteers, community stakeholders, beneficiary focus groups Ark Service Delivery and Utilization Respondents were asked about the range of services and activities offered at Arks. Table 9 includes a list of activities offered across the six sample sites, compiled from respondent data. Noah UGM End of Project Evaluation 12 Table 9: Summary of Activities and Services Offered at Sample Arks Summary of Activities and Services Offered at Sample Arks Type Activities Early childhood development Day care Psychosocial support Bereavement counseling/play therapy Child protection Identification and registration of OVC Assistance applying for an ID or birth certificates Soul Buddyz clubs Life skills education HIV/AIDS prevention TB awareness Teen pregnancy prevention Applications for social grants Education about children’s rights Educational support Homework supervision Numeracy and literacy classes Aftercare Sports Drama Traditional Dance Arts and crafts Field Trips/Excursions Games Music Computer classes Nutrition Meals Food parcels Health Assistance Referrals to clinic or hospital Other Donations (clothing, shoes, school uniforms, etc) Home visits Food gardening Source: Ark staff and volunteer interviews, community stakeholder interviews, beneficiary interviews Across Arks, the most utilized activities or services included meals at Arks and food parcels (86%), HIV/AIDS prevention (85%), home visits (61%), child protection programs (50%), educational support, and food gardening (48%). OVC were least likely to report receiving assistance in applying for social grants (3%) or referrals to health services (5%), perhaps because these services are more often targeting the caregiver. Further details on activity participation within Arks can be found in Appendix X. Monitoring data on service delivery indicators provides another perspective on the extent to which services have been utilized by OVC over the life of the project. Data on key indicators (including child protection, psychosocial support, and ECD) indicates that Noah facilitated access to these essential services in accordance with identified needs and effectively scaled-up these services over time across the six sites. Over the life of the project, child protection interventions were provided most often, followed by ECD and psychosocial support. Table 10: Summary of Noah Performance on Key Indicators, 2009 - 2011 Summary of Performance on Key Indicators over Life of Project Indicator 2009 2010 2011 20127 Number OVC receiving counseling or play therapy 101 221 548 306 Number of OVC receiving child protection interventions 1,975 2,771 9,508 7,548 Number of OVC participating in ECD programs 999 1,397 4,379 1,490                                                              7 Performance data for 2011-2012 is for Quarter 1 and Quarter 2 only. Noah UGM End of Project Evaluation 13 Alignment between Needs and Ark Service Delivery Across beneficiaries and community stakeholders, the most commonly reported “needs” related to caring and supporting OVC include hunger, poverty, coping with the death of parents, and abuse and mistreatment. As described in Table 11, the range of activities and interventions offered at Arks correspond to almost all these and other challenges and concerns of OVC, caregivers, and the broader community. Furthermore, all of these activities are funded by USAID/PEPFAR except the nutrition program, which is supported by Noah through other funding streams. Table 11: Comparison of OVC Needs to Ark Activities and Services Comparison of OVC Needs to Ark Activities and Services Most Reported Needs Activities and Services Offered by Arks to Address the Need PEPFAR Funded Hunger, Food Meals at Arks, food parcels Poverty Assistance with social grants  Death of parent, coping with loss Counseling and play therapy  Unemployment Abuse and mistreatment Child protection interventions  Lack of clothing, shoes, school uniforms Donations  HIV/AIDS HIV/AIDS prevention education  Making friends Aftercare  Behavior problems Life skills  Disinterest in school Educational Support  Respect for others Life skills  Communication and interpersonal skills Counseling and play therapy  School performance ECD, educational support  The only challenge not directly addressed by services to OVC or their families is unemployment. It is apparent from the data that where income generating activities such as food gardening was being implemented, it was not regarded by respondents as an option equivalent to employment. Formal skills acquisition and jobs hold the promise of improved prospects in a way that substantially exceeds what sustainable livelihood interventions are perceived to offer. It is important to point out that Noah (through its Arks) has an indirect impact on unemployment in the community. Firstly, each Ark provides full time employment for about three to five staff people (such as Ark Manager, CACs, cook, gardener, general assistant) who may otherwise not have employment. Secondly, all Arks have community volunteers most of whom who receive formal training and capacity building (including knowledge and skills competencies) on various topics related to caring for OVC. Access to these trainings through the Ark and Noah can potentially assist then in obtaining employment in the child and youth care sector. Feedback from Ark staff and volunteers, Noah key informants, and community stakeholders confirms that programs and services offered at the Ark level respond to the needs of beneficiaries. Almost all Ark staff and Noah key informants (92%) interviewed indicated that activities are designed and implemented to address needs. Other reasons cited for the choices in activities include alignment with the Noah model, funder requirements, and responding to OVC interests. 88% of community stakeholders (such as Ark committee members or community members) stated the services are meeting the needs of OVC and families. Finally, the majority of OVC focus group participants indicated that Noah has helped them to solve a problem or challenge they faced before enrolling in the center. The following comments provide examples of how the Ark met OVC needs.  “I didn’t have money to pay my school fees and the center took the responsibility to pay them for me.” OVC Focus Group Participant, Kliptown Noah UGM End of Project Evaluation 14  “I don’t sleep without food because the center gives me food and even for my family.” OVC Focus Group Participant, Hlangabezu HOW DO STAKEHOLDERS PERCEIVE THE PROGRAM IN TERMS OF QUALITY AND EASE OF ACCESS? Measurement of Quality The perceived quality of services delivered by Arks was measured using six key criteria including accessibility, affordability, age appropriateness, the extent to which services could be accessed without beneficiaries being stigmatized in any way, alignment with established standards and best practices, and beneficiary satisfaction with services. These criteria were generated by the evaluation team, and grounded in early data obtained in key informant interviews and Noah’s definition of quality services, which is informed by provisions in the Children’s Act. Key informants, Ark staff and volunteers, and community stakeholders were asked if and how programs and services offered at their Arks complied with these key criteria. Respondents’ perceptions of quality are denoted in Table 12. Table 12: Themes on Quality of Services across Arks Themes on Quality of Services Across Arks Indicators of Quality Kliptown Swaneville Melusi Nkobongo Zamimpilo Hlangabezu Accessibility Yes Yes Somewhat Yes Somewhat Somewhat Affordability Yes Yes Yes Yes Yes Yes Age appropriate Yes Yes Yes Yes Yes Yes Stigma-free Yes No Stigma related to HIV/AIDS, being an orphan, and poverty No Stigma related to HIV/AIDS, and being an orphan No Stigma related to being an orphan Yes Yes Aligned with best practices Yes Yes Yes Yes Yes Yes Source: Key informant interviews, Ark staff interviews, and beneficiary focus groups Perceptions of Quality: Affordability, Age Appropriateness and Alignment with Best Practices Across all Arks, program and services are affordable, age appropriate for the children served, and aligned with best practices. All participants reported that Ark services are free of charge and therefore affordable to everyone. In some sites, there are sometimes nominal fees for field trips or excursions, but stakeholders believed they were reasonable and families were able to pay in installments. In other sites, all costs for field trips were also covered by the Ark. The majority of staff and community stakeholders suggested that activities at Ark are organized according to age groups to ensure the content and approach is suitable for children. Beneficiaries and OVC also confirmed that activities are structured according to age groups. Staff and community stakeholders across sites reported that activities and services are designed and implemented according to best practices and standards. Most pointed to the focus and integration of principles of the Children’s Act as evidence of their alignment with best practices. Other examples of synergy with best practices include cooperation with other organizations (specifically evidenced by referrals between organizations and the Arks), sharing best practices across sites, screening potential staff and volunteers, respecting children’s rights (a specific example often cited was not allowing children’s photographs to be taken), and utilizing programs/interventions that are developed by reputable and accredited organizations (such as Soul Buddyz, Rob Smetherham Bereavement Noah UGM End of Project Evaluation 15 Counseling, and Lovelife). Finally, some respondents maintained that the regular training received by staff and volunteers also helps to ensure that programs uphold high standards of practice within the children’s sector. Perceptions of Quality: Accessibility Across sample Arks, there were some differences in perceptions of accessibility and stigma associated with services. In half of the sites (Kliptown, Swaneville, and Nkobongo), participants indicated that the Ark was centrally located within the community or close to schools, and as result, services were easily accessible. This was reported despite the fact that the Nkobongo Ark serves children from multiple areas, some of which are far from the Ark. Staff, community stakeholders, and beneficiaries indicated that transport is arranged for OVC or money is provided for bus, taxi, or train fare. As such, despite the distance, the Ark activities and services are accessible. Feedback from participants revealed that services at the remaining three Arks are only somewhat accessible. The Melusi, Hlangabezu, and Zamimpilo Arks, all of which are in rural or peri-urban areas, are reportedly only accessible to some children. The Melusi and Hlangabezu Arks are close to some schools but far from others. In addition, the Hlangabezu and Zamimpilo Arks serve children living in different communities. In these communities, children have to go home before going to the center and because of the distance, only go to the centers a few times per week. In Melusi, children attending far away schools do not go to the center in winter. Community stakeholders in Hlangabezu also expressed concerns about the safety of OVC as they are traveling to the center, given the distances and the fact that some have to cross the highway. In these sites, the lack of proximity to schools and communities where orphans live limits accessibility to and participation in activities. Services are only easily accessible to OVC who live and attend school in the community where the Arks are located. Perceptions of Quality: Stigma Participants had varied perceptions about the extent to which accessing services at Arks exposed beneficiaries to stigma, and how the persistence of stigma impacted the delivery of services. In Kliptown, Zamimpilo and Hlangabezu, staff and community stakeholders believed that there were no challenges with stigma, suggesting that services are delivered in a stigma-free environment in these three sites. In Hlangabezu, community stakeholders suggested there were problems in the past but the Ark was able to address it through education and awareness. The other three Arks reported challenges with different types of stigma in their communities. In Swaneville, participants reported stigma associated with HIV/AIDS, being an orphan and in poverty, and that accessing services to mitigate these exposes them to being stigmatized. Beneficiaries at Nkobongo emphasized the stigma of being labeled an orphan. Respondents at the Melusi Ark also faced challenges with stigma in the community and pointed out that an association with Noah could result in being stigmatized because the organization’s name incorporates the term HIV/AIDS. Apparently, older children at each of the three sites are reluctant to visit the centers because of a greater awareness of how they are perceived by others and not wanting to be labeled as orphans or “vulnerable.” In Swaneville and Melusi, Ark staff is working to address the issues of stigma by emphasizing to the community that they help and support all children, not only those affected by HIV/AIDS. Feedback from key informants indicates that the stance taken by the Ark is a reflection of a shift in Noah’s overall direction to supporting vulnerable children more broadly, not only those orphaned due to HIV/AIDS. In accordance with the strategic shift, the full organizational name is used less often and it is known simply as Noah. Key informants also suggested that Arks are able to combat stigma through consistent community engagement, practicing confidentiality, and sharing experiences and best practices among Arks concerning what has proven effective in dealing with stigma. Overall Quality of Services The consensus among respondents is that, with respect to affordability, age appropriateness, and alignment with best practices, all Arks are providing high quality activities and services to vulnerable children and their families. Further in Kliptown, participants’ perceptions of the quality of services were Noah UGM End of Project Evaluation 16 favorable across all indicators. In all sites, caregivers and OVC themselves reported general satisfaction with Ark activities and had positive interactions with staff characterized by support, mutual trust, love, and care. However, some sites face challenges in ensuring services are accessible due to the location of the Ark (Melusi, Zamimpilo, Hlangabezu). Others have to confront stigma associated with HIV/AIDS, orphans, and poverty (Swaneville, Melusi, Nkobongo). Addressing these barriers (by providing non￾center-based services, as well as more community engagement and education and awareness-raising initiatives around stigma) would help to improve the quality of services in the affected Arks. WHAT WERE THE MOST SIGNIFICANT CHANGES BROUGHT ABOUT BY NOAH IN IMPROVING THE WELL-BEING OF OVC IN TARGETED COMMUNITIES? The outcomes and impact of essential care and support services provide by Arks was assessed based upon the most significant changes realized in target communities. Stakeholders were asked about the changes in well-being observed in beneficiaries as a result of involvement with the Ark in their communities. Changes most frequently reported were considered most significant and verified by OVC self report of the “most significant” change in their lives. These findings are presented in the following sections organized by OVC, family, and community. Changes in OVC Well-being Key informants, Ark staff, volunteers and community stakeholders identified a wide range of change in the well-being of OVC. The top 5 changes across participant groups are shown in Tables 13 and 14. Table 13: Most Reported Perceived Changes in OVC Well-being According to Key Informants, Ark Managers, and Community Stakeholders Most Reported Perceived Changes in OVC Well-being According to Key Informants, Ark Managers, and Community Stakeholders Participant Types Perceived Changes in OVC Well-being Key Informant (n=6 interviewees) Sense of belonging (67%) Decrease in hunger (50%) Positive outlook on life (50%) Ability to communicate and share feelings (33%) Increased happiness (33%) Ark Manager (n=6 interviewees) Better school performance (50%) Improved health (50%) Ability to communicate and share feelings (33%) Access to social grants (33%) Improved life skills (33%) Community Stakeholders (n=13 interviewees) Better school performance (38%) Ability to communicate and share feelings (31%) Pass matric and pursue tertiary studies or job (31%) More respect for others (23%) Increased sense of security (23)% Source: Key informant interviews, Ark staff interviews, and beneficiary focus groups Noah UGM End of Project Evaluation 17 Table 14: Most Reported Perceived Changes in OVC According to CACs, Other Staff and Volunteers, Caregivers, and OVC Most Reported Perceived Changes in OVC Well-being According to CACs, Other Staff and Volunteers, Caregivers, and OVC Participant Types Perceived Changes in OVC Well-being CAC, Volunteers, Other Staff (n=11 group in-depth interviews) Better school performance (36%) Improved life skills (35%) Increased confidence (36%) Ability to communicate and share feelings (27%) Increased interested and motivation for school (27%) Caregiver (n= 7 focus groups) Decrease in hunger, access to food (57%) Better school performance (43%) Increased interest and motivation for school (43%) Increased confidence (43%) Increased happiness (43%) OVC (n=9 focus groups) Decrease in hunger, access to food (67%) Better school performance (67%) Increased confidence (67%) Ark provides clothes, uniforms, shoes, etc. (44%) Knowledge of HIV/AIDS prevention (44%) Source: Key informant interviews, Ark staff interviews, and beneficiary focus groups Among all participants except key informants (site staff, volunteers, community stakeholders, and beneficiaries), better school performance and decrease in hunger were the two most consistently reported changes in the lives of OVC. Between one-third and one-half of site level staff and community stakeholders cited improvement in school performance as a key change in OVC. Participants provided anecdotal evidence of OVC completing their homework and getting better marks in school. At two sites (Hlangabezu and Melusi), community stakeholder respondents who were also teachers or principals confirmed that students attending Arks had shown improvement in their classroom performance. The majority of beneficiaries (57% of caregivers and 67% of OVC) identified access to food as a notable change in OVC well-being since enrolling at Noah. Among key informants, however, the most reported change among children was a general sense of belonging, that despite being an orphan they have a place where they are welcomed and can go for help. Caregivers confirmed this perception, suggesting that Ark staff treat those attending the center like their own children and there is family-like atmosphere within the Ark. Across groups, other changes that were reported among the top five include the ability to communicate and share feelings, increased confidence, and increased happiness. All of the changes reflect a general improvement in emotional health and well-being. It is interesting to note that OVC also report having greater knowledge of HIV/AIDS prevention as a result of the involvement at Arks. Across all focus groups with OVC (n=149 people), only 3% reported having received support from the Ark in accessing a social grant. This most likely reflects the fact that this particular service offering targets caregivers directly, and there is consequently low awareness of this service among OVC. Findings on changes in OVC well-being were similar across sites. As indicated in Table 15, ‘better school performance’ and ‘access to food’ were among the most reported changes irrespective of site. Improvements in school performance were strong themes in Kliptown, Swaneville, and Hlangabezu, while regular access to food emerged as a key individual level outcome in Melusi and Zamimpilo, both of which are in rural or remote areas where food shortages appear to be more prevalent. Noah UGM End of Project Evaluation 18 Table 15: Themes on Perceived Changes in OVC Well-being by Ark for All Participants Themes on Perceived Changes in OVC Well-being By Ark For All Participants Ark Top Themes Kliptown Better school performance Swaneville Better school performance Melusi Decrease in hunger/access to food Nkobongo Respect for others Positive outlook on life Knowledge of HIV/AIDS prevention Better school performance Zamimpilo Knowledge of HIV/AIDS prevention Decrease in hunger/access to food Hlangabezu Better school performance Source: Key informant interviews, Ark staff interviews, and beneficiary focus groups Unlike other sites, several themes surfaced as equally important changes in OVC well-being in Nkobongo. In addition to doing better in school and knowing more about HIV/AIDS prevention, stakeholders and beneficiaries noted having more respect for others and a positive outlook on life among OVC as beneficial outcomes of involvement at the Ark. Surprisingly, despite programming in all Arks, increased HIV/AIDS prevention knowledge was only a change theme (frequently reported) at KZN Arks. Across Arks and participant types, themes indicate that the most common (or frequently reported) change in the lives of OVC are improved performance in school, decrease in hunger, knowledge of HIV/AIDS prevention, greater self confidence, the ability to communicate and express oneself more, and improved life skills such as discipline, responsibility, and decision making. As shown in Table 16 below, these themes generally correspond with what OVC reported as the most significant changes in their lives as a result of their participation in Ark activities. Table 16: Most Significant Changes as Reported by OVC Most Significant Changes As Reported by OVC Themes Access to food, no longer being hungry Better school performance HIV/AIDS prevention knowledge Life skills Increased sense of safety Source: OVC focus groups For the purpose of triangulation and verification, findings from focus groups with OVC were compared with data from Noah Impact Stories, writings by OVC about the impact of Noah in their lives. As noted in Table 17, themes identified in Impact Stories are well aligned with the reported most significant changes. Table 17: Commonly Reported Changes in OVC Well-being Compared to OVC Impact Stories Commonly Reported Changes in Well-being for OVC Focus Group Participants as compared to Noah OVC Impact Stories Focus Group Themes Impact Story Themes Access to food, no longer being hungry No longer hungry, have regular meals at Ark Better school performance Positive outlook on life HIV/AIDS prevention knowledge Sense of belonging, feel welcome at Ark Life skills Increased life skills Increased sense of safety Awareness of the dangers of risk behaviors (drugs, alcohol, sex) Increased confidence Source: OVC focus groups, Noah Impact Stories Changes in Family Well-being Noah UGM End of Project Evaluation 19 Staff, stakeholders, and beneficiaries identified changes in families as a result of the Arks; all participants across sites noted the increase in access to food as the greatest impact Noah has had on the families of OVC. Through the food parcels provided by Arks, families are able to combat hunger due to food shortages for the entire household. In addition to parcels being shared in the household, the provision of meals at centers and parcels to households has the substantial additional benefit of alleviating the burden of provision from caregivers. Improvements in food security on a small scale reflect families’ increased ability to care for OVC (and other children) by meeting basic and immediate needs. Another family-level change attributed to OVC involvement in Arks is described by respondents variously as happiness in the home; better relations between OVC, their siblings, and the family; and increased cooperation from OVC at home. This generalized psychosocial benefit was ubiquitously reported in some form among those respondents answering this question. Children’s happiness and satisfaction with their experience at the Arks extends into the home, resulting in greater happiness and an overall positive impact in the household. Respondents also indicated that children take what they learn at the center in regards to discipline and responsibility and apply it a home. Caregivers confirmed that children are more cooperative at home and readily do their chores; they also observed that children enrolled in the Ark were more caring and related better with their siblings and other family members. One staff member commented on the changes observed at the family level, “Many of the kids participating in Noah’s programs have become more mindful and considerate of their siblings, which has resulted in a better…environment at home, happy children, happy parents.” Aside from these two areas of positive impacts, participants identified very few outcomes of Ark service delivery at the family level. Except for access to food, the family-level outcomes identified do not reflect an increased capacity to care for OVC. The perceived limited impact of program services on their families is likely a reflection of the program model. Noah’s primary focus of intervention is at the individual level with OVC; as such, it is to be expected that any outcomes or impacts beyond the individual are indirect. As one Ark Manager remarked, “changes are seen in the children…they start at the individual and go out.” Similarly, a key informant asked, “Will we see more changes in family once Arks are aligned with the DSD and they can provide more direct services to families through home visits?” More substantive changes in the families of OVC will require more targeted interventions within the household. Changes in Community Well-being When asked about the changes brought about by Arks within the communities where they operate, participants identified the following key changes  greater awareness of OVC  community is taking care of OVC  increased community involvement in Arks Across sites and participant groups, the most common community-level change was an increased awareness of OVC. As a result of Noah’s involvement in the community and the establishment of a community-based organization focused on addressing the needs of OVC, participants believed the community was more aware of orphans and other vulnerable children and the challenges they face. In turn, this awareness and the existence of the Ark (including staff, volunteers, community committee, and resource centers) have caused communities to work together to care for orphans, primarily through the delivery of essential services. Similar to family changes, there was low response among all participants (and across sites) regarding community-level outcomes and impact. Themes in community changes indicate an increase in community involvement in Arks; community stakeholders in different sites noted that engagement at Arks has improved over time. In Nkobongo, a committee member remarked that when the Ark was first established, there was little input and buy-in from people living in the township surrounding the resource center. This has since changed and they are now sitting on the committee, attending meetings and actively participating. In Hlangabezu, several participants reported that the community had recently joined forces to help identify those responsible for breaking into the center and stealing equipment and Noah UGM End of Project Evaluation 20 food, and reported them to the authorities. According to community stakeholders, these actions reflect more interest, buy-in, and active involvement in the activities of the Ark. The fact that community members volunteer to care and support orphans (even if for personal motivations) represents an investment in improving the lives of OVC and strengthening the community at large. Increased awareness of OVC and more involvement in Arks represent important steps in improving community capacity to care for OVC. Before a community can come together to care for OVC, they must be aware of OVC and their needs, community members must be actively involved in providing services and support for OVC, and the community must have means to deliver services (such as space, materials, and resources). In the communities served by Noah, the Arks serve this purpose. Although not reported by participants, the mere existence of an Ark in the community and resource center from which to provide essential services to OVC represents a significant community-level change and is evidence of increased community capacity to care for OVC. Which program services were most valuable to OVC, their families, and the community? In an effort to link outcomes and impact within target communities to program outputs (service delivery), community stakeholders and beneficiaries were asked about the services and activities believed to be most valuable and influential in their lives. Among key informant and community stakeholders, the most valuable service or activity was provision of meals at Arks and food parcels (See Table 18). Beyond the nutrition program, key informants and community stakeholders had very different opinions about which activities had the most influence on changes in well-being. Table 18: Key Informant and Community Stakeholder Perceptions of Most Valuable Activities and Services Key Informant and Community Stakeholder Perceptions of the Most Valuable Activities and Services Key Informants (n=6) Community Stakeholders (n=13) Meals and food parcels (50%) Meals and food parcels (31%) Educational support (50%) Help with ID or birth certificate (23%) Child protection (33%) Traditional dancing and music (23%) Counseling or play therapy (33%) Provision of clothing, uniforms, school uniforms, other donations (23%) Life skills (33%) All services (23%) Source: Key informant and community stakeholder interviews The activities believed to be most valuable among key informants encompass all of Noah’s core PEPFAR-funded services, except for meals and foods parcels. On the other hand, community stakeholders identified activities and services that address OVC’s most immediate needs (including food, identification documents, clothing, shoes, and other basic necessities) as most valuable. Community stakeholders also suggested that aftercare activities focused on the arts and recreation were quite important as they provided OVC with an opportunity to have fun and connect with their cultural heritage through traditional dance. Providing opportunities for fun is particularly important as many vulnerable children have been forced to “grow up” quickly and take on adult responsibilities due to their life circumstance. There are some similarities in the perceptions of key informants, community stakeholders, and OVC regarding the most valuable program services. Table 19 outlines the most valuable activities and services among those offered (top five) as ranked by OVC focus group participants. Noah UGM End of Project Evaluation 21 Table 19: OVC Perceptions of the Most Valuable Activities and Services OVC Perceptions of Most Valuable Activities and Services Ark Five Most Valuable Activities Kliptown Aftercare, Soul Buddyz, food, food parcels, homework supervision, numeracy and literacy program Swaneville Food /food parcels, Soul Buddyz, assistance with social grants, gardening, and home visits Melusi Group 1: Gardening, food/food parcels, home visits, referrals for health services, counseling/play therapy Group 2: Counseling/play therapy, help with ID/birth certificates, food/food parcels, referrals for health services, assistances with social grants Nkobongo Group 1: Food/food parcels, assistance with social grants, help with ID/birth certificates, counseling/play therapy, homework supervision Group 2: Food/food parcels, home visits, assistance with social grants, homework supervision, gardening Zamimpilo Group 1: Food/ food parcels, home visits, gardening, Soul Buddyz, HIV/AIDS prevention Group 2: Gardening, help with ID/birth certificates, assistance with social grants, food/food parcels, homework supervision Hlangabezu Food/food parcels, home visits, counseling/play therapy, sports and recreational activities, assistance with social grants Source: OVC focus groups It is clear that across Arks, OVC believe the most valuable activity or service is the provision of meals and food parcels. This service was ranked first in five out nine focus groups (56%) and among the top three in eight focus groups (89%). In ranking food and food parcels high, OVC confirmed that this service was the most valuable and therefore influential in the changes in their lives. Beyond nutrition, OVC also ranked help with obtaining IDs or birth certificates, assistance with social grants, homework supervision, counseling/play therapy and food gardening as some of the most valuable activities and services offered by the Ark. All of these activities responded to OVC needs and contributed to positive changes in their well-being in OVC. The rationale for the value of these services, as reported by beneficiaries, is outlined in Table 20. Table 20: Beneficiaries' Reasons for Ranking Activities Most Valuable Reasons Provided by Beneficiaries for Ranking Activities as Most Valuable Most Valuable Activities Reasons Food, food parcels Children are hungry and cannot concentrate without a meal Meals are the primary incentive to come to the center Food gardening Provides kids with a skill and a means to deal with food shortage that has the potential to generate income Help with ID/ birth certificates Facilitate access to social grants Assistance with social grants Provide a source of income for the OVC and their family Educational support Provides regular assistance with homework and has helped with performance in the classroom Soul Buddyz Educates OVC about HIV/AIDS, life, and child rights in an interactive child friendly manner Counseling/play therapy Enables OVC to deal with grief, trauma, and loss and improve emotional well-being Source: Beneficiary focus groups Although OVC indicated that help with legal identification documents was a valuable activity because of their role in accessing grants, access to grants did not emerge as a significant change across all Arks. The activities and services identified as most valuable generally correspond to the reported most significant changes among OVC, further evidence that the outcomes observed are directly linked to the interventions provided by Noah’s Arks. This alignment is illustrated in Figure 3. Noah UGM End of Project Evaluation 22 Figure 3: Most Valuable Activities and Services, Most Significant Changes and Expected Outcomes As depicted in Figure 3, the majority of activities perceived to be most valuable were influential in bringing about the expected changes in OVC and their families. The only “valuable” activity that was not directly linked to a reported change was help with accessing identification documents and birth certificates. This finding can be explained partly as a function of how respondents logically understand change. Obtaining documents is simply a step in a process that will facilitate access to social grants, which in turn offer a very effective and highly valued means to affect change. Additionally, the finding can be partly explained as a function of the body of data, which heavily favors OVC and care workers. As a result, the frequency with which obtaining documents and accessing social grants is raised as a benefit is limited. Access to social grants, as has already been noted, was not a major theme in changes at beneficiary level. Nevertheless, given that such documents are required to be registered at Ark (as an OVC), getting assistance with obtaining the documents facilitated access to all other essential services provided by Noah and thus indirectly contributed to the outcomes in OVC and their families. How did Noah and its Arks change the path of OVC through their interventions? Based upon the changes in well-being observed due in part to the interventions implemented by Arks, it is reasonable to conclude that Noah (through its Arks) has changed the immediate path of some OVC. It is possible that without the essential services provided by Arks, many of the challenges faced by OVC may not have been addressed (such as hunger, difficulty coping with parent’s death, poverty, lack of basic necessities, etc.). OVC engaged in Ark programs and services are now well nourished, better able to cope with grief and trauma, more confident, hopeful about the future, respectful, responsible, capable of making good decisions, and doing better in school. OVC experiencing the same challenges, but not involved with an Ark or receiving similar services elsewhere, would presumably have a very different life pathway characterized by continued challenges, unmet needs, and potential engagement in risk behaviors. Some staff and community stakeholders suggested that there are observable differences in the OVC that attend Arks and those that do not.  “Children would be much worse off if they didn’t attend Noah…the community has a lot of challenges. [There is] lots of gangsterism especially in child-headed households. We would see many Noah UGM End of Project Evaluation 23 more school drop outs. It’s because of Noah that they stay in school.” Community Stakeholder, Swaneville  “The Ark provides discipline for children who do not have parents at home. Otherwise they would just stay at outside and play.” Community Stakeholder, Swaneville  “Children not involved in Noah are usually involved in substance use, bad behavior, teen pregnancy, and gangsterism.” Key informant, Noah Head Office Noah has helped to change the path of OVC involved at Arks by providing interventions and services to meet their needs, improve their well-being, and protect them from potential harm. What role did the Arks play in achieving these outcomes/changes? What role did Noah play in mobilizing Arks to fulfill this role? Arks have played a key role in achieving program outcomes. Caregivers and community stakeholders largely credit Ark staff for the changes observed in OVC and their families. 62% of community stakeholders indicated that Ark staff providing direct services (CAC and volunteers) were very influential in improving the well-being of OVC and that the changes would not have occurred without their involvement. As the direct service provider, Arks are the conduits through which interventions are delivered that ultimately contribute to improvements in the lives of OVC, families, and the broader community. Noah has played an instrumental role in mobilizing Arks to affect change in the lives of OVC. As a catalyst and key facilitator, Noah helps communities to create Arks. According to Ark Managers, Noah has helped Arks to establish and expand their services to OVC by assisting them in acquiring a facility; supplying equipment (tables, chairs, kitchen utensils); providing funding to support staff and service delivery; and ongoing training, mentorship, and support to manage the Ark (including governance, project management, community engagement, and staff supervision). In essence Noah has provided the building blocks to establish the Ark, resources to equip staff, and a strong foundation for the Ark to potentially grow into a highly functioning organization. Staff and community stakeholders across sites perceived Noah’s role to be invaluable in achieving outcomes in target beneficiaries. Without the key role played by Noah, it is unlikely that Arks would exist in the various communities. While there may be informal networks of community members independently providing services to OVC, there would not a comprehensive network of a care (staff, volunteers, resource center, oversight committee, relationships) or culture of caring and supporting OVC within target communities. WHAT WERE THE KEY ENABLERS AND BARRIERS IN MEETING PROJECT OBJECTIVES? In assessing the extent to which Noah has met its objectives under the UGM project, it is important to consider factors that have facilitated and hindered program success. The following sections discuss the key programmatic (P) and contextual (C) enablers and barriers that impacted Noah’s efforts to care and support OVC. Key Enablers of Program Achievement Evidence from the evaluation suggests that six critical factors contribute most significantly to the successful achievement of project objectives. 1. The efforts undertaken to involve the community in both the establishment of a local Ark and maximizing the relevance of the Arks programs locally As a core component of Noah’s model, community buy-in is essential in the establishment and functioning of Arks. Across sites, participants (100% of key informants and Ark Managers; 64% of CAC/volunteers) emphasized the importance of consulting with the community and gaining their support for the Ark initiatives. Arks gain and maintain community buy-in by approaching the relevant traditional or political leaders (including Ward Councillors, indunas, and chiefs) for permission and support, holding regular community meetings, establishing an Ark committee with Noah UGM End of Project Evaluation 24 members drawn from the community, ensuring the Ark Committee participates in decision making, and encouraging parental involvement in Ark activities. 2. The evidence-based approach to designing and implementing programs and services, which not only includes adjusting programming based on community inputs on local needs, but adopting a foundation for program design based on childhood development theory Respondents indicated that the design and implementation of Ark programs and services help to facilitate success. Noah uses a developmental approach in caring for OVC which helps to ensure programs are holistic and adequately responsive to the full spectrum of needs OVC face in the various stages of development. Other aspects of program design that have proven to be important include using existing evidence-based programs from reputable organizations (such as Rob Smetherham bereavement counseling and Soul Buddyz) and seeking community input to ensure programs reflect needs. 3. Noah’s investment in training Ark staff and developing the organizational capacity of local Arks Staff and volunteers identified the training and support from Noah as integral to their success. Ark Managers appreciated the mentorship and support from their ADO, especially the guidance on making decisions and effectively running the Ark. Staff and volunteers reported attending a broad range of trainings offered by Noah that provided new skills and competencies (counseling/play therapy, child abuse and child protection, and HIV/AIDS) that helped them in their work at the Ark. The majority of Ark Managers interviewed reported having greater confidence in their ability to care for OVC as a result of training and support from Noah headquarters. The support provided by Noah increased the organizational capacity of Arks and equipped them to provide high quality services. 4. Strategies to maximize resources, enhance cost-effectiveness, and provide services free of charge A number of staff associated program success with having resources including funding and material donations. These resources enabled them to provide the services to OVC, which contributed to changes observed in well-being. Community stakeholders (31%) and to a lesser extent staff emphasized the importance of partnerships in caring for OVC. To varying degrees, all Arks were able to broker partnerships with other civil society organizations, faith communities, and businesses to leverage additional resources and donations to support programs. Establishing and maintaining such partnership will assist Arks as they progress toward independence and sustainability. 5. The establishment of a vehicle—the Ark—through which communities are enabled to take care of their OVC While the objective of increasing the capacity of communities to care for OVC is theoretically sensible, its practicability it is not considered. Impoverished and lower income communities and household are confronted by a massive economic burdens associated with seeing to their own needs; assuming the responsibility for meeting the needs of others exacerbates their existing burden substantially. Providing a community with a mechanism through which to channel resources and volunteer time, to refer those in need and exercise goodwill, is a necessary prerequisite for realizing this objective, central to the national strategy for addressing the OVC crisis in South Africa. 6. Dedication and commitment of staff and their passion for children Key informants and Ark Managers highlighted the role and importance of staff at Arks. Many staff interviewed had been involved with caring for orphans at the Ark or in another capacity for several years. Across all sites, progress towards goals was attributed to staff being fully dedicated to the mission of Noah and equally committed to working as a team to achieve that mission (taking care of OVC). Staff dedication to their work was attributed to a genuine love and passion for children. Some participants believed that this passion for children was essential to making changes in OVC and their communities. There is evidence that these factors represent effective strategies for improving the well-being of OVC in wide range of communities (peri urban, informal settlement, township, and rural) with scarce resources. Noah UGM End of Project Evaluation 25 Participants were asked to identify the factors that had helped to make changes in OVC, families, and communities. Table 21 provides of a summary of these enabling factors across Arks and Stages of Progression. A tick indicates that the factor was indentified in the respective site. It is important to note that the table reflects “enablers” or factors that participants explicitly linked to their success. The table clearly illustrates that community buy-in; staff characteristics like dedication, teamwork, and passion for children; and training and capacity were instrumental in the changes achieved by all Arks. Partnerships and aspects of program design were only important enablers in some Arks. Finally, resources were only identified as a key enabler in one site, which is in the lowest range of Stage 3. This does not mean that the remaining Arks do not have resources; rather, it suggests that staff and community stakeholders do not associate their progress with resources. There do not appear to be any correlations between enablers and stage of progression, most likely because all Arks are at Stage 3. Table 21: Enablers Identified by Ark and Stage of Progression Enablers Identified by Ark and Stage of Progression Enabler Type of Enabler High Stage 3 Mid Stage 3 Low Stage 3 Kliptown Melusi Nkobongo Swaneville Zamimpilo Hlangabezu Community buy-in P       Dedication, commitment, of staff and passion for children C/P       Resources P  Partnerships P     Training and support P       Program design P    Linking enablers to the program model Findings indicate these factors (such as seeking community participation in the development of the Ark and design of programs; training and capacitating staff and volunteers in key content related to care for OVC; and implementing evidence-based programs that reflect the development stages of children) are essential elements of the Noah model. There is evidence that these factors represent effective strategies for improving the well-being of OVC in wide range of communities (peri urban, informal settlement, township) with scarce resources. Key Barriers to Program Achievement Evaluation participants identified a broad range of factors that they believed hindered their ability to positively influence the well-being of OVC, families, and communities (See Table 22). Table 22: Key Barriers to Program Achievement Key Barriers to Program Achievement Programmatic Contextual Shift in management of Ark nutrition program to Noah Head Office (Bulk Buying Initiative) Inadequate facilities (lack of electricity, toilets, not enough space to accommodate service delivery) Budget constraints, lack of funding to expand services Arks are not always secure Not enough staff for service delivery OVC living or attending school far from the Ark Retaining volunteers Evidence from the evaluation suggests that four critical factors most significantly undermine the successful achievement of project objectives. Noah UGM End of Project Evaluation 26 A. While the importance of providing a distinct vehicle (the Ark) through which the community can take care of its OVC is confirmed in the evaluation, there are a number of challenges that emerge as a direct result of the center-based model that undermine effective delivery of services to beneficiaries.  The inadequacy of some facilities in terms of security, basic utilities, and amenities affects the safety, comfort, and predisposition towards interventions of the OVC being serviced. In three communities, Arks lack basic utilities and services that impact service delivery (electricity, stable water supply, and toilets). In Hlangabezu, there is an ongoing problem with cable theft; as a result, Eskom has reportedly abandoned the area. In Melusi, there is a huge challenge with access to clean water. Even though a tunnel has been built to improve the water supply, staff suggested that they still rely on water from the river which is not clean. The Swaneville Ark is challenged by lack of toilets on site. The closest toilets are located across the road from the Ark, which affects service delivery and has the potential impact the Ark’s ability to meet DSD funding requirements. Several Arks identified challenges with space onsite to run programs. In some cases, there is not enough room to hold different activities simultaneously and Ark staff are forced to alternate programs (for instance, by age group or day of the week) to accommodate service delivery. A major concern in four of the six Arks is not having a permanent structure. Three of the four operate the Ark from containers and one currently rents space from a church which they share with a local crèche. Caregivers and community stakeholders were especially interested in seeing Arks move to more permanent facilities. Security is also a barrier for some Arks (Kliptown, Hlangabezu) as there have been issues with break-ins and theft of equipment and food. Community stakeholders in Hlangabezu also expressed concern about the overall safety of the community and the potential dangers to the children traveling to the center. Finally, participants in Swaneville noted the Ark shares space with a facility for the mentally handicapped, which is a safety risk for the children attending the Ark. These security risks make it difficult for Arks to provide to a safe environment in which OVC can receive services, and consequently affect the quality of service delivery.  The inaccessibility of Arks for a portion of the OVC beneficiaries in a community because of their distant location from the center, and the inconsistent exposure to interventions and services that results, affects the realization of outcomes for these OVC. As previously highlighted, accessibility is a barrier to program achievement in three sites. The Ark serves OVC living or attending school in different communities, and services are not easily accessible for all registered children. Limited accessibility affects some children’s attendance and the likelihood that services will have the desired impact. Long distances between children’s homes and the Ark also make it challenging for staff and volunteers to conduct regular home visits. B. Persistent staff shortages and the Ark’s reliance on volunteers Participants in more than half of the sites reported challenges with staffing shortages. In these sites, there are not enough staff members to deliver programs and services as needed, yet they are not able to increase stuff due to insufficient funding. Noah’s reliance on volunteers to support staff in service delivery should supplement staff shortages, but because volunteers are hard to retain (due to lack of pay and, in some cases, incentives), staffing issues are compounded. Volunteer retention was also the most reported theme in barriers to success across sites. In light of the staff shortages and budget constraints, Arks rely on volunteers to assist with service delivery. Key informants indicated that it is not uncommon for volunteers to receive training from Noah, build their knowledge and skills, and leave to work for other opportunities with pay or other incentives. Noah invests in developing volunteers but does not always get to benefit from this investment. C. Programming gaps at family and community level Key intended outcomes for Noah are to increase the capacity of both families and communities to take care of their OVC. There is clear evidence that the necessary prerequisite for improving community capacity is the establishment of a vehicle through which OVC can be cared for, which is at the heart of the Noah model—the establishment of an Ark in a community. Aside from this, the Noah UGM End of Project Evaluation 27 development of family and community capacity to care for OVC is not systematically incorporated into Noah’s programming. D. The persistent inability of Arks to achieve Stage 4 autonomy The Noah model includes a deliberate mechanism to support the maturation of individual Arks through stages towards a self-sustaining, autonomous organization (Stage 4). However, the final transition to a stage 4 organization is proving consistently and inordinately difficult to achieve, largely as a result of the limited success of individual Arks to raise funding and the failure of some Arks to meet DSD facility standards, which would qualify them for DSD subsidies. E. Budget constraints and lack of funding to expand services Several staff (Ark Managers and CACs) indicated that limitations in the current budgets presented challenges in achieving their goals. Some Arks had identified a need to expand their service to reach more OVC, but were unable to do so because of limited funds. Another Ark was forced to scale back services (food parcels for the community) due to budget constraints. Others expressed concern about finding additional donors to support their programs in future; this was of particular concern in rural and peri-urban communities with limited businesses or potential funders. F. Changes to Ark feeding program (Bulk Buying Initiative) Across sites, the most common theme in barriers was the changes to the nutrition program, now referred to as the Bulk Buying Initiative. According to staff and community stakeholders, as of 2010, Arks were no longer allocated funding for purchasing food due to budget constraints. Under the BBI, fortified food was purchased in bulk by the Head Office and distributed to sites. Ark staff and stakeholders identified several concerns with this approach. OVC expressed dissatisfaction with the lack of variety in new food and menu options. An OVC focus group participant commented that “…food has changed. [There] is no more salad or fruit in the meals we have. There is also no variation. We seem to be eating the same type of food all the time. It has become dull.” Reportedly, food is not delivered on time and is no longer fresh when it arrives. Some staff and community stakeholders believed that the food caused health problems for children. Finally, in some sites staff indicated that these changes disempowered the Ark because they were no longer able to make decisions about food. Given the importance of food/food parcels to the changes observed and as an incentive for OVC attendance and participation in all other activities, Ark staff considered the nutrition program in its current form to be major barrier. Participants indicated that Noah is aware of these perceived barriers and is taking measures to resolve the issue. In February 2011, Noah conducted a baseline evaluation of the BBI and introduced a revised program in July 2011 informed by the findings. At the time of this report, Noah was conducting an impact evaluation of the revised nutrition program. According to key informants, Noah is also in the process of seeking input from Arks on how to address remaining issues with the revised program (such as disempowerment of Arks due to lack of involvement in buying foods, variety in menus, quality of foods). Linking barriers to the program model The majority of the reported barriers to improving OVC well-being are directly or indirectly linked to Noah’s center-based model of service delivery (problems with facilities, security, and accessibility, and the perceived challenges with the feeding program). In order to meet the needs of OVC, Arks must have a physical structure or facilities from which to provide services. Ideally, such facilities should have basic utilities (electricity, water, sanitation) required to implement programs with children in a safe environment. In addition, the center must be accessible to registered OVC. Some Arks facilities do not meet these standards and face challenges in providing services to OVC. These factors would not be as great of a challenge to program success if Noah employed a non-center focused model or supplemented this approach with household or family-level interventions delivered in homes by Ark staff. Noah UGM End of Project Evaluation 28 The provision of meals at Arks is facilitated by the fact that OVC come to centers to receive services. Arks may not be facing the same level of challenges with the Bulk Buying Initiative if services were provided using a non-center-based model. With this approach, the feeding program would likely focus more on food parcels. Nevertheless, given the reported impact of the feeding program on OVC and families, it is clearly an essential service that should form part of any program model aimed at care for vulnerable children. Barriers related to human resource challenges (staff shortages and difficulty retaining volunteers) are also indicative of Noah’s model, particularly the types of services delivered and how human resources are utilized in implementation. Noah’s reliance on volunteers to support staff compounds staffing issues due to the difficulty in retaining volunteers. These issues suggest a need for the model to be adapted to ensure more sustainable a human resource structure. WILL ARKS BE ABLE TO CONTINUE SUPPORTING OVC IN FUTURE? WHAT STRUCTURES AND SYSTEMS EXIST TO ENABLE CONTINUED SERVICE PROVISION TO OVC (SUSTAINABILITY) AT ARKS? A core component of the Noah program model is building capacity of Arks to become sustainable community-based organizations. In light of the importance of providing lasting services for OVC, families, and communities, high level Ark staff and stakeholders were asked to comment on their Stage of Progression and the future of the Ark. Among participants asked, there is a general perception that services provided at Arks are sustainable. 64% of Ark Managers and 56% of community stakeholders suggested that essential services to OVC would continue in the future. Participants linked their potential for sustainability to a variety of resources, system, and structures within the Ark such as  facilities and equipment  trained Ark staff and volunteers  governance  financial management  human resources  monitoring and reporting  income generating projects  knowledge and skills in resource mobilization While these resources and systems represent organizational capacity to sustain services, practical challenges remain that compromise the potential for sustainability. As previously noted in the section on Key Barriers, staff and community stakeholders assert that the greatest barriers to sustainability are linked to funding, particularly identifying and securing new donors and satisfying the requirements for DSD funding. Noah UGM End of Project Evaluation 29 Table 23: Overview of Sustainability and Stage Progression Overview of Sustainability by Ark and Stage of Readiness and Progression Ark Stage of Readiness and Progression (2011) Sustainability Score (per Stage of Readiness and Progress Evaluation) Secured Other Funding (Outside of Noah) Income Generating Project Implemented Comments on Sustainability Activities Kliptown High Stage 3 (91%) 81% Yes Yes Planning to engage DSD, not yet submitted proposals Swaneville Mid Stage 3 (84%) 58% Yes No—in the planning stages Submitted a proposal to Edgars; Informal arrangement with Pick n’ Pay for donations Melusi High Stage 3 (95%) 83% Yes Yes Leveraged funding from local businesses and faith community despite the lack of potential funders/partners in the area Nkobongo Mid Stage 3 (86%) 62% Yes No Raised funds for a new facility; Receive in-kind donations from local business community Zamimpilo Low Stage 3 (82%) 60% No No Negotiating a partnership with forest companies, but funding has not yet been released Hlangabezu Low Stage 3 (80%) 53% Yes No—in the planning stages Planning to write letters to funders Source: Ark Manager and community stakeholder interviews As seen in Table 23, five out of six Arks have secured funding from new donors; however, all Ark Mangers reported that even with their current range of donors, additional long term funding is needed to maintain service over time. Furthermore, it is not known if these funds were secured independently or with support from Noah. Independently acquiring funding from new donors is a better predictor of whether Arks could continue fundraising in the future. In addition, two Ark Managers suggested that approaching donors on their own would be difficult given their other management responsibilities. Only two Arks (Melusi and Kliptown) have implemented income generating projects to supplement donor funding; both are High Stage 3 Arks with high sustainability scores, which suggests that this activity may only be feasible for more advanced Arks. All arks reported being engaged in some form of resource mobilization activity, though for some the focus was more on “planning” than “action.” While some Arks indicated plans to engage DSD, no Arks reported having secured a non-profit organization certificate, a prerequisite for DSD funding. Given that DSD is a natural South African Government (SAG) partner for Arks, efforts to address this barrier may need to be intensified to ensure Arks can continue providing services and impart sustainable changes in the lives of OVC and communities. Noah UGM End of Project Evaluation 30 V. CONCLUSIONS This sections summaries the key themes and findings according to the key questions and main purposes of the evaluation. Implications for future interventions by Noah are considered. PROGRAM OBJECTIVE ACHIEVEMENT Responding to the Needs of OVC As an organization, Noah seeks to establish and strengthen the capacity of CBOs and provide essential care and support services to OVC. Based upon the evaluation findings, Noah has responded to and met these broad objectives under the UGM project as follows  Noah has facilitated the provision of essential care and support services related to psychosocial care, HIV/AIDS prevention through implementing appropriate education programs for all OVC beneficiaries, ECD, educational assistance, and child protection through community-based Arks. Examples of such services include counseling/play therapy, HIV/AIDS prevention education, day care, homework supervision, numeracy and literacy programs, Soul Buddyz, help with accessing legal identification documents, and assistance with social grants.  Program and service delivery within Arks (in the sample) are responsive to the major needs and challenges of OVC including poverty (hunger, lack of material necessities like clothing, shoes, and school uniforms), psychosocial well-being (such as death of a parent or relative and coping with the loss, lack of friends), school, (poor performance, disinterest and lack of motivation), behavior problems, abuse and mistreat, and HIV/AIDS. One major challenge that is not directly addressed through Ark interventions in unemployment. However, participants asserted that Noah does contribute to reducing employment in a small way through the jobs and training made available at the Arks.  Stakeholders and beneficiaries generally perceive activities and services to be of good quality. In all sites, services are age appropriate, aligned with the Children’s Act and other standards of practice, and offered free of charge. However, in some communities the quality of services could be improved by addressing stigma associated with receiving services (related to HIV/AIDS, poverty, and being an orphan) and ensuring services are accessible to all registered children, regardless of where they live or attend school. PROGRAM OUTCOMES Changes in the well-being of OVC, families, and communities  Activities and services implemented by Noah’s Arks through PEPFAR funding have contributed to significant changes in the well-being of OVC including better school performance, decreased hunger, greater knowledge of HIV/AIDS prevention, improved psychosocial well-being including increased self confidence and the ability to communicate and express feelings, improved life skills (discipline, responsibility, decision making, and respect for others), and a greater sense of safety.  Although not funded by PEFPAR, the Noah/Ark nutrition program (daily meals and food parcels) has contributed to decreased hunger among OVC and their families; this is one of the most significant changes, from the beneficiary perspective.  OVC participation in Ark activities and services has also led to increased happiness in the household; improved relations between OVC, siblings, and other family members; and increased cooperation from OVC in the home (such as willingness to do chores). However, there were limited family-level changes that indicated increased capacity to care for OVC.  The well-being of communities has been improved due to the existence of Arks equipped to care for OVC. Further, Arks have helped to raise the awareness of OVC and mobilize the community to care for OVC as evident in increased community buy-in and involvement in Arks. Noah UGM End of Project Evaluation 31  Noah has changed the path of OVC by facilitating access to needed services at Arks that improved their well-being. OVC with the same needs that do not receive essential services through the Ark or another organization will likely be forced to cope with unmet needs and are vulnerable to further harm and risk behaviors.  Arks are key drivers in affecting positive changes in the lives of OVC through the work of front line staff and volunteers. As a partner and mentor to Arks, Noah serves as a catalyst for developing and mobilizing the Ark to deliver services through their funding, training, mentorship, and support. KEY ENABLERS AND BARRIERS Program strategies that are effective (or ineffective) in improving the lives of OVC  Key enablers in improving the well-being of OVC include securing community buy-in; sourcing staff that are dedicated, team players, and passionate about children; establishing partnerships with local businesses, civil society, and faith communities to leverage funding and in-kind resources; providing comprehensive training to staff on caring for OVC and ongoing mentorship support; and implementing programs that are evidenced-based, responsive to needs, and reflect a development approach.  Barriers faced by Arks in achieving desired outcomes include not having control over the purchasing of food for meals (due to the Noah Bulk Buying Initiative), an inability to expand services due to funding constraints, staff shortages, retaining volunteers when incentives cannot be provided, inadequate facilities (such as no electricity, toilets, insufficient space), not enough security, and serving children who live far from the site. Given their impact on the delivery and uptake of services, these barriers should be addressed to realize changes in target beneficiaries of OVC support programs. SUSTAINABILITY  As a result of the capacity building and support services provided by Noah, Arks have resources and systems (including financial management, governance, trained staff, a structure/facility, and equipment) in place to assist them in continuing services to OVC.  Arks still face challenges in progressing to fully independent, sustainable community-based organizations, most notably in securing additional funders who can provide long term support for operations and service delivery, and in meeting the requirements for DSD funding. More intensive support around resource mobilization may be required to better assist Arks in their transition to independence. Noah UGM End of Project Evaluation 32 VI. RECOMMENDATIONS Based on the evaluation findings, the following are recommendation for Noah, Ark staff, and others implementers that should be considered to improve existing OVC care and support services as well as enhance scale-up and replication of the model. PROGRAM DESIGN  Intervene at the family level: Given impact of the family on the well-being of a child, further intervention is needed within households, among family members and guardians. Noah should develop or identify interventions with families to increase impact and ensure changes in OVC are reinforced and sustained. For purposes of quality, efficiency, and effectiveness, interventions with the family should align with Arks’ existing service delivery and be responsive to family specific needs. According to a recent RFA concept note, Noah is already considering a comprehensive family centered program with interventions focused on child protection, household economic strengthening, HIV/AIDS and TV awareness, nutrition, and hygiene. Other family-level interventions Noah could consider include child abuse and protection education for guardians and family members, referrals for health and social services, and linking economically active family members with job or skills training opportunities. ORGANIZATIONAL DEVELOPMENT  Engage community volunteers to inform the development of a volunteer retention strategy: Noah invests time and resource into training and building capacity of volunteers because they play an integral role the delivery of services at Arks. As such, it is imperative that Arks have to have a strategy to retain volunteers, especially given the shortage of paid staff. It is recommended that Noah engage volunteers to find out why they stop volunteering and the types of incentives that would motivate them to stay. Based on feedback from volunteers, Noah can identify low or no cost incentives for volunteers. Noah could also investigate the feasibility of providing remuneration for volunteers (transitioning volunteers to causal paid employees). It may also be beneficial for Arks to implement a more selective volunteer recruitment process that includes screening to determine motivations for volunteering and the likelihood of retention.  Provide Arks with more intensive capacity building and support around resource mobilization: Despite training and support from ADOs, Ark staff and committee members report continuing challenges with funding to sustain services for OVC over time. Given these barriers, it is recommended that Noah intensify the resource mobilization component of the CBO development strategy to ensure that Arks have opportunities to apply the skills and competencies from training during the stage progression process. Some strategies to consider include  adopting targets for resource mobilization activities (such as, number of partnerships identified, number proposals written)  providing practical learning opportunities (joint proposal writing the ADO, trial interviews with funders)  attending relationship building engagements or “pitch meetings” with ADO These strategies could be integrated into the new revised stage progression approach (Thuthuka Model) of the Noah Realignment Strategy. PROGRAM IMPLEMENTATION  Continue with efforts to address challenges resulting from the BBI. Decreased hunger was one of the most significant changes brought about in OVC and families, and these outcomes were a direct result of the food/meals provide by Arks. Given the critical importance of food to outcomes and OVC participation in other Ark activities, the weaknesses of the BBI and revised program implemented in July 2011 should be addressed. It is clear from nutrition evaluation reports that Noah UGM End of Project Evaluation 33 Noah is actively pursuing solutions to these challenges. Noah should continue with these efforts and work towards devising a long term solution that adequately responds to nutritional needs of OVC, reflects the recommendations of Arks (including children), promotes empowerment, and is cost effective.  Continue to provide interventions linked to outcomes while addressing indentified gaps in services. Arks should continue to provide the range of essential care and support services that contribute to improving the lives of OVC (such as food, food parcels, psychosocial support services, HIV/AIDS prevention, child protection, educational support). Long term provision of these services will result in communities with safe and emotionally, psychologically, and physically healthy children.  Adapt the current program model to improve accessibility in some rural and peri-urban sites. Noah’s use of a center-based model to provide services to OVC presents some challenges with accessibility for some rural and peri-urban communities. In theses, an assessment should be done to determine the scope of inaccessibility and determine how many children face challenges in accessing services at the center because of the distance from their home or school, and whether these children are clustered in the same communities or coming different areas. These findings would help Arks to plan and deliver essentials services as required to children not able to attend the Ark in their homes. As the findings indicate, the services deemed most important by beneficiaries (help with accessing identification documents and social grants, home visits, homework supervision, and counseling/play therapy) could easily be delivered in a home setting. This approach coincides with Noah’s planned strategic shift towards programming aimed at family preservation and household economic strengthening. In places where accessibility is an issue, services to OVC could be coupled with services to the family.  Develop partnership with South African Police Service (SAPS). To help in addressing security concerns, Noah and the Arks should engage in relationship building and intentional partnership with SAPS during the establishment of Arks. Having an ongoing relationship with police authorities would likely improve SAPS response and resolution of crimes and security issues. Arks could also investigate the possibility of working with SAPS to establish a community policing or neighborhood watch function at the Ark using volunteers.  Improve Ark facilities to meet eligibility criteria for DSD funding. Noah has recently undergone an organizational development process resulting in a new focus on aligning with SAG and preparing Arks/CBOs for partnership with DSD. However, inadequacies in facilities and security issues threaten this potential partnership. Noah should work with Arks to address facilities and security issues to satisfy eligibility requirements. For existing Arks, this may include allocation of resources to make improvements and advocacy/engagement of government and relevant service providers to access basic services (water, sanitation, and electricity). For future Arks, DSD requirements should be used as guidelines for selecting or constructing facilities. Noah UGM End of Project Evaluation 34 APPENDIX I: EVALUATION SCOPE OF WORK Now in the fifth and final year of the project, FHI 360-UGM, at the request of USAID, is commissioning an external evaluation of our grantees. Partner organizations are non-governmental organizations (NGOs) working at national, provincial and local levels in South Africa, primarily implementing services related to services for orphans and vulnerable children (OVC), HIV care and support, HIV counseling and testing, and HIV prevention. These partners have received funding for a period of three to five years under PEFPAR, as well as both organizational and technical capacity building support. Feedback Research and Analytics is being contracted to execute evaluations for two of the UGM Partners: Heartbeat and Noah. The focus of the each partner evaluation will be to:  Determine whether the program objectives under each partner’s program were achieved  Evaluate the key program outcomes and impacts related to improved health and wellbeing of the targeted beneficiaries Most specifically, Feedback Research and Analytics will seek to answer the following key evaluation questions for Noah, utilizing tools, methods, and sub-questions approved reviewed and approved by FHI 360:  What were the most significant changes brought about by Noah improving the well-being of OVC in targeted communities?  To what extent was Noah able to address the needs of children within the community?  How do stakeholders (children, caregivers, DoSD, community representatives) perceive the program; in terms of quality and ease of access? The focus of the evaluation is to assess effectiveness of the partner organizations in addressing the needs of beneficiaries in targeted communities. The evaluators will be required to carefully consider the suitability and feasibility of design options that are likely to offer the best chance of establishing the value of the program in responding to the needs of targeted beneficiaries and communities. Both qualitative and quantitative data collection techniques should be employed. Data will be collected from various sources using appropriate data collection methods and tools for any given evaluation question. The final design to be employed will be determined after the contractor has had a chance to undertake a front-end analysis and is therefore able to select the best design option that specifies; which people or units will be studied; how they will be selected and the kinds of comparison that should be made. Data will be collected from various program sites for each partner. Evaluations will be undertaken in two stages and with expected outcomes for each stage as expressed below: Stage 1: Finalization of Evaluation Protocol Contractor will refine an evaluation protocol which demonstrates:  Understanding the relationship between program stages and the proposed broad evaluation question  Understanding the context for program delivery and key factors that influence program implementation  Understanding the existing theoretical and empirical knowledge about the program and examining program theory  A comprehensive stakeholder analysis and determination of roles of key stakeholders in the evaluation Noah UGM End of Project Evaluation 35  Balancing costs and benefits of the evaluation and advising on the most strategic questions to include in the evaluation  Developing the Finalized Implementation Strategy and Methodology Report Stage 2: Implementation of the Evaluation Contractor will implement the partner evaluation following submission and approval of the Implementation Strategy and Methodology Report:  Pre-test instruments  Train data collectors  Undertake the evaluation data gathering process  Prepare data for analysis  Clean data  Enter data into electronic data analysis systems  Undertake comprehensive data analysis  Formulate the findings During the period of performance of April 9, 2012 – July 31, 2012, payment to Feedback Research & Analytics will be fixed price based on the payment schedule determined by the deliverables below: MILESTONES DELIVERABLES DUE DATE Data collection (April 9 – May 11) 1. Finalized Implementation Strategy and Methodology Report submitted to FHI 360 (Noah) April 13 2. Evaluation Work Plan, including key activities and timeframes submitted to FHI 360 (Noah) April 13 3. Data Analysis Plan, including dummy table/graphs for presenting data submitted to FHI 360 (Noah) May 4 Data analysis and development of PowerPoint Presentation, including summary of evaluation process and results (May 14 – June 1) 4. Oral and PowerPoint Presentation (half-day) of preliminary findings to USAID, FHI 360, and partner (May 28- June 1, 2012) (Noah) Presentations completed by June 1 Development of final written report, including an executive summary with highlights of the evaluation and key findings (June 1 – July 31) 5. Draft written report submitted to FHI 360 (Noah) June 15 6. Final report submitted to FHI 360 (Noah) July 31 Development of brief paper (two-pager) for each partner, targeting community audiences on key findings from the evaluation (June 1 – July 31) 7. Two-page papers submitted to FHI 360 (Noah) July 31 Noah UGM End of Project Evaluation 36 APPENDIX II: DATA COLLECTIONS TOOLS Key Informant Interview Guide  Name of person(s) being interviewed  Designation (current occupational role)  Role at Noah  Length of involvement with Noah (yr) A. GOALS AND OBJECTIVES A1. What is the overall goal of the programs/services of the FHI 360-UGM project? A2. How do the goals of the FHI 360-UGM project fit into the overall organizational vision/mission? A3. What are the objectives of the FHI 360-UGM project? B. PROGRAM DESIGN B1. What are the key programs and services offered to OVC and their families? B2. What is the rationale for providing these services for OVC? B3. Which of these programs/services do you believe has the most influence on OVC well-being? Why do you say so? B4. What programs and services does Noah provide for community organizations and groups? B4.1. How do these interventions enhance service delivery to OVC? B5. How are programs and services structured to ensure the following  Accessible for OVC and their families:  Affordability for OVC and their families:  Child-centeredness (e.g. age appropriate):  Services are delivered in a stigma-free environment:  Alignment with best practice or established standards of care for  OVC (e.g., compliance with Child Act): C. PROGRAM OBJECTIVE ACHIEVEMENT C1. What progress has been made in achieving the objectives of FHI 360-UGM project? Probe for each objective listed:  Providing psychological care to OVC:  Providing child protection services to OVC (e.g., ID document, birth certificate, and  child protection programs):  Conducting home visits to OVC and their families:  Providing education assistance to OVC through aftercare and literacy programs:  Providing early childhood development programs:  Providing HIV/AIDS prevention programs: C1.1. What factors have enabled the program to meet its objectives? Probe for the following:  Organizational structure:  Program design:  Implementation factors:  Partnerships:  Community buy-in: Noah UGM End of Project Evaluation 37  Training and support:  Resources: C1.2. What factors have been barriers to the program meeting its objectives?  Organizational structure:  Program design:  Implementation factors:  Partnerships:  Community buy-in:  Training and support:  Resources D. PROGRAM OUTCOMES AND IMPACT: OVC WELL-BEING D1. What changes have been brought about by the program for OVCs, families, and communities?  PROBE: Can you provide examples of specific changes in OVC related to health, educational readiness, school performance, nutrition, life skills, HIV/AIDS prevention knowledge, or psychosocial well-being?  PROBE: Can you provide examples of specific changes for families?  PROBE: Can you provide specific examples of changes for communities? D2.Have the changes brought about by the program lasted over time? D3. What role did the CAC and volunteers play in achieving these changes in OVC well-being?  PROBE: How much of the changes observed in OVC, families, and the community can be attributed to Child Activity Coordinators and other Ark staff? D4. What structures and systems are in place to ensure programs and services for OVC continue in future?  PROBE: Will support to OVC continue once the FHI 360-UGM project has ended? E. OTHER COMMENTS OR QUESTIONS E1. What lessons have been learned during the implementation of the FHI 360-UGM?  Probe: Program design, implementation, outcomes, sustainability E2. What else should be done in future to improve in the well-being of OVC and their families? E3. Do you have any other comments? Is there anything else we should know about Noah as we conduct this evaluation? Noah UGM End of Project Evaluation 38 Ark Manager Interview Guide  Name:  Designation:  How long have you been involved with the Ark?  Description of the role A. BACKGROUND A1. When did you first become involved in caring for OVC?  PROBE: Was it before or after you became involved with this Ark? A2. Tell me the story of how the Ark was started?  PROBE: When did the partnership with Noah begin? B. ACTIVITIES AND SUPPORT FROM NOAH B1. What activities do you have for OVC and their families? B2. Why did you decide to offer these activities? B3. How are activities structured to ensure the following  Access for OVC and their families:  Affordability for OVC and their families:  Child-centeredness (e.g. age appropriate, space and materials suitable for all age groups):  Services are delivered in a stigma-free environment:  Align with best practice or standards of care for OVC C. SUPPORT FROM NOAH C1. How has Noah assisted in establishing or expanding your services to OVC?  PROBE: What training has been provided to you or your staff?  PROBE: How have you or your staff be mentored?  PROBE: How has Noah supervised your work? C2. What are you personally able to do now that you could not do before the training, support from Noah? C2.1 Have you increased your knowledge or skills? If yes, in what. C2.2 Are you more confident in your ability to take care of orphans in your community? C3. How would you describe the quality of the support provided by Noah?  PROBE: Rating on a scale of 1 to 10. C4. How would you describe the Ark’s relationship with Noah?  PROBE: What are the strengths of the partnership?  PROBE: What challenges have you faced in working with Noah? D. PROGRAM OUTCOMES: CHANGES IN OVC WELL-BEING D1. What changes have been brought about by the program for OVCs, families, and communities?  PROBE: Can you provide examples of specific changes in OVC related to health, educational readiness, nutrition, life skills, HIV/AIDS prevention knowledge, psychosocial well-being?  PROBE: Can you provide examples of specific changes for families?  PROBE: Can you provide specific examples of changes for communities? D2. What has helped you to make these changes in OVC and their families? Noah UGM End of Project Evaluation 39 PROBE on the following:  Training and capacity build support from Noah:  Community support and buy-in:  Resources:  Partnerships:  Other: D3. Will the changes brought about by the activities last over time? D4. What has made it difficult to improve the well-being of OVC? E. SUSTAINABILITY E1. How do you currently fund your OVC care and support activities?  PROBE: Who are your primary funders?  PROBE: Have you sought funding from other sources? Why or why not? E2. Where are you in the process of becoming a Stage 4 Ark (i.e. fully independent OVC organization)?  PROBE: What will assist you in meeting this goal?  PROBE: What challenges do you foresee? E3. Will the activities you provide for OVC continue in future?  PROBE: What systems are in place to ensure this (e.g., fundraising, governance)? F. OTHER COMMENTS OR QUESTIONS F1. What else should be done in future to improve in the well-being of OVC and their families? F2. Do you have any other comments? Noah UGM End of Project Evaluation 40 Child Activity Coordinator, Volunteer, Other Staff Interview Guide  Name of person(s) being interviewed  Designation: ___Child Activity Coordinator ___Volunteer  Length of involvement with the Ark:  Description of role: A. BACKGROUND A1. When did you first become involved in caring for OVC?  PROBE: Was it before or after you became involved with this Ark? B. ACTIVITIES B1. What activities do you provide for OVC and their families? B2. Why does the Ark provide these activities? B3. How do you ensure that the activities you provide are:  Accessible for OVC and their families:  Affordable for OVC and their families:  Child-centeredness (e.g. age appropriate):  Delivered in a stigma-free environment:  In agreement with best practice standards of care for OVC: C. SUPPORT FROM THE ARK C1. How has the Ark helped you in providing activities to OVC?  PROBE: What training or support have you received?  PROBE: Have you increased your knowledge or skills? If yes, in what? C2. How would you describe the quality of the training and support provided by the Ark?  PROBE: Rating on a scale of 1 to 10. D. PROGRAM OUTCOMES: CHANGES IN OVC WELL-BEING D1. What changes have been seen in OVCs, families, and communities as a result of the program?  PROBE: Can you provide examples of specific changes in OVC related to health, educational readiness, nutrition, life skills, HIV/AIDS prevention knowledge, psychosocial well-being?  PROBE: Can you provide examples of specific changes for families?  PROBE: Can you provide specific examples of changes for communities? D1.1. Were the changes immediate or did they take time? D1.2. What impact did the changes in OVC have on their families?  PROBE: Were they negative or positive? Give examples. D2. What has helped you to make these changes in OVC and their families? PROBE for the following:  Training and capacity build support from Noah:  Community support and buy-in:  Resources:  Partnerships:  Other: Noah UGM End of Project Evaluation 41 D3. Have the changes brought about by the activities lasted over time? D4. What challenges have you faced in improving the well-being of OVC? E. OTHER COMMENTS E1. What lessons have been learnt about how to care and support orphans? E2. What else should be done in future to improve in the well-being of OVC and their families? E3. Do you have any other comments? Noah UGM End of Project Evaluation 42 Community Stakeholder Interview Guide  Name:  Designation:  Length of involvement with Noah:  Role at Noah: A. BACKGROUND A1. Describe the challenges faced by OVC in this community? A2. What role does the Ark play in this community? B. ACTIVITIES B1. What activities does the Ark provide to OVC and their families? B2. Do you think these activities are meeting the needs of OVC in the community? B3. Which activities do you believe have the most influence on OVC well-being? B4. Do you know how the Ark’s activities are structured to ensure  Accessibility for OVC and their families:  Affordability for OVC and their families:  Child-centeredness (e.g. age appropriate):  Services are delivered in a stigma-free environment:  Alignment with best practice or standards of care for OVC : C. PROGRAM OUTCOMES AND IMPACT: OVC WELL-BEING C1. What changes have been brought about by the activities for OVCs, families, and communities?  PROBE: Can you provide examples of specific changes in OVC related to health, educational readiness, school performance, nutrition, life skills, HIV/AIDS prevention knowledge, psychosocial well-being?  PROBE: Can you provide examples of specific changes for families?  PROBE: Can you provide specific examples of changes for communities? C2. Will the changes brought about by the activities last over time? Why or why not? C3. How important were Child Activity Coordinators and Volunteers in achieving these changes?  PROBE: Would these changes have occurred with the services and support that Child Activity Coordinator’s provide to OVC and their families? C4. What changes have you observed in the community’s ability to care for OVC? C4.1 To what do you attribute these changes?  PROBE: Has the Ark had any impact on these changes? C5. What has helped the Ark to make these changes in the lives of OVC? Probe for these areas if not mentioned:  Organizational structure (staff arrangement):  Activity design/organization/running:  Community buy-in/ownership:  Partnerships:  Other: C6. What challenges has the Ark faced in addressing the needs of OVC? D. SUSTAINABILITY Noah UGM End of Project Evaluation 43 D1. [Committee Member Only] Will care and support for OVC provided by the Ark be able to continue in future?  PROBE: What systems (e.g., fundraising, governance, and partnerships) are in place to ensure activities for OVC continue? E. OTHER COMMENTS OR QUESTIONS E1. What lessons have been learnt about how to care and support orphans?  Probe: What advice would you give another community wanting to start an Ark? E2. What else should be done in future to improve in the well-being of OVC and their families? E3. Do you have any other comments? Noah UGM End of Project Evaluation 44 OVC Caregiver Focus Group Guide A. WELCOME Welcome Welcome to this focus group discussion. We are here today to learn about your experiences with the programs and services provided at this center (Ark). This discussion will last for about 90 minutes. Please feel free to have some refreshments while we talk. Facilitator Introduction My name is [Facilitator Name] and I will be leading the discussion. I am from Feedback Research and Analytics and we are conducting this focus group as part of an evaluation of Noah, an NGO that supports this center (Ark). I am here today with my colleagues [Insert Names]. They will be helping me to lead the discussion and take notes. We will also be recording this session to make sure we don’t miss anything that you say and to help us write our report. B. INFORMED CONSENT Consent to Participate Your participation in this discussion is voluntary. You have the right to stop participating or leave at any time. The information you share today will be kept anonymous. This means that we will not record or use your name and no one outside of our team will know what you said. Please take time to read the consent form. If you have difficulty reading or understanding the form, please let me know and I will talk you through it. If you are willing to take part in the focus group, sign the consent form and hand to myself or one of my colleagues. C. PARTICIPANT INTRODUCTIONS Some of you might already know each other, but for the purpose of the discussion and for me to get to know you a bit better, please give a short introduction of yourself. Tell me your name, some of your hobbies, and how long you or your child has been involved with the center. Thank you. Now that we have introduced ourselves, let’s start the discussion. From now on, please feel free to speak and remember that there are no right or wrong answers. We are simply interested in your opinions. Finally, let’s all agree to respect each others’ opinions. D. DISCUSSION QUESTIONS D1. Tell me what you know about the activities at this center/Ark? D2. How did you find out about this center/Ark? What made you get involved? D3. What activities does your child participate in though this center/Ark? FACILITATOR: Take a count for each activity.  Homework help:  Food, food parcels:  Gardening:  Home visits:  Day care:  Aftercare:  Numeracy and Literacy program:  Soul Buddyz program (HIV/AIDS and life skills):  Counseling:  Help with ID and birth certificate, child protection:  Other: Noah UGM End of Project Evaluation 45 D4. Before your child was involved with the center/Ark, what concerned you most about taking care of them? D5. How has the center/Ark helped you to deal with these concerns? D6. How is your child’s life different since coming to the center/Ark? PROBE: Changes in:  Nutrition (e.g. having a regular meal)  Health (e.g. identification of health concerns)  School (e.g., readiness for school, marks, homework)  HIV/AIDS prevention knowledge (e.g., what they know about protecting themselves)  Referrals ( e.g., ability to get other needed services)  Emotional and psychosocial well-being (e.g., happiness, confidence, aware of where to go for help if needed)  Access to ID, birth certificates, grants D6.1 Were the changes immediate or did it take time? D6.1 How much did the work of the Child Activity Coordinator/Volunteer contribute to these changes?  PROBE: Would these changes have occurred without the services and support provided by the Child Activity Coordinator and Volunteers? D7. Have these changes lasted over time? Why or Why not? PROBE for each area of change  Nutrition (e.g. having a regular meal)  Health (e.g. identification of health concerns)  School (e.g., readiness for school, marks, homework)  HIV/AIDS prevention knowledge (e.g., what they know about protecting themselves)  Referrals ( e.g., ability to get other needed services)  Emotional and psychosocial well-being (e.g., happiness, confidence, aware of where to go for help if needed) D8. How has your child’s involvement at the center/Ark impacted the rest of the family?  PROBE: Positive and negative impacts D9. Does your child like coming to the center/Ark or participating in activities? Why or why not?  PROBE: What do you think they enjoy most? D10. How do the staff and volunteers interact with the children? D11. Are the activities offered appropriate for children? D12. Do you have any trouble accessing the activities?  PROBE for issues with location, transport E. CLOSING E1. What else should be done in future to improve the well-being of OVC and their families in your community? E2. Are there any comments? Noah UGM End of Project Evaluation 46 OVC Focus Group Guide A. WELCOME Welcome Welcome to this focus group discussion. We are here today to learn about your experiences with the programs and services provided at this center (Ark). This discussion will last for about 90 minutes. Facilitator Introduction My name is [Facilitator Name] and I will be leading this discussion. I am from Feedback Research and Analytics and we are conducting this focus group as part of an evaluation of Noah, an NGO that supports this center (Ark). I am here today with my colleagues [Insert Names]. They will be helping me to lead the discussion and take notes. We will also be recording this session to make sure we don’t miss anything that you say and to help us write our report. B. INFORMED CONSENT Consent to Participate Your participation in this discussion is voluntary. You have the right to stop participating or leave at any time. The information you share today will be kept anonymous. This means that we will not record or use your name and no one outside of our team will know what you said. Please take time to read the consent form. If you find it hard to read or understand the form, please let me know and I will talk you through it. If you are willing to take part in the focus group, sign the consent form and leave it on the table. C. PARTICIPANT INTRODUCTIONS Some of you might already know each other, but for the purpose of the discussion and for me to get to know you a bit better, please could you give a short introduction of yourself: tell everyone your name, where you go to school or work and some of your hobbies. Thank you. Now that we have introduced ourselves, that was the only formal part of our discussion here today. From now on, please feel free to speak and remember that there are no right or wrong answers. We are simply interested in your opinions. Before we get started let’s review the some suggestions for ground rules for our discussion today. 1. One person speaks at a time 2. Raise your hand if you have something to say 3. Listen to what other people are saying 4. No mocking or attacking other people's ideas 5. Respect each other. Would you like to add any others? D. DISCUSSION QUESTIONS D1. Tell me what you know about the activities at this center/Ark? D2. How did you find out about this center/Ark? What made you get involved? D3. What activities do you participate in at the center/Ark? FACILITATOR: Take a count for each activity/service.  Education support  Food, food parcels  Gardening  Home visits  Soul Buddyz (life skills and peer education)  HIV/AIDS prevention education Noah UGM End of Project Evaluation 47  Referrals for to a clinic or doctor  Counseling, social support programs  Child protection programs (about the rights you have as a child)  Help with ID and birth certificate  Help with applying for social grants D 4. CHALLENGES ACTIVITY - Time Frame: 15mins Now, we will do an activity that will help us to understand more about your concerns and worries. We have passed out two different colors of Post-Its or Sticky Notes. Take a few of each color. On the YELLOW Post-Its, write the answer to the following question. Before you were involved in activities at the center/Ark, what were some of the challenges in life? Please think about your life now, since you’ve been participating in the activities as this center/Ark. On the PINK Post-Its or Sticky Notes, please write the answer to the following question. How has the center/Ark helped you to deal with these challenges/problems? When you are finished put your Post-Its on the flip chart paper marked CHALLENGE and HOW THE CENTER HELPED. [FACILITATOR]: Ask some participants to share what they wrote down. [FACILITATOR]: Summarize the challenges and solutions noted on the flip chart papers. [FACILITATOR]: Add an icebreaker here depending on group needs. D5. How is your life different since you became involved with the center/Ark? PROBE: Changes in:  Health (living a healthy lifestyle, no substance use)  School (e.g., marks, progress to the next grade, matriculation  Outlook on life? (e.g., goals for the future)  Confidence (e.g., feelings about oneself and abilities)  Knowledge about protecting yourself from HIV/AIDS  Knowledge about positive living with HIV/AIDS  Knowledge about how HIV/AIDS stigma is damaging  Ability to get social grants and their benefits D6. Of the changes you’ve mentioned, which do you think are the greatest or most significant? D7. GREATEST CHANGE ACTVITY – Time Frame: 10 mins This activity will help us to understand which Ark activities had the greatest impact on the changes in your life. Part 1. We are passing out a packet of cards. Each card is labeled with an activity. As group, identify select the cards for the activities you have participated in through the Ark. [FACILITATOR] Check in with group after 2 minutes. Part 2. Now, using the cards that are left, rank the programs and services in order based on how much they influenced the changes in your life since coming to the Ark. The activity that had the greatest impact on the changes in your life should be ranked first, while the programs or activities that had the least impact should be ranked last. Noah UGM End of Project Evaluation 48 D7.1. Tell me about how you ranked the activities. D7.2. Why were these activities more important to the changes in your life? [FACILITATOR]: Probe participation for consensus and agreement. D8. Have these changes lasted over time? Why or Why not? PROBE for each area of change:  Health (living a healthy lifestyle, no substance use)  School (e.g., marks, progress to the next grade, matriculation)  Outlook on life? (e.g., goals for the future)  Confidence (e.g., feelings about oneself and abilities)  Knowledge about protecting yourself from HIV/AIDS  Knowledge about positive living with HIV/AIDS  Knowledge about how HIV/AIDS stigma is damaging  Ability to get social grants and their benefits D9. How has your involvement at the center/Ark impacted the rest of your family?  PROBE: Positive and negative impacts D10. Do you like coming to the center/Ark or participating in activities? Why or why not? D10.1 What do you enjoy most? D11. How do the staff and volunteers at the center/Ark interact with you?  PROBE: Do you think you are treated with care and respect? D12. Are the activities offered suitable for your age group? D13. Do you have any trouble getting to the activities at the center?  PROBE for issues with location, transport E. CLOSING E1. What else should be done in future to improve the lives of young people in your community? E2. Are there any comments that you would like to make that we didn’t touch on during the discussion?  PROBE: Is there anything you would like to say about the Ark or Noah? Noah UGM End of Project Evaluation 49 APPENDIX III: EVALUATION WORK PLAN EVALUATION WORK PLAN Activity Time Frame FHI 360 Evaluators Workshop March 28–30 Refine Evaluation Questions April 2–6 Develop Evaluation Instruments April 2–6 Develop Evaluation Protocol April 9–13 Submit the Evaluation Protocol (Implementation Strategy, Methodology Report, and Evaluation Plan) April 13 Conduct Key Informant Interviews April 16–20 Scheduling Site Visits April 16–20 Develop Field Worker Training Materials April 9–17 Field Work Training April 18 Conduct field work at Kliptown Ark – Gauteng (Pilot) April 19 Workshop and Revision of Tool April 20 Conduct field work at Swaneville Ark – Gauteng April 23–24 Conduct field work at Melusi Ark - Kwa-Zulu Natal April 25–26 Develop Data Analysis Plan April 30–May 4 Conduct field work at Nkobongo Ark - Kwa-Zulu Natal May 2–4 Submit Data Analysis Plan May 4 Conduct field work at Zamimpilo Ark - Kwa-Zulu Natal May 7–8 Conduct field work at Hlangabezu Ark - Kwa-Zulu Natal May 9–10 Data Analysis May 14–June 1 Oral presentation on Preliminary Findings June 5 Develop Draft Report June 1–June 15 Submit Draft Report June 18 Revise Report and Develop Two Page Findings Summary June 22–29 Submit Final Report and Two Page Summary for Partner June 29 Noah UGM End of Project Evaluation 50 APPENDIX IV: SAMPLING FRAMEWORK Overview of Sample Arks ARK Province Type of Community District Age of Ark Stage of Readiness and Progress Type of Service Kliptown GP Peri-urban (informal settlement) City of Johannesburg 2006 High Stage 3 (91%) Both Swaneville GP Peri-urban West Rand 2004 Mid Stage 3 (84%) Both Melusi KZN Rural uMkhanyakude 2003 High Stage 3 (95%) Aftercare Nkobongo KZN Peri-urban (township) Ilembe 2004 Mid Stage 3 (86%) Aftercare Zamimpilo KZN Peri-urban uThungulu 2004 Low Stage 3 (82%) Both Hlangabezu KZN Peri-urban eThekwini 2003 Low Stage 3 (80%) Aftercare Noah UGM End of Project Evaluation 51 APPENDIX V: COMPOSITION OF EVALUATION TEAM Noah End of Project Partner Evaluation Team Composition Team Member Role Terence Beney Senior Evaluator, Technical Lead Rebekah King Project Manager Mokete Mokone Field Work Coordinator Washington Dudu Field Worker Pule Goqo Field Work Elizabeth Zishiri Evaluation Assistant Fazeela Hoosen Evaluation Assistant Daleen Botha Contracts Management Noah UGM End of Project Evaluation 52 APPENDIX VI: NOAH ORGANOGRAM Noah UGM End of Project Evaluation 53 APPENDIX VII: TYPICAL ARK STRUCTURE Noah UGM End of Project Evaluation 54 APPENDIX VIII: DEMOGRAPHIC PROFILE OF BENEFICIARIES FOCUS GROUP PARTICIPANTS Demographic Profile of OVC Focus Group Participants Number of Focus Groups 9 Total Number of OVC 149 Age Range 12–14 54% 15–17 25% 18+ 3% Missing Data 10% Gender Male 44% Female 34% Missing Data 22% Year Started At Ark Before 2007 21% 2007 – 2011 58% After 2011 5% Missing 15% Source: OVC focus groups Demographic Profile of Caregiver Focus Group Participants Number of Focus Groups with Caregivers 7 Total Number of Caregivers 82 Total Number of Children Have at Ark 1708 Average Number of Children per Caregiver 2.3 ~ 3 Age Range of Children Less than 2 1% 2 – 6 23% 7 – 11 39% 12 -14 47% 15 – 17 13% 18+ 1% Missing Data 49% Gender of Children Male 73% Female 50% Missing Data 47% Year Children Started At Ark Before 2007 11% 2007 – 2011 32% After 2011 11% Missing 46% Source: Caregiver focus group consent forms                                                              8 Data on the number of children at the Ark was missing for 11 caregivers. Noah UGM End of Project Evaluation 55 APPENDIX IX: SUMMARY OF DATA ANALYSIS APPROACH Data Analysis Approach Evaluation Question Data Collection Method Analysis Method and Procedure Comparisons To what extent was Noah able to address the needs of children within the community? Interview Focus Group Thematic Analysis Frequencies of Responses by Theme Arks Which services were most valuable to OVC, their families, and the community? Interview Focus Group Secondary Data Thematic Analysis Frequencies of Responses by Theme Arks Participant Type How do stakeholders perceive the program in terms of quality and ease of access? Interview Focus Group Thematic Analysis Frequencies of Responses by Theme Arks What were the key enablers and barriers in meeting project objectives? Interview Focus Group Thematic Analysis Frequencies of Responses by Theme Arks Participant Type Ark Stage of Progression What were the most significant changes brought about by Noah in improving the well-being of OVC in targeted communities? Interview Focus Group Document Review Thematic Analysis Frequencies of Responses by Theme Arks Participant Type Province What role did the Arks play in achieving these outcomes/changes? What role did Noah play in mobilizing Arks to fulfill this role? Interview Focus Group Thematic Analysis Frequencies of Responses by Theme Not applicable Will Arks be able to continue supporting OVC in future? What structures and systems exist to enable continued service provision to OVC among the Arks? Interview Focus Group Thematic Analysis Frequencies of Responses by Theme Arks Age Stage of Progression Noah UGM End of Project Evaluation 56 APPENDIX X: OVC SELF REPORT OF PARTICIPATION IN ARK ACTIVITIES AND SERVICES Percentage of OVC Participating Activities and Services by Ark Service/Activity Kliptown Swaneville Melusi Nkobongo Zamimpilo Hlangabezu ALL n=149 Education support (Homework help) 80 29 73 20 63 10 48 Food parcels, food 80 100 94 19 59 32 86 Gardening 20 53 58 20 63 20 48 Home visits 100 94 70 23 72 14 61 Soul Buddyz 100 47 0 0 0 3 17 HIV/AIDS prevention education 100 47 94 30 94 25 85 Psychosocial support (counseling, play therapy) 73 35 45 13 41 0 30 Help with ID, birth certificate 47 0 6 3 9 6 13 Child protection (learning about children’s rights) 100 6 48 16 50 17 50 Referrals to clinic or doctor 0 0 21 0 0 0 5 Help with applying for social grant 0 6 6 1 3 0 3 Source: OVC focus groups Noah UGM End of Project Evaluation 57 APPENDIX XI: PERSONS CONTACTED Noah Dr. Gregory Ash, Founder and Vice Chairman of the Board of Directors Rachel Compaan, Programmes Development Manager Sipho Luthuli, Ark Development Officer Prince Magawaza, Ark Development Officer Nomxolisi Malope, M&E Manager Rufaro Musvaire, Assistant M&E Manager Sikhumbuzo Thabede, Programme Implementation Manager Noah UGM End of Project Evaluation 58 REFERENCES Collingwood, C., C. Chames, and D. Lomofsky. Monitoring, Evaluation, and Reporting Framework and Plan: Capacity building and mentoring strategy for CBOs. Southern Hemisphere Consultants, October 2011. Michel, D., N. Malope, B. Sebastian, and R. Musvaire. Noah District Research. Johannesburg, South Africa: The Monitoring and Evaluation Department of Noah, March 2011. Lomofsky, D. Programme framework, program theory and result chain for Noah. Southern Hemisphere Consultants, March 2011. Noah. Concept Paper for RFA-674-12-000006: Johannesburg, South Africa, June 2012. Noah. COP10 Partner Project Plan. Pretoria, South Africa: FHI 360-UGM Project (Undated). Noah. COP11 Partner Project Plan. Pretoria, South Africa: FHI 360-UGM Project (Undated). Noah. Formative Evaluation Report. Johannesburg, South Africa: Michael and Susan Dell Foundation, March 2012. Noah. Noah Bulk Buying Initiative Baseline Evaluation. Johannesburg, South Africa: Michael and Susan Dell Foundation, February 2011. Noah. Summary Document for the Noah Board: The Noah Realignment Strategy. Johannesburg, South Africa (Undated). Noah. Stage Readiness and Progression Evaluation Report. Johannesburg, South Africa: Monitoring and Evaluation Department, September 2011. Noah. Noah Bulk Buying Initiative Baseline Evaluation. Johannesburg, South Africa: Michael and Susan Dell Foundation, February 2011.