The USAID | Project SEARCH, Orphans and Vulnerable Children Comprehensive Action Research (OVC-CARE) Task Order, is funded by the U.S. Agency for International Development under Contract No. GHH-I-00-07-00023-00, beginning August 1, 2008. OVC-CARE Task Order is implemented by Boston University. The opinions expressed herein are those of the authors and do not necessarily reflect the views of the funding agency. Prepared by: Nancy Scott, Melissa Pfaff, Katherine Semrau, JoAnna Leyenaar, Anna Suojanen, Thomas Mugala, Jonathon Simon AUGUST 2010 End-of-Project Evaluation Report Faith-based Regional Initiative For Orphans and Vulnerable Children (FABRIC) Boston University OVC-CARE Project Family Health International Research Assistants (Zambia Based): Portipher Pilingana Yoram Banda Petan Hamazakaza – Costing Component Violet Mwanza Nomsa Manda Aaron Tembo Tunika Zulu Mirriam Phiri Edward Chanda Mwalimu Shuko Mwangala Monde Ndala Clara Bosha Hellen Kumwenda Sinfukwe Ireen Kayamba Christopher Besa Joseph Tembo Research Assistants (South Africa Based): Development Research Agency (DRA) Sarah Laurence – Costing Component, HDA Research Assistants (Namibia Based): Alexia Naris Ingrid Louis Theofilus Joseph Center for Global Health and Development Boston University School of Public Health Boston, Massachusetts, USA and Family Health International, Namibia, South Africa, and Zambia This evaluation and report would not have been possible without the support and commitment of the FABRIC Project Director, Linda Lovick, the Zambia FHI Country Director, Cathy Thompson, and FABRIC project staff including Elizabeth Mwite, Thomas Mugala, Marika Matengu, Eva Weitz, Yvonne Pande, Casper Erichsen, Harold Msiza, and Innocent Ngenzi. We thank our BU data entry team, Dinah Namtowe and James Lungu. We would also like to acknowledge the technical input from Malcolm Bryant, Godfrey Biemba, Candace Miller, Bram Brooks, Gina Etheridge, and our USAID representative, Andrea Halverson. FABRIC End-of-Project Evaluation Final Report C O N T E N T S LIST OF ACRONYMS ...........................................................................................................................................iv EXECUTIVE SUMMARY .......................................................................................................................................i INTRODUCTION ...................................................................................................................................................1 Overview.....................................................................................................................................................................1 Background and Rationale .........................................................................................................................................2 ASSESSMENT OF CHILD WELLBEING: SOUTH AFRICA, ZAMBIA .........................................................7 Overview.....................................................................................................................................................................7 Methods and Study Site..............................................................................................................................................8 Sample Size and Procedures.......................................................................................................................................8 Data Management and Analysis............................................................................................................................. 10 Results ..................................................................................................................................................................... 11 Characteristics of the Sample .............................................................................................................................. 11 FABRIC Service Delivery and Child Wellbeing ...................................................................................................... 12 1. Education ......................................................................................................................................................... 13 2. Psychosocial Support ....................................................................................................................................... 16 3. Food and Nutrition ......................................................................................................................................... 20 4. Healthcare ...................................................................................................................................................... 23 5. Shelter and Care .............................................................................................................................................. 26 6. Child Protection and Legal Aid ........................................................................................................................ 28 ASSESSMENT OF CHILD WELLBEING: NAMIBIA .................................................................................... 33 Overview.................................................................................................................................................................. 33 Study Site and Population........................................................................................................................................ 33 Methods and Procedures......................................................................................................................................... 33 Results ..................................................................................................................................................................... 34 Discussion ................................................................................................................................................................ 40 ORGANIZATIONAL ASSESSMENT: SOUTH AFRICA, ZAMBIA, NAMIBIA.......................................... 45 Overview.................................................................................................................................................................. 45 Methods................................................................................................................................................................... 45 Results ..................................................................................................................................................................... 46 Discussion ................................................................................................................................................................ 51 Conclusion................................................................................................................................................................ 53 COSTING OF OVC SERVICE DELIVERY: SOUTH AFRICA, ZAMBIA ..................................................... 57 Overview.................................................................................................................................................................. 57 Study sites and Methods ......................................................................................................................................... 57 Summary Results and Discussion............................................................................................................................. 60 SUMMARY CONCLUSIONS AND SPECIFIC RECOMMENDATIONS ...................................................... 71 Summary Conclusions.............................................................................................................................................. 71 Specific Recommendations...................................................................................................................................... 73 REFERENCES ...................................................................................................................................................... 75 APPENDICIES ..................................................................................................................................................... 76 FABRIC End-of-Project Evaluation Final Report LIST OF ACRONYMS BU Boston University CAFO Church Alliance for Orphans CGHD Center for Global Health and Development CIHDZ Center for International Health and Development – Zambia CV Children’s Voices DRA Development Research Africa ECR Expanded Church Response, Zambia EoPE End of Project Evaluation FABRIC Faith-based Regional Initiative for Vulnerable Children FHI Family Health International IA Implementing agency IRB Institutional Review Board OGAC Office of the Global AIDS Coordinator OHA Office of HIV/AIDS OVC Orphaned and Vulnerable Children PEPFAR President's Emergency Plan for AIDS Relief PV Positive Vibes SACBC Southern African Catholic Bishops’ Conference SDQ Strengths and Difficulties Questionnaire SR Sub-recipient USAID United States Agency for International Development USG United States Government FABRIC End-of-Project Evaluation Final Report i EXECUTIVE SUMMARY This report provides findings from the End-of-Project Evaluation (EoPE) of Family Health International’s Community Faith Based Regional Initiative for Orphans and Vulnerable Children (FABRIC) in Namibia, South Africa, and Zambia. This independent evaluation was conducted by Boston University’s Center for Global Health and Development (CGHD). Family Health International (FHI) supported the effort with financial resources from the United States Agency for International Development (USAID). FHI implemented a five-year regional PEPFAR-funded program beginning August 2005. The FABRIC project aimed to improve the quality of life for orphans and vulnerable children (OVC) in Namibia, South Africa and Zambia by ensuring delivery of essential services in the PEPFAR OVC domains and by developing the capacity of implementing agencies to coordinate sub-recipients to deliver services effectively. Methods To assess OVC wellbeing in Zambia and South Africa, we conducted a cross-sectional household survey of randomly selected FABRIC beneficiaries and age-range and gender matched (1:2) community sample children in three randomly selected, geographically distinct, FABRIC project sites. A subgroup of beneficiaries and their guardians participated in semi-structured qualitative interviews. In Namibia, we purposefully selected one site in the North and one in the South. We conducted two focus groups and two semi-structured qualitative interviews at each site. For the Organizational Assessment, we reapplied relevant portions of the FHI Technical and Organizational Capacity Assessment Tool (TOCAT) at each of the three Implementing Agencies (IA). We conducted a document review at the FHI level, and conducted semi-structured qualitative interviews at the FHI, IA, and sub-recipient (SR) levels in each of the three countries. We used standard costing methods to estimate the full costs of goods and services used to implement the programs from the SR perspective in South Africa and Zambia only. Key Findings Characteristics of the Sample In South Africa and Zambia, FABRIC households had higher proportions of orphans, higher proportions of guardians with a chronic illness, and lower proportions with a mother as the primary guardian compared to the community sample. The FABRIC and community samples have similar household dependency ratios. Across all samples, the proportion of double orphans increases as age of the index child increases. In Namibia, participants ranged from 8-13 years of age, and all were either HIV infected or an HIV orphan. OVC Service Delivery and Wellbeing Indicators: South Africa and Zambia Of the FABRIC sample, 63% in South Africa and 55% in Zambia reported receiving at least one FABRIC supported service in the preceding 12 months; this was significantly higher than the community samples at 10% and 9%, respectively. Efforts were focused primarily on educational and psychosocial support. In South Africa, 95% of school-aged children in the FABRIC and community samples were registered in school. In Zambia, 97% of the FABRIC sample and 91% of the community sample were registered. When stratified by gender, in Zambia 100% FABRIC females were registered compared to 89% of the community sample. In South Africa, 14% FABRIC females missed more than 2 days of school in the previous 2 weeks compared to 4% in the community sample. Of 5-11 year old girls, 70% of FABRIC sample and 87% of community sample reported average or above average school performance, which may reflect perceived confidence in performance and highlight the ongoing potential vulnerability of girls targeted by the FABRIC project. FABRIC End-of-Project Evaluation Final Report ii Psychosocial support coverage was generally higher in South Africa than in Zambia. Among 0-4 year olds in South Africa, 37% of FABRIC guardians and 29% of community sample guardians reported concerns about the child’s development. Among 0-4 year olds in Zambia 26% and 30% of FABRIC and community guardians respectively reported developmental concerns. Among 5-17 year olds, Zambia generally had more normal psychosocial and emotional wellbeing scores than South Africa; the FABRIC sample in South Africa scored significantly worse than the community sample in the “emotional symptoms” domain. Other PEPFAR OVC domains were less of a focus for the FABRIC project in part due to financial and human resource constraints, and in part because of limitations outlined in donor regulations. We see low rates of acute malnutrition in children under five, but also low rates of dietary diversity in children 5-17. In Zambia, roughly 50% of children in both samples reportedly went without food for 24 hours at least once in the previous week. Approximately 50% of all children in South Africa and 55% in Zambia sleep in a place that gets wet in rain. OVC Service Delivery and Wellbeing Indicators: Namibia Data suggest the Foundation Phase of the Children’s Voices (CV) project enhanced the lives of participants. Children seem to feel more confident and empowered after participating in the CV activities. They expressed confidence in school performance and happiness with extracurricular activities, and perceived that participating in the CV project helped them to make better choices and their own decisions in their lives. While knowledge on health was very general, responses from the children when asked about HIV were much more specific. Knowledge of both modes of transmission and treatment of HIV was fairly high and accurate among all groups. Organizational Assessment: Namibia, South Africa, and Zambia TOCAT scores supported by the interviews reveal a clear self-perception that the organizational capacity at all Implementing Agencies is stronger than before the FABRIC project. Implementing Agencies were able to provide sub-recipients with organizational and program management support but limited technical programmatic guidance. The SRs valued basic assistance including filing, documenting events and report writing. Costing Component: Zambia and South Africa In Zambia, the 2008 average financial cost per OVC across the three SRs was relatively similar, as was the total average costs. The average financial cost was about ZMK 195,000 (close to $40 US annually), while the total average cost was ZMK 279,000 (about $56 US). Because of the increase in numbers of OVC at each site between 2008 and 2009 without similar increases in financial resources, average financial and total costs per OVC in the programs fell over time. In 2009, average financial costs across the three SRs were about ZMK 112,000 (about $22 US annually), and average total costs were about ZMK 170,000 (about $34 US). In South Africa, the three programs varied substantially in terms of total financial expenses: R 2,448,000 million for Sithand’izingane; R 773,000 for Siyathokoza; and R 242,000 for Bophelong. Conclusion Findings suggest the FABRIC recruitment and targeting strategies were successful as the FABRIC sample is generally worse off than the community comparison sample, but also reveal high rates of vulnerability and unmet need in the general communities. At the end of the project, disparities between the two groups seem to be less widespread. We can reasonably assume, though cannot prove, that the initial FABRIC beneficiaries were from households in greatest need. The lack of differences in many of the wellbeing indicators measured by this end-of-project evaluation could be interpreted as the FABRIC project interventions bringing its beneficiaries up to, or maintaining them at, the levels of need and vulnerability of the general population. Though the needs are still profound, the lack of disparity between the two populations can be seen as a proxy measure of success for the FABRIC efforts. Organizational capacity strengthening with respect to administration and M&E, reported by FABRIC End-of-Project Evaluation Final Report iii the sub-recipients (partner organization in Namibia), is undoubtedly beneficial, but further efforts are needed to strengthen the technical capacities of the organizations to implement effective, sustainable OVC services. In Namibia, in general, children learned, retained, and applied information from the PV project. However, our ability to attribute impact or see change over time is limited by the cross-sectional nature of the study, no comparison population, the potential for recall bias when asking children to remember how they were before participating in the project, and the lack of quantitative data to triangulate for validity and reliability. Programmatic Recommendations 1. Invest in developing an appropriate, rigorous, and independent evaluation strategy that is separate but complementary to the routine program management and monitoring activities as early in the project cycle as possible In PEPFAR II far greater emphasis is reportedly placed on evidence based programmatic decisions. This requires using independent, scientifically valid impact evaluations to inform donor officials, policymakers, and program personnel about OVC program effectiveness. Having well-defined specific objectives, linked to a narrow set of outcome measures to accomplish clear program goals is crucial. Standardized process, outcome and costing indicators will ensure we can compare programs and better understand where efforts are successful and where they need improvement. 2. Collect high quality cost data from multiple perspectives To better understand project costs for planning and budgeting purposes, financial and economic cost information, from multiple perspectives (consumer, provider, institutional, donor, societal) should be collected systematically. With agreed upon outcome measures from above, and the more widespread use of standard costing approaches, initial cost-effectiveness calculations can begin to build an evidence base for making programmatic decisions on the more cost- effective intervention mixes. 3. Target populations in need carefully and determine who will serve as comparison populations Invest at the start of the project to create robust, geo-located roster systems with unique identifiers to foster better evaluation data. Randomized evaluation designs, where phased roll-out is expected, and longitudinal evaluations with carefully constructed comparison populations, would improve the quality of evaluation work. Identify and follow comparison populations throughout the project if possible. 4. Institute context, age, and gender-specific programming The FABRIC experience in Namibia, South Africa, and Zambia differed markedly. Even when the approaches were similar (South Africa and Zambia) the underlying economic base and socio-political environment led to major differences in program priorities and outcomes. Disaggregating results by age group and gender elicited important results that may have otherwise been hidden in a grouped analysis. 5. Substantially increase the institutional and individual capacity strengthening efforts PEPFAR II’s strong rhetoric on country-ownership and sustainability is ultimately dependent on the human and institutional capital available for these national efforts. This evaluation highlighted the major organizational development needs including basic core functions at the sub-recipient levels even when the Implementing Agency was relatively strong. 6. Decrease the heavy reliance on unpaid volunteer labor for service provision Though faith-based communities have long had a tradition of providing spiritual, social, and financial support for the sick and the poor, the increasing scale of the global pandemic challenges the faith-based organizations to continue those traditions. Attrition rates among the volunteers at every level of the Implementing Agency operations were high and created program discontinuities, quality of care issues, and generated a need for FABRIC End-of-Project Evaluation Final Report iv regular retraining of a new labor force. It is critical to recognize the additional labor provided by volunteers as a social service with opportunity costs for the worker rather than a assume it is merely a spiritual duty for a member of a faith-based community. Figuring out whether, and if so, how to compensate the community workers assuming responsibility for OVC care is a key social issue. Solutions to this tough issue and the development of realistic expectations of workload must be explored before the need exceeds the capabilities of individual volunteers or faith-based organizations overall. FABRIC End-of-Project Evaluation Final Report 1 INTRODUCTION Overview This report provides findings from the End-of-Project Evaluation (EoPE) of Family Health International’s Community Faith Based Regional Initiative for Orphans and Vulnerable Children (FABRIC) in Namibia, South Africa, and Zambia. This independent evaluation was conducted by Boston University’s Center for Global Health and Development (CGHD). Family Health International (FHI) supported the effort with financial resources from the United States Agency for International Development (USAID). FHI technical staff developed the Terms of Reference for the evaluation and assisted with access and logistics. They also provided comments on draft versions of this report. Full responsibility for the qualitative and quantitative analyses and the interpretation of those results reside with the CGHD team. FHI implemented a five-year PEPFAR-funded program beginning August 2005. The FABRIC project aimed to improve the quality of life for orphans and vulnerable children (OVC) in Namibia, South Africa, and Zambia by ensuring delivery of essential services and developing the capacity of implementing agencies to effectively coordinate its sub-recipients (SR). The FABRIC project objectives were as follows: 1. To increase the number of OVC reached with quality community level services; 2. To strengthen the capacity of partner implementing agencies (IA) (Positive Vibes (Namibia), Southern African Catholic Bishops’ Conference (South Africa), and Expanded Church Response (Zambia)), to target, coordinate, and sustain the programs of local member organizations effectively; 3. To strengthen or create linkages and networks for coordinating OVC coverage and to ensure sharing of lessons learned; 4. To provide timely and reliable information and to meet reporting obligations. The End of Project Evaluation (EoPE) for the FABRIC project was conducted between January and March 2010. The evaluation had the following objectives: 1. To assess if FABRIC strengthened the capacity of its Implementing Agencies to effectively target, coordinate, and sustain programs of community-based organizations (CBOs) that provided direct support to FABRIC project beneficiaries. These CBOs were called sub-recipients (SR) and are separate but programmatically linked to the IAs in South Africa and Zambia. Positive Vibes in Namibia served as the IA and worked with three partners to implement its program. 2. To assess the wellbeing of FABRIC beneficiaries relative to that of similar children living in the same community in each of the following domains: (1) education, (2) psychosocial status, (3) food/nutrition, (4) healthcare, (5) shelter and care, and (6) child protection and legal aid. 3. To describe the experience of services received and the perceived change in personal experience over time in current FABRIC beneficiaries. 4. To compare FABRIC Implementing Agencies’ reports of service delivery to sub-recipients with reported support received by the sub-recipients. 5. To evaluate the costs (using a Full Cost Model approach) of implementing the OVC program annually, from the perspective of the FABRIC sub-recipient organizations. The cost analysis is not an evaluation of FHI’s or their IAs cost to implement the FABRIC project. FABRIC End-of-Project Evaluation Final Report 2 The EoPE consisted of three components: (1) an assessment of wellbeing of FABRIC beneficiary children and a non-beneficiary community sample, (2) an organizational capacity assessment, and (3) an assessment of the full cost of implementing the program at the sub-recipient level. The assessment of orphan wellbeing used mixed methods in South Africa and Zambia, but only qualitative methods in Namibia; the organizational assessment used mixed methods in all three countries; the costing component used standard costing methods and was conducted in South Africa and Zambia Background and Rationale The large and growing number of OVC is a consequence of the global HIV pandemic. Sub-Saharan Africa is most heavily affected by HIV, home to 67% of all people living with HIV and 72% of deaths due to AIDS in 2008.1 It is estimated that in sub-Saharan Africa alone, 14 million children have lost one or both parents to HIV.2 Namibia, South Africa, and Zambia are among the countries most affected by HIV, with adult prevalence estimates of 15.3%, 18.1%, and 15.2% respectively.2 Namibia has an estimated 66,000 orphans due to HIV/AIDS, and 110,000 due to all causes.2 In South Africa, 1.4 million children are reportedly orphaned due to HIV and 2.5 million due to all causes, and in Zambia, an estimated 600,000 children are HIV orphans and 1.1 million have been orphaned due to all causes.2 Support for families caring for OVC is a major component of the President’s Emergency Plan for AIDS Relief (PEPFAR). In 2005, FHI received a five-year grant from PEPFAR for a regional comprehensive care and support program for OVC in Namibia, South Africa, and Zambia. The program, entitled the Community Faith-Based Regional Initiative for Orphans and Vulnerable Children (FABRIC), aimed to improve the quality of life for OVC in these program countries by developing the capacity of implementing partners to effectively allocate resources and ensure essential services reached OVC. The FABRIC project’s key strategy was to strengthen the organizational capacity of large umbrella faith-based organizations to manage small grants programs, which funded OVC care and support activities provided by community level faith-based organizations known as sub-recipients (SR). In South Africa, the umbrella agency was the Southern African Catholic Bishops’ Conference (SACBC); in Zambia, the umbrella agency was Expanded Church Response (ECR). In Namibia, the Church Alliance for Orphans (CAFO) was the primary umbrella agency for the first three years of PEPFAR. In 2008, CAFO graduated into the New Partners Initiative so could no longer receive FABRIC resources. FABRIC selected Positive Vibes (PV) as its Implementing Agency for the final two years of the program. PV worked with three partners to provide services to OVC through child and youth-centered communication and media initiatives that sought to build self-esteem and strengthen coping strategies. FABRIC is a large and complex project. In its 2009 Annual Report, FHI reported that FABRIC had serviced 19,493 OVC across all countries in fiscal year 2009 and reported a total of 62,330 ever served. As part of the rapid “emergency” scale-up after the PEPFAR I legislation was passed, FHI, using Track 1 monies, sought to initiate three country level projects quickly. There were (understandable and appropriate) pressures to initiate the provision of services to OVC as quickly as possible given the needs that were known to exist. These rapid scale￾up demands, combined with the programmatic and budgetary realities created by a system that used annual funding mechanisms that were explicitly tied to “getting the numbers” of beneficiaries, led to a series of program decisions that limit what is possible to accomplish in an evaluation five years later. Consistent, and at times heroic, efforts especially given the lack of history and limited capacity of the SR organizations to systematically collect and report service provision data, were made to establish or improve existing monitoring systems so FABRIC could report on required PEPFAR indicators. Much less consideration or effort was available at the start of the project to prepare for an end-of-project evaluation that would be interested in the impacts of the efforts on the beneficiaries. With no impact evaluation framework in place, FABRIC End-of-Project Evaluation Final Report 3 several decisions (or non-decisions) were made that limit our ability to measure the impacts of the FABRIC investments on their clients. Beneficiary roster systems were constructed in an inconsistent and uneven fashion without assigning unique identifiers to the beneficiary clients. No data on geo-location information were collected that would allow evaluators to identify and return to beneficiary populations over time. No baseline information on beneficiaries or their households was routinely collected. No comparison populations were identified or monitored. Despite these limitations, FHI program and headquarter leadership was keen to learn, and communicate broadly, as much as possible from their effort. Given the large number of beneficiaries served, their own, mostly anecdotal, knowledge on what they have accomplished, and a deep desire that the PEPFAR II OVC programming further improve upon the efforts made during the “emergency phase” of the United States Government (USG) response, FHI welcomed and greatly facilitated, this independent, external evaluation of its efforts over the past five years. Boston University’s CGHD greatly appreciates the intellectual openness and spirit of cooperation provided by our FHI and IA colleagues. We attempted to conduct the best evaluation possible, within the constraints imposed by programmatic decisions made before any of the current FHI staff were engaged in the project. Their guidance was exceptional, their flexibility was remarkable, and their openness to “allow the data to speak” reflects their deep commitment to having the FHI FABRIC experience provide insights that may allow future OVC programming efforts to improve. CGHD conducted the analysis and interpretation, and is thus responsible for any associated mistakes or misinterpretations. However, the hope that readers will use the information in this EoPE report to improve OVC programming in the years to come is fully shared by both organizations. FABRIC End-of-Project Evaluation Final Report 4 FABRIC End-of-Project Evaluation Final Report 5 A S S E S S M E N T O F C H I L D W E L L B E I N G: S O U T H A F R I C A A N D Z A M B I A FABRIC End-of-Project Evaluation Final Report 6 FABRIC End-of-Project Evaluation Final Report 7 ASSESSMENT OF CHILD WELLBEING: SOUTH AFRICA, ZAMBIA Overview FABRIC provided support to an Implementing Agency (IA) in each country, who in turn supported multiple sub￾recipients that provided services at the community level. The organizational structure is shown in Figure 1 and methods are described below. Figure 1: Organizational Framework of the FABRIC project1 Southern African Catholic Bishops’ Conference (SACBC) was the implementing agency (IA) chosen in South Africa. SACBC has assisted the Catholic Church to coordinate its response to HIV/AIDS in the region, facilitate training, scale up programs, establish programs in resource poor setting, provide M&E support and address the needs of OVC. With FABRIC, SACBC worked with 10 community-based sub-recipients. Expanded Church Response (ECR) was the IA in Zambia. ECR was founded in Zambia in 2003 when 220 church leaders met to explore actions the church should take to confront the HIV/AIDS epidemic.3 Since 2003, ECR has worked with several local faith-based organizations (FBOs) to provide advocacy and support services for people living with HIV/AIDS (PLHA) and acted as an attitude and behavioral change agent on stigma and discrimination and the acceptance of PLHA. ECR also advocates and supports the church in poverty alleviation projects. ECR started working with FABRIC in 2005 and aimed to improve quality of life for OVC. ERC worked with 15 community-based sub-recipients. The Church Alliance for Orphans (CAFO) was the IA for the first three years in Namibia. In 2008, CAFO graduated into the New Partners Initiative. FABRIC selected Positive Vibes (PV) as the IA beginning in year four of the five– year project. Positive Vibes is a new Namibian NGO responding to HIV/AIDS with innovative and creative communication approaches. PV works towards “positive social change through innovative, people-centered approaches that articulate the voices and builds the self-esteem of PLHA”, especially children. FABRIC, through PV, worked in six regions of Namibia and collaborated with three partner organizations for implementation, rather than sub-recipients in the other two countries.4 1 Data in this figure are current from fiscal year 2009 2 Survey did not elucidate which group or organization provided the healthcare referral. 3 This section is a summary of the preliminary results from a full costing report prepared by Petan Hamazakaza for Zambia sites and Sarah Laurence in South Africa, with assistance from Jill Costello and Bruce Larson. The full costing report will be • Regional team: Zambia • Country-specific teams: South Africa and Namibia Family Health International (FABRIC) • South African Catholic Bishops Conference (SACBC) • Expanded Church Response Zambia (ECR) • Positive Vibes Namibia (PV) Implementing Agency • South Africa • 10 sub-recipients • Zambia • 15 sub-recipients • Namibia • 3 partner organizations Sub-recipients/Partner Organizations • South Africa: 10,055 • Zambia: 8,937 • Namibia: 501 OVC FABRIC End-of-Project Evaluation Final Report 8 This component of the EoPE aimed to assess the wellbeing of a FABRIC project sample relative to a group of children randomly selected from the same communities in each of the core PEPFAR program areas: (1) education, (2) psychosocial status, (3) food/nutrition, (4) healthcare, (5) shelter and care, and (6) child protection and legal aid. Economic strengthening, commonly known as the “+1” in the USAID 6+1 OVC Service package, was not assessed, as FABRIC did not address this domain. The FABRIC sample was matched on gender and age-range with the community sample at a 1:2 ratio. IRB approval was received from FHI, Boston University, and the Ministry of Health and Social Services in Namibia. Approval was also granted from the Universities of Pretoria and Zambia, both of which are responsible for evaluating applied research in their respective country. Methods and Study Site We conducted a cross-sectional household survey of randomly selected FABRIC beneficiaries and age- and gender-matched community sample children in three randomly selected, geographically distinct, FABRIC project sites in both Zambia and South Africa. An equal number of children in each of three age groups were targeted: 0-4 years, 5-11 years and 12-17 years. A subgroup of the FABRIC sample aged 12-17 years and their guardians participated in semi-structured qualitative interviews focused on their experience with the FABRIC project. Limitations Because there are no baseline wellbeing data available, we conducted a cross-sectional survey to describe the reported wellbeing of each sample at one point in time. We are unable to analyze any changes over time. While we may see differences between the FABRIC and community samples, it is not possible to fully attribute differences to the FABRIC project. Additionally, on the quantitative survey, questions were time-bound to the last year to minimize recall bias. We assumed that FABRIC sample children, once they started receiving services, would stay in the program until they aged out. We cannot be sure that children currently enrolled at the time of the EoPE survey are representative of the full population of FABRIC beneficiaries due to inconsistent annual funding levels, high volunteer turnover and attrition rates, and modifications of the program priorities over time. The survey did not capture whether a child had “ever” received a service, therefore service delivery coverage may under-report services from FABRIC across the 5-year project. Three FABRIC project sites in South Africa and in Zambia were randomly selected to participate in the evaluation. Sites in South Africa included Sithandizingane in Gauteng (532 beneficiaries), Siyathokoza in Free State (654 beneficiaries), and Bophelong in Northwest Province (765 beneficiaries). Sites in Zambia included Bethel Baptist – Samaritan Project in Kafue (422 beneficiaries), Mpatamatu Home-based Care Project in Luanshya (1,050 beneficiaries), and Evangel Oasis of Love Orphans Project in Chingola (387 beneficiaries). In order to reach the desired sample size of children 0-4 years of age in Zambia, an Mpatamatu program extension site (1,000 beneficiaries) was added. A sample of FABRIC project beneficiaries was randomly selected from current program rosters to participate in the evaluation. An equal number of FABRIC beneficiaries were chosen from each of three age categories and the sample of community children was selected from the same communities, matched by gender and age group. A subset of the FABRIC sample (n=24) in the 12-17 years age group participated in an additional in-depth, semi￾structured interview about their experience in the FABRIC project. Informed consent from the guardian and assent from the adolescent were obtained for the quantitative and qualitative components. Sample Size and Procedures In order to contextualize our sampling frame, an understanding of the FABRIC project’s targeting and recruitment strategy is important. Though the targeting strategy is not well documented in either country, current FABRIC project staff was able to ascertain that the project preferentially recruited children that had lost either one or both parents (orphans). Vulnerable children were defined as having suffered some form of abuse, FABRIC End-of-Project Evaluation Final Report 9 dropped out from school, or were living in households headed by grandmothers, children, or chronically ill adults. Among these, households were then prioritized by economic status, with those having no income given support before those with some form of household income. Assuming this targeting strategy was systematic and successful, we can reasonably assume that the FABRIC beneficiary households contained highly vulnerable children. We also can reasonably assume that the community targeting attempted to identify the more marginalized households containing children in greatest need. However, because we are lacking baseline data, it is impossible to conclude without reservation that the FABRIC sample was different from the matched community comparison population. We can, however, examine the current disparities experienced by the FABRIC sample relative to the community sample and suggest a relationship between the FABRIC interventions and the level of disparity. Using this approach and logic, “no difference” between the FABRIC sample and the community sample should be considered positive impact of the program as the base assumption is that the initial beneficiary children were selected because they were considered worse off at time zero. Based on the limited quantitative OVC program evaluation literature and using local expert opinion, we powered the evaluation to detect a 20% difference in psychosocial wellbeing status between the FABRIC sample and the community sample (alpha 95%, power 80%), using a 1:2 FABRIC to community ratio in each age category. An additional 15% was added to account for refusals or incomplete interviews. Table 1 details the actual and projected sample stratified by age group and country. Instrument Development In December 2009, the Boston University and FHI teams met in Lusaka, Zambia to develop the protocol and indicators. Indicators reflected service receipt and wellbeing over the previous 12 months in each of the core PEPFAR program areas: (1) education, (2) psychosocial status, (3) food/nutrition, (4) healthcare, (5) shelter and care, and (6) child protection and legal aid. The indicators we agreed upon to measure service uptake and wellbeing can be seen in the appendix. Surveys were age-specific to make them more appropriate, but measured the same domains. The surveys were developed by the Boston University team, reviewed by FABRIC country and regional staff, and modified as necessary. The 0-4 and 5-11 years of age instruments were answered by the index child’s guardian, and the 12- 17 years of age instrument had sections completed by both the guardian and by the adolescent. The same survey was administered to the FABRIC and community samples. Differences between the instruments reflected age-specific modifications of psychosocial, education, and dietary indicators. The qualitative interview guide was developed to elicit a description of the services received through the program, and the adolescent’s perceived change from involvement in FABRIC. Surveys were translated into local languages (Zulu, Sotho, and Tswana in South Africa; Bemba and Nyanja in Zambia), back-translated, and piloted prior to implementation. Table 1. Actual and (projected) sample size for FABRIC End of Project Evaluation SOUTH AFRICA ZAMBIA Total across FABRIC Sites FABRIC Community FABRIC Community FABRIC Community Age Group 0-4 years 41 (44) 84 (88) 46 (44) 91 (88) 87 (88) 175 (176) 5-11 years 43 (44) 86 (88) 44 (44) 88 (88) 87 (88) 174 (176) 12-17 years 43 (44) 83 (88) 46 (44) 93 (88) 89 (88) 176 (176) Sub-totals 127 (132) 253 (264) 136 (132) 272 (264) 263 (264) 525 (528) Total 380 (396) 408 (396) 788 (792) FABRIC End-of-Project Evaluation Final Report 10 Data Enumerator Training Local enumerators were selected by the FHI national offices in Zambia, in consultation with Boston University’s Center for International Health and Development – Zambia (CIHDZ). FHI contracted data collection services in South Africa to Development Research Africa (DRA). DRA identified experienced data collectors and coordinated all field logistics. All data collectors in South Africa and Zambia had prior experience administering community￾based surveys, were fluent in English and the local languages, and were non-residents of the community. Enumerators attended a five-day training in either Lusaka or Johannesburg, which included sessions on research ethics, data confidentiality, survey implementation, and child protection. Trainings were organized by FHI in Zambia and DRA in South Africa, and were facilitated by a BU faculty member experienced with quantitative and qualitative research methods and familiar with the local context. Data Collection Data collection took place during the month of March 2010, following receipt of all IRB approvals. A member of the BU evaluation team supervised enumerators. An FHI staff member oversaw logistics in Zambia; DRA oversaw logistics in South Africa. The FABRIC beneficiary sample was randomly selected from current program rosters. Households were visited to request participation. If a FABRIC beneficiary declined participation, the next child on the list of randomly selected beneficiaries was approached. At the time of this first household visit, the dwelling was geo-located using a Global Positioning System (GPS) device so it could be traced again for survey implementation. The community sample was randomly selected using the “drop the pen” technique, in which a pen is spun and dropped on the ground. Following the direction of the pen tip, every other house was contacted to see if a child of the appropriate gender and age range resided in the household. This random selection technique was repeated as necessary to locate community comparison households willing to participate in the evaluation. Teams of two enumerators conducted 90-minute qualitative interviews in the respondent’s language of choice. Prior to the interview with each adolescent, we briefly interviewed the primary guardian. Interviews were digitally recorded, transcribed and translated into English by the enumerators. Documents were reviewed for completion by the supervisor and transported back to Lusaka for entry, analysis, and storage. Consent from the guardian and assent from the child were obtained prior to any survey implementation. Data Management and Analysis Data Entry and Cleaning Teleforms (Autonomy Cardiff Software, Vista, CA) was used for survey design and data entry. Teleforms is a data collection and management software that allows questionnaire creation by digitally mapping and encoding all data entry fields. Completed surveys were scanned and the Teleforms optical character recognition software captured data off the paper forms and exported it to a Microsoft Access database. Two research assistants manually verified the electronic data entered. Value, range, and logic checks were conducted for each variable. Data Analysis We used SAS version 9.1.3 (SAS Institute, Cary, NC) for data analysis. We calculated descriptive statistics for all study variables (means, medians, ranges, standard deviations, percent data missing, and frequencies for categorical variables). We assessed each outcome in a bivariate model, comparing the FABRIC and community samples as the exposure of interest. Each analysis was performed for the overall project as well as stratified by age group, country and gender. Chi squared p-values were used when the sample size was large enough; in some cases, where stratification led to small numbers, Fischer’s exact p-values were used to compare the FABRIC End-of-Project Evaluation Final Report 11 FABRIC sample to the community sample. Qualitative interview transcripts were analyzed in NVivo version 8 (QSR International, Cambridge, MA. Themes were identified a priori, according to the focus group and interview guides, and coded using domain analysis. Results Characteristics of the Sample Demographic characteristics of the FABRIC and community samples in South Africa and Zambia are shown in Table 2. Table 2. Demographic characteristics of the FABRIC and community samples participating in the household survey in South Africa and Zambia SOUTH AFRICA ZAMBIA FABRIC n = 127 Community n = 253 FABRIC n = 136 Community n = 272 Mean age (SD) 9.0 (4.9) 8.5 (5.3) 9.1 (4.9) 8.6 (5.0) Child gender Male (%) Female (%) 56 (44.1) 71 (55.9) 111 (43.9) 142 (56.1) 65 (47.8) 71 (52.5) 129 (47.4) 143 (52.6) Orphan status Non-orphan (%) Maternal orphan (%) Paternal orphan (%) Double orphan (%) 59 (46.5) 12 (9.5) 21 (16.5) 35 (27.6) 182 (71.2) 9 (3.6) 42 (16.6) 20 (7.9) 31 (22.8) 9 (6.6) 67 (49.3) 29 (21.3) 210 (77.2) 8 (2.9) 36 (13.2) 18 (6.6) Primary Guardian Mother (%) Father (%) Grandmother (%) With other relative (%) With non-relative (%) 58 (45.7) 2 (1.6) 38 (29.9) 28 (22.0) 1 (0.8) 169 (66.8) 9 (3.6) 42 (16.6) 33 (13.0) 0 (n/a) 76 (55.9) 5 (3.7) 28 (20.6) 26 (19.1) 1 (0.7) 172 (63.2) 22 (8.1) 36 (13.2) 42 (15.4) 0 (n/a) Gender of guardian Male (%) Female (%) 11(8.7) 116 (91.3) 17 (6.7) 236 (93.3) 16 (11.8) 120 (88.2) 41 (15.1) 231 (84.9) Mean age of guardian (yrs) 43.1(sd 13.9) 38.3 (sd 12.0) 39.8 (sd 10.2) 38.5 (sd 11.1) Guardians >60 years of age (%) 21 (16.5) 16 (6.4) 5 (3.7) 11 (4.1) Guardians <18 years of age (%) 0 (n/a) 1 (0.4) 0 (n/a) 0 (n/a) Employment status of the guardian Formal sector (%) Informal sector (%) Domestic/Home (%) Unemployed (%) 14 (11.0) 14 (11.0) 5 (3.9) 85 (66.9) 17 (6.7) 20 (7.9) 13 (5.1) 190 (75.1) 4 (2.9) 47 (34.6) 57 (41.9) 17 (12.5) 12 (4.4) 65 (24.0) 136 (50.2) 29 (10.7) Guardians with a chronic illness (%) 32 (25.2) 47 (18.6) 26 (19.3) 34 (12.5) Mean number of adults living in the household 2.5 (1.3) 2.5 (1.3) 3.2 (sd 1.6) 3.3 (sd 1.5) Mean number of children living in the household 2.8 (sd 1.5) 2.6 (sd 1.3) 3.9 (sd 1.7) 3.9 (sd 1.6) FABRIC End-of-Project Evaluation Final Report 12 The demographic data suggest both the targeting strategy for the program and the matching strategy (age range and gender) for the evaluation were successful. The proportion of males and females in each sample are similar in each country, as are the mean ages of the child, suggesting the community-based matching worked reasonably well. We observed that FABRIC households had higher proportions of orphans than the community sample, suggesting the initial beneficiary targeting strategy was successful. In South Africa, 53.5% of the FABRIC sample and 28.1% of the community sample are known or presumed orphans (p < .0001), where either the mother or father or both is known or presumed deceased. Among orphans, we see a similar proportion of paternal orphans in each sample in South Africa, but the FABRIC sample has nearly double the proportion of maternal orphans and four times the proportion of double orphans. In Zambia, 77.2% of the FABRIC sample and 22.8% of the community sample are known or presumed orphans (p <. 0001), where either the mother or father or both is known or presumed deceased. Among orphans, the proportions of paternal and double orphans are over three times that of the community sample, and maternal orphans are more than double. We also observe a higher proportion of guardians with a chronic illness in the FABRIC samples, and a lower proportion of mothers as the primary caregiver in the FABRIC samples. Household size and composition, as indicated by the mean numbers of adults and children living in the household, is similar across the FABRIC and community samples. Therefore, the dependency ratios are similar. The means are slightly higher in Zambia than South Africa. Though not statistically significant, the proportion of guardians with a chronic illness is higher in the FABRIC sample in both countries than in the community sample. Table 3 shows the breakdown of the type of orphan stratified by country and by age range. In general, the proportion of double orphans increases as children get older across all groups. Table 3. Type of orphan within households in the FABRIC and community samples in South Africa and Zambia by age range SOUTH AFRICA ZAMBIA FABRIC Community FABRIC Community 0-4 years of age Maternal orphan (%) Paternal orphan (%) Double orphan (%) n = 13 (31.7%) 4 (30.8) 7 (53.9) 2 (15.4) n = 8 (9.5%) 0 (n/a) 7 (87.5) 1 (12.5) n = 30 (65.2%) 2 (6.7) 21 (70.0) 7 (23.3 n = 14 (15.4%) 2 (14.3) 10 (71.4) 2 (14.3) 5-11 years of age Maternal orphan (%) Paternal orphan (%) Double orphan (%) n = 21 (48.8%) 6 (28.6) 4 (19.1) 11 (52.4) n = 27 (31.4%) 5 (18.5) 16 (59.3) 6 (22.2) n = 32 (72.7%) 3 (9.4) 18 (56.3) 11 (34.4) n = 15 (17.1%) 2 (13.3) 10 (66.7) 3 (20.0) 12-17 years of age Maternal orphan (%) Paternal orphan (%) Double orphan (%) n = 34 (79.1%) 2 (5.9) 10 (29.4) 22 (64.7) n = 36 (43.3%) 4 (11.1) 19 (52.8) 13 (36.1) n = 43 (93.5%) 4 (5.3) 28 (65.1) 11 (25.6) n = 33 (35.5%) 4 (12.1) 16 (48.5) 13 (39.4) While the data suggest the targeting strategy for enrollment was successful, the high proportion of orphans in the community sample suggests there remained a large unmet need in the community. FABRIC Service Delivery and Child Wellbeing Children could have received supportive services in any of the six of the PEPFAR domains, and it was possible to have received more than one service per domain. Table 4 reflects the number of services reportedly received in FABRIC End-of-Project Evaluation Final Report 13 the preceding 12 months among the FABRIC and community samples. In Zambia, the range of services received was zero to four, with 55.2% (n=75) of the FABRIC sample reporting receiving services over the previous 12 months. In Zambia, the mean number of services received by the FABRIC sample was 0.9 (sd 1.1) and the community sample 0.1 (sd 0.4) (p < .0001). Among the FABRIC sample in Zambia, 24.3% (n=33) reported receiving two or more services. In South Africa, reported services received ranged zero to seven, and 63.0% (n=80) of the FABRIC sample reported that they received some support in the preceding 12 months. The mean number of services received in South Africa reported by the FABRIC sample was 1.4 (sd 1.5); in the community sample it was 0.2 (sd 0.6) (p < .0001). Among the FABRIC sample in South Africa, 40.9% (n=52) reported receiving two or more services. Community samples in both countries reported receiving services, representing either a spillover of FABRIC services to community members or confusion about the source of the services. Table 4. FABRIC supports received among beneficiary and community sample children SOUTH AFRICA ZAMBIA FABRIC n = 127 Community n = 253 FABRIC n = 136 Community n = 272 At least one supportive service received (%) 80 (63.0)* 26 (10.3)* 75 (55.2)* 23 (8.5)* Two or more supportive service received (%) 52 (40.9)* 10 (4.0)* 33 (24.3)* 5 (1.8)* Mean number of supportive services (sd) 1.4 (sd 1.5)* 0.2 (sd 0.6)* 0.9 (sd 1.1)* 0.1 (sd 0.4)* * Significantly different at p < 0.05 The following sections detail services received and outcomes indicators. Results of the quantitative and qualitative data collected are presented together, organized by PEPFAR program domain: (1) education, (2) psychosocial status, (3) food/nutrition, (4) healthcare, (5) shelter and care, and (6) child protection and legal aid. Each domain discusses five areas key to understanding and assessing service delivery, uptake and child wellbeing: 1) General overview of PEPFAR Guidance and Policy 2) Actual FABRIC practices in South Africa and Zambia 3) Coverage results 4) Orphan wellbeing results 5) Discussion 1. Education 1.1.PEPFAR Guidance and Policy PEPFAR programming guidance encouraged diverse programming approaches aimed at removing barriers so that children could attend and perform well in school. The PEPFAR guidance recommended OVC programs give special attention to the vulnerability of girls, addressing the disproportionate level of risk girls face when leaving school at an early age. Interventions to make schools safe for children, especially girls, were encouraged. Vocational training support was also encouraged with recognition that the lack of opportunity to learn a trade or enter vocational networks can compromise adolescents’ long-term economic prospects. FABRIC End-of-Project Evaluation Final Report 14 1.2.FABRIC Actual Practice FABRIC promoted school registration and attendance for OVC, with specific activities including advocacy for reduced school fees and provision of books, uniforms, shoes, and other school supplies. In South Africa, FABRIC provided school materials including uniforms, shoes, and supplies. Because of the Child Welfare Grant available in South Africa, FABRIC did not need to purchase as many books or pay as many school fees as in Zambia. In Zambia, educational support from FABRIC focused on paying school fees, procuring school supplies, homework support, and mentorship by caregivers. More recently, mini libraries were established within sub-recipient organizations. Both countries provided mentorship and career guidance for OVC. 1.3.Education Coverage Results The survey measured the proportion of school-aged children in the FABRIC and community samples who received assistance from the FABRIC project to attend school in the preceding 12 months. In South Africa, 31.4% of the FABRIC sample received assistance compared to 1.2% in the community sample (p < .0001). In Zambia, 53.9% school-aged FABRIC children reported receiving assistance compared to 4.9% in the community sample (p < .0001). These results were substantiated by the qualitative work. Qualitative interview results show that all respondents in Zambia received books or pens, and seven of the twelve adolescents reported receiving school fees or school uniforms. Other frequently reported supports received in Zambia included shoes, stationary, school-books, and pencils. Half of the qualitative respondents in South Africa (n=6) reported receiving homework support. Shoes and uniforms were other commonly reported supports. 1.4.Education Wellbeing Measures Education wellbeing data are presented in Table 5. School enrollment rates are high in both countries. We observed no differences between the FABRIC and community samples in the proportion of school-aged children registered in school. Nor did we observe differences in the number of children reporting average or above average school performance in either Zambia or South Africa. In South Africa, a significantly larger proportion of children in the FABRIC sample reported missing more than two days of school in the preceding two weeks than the community sample. In Zambia, nearly 20% in FABRIC and 30% in the community, reported missing more than two days of school in the previous two weeks. Table 5. Education Wellbeing Indicators of registered school-aged FABRIC and community sample children in South Africa and Zambia SOUTH AFRICA ZAMBIA FABRIC n = 82 (%) Community n = 161 (%) FABRIC n = 86 (%) Community n = 149 (%) School-aged children registered for school missing more than 2 days of school in the last 2 weeks 9 (11.3)* 6 (3.7)* 17 (19.8) 43 (28.9) Children registered in school reporting above average school performance 63 (76.8) 128 (79.5) 67 (83.8) 123 (85.4) * Significantly different from the country-specific community comparison group at p < 0.05 “It has helped her with books but we are still waiting for the school fees… My question is, when are they going to give my child the school fees.” -Guardian of 14-year-old FABRIC beneficiary, Zambia FABRIC End-of-Project Evaluation Final Report 15 We noticed differences when we stratified these outcomes were stratified by gender. First, among females in Zambia, 100% (n = 38) of school-age FABRIC sample children were registered for school compared with 88.9% (n = 64) of community sample children (p = .049). This difference was not seen among males. Second, in South Africa, a greater proportion of girls in the FABRIC sample reported missing more than two days of school in the preceding two weeks relative to the community sample (14.0% and 3.5%, respectively, p = 0.059). Similar differences were not observed in boys or in the Zambian samples. Third, a smaller proportion of guardians of girls 5-11 years of age in the FABRIC sample reported average or above average school performance, compared to the reported school performance in the community sample. No similar differences were observed in boys. Table 6 illustrates parent-reported (5-11 years) and self-reported (12-17 years) school performance, presented as the percentage reporting average or above average school performance. Table 6. Proportion of children with average or above average reported school performance, stratified by age and gender SOUTH AFRICA ZAMBIA TOTAL FABRIC n (%) Community n (%) FABRIC n (%) Community n (%) FABRIC n (%) Community n (%) Female 36 (80.0) 72 (83.7) 28 (73.7) 56 (87.5) 64 (77.1) 128 (85.5) 5-11 years 12-17 years 17 (85.0) 19 (76.0) 27 (84.4) 45 (83.3) 8 (50.0) 20 (90.9) 20 (90.9) 36 (85.7) 25 (69.4)* 39 (82.9) 47 (87.0)* 81 (84.4) Male 5-11 years 12-17 years 27 (73.0) 16 (76.2) 11 (68.8) 56 (74.7) 37 (77.1) 19 (70.4) 43 (89.6) 22 (88.0) 21 (91.3) 71 (83.5) 33 (78.6) 38 (88.4) 70 (82.4) 38 (82.6) 32 (82.1) 127 (79.4) 70 (77.8) 57 (81.4) Total 63 (76.8) 128 (79.5) 71 (82.6) 127 (85.2) 134 (79.8) 255 (82.3) *Significantly different at p = 0.05 level The survey assessed reasons why some school-age children were not enrolled in school. In South Africa, financial poverty did not seem to be a major constraint on school participation. Of the eight school-aged children in the community sample not registered, only one reported no money for school fees. Four guardians reported this was because the children were too young, one reported poor school performance in the preceding year, and reasons were unavailable for the remaining two. Of the unregistered children in the FABRIC sample (n=4), one guardian reported the child was too young. Reasons were unavailable for the remaining three children. In the community sample in Zambia, 14 children were not registered in school. In contrast to South Africa, money for school fees appears to be a constraint in the community sample. Of the 13 community children eligible for school but not enrolled, nine reported a lack of money for school fees, two reported a lack of school uniforms or supplies, one was too young, and one needed to care for another child. Of the three school-aged FABRIC children not registered in school in Zambia, access to funds for school fees was not reported as a problem. One child was too young, another guardian reported the child did not want to attend, and the reason for the third child was unavailable. The survey also assessed reasons for school absenteeism among children registered for school. In South Africa, of the six community sample children missing more than two days of school in the preceding two weeks, five guardians reported illness and one reported lacking school shoes. Among the nine FABRIC sample children who FABRIC End-of-Project Evaluation Final Report 16 missed more than two days of school in the preceding two weeks in South Africa, seven reported illness, one lacked a uniform, and for one the reason was unavailable. Among registered students in Zambia who missed more than two days of school in the past two weeks, the most frequently reported reason was illness (13 community children and 8 FABRIC children). Other reasons included unpaid school fees, (12 community children and 4 FABRIC children), income generating activities (2 community; 1 FABRIC), no shoes or uniform (5 community; 2 FABRIC), household activities (3 community and 1 FABRIC), and child-care (1 community child). Additional reasons provided include too much rain, no soap for washing, a late return from holiday, and no food to eat. 1.5.Education Discussion In Zambia, more than half of the school-aged FABRIC sample surveyed reported receiving support to attend school, consistent with FABRIC’s programmatic focus on education. In South Africa, approximately one-third of FABRIC beneficiaries surveyed reported receiving support to attend school. Although this number is less than that seen in Zambia, it provides evidence of the FABRIC project focus and the probable impact. In both South Africa and Zambia, we observed high levels of school enrollment. Both countries actively support efforts to meet universal primary education goals and are making major efforts to expand educational access to populations that have had restricted access historically. Interestingly, 100% of girls in the FABRIC sample in Zambia reported enrollment in school, a number significantly higher than the 88% in the community sample. Although program impact is difficult to ascribe in the absence of baseline data, this school registration rate among girls, a population group specifically targeted by FABRIC, is noteworthy. Parent- and self-reported school performance suggests the majority of children are perceived to be performing at average or above average levels. When stratified by gender, a smaller proportion of girls 5-11 years of age enrolled in the FABRIC sample reported average or above average performance compared to the community sample. The differences noted in this metric, which may reflect perceived confidence in performance, highlight the ongoing potential vulnerability of girls targeted by the FABRIC project. Lastly, rates of school absenteeism seen in this study are similar to those reported in a learner absenteeism study recently published in South Africa (Learner Absenteeism Report, CASE, 2007). In our study sample, illness was the primary driver of school absenteeism in both the FABRIC and community samples in both countries, again highlighting the widespread vulnerability of these communities. The health status of both the FABRIC and community samples are further discussed below. 2. Psychosocial Support 2.1.PEPFAR Guidance and Policy PEPFAR programming guidance advocated for OVC programs to provide children with age- and situation-specific psychosocial support, recognizing that children often respond differently than adults to trauma and loss. Examples of potential PEPFAR-supported interventions include: (1) referrals to counseling where available and appropriate, particularly for HIV-positive youth; (2) capacity-building activities for caregivers to listen to and talk “I get better marks from school now.” -12-year-old FABRIC beneficiary, South Africa “It has changed him. His life is moving forward since they started helping him. His life is better. They sponsor him in everything at school.” -Guardian of 15-year-old FABRIC beneficiary, Zambia FABRIC End-of-Project Evaluation Final Report 17 with children; and, (3) strengthening local capacities to provide psychosocial support for distressed children, with additional activities supported at the level of the child, family, and care system. 2.2.FABRIC Actual Practice Within the FABRIC project, psychosocial supports provided varied by age range and country. For 0-4 year olds, FABRIC supported home visits by project caregivers, which aimed to identify problems experienced by beneficiaries and to provide encouragement. Some FABRIC caregivers were trained in counseling. In South Africa, early childhood development centers were established for 0-4 year olds; here children could play with toys and receive feeding support. In Zambia, 0-4 year old psychosocial support was limited in the first years of the project, but in 2008, the “Say and Play” tool for all OVC partners was developed and implemented to help guide psychosocial support for 0-4 years of age. In 2009, FABRIC was one of the first projects to scale-up the use of this tool. For 5-17 year olds, South Africa supported daily after-school programs, and Zambia offered “Kids Clubs” once per week. Both countries implemented short-term school holiday camps that focused on psychosocial activities to address emotional distress for OVC including sporting activities, education on children’s rights and abuse, and career guidance. 2.3.Psychosocial Coverage Results The survey captured coverage of psychosocial services differently in each age group. Guardians of children under five years of age were asked about receipt of psychosocial services from the FABRIC project in the preceding three months, rather than 12 months, because young children are highly vulnerable and more recent psychosocial services would be more relevant to wellbeing. Guardians (5-11) and adolescents (12-17) were asked about participation in extracurricular or afterschool activities, and home visits supported by the FABRIC project in the preceding 12 months. Overall, more psychosocial support was reportedly received from the FABRIC project among the FABRIC sample in South Africa than in Zambia. Among children under five, 24.4% in South Africa and 6.5% in Zambia reportedly received some form of psychosocial support in the preceding three months. Among children 5 – 11, 11.6% in South Africa and 9.1% in Zambia reportedly received support in the preceding twelve months. Among children 12-17, 11.6% in South Africa and 6.5% in Zambia reported receipt of psychosocial support from FABRIC in the preceding twelve months. A small proportion of the community sample overall – 0.6% in South Africa, 1.6% in Zambia – also reportedly received some psychosocial service from the FABRIC project in the preceding 12 months. In the qualitative interviews, the most frequently reported psychosocial support activity in both countries was “playing with friends at the FABRIC center.” Adolescents in South Africa also perceived dancing/singing, praying, feeling loved and important, encouragement from FABRIC staff, support groups, and exercise as psychosocial benefits of FABRIC. Adolescents in Zambia perceived counseling from caregivers, sports and exercise, trusting the caregivers, feeling loved and important and praying as psychosocial benefits of FABRIC. “They give us love like we are their own kids and they always give us support and advice. If we feel sad, they treat us good. They always make you happy.” -12-year-old FABRIC beneficiary, South Africa FABRIC End-of-Project Evaluation Final Report 18 2.4.Psychosocial Wellbeing Measures Psychosocial wellbeing indicators used widely accepted, age appropriate, measurement tools from which a summary score is calculated. The wellbeing of children under five was assessed using, with permission, The Parents Evaluation of Developmental Status (PEDS), a developmental screening tool to capture parental concerns about the learning and behavior of children. For children 5-17 years of age, we measured psychosocial and emotional wellbeing using the Strengths and Difficulties Questionnaire (SDQ). The parent-reported version was used for children 5-11 years of age and the self-reported version was used for adolescents 12-17 years of age. Guardians of children under five were asked if they had concerns about their child’s learning or development. In South Africa, among children under five, 36.6% of guardians in the FABRIC sample and 28.6% in the community sample reported concerns about their child’s learning. In Zambia, the numbers were slightly lower, with 26.1% of guardians in the FABRIC sample and 29.7% in the community sample reported concerns about learning. None of the differences between the groups were statistically significant. Child development concerns were measured across six domains, using age-specific questions modified from the Child Developmental Assessment in Zambia (CDAZ). The scores in the community sample were normalized to 1.0 to allow for comparison across the 0-4 years age group. Table 7 illustrates the mean child development index scores in the FABRIC sample relative to the community sample. Across all developmental domains, there were no statistically significant, or programmatically relevant differences, in concerns between the FABRIC and community samples. Table 7. Mean child development index scores in FABRIC beneficiaries aged 0-4 years, relative to scores normalized to 1.0 in the community sample (n = 70) Development component Score Interpersonal development 1.02 Gross motor development 0.97 Fine motor development 1.02 Receptive language development 0.98 Expressive language development 0.98 Self-help development 1.02 The SDQ was administered to guardians of children 5-11 and directly to adolescents 12-17 to assess psychosocial and emotional wellbeing. We calculated scores according to standard scoring algorithms and categorized children as normal, borderline, or abnormal using predetermined published ranges. The proportions of children scoring in the normal range for each domain, by country, are illustrated in Tables 8 and 9. The total difficulties score is the sum of all but the prosocial domain. In South Africa, in general, we see no difference between the FABRIC and community samples in either age group. However, among 5-11 year olds, a significantly smaller proportion of the FABRIC sample (15.0%) compared to the community sample (40.7%) had normal scores on the emotional symptoms domain (p = 0.03). This difference is not seen in the 12-17 year olds. When stratified by gender in South Africa, there was a significant difference between males ages 5 to 11 in the FABRIC and community samples with regard to hyperactivity (p = 0.04). Among females ages 5 to 11, there was a significant difference in the emotional symptoms domain between the FABRIC and community samples (p = 0.03). In this age group, 10.5% (n = 2) of FABRIC females scored in the normal range for emotional symptoms, while 45.7% (n = 16) of the community sample females did. FABRIC End-of-Project Evaluation Final Report 19 In Zambia, in both age groups, the majority of children scored in the normal range in most domains, with no significant differences noted between FABRIC beneficiaries and the community comparison children. Approximately 50% of both samples scored in the normal range for emotional symptoms, again highlighting the vulnerability of the communities. After stratifying by gender, there were no significant differences between the FABRIC and community samples in Zambia. Table 8. FABRIC beneficiaries and community children ages 5 to 11 in South Africa and Zambia scoring in the normal range on the Strengths and Difficulties Questionnaire SOUTH AFRICA ZAMBIA Measure of Wellbeing FABRIC n = 40 (%) Community n = 81 (%) FABRIC n = 44 (%) Community n = 87 (%) Total difficulties 14 (35.0) 31 (38.3) 34 (77.3) 67 (77.0) Normal emotional symptoms 6(15.0)* 33 (40.7)* 24 (54.6) 45 (51.7) Normal conduct problems 16 (40.0) 22 (27.2) 35 (79.6) 62 (71.3) Normal levels of hyperactivity/inattention 27 (67.5) 64 (79.0) 40 (90.9) 77 (88.5) Normal peer relationship problems 16 (40.0) 26(32.1) 31 (70.5) 55 (63.2) Normal prosocial behaviors 31 (77.5) 68 (84.0) 41 (93.2) 75 (86.2) *Significantly different at p = 0.05 level Table 9. FABRIC beneficiaries and community sample children ages 12 to 17 in South Africa and Zambia scoring in the normal range on the Strengths and Difficulties Questionnaire SOUTH AFRICA ZAMBIA Measure of Wellbeing FABRIC n = 39 (%) Community n = 75 (%) FABRIC n = 45 (%) Community n = 90 (%) Total difficulties 24 (61.5) 53 (70.7) 39 (86.7) 79 (87.8) Normal emotional symptoms 23 (59.0) 46 (61.3) 32 (71.1) 72 (80.0) Normal conduct problems 29 (74.4) 51 (68.0) 42 (93.3) 81 (90.0) Normal levels of hyperactivity/inattention 33 (84.6) 66 (88.0) 44 (97.8) 88 (97.8) Peer relationship problems 23 (59.0) 52 (69.3) 35 (77.8) 72 (80.0) Normal prosocial behaviors 34 (87.2) 65 (86.7) 44 (97.8) 87 (96.7) A noticeably smaller proportion of both samples in South Africa scored in the normal ranges than in Zambia. 2.5.Psychosocial Discussion Although psychosocial interventions were a significant component of the FABRIC project, a relatively small proportion of the surveyed FABRIC sample reported receiving services in the preceding 12 months. The most encouraging coverage statistic came out of South Africa, where almost a quarter of guardians of children under FABRIC End-of-Project Evaluation Final Report 20 5 reported receipt of some form of psychosocial support in the preceding three months. This result indicates the potential for higher coverage rates does exist and can be realized. The psychosocial wellbeing indicators show that children less than five years of age in the FABRIC sample, despite being OVC, were meeting the same developmental milestones as children in the community sample. Although project impact is difficult to assess in the absence of baseline data, it is possible that the FABRIC￾supported psychosocial services fostered early childhood development among service recipients. Though there were no significant differences between FABRIC and community samples in this age group, the proportions of guardians reporting concerns about their child’s learning or development were high: around thirty percent across both samples in both countries. This figure is illustrative of the vulnerability of all young children in these communities, regardless of whether were targeted in the FABRIC project. Among 5-11 year olds, there was one significant difference in SDQ scores between FABRIC and community samples, but overall, scores were similar in all domains. The most prominent disparity in this age group was not between samples but rather between countries. The proportion of children scoring in the normal range on the SDQ was markedly lower in South Africa than in Zambia for each wellbeing measure. Given that Zambia had fewer resources and was generally perceived as more vulnerable, these results were unexpected. Coupled with the educational results from South Africa presented earlier, this discrepancy may underscore the need to focus on this age range moving forward. In the older age groups, in both countries, similar proportions of the FABRIC and community samples scored in the normal range on the SDQ. This is reassuring given the presumed heightened vulnerability of the FABRIC target population. However, because there was no baseline, we cannot say with certainty the FABRIC interventions were effective in improving the sample’s psychosocial status, but we can reasonably assume they were beneficial. Future longitudinal assessment of these children using the SDQ may provide further information about the impacts of participating in the FABRIC project. 3. Food and Nutrition 3.1.PEPFAR Guidance and Policy PEPFAR programming guidance restricted direct food support and encouraged programs to leverage food security assistance from other partners. PEPFAR guidance provided for weaning foods and other nutritional support for children less than two years of age, nutritional counseling, information dissemination, and systems supports such as policy and technical guidelines development or coordination to promote food access. Guidance excluded school feeding programs and broad based food-assistance for all school children. 3.2.FABRIC Actual Practice FABRIC supported nutritional education and counseling during the first year of the project. This included nutritional training for caregivers and cooking demonstrations. FABRIC also supported supplementary feeding programs including school programs at community schools in Zambia and at after-school feeding centers in South Africa. Weaning foods were provided to infants in South Africa through leveraged funding. The project provided food parcels through leveraged funding and linked children to other health and nutrition interventions. Greater nutritional supports through leveraged donations were more readily available in South Africa than in Zambia. 3.3.Food and Nutrition Coverage Results The Food and Nutrition domain of our survey instrument collected information from FABRIC guardians about the receipt of food packages or information or to prepare healthy foods from the FABRIC project in the preceding 12 months. FABRIC End-of-Project Evaluation Final Report 21 “They are taught and do homework, and they get enough food at the center.” -Guardian of 15-year-old FABRIC beneficiary In South Africa, 37% (n=47) of the FABRIC sample reported receiving at least one food package and 8% reported receiving information to prepare healthy food in the preceding 12 months. Almost 5% of community sample households in South Africa also reported receiving a food package from the FABRIC project in the preceding twelve months; 1.6% received information on preparing healthy foods. Table 10 stratifies the support by age. Table 10: Receipt of Food and Nutrition services from FABRIC in South Africa FABRIC sample by age Received information from FABRIC about preparing healthy foods in preceding 12 months Received at least one food package from FABRIC in preceding 12 months Average number of food packages received in preceding 12 months All ages 7.9% 37.0% 3.5 0-4 yoa 7.3% 63.4% 2.2 5-11 yoa 11.6% 25.6% 4.9 12-17 yoa 4.7% 23.3% 3.5 In Zambia, 2.2% of the FABRIC sample reported receiving at least one food package in the preceding 12 months and 2.2% reported receiving information on preparing healthy foods. No household in the community sample received food packages and 0.7% received information on preparing healthy foods. Of the twelve adolescents from the FABRIC sample interviewed for the qualitative portion in South Africa, nine reported receiving meals at their FABRIC sub-recipient, and eight received food parcels from the FABRIC project. Only one of twelve adolescents interviewed in Zambia reported receiving food at school and only one reported receipt of a food parcel from FABRIC. 3.4.Food and Nutrition Wellbeing Measures We measured the mid-upper arm circumference (MUAC) to assess nutritional wellbeing in children between 6 months and 5 years. We calculated MUAC-for-age (MFA) z-scores using the 2006 World Health Organization Growth Standards. The proportion of the FABRIC and community samples having MFA z-scores in the normal, moderate malnutrition and severe malnutrition ranges are shown in Table 11. The majority of children in both samples in Zambia and South Africa had normal z-scores, suggesting relatively low rates of malnutrition as assessed by MFA. “They go to FABRIC and have something to eat and will arrive home full.” -Guardian of 14-year-old FABRIC beneficiary FABRIC End-of-Project Evaluation Final Report 22 Table 11. Distribution of mid-upper arm circumference-for-age z-scores in children 6 months to 5 years of age in South Africa and Zambia, relative to World Health Organization Growth Standards (2006) SOUTH AFRICA ZAMBIA FABRIC n = 40 (%) Community n = 75 (%) FABRIC n = 45 (%) Community n = 90 (%) Normal 38 (95.0) 69 (92.0) 40 (88.9) 86 (95.6) Moderate Malnutrition 2 (5.0) 4 (5.3) 4 (8.9) 3 (3.3) Severe Malnutrition 0 (n/a) 2 (2.7) 1 (2.2) 1 (1.1) * Significantly different at p < 0.05 Using standard methods, we calculated individual dietary diversity scores (IDDS), a proxy measure of the nutritional quality of the diet, for all children above 6 months of age.5 The mean IDDS in the FABRIC sample relative to the community sample, by country and age group, are shown in Table 12. The only significant difference was among 12-17 year olds in Zambia, where the FABRIC sample had a lower mean IDDS relative to the community sample. Table 12. Mean dietary diversity scores in the FABRIC and community samples in South Africa and Zambia SOUTH AFRICA ZAMBIA FABRIC Community FABRIC Community All ages 4.5 (1.4) 4.2 (1.5) 3.7 (1.2) 3.9 (1.4) Ages 6 mo – 4 years 4.6 (1.4) 4.2 (1.4) 3.6 (1.3) 3.9 (1.6) Ages 5 -11 years 4.8 (1.5) 4.5 (1.3) 3.8 (0.9) 3.6 (1.1) Ages 12-17 years 4.1 (1.2) 4.2 (1.3) 3.7 (1.2)* 4.2 (1.5)* * Significantly different at p < 0.05 Research suggests a score greater than four reflects adequate micronutrient intake.6, 7 In South Africa, 45.7% of the FABRIC sample and 41.5% of the community sample had an IDDS greater than four. The difference is not statistically significant, even when stratified by age group. In Zambia, 22.2% of the FABRIC sample and 32.7% of the community sample had an IDDS greater than four (p=0.03). Table 13 shows IDDS in Zambia stratified by age group. Statistical differences were seen in the 0-4 year olds and 12-17 year olds, but not in the 5-11 year olds. Guardians and adolescents surveyed were also asked if, in the four weeks preceding the survey, the child had been without food for a full day and night, and how frequently this had occurred. In South Africa, approximately 20% of the FABRIC and community sample in all age groups reported this, with no statistically significant differences. Statistically significant differences were seen in Zambia across the total but this difference disappears when stratified by age-range, as displayed in Table 14. Table 13: Proportion of FABRIC and Community samples in Zambia having an Individual Dietary Diversity Score greater than four FABRIC Community P value Total 22.2% 32.7% 0.03 0-4 yoa 20.0% 37.4% 0.04 5-11 yoa 22.7% 19.3% 0.65 12-17 yoa 23.9% 40.8% 0.05 Table 14: Proportion of FABRIC and Community samples in Zambia having gone without food for a full day and night in the preceding 4 weeks FABRIC Community P value Total 54.4% 43.8% 0.04 0-4 yoa 63.0% 47.3% 0.08 5-11 yoa 56.8% 50.0% 0.46 12-17 yoa 43.5% 34.4% 0.30 FABRIC End-of-Project Evaluation Final Report 23 3.5.Food and Nutrition Discussion The low numbers reporting receipt of food packages and food preparation counseling in Zambia is consistent with the minimal resources focused on this domain as reported by FABRIC project staff. Within South Africa, where more resources were available to leverage support, a greater proportion of the FABRIC sample reported receiving at least one food package in the preceding year. Consistent with the PEPFAR guidance and programmatic focus, the highest rate of receipt of food packages was reported by FABRIC guardians of children less than five years of age, with over half receiving this service. MUAC assessment of the children under- five years of age suggests low levels of severe, acute malnutrition in FABRIC beneficiaries and the community sample in both Zambia and South Africa. Weight- and height-for-age anthropometric measurements, which provide stunting and wasting data, would have provided greater information regarding the long-term nutritional status of the children surveyed, but were not collected due to resource limitations. Nutritional status varies between the two countries. Dietary diversity scores were generally low overall, and worse in Zambia, where limited dietary diversity was observed at all age levels. This is consistent with high rates of food insecurity in Zambia. Within Zambia, FABRIC beneficiaries reported dietary diversity scores that were significantly lower than those reported by the community sample. More than half of the FABRIC sample in Zambia reported to have not eaten for a full day and night in the preceding four weeks, a rate significantly greater than that reported among the community sample. Programmatically, however, a high proportion of both samples reported going a full day and night without food, indicative of the high rates of vulnerability in the communities. Low dietary diversity scores, coupled with the high reported rates of food insecurity, suggest that greater nutritional supports for these OVC may be warranted in subsequent PEPFAR and other USG programming. This cross-sectional household survey was conducted during February and March, which is pre-harvest and the season of greatest food insecurity in Zambia, potentially reflecting the worst-case scenario. 4. Healthcare 4.1.PEPFAR Guidance and Policy PEPFAR OVC programming guidance recommended that OVC projects meet the general health needs of children, with support provided either through direct access to healthcare providers or by referring to existing programs such as PMTCT and HIV providers. It also recommended that vulnerable children receive age￾appropriate, effective HIV prevention messages. Examples of PEPFAR-supported activities included referrals and linkages to healthcare for both routine and acute healthcare, training of caregivers to monitor children’s health, and training of healthcare support staff regarding HIV care and treatment. 4.2.FABRIC Actual Practice FABRIC project staff took measures to meet the primary and general health needs of children at every age level. Activities consisted of indirect supports, such as referrals and linkages to child healthcare, including immunization and growth monitoring. FABRIC also supported preventive health activities including age and context-appropriate HIV prevention education and distribution of insecticide treated mosquito nets (ITNs) from a leveraged source. Using its linkages with Government and other partners, the project relied upon existing health programs to maximize immunization coverage, provide maternal and child healthcare, and train secondary caregivers to monitor children’s health. FABRIC End-of-Project Evaluation Final Report 24 4.3.Healthcare Coverage Results Eight percent of guardians of the FABRIC sample in Zambia and South Africa reported having been referred to (recommended to) access healthcare in the year preceding the survey. 2 In South Africa, 10.2% of FABRIC guardians reported having received a healthcare referral while in Zambia, 6.6% reported having received this service. In South Africa, guardians of FABRIC beneficiaries less than five years of age were most likely to report having received a referral to healthcare, with 22% reporting receipt of this service. A similar proportion of FABRIC beneficiaries reported having received financial or other material assistance from the FABRIC project to access healthcare in year preceding the survey. In South Africa, 11.8% of FABRIC households surveyed reported receiving this service, with 24.4% of guardians of children less than 5 reporting receipt. Reported services received in South Africa included transport assistance, provision of medical fees, and accompaniment to the clinic. In Zambia, 5.2% of FABRIC beneficiaries reported obtaining assistance to access healthcare. None of the guardians of beneficiaries under 5 years of age reported receiving assistance to access healthcare, while 11.4% of guardians of 5-11 year olds and 4.4% of guardians of 12-17 year olds reported receiving this type of assistance. Services reportedly offered in Zambia included provision of medical fees and accompaniment to the clinic. The subset of FABRIC beneficiaries who participated in the semi-structured interviews provided further details about services received through the project. In South Africa, two of the twelve adolescents interviewed reported receiving a healthcare referral, with two adolescents reporting escort or transport assistance to access care. Three of the twelve adolescents interviewed in South Africa also reported they had been given information about health and HIV through the FABRIC project. In Zambia, of the twelve adolescents interviewed about their experiences in FABRIC, four reported having received referrals to healthcare, two reported receiving information about health and HIV, three reported receiving chlorine tablets for water purification, and two received mosquito nets. One adolescent reported that while she had received services through the FABRIC project, the timing was several years ago and would not have been captured in our quantitative survey. Analysis of the semi-structured interviews also suggests that some of the FABRIC implementing partners provided some healthcare directly. 4.4.Healthcare Wellbeing Measures The health wellbeing component of the survey assessed chronic illness prevalence, health-seeking behavior and use of a mosquito net on the night preceding the survey. 2 Survey did not elucidate which group or organization provided the healthcare referral. “Her CD4 count was low. They took care of her. They would give her medicine all the time and loved her until she picked up weight as she looked like a ghost the way she was thin. She is a healthy child now.” -Guardian of 12-year-old FABRIC beneficiary, South Africa “They gave us school items, mosquito nets, jersey for school, but that was a long time (ago), in 2008.” FABRIC End-of-Project Evaluation Final Report 25 In South Africa, 28.4% of the FABRIC sample and 22.1% of children in the community sample were reported to have a chronic illness (p = 0.03). Guardians of FABRIC beneficiaries under five years of age reported the highest rate of chronic illness in their children: 39.0%, compared to 27.4% of the South African community sample (p = 0.04). In Zambia, the FABRIC sample reported the same rates of chronic illness as the community sample. Across all age groups, 16.9% of FABRIC beneficiaries reported chronic illness relative to 14.0% in the community. There were not significant differences within age groups; 17.4% of FABRIC beneficiaries under-five years, 13.6% of beneficiaries 5-11 years, and 19.6% of beneficiaries in the 12-17 years group reported having a chronic illness. The distribution of chronic illnesses reported is illustrated in Figure 2 and reflects illnesses reported by more than one respondent. When asked if they had sought healthcare at a formal health facility for their child’s worst illness in the preceding year, there were no significant differences in health seeking behaviors between the FABRIC and community samples in any age category and the proportions seeking care was fairly high. In South Africa, 83.1% of the FABRIC sample reported seeking healthcare for their worst illness in the preceding 12 months. In Zambia, this number was 85.3% in the FABRIC group relative to 80.1% in the community sample. Government health centers were the most frequently visited formal healthcare establishments (80.0%), followed by Government hospitals (22.1%) and private health facilities (9.0%). Informal sources of healthcare reported included traditional healers, herbalists, the local grocery, community health workers and faith healers, all of which were reported by a small number of respondents. For those FABRIC beneficiaries who did not seek healthcare in the preceding year, the majority reported that the illness was not serious (80%), or that the beneficiary had not been ill (5.4%). Infrequently reported reasons for not seeking care included insufficient money, lack of transportation, and long distance from the healthcare facility. Approximately one quarter (26.3%) of FABRIC beneficiaries in Zambia reported sleeping under a mosquito net on the night preceding the survey. There were no significant differences between the FABRIC and community samples with regards to mosquito net use, and no significant differences between age groups. 23.9% (n = 11) of FABRIC children under five slept under a mosquito net on the night preceding the survey, compared to 31.9 (n = 0 10 20 30 40 50 60 Percentage (%) Figure 2. Distribution of illnesses reported among FABRIC beneficiaries reporting presence of a chronic illness SOUTH AFRICA ZAMBIA FABRIC End-of-Project Evaluation Final Report 26 29) of children in the community sample. The malaria module was not included in the South Africa survey as it was conducted in a non-malarial zone. 4.5.Healthcare Discussion In both Zambia and South Africa, relatively low numbers of the FABRIC sample reported receiving supports to access healthcare, which is consistent with the limited resources allocated to this program area. Children under five years of age in South Africa were most likely to have received healthcare supports, with almost a quarter of those surveyed reporting having received assistance from the FABRIC project to access healthcare. In South Africa, almost one-third of the FABRIC sample reported presence of a chronic illness, with respiratory disease, headaches and chronic diarrhea being the most frequently reported disorders. The FABRIC sample was significantly more likely to report the presence of a chronic illness than the community sample, particularly in children under five years of age, which makes us believe the FABRIC targeting strategy was successful and the most vulnerable were enrolled in the project. Lower rates and fewer types of chronic illness were reported in Zambia, which may reflect hesitancy to report, an inability to recognize chronic illness, or may reflect a different enrollment strategy the Zambia FABRIC project. In both Zambia and South Africa, relatively low rates of HIV were reported, which may be related to reporting bias. It is also possible that voluntary counseling and testing (VCT) rates are low and therefore HIV remains under-diagnosed in these populations. The relatively high number of survey respondents seeking healthcare for their worst illness in the preceding year is reassuring. Of those who did not seek care for their worst illness, most respondents reported not having enough time or that the illness was not serious enough, as opposed to resource limitations. Although the survey was conducted in Zambia during a malarial season, relatively few respondents reported sleeping under a mosquito net on the night preceding the survey. Reasons for this were not explored as part of the survey, and would be beneficial to understand in order to guide future OVC programming. 5. Shelter and Care 5.1.PEPFAR Guidance and Policy In this domain, PEPFAR guided programs to support access to adoption and fostering services, temporary shelter for children in transition, and family reunification when possible. Guidance also included home maintenance support to child or youth-headed households and encouraged innovative community responses to provide care in personalized settings when family options are not available. 5.2.FABRIC Actual Practice This FABRIC project did not focus resources on shelter or home maintenance, due to limited funding and limited ability to leverage support. Sub-recipients did not invest in supplies to repair roofs or houses. Household supplies provided to FABRIC beneficiaries were leveraged from other sources and were often one-time, in-kind donations. These included mosquito nets, water tablets, or supplies such as blankets linked to other psychosocial support and kids camp activities. Limited adoption and fostering services were supported. 5.3.Shelter and Care Coverage Results In this domain, the survey captured supports received in the preceding 12 months including shelter maintenance, household supplies, foster care, and adoption services. Coverage in each area was reportedly low, as this was not a priority domain. FABRIC End-of-Project Evaluation Final Report 27 Shelter Maintenance Assistance Coverage for shelter maintenance assistance was low and seen only in South Africa. Four (3.2%) of FABRIC beneficiaries and one (0.4%) in the community sample in South Africa received FABRIC assistance for home maintenance in the previous 12 months. No support was reported in Zambia. Receipt of Household Supplies The most frequent indicator of support in this domain was receipt of household supplies over the preceding 12 months. In South Africa, 32.3% of the FABRIC sample reported receiving some household supply compared to 4.0% of community comparisons (p < .0001). 8.1% of the FABRIC sample in Zambia reported receiving some sort of household supply in the past 12 months compared to 0.7% in the community sample. The qualitative interviews indicated supplies provided included blankets, liquid chlorine, clothes, and mosquito nets. Foster Care and Adoption Assistance In South Africa, 3.2% of the FABRIC sample and 0.4% of the community sample received some assistance from the FABRIC project to place a child with a foster parent in the preceding 12 months. In Zambia, 4.4% in the FABRIC sample and 0.7% in the community sample received assistance to place a child with a foster parent in the preceding 12 months (p < .0001). 5.4.Shelter and Care Wellbeing Measures We selected standard shelter and care wellbeing measures to measure wellbeing in this domain. Table 15 shows no significant differences in any age range in either country between the FABRIC and community sample when asked if the child sleeps in a place that gets wet in the rain, has a blanket to sleep under, or both. Table 15. Basic shelter and care indicators for FABRIC and community samples in South Africa and Zambia SOUTH AFRICA ZAMBIA FABRIC n = 127 (%) Community n= 253 (%) FABRIC n= 136 (%) Community n= 272 (%) Children who sleep in a place that gets wet in the rain. 54 (42.5) 102 (40.3) 79 (58.1) 151 (55.1) Children who do not have a blanket to sleep under 31 (24.4) 56 (22.1) 64 (47.1) 114 (41.9) Children who sleep in a place that gets wet and do not have a blanket 19 (15.0) 30 (11.9) 40 (29.4) 78 (28.7) 5.5.Shelter and Care Discussion Low coverage in this domain reflects that it was not a priority area for the FABRIC projects even though needs were great. However, one FABRIC project objective was to strengthen networks and linkages. Shelter support was highly dependent on leveraged support, particularly household supplies. South Africa was better able to leverage support than Zambia, in part due to having a stronger in-country network and greater support from SACBC directly, so over 30% of the sample received support. Adoption services were also not prioritized. In Zambia, the lack of differentiation between legal and operational adoption services in the household survey became clear during survey implementation. Legal adoption is very minimal in Zambia, and the majority of adoption supports are unofficial, provided by family members. Unlike in Zambia, in South Africa the family receives a foster grant to support the temporary care of the child. Though the survey did not ask about need for foster care services, this system is widely known and well functioning and it is expected that little assistance from the FABRIC project was necessary. FABRIC End-of-Project Evaluation Final Report 28 Although services were not prioritized, the wellbeing indicators suggest high rates of vulnerability in both countries and both samples. In Zambia, more than half of all children sampled sleep in a place that gets wet in the rain and nearly half did not have a blanket to sleep under. Nearly a third of all children in the Zambia samples slept in a place that gets wet and did not have a blanket. Among children aged 5-11, 50% of the FABRIC sample and 43.9% of the community sample did not possess a blanket to sleep under. This is far higher than the 13% of all children aged 5-11 reported in the 2007 Zambian DHS.8 In South Africa, the numbers were lower (23.7% of the under-five FABRIC sample and 25.7% in the community sample) but showed similar rates of shelter-related vulnerability in both samples. 6. Child Protection and Legal Aid 6.1.PEPFAR Guidance and Policy In this domain, PEPFAR recommended OVC programs confront and minimize the stigma and social neglect faced by OVC, as well as abuse and exploitation, including trafficking, the taking of inherited property, and land tenure. It guided projects to facilitate birth registration, increase access to health and social services, and provide assistance with inheritance claims. Projects were encouraged to strengthen local community systems that protect OVC and to provide training to frontline workers regularly in contact with OVC. 6.2.FABRIC Actual Practice FABRIC activities in this domain included promoting access to social grants, assisting with access to national registration documents such birth certificates and national registration cards or identification (ID) cards. FABRIC referred children to law enforcement for inheritance issues and to social welfare in instances of child abuse. In Zambia, an initiative to register the birth of children was started in October 2009, but did not begin significant scale-up until 2010. The FABRIC project negotiated a reduced fee and paid for birth certificates. 6.3.Child Protection and Legal Aid Coverage Results To measure coverage in this domain, the survey asked about those who received assistance with birth registration and those who received information on children’s rights in the preceding 12 months. Table 16 shows none of the FABRIC or community sample received assistance to register a birth in South Africa. In Zambia, a greater proportion of the FABRIC sample received assistance with birth registration than in the community. In both countries, the FABRIC sample received more information on children’s rights than the community comparison children. Table 16. Proportion of FABRIC and community samples in South Africa and Zambia reporting receiving some child protection and legal aid support from the FABRIC project in the preceding 12 months SOUTH AFRICA ZAMBIA FABRIC n = 127 (%) Community n = 253 (%) FABRIC n = 136 (%) Community n = 272 (%) Reporting receiving assistance to register child’s birth 0 (-) 0 (-) 20 (14.7)* 4 (1.5)* Reporting receiving information on children’s rights 7 (5.5)* 2 (0.8)* 6 (4.4)* 1 (0.4)* * Significantly different at p < 0.05 6.4.Child Protection and Legal Aid Wellbeing Measures Wellbeing indicators in this domain focus on the child having a birth certificate or national ID card (South Africa only), or the guardian knowing how to register a birth, and knowledge of protection laws. In Zambia and South Africa, there was no significant difference between the FABRIC and community samples in their awareness of laws to protect their belongings or their caregivers’ belongings. FABRIC End-of-Project Evaluation Final Report 29 Table 17 shows results of the birth registration wellbeing indicators. There were no statistically significant differences in the number of children with a birth certificate or having registered for a birth certificate between the FABRIC and community samples. In South Africa, the percentage reporting birth certificates was above 90% in both samples and most guardians knew how to register for one. Nearly 76% of the FABRIC sample and 71% of the community sample reported having a national ID card. In Zambia, less than a quarter of each sample reported having a birth certificate or having registered for one. Of guardians of children without birth certificates, 20% in the FABRIC sample and 25% in the community sample reported knowing how to register a child’s birth. Table 17. Child Protection and Legal Aid Wellbeing Indicators of FABRIC and Community sample children in South Africa and Zambia SOUTH AFRICA ZAMBIA FABRIC n = 127 Community n = 253 FABRIC n = 133 Community n = 263 Reporting child has a birth certificate or has registered, but not yet received certificate (%) 122 (96.1) 234 (92.5) 32 (24.1) 60 (22.8) Guardians of children without birth certificate reporting knowledge of how to register a birth (%) 4 (80.0) 10 (52.6) 21 (20.1) 51 (25.1) Reporting having a national ID card (South Africa only) (%) 96 (75.6) 180 (71.2) n/a n/a 6.5.Child Protection and Legal Aid Discussion In general, few respondents reported receiving information on children’s rights from the FABRIC project. In South Africa, children’s rights are a routine part of the curriculum in schools and the FABRIC project did not necessarily need to provide this service. In Zambia, the service was offered, but data show low coverage. The context of birth registration is different in South Africa and Zambia. In South Africa, guardians are incentivized to obtain a birth certificate as early as possible as it gives access to government child welfare grants. This incentive, combined with a history of registering births, means most births are already registered and there was likely little unmet need for the FABRIC project to fill. In Zambia, the government only started issuing birth certificates in the 1970s and there has been very little sensitization around it. Many people feel the birth record at the place of delivery is sufficient. Additionally, a child must be named in order to register a birth, which often does not happen immediately. Delaying registration until 12 months after the birth results in a penalty fee, so many ultimately choose not to register. In the final year of the project, the FABRIC leadership team prioritized assistance with birth registration, primarily by negotiating and paying for a reduced late registration fee. Though the efforts only fully commenced in early 2010, the significant difference in reported birth registration between the FABRIC and community samples suggest these efforts were successful. The 2007 Zambian DHS reports that only 5% of children under 5 five have a birth certificate. The rates in our sample are much higher, with 25% of the FABRIC sample under five years of age and 24% of the community sample under five having a birth certificate. FABRIC End-of-Project Evaluation Final Report 30 FABRIC End-of-Project Evaluation Final Report 31 A S S E S S M E N T O F C H I L D W E L L B E I N G: N A M I B I A FABRIC End-of-Project Evaluation Final Report 32 FABRIC End-of-Project Evaluation Final Report 33 ASSESSMENT OF CHILD WELLBEING: NAMIBIA Overview The evaluation in Namibia used a different approach than Zambia and South Africa and is therefore presented separately. As described earlier, CAFO, the initial IA working with FHI in Namibia graduated from PEPFAR Track I support into the New Partner’s Initiative where received USG monies to continue with their efforts outside of the FABRIC umbrella project. FHI initiated FABRIC activities in year three of the project with a new collaborator, Positive Vibes, which had a different focus and approach than CAFO. At the time of the evaluation, Positive Vibes (PV) Children’s Voices program had been implementing for approximately 14 months. The program is designed in three phases: 1) Disclosure Phase aimed “to enable HIV positive parents to disclose their own and their children’s HIV status to children, and thereby break the silence that often exists around HIV and AIDS within the home.” 2) Foundation Phase aimed “to enable children to explore their lives in the context of HIV and AIDS, and to begin expressing themselves on issues of concern to them.” 3) Children’s Participatory Media Phase aimed “to enable children to use mass media to communicate among themselves, and with a wider audience about their needs and concerns.” At FHI’s request, this evaluation was designed only to assess the self-reported change in the lives of the children participating in the Foundation Phase activities. The Foundation Phase conceptual framework is a series of workshops focused on three domains. Table 18 reflects these domains and the corresponding FABRIC and PRPFAR defined core-service areas. Table 18. Conceptual framework of the Foundation Workshops and the related PEPFAR core service areas in FABRIC/Namibia Outcomes of Children’s Voices Foundation Workshop Core Service Areas of FABRIC/Namibia Confidence and self esteem Psychosocial Support (PSS) Decision-making Protection and Rights Knowledge of HIV Health Education Study Site and Population Two project sites were purposefully selected to represent the different cultures in which PV operates its Children’s Voices project. Omusati was selected in the North of Namibia and Keetmanshoop in the South. Children’s Voices participants were randomly selected from the project rosters and asked to interview. Methods and Procedures This assessment used only qualitative methods. We conducted Focus Group Discussions (FDG) and Key Informant Interviews (KII) in both selected sites. At both evaluation sites, children were divided into age groups 8-10 and 11-13. KIIs were conducted only with children 11-13 years of age, after the guardian was briefly interviewed. Age appropriate activities were employed to engage children and generate discussion around specific topics. FABRIC End-of-Project Evaluation Final Report 34 Instrument development The qualitative FGD and KII guides were developed to elicit a description of the services received through the project and the child’s perceived change from involvement in the FABRIC project though Positive Vibes. Guides were developed by the Boston University team, reviewed by the FHI team, and modified as necessary based on feedback. Guides were translated into the local languages, back-translated, and piloted prior to implementation. Data enumerator training In consultation with BU, the FHI national office in Namibia selected four local enumerators with the intention of two being fluent in the local language at each site. All enumerators were fluent in English, were non-residents of the community, and had experience working with children but minimal experience in qualitative research. In the North, after the enumerators were hired and trained, and data collection commenced, we discovered only one enumerator was fluent in the local language. The other enumerator conducted the sessions and two additional people were hired to assist with translation and transcription. Enumerators attended a three-day training in Oshikati, which included sessions on research ethics, data confidentiality, qualitative research methods, and the roles of the facilitators. A BU faculty member experienced in qualitative methods conducted the training. Data collection, entry, and analysis Data collection took place during March 2010, following receipt of IRB approvals. A member of the BU evaluation team supervised enumerators; an FHI staff member oversaw logistics. Participants were randomly selected from the PV project database and consent from the guardian and assent from the child was obtained. Two enumerators were present to conduct two FGDs and two KIIs. Due to enumerator language difficulties in the North, expanded field notes were not available. In the South a team of two enumerators (one facilitator, one note taker) conducted two FGDs and two KIIs in central locations. We conducted age-appropriate activities in each group to engage the children and generate discussion; activities varied by age group, but all focused on the same domains and all questions asked were the same. All FGDs and KIIs were digitally recorded, transcribed and translated into English by the enumerators and hired translators. Documents were reviewed for completion by the supervisor and transported back to Lusaka for entry, analysis, and storage. Interview transcripts were imported into NVivo version 8.0 (QSR International, Cambridge, MA) for analysis. Themes were identified a priori, as per the facilitation guides, and coded using domain analysis by two coders to ensure validity and reliability. Results Results of the orphan wellbeing assessment in Namibia are presented below and organized by the three domains of the PV Foundation Workshop and their corresponding PEPFAR domain. Results between age groups did not differ and are therefore presented together. KIIs are presented as individual stories. Characteristics of the sample Two Focus Group Discussions were conducted in the North and two in the South. Table 19 describes the basic characteristics of each group. FABRIC End-of-Project Evaluation Final Report 35 Table 19: Characteristics of Focus Group Discussions in Namibia NORTH SOUTH 8-10 years of age (n= 5) 11-13 years of age (n= 11 ) 8-10 years of age (n= 7) 11-13 years of age (n= 7) Number of Females 3 5 4 3 Number of Males 2 6 3 4 Average Age (years) 10.2 11.5 9.9 12.3 Median Grade (range) 4 (2-5) 5 (1-6) 3 (2-5) 6 (5-8) Two KIIs were conducted in the North and two in the South. One interview in the North conducted by the non￾local language speaker was dropped from the analysis due to poor quality and potentially fabricated data. Table 20 displays basic characteristics of the three retained interviews. Table 20: Characteristics of Key Informant Interviews in Namibia NORTH SOUTH Interview #1 Interview #2 Interview #3 Age 12 years 12 years 13 years Gender Male Female Male Caregiver Aunt Grandmother Mother Orphan Status Double Double Paternal HIV Status (if known) Positive Unknown Positive Qualitative results Focus group discussion guides focused on two areas, namely: (1) causes of happiness and sadness for children, and (2) perceived impact of Positive Vibes project. To better understand the context and how our focus group participants perceived their communities and lives, children were asked to self-report causes of happiness and sadness. In the North, children reported things in their community that made them sad were: cemeteries and death, bullies, thieves, drunks and bars, teachers who beat students, nurses who swear at patients, and feeling like you cannot help someone. In the South, the most frequently reported themes were cemeteries and death. Others noted police station, nurse/doctor, teachers who beat students, bullies, and domestic violence. Only the 8-10 year old FGD in the South reported headmen made them sad. Participants in both areas reported that school, friends, family, church, doctors, and “being treated with respect” made them happy. In the second area of focus group discussion, children reported their perceived impact of Positive Vibes on their lives. Results from the FGDs were coded into the three PV domains. Responses, ordered by frequency and stratified by region, are detailed in the Tables 21, 22 and 23. “The cemetery makes me sad because I always think of my uncle who hanged himself. I saw him hanging on a tree when he killed himself” - FGD South, 11-13 yoa “Staying in bars, it makes me sad. Even at night people are in bars getting drunk till they start swearing at each other and start fighting and injuring each other.” FGD North, 11- 13 yoa FABRIC End-of-Project Evaluation Final Report 36 Table 21: Most frequently reported and relevant quotations of perceived impacts of Positive Vibes on confidence and self-esteem as described by Focus Group Discussion participants stratified by region NORTH SOUTH Confidence and Self-esteem (Psychosocial Support) Improvement in school Made friends Learned to play games, draw, etc. Speak loudly Improvement in school Learned not to be shy Made friends Speak loudly “When I was in CVP I started passing well and my performance improved. But also my relationship with teachers has improved even more.” - FGD North, 8-10 yoa “I was very weak in school but now I am doing much better.” - FGD South, 8-10 yoa “We learned new things and we also taught our teachers what they didn’t know.” - FGD North, 11-13 yoa “Before CV, I was very shy and couldn’t mingle easily with other people.” - FGD South, 8-10 yoa Table 22: Most frequently reported and relevant quotations of perceived impacts of Positive Vibes on Decision Making Skills as described by Focus Group Discussion participants stratified by region NORTH SOUTH Decision Making (Protection and Rights) Speak loudly Speak freely Respect adults Respect other children Don’t fight, swear, laugh at others Do things without being told Children’s rights Talk openly Not shy Respect other people/adults Don’t swear or fight Good manners “Ever since CVP I always speak loud and clearly, and my mother’s always happy with me” – FGD North, 8-10 yoa “I have learned to respect my parents but also about my rights.” – FGD South, 8-10 yoa “I was very disrespectful but the workshop has helped me to show respect towards people and my mother” – FGD North, 8-10 yoa “I was very naughty and was lazy. But since joining the program I changed and started respecting people.” – FGD South, 11-13 yoa “I used to be in the streets, throwing stones at children and swearing at people but I stopped when I joined Children’s Voices.” – FGD South, 8-10 yoa “I can choose good friends now.” – FGD South, 11-13 yoa “I attend classes every afternoon. We learn about HIV and how to make our own decisions.” – FGD South, 11-13 yoa FABRIC End-of-Project Evaluation Final Report 37 Table 23: Most frequently reported and relevant quotations - Perceived impacts on Knowledge of HIV/Health Education of Positive Vibes as described by Focus Group Discussion participants stratified by region NORTH SOUTH Knowledge of HIV (Health Education) How people get sick Eat expired/spoiled food No food or drink Smoking or drinking alcohol Coughing spreads TB Eating sugar Contagious – TB or Chicken Pox Cough without covering the mouth Smoking too much How people stay healthy Food health Eat healthy 2X day Wash hands Don’t eat old/spoiled food Don’t touch blood Don’t use a needle used by another person Don’t drink uncovered water Drink medicine as told Keep bad things from your mind Drink medicine as told Don’t drink or smoke Eat healthy Rest and exercise Stay clean (clothes and body) No food in toilet Stay connected to 1 person How people get HIV Touch blood Share toothbrush with wounds Sex without a condom Sleep with too many people Share needles Sex without a condom Sleep with too many people Blood contact “You need to keep bad things away from your mind. Do not feed your mind with bad music, swearing and other dirty things.” – FGD North, 8-10 yoa “We learned that even when you go in the shop to buy an apple, know that someone could have touched it and they might have been to the toilet and did not wash their hands and so you need to wash that apple before you eat it.” –FGD North, 8-10 yoa “People get AIDS if they have sex without a condom with someone who has AIDS…Maybe if you are with someone who has cancer then you can also get cancer….” – FGD South, 8-10 yoa “If you have a husband then you must only stay with him to avoid HIV. If your husband sleeps with someone else and again with you then you can get HIV.” – FGD South, 8-10 yoa “Let’s say someone has HIV and his blood came into contact with your blood then you will get HIV.” – FGD South, 11-13 yoa “If you are HIV positive then you must drink your medication on time, rest and eat healthy food.” – FGD South, 11-13 yoa “I will tell them that we have our rights and that they cannot be violated. I will tell them about the exercise so they can have flexible bodies. And to play soccer and to color.” - FGD North, 11-13 yoa FABRIC End-of-Project Evaluation Final Report 38 “I will tell them to join because it’s a good thing and you learn a lot of things. Things like respecting your parents.” – FGD South, 11-13 yoa “They should speak loud and freely. They shouldn’t laugh at other children.” – FGD North, 8-10 yoa Key Informant Interviews Three KIIs were conducted as part of the Positive Vibes assessment. Below are the summaries of the interviews. Names have been changed to protect the identity of the respondents. Interview !: Dinah is a 12-year-old girl who lives with and is cared for by her grandmother. She and two other children in the household are enrolled in the PV project. Dinah reports that she is “happy” to be participating in PV, and that those children in the house who are not yet a part of the project long to join, so that they can be as happy as she is. Dinah's grandmother reveals that she is also very satisfied with PV. She attributes a number of recent improvements in her granddaughter’s behavior to Positive Vibes. Prior to enrolling in the program, she says, Dinah was “stupid”, “careless”, “lazy” and “stubborn”. She notes that since enrolling, Dinah is less shy, makes more time for chores and homework, and responds eagerly when asked to do something. She also reports that PV has helped Dinah improve her performance at school and “break away from bad friends”. She also observes that her granddaughter has exhibited increased knowledge of HIV/AIDS, though she voices dismay that sex was freely talked about at the workshops. When asked about the program’s effect on Dinah’s health, she reports that there are times when they do not have enough food to eat. Dinah corroborated this assessment, noting that she eats less since having joined PV but did not explain why. Despite these complaints, Dinah’s grandmother is hopeful that the program will continue, as it is the only one of its kind in the area and she believes it helps children stay focused and off the streets. “It is a good program,” she says. “People want their children to join because the effects…are great." FABRIC End-of-Project Evaluation Final Report 39 Interview 2: Justin is a 12-year-old boy who lives with his aunt and three other children. Justin declared that he “felt good” about his enrollment in the Positive Vibes project from the beginning. He reports that through PV he has learned about rights and how to express himself. He even attributed his improved position at school to the project. He did note that his involvement at PV seemed to inspire some resentment and anger among others. Regarding the other children in his home he said, “I know they felt bad. Some even refused to talk to us.” He stated that some of his friends were similarly bitter, noting they “shouted at us” and “prompt us to go home”. Justin’s aunt reports that she is very supportive of the program. She says she is particularly grateful because PV has helped Justin to realize he is not the only HIV positive child without parents and to become more comfortable with his status. She notes that her nephew is happy and always eager to attend PV-sponsored workshops. Since enrolling at PV, she says, he exhibits more confidence and is more open than he had been in the past. In addition to being beneficial for her nephew and other children living with HIV, she suggests that PV workshops are also helpful to caregivers like her, who do not necessarily know a great deal about HIV or caring for someone living with the disease. Interview 3: Joshua is a 13-year-old boy who lives with his mother and three of his five siblings. He is the only member of his household currently enrolled in the Positive Vibes/Children’s Voices project. In an interview with Joshua’s mother, she reveals that her son is eager to attend PV activities and learn about HIV. Though she says she doesn’t note any major changes in his behavior since enrolling, she does say that he now talks openly about what is happening and how he is feeling. She is happy the program keeps him off the streets, and voices her hope that the program will continue, as it helps children to “speak out”. An interview with Joshua reveals that he shares his mother’s sentiments about the positive impact Positive Vibes has had on his wellbeing. Joshua describes his behavior prior to enrolling at PV as “naughty”; he states that he frequently bullied other children and fought in the streets at the urging of peers. Since he began attending PV-sponsored programs in 2009, Joshua says he has learned to speak up for himself and that he has gained confidence. He has new friends and doesn’t “go looking for trouble” anymore. Overall, Joshua says that he is “happy” to be a part of PV. He is hopeful that the program will enroll more children, because “there are still children out there who needs help like we did”. FABRIC End-of-Project Evaluation Final Report 40 Discussion Qualitative data from this evaluation suggest the Foundation Phase of the Positive Vibes Children’s Voices project has enhanced the lives of the participants. Confidence and Self-esteem/Psychosocial Support Children seem to feel more confident and empowered after participating in the PV Children’s Voices Foundation Phase activities. When children were asked to think back to how they felt before they participated in the Children’s Voices project compared to how they feel now, they expressed confidence in their school performance and happiness with their extracurricular activities. Although we have no measure of school performance, the children’s perception of their improved abilities is positive. Children also expressed confidence in their abilities to communicate and interact with others. One child explained they are able to teach teachers, indicating a strong perception of value and self-worth. Very few differences were seen between the response of those in the North and the South, or between age groups. All children perceived an improvement in this domain from what they remember before the project. Decision Making/Protection and Rights When responding to questions designed to elucidate change since the CV project in decision-making or children’s rights, children spoke primarily of the concepts “respect” and “speaking loudly” or “openly.” Responses in the North and South focused on respecting adults and other children and treating them kindly. Children also perceived that participating in the CV project has helped them to make better choices and their own decisions in their lives. Responses did not vary greatly between the North and the South, although those in the North focused more discussion time on respecting all people, particularly adults. The children seem to feel empowered to make their own decisions. Knowledge of HIV/Health Education Children in the North and South had a strong understanding of reasons people get sick and how people can stay healthy. Major themes identified in both regions included food-borne illnesses, infectious diseases, sexual transmission, chronic illness, and personal habits including smoking, hygiene, and eating unhealthy foods. The 8-10 year olds in the North was the only group that specifically mentioned mental health. While knowledge on health was very general, responses from the children when asked about HIV were much more specific. Knowledge of both modes of transmission of HIV and treatment of HIV was fairly high and accurate among all groups. According to facilitators, children comfortably spoke of knowledge of sex and HIV. In general, children learned, retained, and applied information from the PV project. When asked how children who are sad make themselves happy again (coping strategies), children in the North and South responded similarly. In order of frequency, both groups said (1) they play with friends, (2) speak to elders or someone they trust, or (3) go to church. Children in the North also reported going to school. Children in the South specifically noted that they go to a Children’s Voices counselor, or cry alone. Children reported joining the CV project for many reasons. In the North the most frequently reported reason was to learn to exercise and be healthy. Other reasons included because they are orphans and need assistance or they were mistreated at home. In the South, children reported joining primarily to stay out of the streets and to learn new things. Several children said they joined because they were naughty or their mother told them to do so. Responses were generally positive when children were asked what they would tell other children about joining the project, echoing mostly sentiments of rights and respect reported in the “Decision Making” domain. FABRIC End-of-Project Evaluation Final Report 41 In Namibia, in general, children learned, retained, and applied information from the PV project. However, our ability to attribute impact or see change over time is limited by the cross-sectional nature of the study, no comparison population, the potential for recall bias when asking children to remember how they were before participating in the project, and the lack of quantitative data to triangulate for validity and reliability. “ I am thankful to the program and I will continue to support the program. It is a good thing. It is assisting our children in shaping their lives as well as us their caregivers.” - North, Guardian of 12-yoa girl FABRIC End-of-Project Evaluation Final Report 42 FABRIC End-of-Project Evaluation Final Report 43 O R G A N I Z A T I O N A L A S S E S S M E N T: S O U T H A F R I C A , Z A M B I A, A N D N A M I B I A FABRIC End-of-Project Evaluation Final Report 44 FABRIC End-of-Project Evaluation Final Report 45 ORGANIZATIONAL ASSESSMENT: SOUTH AFRICA, ZAMBIA, NAMIBIA Overview One major goal of the FABRIC project was to strengthen the capacity of its implementing agencies (SACBC, ECR, and PV). The organizational assessment aimed to determine if FABRIC strengthened the capacity of its implementing agencies to effectively target, coordinate, and sustain projects of sub-recipients. Specifically, this component sought to: 1. Determine if the organizational capacity of the FABRIC implementing agencies is stronger now than before receiving support; 2. Determine if the FABRIC implementing agencies have the capacity to deliver OVC support to sub￾recipients; 3. Learn from the sub-recipient leadership what FABRIC-specific inputs they associate with improved organizational performance; 4. Identify successes and challenges faced by FABRIC in implementing organizational capacity strengthening activities. For this evaluation, “capacity” was broken into the following two components: 1. Organizational Management: This includes financial management and systems, mentorship, general project management, networking, volunteer management, and resource mobilization skills. 2. OVC Care Technical Support: This includes needs assessment, program implementation, and monitoring and evaluation for quality improvement, mentorship, and OVC technical areas including good practices for OVC care and service delivery. Methods We used a mixed-methods approach to assess organizational capacity. Quantitative and qualitative methods were used at three tiers within the FABRIC project ((FABRIC regional office/country offices, the implementing agencies (IAs), and the sub-recipient level (SR)) in Zambia and South Africa, termed the implementing partners in Namibia. Technical and Organizational Capacity Assessment Tool (TOCAT) For this evaluation, we re-applied TOCAT domains relevant to FABRIC goals at the implementing agency level. The FHI TOCAT and scoring system seen in Table 24 was developed by FHI headquarters and was previously administered in at the Implementing Agency (IA) level in each country. Employees used the tool to assess organizational capacity and reached a consensus score in each TOCAT domain. Complete methods can be found in individual country reports. Assessing the validity and reliability of the TOCAT tool was not the aim of this evaluation. However, since baseline data was available, FHI requested we re-apply the tool. In March 2010, core FABRIC team members self-administered the TOCAT and submitted it electronically to the BU research team. Due to time constraints, individual scores were averaged rather than achieving group consensus. Although the Director of Positive Vibes expressed concern over the quality and validity of the baseline data since it was collected in the very nascent stages of the organization, for comparison we have included it. Table 24. Family Health International TOCAT Scoring System Ranking Implication Percent range 1 Beginning 0-25% 2 Showing Results 26-50% 3 Systematically Achieving 51-79% 4 Model System 80-100% FABRIC End-of-Project Evaluation Final Report 46 Document Review and Qualitative Interviews We conducted a document review of work plans and annual reports to identify successes and challenges related to capacity strengthening activities and determine if activities were successfully implemented. Semi-structured interviews were conducted at the FABRIC, Implementing Agency (IA), and at the sub-recipient levels (partner level in Namibia) to understand capacity strengthening activities conducted throughout the project. Data Management and Analysis Complete TOCAT surveys were entered and analyzed using Microsoft Excel 2007. Themes for the document review and qualitative interviews were determined a priori, and entered and analyzed. Data were triangulated to build a complete picture and determine if FABRIC met its capacity strengthening goals. Results TOCAT Scores South African Catholic Bishops’ Conference (SACBC) and Expanded Church Response (ECR) each have 4 core team members responsible for FABRIC and all were asked to complete the TOCAT. Three of the four (75%) SACBC employees completed and submitted the form. On average, respondents had 19.0 months working for FABRIC. Two of the four (50%) ECR employees completed the form. On average, ECR respondents had 36.0 months working for FABRIC. Positive Vibes (PV) has five core team members responsible for the Children’s Voices FABRIC project. Five of five (100%) employees submitted a completed form. PV became a FABRIC IA in year three of the project so had only been operating for 14 months at the time of data collection; on average, PV respondents had 9.2 months working for FABRIC. Given the high turnover throughout the FABRIC project, the respondents are likely different than those who completed the baseline TOCAT. Table 25 reflects the available pre- and post-FABRIC scores for SACBC, ECR, and PV. Despite repeated attempts to secure the original pre-FABRIC data, only final reports were available; final reports for SACBC and ECR did not reflect all domains. PV has all data points. In domains where we were able to get baseline data, there is a general self-perceived improvement in the functioning of the organization across all IAs. Specifically, all IAs perceived improvements in their capacity for training and mentoring, and in their program planning, monitoring and evaluation from baseline to end line. SACBC rated all available domains equal or superior to their capacity at baseline with all domains as “model system”. SACBC perceived greatest improvements in “Technical quality standards” and “Capacity for training and mentoring”, which ranked 2 at baseline and 4 at end line. At end line, ECR ranked all but two domains as a “model system”. ECR perceived a decrease in “Number, mix, and capacity of technical staff,” but perceived improvements across all other domains. At end line, Positive Vibes perceived all but “Technical Quality Standards” as a “model system”. From baseline to end line, PV perceived improvements in every domain. FABRIC End-of-Project Evaluation Final Report 47 Summary Presentation of Results Tables 26, 27 and 28 present the key points that emerged from the TOCAT, the document review, and the interviews for each country. Each table presents the perceived benefits of collaboration in the FABRIC project, the perceived challenges and limitations, and other general observations emerging in this component of the evaluation at the regional, Implementing Agency and sub-recipient levels. Table 25. Pre and post Technical and Organizational Capacity Assessment Tool (TOCAT) results from FABRIC supported domains in South Africa and Zambia SOUTH AFRICA (SACBC) ZAMIBA (ECR) NAMIBIA (PV) FABRIC Supported Domain Pre-test ranking Post-test ranking Pre-test ranking Post-test ranking Pre-test ranking Post-test ranking 1. Number, mix and capacity of technical staff 3 4 3 3 3 4 2. Technical quality standards 2 4 - 2 2 3 3. Capacity for training and mentoring 2 4 3 4 3 4 4. Governance and leadership - 4 4 4 3 4 5. Management practices 4 4 3 4 3 4 6. Operational planning - 4 3 4 2 4 7. Structure: roles and responsibilities - 4 3 4 3 4 8. Partnering and networking 4 4 3 4 3 4 9. Financial planning and budgeting - 4 4 4 3 4 10. Cash and banking - 4 4 4 3 4 11. Procurement - 4 4 4 3 4 12. Program planning/ Monitoring and evaluation 3 4 2 4 3 4 13. Data collection systems - 4 2 4 3 4 14. Stakeholder communications and reporting 3 4 3 4 3 4 FABRIC End-of-Project Evaluation Final Report 48 Table 26. Organizational Capacity Assessment Results: FABRIC in South Africa REGIONAL PERSPECTIVE: FHI Perceived strengths of collaboration SACBC receives support from multiple funders and had a strong baseline infrastructure Strong and capable organization IMPLEMENTING AGENCY PERSPECTIVE: SOUTHERN AFRICAN CATHOLIC BISHOPS’ CONFERENCE (SACBC) Perceived strengths of collaboration Self-perceived improvements in the following areas of organizational and technical capacity: Number, mix, and capacity of technical staff Technical quality standards Capacity for training and mentoring Program planning/M&E Stakeholder communications and reporting Self-perceived improvements in individual capacity, particularly organizational management and donor regulations Perceived challenges and limitations I. Slow, irregular flow of finances II. High attrition rates among staff III. Physical distances between SACBC and SR IV. Challenge harmonizing needs perceived by the donor with actual needs in the field V. Difficulty translating complex FHI capacity-strengthening trainings to sub-recipient level General observations Capacity strengthening activities focused on organizational management and donor regulations, and less on OVC Care Technical Support Monitoring and evaluation support was noted as the most important contribution to the organizational capacity and program success Capacity development attributed in part to the strong relationship between FHI and SACBC Felt FABRIC improved capacity of sub-recipients from before the project, but capacity still remained low SUB-RECIPIENT (SR) PERSPECTIVE Perceived strengths of collaboration SACBC support was most helpful in the following areas: Assistance with filing, office work, and computers Report writing and documenting events Financial and programmatic trainings according to donor policy Proposal writing and networking Improved staff morale and motivation among SR Improved financial management and accountability Perceived challenges and limitations Donor goals perceived as unrealistic in the face of limited financial resources and overextended staff Insufficient or no remuneration for volunteer caregivers General observations Felt Organizational Management support was more beneficial than OVC Care Technical support FABRIC End-of-Project Evaluation Final Report 49 Table 27. Organizational Capacity Assessment Results: FABRIC in Zambia REGIONAL PERSPECTIVE: FHI Self-perceived benefits of collaboration Strong networking skills at ECR Perceived challenges and limitations High rates of staff turnover, inadequate staff numbers, low individual capacity, and a generally weak infrastructure at ECR FHI regional team doubled as Zambia country team, so less support and attention to ECR General observations Support focused more on Organizational Management and less on OVC Technical Care support. FHI activities supported sustainable networking and grant writing to better position ECR for direct funding IMPLEMENTING AGENCY (IA) PERSPECTIVE: EXPANDED CHURCH RESPONSE (ECR) Self-perceived benefits of collaboration Perceived improvements in the following areas: Training and mentoring capacity Management practices Operational planning Structure Partnering and networking Programming planning/M&E Data collection systems Stakeholder communications and reporting Perceived challenges and limitations Inadequate staff to meet donor expectations and provide quality services Under-supported for program expansion Minimal or no remuneration for caregivers Difficulty translating complex FHI capacity-strengthening trainings to sub-recipient level VI. Limit for support is of 18 years old, so older children starting school were dropped when they turned 18 General observations Reported insufficient financial management support from FHI, but still improved financial management capacity Minimal support in the OVC Technical Care domain aside from monitoring and evaluation Respondents suggested an OVC Specialist and Advocacy Specialist Perceived benefit of the FABRIC project to their organization SUB-RECIPIENT (SR) PERSPECTIVE Self-perceived benefits of collaboration Improvement in overall capacity, particularly donor rules and regulations ECR trainings were helpful in the following areas: Monitoring and evaluation Financial management Psychosocial counseling Proposal writing Entrepreneurship Perceived challenges and limitations Slow flow of finances Maintain quality when having to serve such high numbers General observation All sub-recipients reported a positive impact of the FABRIC project, particularly because of the capacity strengthening activities and the funding. One unforeseen challenge in Zambia and South Africa was the donor restrictions on education assistance to children after they turned 18. This resulted in the termination of support for students who had begun their formal educations at a late age. FABRIC End-of-Project Evaluation Final Report 50 Table 28. Organizational Capacity Assessment Results: FABRIC in Namibia REGIONAL PERSPECTIVE: FHI Self-perceived benefits of collaboration Positive Vibes perceived as “well-respected” organization with strong foothold in community Perceived challenges and limitations Differing styles and methodologies at PV and FHI Persistent lack of M&E capacity, challenges measuring quality Long-term sustainability of PV projects in question Too great a focus on leadership and coordination; too little focus on program implementation Reliance on volunteers Overall observations FHI prioritized the delivery of M&E training from the beginning of the collaboration with PV, but confusion in this domain persists. Long-term sustainability could perhaps be improved through increased human resource capacity. IMPLEMENTING AGENCY PERSPECTIVE: POSITIVE VIBES Self-perceived benefits of collaboration Financial support Administrative support Small-scale technical assistance Increased staff involvement Improved credibility and strengthened relationships Expanded coverage areas Perceived challenges and limitations Relatively low technical capacity Discontinuation of funding might reduce organizational capacity Insufficient staff numbers Insufficient M&E trainings; evaluation needs strengthening Differing styles and methodologies at PV and FHI High turnover rate among freelance facilitators Overall observations FABRIC was perceived to have improved overall capacity at the IA level PARTNER PERSPECTIVE Self-perceived benefits of collaboration Funding support is beneficial Proposal writing and networking support from Positive Vibes Financial planning and budgeting support Establishment of financial management procedures Workshops on finances and accounting Ongoing staff support and mentoring Caregiver trainings Perceived challenges and limitations Slow disbursement of funds; Partners sometimes have to financially support Children’s Voices program Poor communication and clarity, particularly between partner organizations General observations Staff voiced satisfaction that FABRIC services took the needs of the partners into consideration, offering assistance in those domains where it was most needed.. Funding support and financial management skills were the most frequently cited contributions. Overall, staff felt FABRIC increased capacity to deliver services to OVC; likely would not have had the capacity to serve so many OVC without these supports FABRIC End-of-Project Evaluation Final Report 51 ”… FABRIC overall has increased the capacity of ECR to work with OVC.” –ECR respondent “In terms of strengthening our capacity it did assist us in achieving our organizational goals.” –PV respondent “They gave a lot of trainings from FHI to IA level, but never redesigned and applied those to the sub-recipient level” –ECR respondent Discussion The first objective of the FABRIC project, ensuring services reach OVC, was inextricably linked to the second objective of building the capacity of implementing agencies. The FABRIC project was as much an organization building initiative as a service delivery operation. While organization building is a slow process, results suggest efforts were successful in improving the capacity of implementing agencies. However, each IA was in part identified because it was believed capable of delivering services and therefore had a baseline capacity. SACBC had a strong infrastructure prior to FABRIC, with existing policies and procedures. ECR also had organizational documentation, but was less established than SACBC. Although PV in Namibia had only been operating for 14 months at the time of data collection, it also had fairly clear policies and procedures in place. The capacity building contribution of FABRIC, therefore, was heavily weighted toward strengthening the capacity of these organizations to manage and integrate the PEPFAR program. This discussion addresses the four goals of the organizational assessment component of the evaluation: 1. Determine if the organizational capacity of the FABRIC implementing agencies is stronger now than before receiving support Based on the TOCAT scores and supported by the interviews, there is a clear self-perception that the organizational capacity at all implementing agencies is stronger than before the FABRIC project. Measuring change over time in an organization with high staff turnover is a challenge, and a paucity of baseline data limits our ability to confidently remark on these perceived improvements. Regional FHI staff felt an organizational development specialist could have helped to more effectively deliver organizational capacity building services. 2. Determine if FABRIC Implementing Agencies have the capacity to deliver OVC support to sub-recipients Implementing Agencies were able to provide sub-recipients with organizational and program management support but limited technical programmatic guidance. This limited capacity was related both to the significant distance of the SR offices relative to the IAs, as well as limited ability of the SRs to assimilate new information due to low staff numbers and relatively simple baseline administrative systems. Furthermore, capacity strengthening focused more on organizational management and administrative capacity development, and less on OVC care technical support. FABRIC End-of-Project Evaluation Final Report 52 “Positive Vibes has been very helpful. They are monitoring us in terms of supporting, guiding in proposal writing and funding. They create a positive way for us by the guidance they give on day to day running of the organization…” –PV Partner “We have been taught how to deal with donor funds reports, and what is expected of us so that we perform. We have been taught planning and accountability - we have to budget, plan, and eventually come up with a report after you have done the budgeting and the service delivery.” –ECR respondent “The other positive things are that our capacity has been built. Now because of their coming in they have helped us to put things in order, to create files and do thing in the right way and in order.” –ECR sub-recipient Both the qualitative and quantitative data suggest that the implementing agencies have the capacity to deliver OVC support to sub-recipients and are doing it to a degree. However, it is unclear if the full potential of these agencies is being realized. 3. Learn from the sub-recipient leadership what FABRIC-specific inputs they associate with improved organizational performance In South Africa, the most highly valued supports reported by the SR leadership included basic assistance with filing papers, documenting events, report writing, and financial accountability. In Zambia, sub-recipients associated financial management support, counseling skills, program monitoring, and proposal writing with improved organizational performance. In Namibia, financial support was the input most associated with improved performance, but partner organizations also noted the different trainings were valuable. 4. Identify successes and challenges faced by FABRIC in implementing organizational capacity strengthening activities. Most of the challenges faced by FABRIC as they attempted to improve organizational capacity can be attributed to deficits in two domains, namely baseline data and human resources. The lack of baseline data and an evaluation framework from project inception is of particular consequence. As alluded to above, it is impossible to accurately measure changes over time without baseline data. This deficiency made both program monitoring and quality assurance a consistent challenge; projects simply aimed to reach the maximum number of children possible without having any clear outcomes, impact, or quality measures by which they could assess progress. Insufficient human resources were another area of concern for FABRIC staff as they attempted to strengthen organizational capacity. Many of the IA and SR relied heavily on volunteers who received little or no remuneration for their services. Targets were thus perceived as unrealistic by most IAs and SRs. FABRIC End-of-Project Evaluation Final Report 53 “They (SACBC) are very demanding and I personally fear that (eeeh), we are trying the best with the little staff that we have got. At the moment I don't think we have the capacity to meet their expectations but we are doing the best we can.” –SACBC sub￾recipient –SACBC sub-recipien High attrition rates among staff were another human resource-related challenge. FHI proposed that t implementing agencies offer incentives to help offset these turnover rates at the IA level, but neither SACBC nor ECR management was willing to do so, as this approach conflicted with pre-existing institutional policies. SACBC did offer small incentives at the SR level. Staff attrition negatively impacted organizational capacity and has important implications in terms of long-term program sustainability. Conclusion It is valuable to note that all three IA perceived that their organizational management skills improved through the FABRIC project, and, according to monitoring data outlined in annual reports, FABRIC consistently met or exceeded PEPFAR defined project targets of training caregivers and enrolling OVC. However, the quality of the technical services provided, and the reach of service delivery may have been limited by both financial resources and the staff available to provide services. Both the wellbeing assessment as well as the organizational assessment component highlight that the IAs began the FABRIC project with different levels of organizational capacity and systems for service delivery, calling for individualized capacity-strengthening approaches. Across all countries, however, additional resources and efforts must be invested at the sub-recipient level, not just the IA level, to ensure they are adequately equipped to deliver services in the community. Given the PEPFAR II policy of country ownership, it is encouraging that all three implementing agencies perceived themselves to be capacitated and well equipped to manage OVC projects. Whether their performance in the coming years is commensurate with those perceptions remains to be seen and is an area for ongoing monitoring and evaluation. Agreed upon strategies to maximize staff retention and minimize caregiver burnout at all program levels may further strengthen PEPFAR II programs. "There was so much demand from the caregivers who were not paid anything. Should have at least paid someone in charge of reports/accounts. It seemed like a bit of abuse of caregivers because they had to do so much. There were unfair and unrealistic expectations on the caregivers" –ECR Respondent FABRIC End-of-Project Evaluation Final Report 54 FABRIC End-of-Project Evaluation Final Report 55 C O S T I N G O F O V C S E R V I C E D E L I V E R Y: S O U T H A F R I C A A N D Z A M B I A FABRIC End-of-Project Evaluation Final Report 56 FABRIC End-of-Project Evaluation Final Report 57 COSTING OF OVC SERVICE DELIVERY: SOUTH AFRICA, ZAMBIA Overview3 The objective of the costing component of the EoPE was to estimate the full cost of inputs (goods and services) used to implement the FABRIC program at the level of FABRIC sub-recipient partner FBOs in Zambia and South Africa for 2009. A summary of the costing analysis is provided here. Complete costing reports will be available shortly as separate documents. Study Sites and Methods As with the overall EoPE, the three sites in Zambia were: Bethel Baptist – Samaritan Project in Kafue Evangel Oasis of Love Orphans Project in Chingola Mpatamatu Home-based Care Project in Luanshya The three sites in South Africa were: Sithand’izingane Care Project in Gauteng Province Siyathokoza OVC Project in Free State Province Bophelong in Northwest Province Approach. OVC programs or projects are similar to other types of projects, thus standard program costing methods were followed for this analysis. Data for the analysis were based on a review of program financial records, equipment inventories, and interviews with program staff. The purpose was to obtain information to estimate the full costs of goods and services used to implement the programs from the FBO’s perspective. The study did not attempt to capture the costs of ERC, FHI, USAID or others to manage the project above the FBO/community level. Types of costs. Costs are associated with different inputs used to implement a project. These include three basic categories: 1. Direct ‘financial’ expenses used to produce a service ($) for one year. These inputs are purchased and usually expended within the annual time period. 2. Annual equivalent ($) of the service provided by assets/equipment. This is the additional calculation needed to account for the multiple-year lifespan of equipment/assets. Annualizing asset purchases or donations acknowledges that services were provided during the evaluation year even if purchased in previous years. 3. Opportunity cost ($) of unpaid inputs (labor, donations). Because volunteer time and donations are not financial expenses incurred by the organization, one might assume that they were willing to contribute (for their own reasons), so any program payments to volunteers covered the opportunity cost of time 3 This section is a summary of the preliminary results from a full costing report prepared by Petan Hamazakaza for Zambia sites and Sarah Laurence in South Africa, with assistance from Jill Costello and Bruce Larson. The full costing report will be available shortly. FABRIC End-of-Project Evaluation Final Report 58 (otherwise they would not have volunteered). However, if this program were to be scaled up elsewhere, such goods and services may bear financial costs. Thus we include imputed costs for unpaid inputs based on local market rates that would be required to hire or purchase such inputs. Data collection. Data collection at each site therefore aimed to collect information related to all three types of costs. As noted above, this process comprised of interviews with staff coordinating the respective projects in the study sites and review of financial reports and project implementation documents. Where documented information was missing, reasonable estimates based on available information were used (details provided later in the report). In sum, data were collected through the following steps that mirror the types of costs listed above: Step 1. Review annual financial report and categorize expenses. Annual financial reports with itemized expenditures, which are produced by organizations generally for accounting, tax reporting, and/or donor reporting purposes, generally provide significant detail on program specific expenses. However, it is important to keep in mind that financial reports (and underlying electronic datasets) are organized for financial reporting, which is not the same as needed for understanding the structure of program costs. Step 2. Review equipment inventory and annualize equipment costs. We obtained the organization’s asset register (shows purchased equipment, buildings, purchase price, and purchase date). Such information is needed by an organization for insurance and inventory reasons, so organizations typically have it. Based on price, purchase date, expected working life, and a discount rate, we calculated the annual equivalent payment required to pay back a loan equal to the purchase price over the working with the discount rate. We then used consumer price index information for each country to inflate these annual values (at the time of purchase) to 2009 for the costing analysis. Step 3. Add imputed costs for volunteer time and other donations. This information was estimated through a review of program records (on numbers of volunteers) and interviews with program staff and volunteers (on level of effort weekly or monthly). Based on this review, we identified categories of people who contributed time to the project (paid and unpaid) and estimated quantity of time (in days) contributed to project. For the Zambian sites, we used the Zambian “Gazetted Government Cost of rural casual labor per day”, ZMK10,000, as the reasonable ‘price’ per day of volunteer services for the Zambia sites. For the South African sites, reasonable daily wages based on other types of labor paid by the site were used. Finally, we multiplied annual days, the daily wage, and the number of volunteers for each type of volunteer to calculate the total imputed personnel cost of volunteers. This provides an estimate of what it would cost to implement the project in the absence of volunteer labor. For other donations (in goods), the value of the item was estimated by program staff based on current market value. Step 4. Add any other contributions from organization not included in program financial report. FABRIC End-of-Project Evaluation Final Report 59 In general, this would include costs to the organization that are not included in the program financial report. For example, if an organization runs multiple programs (from multiple funders), with a shared Director not accounted for in the program budget, a portion of the Director’s salary would be added at this stage. For the FBOs included in the costing analysis, all organization costs were already included in the financial reports and were associated with the FABRIC program. Sorting and classification of costs. Based on a review of the data, a ‘costing profile’ was developed for each country with a level of detail that would be useful for program management and planning. For the Zambia sites, all costs were categorized into one of the following cost categories: Personnel costs, including program administrative staff and caregivers (i.e. volunteers who have direct contact with OVC and their households); Office costs, which includes rent, utilities, water and telephone, office supplies and materials, transport, project related meetings, costs of organizational project meetings, and so on; Equipment, which provide services to the program over multiple years; Training costs for personnel involved in project implementation; and Direct expenses on key program areas, which for the SRs included in this costing analysis were divided into Education and Kids Clubs. The costing data for each site in Zambia were then compiled into a summary costing profile table. The basic structure of the Zambia cost profile table is provided below. Because of large changes in the number of OVC served by the Zambia sites between 2008 and 2009 and the relatively small size of the programs, cost information for 2009 and 2008 (inflated to 2009 values) are presented. Zambian FBOs Year 2008 Year 2009 OVC Project Cost Category Financial Imputed Total % of Total Costs Financial Imputed Total % of Total Costs 1 Personnel: FBO Program Staff Caregivers 2 Office Administration 3 Annualized Equipment/Asset Services 4 Project Staff Training 5 Direct Expenses on Education 6 Direct Expenses on Kids Clubs Total For the South Africa sites, costs were categorized into one of the following cost categories: FABRIC End-of-Project Evaluation Final Report 60 Labor costs, including program administrative staff and caregivers; Materials for specific program areas such as education, food and nutrition, shelter, health, and psychosocial support; Office Utilities; Equipment; Transport; Buildings (office space); and Training (project staff training). The costing data for each site in South Africa were then compiled into a summary costing profile table. The basic structure of the South Africa cost profile table is provided below. South Africa FBOs Resource Type Financial Costs 2009 Imputed Costs 2009 Total Costs 2009 Percent of Total Costs 2009 1 Labour 2 Materials 3 Utilities 4 Equipment 5 Transport/Vehicle 6 Building/Land 7 Training Total Note that the training activities in both countries were intended to build capacity of the SRs to implement more effectively the project activities during and beyond the FABRIC project. In this respect, expenditures on training are investment in human capital and could be annualized as with equipment. Summary Results and Discussion Zambian Sites The completed cost profile for each site is provided at the end of this section. The details of the analysis are provided in the full costing report. A few key points from completed cost profiles are highlighted here. 1. All three SRs rely on volunteer labor for program administration and implementation. The imputed value of volunteer time, based on a modest daily wage rate, increases the full program costs by about 25-40% for Mpatamatu, 50% for Samaritan, and 65-90% for Evangel depending on the year. 2. Office administration and related costs were a relatively small share of overall program costs for Evangel and Samaritan, and between 15-21% of full program costs for Mpatamatu. Given that the office administration category included transportation as well as all other office related costs, these shares are in line with information from other studies of OVC programs. 3. The annual value of equipment costs is a minor share of total costs. Each SR obtained bicycles for their caregiver volunteers to reduce this transportation time for program implementation in 2009. FABRIC End-of-Project Evaluation Final Report 61 4. Expenditures on project staff training in each year were a relatively large share of program costs. If such expenses are considered investments in human capital, the training would provide services to the project over more than one year. If there is substantial staff turnover, prior investments in training would not provide services to the project in the future. The full costing report will include an additional costing profile for each SR based on annualizing training costs. 5. Expenditures on education for OVC were a large share of overall program costs, while direct expenses on Kids Clubs were minor. Volunteer time was the key input for the Kids Clubs, which is included in the personnel line item under caregivers. As a final summary, the table below provides the reported number of OVC supported by each SR, the financial costs, the total cost (financial plus imputed), and the average cost per OVC (financial and total cost) in 2008 and 2009. Table 29. OVC Costing Outcomes in Three FABRIC Sites in Zambia 2008 and 2009 Samaritan Evangel Mpatamatu Total/Average # OVC 2008 308 308 550 1166 2008 Total Financial Expenses (ZMK) 60,442,915 55,103,024 111,787,206 227,333,145 2008 Average Financial Expense/Child (ZMK) 196,243 178,906 203,249 194,968 2008 Total Costs (ZMK) 90,866,998 92,159,347 141,710,064 324,736,409 2008 Average Total Cost/Child (ZMK) 295,023 299,219 257,655 278,505 2008 Average Total Cost/Child ($ US at 5000 ZMK/USD) 59.00 59.84 51.53 55.70 # OVC 2009 500 405 1050 1955 2009 Total Financial Expenses (ZMK) 71,137,116 48,228,820 98,399,439 217,765,375 2009 Average Financial Expense/Child (ZMK) 142,274 119,084 93,714 111,389 2009 Total Costs (ZMK) 105,817,116 90,468,820 137,279,439 333,565,375 2009 Average Total Cost/Child (ZMK) 211,634 223,380 130,742 170,622 2009 Average Total Cost/Child ($ US at 5000 ZMK/USD) 42.33 44.68 26.15 34.12 For 2008, the average financial cost per OVC across the three SRs was relatively similar, as was the total average costs. Across the three sites, the average financial cost was about ZMK 195,000 (close to $40 US annually), while the total average cost was ZMK 279,000 (about $56 US). FABRIC End-of-Project Evaluation Final Report 62 Because of the increase in numbers of OVC at each site between 2008 and 2009 without similar increases in financial resources, average financial and total costs per OVC in the programs fell over time. In 2009, average financial costs across the three SRs were between ZMK 142,274 to 93,714 (about $22 US annually as an average across the three sites). In 2009, average total costs were between ZMK 211,634 to 130,742 (about $26-44 US per child). We caution that this does not necessarily indicate the project became more efficient over time. We have no measures of quality of services received for 2008, and only the process and outcome measures presented earlier in this report for 2009. It is therefore impossible to draw any conclusions in terms of program efficiency or effectiveness from the cost per OVC data presented in Table 29. Additionally, we caution against drawing conclusions about relative program efficiency by comparing costs per OVC across sites, as there was no standard package of OVC support that would allow us to do so. South African Sites The completed cost profile for each South African site is also provided at the end of this summary. The details of the analysis are provided in the full costing report. A few key points from completed cost profiles are highlighted here. 1. The three programs varied substantially in terms of total financial expenses (rounded): R 2,448,000 million for Sithand’izingane; R 773,000 for Siyathokoza; and R 242,000 for Bophelong. 2. For all three sites, labor and materials for program implementation are the largest shares of financial and total program costs (68% for Sithand’izingane; 75% for Siyathokoza; and 70% for Bophelong). 3. Within the materials category, key sub-categories varied somewhat across the three sites. For Bophelong, the main sub-categories were food and nutrition, educational assistance, psychosocial support, and shelter (in descending order). For Siyathokoza, the main sub-categories were food and nutrition, educational assistance, medical costs, and psychosocial support activities. For Sithand’izingane, the main sub-categories were food and nutrition (about 77% of all materials) and educational assistance (about 11% of all materials). 4. Office administration, utilities, and annualized value of equipment/assets were a relatively small share of overall program costs. As a final summary, Table 30 provides the reported number of OVC supported by each South African SR, the financial costs, the total cost (financial plus imputed), and the average cost per OVC (financial and total cost) in 2009. Again, we caution against drawing conclusions about relative program efficiency by comparing costs per OVC across sites, as there was no standard package of OVC support that would allow us to do so. FABRIC End-of-Project Evaluation Final Report 63 Table 30. OVC Costing Outcomes in Three FABRIC Sites in South Africa in 2009 Sithand’izingane Siyathokoza Bophelong # OVC 2009 645 622 729 2009 Total Financial Expenses (Rand) 2,448,770 773,549 242,146 2009 Average Financial/Child ( Rand) 3,797 1,244 332 2009 Total Costs (Rand) 3,110,574 858,428 258,697 2009 Average Total Cost/Child (Rand) 4,823 1,380 355 2009 Average Total Cost/Child ($ US, 7.5 R/USD) 643.01 184.01 47.32 The three sites included in this analysis for South Africa involve a range of locations, program structures, and different cost profiles, which are reflected in the differing estimates of total program costs per OVC. Please refer to the site-specific costing reports for further explanation and discussion of these issues. FABRIC End-of-Project Evaluation Final Report 64 Table 31. Costing profile for Samaritan: Zambia YEAR 2008 YEAR 2009 OVC Project Cost Category Financial Imputed Total % of Total Program Implementation Costs Financial Imputed Total % of Total Program Implementation Costs 1. Personnel 0 30,424,083 30,424,083 33% 0 34,680,000 34,680,000 33% FBO Program Staff 0 8,527,165 8,527,165 9% 0 9,720,000 9,720,000 9% Caregivers 0 21,896,918 21,896,918 24% 0 24,960,000 24,960,000 24% 2. Office Administration 3,279,453 3,279,453 4% 7,591,721 7,591,721 7% 3. Equipment - Annualized service value 0 0 0% 5,220,025 5,220,025 5% 4. Project Staff Training 39,837,188 39,837,188 44% 28,178,370 28,178,370 27% 5. Direct Expenses on Education 17,326,274 17,326,274 19% 22,707,000 22,707,000 21% 6. Direct Expenses on Kids Clubs 0 0 0% 7,440,000 7,440,000 7% Total 60,442,915 30,424,083 90,866,998 100% 71,137,116 34,680,000 105,817,116 100% FABRIC End-of-Project Evaluation Final Report 65 Table 32. Costing Profile for Evangel Oasis of Love: Zambia YEAR 2008 YEAR 2009 OVC Project Cost Category Financial Imputed Total % of Total Program Implementation Costs Financial Imputed Total % of Total Program Implementation Costs 1. Personnel 0 37,056,323 37,056,323 40% 0 42,240,000 42,240,000 47% FBO Program Staff 0 4,210,946 4,210,946 5% 0 4,800,000 4,800,000 5% Caregivers 0 32,845,377 32,845,377 36% 0 37,440,000 37,440,000 41% 2. Office Administration 4,737,314 4,737,314 5% 5,400,000 5,400,000 6% 3. Equipment - Annualized service value 5,619,867 5,619,867 6% 8,997,530 8,997,530 10% 4. Project Staff Training 26,014,990 26,014,990 28% 17,584,990 17,584,990 19% 5. Direct Expenses on Education 18,498,886 18,498,886 20% 16,246,300 16,246,300 18% 6. Direct Expenses on Kids Clubs 231,967 231,967 0% 0 0 0% Total 55,103,024 37,056,323 92,159,347 100% 48,228,820 42,240,000 90,468,820 100% FABRIC End-of-Project Evaluation Final Report 66 Table 33. Costing Profile for Mpatamatu: Zambia YEAR 2008 YEAR 2009 OVC Project Cost Category Financial Imputed Total % of Total Program Implementation Costs Financial Imputed Total % of Total Program Implementation Costs 1. Personnel 0 29,922,858 29,922,858 21% 0 38,880,000 38,880,000 28% FBO Program Staff 0 7,757,778 7,757,778 5% 0 10,080,000 10,080,000 7% Caregivers 0 22,165,080 22,165,080 16% 0 28,800,000 28,800,000 21% 2. Office Administration 21,703,308 21,703,308 15% 28,200,000 28,200,000 21% 3. Equipment - Annualized service value 3,180,351 3,180,351 2% 7,172,134 7,172,134 5% 4. Project Staff Training 37,322,670 37,322,670 26% 25,822,305 25,822,305 19% 5. Direct Expenses on Education 41,409,966 41,409,966 29% 37,205,000 37,205,000 27% 6. Direct Expenses on Kids Clubs 8,170,911 8,170,911 6% 0 0 0% Total 111,787,206 29,922,858 141,710,064 100% 98,399,439 38,880,000 137,279,439 100% FABRIC End-of-Project Evaluation Final Report 67 Table 35. Costing Profile for Sithand’izingane: South Africa Resource Type Financial Costs 2009 Imputed Costs 2009 Total Costs 2009 Percent of Total Costs 2009 1 Labor R 713,184.00 R 216,000.00 R 929,184.00 30% 2 Materials R 1,196,841.00 R 0.00 R 1,196,841.00 38% 3 Utilities R 156,763.00 R 0.00 R 156,763.00 5% 4 Equipment R64,523.94 R76,540.98 R141,064.92 5% 5 Transport/Vehicle R 275,714.14 R 0.00 R 275,714.14 9% 6 Building/Land R9,621.00 R 369,263.02 R 378,884.02 12% 7 Training R 32,123.00 R 0.00 R 32,123.00 1% Total R 2,448,770.08 R 661,804.00 R 3,110,574.08 100% Table 34. Costing Profile for Siyathokoza: South Africa Resource Type Financial Costs 2009 Imputed Costs 2009 Total Costs 2009 Percent of Total Costs 2009 1 Labor R 225,960.00 R 15,000.00 R 240,960.00 28.1% 2 Materials R 394,337.81 R 0.00 R 394,337.81 46.0% 3 Utilities R 44,685.67 R 0.00 R 44,685.67 5.2% 4 Equipment R 14,044.85 R 18,595.41 R 32,640.26 3.8% 5 Transport/Vehicle R 45,636.99 R 0.00 R 45,636.99 5.3% 6 Building/Land R 22,145.67 R 49,283.92 R 71,429.59 8.3% 7 Training R 26,738.39 R 0.00 R 26,738.39 3.1% Total R 773,549.38 R 82,879.34 R 856,428.72 100.00% FABRIC End-of-Project Evaluation Final Report 68 Table 36. Costing Profile for Bophelong: South Africa Resource Type Financial Costs 2009 Imputed Costs 2009 Total Costs 2009 Percent of Total Costs 2009 1 Labor R 77,280.00 R 0.00 R 77,280.00 29.87% 2 Materials R 102,639.68 R 0.00 R 102,639.68 39.68% 3 Utilities R 12,763.54 R 0.00 R 12,763.54 4.93% 4 Equipment R 8,412.30 R 14,151.05 R 22,563.35 8.72% 5 Transport/Vehicle R 34,683.08 R 0.00 R 34,683.08 13.41% 6 Building/Land R 0.00 R 2,400.00 R 2,400.00 0.93% 7 Training R 6,367.50 R 0.00 R 6,367.50 2.46% Total R 242,146.10 R 16,551.05 R 258,697.15 100.00% FABRIC End-of-Project Evaluation Final Report 69 S U M M A R Y C O N C L U S I O N S A N D S P E C I F I C R E C O M M E N D A T I O N S FABRIC End-of-Project Evaluation Final Report 70 FABRIC End-of-Project Evaluation Final Report 71 SUMMARY CONCLUSIONS AND SPECIFIC RECOMMENDATIONS Synthesizing and summarizing the large and complex FABRIC experience across the three countries is difficult. The context for OVC programming differs markedly across the three countries. The household resource levels, the capacities of the implementing agencies (PV, ECR, and SACBC), and the government inputs into social safety nets across Namibia, South Africa, and Zambia are all very different. As noted earlier, the limitation of post-hoc cross-sectional evaluations are well known and well described. The lack of baseline information and the relative inattention to an evaluation framework by PEPFAR and its Track 1 implementing partners during the Emergency Phase of the USG PEPFAR global response constrains our ability to identify beneficial impacts if they exist. Even when we were able to identify equality or improvement compared to the community comparison children, we are unable to definitively attribute any causative association of these benefits to the FABRIC project inputs. We have organized our closing remarks into summary conclusions and specific recommendations: Summary Conclusions The FABRIC beneficiaries appear to be vulnerable children. Though the specific targeting approaches were not well documented and are unknown to the current FHI project staff, every indication we could measure leads us to conclude the beneficiaries are children in need and that the targeting was successful. However, the levels of vulnerability in the community comparison populations are high as well, as evidenced by high orphan rates, high rates of guardian unemployment (particularly in SA), and relatively high rates of guardian chronic illness. Although we saw a very small proportion of acute malnutrition in children under-five in both the FABRIC and community samples, we still see high rates of food insecurity, high proportions of children without a blanket and/or sleeping in a place that gets wet in rain, and significant rates of chronic illness among children and their guardians. Given the high rates of vulnerability in these communities, it is harder for the evaluation to show statistically significant differences between the FABRIC and community samples. If we make the assumption (which we believe is reasonable, though not provable) that the initial FABRIC beneficiaries were children from households in greatest need, than the lack of difference in many of the wellbeing domains could be interpreted as the FABRIC project interventions bringing the served population up to, or maintaining them at, the levels of need and vulnerability of the general population. Though the needs are still profound, the lack of disparity between the two populations can be seen as a proxy measure of success for the FABRIC efforts. Related to the high levels of needs discussed above, our evaluation reports relatively low rates of service uptake reported by FABRIC beneficiaries in all age groups, particularly in Zambia. Some of this is related to our use of a twelve-month recall period for many services which we used to minimize recall bias, but a larger component of this low and limited intensity coverage is a result of financial constraints, variable annual budgets that made planning extremely difficult, and the dependence on volunteer labor. It is also related to the random sample selected; we chose from all FABRIC communities. FHI staff believed that the monitoring data reported in the Annual Reports was collected more regularly and intensively in the easier to access areas and therefore the routine monitoring data reports higher coverage rates than the estimates generated from our random sample that had greater geographic diversity. It is impossible to estimate what effect, if any, linking annual budget allocations to the number of beneficiaries served had on the data quality of the routine reporting system. Organizational capacity strengthening with respect to administration and M&E, reported by the sub￾recipients (partner organization in Namibia) interviewed, is undoubtedly beneficial, but further efforts FABRIC End-of-Project Evaluation Final Report 72 are needed to strengthen the technical capacities of the organizations to implement effective OVC services. In addition, the push to maximize the number of beneficiaries reached may have compromised the quality of services that could be provided to beneficiaries. The dependence on unpaid volunteer labor as the key labor force for provision of services is problematic. Though faith-based communities have long had a tradition of providing spiritual, social, and financial support for the sick, the increasing scale of the global pandemic challenges the faith-based organizations to continue those traditions. Attrition and turnover rates were high at every level of the FABRIC operation. This has profound effects on the quality of services and limits the impacts of the training and other capacity strengthening investments. The small but significant number of community sample children reporting receipt of services from the FABRIC project warrants further exploration. This result may be related to: incomplete project rosters (if rosters were not up to date, some of the community sample children may have, in fact, been FABRIC beneficiaries who received services through the program), relatively unfocused FABRIC programming, with services provided rather generally to the community instead of to a specific roster of FABRIC beneficiaries, or a lack of understanding of the origin of services received. Given the levels of need in the community sample populations it is hard to argue this spillover is bad, but it does make documenting the value and impacts of the FABRIC interventions among the beneficiary populations all that more difficult to accomplish. Country-specific (and possibly even sub-national) programming and evaluation is essential. The contexts are very different, and results of service uptake/receipt section suggest that the FABRIC approach may have been more effective in South Africa, where there were greater state investments in vulnerable households through national welfare grant programs, where administrative support within service delivery organizations was better. South Africa’s larger economy and wealthier, more active civil society seemed to offer greater opportunities to work in partnership with other local, national, and international organizations to leverage support. Overall, these factors may have resulted in higher rates of effective service delivery than that seen in Zambia. The Namibia program changed directions mid-project and therefore is difficult to fully incorporate into this evaluation. The original IA (CAFO) was similar in structure and basic approach to the IAs in South Africa and Zambia. With CAFO’s graduation to the New Partners Initiative and the selection of a smaller and differently programmatically focused NGO (PV), FABRIC was running essentially a new activity starting late in the FABRIC project cycle. PV’s reach was small in numbers; its programming was intensive among the small numbers, and the outcomes were different from the identified outcomes for South Africa and Zambia. Given the small numbers, short length of time, and high-intensity, high specificity programming of PV, we are cautious to draw many conclusions from the organization’s experience as measured in the EoPE. The FABRIC project was mandated to provide comprehensive OVC services across the full spectrum of the PEPFAR 6 +1 OVC service domains. Neither financial nor human resources were sufficient to operate in all the domains. FABRIC could have chosen to narrow its focus in order to do one or two things well instead of attempting to reach a large number of children across multiple areas of service provision with a limited resource base. Unrealistic programmatic expectations at all levels may have limited the quality and effectiveness of the program. This is an ongoing challenge within OVC programming. FABRIC End-of-Project Evaluation Final Report 73 Specific Recommendations Based on the results of all components of the FABRIC End-of-Project Evaluation, we make the following six programmatic recommendations: 1. Invest in developing an appropriate, rigorous, and independent evaluation strategy that is separate but complementary to the routine program management and monitoring activities as early in the project cycle as possible The Emergency Phase of the USG global response has ended. In PEPFAR II far greater emphasis is reportedly being put on having evidence drive important programmatic decisions. This requires using independent, scientifically valid impact evaluations in order to inform donor officials, policymakers, and program personnel about the effectiveness of OVC programs. Having well-defined specific objectives to accomplish clear program goals is crucial. Once these goals and objectives are agreed upon, we should use a narrower set of outcome measures that utilize common and consistent definitions. Standardized process, outcome and costing indicators will ensure we better understand where programming efforts are successful and where they need improvement to be more effective and efficient. These steps are essential for useful comparative evaluation work. FABRIC was not mandated to build a rigorous evaluation framework into their efforts and invested much of their technical and capacity strengthening efforts to attempts to improve basic management tools and routine monitoring systems rather than evaluation methods and approaches. 2. Collect high quality cost data from multiple perspectives In order to better understand program costs for planning and budgeting purposes, financial and economic cost information from multiple perspectives (consumer, provider, institutional, donor, societal) should be collected systematically. With agreed upon outcome measures from above, and the more widespread use of standard costing approaches, some initial cost-effectiveness calculations can be attempted to begin to build an evidence base for making programmatic decisions on the more cost- effective intervention mixes. 3. Target populations in need carefully and determine who will serve as comparison populations Invest at the beginning of the project to create robust, geo-located roster systems with unique identifiers as an essential first step to fostering better evaluation data. Using randomized evaluation designs, where phased rollout is expected, and using longitudinal evaluation approaches with carefully constructed comparison populations, would make a major difference in the quality of the evaluation work available. Identify, and follow, comparison populations through the life of the project if possible. 4. Institute context-, age-, and gender-specific programming FABRIC’s experiences in Namibia, South Africa, and Zambia differed markedly. Even when the approaches were similar (South Africa and Zambia) the underlying economic base and socio-political environment led to major differences in needs (i.e. birth registration support needed in Zambia was not required in South Africa), program priorities, and outcomes. The disaggregation of the FABRIC results by age group and gender brought forth important results that would have been hidden in a grouped analysis. 5. Substantially increase the institutional and individual capacity strengthening efforts The strong rhetoric on country-ownership and sustainability in PEPFAR II ultimately succeeds or fails largely upon the human and institutional and human capital available for these national efforts. The FABRIC evaluation highlighted the major organizational development needs at the sub-recipient levels even when the Implementing Agency (SACBC and ECR) were relatively strong. The very basic core organizational functions of FABRIC End-of-Project Evaluation Final Report 74 program documentation and monitoring and basic levels of financial control accounting were often deeply lacking. 6. Decrease the heavy reliance on unpaid volunteer labor for service provision Though faith-based communities have long had a tradition of providing spiritual, social, and financial support for the sick and the poor, the increasing scale of the global pandemic challenges the faith-based organizations to continue those traditions. Attrition rates among the volunteers, at every level of the Implementing Agency operations, were high and created program discontinuities, quality of care issues, and generated a need for regular retraining of a new labor force. It is critical to recognize the additional labor provided by volunteers as a social service with opportunity costs for the worker rather than a assume it is merely a spiritual duty for a member of a faith-based community. Figuring out whether, and if so, how to compensate the community workers assuming responsibility for OVC care is a key social issue. Solutions to this tough issue and the development of realistic expectations of workload must be explored before the need exceeds the capabilities of individual volunteers or faith-based organizations overall. FABRIC End-of-Project Evaluation Final Report 75 REFERENCES 1. WHO Ua. 09 AIDS epidemic update. Geneva; 2009 November 2009. Report No.: UNAIDS/09.36E / JC1700E”. 2. (UNAIDS) JUNPoHA. 08 Report on the Global AIDS Epidemic - Executive Summary. Geneva; 2008 July 2008. 3. Tembo Ma. Final TOCAT Report: ECR. Lusaka: Expanded Church Response. 4. International FH. FABRIC Annual Report. Lusaka: Family Health International; 2009. 5. Swindale A BP. Household Dietary Diversity Score (HDDS) for Measurement of Household Food Access: Indicator Guide. Washington DC: Food and Nutrition Technical Assistance Project (FANTA); 2006 September 2006. 6. Steyn NP NJ, Nantel G, Kennedy G, Labadarios D. Food variety and dietary diversity scores in children: are they good indicators of dietary adequacy? . Public Health Nutrition 2006;9:644-50. 7. Kennedy G, Pedro MR, Seghieri C, Nantel G, Brouwer I. Dietary diversity score is a useful indicator of micronutrient intake in non-breast-feeding Filipino children. Journal of Nutrition 2007;137:472-7. 8. Central Statistical Office (CSO) MoHM, Tropical Diseases Research Centre (TDRC),, University of Zambia aMII. Zambia Demographic and Health Survey 2007: Final Report. Calverton, Maryland, USA: CSO and Macro International Inc. ; 2009. FABRIC End-of-Project Evaluation Final Report 76 APPENDICIES I. Maps of FABRIC operational areas II. List of Indicators III. SDQ Scoring Guidelines IV. Instruments available upon request: please email nscott@bu.edu FABRIC End-of-Project Evaluation Final Report 77 I. Maps of FABRIC operational Areas FABRIC End-of-Project Evaluation Final Report 78 FABRIC End-of-Project Evaluation Final Report 79 II. List of Indicators I N D I C A T O R F O R M S S O U R C E (Q U E S T I O N) D E N O M I N A T O R Demographics Mean age of children A A3,A4 FABRIC and Community sample children Number/% of male and female children A A5 FABRIC and Community sample children Number/% of confirmed and presumed paternal orphans B, C, D C10_1 FABRIC and Community sample children *The report also includes the number/% of (paternal, maternal, and double confirmed and presumed orphans, where the denominator is the total number of confirmed and presumed orphans. Number/% of confirmed and presumed maternal orphans B, C, D C7_1 FABRIC and Community sample children; presumed orphans include responses of “don’t know” to mother or father alive. *The report also includes the number/% of (paternal, maternal, and double confirmed and presumed orphans, where the denominator is the total number of confirmed and presumed orphans. Number/% of confirmed and presumed double orphans B,C,D C7_1 and C10_1 FABRIC and Community sample children; presumed orphans include responses of “don’t know” to mother or father alive. *The report also includes the number/% of (paternal, maternal, and double confirmed and presumed orphans, where the denominator is the total number of confirmed and presumed orphans. Number/% of Primary Guardian relationship to child(Mother, Father, Grandmother, Other relative, Non-relative) B,C,D D4 Guardians of FABRIC and Community sample children FABRIC End-of-Project Evaluation Final Report 80 Number/% of male and female guardians B,C,D B4_1 Guardians of FABRIC and Community sample children Mean age of guardians (yrs) B,C,D D3 Guardians of FABRIC and Community sample children Number/% of guardians > 60 years of age B,C,D D3 Guardians of FABRIC and Community sample children Number/% of guardians < 18 years of age B,C,D D3 Guardians of FABRIC and Community sample children Number/% of guardians employed in: Formal Sector Informal sector Domestic/Home Unemployed B,C,D B8_1 Responses 2,4 Responses 3,5 Responses 6,8,9,10 Response 7 Guardians of FABRIC and Community sample children Number/% of guardians with a chronic illness B,C,D D6 Guardians of FABRIC and Community sample children Mean number of adults living in the household B,C,D B4_1-B4_8 FABRIC and Community sample children *Number of sample children serves as proxy for number of households Mean number of children living in the household B,C,D C4_1-C4_8 FABRIC and Community sample children *Number of sample children serves as proxy for number of households Median number of children living in the household B,C,D C4_1-C4_8 FABRIC and Community sample children *Number of sample children serves as proxy for number of households Education Number/% of children whose household received any support from FABRIC to attend school in the preceding 12 months. C, D G11 FABRIC and Community sample children Frequencies: Supports received from FABRIC to attend school C, D G10 FABRIC sample children whose households received support from FABRIC to attend school in the preceding 12 FABRIC End-of-Project Evaluation Final Report 81 months Number/% of school-aged children registered for school (5 years of age in South Africa, 7 years of age in Zambia) C, D G1 School-aged children in FABRIC and Community samples Frequencies: Reasons for not registering in school C, D G2 School-aged children in FABRIC and Community samples not registered for school Number/% of children registered in school reporting > 2 days missed from school in the preceding 2 weeks C, D G5 FABRIC and Community sample children registered for school Frequencies: Reasons for missed school C, D G6 School-aged children registered for school who reported missing 2 or more days of school in the preceding two weeks Number/% of children registered in school reporting average or above average school performance C, D G4 FABRIC and Community sample children of school-age registered for school Psychosocial Status Number/% of children < 5 years of age having received psychosocial services from the FABRIC program in the preceding 3 months B L17 FABRIC sample children < 5 years of age Number/% of children ≥ 5 years of age having participated in extracurricular or afterschool activities by the FABRIC program in the preceding 12 months C, D X2 FABRIC sample children > 5 years of age SDQ: Emotional symptoms score C, D K3, K8, K13, K16, K24 FABRIC and Community sample children > 5 years of age SDQ: Conduct problems score C, D K5, K7, K12, K18, K22 FABRIC and Community sample children > 5 years of age SDQ: Hyperactivity/inattention C, D K2, K10, K15, K21, FABRIC and Community FABRIC End-of-Project Evaluation Final Report 82 score K25 sample children > 5 years of age SDQ: Peer relationship problems score C, D K6, K11, K14, K19, K23 FABRIC and Community sample children > 5 years of age SDQ: Prosocial behavior score C, D K1, K4, K9, K17, K20 FABRIC and Community sample children > 5 years of age SDQ: Total difficulties score (Emotional symptoms, Conduct problems, Hyperactivity, Peer relationship problems) C, D K3, K8, K13, K16, K24, K5, K7, K12, K18, K22, K2, K10, K15, K21, K25, K6, K11, K14, K19 FABRIC and Community sample children > 5 years of age Number/% of guardians of children <5 years of age having concerns about their child’s learning B O1, O2, O3, O4, O7, O8, O9 FABRIC and Community sample children < 5 years of age Number/% of guardians of children < 5 years of age having concerns about their child’s behavior B O5, O6 FABRIC and Community sample children < 5 years of age Mean child development index scores: Interpersonal development Gross motor development Fine motor development Receptive language development Expressive language development B Sev1-Sev3, One1- One3, Two1- Two3,Three1-Three3, Four1-Four3 Sev4-Sev6, One4- One6, Two4-Two6, Three4-Three6, Four4- Four5 Sev7-Sev9,One7- One9, Two7-Two9, Three7-Three9, Four6- Four7 Sev10-Sev14, One10- One12, Two10- Two11, Three10- Three12, Four8- Four10 Sev11-Sev14, One13- One15, Two12- Two14, Three13- Three16, Four11- Four13 Mean of FABRIC sample children > 7 months of age and < 5 years relative to Community sample children > 7 months of age and < 5 years FABRIC End-of-Project Evaluation Final Report 83 Self-help development Sev15-Sev17, One16- One18, Two15- Two17, Three17- Three19, Four14- Four16 Food and Nutrition Number/% of households having received information about preparation of health foods from the FABRIC program in the preceding 12 months B, C, D I 3 FABRIC and community sample children Number/% of households having received at least one food package from the FABRIC program in the preceding 12 months B, C, D I 8 FABRIC and community sample children % of children > 6 months of age with a dietary diversity score of >4, reflecting intake of foods from at least four of eight food categories in the preceding 24 hours B, C, D H1 – H15 FABRIC and Community sample children > 6 months of age Mean/median dietary diversity score in children >6 months of age B, C, D H1 – H15 FABRIC and Community sample children > 6 months of age Frequencies: Dietary diversity scores in children > 6 months of age B, C, D H1 – H15 FABRIC and Community sample children > 6 months of age Number/% of children reported as going without food for a whole day and night in the preceding 4 weeks B, C, D I 11 FABRIC and community sample children Number/% of children 6 months to 60 months of age with severe malnutrition as assessed by mid￾upper-arm circumference B N1 FABRIC and Community sample children 6 to 60 months of age Health Number/% of children referred to a doctor or nurse for healthcare in the preceding 12 B, C, D F6 FABRIC and community sample children FABRIC End-of-Project Evaluation Final Report 84 months Number/% of children provided with assistance by the FABRIC program to access health care for which the household did not have to pay in the preceding 12 months. B, C, D F8 FABRIC sample children Frequencies: Supports provided by a community group or organization to access care B, C, D F7 Children who accessed health care through assistance offered by FABRIC in the preceding 12 months Number/% of children with reported presence of a chronic illness B, C, D F1 FABRIC and community sample children Frequencies: Chronic illnesses reported B, C, D F2 Children with reported presence of a chronic illness % of children seeking healthcare for their worst illness at a formal health facility during the preceding 12 months B, C, D F3, F4 FABRIC and community sample children Frequencies: Establishments at which healthcare was sought for illness during the preceding 12 months B, C, D F4 FABRIC sample children Frequencies: Reasons for not seeking healthcare for the worst illness during preceding 12 months B, C, D F5 FABRIC sample children ZAMBIA ONLY: Number/% of children who slept under a mosquito net on the night preceding the survey B, C, D F10 FABRIC and community sample children Shelter and Care Number/% of households receiving assistance for home maintenance from the FABRIC program in the preceding 12 months B, C, D E1 FABRIC and community sample children FABRIC End-of-Project Evaluation Final Report 85 Number/% of households receiving household supplies from the FABRIC program in the preceding 12 months B, C, D E5 FABRIC and community sample children Number/% of children sleeping in a place that gets wet in rain B, C, D E12 FABRIC and community sample children Number/% of children having a blanket to sleep under B, C, D E13 FABRIC and community sample children Number/% of children sleeping in a place that gets wet with rain and do not have a blanket B,C,D E12,E13 FABRIC and Community sample children Number/% of households having received assistance from the FABRIC program to place a child with a foster parent in the preceding 12 months B, C, D M3 FABRIC and community sample children Number/% of households having received assistance from FABRIC program with adoptions services in the preceding 12 months B, C, D M5, M6 FABRIC sample and community sample children Child Protection and Legal Aid Number/% of guardians having received assistance from the FABRIC program to register birth or obtain birth certificate or identification card B, C, D J6 FABRIC and Community sample children Number/% of guardians reporting that their child has a birth certificate (received or pending) B, C, D J1 FABRIC and Community sample children Number/% of guardians of children without birth certificates reporting knowledge of how to register birth B, C, D J1, J2 FABRIC and Community sample children without birth certificates SOUTH AFRICA: Number/% of children having a national ID card B, C, D J4 FABRIC and Community sample children in South Africa FABRIC End-of-Project Evaluation Final Report 86 Frequencies: Reported reasons for not registering birth of those without birth registration B, C, D J3 FABRIC and Community sample children without birth registration Number/% receiving information on children’s rights from the FABRIC program B, C, D J9 FABRIC and Community sample children FABRIC End-of-Project Evaluation Final Report 87 III. SDQ Scoring Guidelines SCORING THE GUARDIAN-ADMINISTERED STRENGTHS AND DIFFICULTIES QUESTIONNAIRE (ADMINISTERED TO GUARDIANS OF 8-11 YEAR OLD CHILDREN) The 25 items in the SDQ comprise 5 scales of 5 items each. Somewhat True is always scored as 1, but the scoring of Not True and Certainly True varies with the item, as shown below. For each of the 5 scales the score can range from 0-10 if all 5 items were completed. Scale score can be prorated if at least 3 items were completed. QUESTION RESPONSE Not True | Somewhat True | Certainly True Emotional Symptoms Scale My child often complains of headaches, stomach-aches, or sickness. Is this… 0 1 2 My child has many worries or often seems worried. Is this… 0 1 2 My child is often unhappy, depressed, or tearful. Is this… 0 1 2 My child is nervous or clingy in new situations, easily loses confidence. Is this… 0 1 2 My child has many fears, easily scared. Is this… 0 1 2 Conduct Problems Scale My child often loses his/her temper. Is this… 0 1 2 My child is generally well behaved, usually does what adults request. Is this… 2 1 0 My child often fights with other children or bullies them. Is this… 0 1 2 My child often lies or cheats. Is this… 0 1 2 My child steals from home, school, or elsewhere. Is this… 0 1 2 Inattention/Hyperactivity Scale My child is restless, overactive, cannot stay still for long. Is this… 0 1 2 My child is constantly fidgeting or squirming. Is this… 0 1 2 My child is easily distracted, concentration wanders. Is this… 0 1 2 My child thinks things out before acting. Is this… 2 1 0 My child has a good attention span, sees work through to the end. Is this… 2 1 0 Peer Relationship Problems Scale My child is rather solitary, prefers to play alone. Is this… 0 1 2 My child has at least one good friend. Is this… 2 1 0 My child is generally liked by other children. Is this… 2 1 0 My child is picked on or bullied by other children. Is this… 0 1 2 My child gets along better with adults than with other children. Is this… 0 1 2 Prosocial Behavior Scale My child is considerate of other people’s feelings. Is this… 0 1 2 My child shares readily with other children, for example toys, treats, pencils. Is this… 0 1 2 My child is helpful if someone is hurt, upset or feeling ill. Is this… 0 1 2 My child is kind to other children. Is this… 0 1 2 My child often offers to help others (parents, teachers, other children). Is this… 0 1 2 FABRIC End-of-Project Evaluation Final Report 88 Total Difficulties Score The Total Difficulties score is generated by summing the scores from all the scales except the prosocial scale. The resultant score can range from 0 to 40 (and is counted as missing if one of the component scores is missing). Classifying Symptom Scores Normal Borderline Abnormal Total Difficulties Score 0-13 14-16 17-40 Emotional Symptoms Score 0-3 4 5-10 Conduct Problems Score 0-2 3 4-10 Hyperactivity Score 0-5 6 7-10 Peer Problems Score 0-2 3 4-10 Prosocial Behavior Score 6-10 5 0-4 SCORING THE SELF-ADMINISTERED STRENGTHS AND DIFFICULTIES QUESTIONNAIRE (ADMINISTERED TO 12-17 YEAR OLD CHILDREN) The 25 items in the SDQ comprise 5 scales of 5 items each. Somewhat True is always scored as 1, but the scoring of Not True and Certainly True varies with the item, as shown below. For each of the 5 scales the score can range from 0-10 if all 5 items were completed. Scale score can be prorated if at least 3 items were completed. QUESTION RESPONSE Not True | Somewhat True | Certainly True Emotional Symptoms Scale I get a lot of headaches, stomach-aches, or sickness. Is this… 0 1 2 I worry a lot. Is this… 0 1 2 I am often unhappy, depressed, or tearful. Is this… 0 1 2 I am nervous in new situations. I easily lose confidence. Is this… 0 1 2 I have many fears. I am easily scared. Is this… 0 1 2 Conduct Problems Scale I get very angry and often lose my temper. Is this… 0 1 2 I usually do as I am told. Is this… 2 1 0 I fight a lot. I can make other people do what I want. Is this… 0 1 2 I am often accused of lying or cheating. Is this… 0 1 2 I take things that are not mine from home, school, or elsewhere. Is this… 0 1 2 Inattention/Hyperactivity Scale I am restless; I cannot stay still for long. Is this… 0 1 2 I am constantly fidgeting or squirming. Is this… 0 1 2 I am easily distracted; I find it difficult to concentrate. Is this… 0 1 2 I think before I do things. Is this… 2 1 0 I finish the work I’m doing. My attention is good. Is this… 2 1 0 Peer Relationship Problems Scale I would rather be alone that with people of my age. Is this… 0 1 2 I have one good friend or more. Is this… 2 1 0 Other people of my age generally like me. Is this… 2 1 0 Other children or young people pick on me or bully me. Is this… 0 1 2 FABRIC End-of-Project Evaluation Final Report 89 I get along better with adults than with people my own age. Is this… 0 1 2 Prosocial Behavior Scale I try to be nice to other people. I care about their feelings. Is this… 0 1 2 I usually share with other children, for example CDs, games, food. Is this… 0 1 2 I am helpful if someone is hurt, upset or feeling ill. Is this… 0 1 2 My child is kind to younger children. Is this… 0 1 2 I often offer to help others (parents, teachers, other children). Is this… 0 1 2 Total Difficulties Score The Total Difficulties score is generated by summing the scores from all the scales except the prosocial scale. The resultant score can range from 0 to 40 (and is counted as missing if one of the component scores is missing). Classifying Symptom Scores Normal Borderline Abnormal Total Difficulties Score 0-15 16-19 20-40 Emotional Symptoms Score 0-5 6 7-10 Conduct Problems Score 0-3 4 5-10 Hyperactivity Score 0-5 6 7-10 Peer Problems Score 0-3 4-5 6-10 Prosocial Behavior Score 6-10 5 0-4 Center for Global Health and Development Boston University School of Public Health Boston, Massachusetts, USA Email: cihd@bu.edu Phone: +1 617-414-1260 Fax: +1 617-414-1261 Website: http://www.bu.edu/cghd