July 1, 2021 This publication was produced for review by the United States Agency for International Development. It was submitted by the USAID/Uganda OVC Evaluation Contract, 72061719C0000 prepared by SoCha,LLC YEAR 3 FINAL REPORT ORPHANS AND VULNERABLE CHILDREN (OVC) EVALUATION i OVC EVALUATION FINAL REPORT Submitted to: USAID/Uganda Prepared by: SoCha, LLC Disclaimer: The authors’ views expressed in this document do not necessarily reflect the views of the United States Agency for International Development or the United States Government. ii Acknowledgements We would like to thank USAID/Uganda, WEI/B and CRS Uganda for the generous time and support they offered during the course of the evaluation. The SoCha team would like to express our sincere gratitude to the USAID/Uganda staff, especially Kay Leherr, Fiona Waata, Nick Demeter and Joyce Wanican, who showed much appreciated support. Finally, we would also like to thank Sharon Kim and Mahjabeen Raza from New York University and Dr. Wolfgang Stuppert for their technical inputs and guidance. -SoCha, LLC iii Acronyms AIDS Acquired immunodeficiency syndrome ART Antiretroviral Therapy BN Benchmark BOCY Better Outcomes for Children and Youth CBSD Community Based Services Department CBT Community-Based Trainer CDO Community Development Officers CFLO Case Facility Linkage Officer CLHIV Children Living with HIV CMC Case Management Committee COP Country Operation Plan CRS Catholic Relief Services CSO Civil Society Organizations CWC Child Welfare Committee DCWC District Child Well-being Committee DID Difference in Difference DOVCC District Orphans and Vulnerable Children Committee DREAMS Determined, Resilient, Empowered, AIDS-Free, Mentored, and Safe DV Domestic Violence ECD Early Childhood Development ESEM Exploratory Structural Equation Modeling ET Evaluation Team FAR Federal Acquisition Requirements FDG Focus Group Discussions GBV Gender-Based Violence GoU Government of Uganda HAT Household Assessment Tool HES Household Economic Strengthening HH Household HIP Household Improvement Plans HIV Human Immunodeficiency Virus HVAT Household Vulnerability Assessment Tool HVPT Household Orphans and Vulnerable Children Vulnerability Prioritization Tool ICYD Integrated Child and Youth Development IE Impact Evaluation IP Implementing Partner KII Key Informant Interview LCA Latent Class Analysis M&E Monitoring and Evaluation MER Monitoring, Evaluation and Reporting MIS NGO Management Information System Non-Governmental Organization iv OVC Orphans and Vulnerable Children OVCMIS Orphans and Vulnerable Children Management Information System PE Performance Evaluation PEPFAR President's Emergency Plan for AIDS Relief PGA Pre-graduation Assessment PO Probation Officer PSS Psychosocial Support PSW Para Social Worker QCA Qualitative Comparative Analysis SAT Semi-Annual Assessment Tool SAGE Social Assistance Grants for Empowerment SILC Savings and Internal Lending Communities SoCha SoCha, LLC SOCY Sustainable Outcomes for Children and Youth SOVCC Subcounty Orphans and Vulnerable Children Committee SW Social Worker TCS Temporary Consumption Support USAID United States Agency for International Development VHT Village Health Team VL Viral Load VSLA Village Savings and Loan Association WASH Water, Sanitation and Hygiene WEI/B World Education, Inc/Bwantana v Table of Contents Acronyms iii Table of Contents v Executive Summary vii IE Question Findings viii PE Question 2 Findings viii PE Questions 3 and 4 Findings x PE Question 1 Findings x PE Question 5 Findings xi Recommendations xi 1. Introduction 1 OVC Program 1 OVC Evaluation Scope of Work 1 Evaluation Design 2 Methodology 3 Impact Evaluation Methodology 4 Statistical Methods Used to Address Questions 3 and 4 7 Cost Effectiveness Methodology 7 Limitations 13 Limitations to the Impact Evaluation 14 2. Is SOCY contributing to a reduction in household vulnerability? 16 Evaluating Impact through Difference-in-Differences Design 16 Closer Look at the Data 18 Correlation Analysis 18 Conclusion 20 Recommendations 21 3. How is the OVC Program working to support HHs? 22 OVC Program Graduation Benchmark Success 22 Benchmark 1: Known HIV Status 25 Benchmark 2: Virally Suppressed 26 Benchmark 3: Knowledgeable About HIV Prevention 28 Benchmark 4: Not Undernourished for Children 5 Years or Younger 30 Benchmark 5: Improved Financial Stability 31 Benchmark 6: No Violence Reported 33 vi Benchmark 7: Not in a child-Headed Household 37 Benchmark 8: Children in School 37 Conclusion 38 Recommendations 40 4. Graduation rates and benefits from the OVC Program 41 PE Question 3 Qualitative Discussion 41 PE Question 3 Quantitative Findings 41 BOCY Means Testing Results: Gender, Geography and HH size 41 SOCY Means Testing Results: Gender, Geography and HH size 42 BOCY Latent Class Analysis Results 42 SOCY Latent Class Analysis Results 44 Conclusion 45 Recommendations 46 5. What are the processes that facilitate long-term success? 47 HH Case Management: PSWs 47 CMCs 50 SOVCCs 52 DOVCCs 54 Conclusion 57 Recommendations 57 6. The cost-effectiveness of the OVC Program 58 OVC Program Unit Costs per Household 58 Cost Effective Adjustments 59 Conclusion 61 Recommendations 61 Annex 1: OVC Evaluation Scope of Work i Annex 2: Datasets Used in This Report iii Annex 3: SOCY Impact Evaluation Survey Questionnaire iv Annex 4: SOCY and BOCY Cost Models xxx Annex 5: Findings, Conclusions and Recommendations Summary Table xxxi Annex 6: Year 1 Evaluation Report Executive Summary xxxiv Annex 7: Year 2 Evaluation Report Executive Summary xlii vii Executive Summary USAID/Uganda’s Orphans and Vulnerable Children (OVC) Program sought to improve the well￾being of children and youth made vulnerable by HIV and other challenges in Uganda. It employed an integrated approach to address the factors of vulnerability. Two regional activities started in 2015 and covered 46 districts across Uganda. The Better Outcomes for Children and Youth (BOCY) and Sustainable Outcomes for Children and Youth (SOCY) programs aimed to economically empower children, youth, and their caregivers to access core services, strengthen systems to provide core services, and improve the coordination of community-based clinical and socio-economic services for efficiency and effectiveness along the continuum of care. BOCY ran across 22 districts in Eastern and Northern Uganda and SOCY operated in Western and Central Uganda in 24 districts. In each district, OVC Program partners were assisted by the District Orphans and Vulnerable Children Committees (DOVCCs) and their corresponding Subcounty OVC Committees (SOVCCs). This work was carried out in support of and in partnership with the Ugandan Ministry of Gender, Labor, and Social Development (MGSLD). The OVC Evaluation’s purpose is to provide rigorous evidence of the extent to which participation in USAID/Uganda’s OVC Program has reduced Household (HH) vulnerability and improved OVC well-being, and to identify factors or combinations of factors within and outside the program that may explain how results were achieved or what barriers may have prevented greater success. Future programming in Uganda will be informed by the results from this final report. The OVC Evaluation seeks to answer the following Performance and Impact Evaluation (PE/IE) questions: • IE Question: Does participation in SOCY activities contribute to a reduction in HH vulnerability and improve OVC well-being? • PE Question 1: To what extent are processes in place to facilitate the long-term success of program interventions beyond the program’s end? What factors may prevent long-term success? • PE Question 2: What coordination and management mechanisms have worked best to support the integrated OVC implementation approach? What enabling factors do these mechanisms engage with? • PE Question 3: Do HHs or individuals with specific characteristics (e.g., individuals with disabilities, single-parent HHs, regional variation, etc.) progress and graduate at different rates, and if so, why? • PE Question 4: To what extent do females and males experience different benefits from the program? Are there unintended results for either group? • PE Question 5: How cost-effective are different pathways to change? How do interventions/combinations of interventions compare in impact and cost? The OVC Evaluation Team (ET) addresses these questions in coordination with OVC program stakeholders, including USAID/Uganda, SOCY, BOCY and Ugandan government staff. Note that the IE Question builds off previous work under the USAID/Uganda Learning Contract, which involved initial baseline (2016) and extended baseline (2017) surveys. viii IE Question Findings The SOCY IE sought to confirm, through a refined Household Vulnerability Assessment Tool (HVAT), if participation in SOCY activities contributed to a reduction in HH vulnerability and improved OVC well-being. In doing so, the ET conducted four waves of annual survey data collection across 1,922 HHs, including an initial Wave 1 Baseline (FY2017Q1); a subsequent Wave 2 Extended Baseline to confirm both control and treatment HHs travelled parallel and comparable trends (FY2018Q1); a Wave 3 Midline (FY2019Q1); and a Wave 4 Endline (FY2020Q1). Analysis of the Wave 1 to Wave 2 trends confirmed the parallel trend lines assumption and the two groups were thus comparable. The ET unfortunately found no significant difference between control and treatment group HHs moving from Wave 2 to Wave 4. The lack of significance may have been due to potential contagion, contamination and implementation infidelity issues caused by shifting President's Emergency Plan for AIDS Relief (PEPFAR) priorities towards HIV suppression, or due to an overall lack of intervention potency. Further analysis also revealed that while there was no significant difference between control and treatment group HHs regarding the HVAT Assets and Child Health subdimensions, there were in fact significant differences between control and treatment group HHs in the Income Privileges and Child Protection subdimensions, i.e., treatment group HHs demonstrated positive improvements in the former even as they experienced declines in the latter, aka the “SOCY Paradox”. PE Question 2 Findings At the end of the OVC Program, both SOCY and BOCY reported impressive beneficiary outputs and outreach. BOCY reported close to 120,000 beneficiaries served (55 percent of which were female), claiming that 27,175 HHs received services and 20,535 were assessed for graduation. Similarly, SOCY reported close to 140,000 beneficiaries served, claiming that 38,531 HHs received services and 25,946 were assessed for graduation. However, despite these impressive numbers, both SOCY and BOCY averaged around a 41-42 percent graduation rate as per the current requirement to successfully pass all eight PEPFAR OVC graduation benchmarks (BNs). BOCY HHs were relatively successful at achieving BNs related to BN 1 (Known HIV Status), BN 6 (No Violence) and BN 7 (Not a Child-Headed HH) while they struggled to achieve BN 8 (Children in School). SOCY HHs performed relatively the same across BN 1, BN 2 (Viral Suppression), BN 3 (Knowledgeable about HIV Prevention), BN 6 and BN 7, while also struggling to reach BN 8. Counseling is the key to encouraging testing, ensuring adherence, reducing stigmas, and spreading further knowledge across both BOCY and SOCY. The importance of counseling ran across discussions of all three BNs, and oftentimes reflected carefully built trust originating with the Para Social Worker (PSW). Interestingly, PSWs were often seen as the crucial link between the HH and the facility, both in cases where OVC Program volunteers occupied a dual PSW and Village Health Team (VHT) role, and where they were representatives qualified to pass on relevant information to the VHT and health facility. Feedback from focus group discussions (FGDs) on SOCY and BOCY efforts to improve HH nutrition suggests that when considering the successful pathways to HH nourishment, backyard garden education appears to be a viable option. Yet, given that the achievement rate for this BN could benefit from improvement, follow-on interventions must remain mindful of the tendency for beneficiaries to prioritize the sale of produce or use it to primarily address adult and HIV positive ix health concerns over early child health concerns. This tendency should be addressed in future interventions. Both BOCY and SOCY HHs had difficulties achieving BN 5 (Financial Stability). On the positive side, FGD evidence and previous Qualitative Comparative Analysis (QCA) findings suggest that when considering the successful pathways to financial stability, OVC Program savings groups— both Savings and Internal Lending Communities (SILCs) and Village Savings and Loan Association (VLSAs)—can be viable options that improve financial literacy, promote longer-term planning, provide emergency response resources and otherwise cushion against vulnerability. Yet, a key aspect of this model involves investment in livestock to sell at a later date to pay for school fees, and this practice continues as both a sign of prosperity and poverty. As a result, PEPFAR’s BN 6 verification requirements can in fact confuse failure with success. Future interventions should consider more nuanced and/or culturally calibrated operational requirements for this BN, and proxies such as the sale of the last female in the stock may help further define meaningful differences. The ET FGD findings suggest that when considering the successful pathways to non-violence, a very complicated picture of success and struggle emerges. On the one hand, the OVC Program’s various parenting, household economic strengthening (HES) and psychosocial (PSS) counseling undoubtedly provided psychosocial benefits, improved family relations and increased child protection. On the other hand, collecting reliable data on sensitive topics such as domestic violence (DV) is not easy and is prone to bias. Similarly, some beneficiary training may have contributed to increased reporting of violence (when it would not have been reported previously) but the longer-term effects of positive behavior change had not yet occurred. Regarding the SOCY Paradox (which may apply to BOCY as well but cannot be confirmed without control group data), savings groups and other income improvements, while beneficial, may unintentionally contribute to more violence as alcohol consumption increases. Finally, males may respond to HES improvements, especially when led by females, with violence as a means of resisting female empowerment. PEPFAR’s prioritization of primary caregivers may further exacerbate these tendencies as men are systematically excluded from the benefits and jealously respond with violence. The implications are that all HES and female empowerment programming must seriously consider violent reactions to improved incomes and women’s empowerment as a likely, yet unintended, outcome of social improvement. Regarding BN 7, while the OVC Program encouraged adult support to OVC-headed HHs, the demographic factors involved presented challenges in meeting this BN. Nevertheless, BOCY and SOCY services that sought to improve child-headed HH incomes were a viable mediation strategy to potentially reduce vulnerability. Follow-on interventions may wish to consider the development of intermediary measures that distinguish between those HHs with adequate financial resources to meet material and education needs from HHs that lack access to these resources. Regarding BN 8, FGD evidence suggests that sustained school attendance is likely to remain an ongoing challenge for any follow-on interventions, such as the new Integrated Child and Youth Development (ICYD) activity, which integrates OVC Program elements with improved learning outcomes gained through formal education. Yet the OVC Program’s BN 8 performance benefited from the high-value beneficiary HH’s place on education and school attendance. While the education supports favored by BOCY enabled the activity to better achieve this BN, savings group x special allocations such as the EduFund may in fact provide a more sustainable solution to maintaining ongoing school supports now that the OVC Program has ended. PE Questions 3 and 4 Findings The ET disaggregated group differences in two material ways: (i) traditional subgroup analysis and (ii) latent class analysis (LCA). Traditional subgroup analysis drew upon demographic data (e.g., gender of head of HH, etc.) captured within each Implementing Partner (IP) Management Information System (MIS). Interesting IP contrasts emerged using the traditional Chi Squared technique to compare specialized subgroup graduation performance with those of the norm. BOCY HHs that included a member with disabilities fared the highest above the norm, whereas SOCY HHs with disabilities fared the worst against the norm out of any special needs subgroup. BOCY HHs with a higher number of children, both female- and male-dominated, fared better than the norm, while similar SOCY HHs faced graduation rates no different than the norm. However, both activities witnessed similar below-average graduation rates when it came to single-parent HHs and those with few children. Geographic graduation success rates widely varied across both activities, in which 84.6 percent of HHs graduated in BOCY’s Tororo but precious few in Dokolo (1.1 percent), and 83 percent of HHs graduated in SOCY’s Mitooma with no graduations as of yet in Rubanda. The ET then applied LCA to OVC Program Household Assessment Tool (HAT) data to organically build vulnerability profiles. SOCY programming supported HHs in “Overall Least Insecurity” more than those in “Formal Job Insecurity,” and supported both groups more than HHs in the “Overall Most Insecure” class. LCA results indicate that SOCY should consider tailoring formal job opportunity services, such as more skills building, to the “Formal Job Insecurity” and “Overall Most Insecure” groups, while further emphasizing the use of financial savings groups to its “Overall Least Insecure” HH groups. Similarly, BOCY programming better supported classes with “Overall Low Need” and “Income Insecure Need” over “Food Insecure Need” HHs. Yet it is more difficult to make similar suggestions to BOCY, as the clusters of group needs did not significantly differ through the LCA technique. Rather, for BOCY, although the data strongly suggests different clusters of vulnerability groups do exist, more tailored programming to each specific group will require additional investigation to unpack nuances that go beyond the dataset. PE Question 1 Findings The future of the case management system, especially at the lower, subdistrict levels, is currently in flux and in doubt. With the end of the OVC Program, new mechanisms to replace the case management conference (CMC) and SOVCC mechanisms, which may include an expanded membership to “mainstream” services to incorporate community development health, education, agriculture and environment, are still in formation, even as previous relics of the SOVCC and DOVCC system, at best, meet on an ad hoc basis (further constrained by COVID-19). The MGSLD ultimately holds responsibility for the formation of the new mechanism(s), and any guidance provided will ostensibly be informed by the recent 2020 National Child Policy. Yet, while the Policy does mention the formation of a new District Child Well-being Committee (DCWC)—as well as a new Orphans and Vulnerable Children Management Information System (OVCMIS) upgrade, the Child Welfare Information System)—it says nothing about lower-level mechanisms to perform the bulk of case management and provide local solutions, so that only the xi most complex cases requiring special legal or policy changes will reach the district level. Without an effective lower-level filtering mechanism similar to the CMC and SOVCC in place, the ability and capacity of the proposed DCWC should be seriously questioned. To be sure, the newly-awarded USAID/Uganda ICYD activity holds the potential to take the learnings of the OVC Program forward, and may do so as part of its attempt to scale up the case management system. The bulk of these efforts will no doubt initially focus on further system strengthening at the district level, as new roles and responsibilities are defined and tested. Yet, as of this writing, neither MGLSD nor ICYD has put forward a working model of subdistrict case management, even as the latter may be waiting for guidance from the former on subdistrict Child Welfare Committee (CWC) composition while the former may be waiting for the latter to propose financial commitments to ensure subdistrict CWC efficacy. Recalling the lessons identified in the Year 1 OVC Evaluation Report regarding a loss of case management expertise, momentum, and most importantly, continuum of care to HHs, the transition to MGLSD’s new DCWC system with ICYD support runs the similar risk of compiling a backlog of unresolved cases in which many immediate essential needs on the ground remain unaddressed. PE Question 5 Findings To determine cost-effectiveness, the ET built an “activity-based implementation costing” model to establish baseline HH unit costs for OVC Program services implemented across BOCY and SOCY. This model was based upon expenditure and implementation data collected from both activities to calculate project inputs, prices and levels of service demand. Converted into US Dollars, these models suggest an average Unit Cost of Services of $625.60 for BOCY HHs and $475.39 for SOCY HHs. The ET then drew upon QCA findings to identify group and training services as well as qualitative investigation to identify potential areas for cost-effective saving. Once these adjustments were made, the revised cost-effective cost models suggest an average Unit Cost of Services of $555.65 for BOCY HHs and $408.46 for SOCY HHs, which translates to approximately 11 percent and 14 percent unit costs savings per HH, respectively. However, the resulting model should not be viewed as an accountability tool, i.e., it should not be used for audit purposes or to cross-check total expenditures to date with those predicted by the model. Rather, it should be used as a tool to demonstrate the potential cost-saving implications of using complexity management tools like QCA to eliminate irrelevant services, which can be used as a guide to further optimize the cost￾effectiveness of similar follow-on interventions. Recommendations • Given the challenges in evaluating PEPFAR-funded complex programming, USAID may wish to consider more nimble analytical tests better suited to managing complexity and supporting adaptive management rather than impact evaluations. • Similar follow-on interventions should strongly consider replicating the OVC Program practice of enlisting PSWs to serve as the “foot soldiers” of integrated case management services through a balance of HH assessment, monitoring, HIV treatment adherence support, referrals to other services, and inhouse counseling. xii • When seeking to improve HH nourishment through such techniques as backyard gardening, follow-on interventions must remain mindful of the tendency for beneficiaries to prioritize the sale of produce over early child health concerns. • Future interventions should consider more nuanced and/or culturally calibrated operational requirements of PEPFAR’s BN 6—Financially Stable—and further explore other proxies of financial stability that do not restrict all sales of livestock for school fees. • Future interventions, related both to OVC as well as to reducing vulnerability, increasing resilience and HH economic strengthening, must seriously consider violent reactions by men to improved incomes (e.g., through increased alcohol consumption) and women’s empowerment as a likely, yet unintended, outcome of social improvement. • When contemplating resources for school fees and sustained school attendance, similar follow-on interventions that include savings group components should consider establishing special savings group education funds as a means of providing a more sustainable solution to maintaining ongoing school supports after the end of the OVC Program. • Future studies related to similar follow-on interventions should consider investigating whether their services favor the graduation of smaller HH sizes and those HHs containing female majorities. • Similar follow-on interventions may also wish to consider tailoring formal job opportunity services, such as more skills building, to the “Formal Job Insecurity” and “Overall Most Insecure” groups, while further emphasizing the use of financial savings groups to its “Overall Least Insecure” HH groups. • Follow-on interventions similar to the OVC Program should consider advocating with MGLSD a district-level case management model that more systematically shares case resolution procedures with lower levels. This advocacy may also provide further clarity in the National Child Policy. • USAID and follow-on interventions similar to the OVC Program, such as ICYD, should support some version of SOVCC and CMC to advance its OVC Program investments. This should likely include minimal transport allowances for CMC and SOVCC participants. • The newly proposed PSW renumeration scheme should be bolstered with additional training, but studied for cost-effectiveness and sustainability. • Follow-on interventions similar to the OVC Program may wish to consider scaling down district-level case management committees to focus on the resolution of policy decisions, even as the bulk of complex cases that reach the district level can mostly be resolved through the Community Based Services Department (CBSD), which can also maintain new versions of the OVCMIS and advocate for additional funds from the national government. 1 1. Introduction Deep and extensive vulnerabilitiesthat especially affect children exist in Uganda, where 96 percent of children are considered vulnerable. Vulnerabilities take different forms, including poverty, malnutrition and exposure to violence. Low levels of education and the high prevalence of the human immunodeficiency virus/acquired immunodeficiency syndrome (HIV/AIDS) among children themselves, as well as within their families, exacerbate these vulnerabilities. OVC Program USAID/Uganda’s Orphans and Vulnerable Children (OVC) Program sought to improve the well￾being of children and youth made vulnerable by HIV and other challenges in Uganda. It employed an integrated approach to address the factors of vulnerability. Two regional activities started in 2015 and covered 46 districts across the country. The Better Outcomes for Children and Youth (BOCY) and Sustainable Outcomes for Children and Youth (SOCY) programs aimed to economically empower children, youth, and their caregivers to access core services, strengthen systems to provide core services, and improve the coordination of community-based clinical and socio-economic services for efficiency and effectiveness along the continuum of care. BOCY ran across 22 districts in Eastern and Northern Uganda and SOCY operated in Western and Central Uganda in 24 districts. In each district, OVC Program partners were assisted by the District OVC Committees (DOVCCs), and their corresponding Subcounty OVC Committees (SOVCCs). This work was carried out in support of and in partnership with the Ugandan Ministry of Gender, Labor, and Social Development (MGSLD). BOCY and SOCY were designed following the same Theory of Change and implemented according to similar (but not identical) models, aiming to address multiple forms of vulnerability among children and youth through an integrated, comprehensive approach. BOCY and SOCY sought to economically empower orphaned and vulnerable children, youth and their caregivers to: • Result 1: Access core services • Result 2: Strengthen local government, Civil Society Organizations (CSOs), and informal community structures to increase and improve core services • Result 3: Improve coordination of community-based clinical and socioeconomic services for efficiency and effectiveness along the continuum of care. OVC Evaluation Scope of Work USAID selected SoCha, LLC (hereafter SoCha) to conduct performance evaluations for both BOCY and SOCY as well as to continue the quasi-experimental design impact evaluation, collectively referred to as the OVC Evaluation. The purpose of this evaluation is to provide rigorous evidence regarding the extent to which participation in USAID/Uganda’s OVC Program has reduced Household (HH) vulnerability and improved OVC well-being, and to identify factors or combinations of factors, within and outside the program, which may explain how HHs achieved results or what barriers may have prevented greater success. The results from the final performance evaluation of SOCY and BOCY and outcome endline will inform future programming in Uganda and elsewhere, especially as it relates to the case-management and graduation models utilized by the activities. 2 The OVC Evaluation must answer the following performance and impact evaluation questions: • Impact Evaluation (IE) Question: Does participation in SOCY activities contribute to a reduction in HH vulnerability and improve OVC well-being? • Performance Evaluation (PE) Question 1: To what extent are processes in place to facilitate the long-term success of program interventions beyond the program’s end? What factors may prevent long-term success? • PE Question 2: What coordination and management mechanisms have worked best to support the integrated OVC implementation approach? What enabling factors do these mechanisms engage with? • PE Question 3: Do HHs or individuals with specific characteristics (e.g., individuals with disabilities, single-parent HHs, regional variation, etc.) progress and graduate at different rates, and if so, why? • PE Question 4: To what extent do females and males experience different benefits from the program? Are there unintended results for either group? • PE Question 5: How cost-effective are different pathways to change? How do interventions/combinations of interventions compare in impact and cost? The OVC Evaluation Team (ET) addresses these questions in coordination with OVC Program stakeholders, including USAID/Uganda, SOCY, BOCY and Ugandan government staff. Note that the IE Question builds off previous work under the Ugandan Learning Contract, which involved initial baseline (2016) and extended baseline (2017) surveys. Evaluation Design The OVC Evaluation is divided into three deliverable reports: Year 1 Report of Interim Findings (FY 2019), Year 2 Report of Interim Findings (FY 2020) and Year 3 Final Report (FY 2021). The previous Year 1 Report identified the processes in place that may facilitate the long-term success of the OVC Program beyond its end;1 explored through qualitative investigation how HHs experience these processes on the ground and how these experiences varied according to demographic, geographic, and gender characteristics (Q2–4); and built a cost-effectiveness model that identified Para Social Worker (PSW) labor as a dynamic point of implementation with potential to scale (Q5). The Year 1 Report also provided a midline update of the SOCY IE. The Year 2 Report provided the next iteration of findings and analyses of OVC Program implementation regarding the OVC Evaluation Scope of Work. It was primarily geared towards evaluating the OVC Program’s effectiveness. The OVC Program had used two interrelated yet mostly distinct measures of effectiveness. At the results level, the ET evaluated OVC implementing partner (IP) success according to their performance on the President’s Emergency Plan for Aid Relief (PEPFAR) February 2019 graduation benchmarks (BNs). The ET adopted these BNs as the primary measure of success to further address PE Questions 2, 3 and 4. At the 1 The ET found that HH enrollment in the OVC Program assumes a fairly sophisticated division of labor and case management system. On the implementation side, HH enrollment occurs through a diverse labor force arrangement of social workers (SWs), para-social workers (PSWs), monitoring and evaluation (M&E) Officers, specialized service providers, coordinators and, crucially, case facility linkage officers (CFLOs). Once enrolled, HHs enter a wide gamut of services spanning savings and loan, income generation, hygiene, youth and children (especially girls), temporary consumption support, psycho-social, child protection, and, increasingly, facility-referral services. These service packages benefited from BOCY’s and SOCY’s capacity to engage, assess and implement evidence-based programming from activities across the region. This evidence directly informed the design of many of the services offered to HHs. 3 impact level, the OVC Program initially adopted the MGSLD’s “Household Vulnerability Assessment Tool” (HVAT) as a measure of success. The ET adopted a refined version of the HVAT as the primary impact measure used to address the IE question. Adopting two different measures of effectiveness enabled the ET to conduct more robust analyses of the OVC Program’s performance and impact, which facilitated greater opportunities for learning. It should be noted that the Year 2 Report was constrained by the COVID-19 crisis and corresponding restrictions to movement. As a result, the Year 2 Report was primarily driven by quantitative data. First, the final wave of the impact evaluation was completed at the end of 2019 and avoided any constraining issues. The overall impact evaluation results were analyzed and presented in that report. Second, the detailed performance evaluation findings were mostly limited to in-depth analyses of SOCY and BOCY implementation data, captured in each activity’s Management Information System (MIS) and supported by limited remote interviews conducted with relevant staff. In other words, the Year 2 findings were primarily quantitative. This Year 3 Final Report validates the quantitative findings through qualitative investigation conducted from November 2020 until April 2021. This report also takes the learnings and analysis of the two interim reports forward to comprehensively address the primary evaluation question identified above. Methodology The OVC Evaluation rests upon a wide range of analytical and data collection methodologies to model the OVC Program’s implementation and evaluate its results. To build the implementation model, both SOCY and BOCY granted the ET online access to their databases, which contained initial enrollment, implementation, and graduation assessment data, and other relevant information to address questions of effectiveness related to OVC Program implementation and graduation results. The ET used this access to replicate more than 135 GB of data held in the databases, which was processed and analyzed using Python and R scripts. ET staff regularly interacted with SOCY and BOCY staff to confirm a shared understanding of each database. When modeling the BOCY intervention, the ET primarily drew upon the following sources: • All BOCY baseline information came from Household Orphans and Vulnerable Children Vulnerability Prioritization Tool (HVPT), Viral Load, Home Visit, Household Assessment Tool (HAT), and Semi-Annual Assessment Tool (SAT) reports. • All BOCY graduation information came from Graduation Assessment reports as well as separate reports sent by the monitoring and evaluation (M&E) officer. • All BOCY treatment information came from the BOCY Layering reports.2 When modeling the SOCY intervention, the ET primarily drew upon the following sources: • SOCY HAT and Monitoring, Evaluation and Reporting (MER) reports provided all SOCY baseline information. The ET only considered Graduation Assessment reports and the earliest record. 2 Note that BOCY struggled to enter more detailed geographic information (District, Subcounty, Parish and Village). For example, the Layering Report—which best served as an overall summary report—contained 50,353 HHs, of which 20,006 (40 percent) lacked Parish information. 487 HHs completely lacked any type of demographic information. 4 • All SOCY graduation information came from Graduation Assessment reports. The HHs enrolled are the ones appearing in the Graduation Assessment fields with a valid “date of Household Improvement Plans (HIP)” and baseline fields completed. • All SOCY treatment information came from two files: the SOCY Home Visit Member and Community Training reports. The ET collated and managed data through Structured Query Language (SQL) and derived various logical commands to collate and combine numerous data source reports. More information about how SOCY and BOCY were modeled is found in the Year 2 Report (see Annex 6). The ET conducted an analysis of OVC Program implementation data primarily with Qualitative Comparative Analysis (QCA), supported by qualitative investigation. QCA analysis focused both on Truth Table Analysis and minimization.3 The results of these analyses were obtained using R and QCA software (known as fsQCA), and were subsequently investigated qualitatively as part of the larger qualitative component. Qualitative data was collected through 116 focus group discussions (FGDs) across 19 districts, 100+ key informant interviews (KIIs), and call center data (500+ respondents) across the entire area of implementation. This data was processed using Stata and MaxQDA. Impact Evaluation Methodology The IE relied upon a Difference-in-Differences (DID) impact evaluation design. Data was collected via enumerators carrying tablets containing questionnaires tailored to HH primary caregivers, children and youth. Each HH was visited across four annual survey waves (2016, 2017, 2018, 2019) from the end of October into early December. Across all four waves, the survey consistently gathered data from 1,922 HHs in both control and treatment groups across 15 target districts. DID designs compare two types of changes: Changes in the outcome of a treatment group (or groups) before and after an intervention has occurred and changes in the outcome of a control group (or groups) over the same period of time. This design puts in place two controls: The difference in the before-and-after outcomes for the group(s) enrolled in the program controls for factors that are constant over time in that group (as it is comparing the same group to itself over time). This difference is then compared with the difference in before-and-after outcomes for a group that was not enrolled in the program but which was exposed to the same set of environmental conditions over the same time period, to control outside time-varying factors. DID thus combines these two counterfactuals (before-and-after comparisons, and comparisons between those who were enrolled and those who were not enrolled) to better estimate the impact of the program (see Figure 1 below). 3 Truth Table analysis is useful to identify prominent patterns of service delivery combinations, aka “paths,” and to demonstrate how these paths better explain how HHs achieve a BN as opposed to a narrower focus on each individual service. Truth Table also reveals which service pathways HHs seldom take and which produce fewer results. QCA minimization is useful to identify which core service categories may be necessary, sufficient or irrelevant to achieving the outcome of success. In doing so, QCA also identifies which combinations of enabling service categories are most effective in contexts where the provision of other services is lacking. QCA results are best conceived as “essential packages” that are necessary and/or sufficient to achieve graduation even though other relevant core services may be lacking. Overall, the Truth Table and QCA findings suggest that there are varied pathways to meeting each BN, as well as to combining OVC program core service packages to graduation. 5 Crucially, the DID design does not require that each group has statistically insignificant outcome differences at baseline, i.e., statistically significant differences across groups do not pose threats to the internal validity of the design. Rather, the key assumption for DID is commonly referred to as the “parallel trends” assumption in which the outcomes across treatment and control groups are assumed to follow the same trend over time in absence of the treatment. Even though their outcome scores are slightly different between baseline and extended baseline, their trends of improvement remain constant, and no substantive outcome has occurred in the treatment group because it is following a longer-term trend in the same manner as the control group. The two groups were thus comparable. Figure 1 below demonstrates how the aggregate effective size of the control group outcome can be higher than in the treatment group, and yet the impact can be found in the latter between its actual endline scores and what those scores would have been had the project not occurred. Figure 1: DID design in which the control group scores higher than the treatment group but the treatment is effective The IE contained 1,011 HHs from the treatment group and 934 HHs in the control group, for a total of 1,922 HHs.4 This sample size of 1,922 HHs per wave was adequately powered to test the treatment hypotheses against the null.5 Surveys were conducted with the HH primary caregiver as the primary respondent, as well as one sampled index child aged 0–9, and one sampled index youth aged 10–17. Data was also disaggregated according to sex, age (10–17), OVC (0–4, 5–9), youth/non-youth head of HH, and geographic area. Both control and treatment group HHs were visited between November and December across four survey waves, i.e., the initial baseline (FY2016), extended baseline (FY2017), midline (FY2018) and endline (FY2019) surveys. The original IE design did not anticipate an extended baseline wave, but shifts in PEPFAR Country Operation Plan (COP) geographic priorities meant that initial baseline HHs were not enrolled to receive treatment for the 2018 survey wave. The decision was then made to redesignate these HHs as “extended baseline” to enable more rigorous parallel trends assumption testing. The surveys were conducted in Bushenyi, Gombe, Hoima, Isingiro, Kabale, Kampala, Kamwenge, Kanungu, Kasese, Kibaale, Kiruhura, Kyenjojo, Luweero, Mityana, Ntungamo, Rukungiri, and Wakiso 4 The first wave initially surveyed 2,438 HHs and by the end of the fourth wave, 1,922 HHs had been consecutively surveyed, yielding a cumulative HH attrition rate of 20 percent distributed relatively evenly across control and treatment groups. 5 Our calculations were based upon the following assumptions: a relatively conservative standard deviation of 1.2 (there is adequate room for HHs to vary in vulnerability), a mean of zero (i.e. the difference between the control and treatment groups is zero at baseline), and an Average Treatment Effect between .2 and .4 (this is because the HVAT is a categorical variable, which translates to 1/20th and 1/10th, respectively, aka powered to detect a minimal effect size that falls between 5 percent and 10 percent). 6 districts. Data for all four survey waves are in Annex 2. Each survey sampled at least 25 HHs per parish and did not exceed 35 HHs per parish. The primary outcome measure on which the SOCY IE was powered was socioeconomic HH vulnerability status. HH vulnerability was defined as the inability to cope with and thrive during economic and other shocks. Operationally, this was captured through the HVAT. Through the lens of six core programmatic areas (CPAs), the HVAT is set up to connect vulnerable HHs to relevant services. The HVAT is structured to assess a range of vulnerable situations that HHs may be in, such as economic; nutritional; water, sanitation, and hygiene (WASH)-related; education; psychosocial; and child protection. It is a quick vulnerability screener that allows the enumerator to set the HH on a predetermined path to receive support. The HVAT yields a composite score corresponding to “not”, slightly, moderately, and critically vulnerable categories. These scores were previously obtained through simple aggregation of individual scores, ranging from 0 to 4 (where 0 is not vulnerable and 4 is most vulnerable) across 18 questions. However, this simple aggregation technique was misleading. For example, identification of such HH assets as dish washing racks could potentially hold slightly more weight in HH vulnerability scores than instances of rape and defilement. Clearly, the HVAT’s simple aggregate system was misaligned. As such, the components of the HVAT had to be reweighted and restructured using Exploratory Structural Equation Modeling to identify four refined CPAs. 6 These were: • Assets: The combination of assets in and outside the house define this factor. It is about the material wealth of the HH. • Child Health: This factor encompasses the positivity of the items that relate to child emotional and physical well-being. All items except for Main Contributor and Child HIV Status are inversely related. This makes sense because the scale is constructed in a way that lower values indicate better health and access to schooling, and thus the inverse values indicate better outcomes for the child. • Income Privileges: This factor consists of items that indicate the beyond-basic amenities that the HH can afford. As the largest factor, it includes stability of shelter, access to WASH facilities, three meals a day, variety in food types consumed, and a clean source of water. • Child Protection: This factor encompasses the negative items that work against child protection, such as disciplinary action that includes physical or emotional repercussions for the child. Once these new CPAs were validated, the ET then normalized7 their vulnerability scores. The four CPAs were then added together to create a new, more refined HVAT score, 8 which again 6 Using ESEM, the ET yielded considerable improvements to the HVAT design (χ2 (203) = 867, RMSEA = .047, CFI: 0.88, TLI; 0.87) to test how strongly each item is associated with the underlying factor of vulnerability. ESEM is the preferred method for recalibrating the HVAT because each question can contribute to multiple CPAs. This technique generated factor scores which were used to weight the raw scores according to the strength of their relationship to vulnerability. For each of the subsequent waves, the ESEM model of four factors was “confirmed” using confirmatory factor analysis. Factor scores were then generated for each wave. 7 Values between 0-1. 8 The highest aggregated score can be 4 (1 is highest for each factor) because there are 4 factors. 7 corresponded to the four not (0 to 1), slightly (+1 to 2), moderately (+2 to 3) and critically (+3 to 4) vulnerable categories. Statistical Methods Used to Address Questions 3 and 4 A previous analysis modeled overall graduation rates for the entire beneficiary HH population for both BOCY and SOCY, i.e., approximately 41 percent, and established this as the normative graduation rate to be expected across HHs, regardless of their characteristics. “Defining” characteristics of different HHs were then identified through two sources. First, BOCY and SOCY MIS data catalogued various HH demographic features, and these formed the basis of the first type of group definition, including traditional gender, disability, and region subgroups. Traditional means testing techniques were then applied to these demographically defined groups to identify if their graduation differences were significant. Apart from the analysis of traditional gender, disability, and region disaggregation presented above, the ET also sought to understand the HH clustering of needs. Indeed, central to PE Questions 3 and 4 is the idea that subgroups can display vulnerability signs that are unique because of the persons in that subgroup. For instance, HHs with a disabled person(s) have a unique vulnerability—someone in the HH needs extra support, be it a physical, learning or psychological need. Here the ET considers these needs not as individual properties, but as group properties lumped as clusters of needs. To identify these groups clustered by needs and explore their relationship to graduation, the ET deployed Latent Class Analysis (LCA) using BOCY’s HVPT dataset and SOCY’s HAT dataset.9 Similar to the approach applied to the demographic group classifications, LCA group graduation tendencies were analyzed again. LCA is always exploratory because it is not possible to hypothesize how many classes would emerge a priori to the analysis of the data. Instead, classes of group needs emerge “organically” from the underlying data structure. The ET began this analysis with the smallest number of classes (two) and increased the number of classes until the structural model no longer fit. Both SOCY and BOCY LCA yielded three-class models containing robust fit. This analysis created “Overall Low Need”, “Income Insecure” and “Food Insecure” classes for BOCY, and “Overall Least Insecure”, “Formal Job Insecure”, and “Overall Most Insecure” classes for SOCY. Next, the ET conducted classification, or the distinction between classes, based on their homogeneity, which was determined by calculating the odds for each HH to be a member of each class. Homogenous classes emerge from similar odds for HHs. Finally, the ET calculated latent class membership counts and proportions. Cost Effectiveness Methodology To determine cost-effectiveness, the ET used an activity-based implementation costing method, which establishes ongoing costs of implementing OVC services using agreed approaches to delivery costs. The approach is summarized with the following formula: Cost of Service = Input (project) x Price x Quantity(demand) Below we repost the first and second factors of this approach (Input and Price) already taken from the Year 1 Report and finalize the equation with updated Demand data based upon actual HH 9 Previously, the ET conducted ESEM to better understand the way in which the items of the HVAT capture vulnerability. Latent Class Analysis is also a latent variable modeling technique, but instead of examining how items cluster together, LCA indicates how people cluster together. Another way of conceptualizing latent classes is to think of the underlying assumptions of any latent variable analysis—general population is heterogenous. From this heterogeneity emerge homogenous classes that are similar based on their latent or shared traits. 8 participation in each service. The result yields an overall unit cost of services per HHs. We then apply the results of the QCA modeling to remove those services the model identified as irrelevant (already identified in the Year 2 Report) and this Year’s report qualitative findings that suggest revisions to the case management system. The result of these adjustments yields approximate savings, which can be used as a guide to further optimize the cost-effectiveness of similar follow￾on interventions. All costs are presented in Ugandan Shillings (UGX), unless stated otherwise. Building the Unit Cost of Service Model When calculating unit costs, the primary unit of analysis to estimate these costs are fully burdened labor costs. This starts with the costs of the specific staff tasked to directly administer the various services (e.g., a SILC/VSLA administrator to oversee the SILCs/VSLAs, or a PSW to conduct home visits). Additionally, these staff were supervised by higher-level specialized staff categories, and their labor costs were distributed evenly across all the staff they supervised based upon established supervisor to supervisee ratios. Other direct costs, such as the payment of school fees when providing sponsorship services, were then added, as were the indirect overhead costs of the CSO. In all cases, the assumptions behind the cost of labor, the time spent administering a service, the time spent supervising the staff who administer the service, the ratio of supervisor to supervisees, as well as indirect and other direct costs were derived from conversations with SOCY and BOCY staff and from actual expenditure data. That is, the ET made no decisions on the cost model’s content and instead drew upon OVC Program information. First, the cost model identified the supervisory and group management costs for the various services provided by SOCY and BOCY. The BOCY monthly costs of the basic salaries for these labor categories are illustrated in Table 1. Table 1: BOCY monthly costs Labor Category Monthly Salary (UGX) Conversion Salary (USD)* District Coordinator (Project Managers) 11,576,250 $3,215.63 Case Management Tech Officer (SW) 4,762,800 $1,323.00 Economic Strengthening Tech Officer 2,381,400 $661.50 Youth Tech Officer 2,381,400 $661.50 Parenting and ECD Tech Officer 2,381,400 $661.50 SILC Supervisor 1,374,100 $381.69 Youth Field Assistants 550,000 $152.78 Youth Officers 1,374,100 $381.69 Community Based Trainers 345,000 $95.83 District Based Trainers 345,000 $95.83 PSW 30,000 $8.33 *Assumes $1 USD to 3,600 Ugandan Shilling exchange rate. The cost model identified that PSWs sit at the bottom of the labor cost hierarchy. PSWs are paid an average stipend of 30,000 UGX per month. They then enroll and/or visit an average of six HHs per month, yielding an average 72 HHs per year (it is assumed that PSW labor required to enroll 9 and maintain are the same). Further, they are supervised by a SW, who supervises on average 15 PSWs. PSWs also incur bicycle maintenance costs, and these are incurred an average two times per year. When this information is combined, it yields an average PSW cost per HH of 98,006 UGX per year. Total estimated district-level costs are then derived by incorporating annual CSO overhead costs, which vary in their allocation to HHs based upon the total number of HHs enrolled in that district. Table 2: Main services that were costed Group VSLA Sinovuyo Stepping Stones ECD G1C Journeys PEC PEEC # of participants /group 15 10 10 10 10 10 10 10 Months of program, 5 8 8 8 8 8 8 8 Group Facilitator Monthly stipend 60,000 60,000 60,000 60,000 60,000 60,000 60,000 60,000 # groups /facilitator 4 4 4 4 4 4 4 4 CSO Supervisor # meetings 2 2 2 2 2 2 2 2 Transport cost/visit 15,000 15,000 15,000 15,000 15,000 15,000 15,000 15,000 Cost per group 105,000 150,000 150,000 150,000 150,000 150,000 150,000 150,000 Cost per participant 7,000 15,000 15,000 15,000 15,000 15,000 15,000 15,000 In Table 2, the unit costs are fairly standardized across groups. The reason more services were not incorporated in Table 2 was because separate labor costs associated with their delivery could not be identified, typically due to very low levels of attendance. For example, PSWs deliver a variety of services during their HH visits, but these are not directly identified in the CSO invoices submitted to SOCY and BOCY coordinating offices. Instead, “time on task” data collected from the PSW call center survey was relied upon. Finally, case management unit costs were calculated in Table 3, where each respective meeting was the unit of analysis. Table 3: Case management unit costs (in UGX) DOVCC SOVCC Case Conference Number of Members 23 15 50 Number of members that receive a travel fee 4 13 50 Travel/Attendance Fee 15,000 15,000 5,000 Number of government officials attending 19 2 2 Facilitation fee per official 20,000 20,000 20,000 Refreshments per person 20,000 20,000 3,000 Stationery per person 3,000 Cost per event 900,000 535,000 590,000 10 Additional unit costs, such as various trainings, CSO overhead, school fees and total indirect costs per district, are further elaborated in Annex 4. Further technical details are below. The BOCY and SOCY cost models were developed to estimate unit costs for various activities in order to support subsequent analyses that will compare the cost of HH pathways to graduation. Data were collected in the form of payment vouchers from the implementing partners, WEI/B and CRS. Hard copies of vouchers were reviewed at their respective offices in Kampala and electronic copies of vouchers were also shared. Over 80 payment vouchers were reviewed to identify standard input prices that apply. It is clear from the vouchers that CSOs under both projects used similar inputs and that the input prices were standard. This means that it is acceptable to use a standard set of price assumptions for both costing models. An overview of the model and the flow of key information is shown in Figure 2. Figure 2: Overview of the costing model 11 Updating with Demand Data To identify the demand for each relevant service captured in the cost models of each activity, the ET relied upon call center data of PSW time spent on task for home visits, as well as SOCY and BOCY HH activity data collected from each activity’s MIS. Specifically, the ET derived: PSW Time on Task for each HH visit: Average time spent on various referral and counseling service breakdowns were derived from PSW responses to task distribution across an assumed one hour visit. This analysis yielded the following distribution for both BOCY and SOCY shown in Table 4: Table 4: Activities carried out during HH visits Activities carried out during HH visits Percent PSW Time per activity Referrals for domestic violence 5 percent Counseling on domestic violence 10 percent Referrals for violence against children 5 percent Counseling on violence against children 9 percent Other referrals for protection assistance 3 percent Referrals for emergency support 4 percent Referrals for livelihoods training 4 percent Counseling on livelihoods training 7 percent Counseling on early childhood development 8 percent Counseling on malnutrition 8 percent Counseling on HIV testing 13 percent Referrals for HIV management/treatment 9 percent Counseling for HIV 15 percent Average Home Visits: Total number of visits for each HH separated by at least one day including assessment, enrollment, and final graduation visits taken from home visit and graduation logs divided by the total number of HHs enrolled. Note: HH visits unfolding after graduation were not counted. This analysis revealed that BOCY HHs received an average 9.3 visits across their enrollment and SOCY HHs received 7.5.10 Average Trainings provided to the HH: Total participation of HHs and HH members (based upon unique identifiers) taken from home visit logs and training layering reports divided by the total number of HHs serviced to average HH participation and average HH member participation. 10 There are a variety of explanations for why BOCY conducted more sessions, but one strong contributor to the difference is found in the earlier approaches to how graduation was determined. Prior to 2019, BOCY regularly conducted annual and semi￾annual HH assessments to determine graduation, even as SOCY anticipated subcontracting independent organizations to assess graduation, but this was never fully realized. The result, inter alia, was fewer graduation assessments in the SOCY area. 12 This analysis revealed the following average training assumptions received for BOCY HHs in Table 5: Table 5: Average training received per BOCY HHs Training Topic Attendees per HH HIV Prevent Education 0.17 Nutrition Education/Assessment 0.15 Financial Education 0.45 Agribiz Education 0.12 Enterprise Education 0.01 Management Education 0.00 Micro Biz Training 0.03 Inform Education 0.13 OVC Fund Training 0.19 Child Protection Education 0.15 ECD Training 0.22 Parenting 0.45 This analysis revealed the following average training assumptions received for SOCY HHs in Table 6: Table 6: Average training/support received per SOCY HHs Training Topic Attendees per HH OVC Fund Training 0.17 Assessed Nut. 0.45 Parenting 0.81 No Means No 0.12 Child Optimized Financial Ed. 0.04 Enroll Support 0.03 Material Support (school uniforms and books) 0.06 Support Birth Reg. 0.34 13 HH Group Participation: Total participation of HH members (based upon unique identifiers of direct beneficiaries) taken from group logs divided by the total number of HHs enrolled, to yield the average number of HH member participation in each group. This analysis revealed the following average group attendance assumptions received for BOCY HHs in Table 7: Table 7: Average group attendance per BOCY HHs Group Attendees per HH VSLA 1.55 F 1.99 Stepping Stones 1.47 ECD 1.30 G1C 1.75 Journeys 2.51 PEC 1.60 PEEC 1.77 This analysis revealed the following average group attendance assumptions received for SOCY HHs in Table 8: Table 8: Average group attendance per SOCY HHs Group Attendees per HH Cottage 1.36 Garden 1.35 Enterprise 1.33 Coffeelands 1.12 SILC 1.26 Aflateen 1.34 Better Parenting 1.15 Agronomy 1.34 Apprenticeships 1.10 Further details on the costs associated with the above listed assumptions are found in each activity’s cost model attached to Annex 4. Limitations COVID-19 imposed unprecedented constraints and limited the findings of the Year 2 Report, and continued to imposed some constraints on this final report. Due to major travel restrictions, the ET expatriate workforce was only able to travel to Uganda at the end of 2020 and in February to March 2021. Moreover, the 2021 Ugandan General Elections constrained the qualitative investigation in terms of availability and, prior to election results, a willingness to discuss sensitive topics such as criminal case resolutions. Finally, both SOCY and BOCY entered their final closing down phases at the end of 2020 into April 2021, even as BOCY’s prime implementer, World Education Bantwana, entered a new startup phase of the follow-on ICYD intervention as a primary 14 sub-awardee. As a result, the ET interacted mostly with BOCY and SOCY M&E and IT technical staff, although program staff consultations also occurred. The ET managed these limitations through flexible rescheduling, remote interviewing and, in the case of three districts, qualitative team re-investigation after the 2021 Elections were complete. Limitations to the Impact Evaluation HH vulnerability scores may have been influenced by potential contagion, contamination and implementation infidelity. More precisely: • Contagion: In 2016, the ET had originally worked closely with the SOCY implementation team to enroll HHs according to the initial OVC Program screening criteria. To ensure selection was ethical, control group HHs would be enrolled at a later date. However, subsequent COP guidance reallocated SOCY’s geographic priorities at the subcounty level. As a result, some HHs originally designated to the control group received treatment in 2019 while other HHs in the same area remained in the control group. Twenty percent (109 out of 532) of villages contained more than one treatment and control group HH. Although the ET did not find evidence of any control HHs’ members participating in SOCY￾supported groups, control group HHs in these villages may have been affected by SOCY. • Contamination: As the SOCY activity offered an integrated service package containing HH economic strengthening, savings, nutrition, WASH, health, child protection, and psychosocial supports, the likelihood HHs would be “contaminated”—have access to another intervention which also seeks to reduce vulnerability—was high. In fact, the ET anticipated contamination would be a factor and measured its prevalence across every survey wave. For example, in 2018, the extended baseline report found that those control and treatment group HHs who managed to reduce their vulnerability did so through a package of services in which Gender-Based Violence (GBV) counseling sat at the core. By the end of 2019, control group HHs had received an average of 1.46 external services while the treatment had received an average of 1.61 external services. These differences were not significant. As such, the ET determined contamination was relatively consistent for both the control and treatment groups. • Infidelity of implementation: As discussed in the 2019 Year 2 Report, the ET found that the SOCY “treatment” to HH vulnerability changed in response to PEPFAR’s sharper focus on HIV suppression and other 95–95–95 goals, i.e., 95 percent of all people living with HIV know their HIV status; 95 percent of all people with diagnosed HIV infection receive sustained antiretroviral therapy; and 95 percent of all people receiving antiretroviral therapy (ART) have viral suppression. This transition entailed changes to the definition of success for each activity in which graduation defined by HVAT scores (i.e., not vulnerable) was replaced by the eight PEPFAR graduation BNs. Because the IE was not able to deviate from the HVAT outcome measure after preliminary baseline measures had already been taken, its measure of success became increasingly out of line with refined SOCY implementation goals. As a result, the ET was unable to determine if the insignificant differences between the control and treatment group vulnerability scores were due to the potentially confounding issues listed above or due to a lack of implementation potency in the SOCY service delivery model. In what follows below, the ET has taken the liberty to reorder the findings of each EQ and IE result according to a discussion designed to facilitate a more logical flow of results. The next section 15 involves further elaboration of the IE findings to focus on lower-level results. These findings lead to a potential paradox which is discussed further in EQs 2, 3 and 4. EQ 1 and 5 then discuss the mechanisms most likely to continue and potential opportunities for more cost-effective adaptation. 16 2. Is SOCY contributing to a reduction in household vulnerability? To determine if the OVC Program was working or not, USAID/Uganda commissioned a quasi￾experimental design IE to answer the primary question: Does participation in SOCY activities contribute to a reduction in HH vulnerability and improve OVC well-being? The Year 2 Report found that SOCY had not made a significant reduction to HH vulnerability as measured by the HVAT relative to its comparator control group from 2016–2019. The HVAT categorized vulnerability across a spectrum comprising “Not Vulnerable”, “Slightly Vulnerable”, “Moderately Vulnerable”, and “Critically Vulnerable”. The ET further divided the HVAT into four subdimensions of vulnerability: Assets, Child Health, Income Privileges, and Child Protection. This Year 3 Final Report conducted comparisons of control and treatment group differences across these four subdimensions. While no significant differences were found between control and treatment group scores in the Asset and Child Health subdimensions, significant differences were found in the Income Privileges and Child Protection subdimensions. Specifically, Income Privileges changes significantly improved in the treatment group relative to the control group, even as Child Protection changes significantly declined in the treatment group relative to the control group. The ET labels these findings as the “SOCY Paradox”, i.e., Why do SOCY HHs experience less Child Protection during times of improvement relative to the control group? Tentative hypotheses and corresponding evidence in presented in the next section. What follows below is a summary of relevant background IE information already discussed in previous reports, a discussion of the additional analytical steps taken during this Final Report, and a presentation of the results achieved. Evaluating Impact through Difference-in-Differences Design The analysis revealed positive and negative changes to HH vulnerability for both control and treatment groups across the four survey waves. The change modeled between control and treatment groups is 0.039 and indicates that the treatment group became more vulnerable11 than the control group, but this result is not significant. 11 Large values indicate high vulnerability, and vice versa. 17 Graph 1 is a visualization of the average score for both control and treatment groups across all four waves of data collection associated with OVC (note: survey data was collected immediately prior to the year listed, e.g., 2017 data was collected in November/December 2016, etc.). A reminder that higher HVAT scores indicate higher vulnerability. Mean scores for both the control and treatment groups never exceed or drop below the “Slightly Vulnerable” category. The control group started at a higher vulnerability average (1.34) compared to treatment (1.31) in 2017. At 2018 data collection, treatment and control were at 1.46 and 1.47, respectively. This similar trajectory meets the threshold required for the “parallel trends” assumption. Next, as the intervention was slated to start between 2017 and 2018, both groups saw a decrease in vulnerability: the treatment group fell to 1.27 and control to 1.29. Vulnerability rose for both at 2020 data collection, with treatment rising to 1.44 and control to 1.43. In Table 9, the “Vulnerability” column indicates the mean HVAT outcome for each group as well as each group’s individual differences for pre￾intervention and post-intervention. The “|P>|t|” column indicates whether the differences are statistically significant. The “Pre” section (November–December 2015, i.e., FY2016Q1) of Table 1 indicates that both treatment and control groups had different means, 1.317 and 1.346, respectively, and that the difference estimation, -0.029, was statistically significant. The “Post” section (2020) indicates that both groups were different at this time point as well, with treatment at 1.443 and control at 1.433, and that the difference estimation, 0.009, was not statistically significant at post level. MAIN Finding: The DID estimated is 0.039, which indicates that the treatment group is slightly more vulnerable than the control group. The estimate is not statistically significant (though it was very close to meeting p<0.1). Table 9: DID results Vulnerability |P>|t| Pre: control 1.346 treatment 1.317 Diff (T-C) -0.029 0.082* Post: control 1.433 treatment 1.443 Diff (T-C) 0.009 0.576 Diff-in-Diff 0.039 0.104 Inference: *** p<0.01; ** p<0.05; *p<0.1 Graph 1: Average score for both control and treatment groups across all four waves of data collection associated with OVC 18 Closer Look at the Data As mentioned above, the ET refined the HVAT with factor analysis to identify four underlying subdimensions associated with vulnerability: Assets, Child Health, Income Privileges, and Child Protection. In this Final Report, the ET further explored control and treatment group differences across these four subdimensions. While the ET found no significant control/treatment group differences in Assets and Child Health across the four waves, it did find significant group differences in Income Privileges and Child Protection. The annual vulnerability distribution for control and treatment groups is presented in Graph 2, with different vulnerabilities color coded within each annual results bar. As evident, both control and treatment groups experienced relatively similar declines in HH income privileges during the parallel trends year (2017 to 2018). Both then experienced noticeable improvements from 2018 to 2019 (i.e., the first year of implementation), with half of the HHs in the treatment group migrating to the “Not Vulnerable” category, while fewer HHs in the control group did the same. The final 2019 to 2020 year witnessed slight declines in HH income privileges in both groups, yet fewer treatment group HHs declined than control HHs. As shown in Graph 3, Child Protection followed very similar trends, but in the opposite direction. Child Protection declined for both control and treatment groups from FY2017 to FY2018 in a relatively similar fashion, but a noticeably higher number of treatment HHs slide from “Not Vulnerable” to the more severe “Moderately Vulnerable” than in the control group from FY2018 to FY2019. Child Protection slightly improves from FY2019 to FY2020 in terms of a migration from “Moderately Vulnerable” to “Slightly Vulnerable,” but this improvement is visibly stronger in the control group. Correlation Analysis Correlations between the factors: Since the HVAT was the ultimate measure for eligibility into and graduation from OVC and related programming, it was imperative to understand the correlations between the four different dimensions of vulnerability (Assets, Child Health, Income Privileges, and Child Protection). The 0% 50% 100% T C T C T C T C Wave 1 Wave 2 Wave 3 Wave 4 HH Income Privileges Not Vulnerable Slightly Vulnerable Moderately Vulnerable Critically Vulnerable FY2017 FY2018 FY2019 FY2020 Graph 2: Annual vulnerability distribution for control and treatment 0% 50% 100% T C T C T C T C Wave 1 Wave 2 Wave 3 Wave 4 Child Protection Not Vulnerable Slightly Vulnerable Moderately Vulnerable Critically Vulnerable FY2017 FY2018 FY2019 FY2020 Graph 3: Child Protection trends 19 Pearson’s Partial Correlations12 were computed for this purpose. All correlations were statistically significant. The matrix in Table 10 below indicates that there is a correlation between Income Privileges and Assets (0.478). Intuitively, the strong correlation between income and assets reflects the positive logic of SOCY’s Theory of Change, especially regarding the SILCs. On the other hand, Income Privileges and Child Protection have an inverse and weak correlation (-0.12***), i.e., they move in opposite directions. This aligns with the earlier “SOCY Paradox” findings that indicate that Child Protection decreases with an increase in Income Privileges. Child Protection has much lower correlations with both Child Health (0.034**) and Assets (0.021). To be sure, the ET does not suggest that being vulnerable in terms of Child Health or Assets “triggers” sharper child disciplining practices. Rather, privileges that are income contingent bring forth a less child protective posture in caregivers. Table 10: Pearson's Partial Correlations Variable F1. Income Privileges F3.Child Protection F4. Child Health F2. Assets 1. F1. Income Privileges 2. F3. Child Protection -0.120 *** 3. F4. Child Health 0.203 *** 0.034 ** 4. F2. Assets 0.478 *** 0.021 0.101 *** Conditioned on variables: Wave * p < .05, ** p < .01, *** p < .001 Logistic Regression and Odds Ratio Next, using Wave 4 data, the ET examined the Graduation Odds Ratios associated with Income Privileges and Child Protection factors. Since Income Privileges and Child Protection correlated, the first model investigates Child Protection as a covariate with Income Privileges. Graduation Odds Ratios for Income Privileges and Child Protection were modeled separately as well. For all models, the reference class was the 0.25 ≥ group. 13 Recall, a lower value implies lower vulnerability. The odds ratios for each model indicate that the lowest vulnerable group was predicted to do better than all other groups (which increased in the level of specific vulnerability). None of the models below were statistically significant. Income Privileges (Child Protection as covariate) In this model, represented in Table 11 below, the probability to graduate from programming was almost 0.6:1. Including Child Protection indicates that the chance of graduating is higher with a lower vulnerability value. Interestingly, the odds of graduating are better in Group (0.26–0.5) and Group (0.51–0.75) compared to Group (0.25 ≥), implying that HHs that would be categorized as least vulnerable are less likely to graduate from OVC programming than somewhat vulnerable HHs. 12 Pearson’s Partial Correlation is a useful statistic here because it is created by the residuals of each variable—i.e., looking at the correlation between each variable’s uniqueness that cannot be explained or predicted by another variable. Controlled for wave. 13 This breakdown (0.25 ≥ , 0.26-0.5 , 0.51-0.75, 0.76 ≤ ) parallels HVAT graduation guidance. 20 Table 11: Income Privileges (Child Protection as covariate) Odds Ratio (Intercept) 0.592 Child Protection 0.246 Group (0.26–0.5) 0.829 Group (0.51–0.75) 0.695 Group (greater than or equal to 0.76) 1.915e -7 Child Protection In this model, represented in Table 12 below, the probability to graduate from programming was 0.44:1. The highest odds ratio is indicated by Group (0.51-0.75) of 0.5. Table 12: Child Protection Odds Ratio (Intercept) 0.441 Group (0.26–0.5) 0.453 Group (0.51–0.75) 0.521 An important point here is that none of these models should be taken as predictive—all models indicated no sensitivity nor specificity. This means that the variables in the model cannot accurately predict which HH does or does not graduate. Instead, the ET computed these odds ratios to explore a deeper relationship between Income Privileges and Child Protection. In fact, no significant correlation could be found and these concepts appear to move independent of the other. In summary, even though overall SOCY was not significant, there was something about participating in SOCY that increased HH Income Privileges but decreased HH Child Protection relative to HHs in the control group. This finding runs counter to most commonplace theories of change that argue increases to HH Income Privileges should correspond to increases to Child Protection, as stress factors associated with lower incomes are reduced. In the next section, this finding is restated in terms of a qualitative investigative question: Why do SOCY HHs experience more violence during times of economic strengthening relative to the control group? Three hypotheses that seek to explain this phenomenon are offered and empirically tested. Conclusion The SOCY IE sought to confirm, through a refined HVAT measurement tool, if participation in SOCY activities contributed to a reduction in HH vulnerability and improved OVC well-being. In doing so, the ET conducted four waves of annual survey data collection across 1,922 HHs, including an initial baseline (2017); subsequent extended baseline to confirm both control and treatment HHs travelled parallel, and comparable, trends (2018); a midline (2019); and an endline (2020). The ET confirmed that the parallel trends assumption held, i.e., the two groups were comparable. Yet the ET unfortunately found no significant difference between control and treatment group HHs. This lack of significance may have been due to potential contagion, 21 contamination and implementation infidelity issues caused by shifting PEPFAR priorities towards HIV suppression, or due to an overall lack of intervention potency. Further analysis also revealed that while there was no significant difference between control and treatment group HHs regarding the HVAT Assets and Child Health subdimensions, there were in fact significant differences between control and treatment group HHs in the Income Privileges and Child Protection subdimensions, i.e., treatment group HHs demonstrated positive improvements in the former even as they experienced declines in the latter. This “SOCY Paradox” will be explored further in the next section. Recommendations Given the challenges with evaluating PEPFAR-funded complex programming, USAID may wish to consider more nimble analytical tests better suited to managing complexity and supporting adaptive management than impact evaluations. 22 3. How is the OVC Program working to support HHs? While the impact evaluation determined whether or not the SOCY portion of the OVC Program worked, the results did not identify how the OVC Program achieved its associated outcomes (both positive and negative). Answering this “how” question is the focus of this section and addresses PE Question 2: What coordination and management mechanisms have worked best to support the integrated OVC implementation approach? What enabling factors do these mechanisms engage with? This section delves deeper into the counter intuitive findings presented in the previous section through an analysis of the different service mechanism the OVC Program deployed to reduce vulnerability. In doing so, the discussion expands the analysis to also incorporate data from BOCY, and zeroes in on the pathway of services taken to achieve a reduction in vulnerability results. As mentioned above and in previous reports, this definition in vulnerability shifted away from the HVAT and to today’s PEPFAR Graduation BNs, and the Year 2 Report deployed QCA to identify specific pathways of essential service combination to reach each BN. Consequently, this section explores the pathways to OVC Program results through the lens of the eight BNs and qualitatively investigates to what extent the suggested service pathways were experienced on the ground. OVC Program Graduation Benchmark Success The goal of the OVC Program was to improve the well-being of children and youth made vulnerable by HIV and other challenges in Uganda. Implementers sought to reduce these vulnerabilities through a suite of services across health, nutrition, education, child protection, psychosocial functioning and HH economic strengthening to form an integrated case management approach. OVC Program services were provided to HHs and beneficiaries through inhouse counseling, various group formations and support (youth, parenting, savings, etc.), various referrals (health facilities, law enforcement authorities, CSOs, case conferences, etc.), and various material support (temporary food support, transport support, school fees and supplies, etc.) mechanisms. BOCY and SOCY both reported impressive service outputs by the completion of the OVC Program. BOCY reported close to 120,000 beneficiaries served (55 percent of which were female), claiming 27,175 HHs serviced and 20,535 HHs assessed for graduation. Similarly, SOCY reported close to 140,000 beneficiaries served, claiming 38,531 HHs serviced and 25,946 HHs assessed for graduation. As discussed elsewhere, the bulk of these services were initiated and/or provided through HH visits, in which BOCY HHs received an average 9.3 visits across their enrollment and SOCY HHs received 7.5.14 However, despite these impressive numbers, less than half of BOCY and SOCY-assisted HHs managed to achieve graduation as per the current requirement to successfully pass all eight PEPFAR OVC graduation BNs. Table 13 below presents the overall distribution of HH achievements per BN and the overall HH graduation rate based upon achieving all eight BNs. 14 There are a variety of explanations for why BOCY conducted more visits, but one strong contributor to the difference is found in the earlier approaches to how graduation was determined. Prior to 2019, BOCY regularly conducted annual and semi-annual HH assessments to determine graduation, even as SOCY anticipated the use of third-party organizations to assess following pre￾graduation assessments, but this was never fully realized. The result, inter alia, was fewer home visits in the SOCY area. 23 As evident, the much smaller BOCY HH population performed relatively better than the twice as large SOCY HH population on all BNs. Yet, both SOCY and BOCY averaged around a 41–42 percent graduation rate, which may suggest that BOCY individual BN performance could have dropped with a larger HH population even as SOCY individual BN performance could have increased with a smaller HH population.15 Table 6 reveals the relative strengths of each activity. BOCY HHs were relatively successful at achieving BNs 1, 6 and 7 while they struggled to achieve BN 8. BOCYs success with achieving BN 6 is particularly surprising given the domestic violence (DV) challenges discussed in the previous chapter, but this will be discussed further below. SOCY HHs performed relatively the same across BNs 1–4, 6 and 7, and also struggled to reach BN 8. To further understand the nuances of these results, the ET also conducted comparative analyses with the initial HH BN deficiencies identified by each activity’s HH assessment tools (which determined eligibility for enrollment). When comparing initial enrollment BN deficiencies with final graduation BN deficiencies, the ET chose two metrics: • Comparisons of initial average HH enrollment BN deficiencies with those deficiencies that remained to prevent graduation to compare progress across the two activities. • Comparisons of the top three BNs failed at enrollment with the top three at graduation because it was useful to compare progress and persistent challenges within each activity. 15 It must be noted that these comparisons must be viewed as loose approximations of change, as each activity used enrollment criteria that was different from the other, and both activities’ enrolment criteria differed also from the graduation criteria (which was the same for each activity). As a result, any claims one activity outperformed the other cannot be substantiated. Table 13: OVC Program graduation BN assessment results SOCY Success (38,531 HHs) BOCY Success (20,535) BN 1: Know HIV Status 61.98 percent 97.93 percent BN 2: HIV Suppressed 62.31 percent 84.06 percent BN 3: Know HIV Prevention 61.57 percent 89.51 percent BN 4: Nourished 62.39 percent 89.74 percent BN 5: Stable Finance 46.65 percent 64.81 percent BN 6: No Violence 60.63 percent 97.23 percent BN 7: Not Child HH 61.42 percent 93.89 percent BN 8: Child in School 55.75 percent 83.58 percent Graduated 41.06 percent 41.82 percent 24 In regard to overall average deficiencies: Table 14: Comparison of average HH deficiencies: Enrollment vs Graduation Enrollment Graduation Improvement BOCY 2.56 1.71 0.85 SOCY 1.48 1.42 0.06 Difference 1.08 0.29 As can be seen in Table 14, BOCY HHs failed a higher number of BNs at their time of enrollment, i.e., HHs on average fell slightly above 2.56 BNs short of ineligibility to receive OVC Program support, while SOCY HHs fell 1.48 BNs short. Yet because each activity applied different BN definitions to their HH assessment tools for enrollment, the 1.08 BN difference between the two baseline deficiency measures is potentially misleading. By contrast, BOCY HHs failed graduation by a higher deficiency of 1.71 BNs than SOCY’s 1.42 BNs (i.e., 0.29 BNs more deficient) using the same graduate assessment tool, yet BOCY HHs demonstrated a noticeably greater overall improvement from enrollment to graduation compared to SOCY. It may in fact be easier to reduce a higher number of BN deficiencies when enrolling HHs that are more deficient, but the overall graduation rates for both activities suggests that the OVC Program as a whole struggled to meet all eight BNs, regardless of the starting point. Table 15 below presents the changes in the top three BN challenges from enrollment to graduation: Table 15 Comparison of top 3 BNs failed: Enrollment vs Graduation Enrollment Graduation Change in Challenge BOCY 5,3,8 5,2,8 2 replaced 3 SOCY 5,4,3 5,8,6 8 & 6 replaced 4 & 3 In Common 5 & 3 5 & 8 At the time of enrollment, BOCY HHs faced the highest challenges with Stable Finance (BN 5), Knowledge of HIV Prevention (BN 3) and Children in School (BN 8). By the end of the OVC Program, BOCY HHs remained most challenged by Stable Finance and third most challenged keeping Children in School, but Knowledge of HIV Prevention was less of a barrier to graduation and had been replaced by HIV Suppression (BN 2). SOCY HHs also faced the highest challenges with Stable Finance, followed by Nourishment (BN4), and with Knowledge of HIV Prevention. By the end of the OVC Program, SOCY HHs also remained most challenged by Stable Finance but staying Nourished and Knowledge of HIV Prevention had been replaced by keeping Children in School and Violence. Both activities were able to improve Knowledge of HIV Prevention services (which was primarily directed at youth), and SOCY improved on Nourished services, but both struggled to provide services that helped HHs become financially stable. Both BOCY and SOCY HHs found difficulties achieving Stable Finance. The challenges associated with BN 5 will be discussed below but it is useful to point out that a hypothetical exclusion of BN 5 from the graduation criteria would significantly increase both activity’s overall 25 graduation rate. More precisely, should BN 5 be excluded, the BOCY graduation rate would increase to 54.9 percent and SOCY to 50.3 percent. A quick summary of the above discussion includes the following points: • The OVC Program generated an impressive number of vulnerable HHs and individuals across a wide area of operation; • Both activities excelled at ensuring individuals were aware of their HIV status and improving knowledge of HIV Prevention; • Yet, overall, both activities graduated less than half of the HHs they supported; and • Overall, the biggest barrier both activities faced was ensuring that their beneficiary HHs were financially stable (this finding is discussed further below in the BN 5 section). The next section will present qualitative evidence of HH feedback on how the OVC Program was experienced, and will be organized according to each BN. Moreover, special attention is paid to BN 6 (Violence), due to its significance in helping stakeholders understand the paradoxical findings of the IE vulnerability subdimension analysis. HIV BNs The OVC Program BN performance was collectively strongest on the first three BNs related to HIV. The overall average performance of these three BNs exceeded the overall average of the remaining five BNs by more than 10 percent, which is not surprising given the increased programmatic focus on achieving the 95–95–95 goals. Indeed, the previous Year 2 Report’s QCA findings highlighted the key role HIV counseling, HIV literacy, ART adherence support, and health facility referral services played in achieving these results. This year’s 2021 final qualitative investigation confirmed, through the FGDs, that these findings were valued by HH beneficiaries. Counseling is the key to encouraging testing, ensuring adherence, reducing stigmas, and spreading further knowledge across both BOCY and SOCY. The importance of counseling ran across discussions of all three BNs, and oftentimes reflected carefully built trust originating with the PSW. Interestingly, PSWs were often seen as the crucial link between the HH and the facility, both in cases where OVC Program volunteers occupied a dual PSW and Village Health Team (VHT) role, and where they were representatives qualified to pass on relevant information to the VHT and health facility. Benchmark 1: Known HIV Status BN 1 focuses on identifying the first link in the 95–95–95 chain by encouraging HIV testing so that those who live with HIV know their status. BN 1 captures that all children, adolescents, and caregivers in a household have a known HIV status. This BN was met in 62.0 percent of SOCY HHs and 97.9 percent of BOCY HHs. Achieving this BN is partially met through self-reporting responses to the HIV risk assessment and, if deemed at risk, by documented testing evidence. The documentation serves as evidence. Despite numerous HIV testing media outreach campaigns launched by USAID, other donors and the Government of Uganda (GoU), the OVC Program’s case management system created a direct interface with HHs to further encourage testing and awareness. This often started with counseling to overcome fears of positive results and status disclosure. PSW confidentiality was reportedly the key factor encouraging disclosure, and FGD respondents indicated that they trusted that PSWs 26 would not willfully disclose their status to others.16 This counseling also helped overcome previous explanations of poor health offered by respondents, such as witchcraft, which often caused conflict within communities as individuals started accusing community members of bewitching them.17 Respondents emphasized that counseling was most effective when it occurred prior to testing, not after.18 It helped ensure messages related to HIV prevention were spread outside of the session, especially regarding youth. In 89 percent of the FGDs with female adolescents, there was a specific mention by at least one respondent related to receiving counseling to know one’s status in order to prevent HIV.19 Finally, the OVC Program’s approach to integrating testing with other services reportedly helped contribute to wider knock-on health benefits.20 The above discussion suggests that when considering the successful pathways to increased knowledge of HIV status, PSW counseling at the HH prior to testing plays a fundamental role. It not only helps encourage test taking by reducing stigmas and fears of positive results, but has spillover effects in terms of empowering further prevention and wider access to health services. Benchmark 2: Virally Suppressed BN 2 focuses on the remaining two links in the 95–95–95 chain by not only requiring that HIV positive beneficiaries receive treatment, but also that they adhere to the treatment enough to sustain viral suppression. This BN was met in 62.0 percent of SOCY HHs and 97.9 percent of BOCY HHs. Achieving this BN is partially met through self-reporting responses to the HIV risk assessment and, if deemed at risk, by documented testing evidence. The documentation serves as evidence. HIV suppression-related references were raised twice as often in discussions with SOCY HHs than BOCY HHs. Table 16 shows the frequency of discussions on topics related to adherence. 16 For example: ‘After receiving the counselling <> (PSW) gave me, I gained strength to carry on my life because I got to know it is normal to have HIV. I can live for years and years if my ARVs taken on time. I now take ARVs without any difficulty. Patrick ensures privacy when he has medicine to give us or when he needs to visit us. He will call you and meet you in a convenient place.’ (Kibaale, Female Caregiver); and ‘They [PSW] always keep people’s HIV status private from others. They only inform that person who is positive because there might be a situation where I annoy my sister. If she is aware of my status, she could end up announcing that I’m sick to the whole village.’ (Gombe, Female Adolescent) 17 As a female caregiver (Omoro) stated, ‘Testing helped us and our family members to know our HIV status. Those that were sick [HIV positive] did not know what they were suffering from. Now they are healthy and taking medication. Testing has reduced fights with our neighbors as we are no longer accusing them of bewitching us’. Another respondent commented, ‘Before people used to fall sick all the time and thought it was witchcraft. With the help of the PSW, people’s eyes were opened and they learnt that it wasn’t witchcraft but HIV. Up to now, they still go to hospital for HIV testing and take medicines,’ (Gombe, Female Caregiver). 18 As one female respondent from Lira stated, ‘In case someone is called to test without first going through counseling, it will create fear and at times discourage someone from going through with the testing. Counseling has helped that if after testing one finds out he or she is positive, there won’t be a lot of fear as could have been and will therefore start treatment.’ 19 ‘Getting tested and knowing our status made us gain more energy and encouraged us to protect ourselves through living responsible lives so that we are not able to contract the virus when we got to know that we are negative,’ (Gombe, Female Adolescent)’; and ‘It [HIV testing] has helped us know our status and helps prevent us from getting the virus,’ (Kabale, Female Adolescent);’ and ‘It is important to go for HIV testing and know your status because it helps you to protect yourself against HIV,’ (Rakai, Female Adolescent);’ and ‘After testing I start talking to my child so that I protect him/her from also getting HIV. I counsel my child and be open to him/her,’ (Lira, Female Adolescent). 20 For example: ‘SOCY helped me by testing all my family members. It helped us to be aware of our HIV status. It also helped immunize the young children and they were given birth certificates,’ (Bushenyi, Female Caregiver). 27 Table 16: BOCY/SOCY discussion frequency counts related to ART Adherence Program Case Type Frequency Count BOCY HIV Adherence 31 HIV Medicine 78 HIV Viral Load Testing 4 HIV Viral Load (Low/High) 3 SOCY HIV Adherence 73 HIV Medicine 150 HIV Viral Load Testing 11 HIV Viral Load (Low/High) 24 Total 374 Conversations about adherence were mentioned in 67 percent of BOCY and 58 percent of SOCY FGDs. Most respondents reiterated that medicine needs to be taken regularly and consistently. 21 Respondents described how disclosure to family and friends played an important role in reinforcing the message of adherence.22 Yet by far, FGD respondents indicated that the PSW worker played the strongest role by just ‘reminding us to take our medicine,’ (Bugiri, Female Caregiver). This was both in terms of reminders to regularly take the medicine but also to refill the medicine. 23 Other caregivers reported that they take their medicine regularly as they do not want to feel shame when the PSW conducts a follow-up visit, 24 and that PSW reminders also help them feel respected and looked after. 25 Finally, there were limited mentions related to the difficulties of adherence during the COVID-19 governmental lockdown, especially restrictions on public transportation, and the instrumental role the PSWs played in helping those who required refills of medicine to receive them.26 Viral load (VL) testing was less prevalent in BOCY FGDs whereas it was more prevalent throughout all discussions in SOCY. For BOCY, it was discussed in the context of an absence of PSW support in which respondents indicated times when community members pooled 21 For example: ‘Know your status and if you find out that you have the virus, one always needs counseling to let them know that they have to swallow tabs on time,’ (Gombe, Female Adolescent) and ‘Now after the counseling, we take our medicine and on time,’ (Bushenyi, Male Caregiver). 22 For example: ‘It’s very important to tell your family about your status. It is very important because they will always remind you [to take your medicine] and you don’t have to hide anything,’ (Gombe, Female Caregiver) and ‘My husband now days takes care of me and reminds me to take medicines,’ (Kibaale, Female Caregiver). 23 For example: ‘Reminders help us stay healthy because there are times when we forget. With the reminders we are able to know when to take medication and also when to go pick our drugs,’ (Kole, Female Caregiver) and ‘The reminders help in a way that we do not forget the days when we are supposed to take our medication and how to take them,’ (Lira, Male Caregiver). 24 For example: ‘The reminders helped us not to forget taking our medicine because when the PSW finds the medicine that should have been finished in a given period of time it is so ashaming,’ (Omoro, Male Caregiver). 25 For example: ‘[The PSW] reminding me to take medicine made me feel loved and that someone really cares. This makes me work very hard at taking my medicines on time so I don’t disappoint the PSW,’ (Kibaale, Female Caregiver) and ‘When the PSW visits you, you feel so loved and cared for. His [PSW] counselling on how to live a positive life brings a positive energy,’ (Omoro, Male Caregiver). 26 For example: ‘This was made possible because of the transportation facilitation they were getting. PSWs even brought us ARVs when we were in the COVID lockdown as we were stranded and did not have any transportation in order to get to the hospital. They helped us,’ (Kibaale, Female Caregiver) and ‘They did not come back but referred us to hospital now we always go back after 6 months to test us and do a CD4 count. They even used to bring us medicine in Orona when we had no means of transport but the only challenge is that there is sceptrin at the hospital we have to buy from drug shops. And we have no option because it is very important to our health; it heals malaria, reduces chances of getting T B and our health,’ (Kibaale, Female Caregiver). 28 transportation resources so one person could combine VL testing with medicine refills for other community members who needed them. 27 Caregivers commented on helping neighbors understand why some children need medicine to remain healthy.28 By contrast, SOCY respondents typically reiterated their understanding of and compliance with the OVC Program protocols.29 Conversations on adherence also included comments on the role caregivers play in helping positive children follow a treatment plan with the correct medicine.30 The above discussion suggests that when considering the successful pathways to HIV suppression, PSW counseling once again plays a fundamental role. Here, PSWs served as a familiar nudge to adherence, which was often interpreted as a sign that the lives of HIV positive beneficiaries are respected. The above discussion also suggests the family and friend networks further contribute to adherence. Critically, both treatment enforcement arms further define the local HIV positive community as able to coordinate joint VL testing and medicine refills, even in times of government restriction such as during COVID-19, while community adherence structures are likely to continue without PSWs. Benchmark 3: Knowledgeable About HIV Prevention BN 3 focuses on preventing the transmission with a specific reference to adolescents aged 10–17. This BN was met in 61.6 percent of SOCY HHs and 89.5 percent of BOCY HHs. Achieving this BN required the respondent to successfully describe prevention strategies and risks. These responses serve as evidence of knowledge. Discussions with HH respondents stressed the importance of preventing HIV for individuals of all ages. Table 17 shows the frequency of discussions on topics related to prevention with 60 percent and 63 percent (BOCY and SOCY, respectively) of the HIV prevention related to prevention in general. 27 For example: ‘We no longer miss out [HIV] medicine because even if you don’t have transport the one who is going for CD4 counting does that. If she doesn’t have transport we contribute,’ (Gombe, Female Caregiver). 28 For example: I was able to change my neighbor’s mind. He used not to take care of his child who is HIV positive but now he takes him to the hospital for medication and the boy is now healthy and looking good,’ (Omoro, Male Caregiver). 29 For example: ‘I was taught to follow and take proper medication which has helped my viral load to go low,’ (Bushenyi, Female Caregiver) and ‘Keeping proper medication and following instructions has strengthened their [HIV positive people] lives because they always go back for check-up after three months to make sure their viral load has decreased,’ (Kabale, Male Caregiver). 30 For example: ‘I was taught that when I have a positive child, I have to take good care of him or her by giving them medicine on time and feeding them well,’ (Omoro, Female Caregiver) and ‘If you know your child is positive, you start taking good care of them by giving proper medication on time to keep them healthy,’ (Bushenyi, Male Caregiver). 29 Table 17: BOCY/SOCY discussion frequency counts related to HIV/Aids Prevention Program Case Type Frequency Count BOCY HIV Prevention (General) 36 HIV Prevention (Abstinence) 7 HIV Prevention (Condoms) 3 HIV Prevention (Faithfulness) 9 HIV Prevention (Mother to Child) 5 SOCY HIV Prevention (General) 64 HIV Prevention (Abstinence) 11 HIV Prevention (Condoms) 12 HIV Prevention (Faithfulness) 12 HIV Prevention (Mother to Child) 2 Total 162 PSWs were mentioned in eight FGDs31 as providing information on how to prevent HIV.32 PSW’s were largely responsible for teaching several ways of preventing HIV.33 One respondent was clear to suggest that one’s status should not matter in conversations around prevention as the information is useful to all.34 Abstinence as a means of preventing HIV was mentioned most commonly in adolescent discussions during OVC Program group meetings (five out of nine), especially Sinovuyo sessions.35 Adolescents were quick to connect abstinence not only to preventing HIV but also to avoiding early or unwanted pregnancy.36 Under abstinence as a means of prevention, there was only one reference to DREAMS.37 Condoms were noticeably less discussed across the FGDs and viewed as a source of HIV prevention mainly by male caregivers,38 and only two mentions, from the same district, were towards having access to free condoms through the SOCY program.39 The remaining mentions were made more in passing and did not make any connections to the OVC 31 Five female caregiver discussions (50%); two male caregiver discussions (20%); and one female adolescent discussion (11%). 32 For example: ‘They [PSW] have taught us how to live when HIV positive and prevent spreading the virus,’ (Bushenyi, Female Caregiver) and ‘The PSW and LC1 gathered the whole community, both males and females, to teach them how to manage and prevent HIV virus,’ (Tororo, Female Caregiver). 33 For example: ‘Josephine [PSW] taught us how to prevent the spread of HIV by being mindful of things like razor blades and that they shouldn’t be shared, to always use condoms or be faithful to our partners,’ (Gombe, Female Caregiver). 34 She stated that prevention involves ‘all of us: the positive and the negative,’ (Gombe, Female Caregiver). 35 For example: ‘They taught us that we should avoid sex at an early age because one might contract HIV,’ (Gombe, Female Adolescent) and ‘Sinovuyo helped us to protect ourselves most especially in not engaging in sexual activities,’ (Omoro, Female Adolescent). 36 For example: ‘(Sinovuyo) helped to reduce on the spread of HIV and also early pregnancies in girls,’ (Rakai, Female Adolescent). 37 For example: ‘DREAMS helped our children to be focused and change the way they behave and talk to their parents. They are now more into books and helping parents and they no longer sneak to boys houses so now they are free and safe from HIV,’ (Gombe, Female Caregiver). 38 For example: ‘The use of condoms has reduced on the spread of the HIV virus in our community,’ (Kabale, Male Adolescent) and ‘People are now protecting themselves through the use of condoms,‘ (Gombe, Male Adolescent). Condoms were also identified by adolescents as a means to ‘protect ourselves,’ (Omoro, Male Adolescent). 39 These were: ‘Sometimes, VHTs would give us condoms or distribute in our communities especially the busy places like guest houses to enforce prevention against contracting the virus,’ (Kibaale, Female Adolescents) and ‘They gave us condoms at Kagadi Hospital so that we can protect ourselves,’ (Kibaale, Male Caregiver). 30 Program or the case management system, save two comments relating to knowledge about Mother to Child transmission during wider awareness trainings. 40 The above discussion suggests that when considering the successful pathways towards increasing knowledge of HIV prevention, the PSW still plays a role. However, OVC Program youth group sessions, like Sinovuyo, may play a stronger role, and reiterate a general consensus among youth respondents that abstinence is the preferred prevention method. Benchmark 4: Not Undernourished for Children 5 Years or Younger BN 4 focuses on preventing malnutrition in young children. This BN was met in 62.4 percent of SOCY HHs and 89.7 percent of BOCY HHs. The previous Year 2 QCA results suggested that in cases of critical vulnerability, nutrition education combined with HH economic strengthening services were sufficient to meet the BN. Achieving this BN requires Mid-Upper Arm Circumference (MUAC) measurements and visually confirmed lack of bipedal edema. The documentation and visual inspection serve as evidence. BOCY had a higher reference to nutrition (53 percent) compared to SOCY (29 percent). FGD discussions on nutrition did not find the OVC Program’s nutrition training strongly relevant, and most respondents assumed they already provided a balanced diet to children under five. No respondent connected the OVC Program to Early Childhood Development (ECD) or malnourishment services. In some cases, FGD participants who participated in the Backyard Gardens education provided by community development officers (CDOs) made direct connections to nutrition. 41 But respondents were more keen to discuss nutrition in the context of HIV suppression, as they conveyed that learning about nutrition was of more importance for HIV positive individuals in order for them to increase their health and support their immune system. 42 There were also linkages to how nutrition could help strengthen an HIV positive person’s immune system. 43 Otherwise, most FGD respondents viewed backyard gardening also as a means for HHs to earn income. 44 Indeed, some respondents indicated backyard gardening’s purpose was solely for income. 45 As such, most OVC 40 These were: ‘The training helped breastfeeding mothers from transferring the virus to the child,’ (Lira, Female Caregiver) and ‘They taught me how am supposed to treat myself so that I don’t spread HIV to my husband and also give birth to HIV negative children,’ (Rakai, Female Caregiver). 41 For example: ‘Greens have helped me improve my health. I am now strong and can do my work well,’ (Male Caregiver) and ‘When you have your backyard garden with greens, you can get some and mix in the sauce to avoid malnutrition in the household hence promoting good health,’ (Female Caregiver). 42 For example: ‘PSWs helped us realize that when we eat well, our immune system is boosted,’ (Lira, Female Caregiver) and ‘Having a balanced diet helps build our immune system because even if one takes the medication with poor feeding, it might not work out the way it’s supposed to. It also gives someone energy and we are able to work now,’ (Gombe, Male Caregiver). ‘You can’t treat yourself without eating because even in the hospital they make you eat before taking some medication. Food is fuel to the body and makes the body strong,’ (Bugiri, Male Caregivers). 43 ‘For those whose viral load was not suppressed they would be supported to boost their nutrition status. This is done by giving them seeds (cabbage, sukuma wiki—greens) to plant which would later provide nutritious food,’ (Kibaale, Female Adolescent). 44 For example: ‘Greens helped us increase our CD4s, boosted our immune system, and financial status because we grow, eat some and sell some,’ (Gombe, Female Caregiver) and ‘Backyard gardening has helped us in home consumptions and sell to get some money to save,’ (Bugiri, Female Adolescent). 45 ‘After teaching us [backyard gardening] my family can get some money from it which has helped in providing basic needs in the home,’ (Bushenyi, Male Caregiver) and ‘When we grow and harvest our crops, we sell them and get money. In case we need anything, we use that money to help ourselves,’ (Kabale, Female Caregiver) and “We started backyard gardening and from it were able to earn some money. I have gotten enough money to complete my house thus providing shelter to my family,” (Bushenyi, Female Caregiver). 31 Program HHs that achieved this BN most likely did so through the privileges afforded by improved HES. The above discussion suggests that when considering the successful pathways to HH nourishment, backyard garden education appears to be a viable option. FGD feedback certainly demonstrates that the approach is a viable and effective income supplement. Yet, given that the overall achievement rate for this BN could benefit from improvement, follow-on interventions must remain mindful of the tendency for beneficiaries to prioritize the sale of produce or use it to primarily address adult and HIV positive health concerns over early child health concerns. This tendency should be addressed in future interventions. Benchmark 5: Improved Financial Stability BN 5 looks at increasing the overall financial well-being of HHs so they are able to cover school fees and any medical costs without selling assets, such as livestock, or relying on external grants or cash transfers. The previous Year 2 QCA results suggested that savings group attendance, combined with financial literacy, and income generating groups like Backyard Gardens and Sinovuyo, comprised a sufficient combination of services to achieve this BN. Interestingly, the previous analysis also found that other youth group initiatives such as apprenticeships, enterprise clubs, and agro/small business groups were not relevant to achieving this BN. The BN is verified through self-reporting and no other evidence is required, subjecting verification to respondent bias. This BN was by far the most difficult challenge for the OVC Program to achieve. It was met in 46.7 percent of SOCY HHs and 64.8 percent of BOCY HHs, and was the biggest deficiency facing most HHs at enrollment. Below is a discussion on general findings as well as a presentation of evidence that the operational definition of this BN is problematic and should be reconceptualized to be a valid proxy of invulnerability. The most effective service to advance financial stability offered by the OVC Program involved the formation of savings groups. BOCY relied upon a more traditional village savings and loan association (VSLA) model while SOCY relied upon the Catholic Relief Services (CRS) signature “Savings and Internal Lending Communities” (SILC) model. For the FGD respondents, savings was viewed as the most important factor in helping promote HH financial stability, with 42 percent of BOCY and 61 percent of SOCY mentions of financial stability involving positive references to how savings increased the HH’s ability to manage expenses. As the ET expected, FGD respondents confirmed that OVC Program efforts to increase a HH member’s knowledge of budgeting helped promote the concept of saving for the future. 46 In many cases, participation in OVC Program saving schemes provided a financial cushion that helped 46 For example: ‘We used to spend money recklessly without thinking, we would work and spend all of it without saving nor minding about tomorrow, but after the literacy training, we learnt how to save and plan for our money,’ (Omoro, Male Caregiver), and ‘SOCY taught us to see things from far not just from here. I used to only save for not just for festive season but after joining SILC, I know I can save for fees, basic needs and any future emergencies,’ (Kabale, Female Caregiver). “The project lifted me from nothing to something. I used not to have a roofed house and my grandchildren were not able to go to school. Now I am paying school fees for my grandchildren. It is easy since I was taught how to save. From my savings I roofed my house and also paid my grandchildren’s school fees. And now they are able to attend school,” (Lira, Male Caregiver) and “We used to face a lot of challenges before saving since we had no idea of how to and when to start but now, we can even save our children’s school fees. There is no village that does not have a savings group now which was not there before. When they just started informing us, we used to see it as something useless but now we have realized its benefits since we can now pay up our children’s school fees so we have seen that SOCY has really helped us,” (Gombe, Male Caregiver). 32 prevent the sale of assets to cover the emergency costs.47 Conversely, many FGD respondents painted dire pictures of vulnerability and exploitation had they not participated.48 Perhaps the most powerful examples came from adolescent FGDs, all of whom reported how access to these funds helped reduce incidences of transactional sex. 49 School Fees and Livestock The two most widely reported uses of savings group loans were for school fees and livestock. Regarding the former, many male and female caregivers prioritized savings and loan groups as a primary means to help them afford school fees.50 Yet the relationship between the payment of school fees and livestock is complex, and can neither be assumed to reflect increased prosperity nor poverty. Indeed, FGD respondents indicated that an increased ability to purchase (and sell) livestock was a key motivation for participating in a savings group, both as a means for paying school fees as well as an end in and of itself. 51 To be sure, there was limited mention of selling livestock as a means of emergency response (which reinforces the operational conditions needed to fail this BN),52 and the ET would suggest that the selling of the last female of the stock probably better signifies when HHs sell livestock under emergency conditions. But the volume of FGD data suggests that using savings groups to purchase livestock to later sell to pay for school fees is not an anomaly or a sign of vulnerability. Instead, it can be seen as a privileged practice exercised by HHs with more diverse income streams, especially those with higher levels of financial literacy who save, invest and value education. OVC Program savings groups have been key to enabling this process, which, ironically, disqualifies a HH from gradation as BN 5 cannot be met when HHs sell livestock to pay for school fees. 47 For example: ‘Under VSLA we have been able to get loans which we use during difficulties most especially during financial problems,’ (Kibaale, Male Caregiver) and ‘When we get problems we don’t panic because we know that we can get money,’ (Omoro, Female Caregiver). 48 For example: ‘We would have been very poor, and still borrowing salt from neighbors,’ (Omoro, Female Caregiver) and ‘It would have been worse because our boys and girls would have married off at an early age but when SOCY came, we are able to borrow money and sustain our children in school,’ (Kabale, Female Caregiver). 49 ‘When you have money, you can’t be tempted to sleep with men in the community so this have slowed the spread and re-infections because we are busy and can afford food and a few things we need in life,’ (Kibaale, Female Caregiver) and ‘At any time, a girl can easily be deceived by a man with just a hundred thousand Shillings. Therefore, through the training they opened up our mind to saving and helped us realize that patience is key,’ (Gombe, Female Adolescents). 50 For example: ‘Through VSLAs, we are able to save and borrow money which we use to pay up our children’s school fees. Also, we get them scholastic materials and this has helped keep them in school,’ (Kole, Female Caregiver) and‘(SILC) taught us to work hard in terms of savings. We save per week so money accumulates at the end of the year we share the amount which helps us in paying fees,’ (Bushenyi, Male Caregiver). ‘Through the group formation we have been able to save up money which has enabled us send our children to school,’ (Kole, Female Caregiver) and ‘Children’s school fees became very easy to pay because when you are saving you can even pay in advance and still buy other necessities,’ (Gombe, Female Caregiver). ‘It [SILC] taught us how to save and also take children to school through paying school fees,’ (Bushenyi, Male Caregiver) and ‘I was able to learn how to save. If I have saved up my money it can help me send my child to school at the right time,’ (Lira, Male Caregiver). 51 For example: ‘We learnt how to make saving groups, how to save [SILC], and how to manage day to day life. We now save and are even able to pay school fees on time. We bought things like pigs and goats from our savings hence developing our homes and improving our standards of living,’ (Gombe, Female Caregiver) and‘They were given chickens for rearing and they laid eggs and multiplied. Over time they exchanged for goats. They reared the goats and when they multiplied, they exchanged to get cows. When the cows multiplied, they sell for family upkeep, school fees and other necessities,’ (Bugiri, Female Adolescent) and ‘It has helped in a way that parents are now rearing animals like pigs and goats to sell the animal, gets money and pays for the child school fees,’ (Lira, Male Caregiver). ‘Through the skills training my child acquired knowledge in saving. He bought a goat and is now animal rearing. He normally sells them at a higher price and gets profits which he uses to provide himself with supplies for school,’ (Bushenyi, Female Caregiver) and ‘I received goats. And when they reproduced it supported me in paying school fees for the children and upkeep for the family. Some of my children had to drop out of school but with the goats venture I was able to send them back to school,’ (Bugiri, Male Caregiver). 52 For example, an adolescent female in Bugiri put the sale of goats in this context: ‘When I withdraw from my VSLA I buy goats to rear. When there is a problem [financially] I sell. I also buy hens and subsequently sell when there is a problem.’ 33 The above discussion suggests that when considering the successful pathways to financial stability, OVC Program savings groups—both SILCs and VLSAs—can be viable options that improve financial literacy, promote longer-term planning, provide emergency response resources, and otherwise cushion against vulnerability. A key aspect of this model involves investment in livestock to sell at a later date to pay for school fees, and this practice continues as both a sign of prosperity and poverty. As a result, PEPFAR’s BN 6 verification requirements can in fact confuse failure with success. Future interventions should consider more nuanced and/or culturally calibrated operational requirements for this BN, and proxies such as the sale of the last female in the stock may help further define meaningful differences. Benchmark 6: No Violence Reported BN 6 looks at decreasing violence in HHs. This BN was met in 60.6 percent of SOCY HHs and 97.2 percent of BOCY HHs. The previous Year 2 QCA results suggested that PSW psychosocial and GBV counseling, combined with better parenting training, Aflateen and Sinovuyo, comprised a sufficient combination of services to achieve this BN. Interestingly, the previous analysis also found that reporting violence to the police, and special initiatives like Journeys and Stepping Stones, were not relevant to achieving this BN. Achieving this BN requires self-reporting on a lack of physical and sexual abuse in the past six months. This self-reporting is subject to high levels of respondent bias; however, external evidence of police or social service reports of violence can override these reports, as can a reluctance to answer the questions. FGD respondents connected reductions in DV to OVC Program services through HES activities and PSW counseling. Female respondents especially mentioned the important role having an income generating activity has in helping reduce rates of DV as does occupying their time with productive activities.53 SOCY respondents also specifically connected participation in SILC to reduced DV against women. 54 Furthermore, several male respondents also understand that a large portion of DV was the result of poverty. 55 The PSW’s training on DV has been seen as instrumental in helping reduce domestic violence. Couple’s counseling sessions, where the PSW interacted with both the male and female partners, helped both partners relate to one another, but more importantly to address their differences in a socially acceptable manner. 56 The PSW was also viewed by FGD respondents, especially in BOCY, as an authority that helps with counseling 53 For example: ‘You get what to do which has kept us away from domestic violence,’ (Bugiri, Female Caregiver) and ‘These activities have kept us from causing domestic violence like going to the garden and dig,’ (Bushenyi, Female Caregiver). 54 For example: ‘Domestic violence reduced after starting to save because we would borrow money from our SILC group to fund fees for our children and also save for other personal needs. We no longer ask for money from our husbands because it would annoy them a lot and they end up beating us up,’ (Kabale, Female Caregiver) and ‘Before SOCY, we had a lot of domestic violence even the saving groups didn’t lend us any money but after joining SILC, we started borrowing money and now my husband realized there is a change because I can also provide food in the house, hence reduced domestic violence,’ (Kabale, Female Caregiver) and ‘There would be domestic violence in homes. For example if a mother uses money carelessly the father would get annoyed, quarrel and even beat her. But now we were able to teach our parents what we were taught by SOCY about saving and budgeting,’ (Rakai, Female Adolescent). 55 For example: ‘SOCY helped us to reduce domestic violence because at first when a woman would ask for soap and you don’t have it [money] you would just beat her up but after the training especially on income, it helped us to reduce the domestic violence,’ (Bushenyi, Male Caregiver). 56 For example: ‘When married couples are counselled on domestic violence, it helps to reduce domestic fights hence reducing domestic violence in homes,’ (Bushenyi, Male Caregiver) and ‘The PSW gathered us and came with the police officer who taught us about that law and told us to prevent and stop domestic violence,’ (Omoro, Female Caregiver). 34 and addressing cases of DV, and as pivotal in helping scale up cases to other levels of support if required.57 The BN 6 achievement data suggests that BOCY HHs do a better job of mitigating violence than SOCY ones. However, the ET would strongly suggest the reader approach this finding with caution, as issues of DV are highly sensitive topics that are seldom discussed openly during FGD and other similar settings. In fact, BOCY often used a SW to confirm this BN, even though the majority of DV issues were captured and recorded by the PWS (who more regularly interacted with the HH). To support this argument, the ET found, through a comparison of BOCY’s home visit reports with the graduation reports, that 92.6 percent of HHs who had successfully met BN 6 according to graduation assessments records conducted in 2020 had in the past six months received psychosocial counseling from the PSW for issues related to abuse in the HH. Had this evidence been applied to the BOCY BN 6 data, more than 90% of BOCY HHs would not have met this BN. 58 The implications are that accurately confirming BN 6 is extremely difficult when more distant professionals and investigators are the ones administering the assessments. Additional evidence to help resolve the SOCY Paradox FGD respondents indicated that DV was quite common in their areas, although most respondents referred to other HHs instead of theirs. Interestingly, FGD respondents were quick to point out that violence occurred on both sides of the gender divide. Roughly a quarter (23 percent) of the causes of DV discussed were where males suggested that DV is not only caused by them but that females are also perpetrators. 59 This was also raised in several SOVCC discussions. 60 From the above, it should be clear that large sections of the OVC Program HH population benefited from the various psychosocial and HES services, and that, in most cases, improvements brought on by sustainable income practices like savings are beneficial and contribute to reductions in DV. How then can the counter intuitive findings of the previous chapter regarding the relationship between income improvements and Child Protection declines be explained? While the ET did not discover a simplistic, singular answer to this puzzle, three arguments are offered that, when combined, may partially resolve the alleged paradox: 1. FGD feedback suggests a heightened awareness to report violence; 2. Men responded to women’s empowerment with violence; and 3. Higher alcohol consumption resulted from OVC Program HES and lead to more violence. Increased awareness leads to more reporting of violence Another explanation for the rise in income/decline in Child Protection findings may involve a reported increased awareness of the harmful effects of violence, which may also result in an 57 For example: ‘Domestic violence is always solved in the community by calling the two partners for counseling by the help of the Para Social worker, vice chairperson, and local council one,’ (Tororo, Female Caregiver). 58 Further conversations with BOCY staff revealed that a comparison between the graduation assessment and home visit records had not been conducted, but staff suggested that most graduation assessments were conducted by SWs while home visits by PSWs. 59 For example: ‘MUCOBADI has a challenge whereby some women torture their husbands in a household and those men fear to report that my wife has bitten me. So. l am requesting MUCOBADI to also sensitize women to stop torturing the man in their households,’ (Bugiri, Male Caregiver) and ‘Women always become dictators in a home, making a man look useless in a home,’ (Kabale, Male Caregiver). 60 For example: ‘Sometimes you find that a man is being domestically violated at home by his wife. Such men often times feel shy to report their women to the authorities for fear of being stigmatized by fellow men and the rest of the community. So, they pile up anger and depression from continuous abuse and end up committing suicide, (Kabale, SOVCC). 35 increased willingness to report instances.61 While this explanation may run counter to the positive reporting bias suggested above, the recent Literacy Achievement and Retention Activity (LARA) evaluation found that training to reduce violence against children also resulted in higher instances of reporting. Indeed, SOCY and BOCY FGD respondents frequently reported an increased awareness of the negative effects of violent parenting prior to OVC Program training, and recognition that hitting and shouting a child are no longer positive methods of discipline. 62 Some respondents even went so far as to connect the abuse with emotional damage. 63 Adolescents often commented on how the training improved parental awareness of the consequences of mistreating and/or abusing children. 64 Adolescents also reported a change in the way parents disciplined. 65 To the extent the OVC Program facilitated an increased awareness of the negative aspects of child abuse, contributing to an increase in reporting of incidences of violence, the paradoxical reduction to Child Protection in the context of increased HH income is partially explained. As HH incomes increase as a result of participation in savings group programs, they are also more exposed to better parenting training and social pressures, which compels them to report more incidences of violence when they previously would not have. Men respond to women’s empowerment with violence While much of the FGD respondents’ testimonies focused on the positive psychosocial and HES benefits of OVC Program services, the program focus on women’s empowerment had the potential to increase violence.66 Even though income generating activities enabled the women to spend the money, control over financial resources and shared decision making are important in reducing violence. 67 Yet, despite select testimonies, few female participants made these connections. 61 For example: ‘During the parental training we were taught and encouraged to praise our children when they do something good. Before I used not to praise my children and this did not please them. I have a six-year-old who used to sweep the compound and instead of praising and thanking her I would just scold her which always made her feel bad. Nowadays I praise her and this encouraged her to even sweep the compound better,’ (Lira, Male Caregiver) and ‘I really appreciate BOCY so much because before I knew nothing about early childhood development. I benefited a lot because I learnt that as a mother I am supposed to know how to communicate with my child because this helps the child’s language develop. I used to think children have no emotions and hence cared less. I now know that I have to give them time and most especially when they are playing and so that I can learn what they enjoy doing best,’ (Lira, Female Caregiver). 62 For example: ‘They taught the mothers to avoid shouting to the young children, avoid child labor, and always give the child enough time to play with others,’ (Tororo, Female Caregiver) and ‘It helped me to control my anger especially towards the children. I didn’t think it mattered when I hit them. Even now when they annoy me, I give them a chance to explain themselves,’ (Rakai, Male Caregiver). Reference was also made towards no longer using forms of abuse as a way to discipline: ‘We learnt how to properly handle children, talk to them and punish them if we have to in a proper way but not beating them or denying them food,’ (Gombe, Female Caregiver). 63 For example: ‘When a child makes a mistake and you behave harshly on them through harsh beating, he/she may feel emotionally tortured,’ (Rakai, Male Caregiver) and ‘There are abusive words that we [parents] used to use that affects their [children] minds which we said without realizing. The teachings in Sinovuyo helped us a lot, in learning that there are words that we are not supposed to speak before children as they may poison their minds. We also learnt that we must settle our conflicts between us parents without the child knowing,’ (Bugiri, Female Caregiver). 64 For example: ‘Our parents were given awareness on the dangers and outcomes of mistreating their children,’ (Gombe, Female Adolescent). 65 For example: ‘Parents now talk to us calmly even when we make mistakes which has helped us to concentrate,’ (Omoro, Female Adolescent). 66 As one female respondent from Lira said, ‘There has been some domestic violence caused by the trainings because some men take it that they have been left behind and sometimes they feel jealous.’ Similarly, a female respondent from Omoro said, ‘Men feel a bit jealous but don’t beat women when the PSW visits because they fear the PSW will report such things.’ 67 For example: ‘We also work and have our own money but the men would force us to give them the money we have worked for because they are the heads of the households,’ (Gombe, Female Caregiver). The discussion around women not having control over 36 Rather, when ET qualitative investigators probed if some men felt threatened by their empowerment, respondents were more likely to fall silent. In fact, one FGD in Rakai explicitly stated they did not want to discuss this matter and suggested ending the session early. While a variety of explanations for the avoidance of the DV topic are plausible, mistrust and fear of repercussions are quite likely. Higher alcohol consumption results from OVC Program HES and leads to more violence PEPFAR’s preference to provide services to primary caregivers but not heads of HHs builds in a potential gender bias in which women receive violence reduction services while men remain focused on increased incomes. Many end-of-cycle savings group “payouts” involved celebrations in which the main benefactors (and loan takers) invited friends to “drink full”. In Omoro, respondents recalled how VSLA members regularly had get-togethers when they earned income, especially the men, and connected this to DV. 68 To the extent men continue to use the proceeds from HES to support alcohol consumption but do not attend additional violence reduction and Child Protection training, the associated violence is likely to continue. The implications of this line of evidence are that all HES programs need to anticipate a likely increase in alcohol consumption and related violence as an unintended side effect of improved incomes. Savings group payout celebrations may further contribute to these unintended consequences. More problematic, recent PEPFAR emphasis on primary caregivers and the exclusion of other adults, especially heads of HHs, may also result in the systematic exclusion of males from the more positive effects of financial literacy and better parenting. This training imbalance may further contribute to increases in violence and reductions in Child Protection. The above discussion suggests that when considering the successful pathways to non-violence, a very complicated picture of success and struggle emerges. On the one hand, the OVC Program’s various parenting, HES and PSW counseling undoubtedly provided psychosocial benefits, improved family relations, and increased Child Protection. On the other hand, collecting reliable data on sensitive topics such as DV is not easy and is prone to bias. Similarly, some beneficiary training may have contributed to increased reporting of violence (when it would not have previously been reported) but the longer-term effects of positive behavior change had not yet occurred. Regarding what has been framed as the “SOCY Paradox” (but likely applies to BOCY as well), savings groups and other income improvements, while beneficial, may unintentionally contribute to more violence as alcohol consumption increases. Finally, males may respond to HES improvements, especially when led by females, with violence as a means of resisting female empowerment. PEPFAR’s prioritization of primary caregivers may further exacerbate these tendencies as men are systematically excluded from the benefits and jealously respond with violence. The implications are that all HES and female empowerment programming must resources was raised in the SOVCC FGDs. ‘Those are challenges we face too because a man can realize that her wife’s pig has grown and he sells it off without her consent,’ (Kibaale, SOVCC) and: Participant 1: ‘In Buganda culture, a woman is not supposed to own something. When a woman rears her pig, it’s the husband to sell….’ Participant 2: ‘Husbands don’t participate in rearing.’ Participant 1: ‘…and when the pig grows up, ownership turns to the man and disagreements spring from there and when the man beats the woman, the GBV case emerge,’ (Gomba, SOVCC). 68For example: ‘The men would go get drunk, come back home and start a fight with the wife and children and ended up beating each other,’ (Omoro, Female Caregiver). 37 seriously consider violent reactions to improved incomes and women’s empowerment as a likely, yet unintended, outcome of social improvement. Benchmark 7: Not in a child-Headed Household BN7 focuses on confirmation that an adult head of household is present in HHs, which is assumed to reduce vulnerability more than child-headed HHs. This BN was met in 61.4 percent of SOCY HHs and 93.9 percent of BOCY HHs. The previous Year 2 QCA was unable to identify specific services that directly support this indicator and rather assumed that meeting this BN oftentimes was the result of positive spillover effects of meeting the other BNs. Achieving this BN requires self-reporting that an adult has cared for and lived in the HH in the past 12 months. No other evidence is required and the self-reporting is subject to respondent bias. Unfortunately, there was no mention of child-headed HHs in any of the FGDs but there were several references towards orphans being taken in by caregivers, often in the context of why their HH was receiving BOCY/SOCY support. 69 However, FGD respondents did identify adult abandonment of children as a frequent practice, and in Omoro pointed to the positive benefits of BOCY’s parenting training as a mitigating means. 70 Otherwise, conversations with BOCY and SOCY staff revealed that the most common practice involved providing financial support through income generating activities and economic group activities to sustain the child-headed HH until the eldest came of age. While the OVC Program encouraged adult support to OVC-headed HHs, the demographic factors involved presented challenges in meeting this BN. Nevertheless, BOCY and SOCY services that sought to improve child-headed HH incomes were a viable mediation strategy to potentially reduce vulnerability. Follow-on interventions may wish to consider the development of intermediary BN measures that distinguish between those HHs with adequate financial resources to meet material and education needs from HHs that lack access to these resources. Benchmark 8: Children in School BN 8 focuses on the assumption that attendance and grade advancement in the formal school system contributes to a lack of HH vulnerability. The previous Year 2 QCA results suggested that education subsidies and direct enrollment support, as well as special education funds attached to savings groups, both were sufficient pathways to achieving this BN. Interestingly, providing additional material support for books and uniforms was not relevant to achieving this BN (similar HHs not receiving this support achieved it). This BN was met in 55.8 percent of SOCY HHs and 83.6 percent of BOCY HHs. Achieving this BN is self-reported on enrollment, attendance and grade progression. No other evidence is required, and self-reporting is subject to respondent bias. Much of this section has already discussed how many beneficiary HHs prioritized formal school education for their children and mobilized resources from other OVC Program services, especially saving group participation, to pay for school fees. Indeed, qualitative investigations from the previous Year 1 Report explored how early scholarship subsidies were initially well received, but their subsequent withdrawal disrupted HH economies that had already reallocated previous education funds into other pursuits. As the BN success data shows, the lack of these supports 69 For example: ‘Some had lost their parents and care takers could not afford to pay the school requirements,’ (Bugiri, Female Caregiver) and ‘[They] helped the orphans by paying school fees or giving them skills training,’ (Tororo, Female Adolescent). 70 For example: ‘Parent trainings are the right ones because before parents used to just abandon their children and minded little about their children’s future,’ (Omoro, Female Adolescent). 38 continued to pose barriers to graduation as HHs were unable to find the resources needed to pay school fees, supplies and uniforms to sustain attendance. However, both SOCY and BOCY responded by refocusing programmatic efforts to achieve this BN through special education tuition and supply funds as part of their savings group schemes. BOCY implementation data suggested that only a small percentage of HHs opted to establish this fund, and close to 20 percent of HHs assessed for graduation still maintained some type of access to direct school fee support. BOCY FGD participants were grateful for these opportunities, and in particular associated the benefits with one of BOCY’s primary sub awardees, MUCOBADI. 71 By contrast, more than 20 percent of SOCY’s HHs enrolled in a SILC addon option known as the EduFund, which was not loaned out but backed by requiring members with school-aged children to contribute to the EduFund. Reiterating select quotes already presented earlier on in this section, SOCY FGD respondents tended to express the benefits in terms of sustainability. 72 Nevertheless, substantial barriers to maintaining school attendance were not fully overcome by the OVC Program. 73 Indeed, adolescent FGD respondents reported they saw more opportunities dedicating time to other OVC Program services such as backyard gardens than attending school, although these types of connections were very rare. The above discussion suggests that sustained school attendance is likely to remain an ongoing challenge for any follow-on interventions, such as the new ICYD activity, which integrates OVC Program elements with improved learning outcomes gained through formal education. Yet, the OVC Program managed to capitalize on the high-value beneficiary HH’s place on education and school attendance. While the direct education supports favored by BOCY did enable the activity to better achieve this BN, savings group special allocations may provide a more sustainable solution to maintaining ongoing school supports now that the OVC Program has ended. Conclusion At the end of the OVC Program, both SOCY and BOCY reported impressive beneficiary outputs and outreach. BOCY reported close to 120,000 beneficiaries served (55 percent of which were female), claiming 27,175 HHs serviced and 20,535 HHs assessed for graduation. Similarly, SOCY reported close to 140,000 beneficiaries served, claiming 38,531 HHs serviced and 25,946 HHs assessed for graduation. However, despite these impressive numbers, both SOCY and BOCY averaged around a 41–42 percent graduation rate as per the current requirement to successfully pass all eight PEPFAR OVC graduation BNs. BOCY HHs were relatively successful at achieving BNs 1, 6 and 7 but struggled to achieve BN 8. SOCY HHs performed relatively the same across BNs 1–4, 6 and 7, and also struggled to reach BN 8. Counseling is the key to encouraging testing, ensuring adherence, reducing stigmas, and spreading further knowledge about HIV across both BOCY and SOCY. The importance of 71 For example: ‘Some (children) had lost their parents and their caregivers could not afford to pay the school fees. MUCOBADI provided a solution of paying their school fees. The other requirements were left to the caregivers after participating in a skills training,’ (Bugiri, Female Adolescent) and ‘MUCOBADI selected children who were to be supported [skills training] and put into groups to go for studies instead of sitting at home and at being at risk for early marriage,’ (Bugiri, Female Caregiver). 72 For example: ‘We learnt how to make saving groups, how to save (SILC), and how to manage day to day life. We now save and are even able to pay school fees on time,’ (Gombe, Female Caregiver) and ‘SOCY taught us to see things from far not just from here. I used to only save for the not just for festive season but after joining SILC, I know I can save for fees, basic needs and any future emergencies,’ (Kabale, Female Caregiver). 73 For example: ‘Early child marriage is common and this usually happens because the father of the child is always interested in bride price in that even if a case is reported to police, he wouldn’t mind if the girl goes back to school or stays in the marriage as long as he has received his bride price,’ (Bushenyi, Male Caregiver). 39 counseling ran across discussions of all three BNs, and oftentimes reflected carefully built trust originating with the PSW. Interestingly, PSWs were often seen as the crucial link between the HH and the facility, both in cases where OVC Program volunteers occupied a dual PSW/VHT role, and where they were the representatives qualified to pass on information to the VHT. Feedback from FGDs on SOCY and BOCY efforts to achieve BN 4 suggests that when considering the successful pathways to HH nourishment, backyard garden education appears to be a viable option. Yet, given that the achievement rate for this BN could benefit from improvement, follow￾on interventions must remain mindful of the tendency for beneficiaries to prioritize the sale of produce or use it to primarily address adult and HIV positive health concerns over early child health concerns. This tendency should be addressed in future interventions. Both BOCY and SOCY HHs found difficulties achieving BN 5. On the positive side, FGD evidence and previous QCA findings suggest that when considering the successful pathways to financial stability, OVC Program savings groups—both SILCs and VLSAs—can be viable options that improve financial literacy, promote longer-term planning, provide emergency response resources, and otherwise cushion against vulnerability. Yet, a key aspect of this model involves investment in livestock to sell at a later date to pay for school fees, and this practice continues as both a sign of prosperity and poverty. As a result, PEPFAR’s BN 6 verification requirements can in fact confuse failure with success. Future interventions should consider more nuanced and/or culturally calibrated operational requirements for this BN, and proxies such as the sale of the last female in the stock may help further define meaningful differences. The ET FGD findings suggest that when considering the successful pathways to non-violence, a very complicated picture of success and struggle emerges. On the one hand, the OVC Program’s various parenting, HES and PSW counseling undoubtedly provided psychosocial benefits, improved family relations, and increased Child Protection. On the other hand, collecting reliable data on sensitive topics such as DV is not easy and is prone to bias. Similarly, some beneficiary training may have contributed to increased reporting of violence (when it would not have previously been reported) but the longer-term effects of positive behavior change had not yet occurred. Regarding what has been framed as the "SOCY Paradox” (but likely applies to BOCY as well), savings groups and other income improvements, while beneficial, may unintentionally contribute to more violence as alcohol consumption increases. Finally, males may respond to HES improvements, especially when led by females, with violence as a means of resisting female empowerment. PEPFAR’s prioritization of primary caregivers may further exacerbate these tendencies as men are systematically excluded from the benefits and jealously respond with violence. Regarding BN 7, while the OVC Program encouraged adult support to OVC-headed HHs, the demographic factors involved presented challenges in meeting this BN. Nevertheless, BOCY and SOCY services that sought to improve child-headed HH incomes were a viable mediation strategy to potentially reduce vulnerability. Follow-on interventions may wish to consider the development of intermediary BN measures that distinguish between those HHs with adequate financial resources to meet material and education needs from HHs that lack access to these resources. Finally, regarding BN 8, FGD evidence suggests that sustained school attendance is likely to remain an ongoing challenge for any follow-on interventions, such as the new ICYD activity, which integrates OVC Program elements with improved learning outcomes gained through formal education. Yet, the OVC Program managed to capitalize on the high-value beneficiary HH’s place 40 on education and school attendance. While the direct education supports favored by BOCY did enable the activity to better achieve this BN, savings group special allocations such as the EduFund may in fact provide a more sustainable solution to maintaining ongoing school supports now that the OVC Program has ended. Recommendations • Similar follow-on interventions should strongly consider replicating the OVC Program practice of enlisting PSWs to serve as the “foot soldiers” of integrated case management services through a balance of HH assessment, monitoring, HIV medicine adherence support, referrals to other services, and inhouse counseling. • When seeking to improve HH nourishment through such techniques as backyard gardening, follow-on interventions must remain mindful of the tendency for beneficiaries to prioritize the sale of produce over early child health concerns. • Future interventions should consider more nuanced and/or culturally calibrated operational requirements of PEPFAR’s BN 6, Financially Stable, and further explore other proxies of financial stability that do not restrict all sale of livestock for school fees. • Future interventions, related both to OVC as well as to reducing vulnerability, increasing resilience, and HES, must seriously consider violent reactions to improved incomes and women’s empowerment as a likely, yet unintended, outcome of social improvement. • When contemplating resources to pay for school fees and sustain school attendance, follow-on interventions that include savings groups should consider establishing special savings group education funds as a means of providing a more sustainable solution to maintaining ongoing school supports now that the OVC Program has ended. 41 4. Graduation rates and benefits from the OVC Program The previous sections identified the broader effectiveness of services that the OVC Program provides. Yet one should not assume that various subgroups and, crucially, different genders, experienced these successes and their challenges uniformly. This section combines PE Questions 3 and 4 to explore for whom the OVC Program may or may not be working. Specifically, it addresses: Do HHs or individuals with specific characteristics (e.g., individuals with disabilities, single￾parent HHs, regional variation, etc.) progress and graduate at different rates and if so, why? AND To what extent do females and males experience different benefits from the program? These two questions have been previously addressed in the prior Year 1 and 2 reports. In regard to the first question, the ET found little additional evidence, especially regarding beneficiaries with disabilities, during this year’s final 2020/2021 qualitative investigation. The limited evidence is presented below, and combined with the previous analysis from earlier reports. Regarding the second question, additional evidence from this year’s qualitative investigation has already been presented above in the previous section. In summary, the evidence, especially regarding BN 6 and the SOCY Paradox, does suggest that women tended to experience the OVC Program differently due to violent male responses to empowerment, exclusion and alcohol. As a result, summary evidence already presented in the previous reports will be presented here to address this question. Recognizing that this final report provides little additional evidence besides what has already been discussed, this section ends with the same recommendations already offered in the previous reports. PE Question 3 Qualitative Discussion The ET’s FGDs across HHs and case management participants generated very little feedback on the OVC Program’s consideration of HHs with disabilities. To be sure, a HH with disabilities was considered to be a defined characteristic of vulnerability. 74 Yet this type of reference was made in less than 5 percent of the case management conference (CMC) FGD conversations, and always in terms of hypothetical situations that lacked concrete examples. Otherwise, neither HHs nor respondents from other levels of the case management system made any specific mention of inclusion of people living with disabilities, even after prompted. This unfortunate lack of specific qualitative feedback on disability requires that the ET resort to the previously identified findings based upon the quantitative analysis. PE Question 3 Quantitative Findings BOCY Means Testing Results: Gender, Geography and HH size The main criterium the ET used to determine if HHs with specific characteristics experienced a different graduation rate than the others was the average graduation rate. Forty-one percent of the HHs that were once enrolled met the criteria for graduation and this was used as the “norm” or 74 For example: ‘A family with a disease such as malnourishment, tuberculosis, HIV/AIDS, or a disabled person. The family cannot access or afford basic medical help. A woman gives birth in the village. Financially, how much do they earn? Do they have a source of income? Can they afford a nutritious meal that nourishes the body? Look at hygiene and consider a family in a grass thatched hut with five people without a toilet or a house with an iron roof,’ (Gombe, CMC). 42 equilibrium for graduation. To specifically address Questions 3 and 4, the ET looked at the group differences where the size and composition of the HH differed, such as the (i) number of HH members, (ii) number of children in the HH, (iii) number of male and female members of the HH, (iv) gender and status of the primary leader or care provider of the HH and (v) single-parent HH. BOCY group differences for all the HH characteristics of interest were significant. HHs that included a member with disabilities fared the highest above the norm (49.5 percent), followed by larger HHs, i.e., those with more than 5 female (47.7 percent) or male (45.6 percent) children. By contrast, smaller HHs (those with less than 4 children) were below the norm (43.1 percent), and female-headed HHs were further below (42.5 percent), as were single-parent HHs (33.8 percent). Taken together, these sub-group graduation rates indicate that the program did have a gendered focus, and better supported HHs with more children. HH graduation rates varied across districts, but no clear-cut district size patterns emerged. Seven of the 11 districts with fewer than 1,000 HHs were below the norm graduation rate, but five of the ten districts with more than 1,000 HHs were also below the norm graduation rate. Across the districts with fewer than 1,000 HHs, Kamuli (63 percent) graduated the most HHs, whereas in the larger districts (with more than 1,000 HHs), Tororo graduated a staggering 84.6 percent of HHs. While district graduation rates may be a function of the proximity to central program office locations, overall these trends indicate that HHs in larger districts benefited slightly more from BOCY than HHs in districts with smaller number of HHs, but overall these differences do not establish predictable patterns. SOCY Means Testing Results: Gender, Geography and HH size The ET also conducted parallel analyses using the SOCY graduation information. The following analysis further cleaned the SOCY graduation assessment dataset by removing those HHs that lacked internal demographic information. The result is a reduced analytical dataset of approximately 20 percent fewer HHs assessed for graduation cited in the rest of this report. While the dataset is smaller, the analysis findings are more accurate. Group differences for all these HH characteristics of interest in the SOCY dataset were different from the BOCY data. Group differences were significant for geographical region (District), smaller HHs (less than four children in HH) and single-parent HHs, but not significant for large HHs or HHs with a disabled presence.75 As the ET examined the graduation by subgroups of interest, most indicated lower graduation percentages than the overall norm, especially HHs with fewer children (39.8 percent), single-parent HHs (28 percent) and those with a disabled presence (4.3 percent). This last finding stands in stark contrast with BOCY, where HHs with a disabled presence graduated well above the norm. Additionally, SOCY HHs containing high numbers of females graduated at rates no different from the norm. Finally, the highest subgroup graduation percentage was indicated not by a subgroup, but a district, Mitooma, where 83 percent of HHs graduated from the program. BOCY Latent Class Analysis Results For BOCY, the ET found three classes in the HVPT data that it grouped according to various clusters of needs. Table 18 below indicates the distribution of these classes across the HVPT dataset, and class titles were derived according to the clustering of questions in each class. 75 The ET was unable to identify head of HH gender in the SOCY dataset. 43 Table 18: BOCY latent class membership counts and proportions BOCY (HVPT) Latent Class Membership Counts and Proportions Classes Class Counts Proportion of Total Low Need 7324 0.36 Income Insecure 4750 0.23 Food Insecure 8479 0.41 Class 1 is labelled “Overall Low Need” because HHs in this class indicate a high probability that someone in the HH is employed (0.6); that children have at least two meals a day (0.8), have access to water close by (0.8) and have dry, stable shelter (0.8); and the caregiver knows the HIV status of children in the HH (0.78). The HHs in this class also have a low probability to indicate that someone is disabled in the HH (0.2) or that they have children that are not enrolled in school (0.26). This class has a low probability (0.1) to indicate that there is a malnourished child in this HH. Class 2 is labelled “Food Insecure Need” because HHs in this class have a high probability of indicating malnourished children (0.6) and a low probability of indicating that someone is employed in the HH (0.2) and that children have at least two meals a day (0.17). This class indicates higher probability of having a close water source (0.57), stable shelter (0.55), and a caregiver knowing the HIV status of children (0.67) and lower probability of school-age children not enrolled in school (0.47). Finally, Class 3 is labelled “Income Insecure Need” because this class is distinct from the “Overall Low Need” class in one distinguishing manner—it has a critically low probability that someone in the HH is employed (0.06) or that the HH was able to pay for urgent expenses (0.07). While the other two classes also indicated low probabilities (between 0.2 and 0.3) to pay for urgent expenses, Class 3 is insecure on both stable employment and savings, and thus is labelled “Income Insecure”. In a tangible manner, these emergent classes are the clusters of needs that come together to form the type of vulnerability the HHs are experiencing. Mapping each class according to their graduation rates suggests that over 40 percent of HHs across all three classes have graduated. As can be seen in Graph 4, “Overall Low Need” HHs graduated 44 percent of the time, “Food Insecure Need” at 42 percent and “Income Insecure Need” at 46 percent. This is a positive sign, but also indicates little differentiation among the three classes in graduation. Indeed, the ET found Graph 4: BOCY latent classes graduation status 44 no significant group differences when utilizing the “one step”76 approach to evaluate if membership in one of these classes could predict graduation. Although not significant, the ET found that HHs in “Overall Low Need” and “Income Insecure Need” classes were 0.67 and 0.8 times more likely than HHs in the “Food Insecure Need” class to graduate. SOCY Latent Class Analysis Results The ET also applied the same latent class methodology to the SOCY data using parallel items from the HAT measure. The three-class model indicated the best fit and the three classes account for 32, 43 and 25 percent of the HHs, respectively. The results are presented in Table 18. Table 18: SOCY latent class membership counts and proportions SOCY (HAT) Latent Class Membership Counts and Proportions Classes Class Counts Proportion of Total Least Insecure 1303 0.32 Job Insecure 1705 0.43 Most Insecure 978 0.25 All three groups are similar in their low probability of having any financial savings, and thus this item does not serve to differentiate the groups. The key difference between the classes is the way in which the HHs respond to the item of having a person in the HH with a formal job. The ET labeled Class 1 as “Overall Least Insecure” due to their extremely high probability of holding a formal job, alongside very high probabilities of having two meals a day, a working latrine, and caregivers who know HH member HIV statuses. By contrast, Class 2 was labelled as “Formal Job Insecurity” HHs because they have a zero probability of endorsing this item. It is important to note that this is also the largest group of HHs (1,705), although they are not the most vulnerable because they have a higher probability of endorsing other items that would indicate less vulnerability, i.e., (i) their children are enrolled in school (0.82), their living space has a working latrine (1), and the HH has two or more meals in a day (0.89). Finally, the ET labelled Class 3 “Overall Most Insecure” because although they score slightly higher than HHs facing more “Formal Job Insecurity”, they scored significantly lower on all other measures. The difference between classes is statistically significant and allows the ET to make graduation predictions using odds ratios. In particular, HHs in the class “Overall Least Insecure” are 1.4 times more likely to graduate compared to HHs in the class “Overall Most Insecure.” HHs in “Formal Job Insecurity” are 0.6 times more likely to graduate in contrast to the HHs in “Overall Most Insecure.” These differences can be observed in Graph 5 below: 76 Nylund-Gibson, K., Grimm, R. P., & Masyn, K. E. (2019). Prediction from latent classes: A demonstration of different approaches to include distal outcomes in mixture models. Structural Equation Modeling: A Multidisciplinary Journal, 26(6), 967-985. 45 Graph 5: SOCY latent classes graduation status Logically, LCA results indicate that SOCY should consider tailoring formal job opportunity services, such as more skills building, to the “Formal Job Insecurity” and “Overall Most Insecure” groups, while further emphasizing the use of financial savings groups to its “Overall Least Insecure” HH groups. There are parallel findings from both SOCY and BOCY latent classes. First, “Overall Low Need” or “Overall Least Insecure” classes had higher odds to graduate earlier than more vulnerable or specialized needs classes. As BOCY (HVPT) data indicates, HHs in the “Food Insecure Need” class were 0.67 and 0.8 times less likely to graduate, respectively, than HHs in “Overall Low Need” and “Income Insecure Need” classes. However, none of the BOCY (HVPT) findings were statistically significant. In the SOCY (HAT) data, HHs in “Overall Least Insecure” are 1.4 (p<0.1) times more likely to graduate compared to HHs in the class “Overall Most Insecure.” HHs in “Overall Most Insecure” are 0.6 (p<0.1) times less likely to graduate in contrast to the HHs in the “Formal Job Insecurity” class. While the SOCY findings were statistically significant, the BOCY findings were not. As such, there might be other contributing factors to the graduation of HHs not identified in the data we analyzed. Conclusion In summary, the ET disaggregated group differences in two material ways: (i) traditional sub￾group analysis and (ii) latent class analysis. Interesting IP contrasts emerged using the traditional Chi Squared technique to compare specialized subgroup graduation performance with those of the norm. BOCY HHs that included a member with disabilities fared the highest above the norm, whereas SOCY HHs with disabilities fared the worst against the norm out of any special needs subgroup. BOCY HHs with a higher number of children, both female- and male-dominated, fared better than the norm, while similar SOCY HHs faced graduation rates no different than the norm. However, both activities witnessed similar below-average graduation rates when it came to single￾parent HHs and those with few children. Geographic graduation success rates widely varied across 46 both activities, in which 84.6 percent of HHs graduated in BOCY’s Tororo but precious few in Dokolo (1.1 percent), while 83 percent of HHs graduated in SOCY’s Mitooma with no graduations as of yet unfolding in Rubanda. The ET then applied LCA to find that classes that had overall lower need were more likely to graduate in both BOCY and SOCY compared to classes that had either greater or specialized needs. BOCY programming better supported classes with “Overall Low Need” and “Income Insecure Need” over “Food Insecure Need” HHs. Similarly, SOCY programming better supported HHs in “Overall Least Insecurity” than those in “Formal Job Insecurity,” and supported both groups more than HHs in the “Overall Most Insecure” class. Logically, LCA results indicate that SOCY should consider tailoring formal job opportunity services, such as more skills building, to the “Formal Job Insecurity” and “Overall Most Insecure” groups, while further emphasizing the use of financial savings groups to its “Overall Least Insecure” HH groups. It is more difficult to make similar suggestions for BOCY, as the clusters of group needs did not significantly differ through the LCA technique. Rather, for BOCY, although the data strongly suggests the different clusters of vulnerability groups do exist, more tailored programming to each specific group will require further qualitative investigation to unpack nuances that go beyond the dataset. Recommendations • Similar follow-on interventions should consider investigating whether their services favor the graduation of smaller HH sizes and those HHs containing female majorities. • Similar follow-on interventions may also wish to consider tailoring formal job opportunity services, such as more skills building, to the “Formal Job Insecurity” and “Overall Most Insecure” groups, while further emphasizing the use of financial savings groups to its “Overall Least Insecure” HH groups. 47 5. What are the processes that facilitate long-term success? This section is dedicated to answering PE Question 1: To what extent are processes in place to facilitate the long-term success of program interventions beyond the program’s end? What factors may prevent long-term success? Up until the OVC Program’s closure, district case management had been a multi-layered process that connected HHs to a continum of care across community, subcounty and district levels. While the case process is typically initiated by PSWs through initial and follow-on home visits, cases are either immediately addressed or referred to higher levels for resolution up to the district level. Typically, the most complex cases must be addressed at higher levels, and may require either intra￾subcounty coordination or district-level policy changes. A complex division of labor underpins case resolution at each level, with the OVC Program providing key support throughout. This version of the district case management system is unlikely to continue without further donor support. As such, this section presents the results of the ET’s qualitative investigation of the case management process to determine what appeared to function, what is likely to continue, what could be strengthened and what is less necessary. The confounding factor of COVID-19 is also considered. As the OVC Program now completes close down, the findings of this section will potentially influence to what extent the new follow-on activities should adapt, adjust or abandon the now previous case management system to the new operating context. This section begins with a discussion of how PSWs interact with the HHs and balance their time between counseling and referrals; it then continues to present insights gained from conversations with CMC representatives and SOVCCs, until presenting final insights gained from conversations with DOVCCs. This section concludes with overall findings and recommendations for the follow￾on, which will also influence the cost-effective decisions made in the final section of this report. HH Case Management: PSWs HHs assigned to the local case management system typically are enrolled through three entry points. The first is in which individuals such as caregivers, neighbors, or members of the community identify vulnerable HHs. These individuals identify either themselves, on a self￾reported basis, or report on behalf of someone in their HH or community. ‘Sometimes, it is the victim themselves that report directly,’ (Bushenyi, SOVCC FGD). The second is through community service referral pathways such as schools, hospitals and/or clinics, non-governmental organizations (NGOs), or the police. The third is through the case management system where local official staff and/or volunteers such as PSWs, SWs, CDO office, SOVCC, or DOVCC identify cases. 48 Figure 3: Three methods for identifying vulnerable HHs Once HHs are identified as potentially vulnerable, professional or volunteer staff (typically PSWs) conduct HH assessments and, if the HH meets the enrollment critieria, work with the head of the HH and primary caregiver to create a HIP. The plan identifies a number of support services, including referals to OVC Program funding groups, other donors and CSOs, health facilities, law enforcement, social services and, at times, material supports. Periodic PSW HH visits (typically one hour) would monitor implementation of the HIP, make additional referrals, and provide inhouse counseling. HH visit information was then captured in the SOCY or BOCY MIS, and periodically verified through supervisor visits, typically SWs. While the importance of PSW HH counseling was confirmed in various ways by FGD feedback and discussed in the previous section, this section presents “time on task” data that estimates how PSWs typically divide and distribute their time during the HH visit sections.77 Table 18: PSW time on task during HH visit (avg. 1hr) Counseling (2/3 of Visit) Of which Referrals (1/3 of Visit) Of which HIV Testing Advice and Counseling 42.3 percent HIV Management/Treatment 22.6 percent Domestic Violence 30.6 percent Domestic Violence 37.9 percent Nutrition and Non-HIV Health 14.1 percent Livelihoods Training 12.9 percent ECD 10.0 percent Emergency Support (TCS and Transport) 13.6 percent Livelihoods Training 9.4 percent Protection Assistance (e.g. Police) 13.1 percent As Table 18 above suggests, PSWs dedicated approximately two-thirds of their HH visit time to counseling, and one-third to referrals (typically involving collecting information for the referral form). HIV-related counseling and referrals dominate overall, and include friendly “nudges” to HIV positive members about adhering to/refilling medication, even as referrals typically involved a direct handover to a VHT, or coordination with the local facility (especially in cases where the PSW is also a VHT). PSWs typically dedicate the remainder of their HH visit time to DV 77 The data presented below is derived from telephone interviews conducted by the SoCha call center in Uganda. 530 PSWs were contacted, 167 stated that they were no longer supported by the OVC Program and therefore refused to discuss, and the remaining 363 responded. 1. Individuals (self-report and/or on behalf of others) 2. Community Services (schools, hospitals, community organizations identify cases and report) 3. Case Management System (mainly CDO Office and workers under them - PSW & SW) 49 counseling and/or referrals (oftentimes to local leaders, such as religious figures). Additional services and referrals for groups, emergency support, etc., are more distant uses of PSW HH visit time. Members of the higher level CMC and SOVCC commented on the effectiveness of PSWs as a preventative mechanism against increased violence, oftentimes characterizing the OVC Program’s network of PSWs as “the first line of response at the village level” (Bushenyi, KII, SPWO), contributing to a decline in the number of child defilement cases in the years prior to the COVID￾19 pandemic. 78 SOVCC members recognized that PSWs are integral to the case management system for the identification of cases. They highlighted that, ‘since PSWs are living in the community, case identification is easy for them to do,’ (Rakai, SOVCC). In KIIs, with DOVCC members too, the vital role PSWs play was highlighted by 70 percent of interviewees. CMC respondents defined PSWs as the “foot soldiers” of case management who are largely respected and trusted within the community. 79 While there were some exceptions,80 CMC members attributed a large part of the PSWs’ success to their knowledge of the village dynamics and community members.81 Their ability to understand the community context, not only to identify vulnerable HHs but also to understand and address larger stigmas, was generally framed as invaluable. CMC, SOVCC, and DOVCC members reported that younger PSWs work in the role for a few years, gain knowledge through trainings, and then move on to other opportunities. If the role is to be carved into something that extends beyond volunteering, this would help create a consistent volunteer force with an incentive beyond altruism to engage in the role. Leveraging the PSW position, that acts as a springboard for future career opportunities either within government, private sector, or non-governmental organizations, is one strategy that could reduce the rate of replacement, as well as offer a tangible economic incentive for volunteers to enroll in the project and continue the work over a longer period of time. Leveraging this role into a formal career path with the potential for economic earnings after volunteering would help widen the pool of potential 78 For example: ‘When people realized that PSW had been trained and that they were living among them, the rate of child abuse, neglect, rape and defilement reduced … because people feared that they would be reported to the authorities,’ (Rakai, SOVCC), and ‘For example, about 3 years ago, crime rate (defilement) was high but reduced towards the end of last year and this year due to the sensitization by the Para-social workers,’ (Omoro, SOVCC), and ‘Ever since they were recruited, [and] trained PSWs the number of cases in [the] community have reduced. The number of cases that we used to receive in our office has greatly reduced; they have managed to solve cases within their mandate, which shows that they are really trusted,’ (Lira, CMC). 79 For example: “We largely depend on the support of community-based structures. The Para-social workers are a force so dependable at village levels,” (Kabale, SPWO). A member of the CMC told the ET that, ‘They [the community] trust them because it’s the community that chooses them, and after they are trained, they go back to community to tell them what they are going to do. Secondly, they keep the people’s problems confidential, that’s why they a trusted.’ Another said, ‘Para social workers are very much trusted by people and people believe in them, for example if there is domestic violence in a family and a Para-social worker intervenes peace is restored in that family, that alone earns them trust.’ Moreover, ‘They are trusted because they offer nutritional advice, psychosocial support and also work hand-in-hand with VHTs to solve cases, this makes them trusted,” (CMC, Omoro) and ‘When Para-social workers visit households most especially those with children who haven’t been cared for like the HIV positive children and then put on right medication, it earns them a lot of trust from the community members.” 80 For example (all chose to remain anonymous): ‘Not all community members support Para-social workers, they hide some information especially on teenage marriage from Para-social workers. Some community thinks their children are a source of social income,’ and others questioned the confidentiality of information shared with some PSWs: ‘Some are not trusted in the community because they do not keep confidentiality and spread whatever information is told to them for example disclosing people’s HIV status, it makes them not to disclose next time.’ Another questioned the ethics of some PSWs who are, ‘not trusted to some extent because when they get criminal cases they liaise with the offenders and [you] find these cases disappearing.’ 81 As one respondent from the Goma CMC put it, ‘when doing field work, I have only heard praises about Para-social workers because they care about where they are living and are concerned about their lives of the community members.’ 50 candidates. Creating a career path that segues into economic opportunities might also be attractive for the youth, and is one possible method of addressing the high youth unemployment rate, especially for those in the older 25–30 age category. These findings reveal that, in practice, the PSW occupies a unique space between public health and psychosocial functioning. PSWs typically defer medical services to the more health specialized VHT (who also passes upwards to the facility), but at the same time they frequently, and oftentimes repeatedly, provide complex counseling on DV to children, primary caregivers, heads of HHs and other family members. The counseling may range from one-off verbal abuse events to deeply traumatic experiences of repeated physical abuse, rape, and defilement. To be sure, PSWs should refer counseling on deeper traumatic events to qualified professionals, but effectively doing so assumes that the PSW has adequate training to understand when their counseling is appropriate and when more intense potency is required. Yet despite the social complexity of these encounters, PSWs generally receive smaller transport allowances, shortened training and supervision, and less overall recognition of the importance of their role than the VHT. This is to some extent connected to the more prominent role of the Ministry of Health over the MGLSD, as the former’s 20/21 annual operating budget is 1,550.411 (UGX Billions) while the latter’s is 171.476 (UGX Billions).82 Yet it also reflects the long￾standing PEPFAR prioritization of immediate HIV concerns over longer-term psychosocial ones. Should USAID and the GoU wish to elevate PSW status and corresponding capabilities to improve case management, both entities could expand existing PSW curriculums to include generalist topics including interpersonal communication, community mobilization and empowerment, child growth and development, GBV, mental health, and record keeping. These expanded skills could enable PSWs to become better community facilitators and counselors. At the conclusion of this OVC evaluation, the ET has learned of a recent initiative to renumerate PSWs with an equivalent of $50/week based (or prorated) upon a 40-hour work week. While this is an important step to strengthen and advance Uganda’s case management system at the grass roots level, the renumeration increase may raise sustainability concerns for MGSLD should they assume full control of the case management system and provide its resources. Just as crucial, raising PSW remuneration benefits should also correspond to an expanded training and oversight program in which PSWs are given more tools to better deal with the complexities and sensitivities associated with providing crucial psychosocial support (PSS) services, as well as better recognize when both immediate/emergency and regular referrals are appropriate. In doing so, USAID would greatly benefit from an enhanced PSW expansion readiness assessment of what kinds of additional training could feasibly be provided at scale to expand PSW counseling and referral capacities, with special consideration given to the most cost-effective approach to doing so. CMCs Besides PSWs, HHs were also referred to the case management system through other sources, such as schools, CSOs and health facilities. Using a frequency count of CMC members’ responses to questions of how cases are identified under BOCY/SOCY, the ET recorded that most came from PSWs/SWs (23 percent), closely followed by health facilities (22 percent) and VHTs (13 percent). Other referrals through the CMC come from committee members including local leaders such as the chairperson or Local Council 1 (LC1) (16 percent) or other staff from the CDOs office (7 82 See the National Budget Framework found at budget.go.ug/sites/default/files/National 20Budget%20docs/National%20Budget%20Framework%20Paper%20FY%202020-21_0.pdf 51 percent), as well as religious leaders and miscellaneous sources (15 percent). Head teachers and schools (4 percent) were also able to identify vulnerable children such as orphans, dropouts, and those facing malnutrition. Interestingly, police were not mentioned as a pathway through which to identify HHs, but were rather a resource to which cases were reported.83 The case referral system reportedly improved greatly under the OVC Program support of the CMC, as ‘most clients don’t know the right channel, so when a case is brought forth, it’s up to the first CMC line of contact to point the client in the right direction,’ (Rakai, DCDO). CMC respondents in BOCY indicated that, prior to the OVC Program, large numbers of cases were lost and never activated just because no one knew to whom potential cases should be referred. According to the Rakai DCDO (referencing SOCY), referral pathways did not start to become clear until only a few years ago. This has noticeably improved in cases involving referrals to health facilities, as CMC members estimated close to 25 percent of cases referred to their forum related to HIV. The CMC has primarily relied upon two tools used by both BOCY and SOCY in the OVC Program: the HAT and the HIP. The HAT was generally well received by those CMC members who knew of it, but this did not translate into widespread use and endorsement. Indeed, responses from members of the CMC who were interviewed indicate a divergence in understanding of the use of the tool. Some do not feel that the HAT is being implemented according to government policy, but rather as a means for NGOs to identify vulnerable HHs for interventions. 84 A few members also pointed out that the tool might not accurately reflect the perceived vulnerability of a HH, and that the tool removes ambiguity in defining if a HH is vulnerable or not. 85 The HIP, on the other hand, was referred to as being part of case management in all FGDs with CMC members, with just under half of respondents (48 percent) viewing it as an integral part of case management. Some CMC members referred to it as being a document to assess a HH’s ‘status and strength’. Other members use it to identify solutions for issues such as children not having birth certificates. Others use it as a framework to educate the caregiver as ‘they sit and make a plan together’ (Gombe, SOVCC).86 Finally, like the PSWs, CMC members stated that money that trickled down from the subcounty was the primary resource used to fund transport to enable them to be effective in the follow-up of cases. However, the subsequent close-down of both activities left noticeable gaps.87 From these accounts, it seems likely that the current HIP can continue to play a role in guiding what services should be offered at the HH level, but the HAT may likely be replaced with different tools depending on the definition of the new case management system. However, neither tool is likely to continue without additional support for transport. 83 For example: ‘We [PSW] identify them by them reporting at police especially the children who tend to give details of their condition at home,’ (Kole, CMC FGD); or ‘In some families, we [CDO] receive reports that there is a child who was defiled. They report to you so you help them [the family] to go to police,’ (Bugiri, CMC FGD). 84 For example: “It is only used by organizations by data collectors for sampling out community households which they later assess. In government programs it is not common apart from OVC. When SOCY stops, we will not use it,” (Rakai, CMC). 85 For example: “It helps guide us on identifying a vulnerable household based on the responses you get when you ask them the questions on the tool. But sometimes you may think a household is vulnerable but when you use the assessment tool it shows that they are not vulnerable,” (Gomba, SOVCC). 86 Further: “I use the tool as it guides me on what to do after assessing the requirements for enrollment. The improvement tool draws on the care plan which gives information on what interventions of particular thing will be done,” (Bugiri, CMC). 87 For example: “Back then, SOCY would fund the CMC and it was very effective as we could even follow up cases up to court until they were all closed but when funding was cut, the process slowed down,” (Kibaale, SOVCC). 52 SOVCCs The purpose of the SOVCC meeting is to oversee cases that are referred to them from other community offices including hospitals, CDOs, schools, or even NGOs. Cases that are scaled to the SOVCC are those that require more resources or support for resolution, often as a result of a criminal component that requires more facilitation for certain legal or systematic procedures to be conducted, in addition to providing increased support for the victim and family. Most commonly, the SOVCC committee addresses cases of DV, rape or defilement, early pregnancy, and cases associated with preventable school dropouts. Other cases addressed are often in relation to overcoming bureaucratic failures of systems that prevent cases from being referred or resolved. SOVCC members act more as a guide for their community following criminal proceedings or they source additional support for cases. Committee members work together to ensure legal proceedings are facilitated by supporting the referral process and scaling these cases for further review to the DOVCC. Table 19 shows the top three topics for the BOCY and SOCY interventions. The top three cases for BOCY reflect its focus on addressing the vulnerability of children and orphans by addressing common factors associated with limited access to education, whereas SOCY focused more on GBV response though decreasing HH vulnerability. Both program SOVCCs also mentioned early marriage, unwanted pregnancy, and child labor as types of cases most commonly responded too in near to equal frequency as secondary topics. Table 19: Top three case types addressed by the SOVCC, by program Program Case Type Frequency Count BOCY School Dropout 8 Child Abuse/Neglect 6 Rape/Defilement/Sexual Assault 4 SOCY Rape/Defilement/Sexual Assault 5 Child Abuse/Neglect 4 School Dropout 3 Total 10488 All FGDs reported awareness of quarterly meeting targets. One participant [Kabale] pointed out that despite the ‘quarterly [target], it doesn’t happen because of the limited resources. You find that people practically not meeting.’ In 2020, due to COVID-19, several SOVCCs only had the ability to meet twice. The quarterly meeting frequency was supported by members stating that it was a ‘reasonable time’ in which cases could be reviewed and discussed. In the Eastern region, the SOVCC largely addressed cases related to orphans and vulnerable children. In the Western region, the CMC process was largely focused on vulnerability associated with HIV/AIDS. Despite using two different lenses of implementation, the SOVCCs all address 88 In several cases, respondents mentioned more than one classification category of HH vulnerability. 53 similar cases of vulnerability. As cases are scaled to the subcounty, they are often criminal in nature, or more complex cases requiring additional support beyond that of the CDO’s office or implementing partners.89 The SOVCC members have become advocates for children and youth. In one specific memorable case in Tororo, enumerators were told of how the committee worked with a girl who was on the verge of committing suicide after becoming pregnant from her early marriage. The committee worked together with a PSW to encourage her to understand that she would still be able to pursue educational opportunities, and worked at developing a support system for her to return back to school after the child was born. At times, addressing, referring, and closing cases brought to the SOVCC was hindered by a prohibitive subcounty policy that prevent HHs and/or individuals from accessing the services they need. Several mentions were made about advocating to the district to reduce the cost of birth certificates. Birth certificates were identified by the lower levels of the system [HH, CMC] as barriers to enrolment for certain OVC cases. In Bushenyi, the SOVCC was able to advocate for a subsidized price reduction for birth certificates to ensure a nominal processing fee and payment for the form.90 DOVCC members recognized that this made it more manageable for OVCs (or their family members) to access services within the program (although there was some discrepancy in the exact stated figure). 91 Almost all cases that require criminal proceedings are referred to the DOVCC for additional support. Table 20 below presents evidence from the ET FGDs of the four types of cases forwarded by SOVCCs to DOVCCs in the last quarter. Table 20: Frequency of type of cases requiring referral to DOVCC level Level Case Type Frequency Count FGD Accounts of Cases Referred to DOVCC from SOVCC in past Quarter Child Neglect/Abandonment 13 Domestic/Gender Based Violence 11 Rape/Defilement/Sexual Assault Child 10 Teenage Pregnancy 13 Total 47 89 For example: “The role of the SOVCC was not to resolve cases, but rather to receive the identified cases (those handled or referred) and get solutions. Case solving is at the level of the CDO because they are trained and to do so,” (Bushenyi, SOVCC). 90 The Bushenyi SOVCC FGD reported: ‘When we realized that OVCs couldn’t afford the fee for processing a short birth certificate, which is always a requirement even at schools, we appealed to the district committee to reduce the fee specifically for OVCs which they reacted to and reduced it from 10,000 UGX to 2000 UGX in Bushenyi district.’ 91 For example: ‘Whenever OVC cases were to be resolved that needed proof of age, notifications of birth (birth certificates) are a necessity. However, the district charges ten thousand (10,000 UGX) for that document. So the DOVCC pushed for a reduction of that amount to three thousand (3,000 UGX) for easy access,’ (Bushenyi, SPWO). 54 Despite the frequency, the discussion around upwards referrals between SOVCC and DOVCC was not as clear as other system referrals, and SOVCC members expressed frustration with the current approach. They pointed out that there was no formal structure or guidebook to follow when referring cases, and that cases scaled were more of an ad hoc response as the need arises. Yet DOVCC meetings could not respond to ad hoc referrals, and all referred cases were required to wait their turn until DOVCC quarterly meetings properly convened. Yet certain criminal cases that needed immediate address did not have the luxury of waiting.92 SOVCC respondents blamed this lack of responsiveness on the obligations imposed by the OVC Program, who held the power to determine ‘when to convene for a meeting’ (Bugiri, SOVCC). In response, Bugiri SOVCC members created a savings budget so they are able to provide additional financial support to children when necessary. Members of the SOVCC also stressed that one major barrier to case management was the limited feedback, in terms of results, given by the DOVCC for cases referred upward. At the subcounty level, they viewed this as a hindrance to proper case management as they are not aware of case needs. As a result, they were not equipped to decide whether they should continue following up or helping in one shape or another to address the cases. Sometimes the committee members feel at a loss to prevent situations that perpetuate due the ‘delay to give feedback about referred cases’ (Kibaale, SOVCC). These individuals felt ‘left in the dark of what later happened’ when ‘no feedback is given to the person who referred’ (Kabale, SOVCC). Lastly, DOVCCs’ failure to report information back down to the SOVCCs was viewed as a lost opportunity for sharing knowledge to the extent that ‘if information or feedback was shared, you would borrow knowledge on how that case was handled for future reference’ (Kabale, SOVCC). Requests for the trickle-down of information regarding cases referred upwards should be a natural part of the case management process. The above discussion highlights the crucial and flexible role SOVCCs play in resolving the bulk of cases promptly, so that the more serious criminal instances of rape, abandonment, teenage pregnancy and violence can be addressed at the district level. The SOVCC also sits in an advantageous position to push important policy changes, such as the price of birth certificates, so that more cases can be resolved. Yet SOVCC flexibility reaches a limit when referring cases upward, as DOVCC meeting schedules maintained a stricter adherence to pre-established timelines. Moreover, two-way feedback on how DOVCCs resolved cases could help SOVCCs better prepare referrals to ensure faster resolutions. As such, future interventions should consider a district-level case management model that allows a few ad hoc case resolutions for special circumstances, and more systematically shares case resolution procedures with lower levels. DOVCCs The DOVCC, which is scheduled to meet every quarter, is an important gathering of department heads across all sectors, including government staff from health, education, planning, production, police, and community services, facilitating the sharing of resources to provide a more holistic response to the needs of the child and family. DOVCC meetings are chaired by the Chief Administrative Officer (CAO), with the District Community Development Officer (DCDO) as Secretary, and often include other key stakeholders, implementing partners, and invited representatives from CSOs, and occasionally the private sector. The DOVCC plays a crucial role 92 For example: ‘All issues that deal with child protection are important; child neglect, child abuse, all are important. Anything that deals with child protection is not an option, whether there is money or not,’ (Lira, SOVCC). 55 in operations and oversight for the case management system as a whole, and is the final referral pathway for complex cases requiring district-level intervention and/or support.93 The complex and interrelated nature of many cases requires the establishment of a cross￾departmental approach, and while individual offices take responsibility for cases in different sectors, there is a great deal of interdepartmental collaboration in bringing cases to a successful resolution. The general consensus was that coordination between district government systems has improved as a result of the SOCY and BOCY interventions. 94 Three main items tend to dominate these quarterly DOVCC meetings. First, DOVCCs at times discuss wider matters such as challenges in successful case management at the lower levels that often require policy changes; budget requirements for district-supported OVC initiatives; policy proposals to better serve OVC and those working to support them; and specific action points to be taken forward by those offices responsible. Second, DOVCCs offer an opportunity to discuss specific and complex cases requiring immediate intervention, especially criminal cases like those first discussed at SOVCC meetings. Third and finally, members hear and discuss aggregated data taken from collated SOVCC reports, presented by the DCDO, in addition to data and reports submitted by the various officers in attendance. This data feeds into the national OVCMIS, and can be used for advocacy purposes with the national government. Regarding policy changes, DOVCC members spoke of the importance of the committee in leveraging its oversight of the district in a way that better improves the outcomes for OVC. One example of this is the provision of meals during school time to address malnutrition: ‘The district embarked on a discussion that all commuting students in school pack their own lunch to avoid school absenteeism.’ However, it was noted that ‘[t]his suggestion hasn’t yet reached a conclusion,’ (Bushenyi, PCDO). This was echoed in FGDs with members of the SOVCC, who also recognized the important role the DOVCC had played in advocating for district-wide changes.95 Another important policy change that positively affected the case referral process was identified during KIIs in Lira. DOVCC members had been made aware of the challenges involved in securing a medical check-up for cases of defilement, rape, or sexual abuse. This medical check￾up, via Form 3, had become an impasse for many cases, because the form had to be signed off by a doctor, who was not always available, or access to whom was geographically challenging. The difficulty in getting Form 3 signed off was identified as the point where a lot of cases were lost, 93 All districts visited by the ET had functioning DOVCC committees, and all had held at least one DOVCC meeting in the six months prior to data collection, with many having met twice in the two quarters between the easing of the national lockdown in late August 2020, and KIIs conducted by the field team in late-November and December 2020. The constraints on holding meetings during the COVID-19 pandemic, particularly the restrictions on movement and congregations, were attributed to the failure to hold meetings during the five months between March 2020 and August 2020. Some KIIs spoke of the ‘dwindling facilitation’ for DOVVC activities over the past financial year, and most recent meetings had taken the form of ‘round-table’ discussions of sustainability as, towards the end of the project, direct funding to DOVCC and SOVCC activities were gradually reduced, and in some cases, replaced by local sources of funding. 94 For example: ‘In relation to the Hepatitis B case, poor patients who don’t have enough money to take their own samples to health facilities of referral, DOVCC comes in to support by taking them to the Uganda Virus Institute in Entebbe,’ (Bushenyi, SOVCC). 95 For example: “The district passed a resolution and demanded for all parents to pay 10,000 UGX at school for every child to receive meals because some drop out because of hunger. The district has also directed agricultural officers to give seeds to vulnerable families to curb food insecurity. They also made health centers to give free services while interpreting and filling forms while giving testimony/report/ to police and to court at a free cost. Also, police were ordered not to negotiate any case in the community but to handle it right away,” (Gomba, SOVCC). 56 and the barrier to a case being scaled up as a criminal case. The committee had advocated for a greater range and number of healthcare professionals to be able to sign the form. 96 DOVCCs also addressed the remaining cases that fell outside the lower level SOVCC mandate. In doing so, the ET found that the Community Based Services Department (CBSD) was integral to creating and developing a shared sense of purpose for the DOVCC. This department is mandated to deliver community level actions to reduce poverty through the promotion of employment, equity, adult literacy, and community participation in development programs. In fact, most district respondents described the OVC Program’s biggest impact on the DOVCC was to instill a greater appreciation of the important role CBSDs play in community enhancement and Child Protection. This should come as no surprise, as the CBSD held the mandate to address most cases the DOVCC received. CBSD staff are comprised of Probation, Community Development, and Labor Officers; and through them the department promotes issues of social protection, equality, equity, human rights, culture, decent work conditions, and empowerment of the poor and vulnerable groups. Indeed, when one considers that most of the membership needed to address complex cases sits on the CBSD, the need to establish a separate DOVCC is less clear. However, it should be noted that despite the budget increase, CDOs and Probation Officers (POs) remain largely reliant on implementing partners and other district officials for operational support. There is an acceptance that donors are responsible for ensuring that funds are available for OVC case management, with little prospect of support from the national government. In fact, district￾level interviewees all indicated that a great portion of their working time is spent advocating for, requesting, and securing funds from partners in the NGO and private sectors than from the district itself. All interviewees recognized that case management needs funds to fuel the progress of case referrals, and all attributed delays or breakdowns in the case management system to the removal of donor funding for, at the most basic level, transportation. Finally, the DOVCC, supported by the OVC Program, was the main forum to review and compile cases for entry into the OVCMIS. This was more than just a reporting exercise, and KII respondents provided tangible examples of how the data was used for advocacy purposes. For example, the budget for the CBSD was increased in 2019, with some DCDOs reporting a 10 percent increase in the operational budget for OVC activities (Rakai, DCDO). This increase in operational budgets for CDOs and POs was directly attributed to SOCY’s lobbying of the MGLSD by the principal CDO in Bushenyi, although the interviewee was quick to note that, ‘facilitation from the government is still minimal—government has not put in a lot. The Probation Office receives only 800,000 UGX—this is between 15 people—to do case management.’ The above discussion highlights the important role played by DOVCCs in orchestrating district￾level policies that improve the case management system. For these types of solutions to continue, it should be clear that some district-level mechanism should remain in place. However, given that the majority of cases are/can be resolved by the CBSD, a leaner mean district coordination body could likely still coordinate cross-departmental policy changes even as the DCDO can retain ownership of some version of the OVCMIS for advocacy purposes. Moreover, the elimination of 96 The DCDO reported: “Form 3 has been a big change. Form 3 takes information on injuries of the victim, and it can be a big problem if it is delayed—that’s where a lot of cases get lost. Formerly it needed to be signed by a doctor—that they had examined the victim—but it has been revised to include more healthcare professionals—so now the in-charge, or a nurse, they can examine (the victim) and sign that form.” 57 the DOVCC may free up additional transport funds that could be better distributed and applied to the subcounty levels. Conclusion The future of the case management system, especially at the lower, subdistrict levels, is currently in flux and in doubt. With the end of the OVC Program, new mechanisms to replace the CMC and SOVCC mechanisms—which may include an expanded membership to “mainstream” services to incorporate community development health, education, agriculture, and environment—are still in formation, even as previous relics of the SOVCC and DOVCC system, at best, meet on an ad hoc basis (further constrained by COVID-19). MGSLD ultimately holds responsibility for the formation of the new mechanism(s), and any guidance provided will ostensibly be informed by the recent 2020 National Child Policy. Yet while the Policy does mention the formation of a new District Child Well-being Committee (DCWC)—as well as a new OVCMIS upgrade, the Child Welfare Information System—it says nothing about lower level mechanisms to perform the bulk of case management and provide local solutions, so that only the most complex cases requiring special legal or policy changes will reach the district level. Without an effective lower level filtering mechanism similar to the CMC and SOVCC in place, the ability and capacity of the proposed DCWC should be seriously questioned. To be sure, the newly-awarded, USAID/Uganda ICYD activity holds the potential to take the learnings of the OVC Program forward, and may do so as part of its attempt to scale up the case management system. The bulk of these efforts will no doubt initially focus on further system strengthening at the district level, as new roles and responsibilities are defined and tested. Yet, as of this writing, neither MGLSD nor ICYD have put forward a working model of subdistrict case management, even as the latter may be waiting for guidance from the former on subdistrict CWC composition, while the former may be waiting for the latter to propose financial commitments to ensure subdistrict CWC efficacy. Recalling the lessons identified in the Year 1 OVC Evaluation Report regarding a loss of case management expertise, momentum and, most importantly, continuum of care to HHs, the transition to MGLSD’s new DCWC system with ICYD support runs the similar risk of compiling a backlog of unresolved cases in which many immediate essential needs on the ground remain unaddressed. Recommendations • Follow-on interventions similar to the OVC Program should consider a district-level case management model that more systematically shares case resolution procedures with lower levels. • USAID and follow-on interventions similar to the OVC Program, such as ICYD, should support some version of SOVCC and CMC to advance its OVC Program investments. This should likely include minimal transport allowances for CMC and SOVCC participants. • The newly proposed PSW remuneration scheme should be bolstered with additional training but studied for cost-effectiveness and sustainability. • Follow-on interventions similar to the OVC Program may wish to consider scaling down district-level case management committees to focus on the resolution of policy decisions, even as the bulk of complex cases that reach the district level can mostly be resolved through the CBSD, which can also maintain new versions of the OVCMIS and advocate for additional funds from the national government. 58 6. The cost-effectiveness of the OVC Program The question of cost is discussed in this section and addresses the PE Question 5: How cost-effective are different pathways to change? How do interventions/combinations of interventions compare in impact and cost? To determine cost effectiveness, the ET built a cost model to generate the average unit cost of various OVC services at the HH level. These costs were based upon the review of expenditure data collected from both SOCY and BOCY. As such, the cost model should be understood as a tool for estimating future costs based upon a sufficient sample of current expenditures. The resulting model should not be viewed as an accountability tool, i.e., it should not be used for audit purposes or to cross-check total expenditures to date with those predicted by the model. The model itself was designed according to the 80/20 principle: roughly 80 percent of the predicted costs come from roughly 20 percent of the actual expenditures. OVC Program Unit Costs per Household Following the costing methodology identified in the first chapter of this report, the ET was able to generate a unit cost model per HH for each activity based upon the demand data found in the BOCY and SOCY MIS. The overall average unit cost per HH estimates for BOCY are in Table 21: Table 21: Total unit costs per HH, BOCY Total Unit Costs per Household, BOCY Overhead cost per HH 449,060 HH visits and case management 202,543 Group costs per HH 196,877 Training 1,403,669 Total (UGX) 2,252,149 Average USD cost per HH $ 625.60 The overall average unit cost per HH estimates for SOCY are in Table 22: Table 22: Total annual cost per HH, SOCY Total Annual Cost per Household, SOCY Overhead cost per HH 449,060 HH visits and case management 169,811 Group costs per HH 196,531 Training/Support 895,996 Total (UGX) 1,711,398 Average USD Cost per HH $ 475.39 59 Cost Effective Adjustments With the above listed baseline unit costs estimated from each activity’s respective cost model, the ET proceeded to introduce cost-effective adjustments based upon two criteria: QCA Findings from the Year 2 Report The Year 2 Report modeled an array of services used for each BN for QCA and found a number of trainings and group programs offered were not needed to achieve each BN (see Year 2 Report). Based upon QCA findings, the following trainings and groups were identified as irrelevant for BOCY in Table 23: Table 23: Cost-effective adjustments to BOCY Cost Effective Adjustments to BOCY Group Services Irrelevant to BOCY BN Success Stepping Stones Journeys HH Trainings Irrelevant to BOCY BN Success Enterprise Training Informal Education Training Micro Biz Agri Biz Training Based upon QCA findings, the following trainings and groups were identified as irrelevant for SOCY in Table 24: Table 24: Cost-effective adjustments to SOCY Cost Effective Adjustments to SOCY Group Services Irrelevant to SOCY BN Success Apprenticeships Enterprises Agronomy Coffeelands HH Trainings Irrelevant to SOCY BN Success No Means No School Material Support Suggested Case Management Modifications In addition to QCA’s identification of irrelevant services, the ET also incorporated the recommendations of this Report’s case management chapter to reduce costs by limiting the role of DOVCCs, eliminating the payment of travel and facilitation fees for DOVCC participants (in line with current donor regulations), but retaining travel allowances for SOVCC and CMC members, as well as the cost of refreshments for CMC members. This reduction was equally applied to both BOCY and SOCY. As such, the updated case management costs for each activity are in Table 25: 60 Table 25: Refinements to case management costs from qualitative feedback (in UGX) DOVCC SOVCC Case Conference Number of Members 23 15 50 Number of members that receive a travel fee 13 50 Travel/attendance fee 15,000 5,000 Number of government officials attending 19 2 2 Refreshments per person 3,000 Stationery per person 3,000 Cost per event - 195,000 550,000 Number of cases discussed 5 15 30 Cost per HH referred - 13,000 18,333 Combined, the overall cost-effective savings suggested by the QCA model and qualitative investigation for BOCY are as follows in Table 26: Table 26: Total unit costs per HH, BOCY Total Unit Costs per Household, BOCY Overhead cost per HH 431,917 Household visits and case management 184,072 Group costs per HH 137,078 Training 1,247,268 Total (UGX) 2,000,335 Average USD Cost per HH $ 555.65 Original Average USD Cost per HH $ 625.60 Percentage Unit Cost Effective Savings per HH 88.8 percent Combined, the overall cost-effective savings suggested by the QCA model and qualitative investigation for SOCY are shown in Table 27: Table 27: Total annual cost per HH, SOCY Total Annual Cost per Household, SOCY Overhead cost per HH 431,917 Household visits and case management 151,340 Group costs per HH 129,288 Training/Support 757,916 Total (UGX) 1,470,461 Average Cost Effective USD Cost per HH $ 408.46 Original Average USD Cost per HH $ 475.39 Percentage Unit Cost Effective Savings per HH 85.9 percent As can be seen, this application of cost-effectiveness translates into an approximate 11 percent unit costs savings per HH for BOCY and 14 percent unit costs savings per HH for SOCY. 61 Conclusion To determine cost-effectiveness, the ET built an “activity-based implementation costing” model to establish baseline HH unit costs for OVC Program services implemented across BOCY and SOCY. This model was based upon expenditure and implementation data collected from both activities to calculate project inputs, prices and levels of service demand. Converted into U.S. Dollars, these models suggest an average Unit Cost of Services of $625.60 for BOCY HHs and $475.39 for SOCY HHs. The ET then drew upon QCA findings to identify irrelevant group and training services as well as qualitative investigations to identify potential areas for cost-effective saving. Once these adjustments were made, the revised cost-effective cost models suggest an average Unit Cost of Services of $555.65 for BOCY HHs and $408.46 for SOCY HHs, which translates into an approximate 11 percent and 14 percent unit costs savings per HH, respectively. However, the resulting model should not be viewed as an accountability tool, i.e., it should not be used for audit purposes or to cross-check total expenditures to date with those predicted by the model. Rather, it should be used as a tool to demonstrate the potential cost saving implications of using complexity management tools like QCA to eliminate irrelevant services, which can be used as a guide to further optimize the cost-effectiveness of similar follow-on interventions. Recommendations Future interventions may wish to consider combining activity-based implementation costing models with complexity management tools like QCA as an approximate guide to making cost￾effective adjustments to integrated programming. i Annex 1: OVC Evaluation Scope of Work EVALUATION OBJECTIVE AND OUTCOMES The purpose of the OVC Evaluation activity is to provide rigorous evidence regarding the extent to which participation in the program has reduced HH vulnerability and improved OVC well-being and to identify factors or combinations of factors, within and outside the program, which may explain how results were achieved or what barriers may have prevented greater success. This evidence will be used to inform decisions about future efforts to support OVC in Uganda and elsewhere. ● The financial self-sufficiency of a HH in terms of their ability to meet basic needs of children in their care, including nutrition needs, school attendance, children's psychosocial well-being, as well as economic resilience to shocks, such as unexpected medical expenses and others. ● Adequate functioning and/or sufficiency of formal and informal systems and reporting, for example through case follow-up and capacity of service officers. ● Functioning of case management and referral systems. SoCha will provide technical, management and logistical support to achieve the above stated objectives. Specific Tasks For both activities, the evaluation will assess achievement of objectives and will complement these findings using a mixed-methods approach in order to provide additional data on how and why objectives are or are not being achieved. It will provide feedback to the Mission and programs and will include external reviews of data quality of the two activities. SoCha is expected to propose a mixed-method approach to answering the cross-cutting SOCY and BOCY performance related learning questions below. Learning Questions ● Question 1: To what extent are processes in place to facilitate long-term success of program interventions beyond the program’s end? What factors may prevent long-term success? ● Question 2: What coordination and management mechanisms (e.g., structures, operational guidelines and frameworks or forums) have worked best to support the integrated OVC implementation approach? What enabling factors do these mechanisms engage with? ● Question 3: Do HHs or individuals with specific characteristics (e.g., individuals with disabilities, single-parent HHs, regional variation, etc.) progress and graduate at different rates and if so, why? ● Question 4: To what extent do females and males experience different benefits from the program? Are there unintended results for either group? ● Question 5: How cost effective are different pathways to change – how do interventions or combinations of interventions compare regarding impact and cost? ● Impact Evaluation Question: Does participation in SOCYH activities 3 contribute to a reduction in HH vulnerability and improve OVC well-being? SoCha is expected to adhere to a design that: 1) Combines quantitative and qualitative methods as appropriate in order to answer the final questions. 2) Compares and contrasts findings from SOCY and BOCY to produce actionable conclusions and recommendations. ii 3) Uses representative samples that permit conclusions about program achievements to be made with confidence, including disaggregation by sex, disability and vulnerability status, single- vs. two-parent HH, district, and possibly other beneficiary characteristics. SOCY-Specific Considerations The primary outcome measure for the SOCY evaluation is HH vulnerability status. The measurement tools used for the baseline are based on the Household Vulnerability Assessment Tool (HVAT) and Household Vulnerability Prioritization Tool (HVPT), which measure vulnerability across economic, food security, nutrition, health, water, sanitation, and hygiene (WASH), shelter, education, psychosocial support, and child protection dimensions, and yields a composite score corresponding to “not vulnerable”, “slightly”, “moderately”, and “critically” vulnerable categories. The HVAT contains modules administered individually to caregivers, children aged 0-9, and youth aged 10-17. Additionally, nine secondary outcome measures derived from PEPFAR’s MER Essential Survey Indicators for OVC Programs were included in the baseline and should be followed in the mid- and endline. To the extent possible, data collection tools utilized to collect baseline data should be maintained to ensure comparability of mid- and endline with baseline data. However, the contractor will rigorously review the tools and suggest revisions to increase rigor and integrity of the study. USAID will review any suggested modifications to instruments and make final determinations on format and content. iii Annex 2: Datasets Used in This Report Please see attached zip file that contains the following files: • Overall Waves 1-4 SOCY Impact Evaluation Dataset • BOCY and SOCY Compiled Qualitative Coding Scheme iv Annex 3: SOCY Impact Evaluation Survey Questionnaire Household Caregiver Oral Questionnaire Identifier Page DISTRICT: SUBCOUNTY: PARISH: VILLAGE: NAME OF INTERVIEWER: DATE OF INTERVIEW: NAME OF INTERVIEWEE: HOUSEHOLD CODE: GPS: Latitude: S __ __.__ __ __ __º Longitude: E __ __.__ __ __ __º Start Time: End Time: Questionnaire Serial Number: INTERVIEW LOG VISIT 1 VISIT 2 VISITI 3 DATE (day/month/year) INTERVIEWER COMMENTS Interview comment codes: Interview completed 1; Appointment made for later today 2; Appointment made for another day 3; Refused to continue and no appointment made 4; Other (Specify) 5 101 102 103 104 105 106 Line Please give the names of persons who usually live in HH, starting with head of HH. If (NAME of Child), what is the age? less than 1 year=0 If (NAME of child) what is the gender? (Female=1, Male=2) Does (NAME of Child) usually live here? (Y/N) Do you usually cares for/looks after (NAME of Child)? (Yes=1, Other=Give Name, Selfcare=0) If care giver, what is your relationship to (NAME of child)? (see codes below) A B C D E F G H I J K L M N CODES FOR Q105: RELATIONSHIP TO RESPONDENT v 01 = BIOLOGICAL MOTHER 03 = NON-BIOLOGICAL PARENT 05 = AUNT/UNCLE 02 = BIOLOGICAL FATHER 04 = SIBLING 06 = GRANDPARENT vi Household Vulnerability Module Below is the main vulnerability module used for the baseline survey. The four leftmost columns identify from which tool the question was taken, and include MGLSD’s HVAT, Sustainable Outcomes’ HAT, MGLSD’s IPT, and the OVC indicator questionnaire used for PEPFAR indicators. Questions 44-47 were taken from the OVC situational awareness survey. Finally, the subsequent child inventory page and Index child modules were taken from the OVC indicator questionnaires. OVC IPT HAT HVAT ECONOMIC STRENGTHENING 1 1. Is this a child headed HH? Yes No 4 2. Does the HH head, spouse or guardian have any form of severe disability (e.g., physical, speech, visual, hearing, or mental handicap?)? Yes No X 3. Does the HH have ANY member who has been very sick for at least three months during the past 12 months? (By very sick, I mean that the HH head or any member was too sick to work or do normal activities around the house for at least three of the past 12 months) Yes No 2 1.4.4 4. Is there at least one member of the HH who currently has formal or informal employment, is self-employed, has a business, or is engaged in an economically productive activity? Yes No 1 1.5 5. Who is the main contributor to HH income? A Child Grandparent( s) Parent(s) Relatives/ Others 1.6 6. What is the current monthly HH income? (express in Uganda Shillings) <50k 50k – 99k 100k – 149k 150k￾200k >200 k 2 1.2 7. What is the main source of HH income? Formal Job/ Business Informal Job Casual Labour Remittance/ Others 1.5 8. The last time there was an unexpected urgent HH expense (e.g. emergency medical expense or house repair), HH was able to pay that expense? Yes No vii 3 1.5.1 9. If the HH incurred any HEALTH￾related expenses in the past three months, was it able to pay for these expenses? Yes Not always No 4 1.5.2 10. If the HH incurred any SCHOOL￾related expenses in the past three months, was it able to pay for these expenses? Yes Not always No 5 1.5.3 11. If the HH incurred any FOOD-related expenses in the past three months, was it able to pay for these expenses? Yes Not always No 6 1.4.5 12. Does anybody in the HH belong to any financial savings and lending group? Yes No 7 1.4.1 13. Any member of the HH owns an electronic gadget (radio, phone, TV) Yes No 1.4.2 14. Any member of the HH has a functional transport means (bicycle, motor cycle, boat) Yes No 14.3 15. Any member of the HH has vocational/apprenticeship/professional skills? Yes No 1.4.6 16. Household has domestic animals (cow(s), goat(s), Sheep, chicken, pig(s))? Yes No 1.4.7 17. HH has access to land for agriculture/hire? Yes No 7 7 18. Does the HH own any of the following assets: cooking stove, beds, blanket, mattresses, mosquito net, shoes, cooking utensils, furniture. Yes, most of them Yes, some of them None or almost none OVC IPT HAT HVAT FOOD SECURITY AND NUTRITION 8 2.2 19. What does the family usually eat? (at least 3 times a week) Energy foods: (potatoes, banana, oils, posho, millet, rice, maize, bread, cassava) Yes No Body building foods: (beans, meat, soya, peas, milk, eggs, chicken, fish) Yes No viii Protective and regulative foods: (tomatoes, oranges, pawpaw, mangoes, pineapple) Yes No 2.1 20. Over the past month, what has been the MAIN source of food consumed by your HH? Home grown Donated Bought Given for Work 9 2.3 21. How many meals does the HH have in a day? Not everyday One meal per day Two meals per day Three or more X 22. Over the past month, did anyone in the HH ever go without food for a whole day because there wasn’t enough? Yes, more than 5 Times a Month Yes, 1-4 times a month No 5 23. Has this HH eaten at least 2 meals a day, every day, for the last month? OVC IPT HAT HVAT HEALTH, WATER, SANITATION AND SHELTER X 3.1.3 24. What is the distance (in Km) to the health care facility your HH often uses? Kms Don’t Know X 25. When was the last time a member of the HH accessed a health facility? Weeks/Mont hs Don’t Know 9 3.2 26. Does the caregiver know the HIV status for all children in the HH? Yes No X 3.1.8 27. Do all HH members sleep under a mosquito net? Yes Some No X 8 28. Have all the children in the HH been tested for HIV? Yes Some No 29. Are all eligible children who are HIV+ and/ or have TB on treatment? None of the Children Less than half of the Children More than half of the Children All of the Children 10 3.3 30. [If you don’t know of any HIV+ members of the HH, ask this question]: Is there any member of the HH who has a chronic disease? (HIV+, Cancer, TB, Sickle cells, diabetes etc.). If you know Yes No ix there is an HIV+ person in the HH, you do not have to ask this question, but check yes. X 31. What is the main source of water for members of your HH? River, Lake, Pond, Unprotected well Public taps, Bore hole, Rainwater, Protected spring/well Private Connection 6 10 3.1.1 32. Does the HH access drinking water from a safe source within 30 minutes? Yes No 11 33. Does the HH have access to a latrine? Yes owned Yes shared No latrine 3.1.2, 3.1.4- 3.1.7 34. Observe the following: Has a clean compound Yes No Has a drying rack for HH utensils Yes No Has a garbage pit /dust bin Yes No Separate house for animals Yes No Hand washing facility Yes No 7 3.4 35. Observe: Does the HH have a stable shelter that is adequate, safe and dry? Not safe or stable shelter Inadequate, needs major repairs Needs some repairs but adequate Safe, adequate and dry 3.5 36. Observe: What type of a latrine/toilet facility is used by the HH? Bush/None Public Toilet for Pay Private, needs repair Private, adequate, but shared Private, safe, adequate, dry OVC IPT HAT HVAT EDUCATION X 4.1 37. None attend x 11,1 2 How many of the children aged 5–17 years in this HH are not going to school or miss school 3 or more times a week [DO NOT INCLUDE 1-4 years] Less than half attend regularly More than half attend regularly All attend regularly OVC IPT HAT HVAT PSYCHOSOCIAL SUPPORT AND BASIC CARE 5.1 38. In the past 12 months (STATE MONTH), how often has someone in your HH felt so troubled that it was necessary to consult a spiritual, faith or traditional healer, counselor or health worker? Enter number Not at all 5.2 39. Are there any children in this HH who are withdrawn or consistently sad, unhappy or depressed, not able to participate in daily activities including playing with friends and family? Enter number Not at all OVC IPT HAT HVAT CHILD PROTECTION AND LEGAL SUPPORT X 40. Do you think that hitting or beating a child is an appropriate means of discipline or control in the home? Yes No X 41. Do you think that hitting or beating a child is an appropriate means of discipline or control at school? Yes No 12 6.1 42. In the past month, have you or another adult in the HH used the following method of discipline with any child in your HH? (Please select all the methods that apply) Punched, kicked or hit as punishment Withheld a meal as punishment Yelling and screaming 6.2 43. In the past 12 months, has any child in the HH had the following happen to them, in or outside of the HH? [Note: If you see an obvious issue of abuse or you already know about it, then indicate yes]. Indicate Yes/ No Repeated physical abuse Involved in Child Labour Family separation (ran away, chased) Sexually abused, defiled, raped, forced sex Stigmatised/ discriminated due to illness, disability or otherwise In contact/conflict with the law All Below Taken from Situational Awareness Survey HOUSEHOLD SERVICES 44. Has your HH received services or participated in activities from a community based program in the last six Yes If No, then end module xi months? By this I mean, in the last six months have you or someone in your HH been visited by a community worker, or participated in any activities organized by the program such as a savings group or parenting program? and go to next module 45. Are any or all of the services your HH is receiving or participating in provided by Sustainable Outcomes Yes No Don’t Know 46. What type of HH based services or activities (through a visit from a volunteer) has your HH received or participated in the last six months? Circle all that apply Household visits from a volunteer Direct financial support Parenting counseling Early Childhood Development Health and hygiene HIV and GBV prevention Nutrition counseling Pre/post-partum counseling HIV testing Couples counseling Support obtaining a birth certificate Child protection Psychosocial support/basic needs Food security Other_______ None 47. What type of community based services or activities (outside of the home) has your HH received or participated in in the last six months? Savings groups Parenting program Government SAGE program Any other cash transfer xii Circle all that apply Voluntary HIV testing/counseling Food security and Nutrition Skills and employment training Entrepreneurship training Other:_____ None xiii Child Questionnaire aged 10-17 years SECTION 1: BACKGROUND INFORMATION Let’s start out by you telling me a little about yourself. No. Questions Coding Categories 101 Record / Confirm Child’s Name What is your name? 102 Record Child’s Line Letter from Household Schedule (Caregiver Questionnaire) 103* Record / Confirm Child’s Sex Female Male 1 2 104 In what month and year were you born? Month [__|__] Year [__|__|__|__] 105* How old were you at your last birthday? Confirm with 104 and adjust if necessary. Do not leave blank. If child does not know, ask caregiver to estimate age of child. [__|__] years 106 Who takes care of you? Do not read responses. Record one primary response only. Mother and/or father Sister and/or brother Aunt and/or uncle Grandmother and/or Grandfather Other relative Neighbor Friend No one/self Other: __________________ 1 2 3 4 5 6 7 8 66 --- END OF SECTION --- xiv SECTION 2: DIARY I would like you to talk to me about your day yesterday. If yesterday wasn’t a school day, ask about last school day. No. Questions Coding Categories SKIP 201 When did you get up – would you say, before the sun was up/it got light or after the sun was up/it got light? Before sunrise After sunrise 1 2 If After: 203 202 And what did you do after you got up, but before it got light? Anything else? Mark X in all applicable boxes in diary 203 Now, thinking about the time between when the sun came up/it got light and noon/the middle of the day, what did you do? Anything else? Mark X in all applicable boxes in diary 204 And around noon, what did you do? Anything else? Mark X in all applicable boxes in diary 205 Now, let’s think about the time between noon sundown/when it started to get dark, what did you do? Anything else? Mark X in all applicable boxes in diary 206 Now, let’s think about after it got dark. What did you do before you went to sleep? Anything else? Mark X in all applicable boxes in diary Instructions: Ask about the time frames one at a time; probe for additional activities before going on to the next time frame. Every column should have at least one activity box marked. Multiple activities permitted. Do not read response options. Activity Time 202 Before sun-up 203 Sun￾up to noon 204 Noon 205 Noon to sun￾down 206 After sun￾down Sleep Meal Household chores Work on family / HH farm Care for HH member - child Care for HH member - adult School attendance School work Work (excluding HH chores) Informal recreation/leisure xv Organized recreation/club Other: specify ________________________________ SECTION 3: EDUCATION No. Question Coding Category SKIP 301 * Are you currently enrolled in school? Yes (correct diary) No 1 2 If No: 306 302 * During the last school week, did you miss any school days for any reason? Yes No 1 2 If No: 304 303 Why did you miss school days during the last school week? Do not read responses. Circle one primary response. No money for school materials, transport I am too sick to attend school School is too far away / no school I have to work I have to care for HH members Parent/guardian does not want me to go to school I don’t like school School was not in session Other: ___________________________ 1 2 3 4 5 6 7 8 66 304 * What grade/form/year are you in now? [__|__] All: 307 305 Why do you NOT go to school? Do not read responses. Circle one primary response. No money for school materials, transport I am too sick to attend school School is too far away / no school I have to work I have to care for HH members Parent/guardian does not want me to go to school I don’t like school School was not in session Other: ___________________________ 1 2 3 4 5 6 7 8 66 306 Have you ever attended school? Yes No 1 2 If No: 401 307 * Were you enrolled in school during the previous school year? Yes No 1 2 If No: 309 xvi 308 * What grade/form/year were you in during the previous school year? [__|__] All: 401 309 * What is the highest grade/form/year that you have completed? [__|__] SECTION 4: CHORES & WORK No. Questions Coding Categories SKIP 401 Check DIARY. Were the HH chores and/or care for your family or HH, mentioned? Yes No 1 2 If Yes: 403 402 Do you sometimes do HH chores, or care for a member of your HH? Yes (correct diary) No 1 2 If No: 405 403 What HH chores do you usually do? Anything else? Multiple responses possible; circle all mentioned. Probe with response categories if necessary. Corroborate with diary. Prepare food 1 Fetch water 2 Clean toilets 3 Take care of children 4 Plant/tend to/harvest crops 6 Feed, care for animals 7 Wash clothes, blankets 8 Other: ____________________________ _ 66 404 About how much time do you spend per day doing HH or farm chores for your family? Less than 1 hour 1-2 hours 3-4 hours More than 4 hours / most of the day It depends / it is different everyday 1 2 3 4 5 405 Check DIARY Was other work mentioned? Yes No 1 2 If Yes: 407 406 Apart from these chores, do you sometimes do other work outside your home? Yes (correct diary) No 1 2 If No: 411 xvii No. Questions Coding Categories SKIP 407 What kinds of other work do you sometimes do? Anything else? Multiple responses possible; circle all mentioned. Probe with response categories if necessary. Corroborate with diary. Hawk goods 1 Sell food at market 2 Household / farm chores for other families 3 Work in a restaurant or bar 4 Help out in shop 5 Construction 6 Sewing 7 Mechanic 8 Clerk, Delivery, Administrative 9 Other: __________________________ 66 408 How often do you do other work? Would you say….? Read response categories Every day / most days 1 If Every day (1): 409 All others: 410 Several times a week 2 Once a week 3 Once in a while 4 409 About how much time do you spend per day doing this work? Less than 1 hour 1-2 hours 3-4 hours More than 4 hours It depends / it is different everyday 1 2 3 4 5 410 Have you ever received money for any of the work that you do? Yes No 1 2 411 What [else] do you do to get money? Nothing Begging Other: __________________________ 1 2 66 If work mentioned , return to 406-410. 412 What do you do with the money you get? Anything else? Multiple responses possible; circle all mentioned. Probe with response categories if necessary. Give to parents / guardians 1 Pay for my school expenses 2 Pay for school expenses of others 3 Buy food for myself 4 Buy food for others 5 Buy other things for myself 6 Save it 7 Other: __________________________ 66 xviii SECTION 5: FOOD AND ALCOHOL CONSUMPTION Next I would like to ask you about what you eat and drink. No. Question Coding Category SKIP 501 In the past four weeks, did you have to eat a smaller meal than you felt you needed because there was not enough food? Yes No 1 2 If No: 503 502 If yes – How many times did this happen? Read out responses. Rarely (1-2 times in past 4 weeks) Sometimes (3-10 times in past 4 weeks) Often (more than 10 times in past 4 weeks) 1 2 3 503 In the past four weeks, did you have to skip a meal because there was not enough food? Yes No 1 2 If No: 505 504 If yes – How many times did this happen? Read out responses. Rarely (1-2 times in past 4 weeks) Sometimes (3-10 times in past 4 weeks) Often (more than 10 times in past 4 weeks) 1 2 3 505 In the past four weeks did you go to sleep at night hungry because there was not enough food to eat? Yes No 1 2 If No: 507 506 If yes – How many times did this happen? Read out responses. Rarely (1-2 times in past 4 weeks) Sometimes (3-10 times in past 4 weeks) Often (more than 10 times in past 4 weeks) 1 2 3 507 * In the past four weeks did you go a whole day and night without eating anything because there was not enough food to eat? Yes No 1 2 If No: 509 508 If yes – How many times did this happen? Read out responses. Rarely (1-2 times in past 4 weeks) Sometimes (3-10 times in past 4 weeks) Often (more than 10 times in past 4 weeks) 1 2 3 509 Have you ever consumed a drink containing alcohol including beer, spirits – that is a whole glass or drink, not just a taste? Yes No 1 2 If No: 601 510 When was the last time you consumed a drink containing alcohol? Read out responses. Yesterday / a few days ago About a week ago 1 2 xix More than a week ago 3 511 How often does it happen that you consume a drink containing alcohol? Read out responses. Only once in a while At least once a week 1 2 SECTION 6: HEALTH, SUPPORT AND PROTECTION Now I have a few questions about your health and well-being. No. Question Coding Category SKIP 601* Do you have a birth certificate? Yes No Don’t know 1 2 88 If No: 603 If DK: 603 602* Could you please show me your birth certificate? Seen / confirmed Not seen / not confirmed 1 2 603* At any point in the last 2 weeks, have you been too sick to participate in daily activities? Yes No 1 2 604 Do you have a disability that makes it difficult for you to participate in daily activities? Yes No 1 2 If No: 606 605 How would you describe your disability? Blind or partially blind Deaf or partially deaf I have difficulties learning Physical Other _________________________ 1 2 3 4 66 606* I’m going to ask you a few questions about people in your life. Please respond yes or no. Do you have someone in your life to turn to for suggestions about how to deal with a personal problem? Yes No 1 2 607* Do you have someone in your life to help with daily chores if you were sick? Yes No 1 2 608* Do you have someone in your life that shows you love and affection? Yes No 1 2 609* Do you have someone in your life to do something enjoyable with? Yes No 1 2 SECTION 7: HIV/AIDS KNOWLEDGE, ATTITUDES & SEXUAL BEHAVIOR xx Section may be restricted to ages 13-17 only We are nearly done. I have a few short questions on a disease called HIV/AIDS. No. Question Coding Categories SKIP 701 Have you ever heard of an illness called AIDS? Yes No 1 2 If No: 801 702 Can people reduce their chances of getting the AIDS virus by having just one uninfected sex partner who has no other sex partners? Yes No Don’t know / Not sure 1 2 88 703 Can people reduce their chance of getting the AIDS virus by using a condom every time they have sex? Yes No Don’t know / Not sure 1 2 88 704 Is it possible for a healthy-looking person to have the AIDS virus? Yes No Don’t know / Not sure 1 2 88 705 Can people get the AIDS virus from mosquito bites? Yes No Don’t know / Not sure 1 2 88 706 Can people get the AIDS virus by sharing food with someone who has AIDS? Yes No Don’t know / Not sure 1 2 88 707 Can the virus that causes AIDS be transmitted from a mother to her baby: a) During pregnancy? b) During delivery? c) By breastfeeding? Ye s No D K a) During pregnancy 1 2 8 b) During delivery 1 2 8 c) By breastfeeding 1 2 8 708 I have a few more questions about HIV. If you don’t want to answer, that is all right. I don’t want to know the results, but have you ever been tested to see if you have the AIDS virus? Yes No Don’t know 1 2 88 If No: 710 If DK: 710 709 I don’t want to know the results but did you get the results of your test? Yes No 1 2 710 Do you know of a place where people can go to get tested for the AIDS virus? Yes No 1 2 SECTION 8: ACCESS TO HIV PREVENTION, CARE & SUPPORT We have arrived at the last section of the questionnaire. We are almost finished. Thank you very much for your participation so far. xxi Instructions: Respondents should respond only for services that they personally have received. The caregiver or head of HH will also be asked. Data may be cross-checked. OR, this question may be posed to either the adult or the child (instead of both). 801 I am going to read out a list of items and services. Please tell me if YOU have received or accessed any of these items of services in the last 6 months. Read out services. Confirm responses with caregiver. Circle final responses. [ADD / DELETE ITEMS AS RELEVANT TO PURPOSE] a) Health care from a health professional Yes No b) Home visit from a community worker or social worker Yes No c) Free school supplies or a school uniform Yes No d) Mosquito net Yes No Ages 13-17 e) Information on how to prevent HIV and other sexually transmitted infections Yes No f) Information on birth spacing Yes No g) Livelihood training Yes No Ages 15-17 h) Life skills training Yes No --- END OF SECTION --- Thank you! I have come to the end of my questions. Is there anything you would like to add or ask us? I very much appreciate your time today. If you have any further questions about the survey, please use the contact information on your consent form I am leaving with you. Thank you for participating in this interview! 013 END TIME [__|__|:[__|__] xxii Child Questionnaire aged 0-9 years (for Caregiver) SECTION 1: CHILD HEALTH & PROTECTION I am now going to ask you a few questions about [insert child’s name]. No. Question Coding Category SKIP 101 Record / Confirm Child’s Name 102 Record Child’s Line Letter from Household Schedule (Caregiver Questionnaire) 103 * Record / Confirm Child’s Sex Female Male 1 2 104 In what month and year was [NAME] born? Month [__|__] Year [__|__|__|__] 105 * Remind me, how old was [NAME] at their last birthday? Confirm with 104 and adjust if necessary. Do not leave blank. If unknown, ask caregiver to estimate. [__|__] years 106 Would you say that in general [NAME’s] health is……? Read out responses. Excellent 1 Very good 2 Good 3 Fair 4 Poor 5 107 * In the last 2 weeks, has [NAME] been too sick to participate in daily activities? Yes No 1 2 108 Does [NAME] have a disability that makes it difficult for him/her to participate in daily activities? Yes No 1 2 If No: 110 109 How would you describe [NAME’s] disability? Blind or partially blind Deaf or partially deaf I have difficulties learning Physical Other______________ _______ 1 2 3 4 66 xxiii 110 * Does [NAME] have a birth certificate? Yes No Don’t know 1 2 88 If No: 112 If DK: 112 111 * Could you please show me [NAME’s] birth certificate? Seen / confirmed Not seen / not confirmed 1 2 No. Question Coding Category SKIP 112 FILTER. Age of child 5 years or older 0-4 years 1 2 If 5+ years: 128 113 * Do you have a card where [NAME’s] vaccinations are written down? If yes, ask for card. Yes, seen Yes, not seen No Don’t know 1 2 3 88 If No: 113 If DK: 113 114 * Check name on card to make sure card relates to child in question. Document the vaccinations recorded on the card. Only include documented vaccinations here. Yes, documented No a) BCG 1 2 b) OPV 0 1 2 c) OPV 1 1 2 d) OPV 2 1 2 e) OPV 3 1 2 f) DPT 1 1 2 g) DPT 2 1 2 h) DPT 3 1 2 i) Measle s 1 2 If caregiver cannot produce a vaccination card for child, probe for vaccinations below. If you have documented the vaccinations from a card, but there are gaps in the vaccination record, probe with questions below. 115 * Has [NAME] received a vaccine against tuberculosis, that is, an injection in the arm or shoulder, that usually causes a scar? (BCG) Yes No Don’t know 1 2 88 116 * Has [NAME] received the polio vaccine, that is, drops in the mouth? Yes No Don’t know 1 2 88 If No: 121 xxiv If DK: 121 117 * Has the child received OPV0, that is the first polio vaccine normally received in the first two weeks after birth? Yes No Don’t know 1 2 88 118 * Has the child received OPV1, that is the second polio vaccine? Yes No Don’t know 1 2 88 119 * Has the child received OPV2, that is the third polio vaccine? Yes No Don’t know 1 2 88 120 * Has the child received OPV3, that is the fourth polio vaccine? Yes No Don’t know 1 2 88 No. Question Coding Category SKIP 121 * Has the child received the DPT vaccination, that is, an injection given in the thigh or buttocks, sometimes at the same time as polio drops? Yes No Don’t know 1 2 88 If No: 123 If DK: 123 122 * How many times was the DPT vaccine received? Once Twice Three times Don’t know 1 2 3 88 123 * Has the child received a measles injection, that is, a shot in the arm at the age of 9 months or older – to prevent him or her from getting measles? Yes No Don’t know 1 2 88 124 * Has [NAME] had diarrhea in the last 2 weeks? Yes No 1 2 125 * Has (NAME) been ill with a fever at any time in the last 2 weeks? Yes No 1 2 xxv 126 Sometimes adults taking care of children have to leave the house to go shopping, wash clothes, or for some other reasons, and have to leave young children. On how many days in the past week was [NAME] left alone for more than one hour? [__|__] days 127 On how many days in the past week was [NAME] left in the care of another child (that is, someone less than 10 years old) for more than an hour? [__|__] days 128 Did [NAME] sleep under a mosquito net last night? Yes No 1 2 129 I don’t want to know the results, but has [NAME] ever been tested to see if he/she has the AIDS virus? Yes No 1 2 If No: 201 130 I don’t want to know the results, but do you know the result of [NAME’s] test? Yes No 1 2 SECTION 2: CHILD EDUCATION AND WORK No. Question Coding Category SKIP 201 Filter: Age of child (Question 402) 5 years or older 3-4 years 0-2 years 1 2 3 If 3-4 years: 213 If 0-2 years: 301 I now have some questions for you about [NAME’s] schooling and chores. 202 * Is [NAME] currently enrolled in school? Yes No 1 2 If No: 206 203 * During the last school week, did [NAME] miss any school days for any reason? Yes No 1 2 If No: 205 xxvi 204 Why did [NAME] miss school days during the last school week? Do not read responses. Circle one primary response. No money for school fees, materials, transport Child is too sick to attend school School is too far away / no school Child has to work to help family Child needs to care for sick HH members Child does not like school Other: _______________________ 1 2 3 4 5 6 6 6 205 * What grade/form/year is [NAME] in now? [__|__] All: 208 206 Why is [NAME] not enrolled in school? Do not read responses. Circle one primary response. No money for school fees, materials, transport Child is too sick to attend school School is too far away / no school Child has to work to help family Child needs to care for sick HH members Child does not like school Child is too young to attend school Other: ______________________________ _ 1 2 3 4 5 6 7 6 6 207 Has [NAME] ever attended school? Yes No 1 2 If No: 211 208 * Was [NAME] enrolled in school during the previous school year? Yes No 1 2 If No: 210 209 * What grade/form/year was [NAME] in during the previous school year? [__|__] All: 211 210 What is the highest grade/form/year that [NAME] has completed? [__|__] 211 In the past 6 months, has [NAME] worked for money or kind? Yes No 1 2 xxvii If No: 301 No. Question Coding Category SKI P 212 What did [NAME] do to earn these wages? Probe: Anything else? Multiple responses possible. Circle all mentioned. House chores, childcare for other family 1 All: 301 Selling/Hawking goods 2 Labor, e.g., farm, construction 3 Other: ____________________ 66 213 Does [NAME] attend any organized or early childhood education program, such as a private or government facility, including kindergarten or community childcare? Yes No 1 2 If Yes: 301 214 In the past 3 days, did you or any HH member over 15 years of age engage in any of the following activities with [NAME]: Read out a through f one at a time. Yes No a) Read books to or looked a picture books with [NAME]? 1 2 b) Told stories to [NAME]? 1 2 c) Sang songs to [NAME] or with [NAME] including lullabies? 1 2 d) Took [NAME] outside of the home, compound, yard or enclosure? 1 2 e) Played with [NAME]? 1 2 f) Named, counted, or drew things with [NAME]? 1 2 SECTION 3: FOOD CONSUMPTION No. Question Coding Category SKIP 301 Filter: Age of child (Question 402) 2 years or older 0-1 years 1 2 If 0-1 years:401 Next I would like to ask you about what [Name} eats and drinks. xxviii 302 In the past four weeks, did [NAME] have to eat a smaller meal than you felt was needed because there was not enough food? Yes No 1 2 If No: 304 303 If yes – How many times did this happen? Read out responses. Rarely (1-2 times in past 4 weeks) Sometimes (3-10 times in past 4 weeks) Often (more than 10 times in past 4 weeks) 1 2 3 304 In the past four weeks, did [NAME] have to skip a meal because there was not enough food? Yes No 1 2 If No: 306 305 If yes – How many times did this happen? Read out responses. Rarely (1-2 times in past 4 weeks) Sometimes (3-10 times in past 4 weeks) Often (more than 10 times in past 4 weeks) 1 2 3 306 In the past four weeks did [NAME] go to sleep at night hungry because there was not enough food to eat? Yes No 1 2 If No: 308 307 If yes – How many times did this happen? Read out responses. Rarely (1-2 times in past 4 weeks) Sometimes (3-10 times in past 4 weeks) Often (more than 10 times in past 4 weeks) 1 2 3 308 * In the past four weeks did [NAME] go a whole day and night without eating anything because there was not enough food to eat? Yes No 1 2 If No: 401 309 If yes – How many times did this happen? Read out responses. Rarely (1-2 times in past 4 weeks) Sometimes (3-10 times in past 4 weeks) Often (more than 10 times in past 4 weeks) 1 2 3 SECTION 4: ACCESS TO HIV PREVENTION, CARE AND SUPPORT xxix No. Question Coding Category 401 I am going to read out a list of items and services. Please tell me if [child’s name] has received or accessed any of these items or services in the last 6 months. Read out services. Yes No i) (Psychosocial) counseling for a home visitor or social worker 1 2 j) Health care from a health professional 1 2 k) School fees paid for by organization 1 2 l) Free school supplies or a school uniform 1 2 m) Vitamin A supplement from an organization 1 2 n) Supplemental, emergency feeding 1 2 --- END OF SECTION --- Thank you! I have come to the end of my questions. Is there anything you would like to add or ask us? I very much appreciate your time today. If you have any further questions about the survey, please use the contact information on your consent form I am leaving with you. Thank you for participating in this interview! END TIME [__|__|:[__|__] xxx Annex 4: SOCY and BOCY Cost Models Please see attached xls files that contains SOCY and BOCY Cost Models (instructions included in each file). xxxi Annex 5: Findings, Conclusions and Recommendations Summary Table Finding Conclusions Recommendations IE1: Does participation in SOCY activities contribute to a reduction in household vulnerability and improve OVC well-being? -SOCY did not make a significant reduction to HH vulnerability (as measured by the HVAT) relative to its comparator control group from 2016-2019. However, both groups experienced initial reductions to vulnerability from 2017-2018, even as this trend was reversed to witness overall increases to vulnerability for both groups from 2018-2019 -The lack of significance may have been due to potential contagion, contamination and implementation infidelity issues caused by shifting President's Emergency Plan for AIDS Relief (PEPFAR) priorities towards HIV suppression, or due to an overall lack of intervention potency -There were significant differences between control and treatment group HHs in the Income Privileges and Child Protection subdimensions, i.e. treatment group HHs demonstrated positive improvements in the former even as they experienced declines in the latter. -Explanations involve increased reporting of violence issues, increased alcoholism and resistance to Women’s Empowerment -The SOCY intervention was not potent enough to significantly reduce HH vulnerability relative to the control group -PEPFAR COP shifts in geography make contagion more likely; rendering original measures irrelevant -Although IE result was not significant, SOCY did open eyes to broader concepts of violence, empowering better reporting, even as alcoholism undermined SOCY-sponsored HES. -The pathway to reduced vulnerability can be marked by violence, especially for women -When integrated programming is evaluated, consider more nimble analytical tests better suited to manage complexity and support adaptive management -ALL HES, Livelihood and Resilience programming MUST elevate stronger GBV and Child Protection concerns as integral to their intervention. consider ways to better incentivize men to attend and identify BCC support EQ1: What processes are in place to facilitate long-term success? -The Case Management system, administered by District OVC Committees (DOVCCs), Subcounty OVC Committees (SOVCCs) and Case Management Conferences (CMCs), and supported by the various OVC Program group options and ongoing Parasocial Worker (PSW) visits, constituted the best process needed to facilitate long-term graduation from vulnerability -PSWs served as The “foot soldiers” of OVC Case Management, and The bulk of cases were resolved at The CMC level -SOVCCs assisted with The presentation of more complex cases, typically criminal in nature, that were resolved at The DOVCC level. -DOVCCs also solved cross-subcounty issues, shaped policy relevant to Case Management, compiled and reviewed OVCMIS data, and lobbied with central authorities for additional funding. -The recent 2020 National Child Policy says nothing about lower-level mechanisms to perform the bulk of case management and provide local solutions, so that only the most complex cases requiring special legal or policy changes will reach the district level. -OVC Program clarified referral pathways from ground to the DOVCC level; elevated importance of DCDO, supported case compilation (e.g. OVCMIS)-The future of the case management system, especially at the lower, subdistrict levels, is currently in flux and in doubt. -Without an effective lower-level filtering mechanism similar to the CMC and SOVCC in place, the ability and capacity of the proposed DCWC should be seriously questioned. -National Child Protection Policy should further elaborate local govern case management mechanisms and commit subdistrict resources -USAID should support some version of SOVCC and CMC to advance its OVC program investments EQ2: What mechanisms have worked best to support the integrated OVC implementation approach? xxxii Finding Conclusions Recommendations -Overall OVC Program Graduation Success was around 41% -BOCY’s HV referrals; and SOCY’s HH assessment + HIV counseling services worked best to help HHs know their HIV status -PSW “nudges” was the most effective mechanism to ensure HIV treatment adherence; more than clinical counseling -BOCY’s Sinovuyo combined with HIV prevention education; and SOCY’s HIV prevention education worked best to increase knowledge about HIV prevention -BOCY’s VSLA; and SOCY SILC plus nutrition assessments and birth registration were the most effective combinations to ensuring nourishment. -BOCY VLSAs combined with finance literacy; even as SOCY’s SILC worked to improve financial stability. However, achieving Stable Finance was the biggest challenge for both SOCY and BOCY. PEPFAR’s measurement guidance for this benchmark is flawed. -BOCY’s psychosocial and gender-based violence; and SOCY family group supports worked to reduce violence. Yet actual violence may be underreported. -Both SOCY and BOCY saving group models, combined with PSW confirmations of enrollment, worked best to ensure children were in schools. -A new PSW renumeration scheme is in place to offer additional incentives to PSWs. -OVC program was essential to advancing 95-95-95 goals and “holding the line” against further vulnerability -However, less than half of enrolled HHs achieved PEPFAR BN-based graduation -OVC services and groups supports worked in various combinations across contexts for respectable levels of Individual BN achievement -Some PEPFAR measures, e.g. financial stability, are mis-calibrated to Uganda and underrepresent OVC program success -Domestic violence is no longer an “unintended consequence” but can be a likely outcome of HH improvement -PSWs are first responders to HH violence. ‘-New PSW renumeration scheme should be bolstered with additional training but studied for cost effectiveness and sustainability -ALL HES, Livelihood and Resilience programming MUST elevate stronger GBV and Child Protection concerns as integral to their intervention. Consider ways to better incentivize men to attend and identify BCC support -Future integrated activities of similar or greater size and scope should dedicate resources to managing complexity, independently monitor and conduct nimble analyses on the fly to test and refine hypotheses EQ3: Do households or individuals with specific characteristics (e.g., individuals with disabilities, single-parent households, regional variation, etc.) progress and graduate at different rates? -Low Need and Income Insecure HHs were 0.8 and 0.7 times more likely to graduate -Least Insecure were 1.4 times more likely to graduate than Most Insecure -Job Insecure HHs were 0.6 times more likely to graduate than Most Insecure -BOCY HHs with disabilities fared better than the norm, single-parent HHs far below the norm -SOCY HHs with disabilities graduated below the norm -OVC graduated and non-graduated households do not significantly differ when more than 50% of the household members are HIV+. Suppression is the largest issue -OVC Program struggled to graduate large HH-sized subgroups -OVC Program struggled to graduate HHs with specific characteristics, but not by much -Similar interventions should consider providing additional support to single-parent HHs and those with disabilities to address the imbalance. -These interventions should also continue the practice of prioritizing services to food insecure HHs in need of support. EQ4: To what extent do females and males experience different benefits from the program? -Through a comparison of graduation rates of predominantly female HHs with the average rates of graduation across BOCY, HHs with more than five females did slightly better than the norm (and better than those with more than five males). However, adult female headed HHs graduated roughly 5% less than the norm. -In somewhat of a contrast to BOCY, SOCY HHs that graduated below the norm include those with more than five females graduated (almost none of these HHs graduated) and female headed households -Households with a higher number of non￾HH head females tend to graduate above the norm for BOCY, but not so with SOCY. -Female headed households also tend to graduate below the norm across the entire program. Gender imbalances do exist across HHs that participate in the OVC Program. Similar interventions in the future should consider providing additional support to female-headed households to address the gender imbalance. EQ5: How cost-effective are different pathways to change? xxxiii Finding Conclusions Recommendations -The ET built an “activity-based implementation costing” model, which suggests an average Unit Cost of Services of $625.60 for BOCY HHs and $475.39 for SOCY HHs. -The ET then drew upon QCA findings to identify group and training services as well as qualitative investigations to identify potential areas for cost-effective saving. Once these adjustments were made, the revised cost-effective cost models suggest an average Unit Cost of Services of $555.65 for BOCY HHs and $408.46 for SOCY HHs, which translates into an approximate 11 percent and 14 percent cost savings per HH, respectively. -However, the resulting model should not be viewed as an accountability tool, i.e. it should not be used for audit purposes or to cross-check total expenditures to date with those predicted by the model. -Using QCA-based cost effectiveness modelling, the OVC program could have introduced 10-15% cost effective savings. ‘-Future interventions may wish to consider combining activity-based implementation costing models with complexity management tools like QCA as an approximate guide to making cost-effective adjustments to integrated programming. xxxiv Annex 6: Year 1 Evaluation Report Executive Summary The purpose of this evaluation is to provide rigorous evidence regarding the extent to which participation in USAID/Uganda’s Orphans and Vulnerable Children (OVC) program has reduced household vulnerability and improved OVC well-being, and to identify factors or combinations of factors, within and outside the program, which may explain how results were achieved or what barriers may have prevented greater success. The results from the final performance evaluation of Sustainable Outcomes for Children and Youth (SOCY) and Better Outcomes for Children and Youth (BOCY), including outcome endline, will inform future programing in Uganda and elsewhere as it relates to the case-management and graduation models utilized by the activities. OVC program The two projects comprising the OVC program, i.e. BOCY and SOCY, were designed following the same Theory of Change and implemented according to similar (but not identical) models, aiming to address multiple forms of vulnerability among children and youth through an integrated, comprehensive approach. This is presented below in the following figure: BOCY and SOCY seek to economically empower orphaned and vulnerable children, youth, and their caregivers to access core services (Result 1); strengthen local government, CSOs, and informal community structures to increase and improve core services for orphaned and vulnerable children, youth, and their caregivers (Result 2); and improve coordination of community-based clinical and socio-economic services for efficiency and effectiveness along the continuum of care (Result 3). Evaluation Scope of Work The OVC Evaluation is tasked with answering the following performance and impact evaluation questions: • Question 1: To what extent are processes in place to facilitate long-term success of program interventions beyond the program’s end? What factors may prevent long-term success? • Question 2: What coordination and management mechanisms have worked best to support the integrated OVC implementation approach? What enabling factors do these mechanisms engage with? xxxv • Question 3: Do households or individuals with specific characteristics (e.g., individuals with disabilities, single-parent households, regional variation, etc.) progress and graduate at different rates & if so, why? • Question 4: To what extent do females and males experience different benefits from the program? Are there unintended results for either group? • Question 5: How cost-effective are different pathways to change? How do interventions/ combinations of interventions compare in impact & cost? • Impact Evaluation Question: Does participation in SOCY activities contribute to a reduction in household vulnerability and improve OVC well-being? Methodology Data collection and analysis for this first report took place from October 2018 until August 2019. Data collection involved a midline household survey of SOCY areas to support the impact evaluation, subsequent document review of SOCY and BOCY documents, several introductory and coordination calls with both SOCY and BOCY teams, and extended data collection and collaboration sessions in Kampala and across the OVC program area from May to July 2019. Analysis was ongoing, but the bulk of the analysis and report writing was conducted from June to August 2019. Data Collection Exercises related to the Year 1 Report SOCY Survey Midline Data Collection Nov. – Dec. 2018 Surveyed 2,730 HHs Document Collection and Desk Review Feb. - Mar. 2019 Met will all senior staff from both projects; field visit in March and collected 47 SOCY and BOCY programmatic and implementation documents Process Mapping and Cost Modelling May - Jun. 2019 Met approximately 30 senior staff from both projects across 3 weeks Implementation Databases and QCA Thresholding May - Jun. 2019 Collected more than 1gb of data total containing more than 100 million data points across both projects; identified binary cut-off points, i.e. "thresholds," in the data with approximately 30 senior staff Qualitative Data Collection Jun. 2019 Conducted 39 FGDs and 2 individual interviews with OVC beneficiaries in 10 districts (5 in BOCY and 5 in SOCY) Building of Process Maps with BOCY and SOCY staff Jun. - Jul. 2019 49 CSOs and organizations mentioned by the CSOs; data collected through face to face interviews and by phone Findings Question 1 The OVC program’s Household (HH) enrollment process assumes a fairly sophisticated division of labor and case management system. This system rests upon a diverse labor force arrangmenet of social workers (SWs), para social workers (PSWs), monitoring and evaulation (M&E) Officers, specialized service providers, coordinators and case facility linkage officers (CFLOs). While any follow-on OVC service provider will be able to rest upon the enrollment system identified above, the main question for the next wave of this evaluation’s data collection will be to what extent this division of labor can be streamlined and simplified. xxxvi Once enrolled, HHs enter a wide gamut of services spanning across savings and loan, income generation, hygiene, youth and children (espescialy girls), temporary consumption support, psycho-social, child protection, and, increasingly, facility-referral services, esp. HIV testing, treatment and counciling services. These service packages benefited from BOCY’s and SOCY’s capacity to engage, assess and implement evidence-based programming from activities across the region. This evidence directly informed the design of many of the services SOCY and BOCY currently offer households. HH case management currently follows a complex stream of services. The delivery of starts with the PSW ath the HH level, and, depending on the demographic of the HH member, branches off into various groups and community clubs. HH case management also benefits from regular SOVCC (monthly and quarterly) and DOVCC (quarterly) coordination meetings that manage the flow of case information, and further facilitate referral processes. Yet the evaluation team found that some of these local government structures failed to operate when the previous OVC-support activities (SUNRISE and SCORE) ceased operating, which lead to start up delays and inefficiencies as both SOCY and BOCY began. HH case management implmentation benefits SOCY and BOCY national capacity building and advocacy efforts. Current implementers believe this may help contribute to the OVC program’s longer term success. Yet the evaluation team also found that HH case management unfolds in wider networks of collaborative resource sharing across local and regional levels, espescially in the local organizations MUCOBADI and LADA, as well as across activities such as LARA and RHITES-North. These organizaitons and activities may be potential candidates to explore conversations around greater collaborative resource sharing of beneficiary, technical assistance, trainer, and administrative resources. Impact Evaluation Question The evaluation team found that, through a more refined and accurate approach to calculating the HVAT, HHs enrolled in the SOCY program appear to be gradually moving from more vulnerability to less vulnerability. From the perspective of the impact evaluation, HHs in the control group also appear to be reducing their vulnerability over the same time period. As a result, the evaluation team has not yet found evidence to prove that declines in HH vulnerability can be attributed to SOCY in this midline survey. However, these results are not yet conclusive, as many of the HHs enrolled in the treatment group have not yet completed the SOCY program. The implications are that it is still plausible to expect that SOCY treatment HHs will not only continue to decline in vulnerability but will do so at a faster rate than those in the control group. As there are no substantive recommendations that can be made until the results are conclusive, the endline survey should therefore continue. The evaluation team also reviewed progress on several PEPFAR OVC indicators. Through means-testing exercises, the evaluation team found that conceptualizing the primary caregiver as the main force of change in the household is not enough. The non/significant differences across groups on indicators related to child and youth point to a more complex picture than the simple one of the primary caregivers being the primary influencer of change in the household. This may suggest that some of the benefits the primary caregiver gains while attending SOCY are not significantly passed down to the children and youth of the household. Put differently, the distribution of SOCY benefits, where they exist, are uneven from caregiver to child. As with the HVAT comparisons, the OVC indicator results are not yet conclusive, especially because many of the households enrolled in the treatment group have not yet completed the xxxvii SOCY program. The implications are that it is still plausible to expect that SOCY treatment households will improve more than the control group households. Question 2a: QCA Model In this wave, the evaluation team extracted the relevant database information needed to conduct the QCA and cleaned the dataset so that an effectiveness evaluation of the OVC program can be taken forward into the next wave of QCA analysis. However, the monitoring data on SOCY and BOCY program implementation requires further refinement. In the case of SOCY, data on service delivery are only available from October 2017 onwards, and the evaluation cannot model SOCY’s implementation data against graduation until the upcoming Pre-Graduation Assessment (PGA) has occurred. When that takes place, the QCA analysis will be conducted. In the case of BOCY, the evaluation team identified 3 improvements to the data: • First, the current BOCY graduation approach leads to the graduation of HHs that substantially improved their resilience, but also designates this status to HHs that have experienced little or no improvements in vulnerability. In the next wave, the evaluation team will work with BOCY to develop different types of graduation outcomes that separates substantial improvements from minor improvements to reach graduation. This will facilitate more differentiated learning around the various pathways out of vulnerability. • Second, BOCY monitoring data suggests that participation in group-based services is short-lived. Few HHs seem to have participated in groups such as Girls First Clubs or VSLAs long enough to justify the assumption that they had any impact on them, i.e. they typically dropped out after attending only one meeting. Modeling this level of participation in services with HHs who participate more extensively can create misleading results. The evaluation team will work with BOCY to determine if future attendance will be forthcoming and, if not, determine if these low levels of attendance should be considered adequate. • Third, a substantive share of services is being delivered to HHs that have already fulfilled BOCY’s graduation criteria. This distorts the ability of our analysis to identify which combination of services are sufficient to move HHs to graduation. • Fourth, based on the monitoring data, we are unable to detect any patterns in service delivery that are closely tailored to specific HH vulnerabilities. Many HHs received a general package, but not the tailored services the BOCY HAT suggested they receive. That is, there is room to improve the alignment of the solutions BOCY provides to the problems HHs face. Implementing these improvements will make it easier (for QCA analysts as well as program managers) to identify which sets of essential packages best reduce vulnerability across contexts. Given these midpoint challenges, the evaluation team optimistically looks to the next wave of data collection to conduct the QCA analysis to identify which services combine in which ways to help HHs overcome different forms of vulnerability. Fortunately, as graduation and data entry by the BOCY field team proceeds and improves over time, the project’s OVC database will contain rich enough information to overcome these challenges. Question 2b: Qualitative Findings Focus Group Discussion (FGD) participants found most OVC program services beneficial; however they did identify individual and group hierarchies that were more valuable, which was important as the most valuable services were not always the most spoken about. Overall, Savings and Loans models (i.e. Village Savings and Loan (VSLA) and Savings and Internal Lending Communities (SILC)) services are the most valued, closely followed by HIV Testing and Counseling services. These findings are fortunate, not only because they sit at the heart of xxxviii the OVC program theory of change, but they also illustrate that the Ministry of Gender, Labor, and Social Development (MGLSD) goal of reducing economic vulnerability is not mutually exclusive of the Presidents Emergency Plan for Aids Relief (PEPFAR) goal of reducing HIV incidents. Summarized in tabular form, a loose hierarchy of service delivery combinations emerge across both projects: Configurations of Services That Reduce Vulnerability and Their Spoilers (based upon individual, group and social hierarchies) SOCY Priorities BOCY Priorities Essential Combinations of Services S&L + Sponsorship + HIV T&C + Backyard Gardens HIV T&C + S&L + Backyard Gardens + Either Hygiene/Sanitation OR Parenting/Sinovuyo Complementary Combinations Hygiene Sanitation + Training&Skills + Youth Groups + Either DREAMS OR Referrals Sponsorship + ECD Training + Community Meetings + Nutrition Supplemental Service Combinations Emergency Fund + TCS OVC/Emergency Fund + Nutrition SOCY Spoilers BOCY Spoilers Main Spoiler Sponsorship offered, esp. payment of school fees, then suspended (in both SOCY & BOCY) "Promises Not Delivered" and Non￾Substantive Spoilers Lack of Temp. Consumption Support + Unprofessional Trainers + Training Promised but not delivered Inadequate Capital to Implement + Incomplete Training The biggest areas of concern are twofold: the first revolves around what FGD respondents called “failed promises”, i.e. situations where services were promised but then not fully delivered. This was especially problematic in situations where the HH made important decisions based upon these expectations, such as further school attendance, but then had to make uncomfortable adjustments when that service was not delivered. While these situations were not completely within the control of SOCY and BOCY, greater emphasis on PSW management of HH expectations would be useful. Questions 3 & 4 The evaluation team constructed eight HH subcategories and two demographic ones. We further found that five of the eight HH subgroup categories met or exceeded the norm. Those households with (i) less than 6 members (13.4%), (ii) less than 3 children (14.7%), (iii) households with more females, and (iv) households with less than 30% HIV+ members (12.9%) all indicated higher or equal to the 12% norm in terms of graduation rates. The main takeaways from these “above the norm” subgroups are that the foci of the programming are geared to (i) female-majority and (ii) smaller households, particularly HHs with smaller proportions of HIV prevalence. The subgroups that did not meet the criteria were predominantly (i) large households (10.4%), (ii) with more than 5 children (10.2%), (iii) with more children in the household than adults (11%), and (iv) more than half the household members were HIV+ (10.6%). The main takeaway from these findings is that BOCY struggles to graduate these large HH-sized subgroups relative to the overall norm of graduation and the smaller subgroups. Large HH￾sized subgroups may require additional support that the project is not able to provide. xxxix Outside of HH characteristics, we applied the same approach to geographic and enrollment characteristics of the households. Those HH enrolled in the initial two years of BOCY are, not surprisingly, ahead of the norm in terms of graduation, while the others fall below (no HH should be graduating in 2019). There is no discernable graduation pattern in regard to geography. A larger number of districts witnessed HHs that graduate below the norm, but some of these are newly created districts and push the overall number of “below-average” graduation districts up. Question 5 The evaluation team built a unit cost model of services across BOCY and SOCY projects. One of the most interesting findings is the high-level disproportion of between the costs of services delivered by a PSW and their corresponding supervisory and overhead costs. The average PSW monthly cost is 30,000 Ugandan Shillings (UGS), while the average monthly SW costs, who supervise up to 15 PSWs, is 4,762,800 UGS. This large imbalance is due mostly to the fact that PSWs are volunteer labor, while SWs are full time employees. Should the OVC program wish to sustain the social services structure to maintain progress, findings from the previous sections suggested that an expended PSW workforce to intensify services and encourage greater HH participation in group activities may be beneficial. Indeed, the marginal costs of additional PSWs is minimal and therefore the best way to optimize value for money would be to expand the PSW labor force while keeping their monthly labor costs constant but at the same time increasing the number of PSWs monitored by SWs. Allowing the SWs to supervise more PSWs then mitigates the very substantial cost increase associated with hiring more SWs. Similar expansions to the PSW labor force to support various group activities (and encourage HH members to attend these group meetings more frequently) would also add more value for money, assuming that PSW unit costs remain constant while the ratio of group supervisors to number of PSWs increases. The justification for expanding the number of PSWs supervised by the higher-level staff categories is that, even though PSWs are the cheapest labor cost, they are also the main point of entry for HH service delivery. They therefore should be optimized. However, the evaluation team also found that PSWs require additional training, mentoring and supervision. Considering the prohibitive monthly salary costs of additional SWs, alternative, intermediary supervisors, such as the Local Council 1 (LC1), could potentially fulfill this requirement in a cost-effective manner. Additionally, the development of a PSW career-track may also serve to help professionalize the PSW workforce, which could also increase service deliver quality. Looking forward to the Year 2 Survey report, the evaluation team will further break down the various services by generating “time on task” estimates captured through discussions and SMS surveys with PSWs and other staff. These results will enhance the unit cost model with more precise estimates of the incremental costs associated with progress towards the implementation of the Household Improvement Plans (HIP) and graduation. The evaluation team also optimistically looks to the next wave of data collection to finalize the impact evaluation as well as conduct the QCA analysis to identify which services combine in which ways to help HHs overcome different forms of vulnerability and to what extent these services are experienced differently across subgroups and between males and females. This will in turn reveal with implementation mechanisms hold the most potential to sustain long￾term results, and the most cost-effective approaches in doing so. Recommendations xl • Any potential OVC program follow-on partners, including local government stakeholders, who continue to deliver the benefits of the OVC case management system may wish to consider building upon the current OVC enrollment system. • Any potential follow-on partners, including local government stakeholders, who continue to deliver the benefits of the OVC case management system may wish to consider building their global capacity to engage, assess and implement evidence-based programming, which would most likely improve the overall probability that longer￾term success could be replicated and sustainted. • Considering that many of the DOVCCs and SOVCCs ceased to exist after the conclusion of the previous OVC program (SUNRISE-OVC) and the withdrawal of direct funding, USAID/Uganda may wish to hold a discussion with MGLSD to identify how these or equivalent local government structures may continue, because the resources required to reactivate them will delay future programming. • When considering collaborative partner organizations to take forward and sustain any potential success achieved by the OVC program, the OVC program may wish to moblizie the current OVC service provider catalysts to leverage various resources, expertise and experience across regions and activities. • Regarding the impact evaluation, there are no substantive recommendations that can be made until the results are conclusive and the endline survey should therefore continue. • BOCY is encouraged to review their graduation criteria to refrain from graduating households if they are unable to stabilize their resilience gains. • OVC program implementers should identify graduation from some type of “more vs less” vulnerability starting points to allow more differentiated learning about pathways out of vulnerability. • OVC program implementers may wish to consider incentivizing the main administrators of their HH visits (PSWs) and various group activities (SILC Supervisors, Youth Officers, etc.) to further encourage greater participation in the various group-based services. • OVC program implementers should set minimum “attendance thresholds” for participation in the various groups. Those HHs with members that do not meet this threshold should then not be counted as receiving that service. • OVC program implementers should stop providing services to HHs that have already graduated. • SOCY is encouraged to conduct the next Pre-Graduation Assessment (PGA) so that the rigorous analysis of their project effectiveness can occur. • BOCY may wish to consider reviewing these findings and investigating whether their services favor the graduation of smaller HH sizes and those HHs containing female majorities. • BOCY may wish to examine why HHs with a higher percentage of HIV prevalence appear sit right within the norm, while those HHs with a lower percentage of prevalence are above the norm. This may suggest more intensification of services for the high HIV prevalence HHs. • Should the OVC program wish to scale up its implementation in a cost-effective manner, implementers may wish to consider allocating more PSWs to the implementation process to support more group services (and encourage more frequent group attendance), as well as provide more intense and frequent HH visits. The reason for this is the PSW adds the most value for money at the lowest costs, even though it will require that SWs supervise a higher number of PSWs. However, this must also xli entail additional PSW training and mentoring to reduce the SW supervision burden, as well as potentially introducing intermediary supervisors, such as LC 1 officials. • Although the evaluation team will estimate the “time on task” PSWs spend providing each service at the HH level, the evaluation team suggests SOCY and BOCY derive their own estimates in order to generate more detailed estimates of the incremental costs associated with progress towards the implementation of the HIP and graduation. • Should the OVC program wish to sustain the social services structure to maintain progress, implementers may wish to consider allocating more PSWs to the implementation process to support more group services (and encourage more frequent group attendance), as well as provide more intense and frequent HH visits. The reason for this is the PSW adds the most value for money at the lowest costs, even though it will require that SWs supervise a higher number of PSWs. • Concurrently, the OVC program, in collaboration with the MGLSD, should also explore how an intermediary supervisory labor force, such as members of the LC1, could better ensure PSW service delivery quality. • Although the evaluation team will estimate the “time on task” PSWs spend providing each service at the HH level, the evaluation team suggests SOCY and BOCY derive their own estimates in order to generate detailed estimates of the incremental costs associated with progress towards the implementation of the HIP and graduation. xlii Annex 7: Year 2 Evaluation Report Executive Summary The Orphans and Vulnerable Children (OVC) Evaluation’s purpose is to provide rigorous evidence regarding the extent to which participation in USAID/Uganda’s Orphans and Vulnerable Children (OVC) program has reduced Household (HH) vulnerability and improved OVC well-being and to identify factors or combinations of factors within and outside the program that may explain how results were achieved or what barriers may have prevented greater success. Future programming in Uganda will be informed by the results from the final performance evaluation of Sustainable Outcomes for Children and Youth (SOCY) and Better Outcomes for Children and Youth (BOCY) in relation to the case-management and graduation models utilized by the activities. The OVC Evaluation seeks to answer the following Performance and Impact Evaluation (PE/IE) questions: ● PE Question 1: To what extent are processes in place to facilitate long-term success of program interventions beyond the program’s end? What factors may prevent long-term success? ● PE Question 2: What coordination and management mechanisms have worked best to support the integrated OVC implementation approach? What enabling factors do these mechanisms engage with? ● PE Question 3: Do HHs or individuals with specific characteristics (e.g., individuals with disabilities, single-parent HHs, regional variation, etc.) progress and graduate at different rates, and if so, why? ● PE Question 4: To what extent do females and males experience different benefits from the program? Are there unintended results for either group? ● PE Question 5: How cost-effective are different pathways to change? How do interventions/ combinations of interventions compare in impact & cost? ● IE Question: Does participation in SOCY activities contribute to a reduction in HH vulnerability and improve OVC well-being? The OVC Evaluation Team (ET) will address these questions in coordination with OVC program stakeholders, including USAID/Uganda, SOCY, BOCY and Ugandan government staff. Note that the IE Question builds off previous work under the Ugandan Learning Contract, which involved initial baseline (2016) and extended baseline (2017) surveys. OVC Program BOCY and SOCY were designed following the same Theory of Change and are implemented according to similar (but not identical) models, aiming to address multiple forms of vulnerability among children and youth through an integrated, comprehensive approach. The following figure presents this below: xliii Figure 1. Results for Addressing Multiple Forms of Vulnerability (Children and Youth) BOCY and SOCY seek to economically empower orphaned and vulnerable children, youth and their caregivers to: ● Result 1: Access core services ● Result 2: Strengthen local government, Civil Society Organizations (CSOs) and informal community structures to increase and improve core services ● Result 3: Improve coordination of community-based clinical and socioeconomic services for efficiency and effectiveness along the continuum of care. Evaluation Design The OVC Evaluation is divided into three deliverable reports: ● Year 1 Report of Interim Findings (FY2019) ● Year 2 Report of Interim Findings (FY2020) ● Year 3 Final Report (FY2021) PE Q1 in the previous Year 1 Report identified the processes in place that may facilitate long-term success of the OVC program and also provided a midline update of the SOCY IE. Through qualitative investigation, PE Q1 also explored how HHs on the ground experienced these processes xliv (see Annex 7 for more details on PE Q1).97 PE Q2–4 demonstrated the variation of these experiences in accordance with demographic, geographic and gender characteristics, including a cost effectiveness model that identified PSW labor as a dynamic point of implementation with potential to scale (PE Q5). This Year 2 Report provides the next iteration of findings and analyses of OVC program implementation in regard to the OVC Evaluation Scope of Work. It is primarily geared towards evaluating the OVC program effectiveness. The OVC Program uses two interrelated yet distinct measures of effectiveness: results and impact. At the results level, the OVC Program’s Implementing Partners (IPs) are evaluated according to their performance on the President’s Emergency Plan for Aid Relief (PEPFAR) February 2019 graduation benchmarks (BN). The ET adopted these BNs as the primary measure of success to further address PE Q2–4. The ET employed two different measures of effectiveness to conduct a more robust analyses of the OVC Program’s performance and impact which facilitated greater learning opportunities. At the impact level, the OVC program initially adopted the Ministry of Gender, Labor and Social Development’s “Household Vulnerability Assessment Tool” (HVAT) as a measure of success. The ET subsequently refined and adopted the HVAT as the primary impact measure for addressing the IE question. We expect the Year 3 Final Report will comprehensively address the question of what mechanisms will facilitate the long-term, cost-effective success of OVC goals across genders and variations in HH characteristics. The ET will achieve this by drawing from the learnings and analysis of the two interim reports across genders and variations in HH characteristics. Limitations COVID-19 imposed unprecedented constraints and limited the findings of this Year 2 Report. Due to major travel restrictions, the ET expatriate workforce was unable to travel to Uganda to support quantitative findings with qualitative investigation. Moreover, COVID-19 restrictions required extended periods of remote work across all stakeholders, which contributed to a variety of IT￾related clarification and data transfer delays involving both SOCY and BOCY implementation databases. As a result, the ET engaged in interactions mostly with BOCY and SOCY M&E and IT technical staff, although program staff consultations also occurred. These limitations implicate the findings of both this Year 2 Report and the IE, being limited to those gained through quantitative data exploration and analysis. Consequently, policy makers should suspend judgement until the ET completes and submits the results of the qualitative investigation. The Year 3 Final Report will subsequently integrate these results and their interpretation. The ET anticipates further qualitative investigation will unfold at the beginning of FY2021. 97 The ET found that HH enrollment in the OVC program assumes a fairly sophisticated division of labor and case management system. On the implementation side, HH enrollment occurs through a diverse labor force arrangement of SWs, PSWs, M&E Officers, specialized service providers, coordinators and, crucially, CFLOs. Once enrolled, HHs enter a wide gamut of services spanning across savings and loan, income generation, hygiene, youth and children (especially girls), temporary consumption support, psycho-social, child protection, and, increasingly, facility-referral services. These service packages benefited from BOCY’s and SOCY’s capacity to engage, assess and implement evidence-based programming from activities across the region. This evidence directly informed the design of many of the services they offer HHs. xlv Data Collection and Analysis Methodology This Year 2 Report draws upon extensive quantitative data to address Questions 2-4 and the IE question. Both SOCY and BOCY granted the ET access to their online databases to address questions of effectiveness related to OVC program implementation and graduation results.98 When addressing the SOCY IE question, the ET deployed the end line survey at the end of 2019. This survey was the fourth and final survey of the same “control” and “treatment” group HHs conducted annually (Nov-Dec 2016, Nov-Dec 2017, Nov-Dec 2018 and Nov-Dec 2019). Across all four waves, the survey consistently gathered data from 1,922 HHs in both control and treatment groups across 15 target districts. This final survey wave implementation experienced no notable difficulties or limitations. The ET conducted analysis of OVC program implementation data primarily with descriptive statistics, logistical and other regression packages such as Binomial Chi Square tests, as well as Qualitative Comparative Analysis (QCA). QCA analysis focused both on Truth Table Analysis as well as Minimization. The ET conducted most analyses using Python scripts and a QCA specialized software package known as fsQCA. The ET conducted analysis of IE data using Difference in Differences (DiD) techniques, factor analysis and various regression techniques to address the impact evaluation question. It conducted this analysis primarily using the MPlus, the R programing language and SPSS. Findings Overall, the ET found mixed results using both the current eight graduation BNs and the HVAT measure used for the impact evaluation. In regard to meeting the eight graduation BNs, the ET identified 24,481 HHs in the database that BOCY had assessed for graduation, of which 11,045 had graduated while the remaining had not, yielding a 44.5 percent graduation rate. The ET also identified 4,452 HHs in the database that SOCY had assessed for graduation; of which 2,210 had graduated while the remaining had not, yielding a 49.6 percent graduation rate. The ET was also anecdotally informed that some of the new requirements introduced to meet these BNs are difficult to achieve in the Ugandan context. For example, BN 5 defines a financially stable HH as one that does not have to sell HH assets to meet health and education expenses. However, for those HHs who often sell HH livestock, such as goats, to regularly cover these reoccurring expenses, meeting the BN requirement may run counter to their regularly established financial practices. The ET will investigate these and other context-specific claims in the Year 3 investigation. In regard to the HVAT used for the Impact Evaluation, the ET found that SOCY has not made a significant reduction to HH vulnerability relative to its comparator control group from 2016-2019. Both control and SOCY treatment HHs experienced an equivalent level of additional benefits from non-SOCY service providers (aka “contamination”), in which SOCY HHs experienced slightly more benefits. 98 The ET replicated and managed these databases through SQL and Python programming language scripts and derived. various logical commands to collate and combine various data source reports. xlvi However, both control and treatment groups experienced initial reductions to vulnerability from 2017-2018, although this trend reversed and both groups witnessed overall increases to vulnerability from 2018-2019. As such, participation in SOCY can contribute to “holding the line” against further increases to vulnerability, but not significantly more so than those in the control group. PE Question 1 To what extent are processes in place to facilitate long-term success of program interventions beyond the program’s end? What factors may prevent long-term success? The ET found that the OVC program transition from PEPFAR 2.0 to 3.0 required significant adjustments to the SOCY and BOCY processes across the treatment cycle.99 Whereas the previous focus was on a wider reduction of human immunodeficiency virus (HIV)-related vulnerabilities, the OVC program has since increased emphasis on containing the disease through strict adherence to meeting the 95–95-95 goals, i.e., 95 percent of people with HIV diagnosed, 95 percent of them on antiretroviral therapy (ART) and 95 percent of them virally suppressed. This transition has entailed changes to the definition of success for each activity. These changes break down into adjustments to enrollment, service provision and graduation tracking and adjustments to the way this information was managed and acted upon. In regard to the first adjustment, the ET constructed a “Needs to Treatment to Graduation” (NTG) matrix based upon IP activity data (see Annex 6). The NTG matrix findings suggest that BOCY was overall effective at maintaining strict adherence to the OVC Program Treatment Cycle, despite shifting enrollment and graduation requirements. SOCY was effective at maintaining adherence to the OVC Program Treatment Cycle as well. Yet SOCY delivered services to some HHs without first assessing their needs, and it remains to be seen if a more substantial portion of these HHs graduate. Turning to the overall activity databases, the ET found IT challenges regarding logical calculation errors (in BOCY), user-entry and syntax errors, lack of documentation, early database operation issues and lack of comprehensive “one-stop shop” services received reports. These challenges have required the M&E teams (as well as the ET) to dedicate effort to data cleaning. SOCY utilized advanced Python programing skills to better mitigate these challenges. The overall implications are that it is difficult to precisely identify how many services HH may have received, which may be associated with potential misreporting. Annex 5 of this report offers potential technology recommendations to help mitigate these challenges moving forward. PE Question 2 What coordination and management mechanisms have worked best to support the integrated OVC implementation approach? What enabling factors do these mechanisms engage with? 99 For PEPFAR 3.0, see https://www.state.gov/wp-content/uploads/2019/08/PEPFAR-3.0-%E2%80%93-Controlling-the￾Epidemic-Delivering-on-the-Promise-of-an-AIDS-free-Generation.pdf for a summary. For PEPFAR 2.0, see https://www.usaid.gov/sites/default/files/documents/1864/PY6_O3.9_Fostering_PEPFAR_Sustainability_through_LMG_FINAL _09.22.17-508.pdf xlvii The OVC program integrates a variety of services to achieve success across graduation BNs. The ET examined graduation BN success rates for both BOCY and SOCY. As of July 1st, 2020, the ET had identified 24,481 BOCY HHs that had been assessed for graduation; of which 11,045 had graduated while the remaining had not, yielding a 44.5 percent graduation rate. As of July 27th, 2020, the ET had identified 4,452 SOCY HHs that had been assessed for graduation; of which 2,210 had graduated while the remaining had not, yielding a 49.6 percent graduation rate. A summary of BOCY and SOCY success in supporting HHs to achieve each BN is as follows: Figure 2 Results for Addressing Multiple Forms of Vulnerability (Children and Youth) OVC Program Graduation BN Assessment Results # of HHs BN Met Unmet Success Rate (%) BN 1 BOCY 5777 5638 139 97.6 BN 1 SOCY 578 569 9 98.4 BN 2 BOCY 1732 958 774 55.3 BN 2 SOCY 25 20 5 80.0 BN 3 BOCY 6163 5548 615 90.0 BN 3 SOCY 433 404 29 93.3 BN 4 BOCY 8611 7081 1530 82.2 BN 4 SOCY 383 359 24 93.7 BN 5 BOCY 14062 9177 4885 65.3 BN 5 SOCY 1055 675 380 64.0 BN 6 BOCY 5899 5601 298 94.9 BN 6 SOCY 113 99 14 87.6 BN 7 BOCY 741 719 22 97.0 BN 7 SOCY 16 14 2 87.5 BN 8 BOCY 10530 8489 2041 80.6 BN 8 SOCY 63 39 24 61.9 The ET used Truth Table and Qualitative Comparative Analysis (QCA) to analyze SOCY and BOCY treatment effectiveness. Truth Table analysis is useful to identify prominent patterns of service delivery combinations, aka “paths,” and to demonstrate how these paths better explain how a BN is achieved as opposed to a narrower focus on each individual service. Truth Table also reveals which service pathways are seldom taken and which produce fewer results. QCA is useful to identify which core service categories may be necessary, sufficient, or irrelevant to achieving the outcome of success. In doing so, QCA also identifies which combinations of enabling service xlviii categories are most effective in contexts where the provision of other services is lacking. QCA results are best conceived as “essential packages” that are necessary and/or sufficient to achieve graduation even though other relevant core services may be lacking. Overall, the Truth Table and QCA findings suggest that there are varied pathways to meeting each BN and achieving HH graduation. Focusing first on Truth Table analysis, the ET found the following trends: ● For BN 1, in which all members of the HH know their HIV status, BOCY’s Home Visit referrals were associated with the highest level of success helping HHs achieve this BN, whereas SOCY’s HH risk assessment combined with HIV disclosure support proved to be their most effective support achieving this BN. ● For BN 2, in which all HIV-positive youth, children and primary caregivers are virally suppressed, both BOCY and SOCY experienced their highest levels of success when their para-social workers confirmed that patients took ART medication alongside the provision of adherence support. , yet both provided additional support in which BOCY also worked in closer coordination with health facilities while SOCY provided HIV literacy support. ● For BN 3, in which all HH youth (ages 10-17) are knowledgeable about HIV prevention, BOCY’s Sinovuyo group combined with additional HIV prevention education was the most common path to success, just as SOCY was able to achieve comparable levels of success only providing the HIV prevention education service. ● For BN 4, in which all children in the HH under 5 have no signs of malnourishment, both BOCY and SOCY experienced the most success providing HH economic support services, but SOCY did so in combination with additional nutrition assessment services and the provision of birth registration support. ● For BN 5, in which the HH has improved financial stability, various savings group schemes proved to be the most common pathway to achieving this BN, and BOCY did so in combination with additional financial literacy support. ● For BN 6, in which the HH reports no violence, a combination of BOCY’s psychosocial and gender-based violence services were the most effective pathway to meeting this BN, while SOCY’s family group supports was the most prevalent path its HHs traveled. ● For BN 7, in which there is no child headed HH, neither SOCY nor BOCY offered a significant level of direct services to achieve this BN, but both did offer various other supports until the eldest youth came of adult age. ● Finally, for BN 8, in which all eligible children and youth (ages 6-17) in the HH attend school, BOCY’s previously-offered education subsidies, followed by Village Savings and Loans (VSLAs) that contain education fund allotments, were the most common paths HHs traveled, while SOCY’s enrollment supports, combined with home visit confirmations of school attendance, was its most common path. Using QCA, the ET also found various combinations of direct services across BNs were associated with HH graduation. The graphic below summaries these trends: xlix Figure 3 BOCY Core Service Pathways to Graduation Success BOCY Core Service Pathways to Graduation Success Since 2019? HIV Test HIV Suppress HIV Prevent Nutr ition HES CP/P SS School Atten % on Path % on Path meet BN Core Service Package 1 Prior + + + + ~CON TEXT 33.2% 56.4% Core Service Package 2 ~CONT EXT + + 29.0% 56.1% Core Service Package 3 + + + ~CONT EXT + + 30.1% 55.8% % of total HHs Covered by Solutions 63.0% % Overall Success of Above Paths 51.6% The most common core service path to HH graduation for BOCY occurred in HHs enrolled prior to 2019. During this time, a combination of HIV prevention, HH economic strengthening, nutrition support, and CP/PSS services was successful more than half of the time. Also, in HIV negative HHs, HH economic strengthening (such as through VSLAs) and CP/PSS services were often sufficient to meet graduation (the other services were not needed). Otherwise, a less frequented path to graduation occurred in the context of a lack of HIV prevention services, whereby combining the HIV suppression and awareness services with psychosocial, child protection, and nutrition support services was slightly successful 55.8% of the time. These three pathways represent about two thirds of the HHs assessed for graduation, and, combined, witnessed a 52% success rate. The most common core service path to HH graduation for SOCY actually unfolded when some of its core services were lacking, especially when it did not offer nutrition-specific focused services to HHs. The graphic below summaries these trends: Figure 4 SOCY Core Service Pathways to Graduation Success SOCY Core Service Pathways to Graduation Success HIV Test HIV Suppres s HIV Preven t Nutritio n HE S PSS /CP Scho ol Atten d % on Path % on Path Graduate l Core Service Package 1 + + + ~CON TEXT + 29.0 % 60.9% Core Service Package 2 + + + ~C ON TE XT + + 9.5% 63.7% Core Service Package 3 + + + + ~CO NTE XT 7.9% 62.1% % of total HHs Covered by Solutions 48.5% % Overall Success of Above Paths 61.8% In the no nutrition-support contexts, SOCY’s combination of the full gamut of HIV services (testing support, suppression support and prevention training) with HH economic strengthening was sufficient to help HHs achieve graduation 61% of the time. Otherwise, a less frequented path (9.5% of graduating HHs) to graduation occurred in the context of a lack of HH economic strengthening services, whereby combining all three HIV core services packages with psychosocial, child protection and school attendance services was successful 63.7% of the time. Finally, in context where school attendance services were lacking, 7.9% of HHs still managed to graduate when they received only psychosocial, child protection and nutrition services combined with a slimmer set of HIV core packages focused on testing and suppression (but not prevention). The above pathways represent about half of the HHs assessed for graduation, and, combined, tended to witness around a 62% success rate. These suggestive findings hold strong implications for further tailoring of SOCY and BOCY services into essential packages of effective support combinations. The ET will unpack the causal mechanisms underpinning these packages in the Year 3 Final Report. PE Question 3 & 4 Do HHs or individuals with specific characteristics (e.g., individuals with disabilities, single￾parent HHs, regional variation, etc.) progress and graduate at different rates, and if so, why? PE Question 4: To what extent do females and males experience different benefits from the program? Are there unintended results for either group? The ET used statistical techniques to identify different levels of graduation across groups and genders. We compared the overall graduation rates of predominantly female HHs (either in terms of HH head or in terms of HH composition) with the average rates of graduation across BOCY. We found that HHs with more than five females did slightly better than the norm (and better than those with more than five males). However, the ET also found that adult female headed HHs graduated roughly 5% less than the norm. Moreover, HHs with disabled members had the highest rate of graduation (more than 10% above the norm), while single-parent HHs graduated around li 25% less than the norm. HHs in Arua and Tororo Districts by far had the highest graduation rates; while those in Agago and Apac districts faced the lowest graduation rates. We conducted the same analysis for SOCY HHs. In somewhat of a contrast to BOCY, SOCY HHs that graduated below the norm include those with more than five females graduated (almost none of these HHs graduated), as well as single-parent HHs (roughly half of the norm graduation rate) and those with disabilities (less than 10% of these HHs graduated). However, these differences are not significant. HHs in Mitooma and Rukiga Districts graduated well above the norm, while no HHs in Bushenyi and Rubanda Districts witnessed graduation, and these differences are significant. Based upon these findings, BOCY HHs that are predominately female and those with disabilities fared better than the norm and those in SOCY, even as female headed and single-parent HHs also fared better than those in SOCY, but below the overall BOCY norm. The ET also used data-driven statistical techniques (latent class analysis) to identify other types of groups. In BOCY, The ET also found three distinct “needs-based” groups in both SOCY and BOCY. First, “Overall Low Need” HHs had higher odds to graduate earlier than more vulnerable or specialized needs classes. Specifically, “Food Insecure” HHs were 0.67 and 0.8 times less likely to graduate than HHs in “Overall Low Need” and “Income Insecure Need” HHs, respectively. In SOCY, the “Overall Least Insecure” HHs were 1.4 times more likely to graduate compared to “Overall Most Insecure” HHs. Further, this latter group was 0.6 times less likely to graduate in contrast to the middle “Formal Job Insecurity” group of HHs. Notably, the SOCY findings were statistically significant, even as the BOCY findings were not. This is most likely due to the content of each IP’s vulnerability assessment tools, in which BOCY’s Household Vulnerability Prioritization Tool (HVPT) captured fewer aspects of vulnerability than SOCY’s revised Household Assessment Tool (HAT). While the above gender, socio-economic demographic, ability and geographic group differences are significant and may guide more tailored programming, the ET will confirm these findings through qualitative investigation in the next report. Impact Evaluation Question Does participation in SOCY activities contribute to a reduction in HH vulnerability and improve OVC well-being? Overall, the SOCY Impact Evaluation found that SOCY has not made a significant reduction to HH vulnerability (as measured by the HVAT) relative to its comparator control group from 2016- 2019. However, both groups experienced initial reductions to vulnerability from 2017-2018, although this trend reversed and both groups witnessed overall increases to vulnerability from 2018-2019. The ET also identified potential “contamination” of both control and treatment groups in which they may have received additional services from non-OVC program service providers which may distort the results. In fact, the ET found SOCY-treatment HHs received a higher percentage and intensity of additional services than control HHs, although contamination is unlikely to explain the lii larger insignificant change in HH vulnerability observed in the two groups over time. While there may have been unobserved influences on the SOCY treatment group as the study progressed, the lack of significant differences from the control group may have more to do with SOCY’s shifting programmatic emphasis away from broader dimensions of OVC vulnerability and towards meeting the 95-95-95 goals. Recommendations Given the travel restrictions in place due to the COVID-19 outbreak, the following recommendations should receive further attention after subsequent qualitative investigation and discussion has occurred: ● Moving forward, the next iteration of large scale OVC support would do well to invest resources in capacity-building of local staff to have more information management skills, especially programming, to better navigate shifting graduation requirements. Specifically, stronger “big data” management skills such as Python could be cost-effective solutions to mitigate these challenges. ● Complexity management tools such as QCA can be used to produce rigorous evidence on which specific services and combinations of services, are most likely to achieve success on various benchmarks and graduation from OVC program support. USAID should consider embedding a similar approach within the adaptive management toolkit of similar follow￾on activities. ● BOCY’s layering report provides useful summary information that the OVC Program could potentially apply to any follow-on implementation activities.(Annex 5 for further database technical recommendations). ● SOCY and BOCY should consider reviewing these findings and investigating whether their services favor the graduation of smaller HH sizes and those HHs containing female majorities. ● SOCY may wish to consider tailoring formal job opportunity services, such as more skills building, to the “ Formal Job Insecurity” and “Overall Most Insecure” groups, while further emphasizing the utility of financial savings groups to its “Overall Least Insecure” HH groups. ● USAID should qualitatively investigate the various subgroups identified within SOCY and BOCY beneficiary data better identify how HHs experience these various vulnerability profiles in different manners. ● No definitive judgement on the Impact Evaluation results should rendered until the ET further investigates the suggested findings qualitatively on the ground. liii