1 SUSTAINABLE OUTCOMES FOR YOUTH AND CHILDREN EXTENDED BASELINE REPORT June 25, 2018 This publication was produced for review by the United States Agency for International Development. It was submitted by the USAID/Uganda Monitoring, Evaluation and Learning Program, AID-617-C-13-00007, prepared by SoCha, LLC under subcontract to The QED Group, LLC. SUSTAINABLE OUTCOMES FOR YOUTH AND CHILDREN EXTENDED BASELINE REPORT Submitted to: USAID/Uganda Prepared by: SoCha, LLC, under subcontract to The QED Group, LLC Disclaimer: The authors’ views expressed in this document do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Acknowledgements SoCha, LLC would like to thank USAID/Uganda, The QED Group, LLC, and CRS Uganda for the generous time and support they offered during the implementation of the extended baseline survey. The SoCha team would like to express our sincere gratitude to the USAID/Uganda staff and the Learning Contract team for the support they have shown – in particular: Samuel Hargadine, Anant Singh, Zachary Dunn, and Gorretti Kiiza Mbabazi. In addition, we are grateful to Katharina Erxleben, Christine Muwanga, Phil Greene, and Douglas Blanton for endorsing the use of QCA as a rigorous impact assessment tool and inviting SoCha to participate. We also are grateful to the QEDGroup team – particularly Johan Kleinhans, Denis Owor, Augustine Wandera and Ruth Ankunda for their support throughout Phase 2 of our visit. Staff at CRS– especially Sasha Angelevski, Olive D’Mello, Fred Werikhe, Benjamin Aisya, Tadeo Byemukama and– generously offered their time straight away, and have since been crucial to the development of our understanding of Sustainable Outcomes. Finally, we’d also like to thank Sharon Kim and Mahjabeen Raza from New York University and Wolfgang Stuppert for their technical inputs and guidance on factor analysis and high-volume QCA. Acronyms ACODEV Action for Community Development AMELP Activity Monitoring, Evaluation and Learning Plan ANPPCAN African Network for Prevention & Protection Against Child Abuse & Neglect CAO Chief Administration Officers CDO Community Development Officers CRS Catholic Relief Services CSO Civil Society Organizations DCDO District Community Development Officers DOVVC District OVC Coordination Committees ECD Early Childhood Development HIV Human Immunodeficiency Virus HVAT Household Vulnerability Assessment Tool HVPT Household Orphans and Vulnerable Children Vulnerability Prioritization Tool IPs Regional Implementing Partners MEEPP Monitoring and Evaluation of Emergency Plan Progress MGLSD Ministry of Gender Labor and Social Development OLS Ordinary Least Squares OVC Orphans and Vulnerable Children OVCMIS OVC Management Information Science PEPFAR President's Emergency Plan for AIDS Relief PSWs Para Social Workers QCA Qualitative Comparative Analysis SAGE Social Assistance Grants for Empowerment SCORE Sustainable Comprehensive Responses for OVC and their Families SDS Strengthening Decentralized Systems SILCs Savings and Internal Lending Communities SOCY Sustainable Outcomes for Children and Youth (aka Sustainable Outcomes) SOVCCs Subcounty OVC Coordination Teams SRMP Sub Recipient Management Policy TPO Transcultural Psychosocial Organization UNICEF The United Nations Children's Fund USAID United States Agency for International Development VHTs Village Health Teams VI Vulnerability Index Table of Contents Acknowledgements............................................................................................................................................................. ii Acronyms............................................................................................................................................................................ iii Table of Contents.............................................................................................................................................................. iv Executive Summary........................................................................................................................................................... vi Background .................................................................................................................................................................... vi Goals and Design.......................................................................................................................................................... vi Extended Baseline Implementation .......................................................................................................................... vii Findings.......................................................................................................................................................................... vii HVAT Distribution and Refinement ..................................................................................................................viii Using QCA to Identify Service Pathways Out of Vulnerability ............................................................................ ix Conclusions and Recommendations........................................................................................................................... x Introduction.......................................................................................................................................................................12 Evaluation Purpose Aims and Objectives................................................................................................................14 Research Strategy..........................................................................................................................................................14 Research Design: Impact Evaluation ........................................................................................................................15 Research Design: QCA................................................................................................................................................16 Outcome: Defining Vulnerability ..............................................................................................................................17 Survey Implementation ...............................................................................................................................................18 Issues Encountered......................................................................................................................................................20 Extended Baseline Survey Results..................................................................................................................................23 Survey Data Distribution ............................................................................................................................................23 Means Testing of PEPFAR Indicators.................................................................................................................24 Vulnerability Assessment and HVAT Scoring ........................................................................................................27 HVAT Means Testing.............................................................................................................................................28 HVAT Distribution.................................................................................................................................................29 Changes to the HVAT Distribution.....................................................................................................................30 Using QCA to Identify Service Pathways Out of Vulnerability ...........................................................................37 Results of the QCA.................................................................................................................................................38 Conclusions and Recommendations..............................................................................................................................47 Annex 1: Scope of Work .................................................................................................................................................48 Annex 1a: Changes to the Scope of Work....................................................................................................................56 Annex 2: Survey Instrument ...........................................................................................................................................59 Annex 3: Fully-Refined HVAT Model Structure.........................................................................................................83 Annex 4: Cluster analysis.................................................................................................................................................84 Annex 5: Frequencies of specified conditions .............................................................................................................86 Annex 6: Various Datasets in Excel and SPSS.............................................................................................................88 Annex 7: Correlational analysis.......................................................................................................................................89 Annex 8: Specifications of configurational analyses for the absence of vulnerability changes............................96 Annex 9: Full Display of QCA Solutions .....................................................................................................................97 vi Executive Summary Background USAID/Uganda’s Orphans and Vulnerable Children (OVC) programming employs an integrated approach to reducing OVC and Youth vulnerability in Western, Southwestern and Central Uganda. Specifically, Sustainable Outcomes aims to achieve the following three Results: • Result 1: Orphaned and vulnerable children, youth and their caregivers are better economically empowered to access core services; • Result 2: Local government and CSOs and informal community structures increase and improve core services for orphaned and vulnerable children, youth and their caregivers; • Result 3: Improved coordination of community-based clinical and socio-economic services for efficiency and effectiveness along the continuum of care. These three results are designed to be mutually supporting, in which SILC households not only benefit from the expected increase in knowledge, access, and support, but also experience improved access to quality OVC services across an improved body of coordinated service providers operating at local (CSO and informal), social service worker, local government and other service provider (other programs and NGOs) levels. The report is focused on the second survey wave of the same households conducted in November and December 2017. The initial baseline was first conducted in November and December 2016, but the planned baseline households were not enrolled in the Sustainable Outcomes activity. As such, what was originally referred to as the “midline survey” is now referred to as the “extended baseline survey.” Changes to the Sustainable Outcomes implementation and the implications for this study’s scope of work are discussed in Annex 1a (Changes to the Scope of Work). Goals and Design The purpose of this extended baseline is to help USAID/Uganda answer overarching questions regarding the effectiveness of Orphans and Vulnerable Children (OVC) programs; understand how different interventions or combinations of interventions of the program contribute to program outcomes; and generate evidence to inform decisions about future OVC programs. The aim is to assess and understand the impact of participating in programs that seek to improve household socioeconomic security through savings and strengthening institutional childcare service delivery on OVC caregivers, households and children over time. The primary objectives of this extended baseline are to lay the foundation to answer two complementary and inter-related questions: • Has caregiver participation in Sustainable Outcomes reduced household vulnerability and improved OVC well-being? • What combinations of factors (Including those within and outside of Sustainable Outcomes’ control) best identify how this may or may not have happened? Whereas a quasi-experimental evaluation design will confirm whether or not Sustainable Outcomes has worked, a qualitative comparative analysis (QCA) approach will identify how it has worked. QCA is a data analysis technique for identifying which combinations of factors best explain how an outcome was (or was not) achieved. This combined QCA/quasi-experimental design approach presents a unique opportunity to test Sustainable Outcomes’ underlying assumption that caregiver participation in a SILC group, combined with additional support services and an improved referral system, improves household economic status which in turn leads to improvements in child well-being. vii Extended Baseline Implementation The survey team conducted household surveys for 2,899 households across 94 parishes in 15 districts across the three regions. A copy of the full survey tool is found in Annex 2, and the dataset can be found in the dataset attachment Annex 6. The control group included 838 households, Treatment group 1 (R3 group) included 978 households and the Treatment group 2 (R1R3) included 1083 households. Combined, the two treatment groups yield 2,061 households. The distribution of households across regions is the following: HH Group Distribution by Region Region Control Group R3 group (Treatment 1) R1R3 group (Treatment 2) Combined Treatment Group Total Central 781 572 0 572 1353 Western 0 324 15 339 339 Southwest 57 82 1068 1150 1207 Total 838 978 1083 2061 2899 Household sampling selection was aligned to Sustainable Outcomes’ enrollment target areas across cohort’s 2 and 3, and used a two-stage cluster sampling approach to select households. In the first stage, 31 parishes were selected from each treatment and control group’s sampling frame. In the second stage, 27 to 30 households in the control group and R3 treatment were selected in each Parish using the “random route” technique. The baseline survey was designed to collect sampling information at three levels of analysis: Household, OVC (0-9), and Youth (10-17). It collected information across three dimensions: Vulnerability, OVC/Youth Status and Situational Analysis. Each dimension uses a questionnaire module that was already been tested and IRB approved in Uganda. Data was collected using handheld tablets (with power banks) using Survey CTO. Survey instruments and consent forms were translated into Luganda, Runyankole, and Runyoro/Rutoro. Consent forms were created for primary care givers, emancipated youth respondents (age 10-17), and child assent forms (signed by the primary care giver). Findings We ran statistical comparisons across the control and combined treatment groups to test for significant differences. For the secondary outcomes, thirteen had no significant differences across the two groups, and details are presented in the main body of this report. For the remaining 11 indicators in which there were significant differences – i.e. the control and treatment groups appear to be at different starting points – the reasons appear to mostly involve economic geography. The ability to meet unexpected and ongoing expenses, such as in health, food and education, stand out as significantly differences across groups. Cultural differences across geographies regarding OVC discipline may also contribute significant differences in caregiver views regarding the appropriateness of harsh physical punishment at home or in school. Recent fever, child attendance in school, and caregiver knowledge of OVC HIV status are the other significant differences. For the HVAT, we found significant difference in the mean average vulnerability score and CPA ranking on all components save Health/WASH and Psychosocial across both groups, again potentially reflecting the similar economic geography and cultural differences speculated above. viii HVAT Distribution and Refinement Turning to the distribution of vulnerability, we find the following distribution using the HVAT: Distribution of Vulnerability across Control and Treatment Groups HVAT Category Control Treatment Total Not Vulnerable 171 (20.4%) 285 (13.8%) 456 (15.7%) Slightly Vulnerable 617 (73.6%) 1607 (78.0%) 2224 (76.7%) Moderately Vulnerable 50 (6.0%) 169 (8.2%) 219 (7.6%) Critically Vulnerable 0 0 0 Total 838 2061 2899 Overall, we find a high concentration of “slightly vulnerable” across the three groups, 76.7% of the sample (the previous survey was similar at 73% of the total sample), followed by a smaller concentration of “not vulnerable” households, and the smallest percentage of “moderately vulnerable” households, and a complete lack of “critically vulnerable” households across the entire sample. Overall, the HVAT scores suggest that Sustainable Outcomes is supporting a higher number of “less vulnerable” households than those that score as “more vulnerable,” and that many “not vulnerable” households should potentially not be supported at all in favor of enrolling more vulnerable HH. This raises the question: Has Sustainable Outcomes been supporting HH in regions of Uganda that don’t need assistance? Or is the HVAT tool used to measure vulnerability mis-calibrated and should be refined to better capture the dynamics of vulnerability?” In the previous baselines survey report, we explored these questions extensively using a number of tools. Overall, we found that Sustainable Outcomes was being implemented in appropriate areas in need of assistance and that the survey’s sampling approach was valid. Rather, it was the design of the HVAT tool that suffered from some challenges that impacted the resulting vulnerability scores and status distribution; specifically, the weighting scale assigned to the questionnaire scheme was mis-calibrated. For this wave of the survey, we used factor analysis to generate index scores (as opposed to simple aggregation) and a “weighting scheme” that can be applied to each question’s score. In doing so, we conducted exploratory and then confirmatory factor analyses to recalibrate the HVAT’s original five “Core Programme Areas” into a new “four factor” structure. This new model represents an optimal fit of the data in terms of its parameters of fit, and the results of our measurement invariance testing reveal that this new, refined HVAT structure can be consistently applied across the various groups in the study. ix With the refined HVAT weighting scores and relationships both identified and validated by using factor analysis, we then generated new, refined individual HH vulnerability scores.1 These results are presented below in conjunction with the original HVAT raw scores to illustrate the differences of using Factor Analysis to appropriately weight each item on the questionnaire and the effects on the sampled vulnerability distribution: Change in Vulnerability Distribution from Original to Refined HVAT Vulnerability Category Refined Score Distribution and Percentages Original Score Distribution and Percentages Change from Original to Refined HVAT Critically Vulnerable 109 (3.76%) 0 109 were Added Moderately Vulnerable 1104 (38.08%) 219 (7.6%) 885 were Added Slightly Vulnerable 1488 (51.33%) 2224 (76.7%) 736 were Removed Not Vulnerable 198 (6.83%) 456 (15.7%) 258 were Removed Total 2,899 2,899 As can be seen, moving away from the raw aggregated vulnerability scores to the more refined HVAT approach that incorporates score weightings generated by factor analysis yields significantly different results. Specifically, the “critically vulnerable” category grew from zero households to 109, while the “not vulnerable” category reduced from 456 households to 198; thus suggesting that original HVAT significantly under represented overall vulnerability.2 As such, the refined results yield a more accurate recalibration of the sampled HVAT vulnerability distribution and weight the value of each response more appropriately. Using QCA to Identify Service Pathways Out of Vulnerability Whereas the DID design will confirm whether or not Sustainable Outcomes has worked, this study uses QCA to identify how it has worked. The two baseline waves of data collection represent a unique opportunity to analyze which combinations of factors outside of the program’s control drive changes in household vulnerability. To do this, QCA was used to identify how different combinations of social services might help reduce (or increase) household vulnerability to identify which combinations of factors best explain how an outcome was (or was not) achieved.3 By modelling the changes in vulnerability outcomes from 2016 to 2017, we found that 375 households reduced their vulnerability without any assistance from Sustainable Outcomes. QCA identified how these households travelled across three types of pathways out of vulnerability:4 • Path 1 shows that in 17.2% of the households that reduced their vulnerability from 2016 to 2017, they had more than three children, were members of savings groups, gained access to direct cash transfers recently, and had not lost access to HIV and Gender-Based Violence prevention since Wave 1 (2016). • Path 2, in contrast to Path 1, was traveled by 25.7% of successful households and refers to those who house three children/youth or less. Path 2 shows that among these households, NOT losing access to 1 To do this, we applied the regression method to the cluster of items found under each factor. This created four sets of weighted factor scores, one for each revised CPA, which were then simply aggregated to yield sum scores for factor scores. 2 One of the reasons why the original HVAT scores may be so misaligned with the weighted scores is connected to a potential “positive social response” bias around questions involving abuse and socio-psycho well-being. Respondents may be more reluctant to share this type of information with enumerators and inflate their responses towards less vulnerable categories. Factor Analysis helps correct for this by drawing upon the underlying structure of the data to better tease out the relationships among questions across the sample and to some extent bypass response bias. 3 QCA was used to draw from a comprehensive list of surveyed factors that characterize the households, their context (including services provided by other programs), and the program activities they participate in. Similar to the way we build “trend lines” to facilitate a Difference-in-Difference comparison of control and treatment groups in the impact evaluation design, here we build “service lines” through QCA to identify various service trajectories already underway in the implementation areas. 4 Eighty of the total 375 households may have travelled more than one pathway out of vulnerability. x key services - specifically not losing access to HIV and Gender Based Violence prevention services - is sufficient to achieve at least slight reductions in vulnerability. One peculiar aspect of this solution is how NOT being a member of a savings group is also highlighted as an essential part of the pathway to improvement (which is different than saying that membership in a savings group doesn’t matter either way; here, it states, NON-membership appears to matter). Further investigation into why households who withdraw from a savings group tend to reduce their vulnerability is warranted. • Path 3 expresses the strongest trend towards reducing vulnerability, 57.1% of all successful households. For these households, retaining access to HIV and GBV prevention services (including community testing of HIV) remains important, but the unique aspect of this pathway is the reliance on sustained access to food support. This pathway is indifferent to membership in a savings group, or lack thereof. Finally, more than 20% of HH in the survey experienced no change in their vulnerability from 2016 to 2017. QCA found that the common trend among these households was their lack of access to services to begin with (as of Wave 1) and that this lack of access continued through 2017. Such a finding reinforces the Sustainable Outcomes Theory of Change around the importance of access to and continued receipt of various services. Subsequent waves of the impact evaluation will help determine if the types of services offered by Sustainable Outcomes have a positive effect. Conclusions and Recommendations As can be seen, the second wave of the extended baseline survey generated some interesting findings and has laid the groundwork for a new, more advanced approach to determining impact by combining both indirect methods of attribution (through a Difference-in-Difference quasi-experimental design) and direct methods of contribution (through an expanded application of Qualitative Comparative Analysis to much larger datasets). Along the way, the analysis has greatly improved the validity and accuracy of the HVAT tool, yielding a much more appropriate weighting scale of HVAT items and useful vulnerability categorization system. As such, the following recommendations are offered by the SoCha team for USAID/Uganda and Sustainable Outcomes staff to consider taking forward: Recommendation: Sustainable Outcomes should strongly consider enrolling the surveyed treatment households to receive treatment so that the impact evaluation can be conducted, and its learning questions addressed. Furthermore, the surveyed control group households should also be enrolled at a later date so as to avoid ethical issues and not harm the implementer’s legitimacy and reputation. Recommendation: Sustainable Outcomes should increase their quality control of the SOCY MIS implementation database, examine and revise their current MIS implementation protocols, provide new and more comprehensive user training, regularly monitor and clean the MIS as appropriate, and address the technical and design issues identified above, especially the lack of a unique HH identifier for each household. Recommendation: Review the Control/Treatment group allocations in lieu of the recent change to implementation strategy and determine if reallocation of HH to regional boundaries would add value to the study’s learning questions. Recommendation: Sustainable Outcomes and other HVAT users consider a workshop to discuss a potential revised HVAT using these results as the basis for a more refined HVAT applied across the country. Recommendation: Sustainable Outcomes and USAID/Uganda staff should consider conducting additional formative and exploratory analyses on some of key mediators channeling vulnerability in different ways. This type of analyses should be results driven in that the findings can be directly used to inform more targeting OVC programming taken to scale. xi Recommendation: Sustainable Outcomes should consider encouraging greater participation in a savings group (hopefully SILC) combined with encouraging additional access to Cash Transfers and ongoing access to HIV and GBV prevention services as one viable pathway out of vulnerability for moderately vulnerable households that house four or more youth/children. Recommendation: Bearing in mind the findings of the QCA analysis, Sustainable Outcomes may also consider whether or not membership in savings group by HHs with less than four youth/children may entail financial or other types of burdens that overcome the benefits of savings group membership; thus raising the possibility that withdrawal from savings groups by these types of HH may reduce their vulnerability status. Recommendation: Sustainable Outcomes may wish to consider how ongoing access to HIV and GBV prevention services, combined with regular HIV community-testing services and, perhaps most importantly, with ongoing access to steady food support may form a viable pathway out of vulnerability, even without joining a savings group (this proposition could of course also be rigorously tested by reinstating the original dual treatment group design identified in 2016). Recommendation: Sustainable Outcomes should strongly consider enrolling the households surveyed across both waves into the program for services so that a rigorous quasi-experimental design impact evaluation can still be completed and a rigorous QCA can further unpack what combinations of services can lead to pathways out of vulnerability. 12 Introduction Uganda is one of the fastest growing nations in Africa. However, deep and extensive vulnerabilities exist that especially affect children: the majority of children are considered vulnerable, and 62% of those living in poverty are children. Vulnerabilities take different forms, including poverty, malnutrition, and exposure to violence. Low levels of education and high prevalence of HIV/AIDS among children themselves as well as within their families exacerbate these vulnerabilities. USAID/Uganda’s Orphans and Vulnerable Children (OVC) programming employs an integrated approach to address these factors. The Sustainable Outcomes for Children and Youth (Sustainable Outcomes) activity aims to economically empower children, youth, and their caregivers to access core services, strengthen systems to provide core services, and improve coordination of community-based clinical and socio-economic services for efficiency and effectiveness along the continuum of care. The activity is a five-year cooperative agreement implemented by a consortium led by Catholic Relief Services (CRS) across 17 districts in Central and Western Uganda. These districts were identified by the United States Agency for International Development (USAID) due to high prevalence rates of HIV and availability of trained community workers to deliver program services. Sustainable Outcomes’ goal is to help a minimum of 625,000 OVC, youth and 101,500 households to access core services for improving health, nutrition, education, and psychosocial well-being to reducing abuse, exploitation and neglect. Sustainable Outcomes deploys an integrated approach to reduce OVC and Youth vulnerability that operates across multiple levels. Specifically, Sustainable Outcomes aims to achieve the following three Results: • Result 1: Orphaned and vulnerable children, youth and their caregivers are better economically empowered to access core services; • Result 2: Local government and CSOs and informal community structures increase and improve core services for orphaned and vulnerable children, youth and their caregivers; • Result 3: Improved coordination of community-based clinical and socio-economic services for efficiency and effectiveness along the continuum of care. Each Result corresponds to a different level of implementation, beneficiary and support package. For Result 1, OVCs and their care givers at the household level are the main beneficiaries of support, primarily through the formation of savings and internal lending communities (SILCs).5 SILCs are hypothesized to increase household income and savings, which will enable the household and OVC caregiver to better provide for OVCs as well as better access core OVC services outside of the home.6 Activities include increased access to temporary consumption support (by linking them to external resources such as SAGE, Give Directly, and WFP, as well as to groups within the community); enroll households in SILCs (in which 15-30 households pool savings on a weekly basis to create social funds and access loans; the returns from which are paid out to members after 12 months); train SILC members on a variety of topics, such as financial management, HIV 5 For a description of SILCs, see Vanmeenen G. (2010). Savings and Internal Lending Communities (SILC) Voices from Africa: The benefits of integrating SILC into development programming. Baltimore, MD: Catholic Relief Services. Drawing upon various descriptions in SILC evaluations, here SILCs can be summarized as model developed by CRS for user-owned, self-managed savings and credit groups that offer households a way to protect assets, smooth cash flow, and increase income. Community members are encouraged to self-select into groups of 15-30 people they know and trust. Compared to traditional accumulating savings and credit associations, SILC groups are more accessible, transparent, and flexible. Each group determines its meeting schedule, regular contributions to a loan fund, loan duration, interest, and maximum loan amount. Members can borrow from the fund at this predetermined interest rate and term. Interest and fines allow the common fund to grow. At the end of a predetermined time period, all or part of the common fund returns to the group members based on the total amount saved by each member. SILC groups are initially supported by project “field agents,” but the goal is institutional and financial independence (note: this was taken from the SILC Impact Evaluation of the STEPS OVC Project in Zambia Final Evaluation Report, found at www.silcevaluation.com). 6 The key assumption in this argument is that if SILCs do in fact increase household income and savings for members who are caregivers of OVCs, some of the benefits will also reach the OVC, e.g. caregivers will use these resources to benefit the child and not use them for something else. 13 prevention and treatment, skills building, and better parenting/caregiving; link girls to access to better education opportunities; and provide additional technical assistance, market access, and training to farmer-focused SILCs to improve income. For Result 2, district government and CSOs are the main beneficiaries and here sustainable outcomes work with relevant district staff to create functioning OVC coordination mechanism, allocate resources for OVCs and youth, fill key OVC-support positions, conduct performance appraisals of relevant staff, use the OVCMIS for planning, create alternative care panels and transition OVCs from institutional to family based alternative care. The program will also build the capacity of CSOs to comply with national OVC and youth standards, conduct protection pilots and implement improved OVC service delivery. Finally, for Result 3, the primary beneficiary is the referral system, in which the Para-social workers (PSWs) and village health teams (VHTs) sit as the main actors who refer youth and OVCs to the appropriate provider in the service network. Here, activities include a mapping of district level services; training PSWs and VHTs on how to make referrals to multiple types of services, track those referrals, and assist vulnerable households with the development of household action plans; and conducting pilot projects on an Early Childhood Development (ECD) tracking component, ECD development spaces at health centers, and creating very young adolescent groups to discuss sexual reproductive health issues. Figure 1: Sustainable Outcomes Graphic Representation These three results are designed to be mutually supporting, in which SILC households not only benefit from the expected increase in knowledge, access, and support, but also experience improved access to quality OVC and services across an improved body of coordinated service providers operating at local (CSO and informal), social service worker, local government and other service provider (other programs and NGOs) levels. 14 Evaluation Purpose Aims and Objectives The purpose of this evaluation is to help USAID/Uganda answer overarching questions regarding the effectiveness of Orphans and Vulnerable Children (OVC) programs; understand how different interventions or combinations of interventions of the program contribute to program outcomes; and generate evidence to inform decisions about future OVC programs. The aim of this evaluation is to assess and understand the impact of participating in programs that seek to improve household socioeconomic security through savings and strengthen institutional childcare service delivery on OVC caregivers, households and children over time. The primary objectives of the evaluation are to answer two complementary and inter-related questions: • Has caregiver participation in Sustainable Outcomes reduced household vulnerability and improved OVC well-being? • What combinations of factors (including those within and outside of Sustainable Outcomes’ control) best identify how this may or may not have happened? Answering the first question suggests a counterfactual research design that compares changes in vulnerability status across various control and treatment groups to demonstrate “what would have been the change if households had NOT participated in Sustainable Outcomes.” This is commonly referred to as an impact evaluation. Answering the second question suggests a comparative research design that combines the diversity of implementation with data on relevant external factors into an analytical model that identifies how various pathways to the outcome can be achieved. Qualitative Comparative Analysis is the method commonly associated with this approach. In what follows below, we review the overall design strategy, the type of quasi￾experimental design that has been used, the approach to QCA, and finish with a report on vulnerability results. Research Strategy This impact evaluation deploys a combined quasi-experimental design and qualitative comparative analysis strategy (QCA). Combining a QCA-approach with an impact evaluation presents a unique opportunity to test Sustainable Outcomes’ underlying assumption that caregiver participation in a SILC group, combined with additional support services and an improved referral system, improves household economic status which in turn leads to improvements in child well-being. The combination of this approach yields a design strategy oftentimes referred to as “lumping and splitting”. For the sake of policy decision making, it is oftentimes necessary to “lump” a wide array of outcomes and pathways into two groups (control and treatment) represented with summary statistics based upon measures of central tendency (e.g. the mean, the regression line, etc.). This allows us to make clear, testable statements around program attribution and causal inference (aka internal validity). While useful, lumping seldom includes information on how the multitude of household pathways converged on these results across contexts, and so we are left with little information on how to reproduce the same results elsewhere. It is therefore necessary to “split” the control and treatment groups up into more refined subgroups to identify how local context matters and how various aspects of a program combine and interact in different ways. This then allows us to better understand how various pathways to the outcome occurred and can occur elsewhere (i.e. external validity). While also useful, splitting does run the risk of identifying pathways that may have been the result of random chance, and therefore our ability to attribute the outcome to these explanations is reduced. As such, combining the strengths of both approaches compensates for their respective weaknesses. This evaluation relies upon a longitudinal, quasi-experimental and QCA design using a multi-stage cluster sampling approach in which a representative group of households that participate in Sustainable Outcomes will be compared with households that have not yet participated (but will be eligible in the future). These groups have been selected from Cohorts 2 and 3, respectively. The difference between these groups will contribute to the estimates of Sustainable Outcomes’ effects. The evaluation also incorporates a wide array of implementation and environmental data. Participants and households in both groups have taken part in an annual interviewer-administered survey for 2016 and 2017, with additional waves of data collection anticipated. Looking forward, systematic follow up during and after Sustainable Outcomes implementation allows for 15 retrospective analysis of dose-response, sub-group analysis, and evaluation of the sustainability of both SILC and outcomes of participation. Research Design: Impact Evaluation This extended baseline is designed to support a difference-in-difference (DID) impact evaluation design. DID designs compare two types of changes: The changes in outcome of a treatment group (or groups) before and after an intervention has occurred with the changes in outcome of a control group (or groups) over the same period of time. This design puts in place two controls: The difference in the before-and-after outcomes for the group(s) enrolled in the program controls for factors that are constant over time in that group (as it is comparing the same group to itself over time); Then, this difference is compared with the difference in before-and-after outcomes for a group that was not enrolled in the program but was exposed to the same set of environmental conditions over the same time period to control outside time-varying factors. DID thus combines these two counterfactuals (before-and-after comparisons, and comparisons between those who were enrolled and those who were not enrolled) to produce a better estimate of the counterfactual to identify the impact of the program (see Figure 2 below). Crucially, the DID design does not require that each group have statistically insignificant outcome differences at baseline, i.e. statistically significant differences across groups do not pose threats to the internal validity of the design (we find several statistically differences across groups, which is discussed below). Rather, the key assumption for DID is commonly referred to as the “parallel trends” assumption in which the outcome across treatment and control groups are assumed to follow the same trend over time in absence of the treatment. Even though their outcome scores are different between baseline and extended baseline, their trends of improvement remain constant, and no substantive outcome has occurred in the treatment group because it is following a longer-term trend in the same manner as the control group. The two groups are thus comparable. Conversely, deviations from the parallel trend line by the treatment but not in the control group can be attributed to the impact of the program (if significant). Indeed, figure 2 below demonstrates how the aggregate effective size of the control group outcome can be higher than in the treatment group, and yet the impact can be found in the latter between its actual end line scores and what those scores would have been had the project not occurred.7 Figure 2: Difference in Difference Design in which the Control Group Scores Higher than the Treatment Group, but the Treatment is Effective 7 Originally, this impact evaluation had planned to establish trendline scores using earlier results from the “vulnerability assessment” tool, which was hosted by the Ministry of Gender, Labour, and Social Development (MGLSD) and formed the basis of the revised Household Vulnerability Assessment Tool (HVAT- see below) used for this study. However, conversations with MGLSD staff revealed that this data were not available at lower levels (e.g. below the county-level) and the sample sizes too small to legitimately establish control and treatment groups. Fortunately, changes to Sustainable Outcomes implementation have presented the opportunity to establish a much stronger parallel trend using actual HVAT data from one year to the next, i.e. using the baseline and what was to be midline data to establish the trends. These changes will be discussed in more detail below. 16 Changes to the Sustainable Outcomes implementation and the implications for this study’s scope of work are discussed in Annex 1a (Changes to the Scope of Work). Here, it is sufficient to note that the original impact evaluation design poses three different impact evaluation questions: • Question 1: Does caregiver participation in activities under Results 1 and 3 reduce household vulnerability and improve OVC well-being? • Question 2: Does caregiver participation in activities under Results 1 reduce household vulnerability and improve OVC well-being? • Question 3: Do the combined activities under Results 1, 2 and 3 reduce rates of HIV prevalence and incidence in participating sub-counties? For Question 1, households and OVC/Youth in Cohort 2 sub-counties that receive both R1 and R3 treatments (treatment group 1) were to be compared to households and OVC/Youth in Cohort 3 sub-counties (the control group) who do not receive any treatment until the end of 2017 and 2018. However, for Question 2, Sustainable Outcomes plans to eliminate the distinctions between an R1+R3 group and an R1 only group by offering R3 services to all beneficiaries. The result, to the extent it is fully implemented, eliminates Question 2. Finally, Question 3 remains the same and will rely upon data that compares HIV prevalence and incidence rates at the participating sub-county level. This question will be addressed during the endline. In doing so, the study will estimate the combined effects of R1+R2+R3 by comparing HIV Prevalence and Incidence rates across Sustainable Outcomes sub-counties and various high-risk groups, such as girls age 10-24, within those sub￾counties with rates in non-Sustainable Outcomes sub-counties across the 17 districts. This data will be collected from secondary sources; namely, the Ministry of Health DHMIS2 database, and analyzed in the final phase. Research Design: QCA Whereas the DID design will confirm whether or not Sustainable Outcomes has worked, this study uses QCA to identify how it has worked. QCA is a data analysis technique for identifying which combinations of factors best explain how an outcome was (or was not) achieved. Factors here refer to a wide range of potential variables, e.g. continuous variables, categorical variables, qualitative changes in condition, M&E indicators, program outputs, relevant external variables outside of the program’s control, context variables, etc. These factors are built into a model that identifies how every possible combination (both observed and unobserved) can be associated with the outcome of interest. QCA then applies an algorithm to this model to identify sets of necessary and sufficient combinations of conditions that can be logically associated with explaining how the outcome was achieved. Crucially, QCA typically identifies several combinations of factors simultaneously, which can be used to account for various contexts and explain how there may be multiple pathways to achieving the same outcome. As with the impact evaluation, QCA is able to address these questions at the household, OVC aged 0-9, and youth aged 10-17 and sub county levels. QCA uses the same outcome indicators as the impact evaluation. Yet unlike the impact evaluation, QCA incorporates a much larger number of factors to explain the outcome without requiring an increase in sample size. Whereas the impact evaluation is limited to testing the impact of only a few treatment arms at the Results level, QCA unpacks how the various activities that fall underneath each result combine to achieve success. For example, Result 1 consists of eight different activities. When the HHs are enrolled, QCA will determine if all of these are necessary for success, or if a “leaner and meaner” essential package, or packages, can be identified. Moreover, using additional context variables, such as the presence of other donor-supported OVC programs, QCA identifies under which conditions one essential package is more preferable to another. For this extended baseline, QCA was still used to identify how some HH manage to pull themselves out of vulnerability before being enrolled into Sustainable Outcomes (see below in the results section). 17 Outcome: Defining Vulnerability The primary outcome measure is socioeconomic household vulnerability status. Household vulnerability is defined as the inability to cope with economic and other shocks. Operationally this is captured through the Household Vulnerability Assessment Tool (HVAT), which was administered at baseline, has been re-administered to the same households across all three groups at the end of 2017, and will hopefully continue in 2018. The HVAT measures vulnerability across economic, food security, nutrition, health, WASH, shelter, education, psychosocial support, and child protection dimensions, and yields a composite score corresponding to “slightly”, “moderately”, and “critically” vulnerable categories.8 The HVAT contains question modules directed to the caregiver, children aged 0-9, and youth aged 10-17. The HVAT was chosen because it is the official vulnerability assessment tool currently endorsed by the Ministry of Gender, Labor and Social Development. The HVAT is the primary household vulnerability assessment tool used by the national OVCMIS, to which all OVC and youth-related programs are required to report.9 Finally, the HVAT has been the primary outcome indicator used by Sustainable Outcomes to gauge their overall success, was captured under Cohort 1 and is regularly reported on as per the SOCY Monitoring, Evaluation and Learning Plan. It should be noted that during the initial baseline, a number of challenges emerged around the HVAT’s design, household selection and computation. Specifically, a seemingly low number of more vulnerable households as well as a higher percentage of “not vulnerable households” captured in the baseline was a cause for concern. This raised questions about the household selection approach and about the design of the HVAT tool; both of which were further investigated. In regards to household selection, households that were selected through pure random selection without the assistance of Sustainable Outcome stakeholders had no significant differences from those who were selected from lists generated by the Sustainable Outcomes team. However, two of the HVAT’s “core program areas” (CPAs) – the questions regarding psycho/social and child protection functioning– most likely suffers from a positive social response bias (especially in regards to self-reporting of abuse and mental well-being), which may drive vulnerability scores downward (i.e. households appear less vulnerable than they actually are). Dropping these two CPAs pushes overall vulnerability upward (households are scored as more vulnerable). In regards to the design of the HVAT tool, it was found that the way the tool is scaled has an impact on the resulting vulnerability scores and status. More precisely, a review of the HVAT tool scaling structure reveals the following: • Contradictions in scoring, due in part to inconsistent units of measurement; • Arbitrary spaces in the scoring of each item can bias household vulnerability scores upward (i.e. they are less vulnerable); • Many of the qualitative values found under each item either do not logically follow the structure of the data or do not have a relationship to other elements; • The items on the HVAT are not equal in relevance and therefore should not be scored equally; • The items fall across different underlying dimensions and therefore are not easily aggregated using the current scoring method. 8 It should be noted that the aforementioned Impact Evaluation that focused more narrowly on caregiver participation in SILCs adopted percentage of children aged 2-17 who have gone a whole day and night without eating in the last four weeks as its primary outcome measure, which was an item that comprised the Household Food Insecurity Access Scale (HFIAS) and similar to the HVAT. For this evaluation, we adopt the wider measure to account for the system-strengthening and improved service delivery aspects of Sustainable Outcomes, which include but go beyond the more narrow SILC focus. 9 The OVCMIS database can be found here: http://ovcmis.mglsd.go.ug/home.php?linkvar=Data%20Collection%20Tools&&action=Data%20Collection%20Tools 18 Principle Component Analysis (PCA) was used to test the internal consistency of each item across the HVAT index and its six constituent “Core Program Areas” (CPAs, i.e. economic strengthening, food and nutrition, health and WASH, education, child protection and psycho-social). The results suggest a very poor fit in terms of how well all HVAT items combine to form a uni-dimensional scale. Less than 16% of all variance could be explained on a combined HVAT component, and 9 of the HVAT’s 17 items drop out. These results suggest that the HVAT cannot be easily reduced to a single dimension score, as suggested by the original design. The implications are that HVAT scores can be a misleading proxy for comparing underlying vulnerability across households (for example, two households that hold the same overall vulnerability score but differ in their CPA distribution in fact may differ significantly). In the following sections, we discuss how we addressed this issue using Confirmatory Factor Analysis and some of the suggested changes this analysis reveals. Additionally, twenty-one secondary outcome measures are being captured at the OVC and Youth wellbeing level for children aged 0-9 years (with questions directed to the primary caregiver) and youth aged 10-17 years. These measures are derived from PEPFAR’s MER Essential Survey Indicators for OVC Programs document (2014) and elaborated using the MEASURE Evaluation Guide (2015) for OVC program outcomes. These include: • Percent of children whose primary caregiver knows the child’s HIV status • Percent of children < 5 years of age who are undernourished • Percent of youth too sick to participate in daily activities • Percent of children too sick to participate in daily activities • Percent of youth who have a Birth Certificate • Percent of children who have a Birth Certificate • Percent of youth enrolled in school • Percent of children enrolled in school • Percent of youth regularly attending school • Percent of children regularly attending school • Percent of children <5 years with recent diarrhea • Percent of children <5 years with recent fever • Percent of youth >10 to 17 years reporting irregular food intake • Percent of children >2 to 9 years reporting irregular food intake • Percent of caregivers who feel harsh physical punishment is appropriate means of discipline in the home or school • Percent of households able to access money to pay for unexpected expenses • Percent of households able to access money to pay for health expenses in past 3 months • Percent of households able to access money to pay for school expenses in past 3 months • Percent of households able to access money to pay for food expenses in past 3 months • Percent of children 1-5 years fully immunized • Percent of youth aged 10-17 years reporting basic support Finally, two HIV-specific measures are captured at the sub county level over time. These are HIV prevalence and HIV incidence rates, and can be broken down into specific subgroups, such as girls ages 10-17. The data for this will come from the DHMIS2 database at the endline. Note, power calculations for this evaluation are found in the previous baseline report. Survey Implementation Team Mobilization In November 2017 until middle of December 2017, the survey team was mobilized for implementation. Thirty￾five enumerators and seven field managers were mobilized and attended the one week training and field testing 19 in Kampala and Wakiso. Out of these, twenty-eight of the enumerators and five of the field managers were returning staff from the previous baseline survey, with a ratio of one field supervisor to five enumerators. Participants also received ethics, confidentiality, child protection and rights of the respondent training. Finally, enumerators were again trained on how to measure upper arm circumference (MUAC) on children under five years old using MUAC tape by a health officer from the Mulago National Referral Hospital. Survey Instruments The baseline survey was designed to collect sampling information at three levels of analysis: Household, OVC (0-9), and Youth (10-17). It also collected information across three dimensions: Vulnerability, OVC/Youth Status and Situational Analysis. Each dimension used a questionnaire module that was already been tested and IRB approved in Uganda. They are as follows: • Vulnerability: As mentioned above, we used the HVAT as the primary tool for collecting information on household and youth/child vulnerability. Additionally, we incorporated questions from the Household Orphans and Vulnerable Children Vulnerability Prioritization Tool (HVPT), which contains additional items on household vulnerability. Combined, these tools contain items across six areas of inquiry.10 • OVC/Youth Status: We also included questions to capture the eight OVC/Youth PEPFAR outcome indicators. The tools have already been tested in various African countries and recently MEEPP collected these indicators on a sample of Cohort 1. • Situational Awareness: Finally, we again incorporated a situational awareness module that captures current levels of social assistance provided from external sources. It contains questions about financial service, health facility, community based and government assistance access and use. The current situational awareness tool was applied by Sustainable Outcomes to Cohort 1. We incorporated additional questions regarding specific services provided by other activities and initiatives, which can be identified using the OVC MIS. The survey instrument was administered using handheld tablets (with power banks) using Survey CTO. Survey CTO is a dual offline/online survey platform based upon Open Data Kit (ODK), and is designed for field settings with little or no connectivity. Forms were programmed and designed to synchronize with the main server either in real-time or at the end of each workday when connectivity was limited. Survey instruments and consent forms were translated into Luganda, Runyankole, and Runyoro/Rutoro and programmed into the forms. Consent forms were created for primary care givers, emancipated youth respondents (age 10-17), and child assent forms (signed by the primary care giver). Each respondent was given a hard copy of the consent form and the enumerator kept an additional hard copy that was submitted to the field supervisor at the end of each day and was brought to Kampala at the end of each week. Quality Control For quality control purposes, we put into place several measures. First, field supervisors accompanied enumerators as observers on close to 10 percent of the interviews. Second, field supervisors also revisited households and asked questions on a select number of items for 10% of the households surveyed. Third, the survey manager rechecked and observed field supervisor performance for five households per field supervisor. Fourth, the survey manager performed daily consistency checks on enumerator performance and reviewed enumerators who fell outside of two standard deviations of the norm of responses. We also successfully included a better time stamp system during this wave, which allowed us to also monitor average interview time and flag interviews that fall outside of two standard deviations of the norm. Finally, enumerators were required to record the GPS coordinates of each household according to a 10-meter accuracy rating and retake 10 These are Economic Strengthening, Food Security and Nutrition, Health, WASH and Shelter, Psychosocial Support and Basic Care, Child Protection and Legal Support, Education. 20 coordinates until this was achieved. All interviews thus achieved this rating and close to one fourth achieved an “exceptional” rating of 5-meter accuracy or less. Issues Encountered 1. A Lack of Enrolment of Baseline Households As mentioned earlier, the previously-surveyed households in the original baseline were not enrolled. More details are discussed in Annex 1a (Changes to the Scope of Work). Recommendation: Sustainable Outcomes should strongly consider enrolling the surveyed treatment households to receive treatment so that the impact evaluation can be conducted and its learning questions addressed. Furthermore, the surveyed control group households should also be enrolled at a later date so as to avoid ethical issues and not harm the implementer’s legitimacy and reputation. 2. Correlating SOCY Implementation Data to Vulnerability The second issue involved a lack of implementation data to correlate with the sampled HHs’ vulnerability statuses for the QCA analysis is more complicated. As the method is used to identify which combinations of Sustainable Outcomes activities can be associated with success (or lack thereof), a lack of activity data renders QCA analysis impossible. However, the two waves of baseline survey data do contain additional data points that could be modeled using QCA to identify any “endogenous” changes to HH vulnerability that have occurred independent of the Sustainable Outcomes intervention. In particular, both survey waves captured additional information from each HH regarding what types of additional services they currently receive that are relevant to the SOCY theory of change. Examining the change in volume and composition of HH access to relevant services thus allows us to explore how these services may or may not be associated with varying levels of HH vulnerability. This new analytical scope is further discussed and demonstrated in the QCA section of this report. 3. Related: SOCY MIS-Issues It should be noted that the SoCha team, in cooperation with staff at the Learning Contract (collectively referred hereafter in this section to as the “Team”), also mined the SOCY MIS data to see what could be used for the QCA analysis. SoCha had already examined the initial Cohort 1 dataset, containing 132,052 respondents (see the Baseline Report for more information), and the Team further analyzed the subsequent Pre-Graduation Assessment (PGA) Dataset, which contains more than 84,000 respondent line items. Sustainable Outcomes staff stated that data assigned to the same households found across the initial Cohort 1 dataset, the PGA and the SOCY MIS were linked together by the same HH ID and primary key, but the Team discovered that this turned out to be false and there was no straightforward way to merge the tables across each dataset to provide a comprehensive picture of each HH’s vulnerability scores, household dynamics and services received. After several weeks of data-mining and linking between the MIS and PGA, close to 24,000 HH had been identified in both MIS and PGA datasets, but additional steps would be needed to fully connect all the implementation held within the MIS to their corresponding households. Further linking these HH to the Cohort 1 dataset was beyond this scope of work and will most likely require considerable effort. As already mentioned, it was clear that the MIS suffered from a number of data cleaning and quality control issues (see footnote above), but the Team further uncovered a number of underlying problems: • Many entries are incomplete and household beneficiaries are not fully identified; • The date and time stamp used was not standardized, and analysis based on dates is a challenge (if not impossible), especially when activities took place on days under the 13th of any month; • There are very high levels of mismatch in geographic locations across various zones. For example, some households with coordinates for Subcounty A were in fact logged in other Subcounty B; 21 • There is no geographic hierarchy in the menu options, and thus users can mistakenly select districts when they wish to select parishes; • Confidential test records were found at various places throughout the MIS; • There is significant variation in the identity of the household head within the same household; and • Data collection comments were included as part of the household ID number. Perhaps the most significant data linking and analysis issue is a flaw in HH ID system. There is no way to identify HH geography in these IDs (e.g. no way to identify district, parish, etc.), which in and of itself is not necessarily a problem, but leads to major issues because the same HH ID can be used to reverence multiple HHs across several different locations. The implications are that households in different sub-counties that are serviced even by the same CSO can have the exact same HH ID number. This issue was flagged by the SoCha team in 2016 (e.g. see p. 60 of the 2016 Baseline report), but has not yet been addressed. As a result, many significant issue not only still exist but have compounded. Overall, these data issues suggest a severe lack of data cleaning and management. A lack of oversight is also highly likely, as the volume of user entry errors suggest they not only don’t understand the data entry protocols (assuming there was a training), but also are not being reviewed by supervisors to see if the protocols are being implemented. Ultimately, the explanation for how this many and types of errors could accumulate and compound over time unfortunately suggests that no one with access to the MIS has attempted to conduct any form of quantitative analysis. If such is the case, it begs the question as to why an elaborate (and overall, well￾designed) MIS was funded and implemented in the first place. Recommendation: Sustainable Outcomes should increase their quality control of the SOCY MIS implementation database, examine and revise their current MIS implementation protocols, provide new and more comprehensive user training, regularly monitor and clean the MIS as appropriate, and address the technical and design issues identified above, especially the lack of a unique HH identifier for each household. 4. Delay in Hoima One issue encountered during the implementation of the extended baseline survey occurred in Hoima. Hoima has recently experienced a significant rise in child trafficking and District officials have been on high alert. There, the survey team was extensively questioned to ensure there were no child trafficking concerns, and as a result were slightly delayed. The survey team also delivered a larger sample size, thus increasing the overall power of the analysis. The survey team conducted household surveys for 2,899 households from November 2017 to December 2017. This second baseline wave fell with the same time frame as the first wave in 2016. That year, 2,630 households were sampled, which included an additional 200 surveyed HH beyond the original 2,410 target (see above and the original 2016 baseline survey report). The main reason for the excess was a miscommunication between survey and field team management regarding the targeted average number of households required per parish. The target across both waves has been a minimum of 26 HH per parish, but the initial understanding in parts of the field team was that this number was much lower, and enumerators proceeded to survey additional parishes to reach their quota targets. Approximately half way into the survey this miscommunication was discovered, and those enumerators returned to the earlier parishes to survey the remaining households. The result was a much larger sample size, although this incurred no additional cost to the task order and the expense was born by SoCha. 5. HH Attrition HH attrition was the final issue, as not all of the original households were resampled. In total, 2,201 of the original 2,630 households were resampled. The drop in revisited households was due to two main reasons. First, 82 of the original 2,630 households in 2016 were in the Kasese district, parts of which were under siege by a militant group who explicitly forbade SoCha enumerators from completing their work and alternative parishes had to be selected from the district. These households were not resampled during the second wave 22 and leaving a total of 2,548 households to resample. Second, the remaining 347 households were visited by enumerators but were not surveyed because either the family had moved, or a consenting adult was not at home during the time. Enumerators were instructed to revisit empty households up to two more times (for a total of three attempts); after which they would randomly select an alternative HH in the village. When the Kasese households are removed, the resulting attrition rate from first to second wave was thus 13.6% when the oversampled HH are included, and less than 10% when the original power analysis estimates of 2,430 HH are used (recall that this estimate assumed a 10% attrition rate). As a result, the current attrition rate does not pose a threat to the validity of the original impact evaluation design. Nevertheless, this sample size still remains significantly larger than found in practically every published QCA survey we discovered.11 11 The only exception is Greckhamer 2008, who used 2,841 cases to evaluate business performance. See Greckhamer, T., Misangyi, V. F., Elms, H., & Lacey, R. (2008). Using qualitative comparative analysis in strategic management research: An examination of combinations of industry, corporate, and business-unit effects. Organizational Research Methods, 11(4), 695-726. 23 Extended Baseline Survey Results Survey Data Distribution The survey team conducted household surveys for 2,899 households across 94 parishes in 15 districts across the three regions. A copy of the full survey tool is found in Annex 2, and the dataset can be found in the dataset attachment Annex 6. The control group included 838 households, Treatment group 1 (R3 group) included 978 households and the Treatment group 2 (R1R3) included 1083 households. Combined, the two treatment groups yield 2,061 households. The distribution of households across regions is the following: HH Group Distribution by Region Region Control Group R3 group (Treatment 1) R1R3 group (Treatment 2) Combined Treatment Group Total Central 781 572 0 572 1353 Western 0 324 15 339 339 Southwest 57 82 1068 1150 1207 Total 838 978 1083 2061 2899 Most of the households from the control group are selected from Central Region (781), with the reminder found in the Southwest (57). Households from the now combined treatment group run across all three regions (572 in Central, 339 in Western and 1150 in the Southwest Region). Broken down by district, we find the following distribution: Region District Control Group R3 Group R1R3 Group Combined Treatment Total Central Gomba 58 153 153 364 Hoima 173 88 88 349 Luweero 227 95 95 417 Mityana 73 236 236 545 Rakai 250 250 Southwestern Bushenyi 112 112 224 Isingiro 57 46 14 60 177 Kabale 36 169 205 410 Kanungu 231 231 462 Kiruhura 147 147 294 Ntugamo 259 259 518 Rukungiri 136 136 272 Western Kamwenge 70 15 85 170 Kasese 92 92 184 Kibaale 162 162 324 It should be noted that one district from the Southwest region falls under the Control group designation while four districts from the Central region fall under the Combined Treatment group( see above table). Recall that this sampling was originally done under the assumption of two treatment group arms and had contained a much more balanced distribution of HH across geographies. Moreover, Control/Treatment Group designations were based upon the originally implementation timelines of the Sustainable Outcomes workplan. Given the 24 recent changes to implementation (i.e. there will most likely be only one treatment arm) and that no surveyed HH have yet been enrolled, it may make sense to reallocate HH sampled in Central region away from the Treatment group and to the Control group, while reallocating Southwestern HH originally allocated to the Control group to the Combined Treatment. The result would be a new balance of 1,353 HH in the Control Group and 1,546 HH allocated to the Combined Treatment Group. However, it should be recognized that regional boundaries in Uganda are mostly symbolic and do not meaningfully correspond to actual administrative boundaries, and that delays in implementation may again yield the relevance of a second treatment group. As such, these options and their implications should be discussed with Sustainable Outcomes and USAID/Uganda staff to determine the best course of action. Recommendation: Review the Control/Treatment group allocations in lieu of the recent change to implementation strategy and determine if reallocation of HH to regional boundaries would add value to the study’s learning questions. Means Testing of PEPFAR Indicators Below we present the results of this recent baseline survey in terms of PEPFAR indicator and HVAT scores. As Sustainable Outcomes recently informed us that there will likely be no second treatment arm, the descriptive data below is limited to a comparison of Control and Combined Treatment groups. The dataset can be found in the dataset attachment, Annex 6. It should be noted that the comparison of means between the two groups involved a recording of some of the original survey variables used to construct the HVAT (see below) into binary variables to comply with PEPFAR’s “yes/no” structure. We ran statistical comparisons across the control and combined treatment groups to test for significant differences. For continuous variables, we ran F-tests. Otherwise, we used Chi-Squared (χ²) tests for categorical variables. The actual value of each test should not be directly interpreted, but instead should be understood in terms of the significance level of the result, represented as * p≤0.05, ** p≤0.01, *** p≤0.001. Results with none of these three designations should be understood as insignificant differences across the groups. For this report, we reported on PEPFAR Indicators and HVAT scores. For more generic comparisons of HH demographic characteristics, such as age of HH head and total HH side, please refer to the original baseline report. Twenty-one secondary outcome measures are captured at the OVC and Youth wellbeing level for children aged 0-9 years (with questions directed to the primary caregiver) and youth aged 10-17 years. These include: • Percent of children whose primary caregiver knows the child’s HIV status • Percent of children < 5 years of age who are undernourished • Percent of youth too sick to participate in daily activities • Percent of children too sick to participate in daily activities • Percent of youth who have a Birth Certificate • Percent of children who have a Birth Certificate • Percent of youth enrolled in school • Percent of children enrolled in school • Percent of youth regularly attending school • Percent of children regularly attending school • Percent of children <5 years with recent diarrhea • Percent of children <5 years with recent fever • Percent of youth >10 to 17 years reporting irregular food intake • Percent of children >2 to 9 years reporting irregular food intake • Percent of caregivers who feel harsh physical punishment is appropriate means of discipline in the home or school 25 • Percent of households able to access money to pay for unexpected expenses • Percent of households able to access money to pay for health expenses in past 3 months • Percent of households able to access money to pay for school expenses in past 3 months • Percent of households able to access money to pay for food expenses in past 3 months • Percent of children 1-5 years fully immunized • Percent of youth aged 10-17 years reporting basic support For the 21 secondary outcomes, thirteen had no significant differences across the two groups. For those indicators in which there were significant differences – i.e. the control and treatment groups appear to be at different starting points – mostly involve economic geography. The ability to meet unexpected and ongoing expenses, such as in health, food and education, stand out as significantly differences across groups. Cultural differences across geographies regarding OVC discipline may also contribute significant differences in caregiver views regarding the appropriateness of harsh physical punishment at home or in school. Recent fever, child attendance in school, and caregiver knowledge of OVC HIV status are the other significant differences. The results are presented below: Outcome Indicator 1: Percent of children whose primary caregiver knows the child’s HIV status Control Group(838) Treatment Group(2061) Total(2899) Chi-Sqd No 387 (46.18%) 838 (40.66%) 1225 (42.26%) 7.444** Yes 451 (53.82%) 1223 (59.34%) 1674 (57.74%) Outcome Indicator 2: Percent of children < 5 years of age who are undernourished Control Group(191) Treatment Group(392) Total(583) Chi-Sqd No 116 (60.73%) 234 (59.69%) 350 (60.03%) 0.058 Yes 75 (39.27%) 158 (40.31%) 233 (39.97%) Outcome Indicator 3a: Percent of youth too sick to participate in daily activities Control Group(714) Treatment Group(1765) Total(2479) Chi-Sqd No 525 (73.53%) 1349 (76.43%) 1874 (75.59%) 2.319 Yes 189 (26.47%) 416 (23.57%) 605 (24.41%) Outcome Indicator 3b: Percent of children too sick to participate in daily activities Control Group(695) Treatment Group(1652) Total(2347) Chi-Sqd No 459 (66.04%) 1113 (67.37%) 1572 (66.98%) 0.391 Yes 236 (33.96%) 539 (32.63%) 775 (33.02%) Outcome Indicator 4a: Percent of youth who have a Birth Certificate Control Group(705) Treatment Group(1757) Total(2462) Chi-Sqd No 429 (60.85%) 1104 (62.83%) 1533 (62.27%) 0.842 Yes 276 (39.15%) 653 (37.17%) 929 (37.73%) Outcome Indicator 4b: Percent of children who have a Birth Certificate Control Group(687) Treatment Group(1640) Total(2327) Chi-Sqd No 439 (63.90%) 1018 (62.07%) 1457 (62.61%) 0.691 Yes 248 (36.10%) 622 (37.93%) 870 (37.39%) Outcome Indicator 5a: Percent of youth enrolled in school Control Group(714) Treatment Group(1765) Total(2479) Chi-Sqd No 157 (21.99%) 435 (24.65%) 592 (23.88%) 1.974 26 Yes 557 (78.01%) 1330 (75.35%) 1887 (76.12%) Outcome Indicator 5b: Percent of children enrolled in school Control Group(459) Treatment Group(1122) Total(1581) Chi-Sqd No 69 (15.03%) 191 (17.02%) 260 (16.45%) 0.939 Yes 390 (84.97%) 931 (82.98%) 1321 (83.55%) Outcome Indicator 6a: Percent of youth regularly attending school Control Group(705) Treatment Group(1757) Total(2462) Chi-Sqd No 238 (33.76%) 587 (33.41%) 825 (33.51%) 0.028 Yes 467 (66.24%) 1170 (66.59%) 1637 (66.49%) Outcome Indicator 6b: Percent of children regularly attending school Control Group(687) Treatment Group(1640) Total(2327) Chi-Sqd No 435 (63.32%) 964 (58.78%) 1399 (60.12%) 4.159* Yes 252 (36.68%) 676 (41.22%) 928 (39.88%) Outcome Indicator 7: Percent of children <5 years with recent diarrhea Control Group(236) Treatment Group(517) Total(753) Chi-Sqd No 177 (75.00%) 375 (72.53%) 552 (73.31%) 0.504 Yes 59 (25.00%) 142 (27.47%) 201 (26.69%) Outcome Indicator 8: Percent of children <5 years with recent fever Control Group(236) Treatment Group(517) Total(753) Chi-Sqd No 133 (56.36%) 332 (64.22%) 465 (61.75%) 4.239* Yes 103 (43.64%) 185 (35.78%) 288 (38.25%) Outcome Indicator 9a: Percent of youth reporting irregular food intake Control Group(714) Treatment Group(1765) Total(2479) Chi-Sqd No 653 (91.46%) 1639 (92.86%) 2292 (92.46%) 1.438 Yes 61 (8.54%) 126 (7.14%) 187 (7.54%) Outcome Indicator 10b: Percent of children reporting irregular food intake Control Group(644) Treatment Group(1534) Total(2178) Chi-Sqd No 608 (94.41%) 1456 (94.92%) 2064 (94.77%) 0.233 Yes 36 (5.59%) 78 (5.08%) 114 (5.23%) Outcome Indicator 11: Percent of caregivers who feel harsh physical punishment is appropriate means of discipline in the home or school Control Group(838) Treatment Group(2061) Total(2899) Chi-Sqd No 314 (37.47%) 880 (42.70%) 1194 (41.19%) 6.721** Yes 524 (62.53%) 1181 (57.30%) 1705 (58.81%) Outcome Indicator 12: Percent of households able to access money to pay for unexpected expenses Control Group(838) Treatment Group(2061) Total(2899) Chi-Sqd No 422 (50.36%) 1226 (59.49%) 1648 (56.85%) 20.234*** Yes 416 (49.64%) 835 (40.51%) 1251 (43.15%) Outcome Indicator 13: Percent of households able to access money to pay for health expenses in past 3 months Control Group(838) Treatment Group(2061) Total(2899) Chi-Sqd 27 No 406 (48.45%) 1188 (57.64%) 1594 (54.98%) 20.343*** Yes 432 (51.55%) 873 (42.36%) 1305 (45.02%) Outcome Indicator 14: Percent of households able to access money to pay for school expenses in past 3 months Control Group(838) Treatment Group(2061) Total(2899) Chi-Sqd No 501 (59.79%) 1362 (66.08%) 1863 (64.26%) 10.294*** Yes 337 (40.21%) 699 (33.92%) 1036 (35.74%) Outcome Indicator 15: Percent of households able to access money to pay for food expenses in past 3 months Control Group(838) Treatment Group(2061) Total(2899) Chi-Sqd No 371 (44.27%) 809 (39.25%) 1180 (40.70%) 6.219** Yes 467 (55.73%) 1252 (60.75%) 1719 (59.30%) Outcome Indicator 16: Percent of children 1-5 years fully immunized Control Group(426) Treatment Group(977) Total(1403) Chi-Sqd No 43 (10.09%) 116 (11.87%) 159 (11.33%) 0.935 Yes 383 (89.91%) 861 (88.13%) 1244 (88.67%) Outcome Indicator 17: Percent of youth aged 10-17 years reporting basic support Control Group(838) Treatment Group(2061) Total(2899) Chi-Sqd No 240 (28.64%) 591 (28.68%) 831 (28.67%) .000 Yes 598 (71.36%) 1470 (71.32%) 2068 (71.33%) *p<0.05, **p<0.01, *** p<0.001 Vulnerability Assessment and HVAT Scoring The primary impact indicator used for this evaluation is the Household Vulnerability Assessment Tool (HVAT). The HVAT is the official tool used by the Ministry of Gender Labor and Social Development (MGLSD) to obtain in-depth information about a household’s level of vulnerability. The HVAT is applied nationally and reported quarterly in the OVC MIS. A leaner variant of the HVAT is used by Sustainable Outcomes as the primary outcome indicator in the AMELP to gauge changes to household vulnerability over time (Result Area Indicator #2), as well as an implementation tool for household improvement and case management plans. As such, the HVAT is the logical choice for the impact evaluation, as it is aligned to Sustainable Outcomes program goals and performance metrics, is in line with the National MGLSD strategy and is accounted for in the national OVCMIS system. The HVAT is laid out according to six dimensions of vulnerability known as Core Programme Areas (CPAs). Each CPA is further divided according to a varying number of items; each of which is scaled across an ordinal spectrum of 0 to 4 in which 0 corresponds to some acceptable level of resilience while 4 corresponds to high levels of vulnerability. The national HVAT contains 18 items in total, which are distributed across the CPAs as follows: • CPA 1: Economic Strengthening (5 items) • CPA 2: Food Security and Nutrition (3 items) • CPA 3: Health, WASH and Shelter (5 items) • CPA 4: Education (1 item) • CPA 5: Psychosocial Support and Basic Care (2 items) • CPA 6: Child Protection and Legal Support (2 items) • Total: 18 items that span from 0 to 4 for a potential total vulnerability score of 72 28 HVAT scores are given as a percentage. To compute overall vulnerability, scores for each item are tallied across all CPAs to form the numerator; and are divided by the highest possible vulnerability score (i.e. a score of 4 across all 18 items = 72). The result is a percentage score falling across 0 to 100%. Vulnerability is then assigned according to placement on the percentage score across potential scores across quartiles as follows: • Not Vulnerable: 0-24% • Slightly Vulnerable: 25-49% • Moderately Vulnerable: 50-74% • Critically Vulnerable: 75-100% HVAT Means Testing For this survey, we found significant difference in the mean average vulnerability score and CPA ranking on all components save Health/WASH and Psychosocial across both groups, again potentially reflecting the similar economic geography and cultural differences speculated above. HVAT and CPA Score Means Testing (ANOVA) Category Group # Mean Std. Deviation F-Test CPA1: Economic Strengthening Control 838 11.42 3.34 28.79*** Treatment 2061 12.09 2.94 Total 2899 11.90 3.08 CPA2: Food Security Control 838 2.24 1.89 58.16*** Treatment 2061 2.89 2.17 Total 2899 2.70 2.11 CPA3: Health and WASH Control 838 7.05 3.40 3.21 Treatment 2061 7.29 3.20 Total 2899 7.22 3.26 CPA4: Education Control 838 1.19 1.62 4.92* Treatment 2061 1.34 1.69 Total 2899 1.30 1.67 CPA5: Psychosocial Control 838 1.12 1.93 0.02 Treatment 2061 1.13 1.93 Total 2899 1.13 1.93 CPA 6: Child Protection Control 838 1.95 1.39 17.95*** Treatment 2061 1.71 1.35 Total 2899 1.78 1.36 HVAT Score Control 838 24.96 7.57 25.22*** Treatment 2061 26.46 7.13 Total 2899 26.02 7.29 *p<0.05, **p<0.01, *** p<0.001 As stated before, this is not an issue for the difference-in-difference design as long as the parallel trends assumption still holds. 29 HVAT Distribution Turning to the distribution of vulnerability, we find the following distribution using the HVAT: Distribution of Vulnerability across Control and Treatment Groups HVAT Category Control Treatment Total Not Vulnerable 171 (20.4%) 285 (13.8%) 456 (15.7%) Slightly Vulnerable 617 (73.6%) 1607 (78.0%) 2224 (76.7%) Moderately Vulnerable 50 (6.0%) 169 (8.2%) 219 (7.6%) Critically Vulnerable 0 0 0 Total 838 2061 2899 Overall, we find a high concentration of “slightly vulnerable” across the three groups, 76.7% of the sample (the previous survey was similar at 73% of the total sample), followed by a smaller concentration of “not vulnerable” households, and the smallest percentage of “moderately vulnerable” households, and a complete lack of “critically vulnerable” households across the entire sample. Overall, the HVAT scores suggest that Sustainable Outcomes is supporting a higher number of “less vulnerable” households than those that score as “more vulnerable,” and that many “not vulnerable” households should potentially not be supported at all in favor of enrolling more vulnerable HH. This raises the question: Has Sustainable Outcomes been supporting HH in regions of Uganda that don’t need assistance? Or is the HVAT tool used to measure vulnerability mis-calibrated and should be refined to better capture the dynamics of vulnerability?” In the previous baselines survey report, we explored these questions extensively using a number of tools. Overall, we found that Sustainable Outcomes was being implemented in appropriate areas in need of assistance and that the survey’s sampling approach was valid. Rather, it was the design of the HVAT tool that suffered from some challenges that impacted the resulting vulnerability scores and status distribution. More precisely, the review revealed the following: • Contradictions in scoring, due in part to inconsistent units of measurement; • Arbitrary spaces in the scoring of each item can bias household vulnerability scores upward (i.e. they are less vulnerable); • Many of the qualitative values found under each item either do not logically follow the structure of the data or do not have a relationship to other elements; • The items on the HVAT are not equal in relevance and therefore should not be scored equally; • The items fall across different underlying dimensions and therefore are not easily aggregated using the current scoring method. Principle Component Analysis (PCA) was used to test the internal consistency of each item across the HVAT index (i.e. the validity of the ordinal scaling of qualitative responses set up under each question) and its six constituent “Core Program Areas” (CPAs, i.e. economic strengthening, food and nutrition, health and WASH, education, child protection and psycho-social). The results suggested a very poor fit in terms of how well all HVAT items combine to form a unidimensional scale. For instance, less than 16% of all variance could be explained on a combined HVAT component, and almost half of the items dropped out with scores below the minimum threshold. This suggested that the HVAT cannot be easily reduced to a single dimension score. The implications are that HVAT scores can be a misleading proxy for comparing underlying vulnerability across households. For instances, two households that hold the same overall vulnerability score may in fact differ significantly in the types of vulnerability, e.g. one may be vulnerable in terms of food security, while the other may be vulnerable in terms of the lack of child protection. Building upon these lessons, we adopted a different 30 analytical approach to this survey wave; and refined the HVAT scores to build a more accurate model. To this discussion we now turn. Changes to the HVAT Distribution To address this issue, we used factor analysis to appropriately weight the individual items in the survey and generate more accurate HH vulnerability scores.12 These results are presented below in conjunction with the original HVAT raw scores to illustrate the differences of using Factor Analysis to appropriately weight each item on the questionnaire and the effects on the sampled vulnerability distribution: Change in Vulnerability Distribution from Original to Refined HVAT Vulnerability Category Refined Score Distribution and Percentages Original Score Distribution and Percentages Change from Original to Refined HVAT Critically Vulnerable 109 (3.76%) 0 109 were Added Moderately Vulnerable 1104 (38.08%) 219 (7.6%) 885 were Added Slightly Vulnerable 1488 (51.33%) 2224 (76.7%) 736 were Removed Not Vulnerable 198 (6.83%) 456 (15.7%) 258 were Removed Total 2,899 2,899 As can be seen, moving away from the raw aggregated vulnerability scores to the more refined HVAT approach that incorporates score weightings generated by factor analysis yields significantly different results. First, there is a substantial exodus of 258 HH – more than half of the original “not vulnerable” category – to the more vulnerable categories below it. Several HH in the “slightly vulnerable” category embarked on a similar exodus (736 were re-assigned to more vulnerable categories). Correspondently, membership in the “moderately vulnerable” grew and the “critically vulnerable” category is now populated with the most vulnerable households. As such, the refined results yield a more accurate recalibration of the sampled HVAT vulnerability distribution and weight the value of each response more appropriately. These results also suggest that original HVAT significantly under represents overall household vulnerability. Below we provide more detailed information about the steps taken to refine the HVAT analysis. Readers less interested in this information can skip to the Using QCA to Identify Service Pathways Out of Vulnerability section below. More Detailed Look at What was Done: Refining the HVAT through Factor Analysis As discussed above, the traditional HVAT determines HH vulnerability by simply adding the values of each score on the HVAT tool and dividing this total by the worst possible vulnerability score (72).13 One of the main problems with way of combining scores on each question is that it assumes all questions, and the concepts they seek to represent (called CPAs) are equal. One could logically ask “why should a HH that lives in unsafe conditions where the roof is about to collapse be given the same vulnerability score as someone who doesn’t know the HIV status of their children as well as someone who or as someone who thinks it is appropriate to beat children at home and school?” This and similar types of questions basically disagree with the simple aggregation approach of the HVAT, but then the problem arises: Then how should the scores on the HVAT tool be combined? 12 The steps taken in this approach are detailed in the next section for interested readers to examine. 13 These percentages are then converted into categories based upon where they fall across the quartiles of percentages (Not Vulnerable: 0-24%, Slightly Vulnerable: 25-49%, Moderately Vulnerable: 50-74%, Critically Vulnerable: 75-100%). 31 This report advances the use of factor analysis to generate index scores (as oppose to simple aggregation).14 To oversimplify things a bit, factor analysis can be used to generate a “weighting scheme” that can be applied to each question’s score to generate new scores that better reflect how important it is relative to the overall concept of HH vulnerability, and these scores can then be aggregated to produce a more accurate and refined vulnerability index. In the simplest terms, factor analysis is a statistical method for measuring so-called latent variables, which are “constructs” that cannot be directly observed but still can be captured by measuring its various types of questions. For example, HVAT concepts such as “Economic Strengthening,” “Food Security” and “Vulnerability” are all latent constructs that cannot be directly observed but can be captured through various questions on the HVAT tool. The key to understanding a latent variable is through using multiple indicators (here in the form of survey questions) to capture it. The main assumption behind this is that, if there is a latent variable in effect, then multiple questions should have similar patterns of responses because they are all associated with that underlying latent construct (i.e. not directly measured) variable. Without factor analysis, it is difficult to know if the measure is internally valid and indeed measuring what it purports to do so. Below we report on our results of the factor analysis and generate new HH vulnerability scores based upon a refined HVAT.15 But first it must be mentioned that there are certain conditions that must be met for a factor analysis to be performed. In this case, the HVAT measure met the requirements: 1. Severely skewed items: Such items are problematic for factor analysis as they lack variation. In this case, none of the items were deemed as severely skewed to be excluded. 2. N:k ratio: The 10:1 ratio (data points: each item) is a conservative estimate to ensure that the factor model has the needed power to conduct factor analyses. The basic rule of thumb is at least 10 data points (e.g. HH responses) are needed for each item (e.g. questions on the survey).16 With 22 items (questions) and a dataset of 2,899 households, this condition was also met.17 In fact, having such a large dataset gave us the opportunity to conduct both exploratory and confirmatory factor analyses on separate datasets. Exploratory analysis is typically used first to identify the underlying patterns of the data and if/how well the questions fit to the latent constructs. Exploratory analysis also gives suggestions to improve the model and drops questions that don’t fit. Once this refined model is found, confirmatory analysis is run to test it and confirm that the new model is in fact a strong fit. Crucially, these two analyses should not be done on the same dataset, because the solutions suggested by the exploratory analysis will of course come out perfect using confirmatory analysis. Instead, the refined model tested by the confirmatory analysis should be done using a different dataset. How to have two different datasets for this report? To do this, we randomly divided the 2,899 dataset into two sub-datasets, running exploratory factor analysis the first and testing the refined model using confirmatory factor analysis on the second. All analyses were conducted using MPlus version 7, and the results are reported below. 14 Factor scores are particularly useful for measuring so-called latent variables, which are “constructs” that cannot be directly observed but still can be captured by measuring its various types of questions. For example, HVAT concepts such as “Economic Strengthening,” “Food Security” and “Vulnerability” are all latent constructs that cannot be directly observed but can be captured through various questions on the HVAT tool. The key to understanding a latent variable is through using multiple indicators to capture it. The main assumption behind this is that, if there is a latent variable in effect, then multiple observed variables should have similar patterns of responses because they are all associated with that underlying construct (i.e. not directly measured) variable. 15 For this wave, we retained the original HVAT questionnaire structure, added additional questions and applied a different analytical approaching using the factor analysis (measurement model) aspect of structural equation modelling (SEM). SEM is a broad and very powerful set of statistical operations that seek to uncover underlying relations between measured and unmeasured variables. The factor analysis aspect of SEM is especially useful for creating and confirming indexes. For this round of the baseline, we included additional questions under the Economic Strengthening CPA (borrowing questions about household items from the previous HAT), modified the structure of access to drinking water and views on beating questions (guided by the results of the PCA), added an additional module about caregiver/child relationships and perceived self-esteem. These were incorporated into the expanded HVAT in an attempt to mitigate potential social response biases. They can be found as part of the attached questionnaire in Annex 2. 16 Tabachnick, B. G., & Fidell, L. S. (2007). Using multivariate statistics. Allyn & Bacon/Pearson Education. 17 For more information on this best practice, see Ford, J. K., MacCallum, R. C., & Tait, M. (1986). The application of exploratory factor analysis in applied psychology: A critical review and analysis. Personnel Psychology, 39(2), 291. 32 Exploratory factor analysis (EFA) Though the HVAT has 5 “Core Programme Areas” (CPAs), we conducted an EFA to allow the questions to come together based upon their structure and without any theoretical model constraints (i.e. to use EFA to find the “natural” clustering of the questions). Although EFA is typically a good practice when testing the psychometric properties of a new measure or if a measure has been implemented in a different culture or country context, as far as we know, no EFA had been conducted on the HVAT to this point.18 The first results of the EFA generated a “Scree plot,” which is a graphic visualization of amount of variation in the data explained (the eigenvalues) on the Y axis, and the number of factors (e.g. latent constructs) used to explain that variation on the X axis. The Scree Plot shows that up to 22 factors could be identified, translating into 22 CPAs, and probably more as well. However, the “elbow” or largest drop in the difference of variation explained by the next factor indicates the probable number of factors that should be extracted. In the case of the HVAT, the Scree Plot suggested either a 3 or 4 factor solution. The next step in the EFA was based on the model’s goodness-of-fit statistics, mostly focused on what are known as the comparative fit index (CFI), the Tucker-Lewis Index (TLI), and the Root Squared Error Approximation (RSMEA).19 Using these measures, the four-factor solution scored a much better fit than the three-factor model, so the former was chosen. We then examined to what extent these new factors made sense and corresponded to the CPAs of the HVAT. In fact, the items loaded in a somewhat different manner than in the HVAT, which means that original questions that fell under the previous CPAs were misaligned. We then reformulated the four factors into new CPAs with corresponding titles that seemed to represent what the CPAs were capturing. These are: Household Assets, Income Privileges, Child Health, and Child Protection. The table below indicates each item loading on the respective factor with the new CPA descriptions above: Table 1: Item loadings in exploratory factor analysis Item Factor Description*20 Description ID Household Assets Income Contingent Privileges Child Health Child Protection 18 Because the CPAs in the HVAT are believed to be connected to each other, we used the Oblique rotation for this EFA. Oblique rotations allow higher level factors (think latent concepts, such as CPAs) to be correlated with each other. Therefore, when interpreting the correlation matrix, item loadings and the factor correlations, the rotation must be kept in mind. 19 In their seminal paper, Hu and Bentler (1999) indicate that multiple indices, in addition to be used to measure goodness of fit in structural models. Each indices has a cut off criterion for good fit; in which the CFI and TLI scores should be 0.90 or greater indicates a good fit, and RMSEA should be less than .05. The four-factor solution indicated the better fit (χ2 (231) = 5245, RMSEA = .041, CFI: 0.92, TLI; 0.89) and thus the four factor solution was retained. See Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural equation modeling: a multidisciplinary journal, 6(1), 1-55. 20 Asterisk indicates statistical significant factor loading 0 1 2 3 4 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 Eigenvalue Number of factors that can be extracted Fig 1. Scree Plot 33 Main Contributor CPA1_1 -0.179* Source of Income CPA1_2 0.328* Monthly Income CPA1_3 0.516* Assets CPA1_4 0.760* Urgent Expenses CPA1_5 0.369* Home Assets CPA1_7 0.615* Source of Food CPA2_1 0.380* Meals in a day CPA2_2 0.444* Food types CPA2_3 0.468* WASH facilities CPA3_1 0.460* Child HIV status CPA3_2 -0.163* Receiving Treatment CPA3_3 0.206* Stable Shelter CPA3_4 0.794* Toilet Facilities CPA3_5 0.703* Source of Water CPA3_6 0.221* Schooling CPA4 0.289* Health Access CPA5_1 0.691* Depression CPA5_2 0.752* Child Discipline *Physical CPA6_1 0.629* Child Discipline *Withholding Meals CPA6_2 0.354* Child Discipline *Yelling/Screaming CPA6_3 0.166* Views on Beating CPA6_4 0.847* Some of the factor correlations between the factor (the latent construct) and the questions were weak (less than the absolute value of .4), but still informative. To improve some of these lower correlations, one solution could be to split the larger factors into smaller ones, but doing (we tried) reduces the overall goodness of fit of the entire model (TLI, CFI and RMSEA) to below acceptable levels. Thus, the four-factor solution remains the optimal model of the data. Table 2 below presents the higher-level correlations among factors. It should be noted that these correlations test the relationship of latent constructs to each other, which is different than the above where the tests were around how well each group of questions correlates to its underlying latent constructs. The correlation between household assets and income contingent privileges holds with a decent sized correlation and is significant, thus confirming common sensical views that a child living in a home with assets and higher income would also be less likely to face health and discipline issues. Nonetheless, some of the weaker correlations amongst factors suggests that the new factors may be tapping into two or more different latent constructs at the same time when in fact they are different from each other and should be separated. Moreover, the child protection construct’s relationship to the rest of the constructs lack significance, which may suggest that the inner￾workings of child protection are so different to the inner-workings of the other, more “material” factors that no significant relationship across them exists. Put differently, vulnerability under child protection might be relatively unrelated to the types of vulnerability that manifest themselves under other concepts such as assets, privileges and health. 34 Table 2: EFA – Factor Correlations Household Assets Income Contingent Privileges Child Health Child Protection Household Assets 1 Income Contingent Privileges 0.311* 1 Child Health -0.101* 0.081* 1 Child Protection -0.083 0.018 0.071 1 Here, it is important to mention that we ran multiple models by dropping various questions from each to test if this would improve the overall goodness of fit of the model. The EFA suggested that the most promising would be models that dropped the HVAT questions regarding who is the main contributor, the reported monthly income, and household assets. In non-technical terms, the inclusion of these questions does create some noise in the way that the HVAT works. Since these items do not contribute to any latent or underlying cause (for example, monthly income can help alleviate or exacerbate vulnerability, but is not likely a cause of vulnerability itself), a decision to revise the HVAT should consider dropping them in subsequent factor analysis. Although it may seem counter intuitive to drop such items as estimated HH income from a vulnerability index, doing so doesn’t suggest that these items are not important. Rather, items such as income, assets and main contributors are probably better conceptualized as “mediating” and “moderating” factors that channel vulnerability in different ways, but in fact are not part of the integral definition of what constitutes vulnerability. We did run alternative models that dropped these questions from the revised HVAT, but in fact these suggestions translated into only very marginal differences in the model goodness of fit. Thus, we kept the questions in the refined HVAT model to retain the original structure of the HVAT. Although the EFA suggests that these questions should not be included as constituent elements of vulnerability and that they don’t fit very well by including them (i.e. they don’t add value), we don’t gain any meaningful statistical improvements by dropping these questions (they don’t subtract value either). Confirmatory Factor Analysis The next step taken was to confirm the exploratory factor structure. Using the other half of the dataset, we conducted confirmatory factor analyses to replicate the 4-factor model, essentially testing how well our refined HVAT could “travel” to other contexts and households and still remain valid. Fortunately, the confirmatory model indicated similar goodness of fit statistics (χ2 (203) = 867, RMSEA = .047, CFI: 0.88, TLI; 0.87) to the exploratory model. Thus, the model was confirmed to be statistically valid. 35 The resulting four confirmed factors can be defined as new CPAs: 1. Assets: The combination of assets in and outside the house define this factor. It is about the material wealth of the household 2. Income contingent privileges: This factor consists of items that indicate the beyond basic amenities that the household can afford. The largest factor, it includes stability of shelter, access to WASH facilities, 3 meals a day, variety in food types consumed, clean source of water. 3. Child Health: This factor encompasses the positivity of the items that relate to child emotional and physical wellbeing. All items except for Main contributor and Child HIV status are inversely related. This makes sense because the scale is constructed in a way that lower values indicate better health and access to schooling and thus the inverse values indicate better outcomes for the child. 4. Child Protection: This factor encompasses the negative items that work against Child Protection, such as disciplinary actions that includes physical or emotional repercussions for the child. Though the factor and item loadings are similar in both the EFA and CFA models, there are two significant differences. First, the Asset and Income Contingent Privileges factors strongly correlate with each other (r= 0.9). Looking at the items in both factors, it is rather intuitive that both factors have to do with the role of income in accessing better facilities. Second, the correlation between the factors of Child Health and Child Protection have a much stronger correlation in the CFA dataset (-.38) than in the EFA (-.071), suggesting that the relationship is much stronger across wider segments of Uganda than originally predicted by the EFA. Measurement Invariance between Control and Treatment Groups Before factor scores can be generated, it is important to calculate if the factor structure of the HVAT is “invariant” between control and treatment groups, i.e. that the HVAT model relationships and weighted scores are consistent across the two groups. Moreover, considering that these groups also tend to follow geographic, cultural and even language divisions, the test for invariance is a strong predictor of how “universal” the refined HVAT is when measuring vulnerability across these regions and groups in Uganda. Measurement invariance testing is typically run across three models – unconstrained, measurement weights and structural covariance – with the structural covariance model as the most stringent and difficult to achieve for any index. Across all three models, acceptable CFI, TLI and RMSEA scores suggest that the factor structure, HVAT questions and refined weights assigned to those questions are understood the same way across the different groups in the dataset (i.e. the measure is more “universal”). As such, measurement invariance testing was run across the 36 entire dataset (instead of limiting it to the EFA or CFA subsets) to see if the model worked differently across the control and treatment groups. Unconstrained model Measurement Weights model Structural Covariances Model CMIN/DF 5.608 5.398 6.292 CFI .861 0.859 0.814 TLI .825 0.833 0.800 RMSEA .040 0.039 0.043 Overall, the above table indicates what can be categorized as an “acceptable fit”21 across the three different measurement invariance models. The HVAT passes the first two tests regarding unconstrained and measurement weights models. However, the structural covariances model, the most stringent and difficult model testing for measurement invariance, is the weakest model of all and implies that both control and treatment groups are variant. It should be noted that this is also the most restrictive and constrained model statistic and not required for testing means difference related to an impact evaluation. Further testing of the HVAT using data from other parts of the country is most likely needed to determine how universal it is in Uganda or if it is more limited to the specific regions, languages and cultures of the Southwest, Central and West. Yet considering that these samples were drawn using random sampling techniques across three regions and 17 districts across Uganda, we hypothesize that this new, refined HVAT model (with its weighted scales) could potentially be universally applied across the country, and comparable relationships are likely to emerge. Recommendation: Sustainable Outcomes and other HVAT users consider a workshop to discuss a potential revised HVAT using these results as the basis for a more refined HVAT applied across the country. Factor Scores With the refined HVAT weighting scores and relationships both identified and validated by using factor analysis, the final step in refining the HVAT is to apply the factor weightings to the survey data to generate new, refined individual HH vulnerability scores. To do this, we applied the regression method to the cluster of items found under each factor (this takes into account not only the correlation between the factors and between factors and observed variables (via item loadings), but also the correlation among observed variables)22 and used SPSS to generate the scores.23 This created four sets of weighted factor scores, one for each revised CPA, which were then simply aggregated to yield sum scores for factor scores. 24 These results are presented below in conjunction with the original HVAT raw scores to illustrate the differences of using Factor Analysis to appropriately weight each item on the questionnaire and the effects on the sampled vulnerability distribution: 21 Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9(2), 233-255 22 See DiStefano, C. et. al. (2009), “Understanding and Using Factor Scores: Considerations for the Applied Researcher,” in Practical Assessment, Research and Evaluation, Vol. 14, No. 20 for a discussion on the pros and cons of each technique. 23 To do so, we used the “factor scores” function under Dimension Reduction selecting Promax rotation and Principle Axis Factoring under a fixed number of factors (1) for a correlation matrix using the regression method. 24 Crucially, we introduced two “dummy” cases into the dataset in which the first had “perfect vulnerability” (all the scores on each question were originally coded as 4 – the most extreme ranking), and “perfect non-vulnerability” (all the scores on each question were originally coded as 0 – the most non-vulnerable ranking). The distance between these two sum scores was then divided into quartiles to correspond to the critical, moderately, slightly and not vulnerable structure in the original HVAT. It should be noted that using these dummies to determine the bandwidth of each vulnerability category is necessary to avoid categories that are completely dependent upon the data. For example, if all HH were sampled from an extremely rich neighborhood in Kampala (e.g. Kololo), using a data-driven approach to the categories would class the bottom 25% of the sampled HH as critically vulnerable. We avoid this problem by pinning “objective” borders to the vulnerability spectrum that exist independent of the HHs evaluated by the HVAT. 37 Change in Vulnerability Distribution from Original to Refined HVAT Vulnerability Category Refined Score Distribution and Percentages Original Score Distribution and Percentages Change from Original to Refined HVAT Critically Vulnerable 109 (3.76%) 0 109 were Added Moderately Vulnerable 1104 (38.08%) 219 (7.6%) 885 were Added Slightly Vulnerable 1488 (51.33%) 2224 (76.7%) 736 were Removed Not Vulnerable 198 (6.83%) 456 (15.7%) 258 were Removed Total 2,899 2,899 As can be seen, moving away from the raw aggregated vulnerability scores to the more refined HVAT approach that incorporates score weightings generated by factor analysis yields significantly different results. First, there is a substantial exodus of 258 HH – more than half of the original “not vulnerable” category – to the more vulnerable categories below it. Several HH in the “slightly vulnerable” category embarked on a similar exodus (736 were re-assigned to more vulnerable categories). Correspondently, membership in the “moderately vulnerable” grew and the “critically vulnerable” category is now populated with the most vulnerable households. One of the reasons why the original HVAT scores may be so misaligned with the weighted scores is connected to a potential “positive social response” bias around questions involving abuse and socio-psycho well-being. Respondents may be more reluctant to share this type of information with enumerators and inflate their responses towards less vulnerable categories. Factor Analysis helps correct for this by drawing upon the underlying structure of the data to better tease out the relationships among questions across the sample and to some extent bypass response bias. As such, the refined results yield a more accurate recalibration of the sampled HVAT vulnerability distribution and weight the value of each response more appropriately. The logical next step in this type of analysis will be to explore the relationship between vulnerability – as measured by the refined HVAT – to various intervening and “lived experiences” that can be captured qualitatively and potentially quantified. For example, female vs male headed households may be correlated differently to this vulnerability structure, as might households with higher or lower numbers of youth and children. In fact, the number of relevant “lived experiences” factors may grow to a complexity that will be difficult for correlation-based approaches to meaningfully unpack, just as it will also make it more difficult to translate into more tangible policy recommendations (e.g. using the results to design a more tailored and differentiated OVC assistance strategy that can still be nonetheless reasonably managed across wider geographic areas and taken to scale). The next section of this report takes a necessary step in this direction using QCA analysis to identify how configurations of key services combine in different ways to help reduce a household’s vulnerability. Yet more opportunities for learning exist, and we therefore recommend that: Recommendation: Sustainable Outcomes and USAID/Uganda staff should consider conducting additional formative and exploratory analyses on some of key mediators channeling vulnerability in different ways. This type of analyses should be results driven in that the findings can be directly used to inform more targeting OVC programming taken to scale. Using QCA to Identify Service Pathways Out of Vulnerability The two baseline waves of data collection represent a unique opportunity to analyze which combinations of factors outside of the program’s control drive changes in household vulnerability. Specifically, the data allows us to look at which sets of services provided by other programs both on the household and the community level are associated with increasing or decreasing vulnerability. In the first instance, different combinations of services that are found to contribute to household wellbeing (or lack thereof) represent alternative paths to 38 reducing and increasing household vulnerability, respectively. Its results could provide useful information on adverse and conducive contexts for the program’s desired effects on vulnerability at the household level.25 We used QCA to identify how different combinations of social services might help reduce (or increase) household vulnerability to identify which combinations of factors best explain how an outcome was (or was not) achieved. QCA was used to draw from a comprehensive list of surveyed factors that characterize the households, their context (including services provided by other programs), and the program activities they participate in. At the same time, the data collection of this evaluation was not designed to rigorously assess the impact of services provided by other programs or organizations. In particular, we face an “enrollment problem,” in that we do not know why these sampled households were enrolled in the services we surveyed and the criteria those organizations applied to select these households. Not receiving the service may indicate an unfulfilled need of the household that could contribute to household vulnerability. Alternatively, not receiving the service may instead indicate that the household is comparatively less vulnerable. Given these limitations, the primary objective of the present QCA is to identify alternative pathways out of vulnerability for HH that will hopefully be enrolled in Sustainable Outcomes under the impact evaluation. Similar to the way we build “trend lines” to facilitate a Difference-in-Difference comparison of control and treatment groups in the impact evaluation design, here we build “service lines” through QCA to identify various service trajectories already underway in the implementation areas. Moreover, these findings may help Sustainable Outcomes improve or refine their implementation design. Specifically, QCA identifies configurations of factors outside the program that contribute to households increasing or decreasing their vulnerability. Furthermore, to reduce the influence of random change, paths to reducing vulnerability that other service providers offer to the surveyed households will only be identified by QCA if their services are particularly common or exceptionally strongly associated with the outcome. Fortunately, the very large dataset (from a QCA perspective) allows us to do this. Results of the QCA Overall, we found that 375 households reduced their vulnerability without any assistance from Sustainable Outcomes. QCA identified how these households travelled across three types of pathways out of vulnerability. Eighty of the total 375 households may have travelled more than one pathway out of vulnerability, and thus the corresponding “coverage” percentage scores add up to more than 100%. These pathways are: Path 1 shows that in 21% of the cases where moderately vulnerable households achieved at least slight improvements in vulnerability, they had more than three children, were members of savings groups, gained access to direct cash transfers recently, and had not lost access to HIV and GBV prevention since Wave 1. In comparison with path 2 and 3, the requirements with regard to service provision for path 1 are higher. This indicates that a higher number of children represents a greater risk to household vulnerability that needs to be offset by participation in savings groups and receipt of direct cash transfers to achieve reductions in household vulnerability. Put differently, among this group of households these services are of particular relevance to reduce household vulnerability, and the role of membership in savings groups should be emphasized for HH with four or more children. 25 As mentioned under the “limitations” section of this report, the sampled HH of the initial baseline were not enrolled in the Sustainable Outcomes activity and thus the original intent to use QCA to identify which combinations of Sustainable Outcomes-administered support services pulled HH out of vulnerability could not be realized. Once implementation data on households that are participating in Sustainable Outcomes becomes available, the findings of the present QCA will be used to support a comprehensive configurational impact analysis of the program. See Annex1a for more details. 39 Path 2 shows that among moderately vulnerable households with fewer children, the absence of losing access to at least one external service, here in the form of not losing access to HIV and Gender Based Violence prevention, is sufficient for them to achieve at least slight reductions in vulnerability. One peculiar aspect of this solution is how NOT being a member of a savings group is highlighted as an essential part of the pathway to improvement (which is different than saying that membership in a savings group doesn’t matter either way; here, it states, NON-membership appears to matter). This could be because the financial burden of membership for households with less than four children/youth outweighed the benefits of membership, or because the savings group in practice lost money and therefore withdrawal was because the group collapsed. Not gaining access to parent coaching services also seems to play a role, but again this could be because the HH no longer required these services (as opposed to the act of deliberately avoiding these services as part of the HH’s vulnerability reduction strategy). Overall, this pathway suggests that access to HIV and GBC services may have helped HH “hold the line” and avoid further increases to their vulnerability. Path 3 expresses the strongest trend towards reducing vulnerability: Among moderately vulnerable households, vulnerability is decreasing over time. This appears to be connected to again “holding the line” in terms of access to HIV and GBV prevention services (including community testing of HIV) but also in terms of sustained access to food support. Thus, for 69% of moderately vulnerable households that achieved a reduction in household vulnerability, the absence of losing access to services from 2016 to 2017 provided a sufficient basis for improvements in vulnerability. While slight improvements in vulnerability among moderately vulnerable households is the only outcome that could be determined positively by patterns in service provision, the configurational analyses were able to identify paths that led to the absence of changes in vulnerability in several cases. Put differently, what combinations of Paths to slight improvements in vulnerability among moderately vulnerable households (consistency threshold .65, α < .05) Path HH context Access to services Mildly less vulnerab le Cover￾age Saving groups Gained access Did not lose access Child￾ren (4+) Cash HIV test (HHL) Parent Food HIV￾GBV HIV test (CL) 1 21% 2 31% 3 69% Notes: Outcome represents a 10% reduction in the HVAT vulnerability score. Solution coverage: 83%. Solution consistency: 83%. Directed assumptions: Saving groups (+), Direct cash transfers (+), HIV testing (household level) (+), Parenthood-related services (+), Losing access to food security services (-), Losing access to HIV-GBV prevention (-), Losing access to HIV testing (community level) (-). Also note: the column colors refer to the type of variable used, e.g. Tan is context, rouge is type of service, and green is outcome. Green circles indicate the presence of a variable, and red circles indicate its absence. Blank spaces indicator presence or absence of the variable is irrelevant. 40 services did nothing to change HH vulnerability?26 For slightly vulnerable households and the full sample of all vulnerable households, we obtained QCA solutions for the absence of strong reductions and the absence of strong increases in vulnerability. For moderately vulnerable households, we were also able to identify paths that led to the absence of slight increases and the absence of strong increases in vulnerability. Needless to say, these results – how to stay in vulnerability (as opposed to how to move out of it) - are less informative than explanations of how vulnerability is reduced. The pathways identified in this “no change” analysis consist almost entirely of HH that lack any access to any services or experienced no change in their access from one year to the next. What then could be the role of not having these services? It could be that not having access to these services keeps a household locked into vulnerability and prevented the HH from improving; but another interpretation could be also that not having access to services kept households from falling deeper into vulnerability. Answering this question would require more in-depth research into HH that declined further into vulnerability to see if it was because the services they did receive weren’t strong enough to overcome the decline or because the services actually helped push them deeper. But for this survey, these pathways most likely appear because there were so many households in the sample that received no services at all combined with so many households that witnessed no meaningful change in their vulnerability score. As a result, this over representation of the “banal” lack of changes can still lead to significant relevant results in statistical terms and valid configurations of a lack of services in QCA terms. As such, these paths are probably best understood through the argument that those households who lacked access to services to begin with (as of Wave 1) and experienced no change after (in Wave 2) do not experience any meaningful change in their vulnerability. Such a finding reinforces the Sustainable Outcomes Theory of Change around the importance of access to and continued receipt of various services, and subsequent waves of the impact evaluation will help determine if the types of services offered by Sustainable Outcomes indeed have a positive effect. The dataset can be found in the dataset attachment, Annex 6 and more extended graphical presentations can be found in Annexes 8 and 9. Meanwhile, the QCA results do offer the following programmatic recommendations: Recommendation: Sustainable Outcomes should consider encouraging greater participation in a savings group (hopefully SILC) combined with encouraging additional access to Cash Transfers and ongoing access to HIV and GBV prevention services as one viable pathway out of vulnerability for moderately vulnerable households that house four or more youth/children. 26 The model specifications of these analyses are reproduced in Annex 4. How to read the Path Diagram The diagram depicts all paths among households of the respective vulnerability category that led to the outcome under study. Each numbered row represents one path. The grey cells on the right show the outcome the paths are leading to, and the share of households that reached the outcome on the respective path. The outcome is the same for all paths identified by the same analysis. A green plus sign indicates that the presence of the outcome was analyzed; a red minus sign indicates that the absence of the outcome was analyzed. The light-green and orange cells contain information on the conditions that were employed in the analysis. A green plus sign indicates that the presence of the condition was causally relevant for the given path; a red minus sign indicates that the absence of the condition was causally relevant. If the cell is empty, the condition was not relevant for the path. This means that the respective path to the outcome was open to households where the respective condition was present as well as to households where it was absent. The first line of the diagram above reads as follows: In 21% of the cases where moderately vulnerable households achieved at least slight improvements in their vulnerability, households had more than three children, were members of savings groups, gained access to direct cash transfers recently, and had not lost access to HIV and GBV prevention since wave 1. So on and so forth. 41 Recommendation: Considering the findings of the QCA analysis presented above, Sustainable Outcomes may also consider whether or not membership in savings group by HHs with less than four youth/children may entail financial or other types of burdens that overcome the benefits of savings group membership; thus raising the possibility that withdrawal from savings groups by these types of HH may reduce their vulnerability status. Recommendation: Sustainable Outcomes may wish to consider how ongoing access to HIV and GBV prevention services, combined with regular HIV community-testing services and with ongoing access to steady food support may form a viable pathway out of vulnerability, even without joining a savings group (this proposition could of course also be rigorously tested by reinstating the original dual treatment group design identified in 2016). Recommendation: Sustainable Outcomes should strongly consider enrolling the households surveyed across both waves into the program for services so that a rigorous quasi-experimental design impact evaluation can still be completed and a rigorous QCA can further unpack what combinations of services can lead to pathways out of vulnerability. Below we provide more detailed information about the steps taken to refine the HVAT analysis. Readers less interested in this information can skip to the final Conclusions and Recommendations section below. More Detailed Look at Using QCA: Steps Taken The evaluation presents both a challenge and an opportunity to QCA. QCA is a qualitative method that maximizes leverage to a researcher who has an intimate knowledge of the cases under study. The large number of cases covered by the present analysis means that the information available to differentiate between households that reduced or increased their vulnerability over time and those that did not is limited to the data obtained in the surveys. Differences between households that are not captured by survey items may blur the lines of otherwise distinctive patterns of association between services and outcomes, which may lead to what are known as “contradictions” applicable to the type of QCA used here (crisp￾set). The effect of contradictions is clearly seen in one of QCA’s strongest parameters of fit – consistency scores. If consistency scores drop below acceptable thresholds (e.g. .8 for small sample sizes), then the QCA model is dropped as nonsensical and lacking utility. As such, the threat of contradictions threatening the legitimacy of such a large dataset are very real and could threaten the overall model. Snap Shot on How QCA Works: From Cases to Paths In QCA, cases are represented as configurations of conditions. Conditions are characteristics of cases that are deemed relevant to the explanation of how the outcome occurred. In our case, such characteristics could be geographic location, household composition, or social services received. For each household surveyed, the respondents indicated if the service was provided or not and for how long. To make this decision consistently across cases, a threshold is defined for each condition. The threshold for receiving a service was, e.g. Parent Counseling, defined as having received the respective service at least once in the last six months. If the household did receive the respective service, the condition was designated as present in this instance. Thus, each household is characterized by a combination of conditions that are either present or absent. Households that are characterized by the same configuration of present and absent conditions are treated as instances of the same logical combination. It is crucial for the analysis that the majority of cases with the same logical combination of conditions displays the same outcome. That is, the conditions included in the analysis must enable us to discern between cases where the outcome is present and those where it is not. In the present analysis, we want to differentiate between households that reduced their vulnerability and those that did not, based on the services they received. If all cases with the same combination of conditions show the same outcome, their configuration is called perfectly consistent. QCA applies an algorithm to consistent configurations to decide which of these configurations (or parts thereof) provide sufficient explanations for the outcome. A combination is called sufficient if whenever it is present, the outcome is present, too. Such sufficient explanations are called paths to the outcome. 42 To mitigate the impact of contradictions, we adjust the consistency threshold for the inclusion of configurations in the logical minimization process to “usually consistent”, or 65% of cases showing the outcome under study (Ragin 2000, p. 109ff.). In fact, QCA requires full datasets and cannot function with missing data items. To manage this, the datasets from both Wave 1 (in 2016) and Wave 2 (in 2017) were merged to create 2,201 households. Of these, 69 HH needed to be dropped due to missing data, yielding a total of 2,132 households that were sampled consistently and completely across both waves. As far as we know, only one other study has applied QCA to such a large sample size.27 Moreover, this large number of cases and their random sampling affords us with a unique opportunity as well. As a qualitative method typically applied to a small number of cases, the conclusions drawn by QCA are often limited to the cases under study. With the present analysis, however, we are able to identify paths to reduced vulnerability that have bearing for the study population as a whole. To facilitate generalizability, we have included only those configurations of conditions in the logical minimization process that significantly pass our consistency threshold of 0.65. Statistical significance was assessed by means of binominal probability tests (Greckhamer et al. 2008, p. 715).28 In what follows, we describe how we modelled the changes in vulnerability outcomes from 2016 to 2017, how we modelled a wide array of household and community services over the same time period, what were the results and the identified corresponding service pathways out of vulnerability, and the potential programmatic implications for Sustainable Outcomes moving forward (presented as recommendations). Modeling the Outcome QCA requires us to turn case characteristics into conditions and define their presence and absence. This applies to outcomes as well and for this analysis, the primary outcome measure is socioeconomic household vulnerability status captured through the HVAT. As mentioned before HVAT measures household vulnerability across economic, food security, health/WASH, education, psychosocial support, and child protection dimensions, and yields a categorical score of no vulnerability to critical vulnerability. Households can be categorized as “slightly” (25% to 49% of the maximum score), “moderately” (50-74%), and “critically” (75-100%) vulnerable. For the QCA model, we moved away from static, “point” measures of HH vulnerability that only examine vulnerability status through single snapshots in time. Instead, we were in a position to look at how vulnerability changes over time and therefore used the difference in vulnerability scores from Wave 1 to Wave 2 as the main outcome to be modeled. Percentage changes allow for a more consistent assessment of what constitutes comparable changes across the spectrum of household vulnerability. Conversely, point changes in raw vulnerability scores would bias the analysis towards higher scores of vulnerability. This is because, all else being equal, it is more likely for high-scoring households to reduce their vulnerability by one point than for low-scoring ones. We also decided against category changes in vulnerability as an outcome measure because these changes would denote an improvement in vulnerability of anywhere between 1 and 17 points. It should be noted that in Wave 1 we were not aware of the HVAT scaling issues and did not collect the additional vulnerability data as we did in Wave 2. As a result, the QCA analysis we discuss here uses the original HVAT raw score adding system so we can compare vulnerability scores across both waves. 27 See Greckhamer, T., Misangyi, V. F., Elms, H., & Lacey, R. (2008). Using qualitative comparative analysis in strategic management research: An examination of combinations of industry, corporate, and business-unit effects. Organizational Research Methods, 11(4), 695-726. 28For each configuration, we calculated the probability that the consistency score would be observed if the underlying relation was random. If this probability was less than 5% and the consistency threshold of .65 was passed, the configuration was included. For configurations that cover less than five cases, even a perfect consistency score could be observed with a probability higher than 5%. Such configurations were therefore excluded. 43 Examining the changes in vulnerability from 2016 to 2017 generated a new distribution of changes to household vulnerability categories based not on their static scores but instead on their level and direction of changes: Distribution of Vulnerability Score Changes from 2016 to 2017 Strongly less vulnerable (Minimum of 25% decrease) 586 Mildly less vulnerable (Minimum of 10% decrease) 434 Stable (HVAT score change between -10% and +10%) 450 Mildly more vulnerable (Minimum of 25% increase) 257 Strongly more vulnerable (Minimum of 25% increase) 404 Total 2131 Changes in vulnerability score are thus divided into five different types of outcomes. These are: “Mildly less vulnerable” (in which there was a minimum of 10% decrease in the HVAT vulnerability score between wave 1 and wave 2), “Strongly less vulnerable” (minimum of 25% decrease in the HVAT vulnerability score between wave 1 and wave 2), “Mildly more vulnerable” (in which there was a minimum of 10% increase in the HVAT vulnerability score between wave 1 and wave 2), and “Strongly more vulnerable” (minimum of 25% increase in the HVAT vulnerability score between wave 1 and wave 2). Of course, a fifth category also emerged, “No change in vulnerability status” (in which increases or decreases in vulnerability score were less than 10%). With these “change categories” now in place, we re-incorporated the original HVAT category distributions according to how they are distributed across the change categories. This distribution is presented below. (Note: there were 450 static HHs that had no meaningful change in score from 2016 to 2017, and thus were assigned to the “absent” categories of each type of outcome. As a result, non-moving HH’s appear across all “absent” categories. Moreover, all “strongly” characterized categories are subsets of their “Mildly” counterparts): HVAT Category Mildly less vulnerable (min. 10%) Strongly less vulnerable (min. 25%) Present Absent Present Absent Not vulnerable 7 (6%) 111 (94%) 4 (3%) 114 (97%) Slightly vulnerable 638 (41%) 917 (59%) 336 (22%) 1219 (78%) Moderately vulnerable 375 (82%) 83 (18%) 246 (54%) 212 (46%) Critically vulnerable 0 0 0 0 Total 1020 (48%) 1111 (52%) 586 (27%) 1545 (73%) HVAT Category Mildly more vulnerable (min. 10%) Strongly more vulnerable (min. 25%) Present Absent Present Absent Not vulnerable 104 (88%) 14 (12%) 94 (80%) 24 (20%) Slightly vulnerable 539 (35%) 1016 (65%) 306 (20%) 1249 (80%) Moderately vulnerable 18 (4%) 440 (96%) 4 (1%) 454 (99%) Critically vulnerable 0 0 0 0 Total 661 (31%) 1470 (69%) 404 (19%) 1727 (81%) Once again, we saw no movement into Critical Vulnerability in Wave 1 or in Wave 2 using the traditional HVAT scoring method, as this QCA analysis relied upon the original HVAT scoring method to compare both waves. Assuming subsequent survey waves will use the refined approach, changes in HH vulnerability across waves can be easily conducted. As the distributions of the outcomes indicate, changes in vulnerability differ markedly between households of different vulnerability categories. Whereas among slightly vulnerable households, mild and strong increases in vulnerability occur with a similar frequency as in mild and strong decreases in vulnerability. Among moderately vulnerable households, decreases in HH vulnerability scores are substantially more frequent than increases, across both mild and strong changes. The point of highlighting these different levels of change across 44 vulnerability categories is to suggest that slightly and moderately vulnerable households are influenced by different external factors. Put different, the pathways out of vulnerability are not linear, but can qualitatively vary depending on where a HH falls across the vulnerability spectrum. Unpacking these causal dynamics then requires a causal analysis that differentiates between these two categories, rather than assuming they are both expressions of “more or less the same.” This will be discussed further in the results section, and to the discussion of how we modelled access to services we now turn. Modeling the Services HHs Received Over Time The survey data provides information on a total of 24 individual services on the household and community level. A breakdown of these services can be found in Annex 5. Fifteen of these services were household-based and nine community-based. If we were to include a condition for each service in the analysis, we would arrive at a total of 24² (=576) logically possible configurations of present and absent conditions for service provision alone. Not only would this probably go beyond the ability of the current QCA algorithm to process, but it would also lead to a high number of configurations that we could not validate empirically, both because similar services tend to co-occur across the community and household levels; thus, there would be too much “noise” in the model which couldn’t be filtered out with the above-mentioned binominal probability test (the noise would create too many discrete cases on the one hand, and the sample size of each discrete case would be too small to eliminate using statistical methods). We have therefore conducted a cluster analysis of the variables that represent the provision of individual services.29 We arrived at ten clusters of services that tend to co-occur across surveyed households. If at least one of the services of a cluster was provided to the household at least once in the past six months (captured between wave 1 and 2), the respective condition was coded as present. These were the resulting categories of services: • Direct cash transfers • Donations to household • Food security • Health and hygiene • HIV and GBV prevention • HIV testing (household level) • Savings groups • HIV testing (community level) • Parenthood-related services • Other household-based services We also compared service provision in Wave 1 with service provision in Wave 2 and constructed two additional conditions for each service cluster. If a household was recorded as receiving at least one service of a given cluster in Wave 1 and did not receive any services of that cluster in Wave 2, we created a variable known as “loss of access to services” was coded as present. In turn, if a household was recorded as receiving none of the services of a given cluster in wave 1, but did receive at least one service in Wave 2, the condition “gain of access to services” was coded as present. In fact, this led to three new conditions: gaining access, losing access, no change to access. In this way, we captured to what extent new services might play a role in changing a HH 29 We conducted an agglomerative hierarchical cluster analysis. This analysis is most often used to identify homogenous groups of cases in a sample. Step by step, it joins clusters that are most similar, starting with the two most similar cases and ending with one cluster that contains all cases. The larger the clusters, the more dissimilar are the cases in each cluster, and the more information about each case is lost when clusters are used to represent them. The loss of information at each step is expressed by an agglomeration coefficient. Jumps in this coefficient indicate which number of clusters to retain for further analysis. Here, we used cluster analysis to join variables instead of cases. Different linkage methods and distance metrics are available to decide which clusters to join at each step. We employed Ward’s linkage and squared Euclidean distance. Agglomeration coefficients are reproduced in Annex 4. 45 vulnerability status. However, as mentioned above under the “enrollment problem,” we do not know the reason for these changes in access to service, i.e. if gaining or losing access to a given service cluster should be interpreted as due to positive or negative shocks to the household, respectively (e.g. their situation improved so they didn’t need the service, their situation worsened so they now qualified for it, the service stopped being implemented, the service was recently added in the area, etc.). In addition to conditions pertaining to service provision, the number of children in the household was included as well. The condition was coded as present when the household had more than three children. The frequencies of all specified conditions are provided in Annex 5. Given the high volume of potential configurations and the high sample size, we also used correlational analysis to help identify various sets of conditions of particular relevance for reductions in household vulnerability.30 To determine these configurations, we tested the correlations of the number of children in the household, all 24 individual services, and the three measures (gaining access, losing access, no change to access) for each of the 10 service clusters with the four outcome measures “Slightly less vulnerable”, “Strongly less vulnerable”, “Slightly more vulnerable”, and “Strongly more vulnerable”. This analysis was repeated for the full set of vulnerable households, slightly vulnerable households, and moderately vulnerable households. To be included in the set of conditions of particular relevance, the respective condition had to significantly correlate with at least one of the outcome measures for at least one of the three different subsamples.31 This set formed the core of the starting model for the configurational analyses.32 The results of the correlational analysis are provided in Annex 7. The markedly different dynamics in vulnerability changes across categories of household vulnerability suggested that we build separate models testing how the various services influence the different types of vulnerability. Specifically, we conducted separate analyses of the causes of vulnerability change for a. all vulnerable households, b. slightly vulnerable households, and c. moderately vulnerable households. We therefore conducted 24 qualitative-comparative analyses in total. Eight were conducted with the full sample of vulnerable households. They included a condition that differentiated between slightly and moderately vulnerable households. Eight were also conducted for slightly vulnerable households, and eight for moderately vulnerable households. Each analysis focused on the presence or absence of one of the four outcome measures. The limits of the models are discussed first. The QCA modelling showed that services at the household and community level offer paths to mildly reduced vulnerability (10% or less) to households that are class as “moderately vulnerable” by the HVAT. Other changes in vulnerability among moderately vulnerable households could not be sufficiently explained by the services that they receive.33 For slightly vulnerable households, unfortunately none of the changes in household vulnerability could be positively determined by patterns in service provision. What this means is that the QCA model could not identify pathways consistent enough to be considered useful for slightly vulnerable households. It was also not possible to explain the presence of vulnerability changes across categories of household vulnerability in joint analyses of all vulnerable households. As 77% of vulnerable households are slightly vulnerable, the results of the joint analysis are a confirmation that vulnerability changes among slightly 30 While correlation analysis follows a different logic than the logic of QCA causal asymmetry, a perfect correlation does imply a perfect set relation (consistency score). By identifying conditions that are particularly promising (having high correlation scores), a correlation analysis can therefore be a good starting point for an analysis of configurations. The reverse is not true (i.e. . 31 We used Phi coefficients to assess statistical significance at p≤0.05. 32 If both an individual service and a cluster that contained the service entered the set, clusters were given preference. Likewise, gaining access or losing access to a cluster took preference over having access to a service cluster at wave 2. 33 In cases where a configurational analysis did not lead to a sufficient explanation, one of two situations occurred: (1) The available information on households and the services provided to them could not clearly differentiate between cases that showed the outcome and those that did not. This means that it was not possible to identify any combinations of conditions that passed the consistency threshold. Thus, none could be subjected to QCA’s minimization algorithm. (2) Taken together, the paths identified by the algorithm did not provide a sufficiently consistent explanation with a meaningful coverage of cases (those pathways were exceptions and very low in sample size. In QCA technical terms, they yielded a so-called “solution” with consistency and coverage significantly below 0.8). 46 vulnerable households are unrelated to the services they receive. Taken together, these (non-)results suggest that household- and community-based services, such as counseling and savings groups, may facilitate reductions in vulnerability among households with pronounced vulnerabilities, but are less adequate to support households in graduating out of vulnerability.34 34 Additional information on the context of households, such as regional developments or changes in their communities, would likely improve the analyses and help identify conducive and adverse contexts under which services can or cannot make a difference. Most of the information on household characteristics that was available for the configurational analyses was part of the HVAT vulnerability score and thus could not be employed to explain changes in household vulnerability. 47 Conclusions and Recommendations The second wave of the baseline survey generated some interesting findings and has laid the groundwork for a new, more advanced approach to determining impact by combining both indirect methods of attribution (through a Difference-in-Difference quasi-experimental design) and direct methods of contribution (through an expanded application of Qualitative Comparative Analysis to much larger datasets). Along the way, the analysis has greatly improved the validity and accuracy of the HVAT tool, yielding a much more appropriate weighting scale of HVAT items and useful vulnerability categorization system. As such, the following recommendations are offered by the SoCha team for USAID/Uganda and Sustainable Outcomes staff to consider taking forward: • USAID/Uganda and Sustainable Outcomes should strongly consider enrolling the surveyed treatment households to receive treatment so that the impact evaluation can be conducted and its learning questions addressed. Furthermore, the surveyed control group households should also be enrolled at a later date so as to avoid ethical issues and not harm the implementer’s legitimacy and reputation. • Sustainable Outcomes should increase their quality control of the SOCY MIS implementation database, examine and revise their current MIS implementation protocols, provide new and more comprehensive user training, regularly monitor and clean the MIS as appropriate, and address the technical and design issues identified above, especially the lack of a unique HH identifier for each household. • USAID/Uganda and Sustainable Outcomes should review the Control/Treatment group allocations in lieu of the recent change to implementation strategy and determine if reallocation of HH to regional boundaries would add value to the study’s learning questions. • USAID/Uganda and Sustainable Outcomes and other HVAT users consider a workshop to discuss a potential revised HVAT using these results as the basis for a more refined HVAT applied across the country. • Sustainable Outcomes and USAID/Uganda staff should consider conducting additional formative and exploratory analyses on some of key mediators channeling vulnerability in different ways. This type of analyses should be results driven in that the findings can be directly used to inform more targeting OVC programming taken to scale. • USAID/Uganda and Sustainable Outcomes should consider encouraging greater participation in a savings group (hopefully SILC) combined with encouraging additional access to Cash Transfers and ongoing access to HIV and GBV prevention services as one viable pathway out of vulnerability for moderately vulnerable households that house four or more youth/children. • USAID/Uganda and Sustainable Outcomes may also consider whether or not membership in savings group by HHs with less than four youth/children may entail financial or other types of burdens that overcome the benefits of savings group membership; thus raising the possibility that withdrawal from savings groups by these types of HH may reduce their vulnerability status. • USAID/Uganda and Sustainable Outcomes may wish to consider how ongoing access to HIV and GBV prevention services, combined with regular HIV community-testing services and, perhaps most importantly, with ongoing access to steady food support may form a viable pathway out of vulnerability, even without joining a savings group (this proposition could of course also be rigorously tested by reinstating the original dual treatment group design identified in 2016). • USAID/Uganda and Sustainable Outcomes should strongly consider enrolling the households surveyed across both waves into the program for services so that a rigorous quasi-experimental design impact evaluation can still be completed and a rigorous QCA can further unpack what combinations of services can lead to pathways out of vulnerability. 48 Annex 1: Scope of Work SCOPE OF WORK: BASELINE EVALUATION OF SUSTAINABLE OUTCOMES FOR CHILDREN AND YOUTH PURPOSE OF THE EVALUATION Uganda is currently the second youngest population in the world and the third fastest growing nation in Africa. However, deep and extensive vulnerabilities exist that especially affect children: 96% of children are considered vulnerable, and 62% of those living in poverty are children. Vulnerabilities take different forms, including poverty, malnutrition, and exposure to violence. Low levels of education and high prevalence of HIV/Aids among children themselves as well as within their families exacerbate these vulnerabilities. USAID/Uganda’s Orphans and Vulnerable Children (OVC) programming employs an integrated approach to address these factors. The Sustainable Outcomes for Children and Youth (Sustainable Outcomes) program aims to economically empower children, youth, and their caregivers to access core services, strengthen systems to provide core services, and improve coordination of community-based clinical and socio-economic services for efficiency and effectiveness along the continuum of care. USAID/Uganda wants to a) establish a baseline regarding key indicators and b) establish procedures that may be used for future monitoring and evaluation of OVC programs using Qualitative Comparative Analysis (QCA). BACKGROUND Program Summary Activity Name Sustainable Outcomes for Children and Youth (Sustainable Outcomes) Implementer Catholic Relief Services (CRS) Cooperative Agreement # AID-617-A-15-00005 Total Estimated Ceiling of the Evaluated Activity (TEC) $40,526,948 Life of Activity April 2015 – April 2020 Active Geographic Regions 17 districts in Central and Western Uganda USAID Office Education, Youth, and Childhood Development Sustainable Outcomes for Children and Youth is implemented by Catholic Relief Services (CRS) in partnership with a consortium of organizations, including Action for Community Development (ACODEV), African Network for Prevention and Protection against Child Abuse and Neglect (ANPPCAN), TPO Uganda, and Palladium (formerly Futures Group). 49 Figure 1: Sustainable Outcomes Results Framework 50 Development Hypothesis and Theory of Change Sustainable Outcomes for Children and Youth aims to address multiple forms of vulnerability among children and youth through an integrated, comprehensive approach (Figure 1). Specifically, it will economically empower orphaned and vulnerable children, youth, and their caregivers to access core services (Result 1), strengthen local government, CSOs, and informal community structures to increase and improve core services for orphaned and vulnerable children, youth, and their caregivers (Result 2), and improve coordination of community-based clinical and socio-economic services for efficiency and effectiveness along the continuum of care (Result 3). Sustainable Outcomes’ Development Hypothesis states that if local governments and CSOs increase and improve core services for OVC, youth and their caregivers, and community-based clinical and socio-economic services become more responsive and collaborate more effectively, then an enhanced, social support system will develop through which OVC, youth, and households will access core services, for improving health, nutrition, education, and psychosocial well-being and reducing abuse, exploitation and neglect. Overview of Interventions Households are enrolled into the program on the basis of certain criteria of vulnerability, which are assessed using the Ministry of Gender, Labour and Social Development-approved national tool: the Household Orphans and Vulnerable Children Vulnerability Prioritization Tool (HVPT). Alternatively, households are enrolled via referral from health facilities. Information from the vulnerability assessment then also serves as the basis for a tailored household action plan for referrals and interventions. Parasocial workers are trained to supervise the implementation of household action plans. Households are expected to graduate out of vulnerability within one or two years, but may be supported for up to three years. Depending on the circumstances of the household, participation in a savings and internal lending communities (SILC) savings group may be offered to them. SILC groups meet weekly and these meetings also provide a platform for financial education and training in parenting and communications within families, regarding for example gender roles and how to make joint decisions. Sustainable Outcomes also works with private sector to link households to markets and internship or apprenticeship opportunities. Sustainable Outcomes trains SILC agents for 12 – 18 months, after which SILC groups are expected to be self-sustaining. Households that don’t have the means to participate in a SILC group are linked to cash transfer opportunities. System strengthening interventions include support to district and sub-county OVC coordination committees (DOVCCs and SOVCCs) to develop operational plans, undertake support supervision, monitor and assess OVC program performance, carry out quality assessments and support supervision, data analysis, dissemination and advocacy. In addition, system strengthening interventions focus on optimal utilization of available resources for OVC services within local government, CSOs and informal community structures, tracking staffing levels and training Para-social workers, and support for the use of data. Interventions also aim to improve the functioning of the referral network, increase utilization of core services, and expand the case management approach to build long term sustainability. Through enhanced coordination, user-friendly directories, effective follow-up mechanisms and improved service provider capacities, OVC and caregivers who access one key core service will be linked to multiple core services that pave the way to sustained health outcomes and graduation from social assistance. Current Status of Implementation Sustainable Outcomes is intended to operate in 17 districts in Western and Central Uganda. These districts were identified by the United States Agency for International Development (USAID) due to high prevalence rates of HIV and availability of trained community workers to deliver program services. System strengthening interventions under Result 2 have been rolled out in all target communities since October 2015. Household level interventions (Result 1 and Result 3) are being phased in according to the following plan: • Year 1 (since May 2016): 25% of communities • Year 2 (planned for October 2016): 40% of communities • Year 3 (planned for October 2017): 35% of communities Critical assumptions are that there is sufficient and quality of health and education services for the OVCs to access services and in addition, other Implementing Partners and USAID supported IPs to provide HIV/AIDS services to beneficiaries, and that local governments have adequate staff to support supervise and ensure quality of services to vulnerable children by service providers. Sustainable Outcomes Activity Monitoring, Evaluation and Learning Plan The evaluator will review information available through Sustainable Outcomes’ own data collection efforts, including vulnerability assessments and AMELP data, as well as other existing data sources, and utilize available data where 51 possible. Sustainable Outcomes has undertaken a baseline survey in all three regions of the program (Central, South￾Western and Western) in a random selection of the 17 project districts to establish pre-intervention conditions to inform the development of interventions and provide a basis for project monitoring and outcome monitoring indicators. The survey focuses on obtaining information on (1) characteristics of the households (economic status; clinical, community and social protection services received); (2) characteristics of the selected index child (caregiver’s knowledge of child’s HIV status, possession of a birth certificate for child, child’s school attendance, services currently receiving); and (3) attitudes of the caregiver towards physical punishment. EVALUATION QUESTIONS, DESIGN, AND METHODOLOGY With this Scope of Work (SOW), USAID/Uganda seeks to establish a baseline regarding key program indicators. Program outcome indicators of interest are linked to the three areas of interventions and are to be refined in collaboration between USAID/Uganda, Sustainable Outcomes, and the evaluator: • Result 1: financial self-efficacy in terms of a household's ability to meet basic needs of children in their care, including nutrition needs, school attendance, children's psychosocial wellbeing, as well as economic resilience to shocks, such as unexpected medical expenses and others. • Result 2: adequate functioning and/or efficiency of formal and informal systems and reporting, for example through case follow-up and capacity of probation officers. • Result 3: functioning of case management and referral systems. As stated above, the evaluator shall review and utilize existing data sources to the extent possible. However, the evaluator is also expected to develop survey instruments to address data needs not covered by existing sources. Data should be collected from two groups of beneficiaries: those who receive system strengthening only during year 2 and those who also receive household level interventions (see section 2). Data collection is expected to take place before program interventions begin in cohort 2, so that a clean baseline for cohort 2 can be established, but may include other cohorts to be determined in collaboration between USAID/Uganda, Sustainable Outcomes, and the evaluator. Data should be disaggregated by sex, disability status, single- vs. two-parent household, district, and possibly other beneficiary characteristics to be determined in collaboration between USAID/Uganda, Sustainable Outcomes, and the evaluator. Sample size calculations must take the disaggregation into account. Selection of indicators, instruments, and methods must be suitable to enable the use of Qualitative Comparative Analysis (QCA). QCA is a method to identify necessary and sufficient factors that can combine in various ways to produce an outcome. In short, QCA defines desired outcomes (such as for example school enrollment) and analyzes the presence or absence of various factors (such as specific interventions) in conjunction with occurrence of the outcome. The method is superior to more traditional evaluation methods for evaluating a program with highly integrated interventions such as Sustainable Outcomes, because it allows identifies the contribution of each intervention as well as combinations of interventions. The evaluator will establish and document procedures that can be used for future monitoring and evaluation of Sustainable Outcomes and other OVC programs using QCA. The procedures developed under this SOW may be used to adapt Sustainable Outcomes’ own data collection activities as well as future evaluations of this and similar programs. This work will help USAID/Uganda in the long term to better answer overarching questions regarding the effectiveness of OVC programs, such as: • To what extent do program outcomes improve over time? • To what extent do household level interventions (as opposed to system strengthening alone) contribute to these changes? • How do program interventions affect change? • How cost effective are different pathways to change? • Sustainability: To what extent are processes being put in place to facilitate long-term success of program interventions beyond the program’s end? Which factors exist that may prevent long-term success? The ability to answer these questions will help USAID/Uganda understand to what extent the full set of program interventions has a positive effect on children and youth wellbeing, and how different interventions or combinations of 52 interventions of the program contribute to program outcomes. This information will be crucial to inform decisions about future OVC programs. DELIVERABLES UNDER THIS SOW 1. Inception meeting: Within 1 week of award, the evaluation team will meet with USAID/Uganda to discuss the team’s understanding of the assignment, initial assumptions, evaluation questions, design, methodology, and work plan, and/or to adjust the Statement of Work (SOW), if necessary. 2. Concept note and work plan: Within 2 weeks of award but no later than July 29, 2016, the evaluation team must submit to the Agreement Officer’s Representative/Contracting Officer’s Representative (AOR/COR) a concept note and work plan, which will include: (1) a set of proposed indicators that will be measured, including indicator definitions and rationale for selecting them linking back to the evaluation purpose, (2) a detailed data collection plan that explains how indicators are going to be measured and how the data collection plan enables use of QCA; (2) draft questionnaires and other data collection instruments or their main features; (3) the list of potential interviewees and sites to be visited and proposed selection criteria and/or sampling plan (must include calculations and a justification of sample size, plans as to how the sampling frame will be developed, and the sampling methodology); (4) known limitations to the design, (5) the anticipated schedule and logistical arrangements; and (6) a list of the members of the evaluation team, delineated by roles and responsibilities, and (7) level of effort and cost information. USAID offices and relevant stakeholders are asked to take up to 5 business days to review and consolidate comments through the AOR/COR (by August 5, 2016). Once the evaluation team receives the consolidated comments on the initial concept note and work plan, they are expected to return a revised concept note and work plan within 5 business days (August 12, 2016). 3. Protocol for IRB submission: Within 4 weeks of award but no later than August 12, 2016, the evaluation team must submit to the Agreement Officer’s Representative/Contracting Officer’s Representative (AOR/COR) a data collection protocol ready for submission to the appropriate IRB and initiate the IRB process, expected to take up to one month. 4. Data collection: The evaluation team must complete all necessary preparations, such as enumerator recruitment and training and translations into local language, as well as the data collection according to the approved protocol no later than by the end of September 2016 given the anticipated enrollment of cohort 2 into the program in October 2016. 5. Draft baseline report: The draft evaluation report should be consistent with the guidance provided in Section IX: Final report format. The submission date for the draft evaluation report will be determined in the evaluation work plan. Once the initial draft evaluation report is submitted, USAID/Uganda will have 10 business days in which to review and comment on the initial draft, after which point the AOR/COR will submit the consolidated comments to the evaluation team. The evaluation team will then be asked to submit a revised final draft report 10 business days hence, and again USAID/Uganda will review and send comments on this final draft report within 10 business days of its submission. 6. Final baseline report: The evaluation team will be asked to take no more than 10 business days to respond/incorporate the final comments from USAID/Uganda. The evaluation team leader will then submit the final report to the AOR/COR. All data and records will be submitted in full and should be in electronic form in easily readable format in line with all applicable provisions regarding privacy and protection of participants, organized and documented for use by those not fully familiar with the project or evaluation, and owned by USAID. 7. Final presentation: The evaluation team is expected to hold a final presentation in person/by virtual conferencing software to discuss the summary of findings and recommendations to USAID. 53 EVALUATION TEAM COMPOSITION The evaluation team should consist of a team leader with expertise in evaluations. The evaluation team should further demonstrate experience in M&E of OVC programs, knowledge of the Ugandan context, as well as strong skills in qualitative and quantitative research, including survey design and analysis, and the ability to oversee any necessary translation of survey tools into local language. The evaluation team must include a specialist in the QCA method. A point of contact from USAID/Uganda will work closely with the evaluation team and may observe some of the data collection efforts. FINAL REPORT FORMAT The final baseline report should include an executive summary; introduction; background of the local context and the program being evaluated; the main evaluation purpose; the methodology or methodologies; the limitations to the design; findings, conclusions, and recommendations; and lessons learned (if applicable). The report should focus on a) presenting baseline findings, and b) explaining how the procedures used can inform the design of future monitoring and evaluation activities using QCA. The executive summary should be not more than 3 pages in length and summarize the purpose, background of the project being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations and lessons learned (if applicable). The baseline data collection methodology shall be explained in the report in detail. Limitations shall be disclosed in the report, with particular attention to the limitations associated with the methodology (e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.). The annexes to the report shall include: • The Evaluation SOW; • Any statements of difference regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team; • All tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides; • Sources of information, properly identified and listed; and • Disclosure of conflict of interest forms for all evaluation team members, either attesting to a lack of conflicts of interest or describing existing conflicts of. CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT Per the USAID Evaluation Policy and USAID ADS 203, draft and final evaluation reports will be evaluated against the following criteria, as applicable, to ensure the quality of the evaluation report. 35 • The evaluation report should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate what worked in the project, what did not, and why. • Evaluation reports shall address all evaluation questions included in the SOW. • The evaluation report should include the SOW as an annex. All modifications to the SOW—whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline—need to be agreed upon in writing by the AOR/COR. • The evaluation methodology shall be explained in detail. All tools used in conducting the evaluation—such as questionnaires, checklists, and discussion guides—will be included in an annex in the final report. • Evaluation findings will assess outcomes and impact on males and females. • Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). 35 See Appendix I of the Evaluation Policy and the Evaluation Report Review Checklist from the Evaluation Toolkit for additional guidance. 54 • Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or the compilation of people’s opinions. Findings should be specific, concise, and supported by strong quantitative or qualitative evidence. • Sources of information need to be properly identified and listed in an annex. • Recommendations need to be supported by a specific set of findings. • Recommendations should be action-oriented, practical, and specific, with defined responsibility for the action. OTHER REQUIREMENTS All quantitative data collected by the evaluation team must be provided in machine-readable, non-proprietary formats as required by USAID’s Open Data policy (see ADS 579). The data should be organized and fully documented for use by those not fully familiar with the project or the evaluation. USAID will retain ownership of the survey and all datasets developed. All modifications to the required elements of the SOW of the contract/agreement, whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline, need to be agreed upon in writing by the COR. Any revisions should be updated in the SOW that is included as an annex to the Evaluation Report. 55 LIST OF ANNEXES • ANNEX A: SUSTAINABLE OUTCOMES DISTRICTS • ANNEX B: SUSTAINABLE OUTCOMES BASELINE INDICATORS • ANNEX C: ESSENTIAL PEPFAR INDICATORS FOR OUTCOME MONITORING ANNEX A: SUSTAINABLE OUTCOMES DISTRICTS Bushenyi Gombe Hoima Isingiro Kabale Kampala Kamwenge Kanungu Kasese Kibaale Kiruhura Kyenjojo Luweero Mityana Ntungamo Rukungiri Wakiso ANNEX B: SUSTAINABLE OUTCOMES BASELINE INDICATORS 1 DSD OVC_SERV: Number of active beneficiaries receiving support from PEPFAR OVC programs to access HIV services 2 OVC_MONEY: Percent of households able to access money to pay for unexpected household expenses 3 Number of OVC and youth whose households were linked to social protection assistance 4 Number of project-supported savings groups 5 Number of girls supported with financial support for further education 6 Number of youth participating in agro-enterprise training 7 OVC_CP: Percent of caregivers who agree that harsh physical punishment is an appropriate means of discipline or control in the home or school 8 OVC_BCERT: Percent of children who have a birth certificate 9 Proportion of districts with functional district OVC coordination mechanisms 10 Proportion of annual local government resource allocation for children and youth from local revenues sources and unconditional grants 11 Proportion of districts with 80% of key positions substantively filled 12 Proportion of districts using OVCMIS for planning 13 Number of district alternative care panels supported 14 OVC_HIVST Proportion of beneficiary children whose primary caregiver knows the child’s HIV status 56 15 Proportion of referred OVC, youth and caregivers that complete a referral 16 Number of very young children reached with ECD services 17 DSD OVC_ACC: Number of active beneficiaries receiving support from PEPFAR OVC programs to access HIV services ANNEX C: ESSENTIAL PEPFAR INDICATORS FOR OUTCOME MONITORING Annex 1a: Changes to the Scope of Work Quick Summary The initial design of this impact evaluation occurred in August 2016, and baseline data collection occurred shortly after in November and December 2016. Subsequently, a midline data collection exercise was scheduled to occur in November and December 2017. Data collection for the midline proceeded as planned at the end of 2017, but subsequent information changed its overall intention. Specifically, upon completing the second survey wave, SoCha learned that Sustainable Outcomes did not enroll the original households surveyed under the baseline. Hereafter, what was to be referred to as “midline” is referred to as “extended baseline.” Background 57 As per SoCha’s protocol, our evaluation team leader contacted Sustainable Outcome staff and USAID/Uganda Mission staff prior to the launch of what was previously called the midline survey. Sustainable Outcome staff were occupied with field visits and other reporting requirements; and could not attend. Nor were Sustainable Outcome staff available during the survey’s implementation. As a result, there was no official updating of implementation progress or changes to the design during survey implementation communicated to SoCha. The evaluation team leader returned to Uganda to meet with Sustainable Outcome’s staff in January 2018 to collect implementation data from their newly-created implementation MIS database as well as discuss how the previously enrolled households under the baseline could be identified. During this visit, the evaluation team leader learned that the HH identified in the baseline treatment groups had not been enrolled in Sustainable Outcomes and had not received any Sustainable Outcomes services at any point. As such, none of these households appeared in the new implementation MIS, and the evaluation team ran a fuzzy matching analysis of all the households in the MIS to confirm this was true (it was).36 The immediate result of these updates was that neither the planned QCA section nor the midline analysis of the three comparison groups were possible. As mentioned, the second survey – the midline – now becomes a second wave of the original baseline – referred to as the “extended baseline” - as the households remain relatively “uncontaminated” by the treatment. More importantly, the assumptions underpinning the Difference-in-Difference Design will be significantly strengthened. Although it may appear that conducting two waves of baseline survey of the same households is an inefficient allocation of resources from a programmatic perspective, in fact a “double baseline” approach strengthens the ability of the Difference in Difference design to demonstrate the current trends of vulnerability running across control and treatment groups (i.e. the parallel trends assumption). In practice, few quasi-experimental Difference in Difference designs achieve this level of rigor and evidence to justify their assumptions, thus placing the Sustainable Outcomes evaluation in a far higher class of analysis than similar designs. Hereafter what was previously referred to as the midline survey will now be referred to as the extended baseline. Other Changes The evaluation team also learned of an additional significant change: All households enrolled in the previous cohorts are now eligible to receive BOTH R1 and R3 services; thus eliminating the need to have an additional treatment arm in the impact evaluation that was originally planned. Moreover, while searching for implementation data that could be used in the current analysis, the evaluation team reviewed Sustainable Outcome’s Implementation MIS and other datasets. In searching for a solution, the team also discovered that there was no straightforward way to merge the tables across each dataset to provide a comprehensive picture of each HH’s vulnerability scores, household dynamics and services received. Further attempts to link Cohort 1 Assessment and SOCY services data were unsuccessful. The immediate result of these updates was that neither the planned QCA section nor the midline analysis of the three comparison groups were possible, as the households remain relatively “uncontaminated” by the treatment. The original impact evaluation design poses three different impact evaluation questions: • Question 1: Does caregiver participation in activities under Results 1 and 3 reduce household vulnerability and improve OVC well-being? • Question 2: Does caregiver participation in activities under Results 1 reduce household vulnerability and improve OVC well-being? • Question 3: Do the combined activities under Results 1, 2 and 3 reduce rates of HIV prevalence and incidence in participating sub-counties? For Question 1, households and OVC/Youth in Cohort 2 sub-counties that receive both R1 and R3 treatments (treatment group 1) were to be compared to households and OVC/Youth in Cohort 3 sub-counties (the control group) who do not receive any treatment until the end of 2017 and 2018. However, for Question 2, Sustainable Outcomes 36 Fuzzy matching was necessary as the MIS as of yet has not standardized its HH identification protocol, and has not undertaken data cleaning or quality control of the entries. As a result, there are a substantial number of user entry errors in the MIS, some of which are easily fixed (such as a misspelling of hyphens with dashes, which can be easily fixed with a “Find/Replace” function), while others are more difficult (multiple spellings of the same name). Fuzzy matching was initially set at 80% but returned no results. Six HH were returned when the threshold was set at 60% but further scrutiny revealed these were not matched. We therefore confidently concluded that no households from the samples have been enrolled in Sustainable Outcomes. 58 plans to eliminate the distinctions between an R1+R3 group and an R1 only group by offering R3 services to all beneficiaries. The result, to the extent it is fully implemented, eliminates Question 2. However, the implications for the Difference in Difference design are that this new “extended baseline” approach strengthens its ability to demonstrate the current trends of vulnerability running across control and treatment groups (i.e. using “trend lines” to test the parallel trends assumption). Moreover, the impact evaluation’s original power analysis was conducted under the assumption of three groups (one control and two treatments). Should Sustainable Outcomes return to its earlier two treatment arm design (e.g. due to a change in strategy or delays in implementation), the multiple treatment arm analysis can continue using the current datasets. Should the distinction between the two treatment arms vanish, the result will be a larger treatment group to compare to the control group; i.e. the statistical power of the impact evaluation is increased, especially because the need to adjust for the Bonferroni effect will no longer be necessary. In regards to the QCA analysis, the two waves of baseline survey data do contain additional data points that could be modeled using QCA to identify any “endogenous” changes to HH vulnerability that have occurred independent of the Sustainable Outcomes intervention. Examining the change in volume and composition of HH access to relevant services allows an analysis of how these services may or may not be associated with varying levels of HH vulnerability. As a result, the scope has been changed to: • Improve the “trend lines” assumption for subsequent testing in the future midline and endline; • Eliminate Question 2 under the Impact Evaluation Design • Conduct QCA analysis using two waves of HH access to service data to explore any potential impacts on HH vulnerability, and • Improve and revise the measurement of household vulnerability to yield a more accurate weighting system. 59 Annex 2: Survey Instrument Household Caregiver Oral Questionnaire Identifier Page DISTRICT: SUBCOUNTY: PARISH: VILLAGE: NAME OF INTERVIEWER: DATE OF INTERVIEW: NAME OF INTERVIEWEE: HOUSEHOLD CODE: GPS: Latitude: S __ __.__ __ __ __º Longitude: E __ __.__ __ __ __º Start Time: End Time: Questionnaire Serial Number: INTERVIEW LOG VISIT 1 VISIT 2 VISITI 3 DATE (day/month/year) INTERVIEWER COMMENTS Interview comment codes: Interview completed 1; Appointment made for later today 2; Appointment made for another day 3; Refused to continue and no appointment made 4; Other (Specify) 5 101 102 103 104 105 106 Line Please give the names of persons who usually live in household, starting with head of household. If (NAME of Child), what is the age? less than 1 year=0 If (NAME of child) what is the gender? (Female=1, Male=2) Does (NAME of Child) usually live here? (Y/N) Do you usually cares for/looks after (NAME of Child)? (Yes=1, Other=Give Name, Selfcare=0) If care giver, what is your relationship to (NAME of child)? (see codes below) A B C D E F G H I J K L M N CODES FOR Q105: RELATIONSHIP TO RESPONDENT 01 = BIOLOGICAL MOTHER 03 = NON-BIOLOGICAL PARENT 05 = AUNT/UNCLE 02 = BIOLOGICAL FATHER 04 = SIBLING 06 = GRANDPARENT 60 Household Vulnerability Module ECONOMIC STRENGTHENING 1. Is this a child headed household? Yes No 2. Does the household head, spouse or guardian have any form of severe disability (e.g., physical, speech, visual, hearing, or mental handicap?)? Yes No 3. Does the household have ANY member who has been very sick for at least three months during the past 12 months? (By very sick, I mean that the household head or any member was too sick to work or do normal activities around the house for at least three of the past 12 months) Yes No 4. Is there at least one member of the household who currently has formal or informal employment, is self-employed, has a business, or is engaged in an economically productive activity? Yes No 5. Who is the main contributor to household income? A Child Grandparent(s) Parent(s) Relatives/ Others 6. What is the current monthly HH income? (express in Uganda Shillings) <50k 50k – 99k 100k – 149k 150k-200k >200k 7. What is the main source of household income? Formal Job/ Business Informal Job Casual Labour Remittance/ Others 8. The last time there was an unexpected urgent household expense (e.g. emergency medical expense or house repair), HH was able to pay that expense? Yes No 9. If the household incurred any HEALTH-related expenses in the past three months, was it able to pay for these expenses? Yes Not always No 10. If the household incurred any SCHOOL-related expenses in the past three months, was it able to pay for these expenses? Yes Not always No 11. If the household incurred any FOOD-related expenses in the past three months, was it able to pay for these expenses? Yes Not always No 12. Does anybody in the HH belong to any financial savings and lending group? Yes No 13. Any member of the HH owns an electronic gadget (radio, phone, TV) Yes No 14. Any member of the HH has a functional transport means (bicycle, motor cycle, boat) Yes No 15. Any member of the HH has vocational/apprenticeship/professional skills? Yes No 16. Household has domestic animals (cow(s), goat(s), Sheep, chicken, pig(s))? Yes No 17. HH has access to land for agriculture/hire? Yes No 18. Does the household own any of the following assets: cooking stove, beds, blanket, mattresses, mosquito net, shoes, cooking utensils, furniture. Yes, most of them Yes, some of them None or almost none FOOD SECURITY AND NUTRITION 19. What does the family usually eat? (at least 3 times a week) Energy foods: (potatoes, banana, oils, posho, millet, rice, maize, bread,cassava) Yes No Body building foods: (beans, meat, soya, peas, milk, eggs, chicken, fish) Yes No Protective and regulative foods: (tomatoes, oranges, pawpaw, mangoes, pineapple) Yes No 20. Over the past month, what has been the MAIN source of food consumed by your household? Home grown Donated Bought Given for Work 61 21. How many meals does the household have in a day? Not everyday One meal per day Two meals per day Three or more 22. Over the past month, did anyone in the household ever go without food for a whole day because there wasn’t enough? Yes, more than 5 Times a Month Yes, 1-4 times a month No 23. Has this household eaten at least 2 meals a day, every day, for the last month? HEALTH, WATER, SANITATION AND SHELTER 24. What is the distance (in Km) to the health care facility your household often uses? Kms Don’t Know 25. When was the last time a member of the HH accessed a health facility? Weeks/Months Don’t Know 26. Does the caregiver know the HIV status for all children in the household? Yes No 27. Do all HH members sleep under a mosquito net? Yes Some No 28. Have all the children in the household been tested for HIV? Yes Some No 29. Are all eligible children who are HIV+ and/ or have TB on treatment? None of the Children Less than half of the Children More than half of the Children All of the Children 30. [If you don’t know of any HIV+ members of the household, ask this question]: Is there any member of the household who has a chronic disease? (HIV+, Cancer, TB, Sickle cells, diabetes etc). If you know there is an HIV+ person in the household, you do not have to ask this question, but check yes. Yes No 31. What is the main source of water for members of your household? River, Lake, Pond, Unprotected well Public taps, Bore hole, Rainwater, Protected spring/well Private Connection 32. Does the household access drinking water from a safe source within 30 minutes? Yes No 33. Does the household have access to a latrine? Yes owned Yes shared No latrine 34. Observe the following: Has a clean compound Yes No Has a drying rack for HH utensils Yes No Has a garbage pit /dust bin Yes No Separate house for animals Yes No Hand washing facility Yes No 35. Observe: Does the HH have a stable shelter that is adequate, safe and dry? Not safe or stable shelter Inadequate, needs major repairs Needs some repairs but adequate Safe, adequate and dry 36. Observe: What type of a latrine/toilet facility is used by the HH? Bush/None Public Toilet for Pay Private, needs repair Private, adequate, but shared Private, safe, adequate, dry EDUCATION 1. How many of the children aged 5–17 years in this HH are not going to school or miss school 3 or more times a week [DO NOT INCLUDE 1-4 years] None attend Less than half attend regularly More than half attend regularly 62 All attend regularly PSYCHOSOCIAL SUPPORT AND BASIC CARE 37. In the past 12 months (STATE MONTH), how often has someone in your HH felt so troubled that it was necessary to consult a spiritual, faith or traditional healer, counselor or health worker? Enter number Not at all 38. Are there any children in this HH who are withdrawn or consistently sad, unhappy or depressed, not able to participate in daily activities including playing with friends and family? Enter number Not at all CHILD PROTECTION AND LEGAL SUPPORT 39. Do you think that hitting or beating a child is an appropriate means of discipline or control in the home? Yes No 40. Do you think that hitting or beating a child is an appropriate means of discipline or control at school? Yes No 41. In the past month, have you or another adult in the household used the following method of discipline with any child in your household? (Please select all the methods that apply) Punched, kicked or hit as punishment Withheld a meal as punishment Yelling and screaming 42. In the past 12 months, has any child in the HH had the following happen to them, in or outside of the HH? [Note: If you see an obvious issue of abuse or you already know about it, then indicate yes]. Indicate Yes/ No Repeated physical abuse Involved in Child Labour Family separation (ran away, chased) Sexually abused, defiled, raped, forced sex Stigmatised/ discriminated due to illness, disability or otherwise In contact/conflict with the law HOUSEHOLD SERVICES 43. Has your household received services or participated in activities from a community based program in the last six months? By this I mean, in the last six months have you or someone in your household been visited by a community worker, or participated in any activities organized by the program such as a savings group or parenting program? Yes If No,then end module and go to next module 44. Are any or all of the services your household is receiving or participating in provided by Sustainable Outcomes Yes No Don’t Know 45. What type of household based services or activities (through a visit from a volunteer) has your household received or participated in the last six months? Circle all that apply Household visits from a volunteer Direct financial support Parenting counselling Early Childhood Development Health and hygiene HIV and GBV prevention Nutrition counselling Pre/post-partum counselling HIV testing Couples counselling Support obtaining a birth certificate Child protection Psychosocial support/basic needs Food security Other_______ None 46. What type of community based services or activities (outside of the home) has your household received or participated in in the last six months? Circle all that apply Savings groups Parenting program Government SAGE program Any other cash transfer Voluntary HIV testing/counselling 63 Food security and Nutrition Skills and employment training Entrepreneurship training Other:_____ None 64 Child Inventory Page You told me at the beginning of the interview that you are responsible for [refer to total # of children from Question 105 on the Identifier page] children (0-17 years). Starting with the oldest, please tell me the first names and ages of these children. Make sure the total number of children is the same as question 105 and Proceed to fill out chart HOUSEHOLD CODE: INTERVIEW DATE ID Name Y.O.B Sex B.Regi str Disabl e Immu nized HIV status ART Works ? Attend s School Educat ion Sexuall y Active? Protect ion Pregna nt Ever given birth Marital Status (M.F) (Y/N) (Y/N) or MD/P D (Y/N) (+/-/?) (Y/N) (S, W, UE, NA) (Y/N) Class comple ted (Y/N) PA/EP / EM,SA (Y/N) (Y/N) M or S 01 02 03 04 06 07 08 09 10 11 Assessor’s Comment: 65 Index Child and Kish Grid Page Information is required for only one child in each age group. If there is only one child in a given age group, ask the caregiver to reference that child in his/her responses to the questions pertinent to that age group. If there is more than one child in any age group, use the Kish grid to randomly select one child from each age group. Once you have selected the reference child in each age group remind the caregiver that his/her responses pertain to that child only. If there is no child of a given age group in the household, skip the module for that age group. Kish Grid In each age-group specific table below, list the names of all eligible children from oldest to youngest. Using the last digit of the serial number of the questionnaire, find that number along the top row of the table. Follow that number down to the last line where a child is listed. The number that you come to is the number of the child that should be surveyed (as indicated in the number column on the far left). Circle the index child selected from each age group or indicate in writing if there are no children in a given category. # Name of eligible individuals 0-9 years listed from oldest to youngest Age (0-9) Last digit of questionnaire serial number 1 2 3 4 5 6 7 8 9 0 1 1 1 1 1 1 1 1 1 1 1 2 1 1 2 2 1 1 2 2 1 1 3 3 2 1 3 2 1 3 2 1 3 4 2 3 4 1 2 3 4 1 2 3 5 5 4 3 2 1 5 4 3 2 1 1 2 3 4 5 # Name of eligible individuals 10-17 years listed from oldest to youngest Age (10-17) Last digit of questionnaire serial number 1 2 3 4 5 6 7 8 9 0 1 1 1 1 1 1 1 1 1 1 1 2 1 1 2 2 1 1 2 2 1 1 3 3 2 1 3 2 1 3 2 1 3 4 2 3 4 1 2 3 4 1 2 3 5 5 4 3 2 1 5 4 3 2 1 1 2 3 4 5 66 Child Questionnaire aged 10-17 years SECTION 1: BACKGROUND INFORMATION Let’s start out by you telling me a little about yourself. No. Questions Coding Categories 101 Record / Confirm Child’s Name What is your name? 102 Record Child’s Line Letter from Household Schedule (Caregiver Questionnaire) 103* Record / Confirm Child’s Sex Female Male 1 2 104 In what month and year were you born? Month [____] Year [__|__|____] 105* How old were you at your last birthday? Confirm with 104 and adjust if necessary. Do not leave blank. If child does not know, ask caregiver to estimate age of child. [____] years 106 Who takes care of you? Do not read responses. Record one primary response only. Mother and/or father Sister and/or brother Aunt and/or uncle Grandmother and/or Grandfather Other relative Neighbor Friend No one/self Other: __________________ 1 2 3 4 5 6 7 8 66 --- END OF SECTION --- 67 SECTION 2: DIARY I would like you to talk to me about your day yesterday. If yesterday wasn’t a school day, ask about last school day. No. Questions Coding Categories SKIP 201 When did you get up – would you say, before the sun was up/it got light or after the sun was up/it got light? Before sunrise After sunrise 1 2 If After: 203 202 And what did you do after you got up, but before it got light? Anything else? Mark X in all applicable boxes in diary 203 Now, thinking about the time between when the sun came up/it got light and noon/the middle of the day, what did you do? Anything else? Mark X in all applicable boxes in diary 204 And around noon, what did you do? Anything else? Mark X in all applicable boxes in diary 205 Now, let’s think about the time between noon sundown/when it started to get dark, what did you do? Anything else? Mark X in all applicable boxes in diary 206 Now, let’s think about after it got dark. What did you do before you went to sleep? Anything else? Mark X in all applicable boxes in diary Instructions: Ask about the time frames one at a time; probe for additional activities before going on to the next time frame. Every column should have at least one activity box marked. Multiple activities permitted. Do not read response options. Activity Time 202 Before sun-up 203 Sun-up to noon 204 Noon 205 Noon to sun-down 206 After sun- down Sleep Meal Household chores Work on family / household farm Care for household member - child Care for household member - adult School attendance School work Work (excluding household chores) Informal recreation/leisure Organized recreation/club Other: specify ____________- ____________________ SECTION 3: EDUCATION No. Question Coding Category SKIP 68 301* Are you currently enrolled in school? Yes (correct diary) No 1 2 If No: 306 302* During the last school week, did you miss any school days for any reason? Yes No 1 2 If No: 304 303 Why did you miss school days during the last school week? Do not read responses. Circle one primary response. No money for school materials, transport I am too sick to attend school School is too far away / no school I have to work I have to care for household members Parent/guardian does not want me to go to school I don’t like school School was not in session Other: ___________________________ 1 2 3 4 5 6 7 8 66 304* What grade/form/year are you in now? [____] All: 307 305 Why do you NOT go to school? Do not read responses. Circle one primary response. No money for school materials, transport I am too sick to attend school School is too far away / no school I have to work I have to care for household members Parent/guardian does not want me to go to school I don’t like school School was not in session Other: ___________________________ 1 2 3 4 5 6 7 8 66 306 Have you ever attended school? Yes No 1 2 If No: 401 307* Were you enrolled in school during the previous school year? Yes No 1 2 If No: 309 308* What grade/form/year were you in during the previous school year? [____] All: 401 309* What is the highest grade/form/year that you have completed? [____] SECTION 4: CHORES & WORK No. Questions Coding Categories SKIP 401 Check DIARY. Were the household chores and/or care for your family or household, mentioned? Yes No 1 2 If Yes: 403 69 No. Questions Coding Categories SKIP 402 Do you sometimes do household chores, or care for a member of your household? Yes (correct diary) No 1 2 If No: 405 403 What household chores do you usually do? Anything else? Multiple responses possible; circle all mentioned. Probe with response categories if necessary. Corroborate with diary. Prepare food 1 Fetch water 2 Clean toilets 3 Take care of children 4 Plant/tend to/harvest crops 6 Feed, care for animals 7 Wash clothes, blankets 8 Other: _____________________________ 66 404 About how much time do you spend per day doing household or farm chores for your family? Less than 1 hour 1-2 hours 3-4 hours More than 4 hours / most of the day It depends / it is different everyday 1 2 3 4 5 405 Check DIARY Was other work mentioned? Yes No 1 2 If Yes: 407 406 Apart from these chores, do you sometimes do other work outside your home? Yes (correct diary) No 1 2 If No: 411 70 No. Questions Coding Categories SKIP 407 What kinds of other work do you sometimes do? Anything else? Multiple responses possible; circle all mentioned. Probe with response categories if necessary. Corroborate with diary. Hawk goods 1 Sell food at market 2 Household / farm chores for other families 3 Work in a restaurant or bar 4 Help out in shop 5 Construction 6 Sewing 7 Mechanic 8 Clerk, Delivery, Administrative 9 Other: __________________________ 66 408 How often do you do other work? Would you say….? Read response categories Every day / most days 1 If Every day (1): 409 All others: 410 Several times a week 2 Once a week 3 Once in a while 4 409 About how much time do you spend per day doing this work? Less than 1 hour 1-2 hours 3-4 hours More than 4 hours It depends / it is different everyday 1 2 3 4 5 410 Have you ever received money for any of the work that you do? Yes No 1 2 411 What [else] do you do to get money? Nothing Begging Other: __________________________ 1 2 66 If work mentioned, return to 406-410. 412 What do you do with the money you get? Anything else? Multiple responses possible; circle all mentioned. Probe with response categories if necessary. Give to parents / guardians 1 Pay for my school expenses 2 Pay for school expenses of others 3 Buy food for myself 4 Buy food for others 5 Buy other things for myself 6 Save it 7 Other: __________________________ 66 SECTION 5: FOOD AND ALCOHOL CONSUMPTION Next I would like to ask you about what you eat and drink. No. Question Coding Category SKIP 71 501 In the past four weeks, did you have to eat a smaller meal than you felt you needed because there was not enough food? Yes No 1 2 If No: 503 502 If yes – How many times did this happen? Read out responses. Rarely (1-2 times in past 4 weeks) Sometimes (3-10 times in past 4 weeks) Often (more than 10 times in past 4 weeks) 1 2 3 503 In the past four weeks, did you have to skip a meal because there was not enough food? Yes No 1 2 If No: 505 504 If yes – How many times did this happen? Read out responses. Rarely (1-2 times in past 4 weeks) Sometimes (3-10 times in past 4 weeks) Often (more than 10 times in past 4 weeks) 1 2 3 505 In the past four weeks did you go to sleep at night hungry because there was not enough food to eat? Yes No 1 2 If No: 507 506 If yes – How many times did this happen? Read out responses. Rarely (1-2 times in past 4 weeks) Sometimes (3-10 times in past 4 weeks) Often (more than 10 times in past 4 weeks) 1 2 3 507* In the past four weeks did you go a whole day and night without eating anything because there was not enough food to eat? Yes No 1 2 If No: 509 508 If yes – How many times did this happen? Read out responses. Rarely (1-2 times in past 4 weeks) Sometimes (3-10 times in past 4 weeks) Often (more than 10 times in past 4 weeks) 1 2 3 509 Have you ever consumed a drink containing alcohol including beer, spirits – that is a whole glass or drink, not just a taste? Yes No 1 2 If No: 601 510 When was the last time you consumed a drink containing alcohol? Read out responses. Yesterday / a few days ago About a week ago More than a week ago 1 2 3 511 How often does it happen that you consume a drink containing alcohol? Read out responses. Only once in a while At least once a week 1 2 SECTION 6: HEALTH, SUPPORT AND PROTECTION Now I have a few questions about your health and wellbeing. No. Question Coding Category SKIP 601* Do you have a birth certificate? Yes No Don’t know 1 2 88 If No: 603 If DK: 603 72 602* Could you please show me your birth certificate? Seen / confirmed Not seen / not confirmed 1 2 603* At any point in the last 2 weeks, have you been too sick to participate in daily activities? Yes No 1 2 604 Do you have a disability that makes it difficult for you to participate in daily activities? Yes No 1 2 If No: 606 605 How would you describe your disability? Blind or partially blind Deaf or partially deaf I have difficulties learning Physical Other _________________________ 1 2 3 4 66 606* I’m going to ask you a few questions about people in your life. Please respond yes or no. Do you have someone in your life to turn to for suggestions about how to deal with a personal problem? Yes No 1 2 607* Do you have someone in your life to help with daily chores if you were sick? Yes No 1 2 608* Do you have someone in your life that shows you love and affection? Yes No 1 2 609* Do you have someone in your life to do something enjoyable with? Yes No 1 2 SECTION 7: HIV/AIDS KNOWLEDGE, ATTITUDES & SEXUAL BEHAVIOR Section may be restricted to ages 13-17 only We are nearly done. I have a few short questions on a disease called HIV/AIDS. No. Question Coding Categories SKIP 701 Have you ever heard of an illness called AIDS? Yes No 1 2 If No: 801 702 Can people reduce their chances of getting the AIDS virus by having just one uninfected sex partner who has no other sex partners? Yes No Don’t know / Not sure 1 2 88 703 Can people reduce their chance of getting the AIDS virus by using a condom every time they have sex? Yes No Don’t know / Not sure 1 2 88 704 Is it possible for a healthy-looking person to have the AIDS virus? Yes No Don’t know / Not sure 1 2 88 705 Can people get the AIDS virus from mosquito bites? Yes No Don’t know / Not sure 1 2 88 73 706 Can people get the AIDS virus by sharing food with someone who has AIDS? Yes No Don’t know / Not sure 1 2 88 707 Can the virus that causes AIDS be transmitted from a mother to her baby: a) During pregnancy? b) During delivery? c) By breastfeeding? Yes No DK a) During pregnancy 1 2 8 b) During delivery 1 2 8 c) By breastfeeding 1 2 8 708 I have a few more questions about HIV. If you don’t want to answer, that is all right. I don’t want to know the results, but have you ever been tested to see if you have the AIDS virus? Yes No Don’t know 1 2 88 If No: 710 If DK: 710 709 I don’t want to know the results but did you get the results of your test? Yes No 1 2 710 Do you know of a place where people can go to get tested for the AIDS virus? Yes No 1 2 SECTION 8: ACCESS TO HIV PREVENTION, CARE & SUPPORT We have arrived at the last section of the questionnaire. We are almost finished. Thank you very much for your participation so far. Instructions: Respondents should respond only for services that they personally have received. The caregiver or head of household will also be asked. Data may be cross-checked. OR, this question may be posed to either the adult or the child (instead of both). 801 I am going to read out a list of items and services. Please tell me if YOU have received or accessed any of these items of services in the last 6 months. Read out services. Confirm responses with caregiver. Circle final responses. [ADD / DELETE ITEMS AS RELEVANT TO PURPOSE] a) Health care from a health professional Yes No b) Home visit from a community worker or social worker Yes No c) Free school supplies or a school uniform Yes No d) Mosquito net Yes No Ages 13-17 e) Information on how to prevent HIV and other sexually transmitted infections Yes No f) Information on birth spacing Yes No g) Livelihood training Yes No Ages 15-17 h) Life skills training Yes No --- END OF SECTION --- Thank you! I have come to the end of my questions. Is there anything you would like to add or ask us? I very much appreciate your time today. If you have any further questions about the survey, please use the contact information on your consent form I am leaving with you. Thank you for participating in this interview! 74 013 END TIME [__|__|:[____] 75 Child Questionnaire aged 0-9 years (for Caregiver) SECTION 1: CHILD HEALTH & PROTECTION I am now going to ask you a few questions about [insert child’s name]. No. Question Coding Category SKIP 101 Record / Confirm Child’s Name 102 Record Child’s Line Letter from Household Schedule (Caregiver Questionnaire) 103* Record / Confirm Child’s Sex Female Male 1 2 104 In what month and year was [NAME] born? Month [____] Year [__|__|____] 105* Remind me, how old was [NAME] at their last birthday? Confirm with 104 and adjust if necessary. Do not leave blank. If unknown, ask caregiver to estimate. [____] years 106 Would you say that in general [NAME’s] health is……? Read out responses. Excellent 1 Very good 2 Good 3 Fair 4 Poor 5 107* In the last 2 weeks, has [NAME] been too sick to participate in daily activities? Yes No 1 2 108 Does [NAME] have a disability that makes it difficult for him/her to participate in daily activities? Yes No 1 2 If No: 110 109 How would you describe [NAME’s] disability? Blind or partially blind Deaf or partially deaf I have difficulties learning Physical Other________________ _____ 1 2 3 4 66 110* Does [NAME] have a birth certificate? Yes No Don’t know 1 2 88 If No: 112 If DK: 112 111* Could you please show me [NAME’s] birth certificate? Seen / confirmed Not seen / not confirmed 1 2 No. Question Coding Category SKIP 76 112 FILTER. Age of child 5 years or older 0-4 years 1 2 If 5+ years: 128 113* Do you have a card where [NAME’s] vaccinations are written down? If yes, ask for card. Yes, seen Yes, not seen No Don’t know 1 2 3 88 If No: 113 If DK: 113 114* Check name on card to make sure card relates to child in question. Document the vaccinations recorded on the card. Only include documented vaccinations here. Yes, documented No a) BCG 1 2 b) OPV 0 1 2 c) OPV 1 1 2 d) OPV 2 1 2 e) OPV 3 1 2 f) DPT 1 1 2 g) DPT 2 1 2 h) DPT 3 1 2 i) Measles 1 2 If caregiver cannot produce a vaccination card for child, probe for vaccinations below. If you have documented the vaccinations from a card, but there are gaps in the vaccination record, probe with questions below. 115* Has [NAME] received a vaccine against tuberculosis, that is, an injection in the arm or shoulder, that usually causes a scar? (BCG) Yes No Don’t know 1 2 88 116* Has [NAME] received the polio vaccine, that is, drops in the mouth? Yes No Don’t know 1 2 88 If No: 121 If DK: 121 117* Has the child received OPV0, that is the first polio vaccine normally received in the first two weeks after birth? Yes No Don’t know 1 2 88 118* Has the child received OPV1, that is the second polio vaccine? Yes No Don’t know 1 2 88 119* Has the child received OPV2, that is the third polio vaccine? Yes No Don’t know 1 2 88 120* Has the child received OPV3, that is the fourth polio vaccine? Yes No Don’t know 1 2 88 77 No. Question Coding Category SKIP 121* Has the child received the DPT vaccination, that is, an injection given in the thigh or buttocks, sometimes at the same time as polio drops? Yes No Don’t know 1 2 88 If No: 123 If DK: 123 122* How many times was the DPT vaccine received? Once Twice Three times Don’t know 1 2 3 88 123* Has the child received a measles injection, that is, a shot in the arm at the age of 9 months or older – to prevent him or her from getting measles? Yes No Don’t know 1 2 88 124* Has [NAME] had diarrhea in the last 2 weeks? Yes No 1 2 125* Has (NAME) been ill with a fever at any time in the last 2 weeks? Yes No 1 2 126 Sometimes adults taking care of children have to leave the house to go shopping, wash clothes, or for some other reasons, and have to leave young children. On how many days in the past week was [NAME] left alone for more than one hour? [____] days 127 On how many days in the past week was [NAME] left in the care of another child (that is, someone less than 10 years old) for more than an hour? [____] days 128 Did [NAME] sleep under a mosquito net last night? Yes No 1 2 129 I don’t want to know the results, but has [NAME] ever been tested to see if he/she has the AIDS virus? Yes No 1 2 If No: 201 130 I don’t want to know the results, but do you know the result of [NAME’s] test? Yes No 1 2 SECTION 2: CHILD EDUCATION AND WORK No. Question Coding Category SKIP 201 Filter: Age of child (Question 402) 5 years or older 3-4 years 0-2 years 1 2 3 If 3-4 years: 213 If 0-2 years: 301 I now have some questions for you about [NAME’s] schooling and chores. 78 202* Is [NAME] currently enrolled in school? Yes No 1 2 If No: 206 203* During the last school week, did [NAME] miss any school days for any reason? Yes No 1 2 If No: 205 204 Why did [NAME] miss school days during the last school week? Do not read responses. Circle one primary response. No money for school fees, materials, transport Child is too sick to attend school School is too far away / no school Child has to work to help family Child needs to care for sick household members Child does not like school Other: _______________________ 1 2 3 4 5 6 66 205* What grade/form/year is [NAME] in now? [____] All: 208 206 Why is [NAME] not enrolled in school? Do not read responses. Circle one primary response. No money for school fees, materials, transport Child is too sick to attend school School is too far away / no school Child has to work to help family Child needs to care for sick household members Child does not like school Child is too young to attend school Other: _______________________________ 1 2 3 4 5 6 7 66 207 Has [NAME] ever attended school? Yes No 1 2 If No: 211 208* Was [NAME] enrolled in school during the previous school year? Yes No 1 2 If No: 210 209* What grade/form/year was [NAME] in during the previous school year? [____] All: 211 210 What is the highest grade/form/year that [NAME] has completed? [____] 211 In the past 6 months, has [NAME] worked for money or kind? Yes No 1 2 If No: 301 No. Question Coding Category SKIP 79 212 What did [NAME] do to earn these wages? Probe: Anything else? Multiple responses possible. Circle all mentioned. House chores, child care for other family 1 All: 301 Selling/Hawking goods 2 Labor, e.g., farm, construction 3 Other: ____________________ 66 213 Does [NAME] attend any organized or early childhood education program, such as a private or government facility, including kindergarten or community child care? Yes No 1 2 If Yes: 301 214 In the past 3 days, did you or any household member over 15 years of age engage in any of the following activities with [NAME]: Read out a through f one at a time. Yes No a) Read books to or looked a picture books with [NAME]? 1 2 b) Told stories to [NAME]? 1 2 c) Sang songs to [NAME] or with [NAME] including lullabies? 1 2 d) Took [NAME] outside of the home, compound, yard or enclosure? 1 2 e) Played with [NAME]? 1 2 f) Named, counted, or drew things with [NAME]? 1 2 SECTION 3: FOOD CONSUMPTION No. Question Coding Category SKIP 301 Filter: Age of child (Question 402) 2 years or older 0-1 years 1 2 If 0-1 years:401 Next I would like to ask you about what [Name} eats and drinks. 302 In the past four weeks, did [NAME] have to eat a smaller meal than you felt was needed because there was not enough food? Yes No 1 2 If No: 304 303 If yes – How many times did this happen? Read out responses. Rarely (1-2 times in past 4 weeks) Sometimes (3-10 times in past 4 weeks) Often (more than 10 times in past 4 weeks) 1 2 3 80 304 In the past four weeks, did [NAME] have to skip a meal because there was not enough food? Yes No 1 2 If No: 306 305 If yes – How many times did this happen? Read out responses. Rarely (1-2 times in past 4 weeks) Sometimes (3-10 times in past 4 weeks) Often (more than 10 times in past 4 weeks) 1 2 3 306 In the past four weeks did [NAME] go to sleep at night hungry because there was not enough food to eat? Yes No 1 2 If No: 308 307 If yes – How many times did this happen? Read out responses. Rarely (1-2 times in past 4 weeks) Sometimes (3-10 times in past 4 weeks) Often (more than 10 times in past 4 weeks) 1 2 3 308* In the past four weeks did [NAME] go a whole day and night without eating anything because there was not enough food to eat? Yes No 1 2 If No: 401 309 If yes – How many times did this happen? Read out responses. Rarely (1-2 times in past 4 weeks) Sometimes (3-10 times in past 4 weeks) Often (more than 10 times in past 4 weeks) 1 2 3 SECTION 4: ACCESS TO HIV PREVENTION, CARE AND SUPPORT No. Question Coding Category 401 I am going to read out a list of items and services. Please tell me if [child’s name] has received or accessed any of these items or services in the last 6 months. Read out services. Yes No i) (Psychosocial) counselling for a home visitor or social worker 1 2 j) Health care from a health professional 1 2 k) School fees paid for by organization 1 2 l) Free school supplies or a school uniform 1 2 m) Vitamin A supplement from an organization 1 2 n) Supplemental, emergency feeding 1 2 --- END OF SECTION --- Thank you! I have come to the end of my questions. Is there anything you would like to add or ask us? 81 I very much appreciate your time today. If you have any further questions about the survey, please use the contact information on your consent form I am leaving with you. Thank you for participating in this interview! END TIME [__|__|:[____] Questions Added to Wave 2: 6.g.8 In the past month, have you or another adult in the household used the following method of discipline with any child in your household? (Please select all the methods that apply) • Punched, kicked or hit as punishment • Withheld a meal as punishment • Yelling and screaming For each response of Yes, open up this option: How often did this happen in the past 30 days? 1 = Rarely (1-2 times) 2 = Sometimes (3-10 times) 3 = Often (more than 10 times) 6.g.9 In the past 12 months, has any child in the HH had the following happen to them, in or outside of the HH? [Note: If you see an obvious issue of abuse or you already know about it, then indicate yes]. Indicate Yes/ No • Repeated physical abuse • Involved in Child Labour • Family separation (ran away, chased) • Sexually abused, defiled, raped, forced sex • Stigmatised/ discriminated due to illness, disability or otherwise • In contact/conflict with the law • Other___ If Index Child is 0-9, OR if Adult is responding for INDEX Youth, THEN use this Module: I’m going to read you some statements describing the different ways that parents behave towards their children. Please indicate how often has this happened IN THE PAST YEAR with the with _____ Index Child? Never 0 Not Often 1 Often 2 All the time 3 1. I show my child that I am proud of him/her. 0 1 2 3 2. I take an interest in my child’s activities. 0 1 2 3 3. I listen to my child when he/she talks to me. 0 1 2 3 82 4. My child can count of on me to be there when he/she needs me. 0 1 2 3 5. My child and I talk about the things that are important 0 1 2 3 6. My child is comfortable sharing his/her thoughts and feelings with me. 0 1 2 3 7. Even if my child knowsI’d be disappointed, he/she can come to me for help with a problem. 0 1 2 3 If Adolescent is responding with adult in room OR if Adolescent is responding with no adult present, THEN use this module: I’m going to read you some statements describing the different ways that parents behave towards their children. Please indicate how often has this happened IN THE PAST YEAR with the with your parents/Primary caregivers Section 6: Support Never 0 Not Often 1 Often 2 All the time 3 1. they tell me that they are proud of me 0 1 2 3 2. They take an interest in my activities. 0 1 2 3 3. They listen when I talk to them. 0 1 2 3 4. I can count on them to be there when I need them. 0 1 2 3 5. We talk about the things that are important 0 1 2 3 6. I am comfortable sharing my thoughts and feelings with them. 0 1 2 3 7. Even if I know my they are disappointed, I can go to them for help with a problem. 0 1 2 3 83 Annex 3: Fully-Refined HVAT Model Structure 84 Annex 4: Cluster analysis We conducted an agglomerative hierarchical cluster analysis of the 24 variables that represent individual services at the household and community level. Ward’s linkage was employed as the linkage method, squared Euclidean distance as the distance measure. At each stage, the two most similar clusters are joined, beginning with the two most similar variables. A jump in the agglomeration coefficient can be observed between stage 14 and 15. Here, the coefficient change increases by 22 points over the coefficient change between stage 13 and 14, whereas changes between earlier stages were of a maximum of 12 points. At stage 14, ten clusters were formed of the 24 variables. Later jumps can be observed between stage 19 and 20, and 21 and 22. However, a five- or three-cluster solution would have retained very little information about patterns in service provision for the subsequent analysis. The second table shows the services that comprise each cluster. Stage Agglomeration coefficient Coefficient change between current and next stage Number of clusters after joining 1 5.000 7 23 2 12.000 18.75 22 3 30.750 21.65 21 4 52.400 22.933 20 5 75.333 34.381 19 6 109.714 38 18 7 147.714 45.661 17 8 193.375 46.5 16 9 239.875 51.333 15 10 291.208 54.917 14 11 346.125 60.55 13 12 406.675 72.594 12 13 479.269 75.5 11 14 554.769 97.5 10 15 652.269 103.16 9 16 755.429 109.804 8 17 865.233 119.834 7 18 985.067 124.5 6 19 1109.567 133.977 5 20 1243.544 278 4 21 1521.544 293.229 3 22 1814.773 488.019 2 23 2302.792 1 Cluster Household-based services in cluster Community-based services in cluster Direct cash transfers Direct financial support (cash), direct financial support (loan) Donations to household Cash donations to household Food security Food security Food security and nutrition Health and hygiene Health and hygiene HIV and GBV prevention HIV and GBV prevention HIV testing (household level) HIV testing 85 Savings groups Savings groups HIV testing (community level) Voluntary HIV testing/counseling Parenthood￾related services Parenting counseling Early Childhood development Nutrition counseling Pre/post-partum counseling Couples counseling Support obtaining a birth certificate Child protection Psychological support/basic needs Parenting program Skills and employment training Entrepreneurship training Government SAGE program Any other cash transfer Other household￾based services Other services 86 Annex 5: Frequencies of specified conditions Type Condition Presence Absence Household context Children (4+) 951 (47.2%) 1062 (52.8%) Individual household￾based services Direct financial support (cash) 65 (3.2%) 1948 (96.8%) Direct financial support (loan) 115 (5.7%) 1898 (94.3%) Cash donations 116 (5.8%) 1897 (94.2%) Parenting counseling 59 (2.9%) 1954 (97.1%) Early childhood development 46 (2.3%) 1967 (97.7%) Health and hygiene 468 (23.2%) 1545 (76.8%) HIV and GBV prevention 122 (6.1%) 1891 (93.9%) Nutrition counseling 70 (3.5%) 1943 (96.5%) Pre/post-partum counseling 8 (0.4%) 2005 (99.6%) HIV testing 271 (13.5%) 1742 (86.5%) Couples counseling 24 (1.2%) 1989 (98.8%) Obtaining a birth certificate 22 (1.1%) 1991 (98.9%) Child protection 74 (3.7%) 1939 (96.3%) Psychosocial support 39 (1.9%) 1974 (98.1%) Food security 112 (5.6%) 1901 (94.4%) Other household-based 91 (4.5%) 1922 (95.5%) Individual community-based services Savings groups 414 (20.6%) 1599 (79.4%) Parenting program 66 (3.3%) 1947 (96.7%) Government SAGE program 7 (0.3%) 2006 (99.7%) Any other cash transfer 2 (0.1%) 2011 (99.9%) Voluntary HIV testing 225 (11.2%) 1788 (88.8%) Food security and nutrition 129 (6.4%) 1884 (93.6%) Skill and employment 25 (1.2%) 1988 (98.8%) Entrepreneurship training 56 (2.8%) 1957 (97.2%) Clusters (w2) Direct cash transfers 135 (6.7%) 1878 (93.3%) Donations to household 116 (5.8%) 1897 (94.2%) Food security 190 (9.4%) 1823 (90.6%) Health and hygiene 468 (23.2%) 1545 (76.8%) HIV and GBV prevention 122 (6.1%) 1891 (93.9%) HIV testing (household) 271 (13.5%) 1742 (86.5%) Savings groups 414 (20.6%) 1599 (79.4%) HIV testing (community) 225 (11.2%) 1788 (88.8%) Parenthood-related services 287 (14.3%) 1726 (85.7%) Other household-based 91 (4.5%) 1922 (95.5%) Clusters (gain) Direct cash transfers 128 (6.4%) 1885 (93.6%) Donations to household 115 (5.7%) 1898 (94.3%) Food security 159 (7.9%) 1854 (92.1%) Health and hygiene 384 (19.1%) 1629 (80.9%) HIV and GBV prevention 112 (5.6%) 1901 (94.4%) HIV testing (household) 237 (11.8%) 1776 (88.2%) Savings groups 310 (15.4%) 1703 (84.6%) HIV testing (community) 206 (10.2%) 1807 (89.8%) Parenthood-related services 229 (11.4%) 1784 (88.6%) Other household-based 87 (4.3%) 1926 (95.7%) Clusters (lost) Direct cash transfers 84 (4.2%) 1929 (95.8%) Donations to household 9 (0.4%) 2004 (99.6%) Food security 245 (12.2%) 1768 (87.8%) Health and hygiene 228 (11.3%) 1785 (88.7%) HIV and GBV prevention 117 (5.8%) 1896 (94.2%) 87 HIV testing (household) 238 (11.8%) 1775 (88.2%) Savings groups 254 (12.6%) 1759 (87.4%) HIV testing (community) 197 (9.8%) 1816 (90.2%) Parenthood-related services 267 (13.3%) 1746 (86.7%) Other household-based 33 (1.6%) 1980 (98.4%) 88 Annex 6: Various Datasets in Excel and SPSS Please see attached Excel and SPSS files for datasets and corresponding results. 89 Annex 7: Correlational analysis All households Type Condition Slightly less vulnerable Strongly less vulnerable Slightly more vulnerable Strongly more vulnerable Included in initial models Household context Children (4+) ns ns ns ns no Individual household￾based services Direct financial support (cash) ns ns ns ns no Direct financial support (loan) ns ns ns ns no Cash donations ns ns ns ns no Parenting counseling ns ns ns ns no Early childhood development ns ns ns ns no Health and hygiene ns ns ns ns no HIV and GBV prevention ns ns ns ns no Nutrition counseling 0.039 ns ns ns yes Pre/post-partum counseling ns 0.026 ns ns yes HIV testing ns ns ns ns no Couples counseling ns ns ns ns no Obtaining a birth certificate ns ns ns ns no Child protection ns ns ns ns no Psychosocial support ns ns ns ns no Food security ns ns ns ns no Other household-based ns ns ns ns no Individual community￾based services Savings groups 0.043 ns ns ns yes Parenting program ns ns ns ns no Government SAGE program ns ns ns ns no Any other cash transfer ns ns ns ns no Voluntary HIV testing ns ns ns ns no Food security and nutrition ns ns ns ns no Skill and employment ns ns ns ns no 90 Entrepreneurship training ns ns ns ns no Clusters (w2) Direct cash transfers ns ns ns ns no Donations to household ns ns ns ns no Food security ns ns ns ns no Health and hygiene ns ns ns ns no HIV and GBV prevention ns ns ns ns no HIV testing (household) ns ns ns ns no Savings groups 0.043 ns ns ns yes HIV testing (community) ns ns ns ns no Parenthood - related services ns ns ns ns no Other household -based ns ns ns ns no Clusters (gain) Direct cash transfers ns ns ns ns no Donations to household ns ns ns ns no Food security ns ns ns ns no Health and hygiene ns ns ns ns no HIV and GBV prevention ns ns ns ns no HIV testing (household) ns ns ns ns no Savings groups ns ns ns ns no HIV testing (community) ns ns ns ns no Parenthood - related services ns ns ns ns no Other household -based ns ns ns ns no Clusters (lost) Direct cash transfers ns ns ns ns no Donations to household ns ns ns 0.044 yes Food security ns ns ns ns no Health and hygiene ns ns ns ns no 91 HIV and GBV prevention ns ns ns ns no HIV testing (household) ns ns ns ns no Savings groups ns ns ns ns no HIV testing (community) ns ns ns ns no Parenthood￾related services ns ns ns ns no Other household-based ns ns ns ns no Slightly vulnerable households Type Condition Slightly less vulnerable Strongly less vulnerable Slightly more vulnerable Strongly more vulnerable Included in initial models Household context Children (4+) ns ns ns ns no Individual household￾based services Direct financial support (cash) 0.039 ns 0.006 0.003 yes Direct financial support (loan) ns ns ns ns no Cash donations ns ns ns ns no Parenting counseling ns ns ns ns no Early childhood development ns ns ns ns no Health and hygiene ns ns ns ns no HIV and GBV prevention ns ns ns ns no Nutrition counseling ns ns ns ns no Pre/post-partum counseling ns 0.001 ns ns yes HIV testing ns ns ns ns no Couples counseling ns ns ns ns no Obtaining a birth certificate ns ns ns ns no Child protection ns ns ns ns no Psychosocial support ns ns ns ns no Food security ns ns ns ns no Other household-based ns 0.017 ns ns yes 92 Individual community - based services Savings groups ns ns ns 0.045 yes Parenting program ns 0.048 ns ns yes Government SAGE program ns ns ns ns no Any other cash transfer ns ns ns ns no Voluntary HIV testing ns ns ns ns no Food security and nutrition ns ns ns ns no Skill and employment ns ns ns ns no Entrepreneurship training 0.048 ns ns ns yes Clusters (w2) Direct cash transfers ns ns ns 0.042 yes Donations to household ns ns ns ns no Food security ns ns ns ns no Health and hygiene ns ns ns ns no HIV and GBV prevention ns ns ns ns no HIV testing (household) ns ns ns ns no Savings groups ns ns ns 0.045 yes HIV testing (community) ns ns ns ns no Parenthood - related services ns ns ns ns no Other household -based ns 0.020 ns ns yes Clusters (gain) Direct cash transfers ns ns ns ns no Donations to household ns ns ns ns no Food security ns ns ns ns no Health and hygiene ns ns ns ns no HIV and GBV prevention ns ns ns ns no HIV testing (household) ns ns ns ns no Savings groups ns ns ns ns no 93 HIV testing (community) ns ns ns ns no Parenthood￾related services ns ns ns ns no Other household-based ns 0.019 ns ns yes Clusters (lost) Direct cash transfers ns ns ns ns no Donations to household ns ns ns ns no Food security ns ns ns ns no Health and hygiene ns ns ns ns no HIV and GBV prevention ns ns ns ns no HIV testing (household) ns ns 0.022 ns yes Savings groups ns ns ns ns no HIV testing (community) ns ns ns ns no Parenthood￾related services ns ns ns ns no Other household-based ns ns ns ns no Moderately vulnerable households Type Condition Slightly less vulnerable Strongly less vulnerable Slightly more vulnerable Strongly more vulnerable Included in initial models Household context Children (4+) ns ns ns ns no Individual household￾based services Direct financial support (cash) ns ns ns ns no Direct financial support (loan) ns 0.037 ns ns yes Cash donations ns ns ns ns no Parenting counseling ns ns ns ns no Early childhood development ns ns ns ns no Health and hygiene ns ns ns ns no HIV and GBV prevention ns ns ns ns no Nutrition counseling ns ns ns ns no Pre/post-partum counseling 0.033 ns ns ns yes HIV ns ns ns ns no 94 testing Couples counseling ns ns ns ns no Obtaining a birth certificate ns ns ns ns no Child protection ns ns ns ns no Psychosocial support ns ns ns ns no Food security ns ns ns ns no Other household -based ns ns 0.003 ns yes Individual community - based services Savings groups 0.033 ns ns ns yes Parenting program ns ns ns ns no Government SAGE program 0.033 ns 0.000 ns yes Any other cash transfer ns ns ns ns no Voluntary HIV testing ns ns ns ns no Food security and nutrition ns ns ns ns no Skill and employment ns 0.008 ns ns yes Entrepreneurship training ns ns ns ns no Clusters (w2) Direct cash transfers ns 0.016 ns ns yes Donations to household ns ns ns ns no Food security ns ns ns ns no Health and hygiene ns ns ns ns no HIV and GBV prevention ns ns ns ns no HIV testing (household) ns ns ns ns no Savings groups 0.033 ns ns ns yes HIV testing (community) ns ns ns ns no Parenthood - related services ns ns ns ns no Other household -based ns ns ns ns no Clusters (gain) Direct cash transfers ns 0.023 ns ns yes 95 Donations to household ns ns ns ns no Food security ns ns ns ns no Health and hygiene ns ns ns ns no HIV and GBV prevention ns ns ns ns no HIV testing (household) ns ns ns ns no Savings groups ns ns ns ns no HIV testing (community) ns ns ns ns no Parenthood - related services ns ns ns ns no Other household -based ns ns ns ns no Clusters (lost) Direct cash transfers ns ns ns ns no Donations to household ns ns ns ns no Food security ns 0.009 ns ns yes Health and hygiene ns ns ns ns no HIV and GBV prevention ns ns ns ns no HIV testing (household) ns ns ns ns no Savings groups ns ns ns ns no HIV testing (community) ns ns ns ns no Parenthood - related services ns ns ns ns no Other household -based ns ns ns ns no 96 Annex 8: Specifications of configurational analyses for the absence of vulnerability changes Model specifications Sample All vulnerable HH Slightly vulnerable HH Moderately vulnerable HH Outcome Absence of strong reductions I Absence of strong increases II Absence of strong reductions III Absence of strong increases IV Absence of slight increases V Absence of strong increases VI Conditions and directed assump￾tions Moderately vulnerable HH (/) (/) Nutrition counseling (-) (+) (-) (+) (+) (+) Pre/post￾partum counseling (-) (+) (-) (+) (+) (+) SAGE program (+) (+) Skill and employ￾ment (+) (+) Parenting program (-) (+) (-) (+) Entrepre￾neurship training (-) (+) (-) (+) Savings groups (-) (+) (-) (+) (+) (+) Direct cash transfer (gain) (-) (+) (-) (+) (+) (+) Other HH￾based services (gain) (-) (+) (-) (+) (+) (+) Donations to HH (lost) (+) (-) (+) (-) (-) (-) Food security (lost) (+) (-) (+) (-) (-) (-) HIV testing (lost) (+) (-) (+) (-) (-) (-) Solution coverage 0.77 0.95 0.91 0.99 0.99 0.99 Solution consistency 0.80 0.85 0.80 0.80 0.96 0.99 Notes: The signs in parentheses denote the nature of directed assumptions that were specified for the conditions: (/) none has been specified, (-) the absence of the condition contributes to the outcome, (+) the presence of the condition contributes to the outcome. Empty cells denote that the condition has not been included in the model. 97 Annex 9: Full Display of QCA Solutions Table 1 - Paths to less vulnerability for moderately vulnerable households (consistency threshold .65, α < .05) Path HH context Access to services Less vulnerab ility Coverag e Saving groups Gained access1 Did not lose access2 Children (4+) Cash transfers HIV test Parental Food HIV￾GBV Volunta ry HIV testing 1 21% 2 31% 3 69% Notes: Outcome represents a 10% reduction in the HVAT vulnerability score. Solution coverage: 83%. Solution consistency: 83%. Directed assumptions: Cash transfers (+), Saving groups (+), HIV test (+), Parental (+), Losing access to food (-), Losing access to HIV-GBV (-), Losing access to voluntary HIV testing (-). 1“Gained access” is positive if HH gained access to the service between W2 and W1. It is negative if it did not gain access in that time. 2“Did not lose access” is positive if HH did not lose access to the service between W2 and W1. It is negative if HH lost access to the service in that time. Trying to interpret this: • [Is “age of parents” available? Is there data on polygamy? Given the high reproduction rate in Uganda, larger households may be simply an effect of years since woman entered reproductive age. Or an effect of number of women in household.] • More children => more vulnerable. Cash transfers and access to saving groups needed to reduce vulnerability. (Path 1) • Smaller household, young parents – material vulnerability will likely reduce over time. Functional vulnerability in focus. HIV-GBV services help to reduce functional vulnerability. (Path 2) • Vulnerability is generally decreasing over time (average W2: 38%, average W1: 41%). Path 3 shows that if no major disruptions happen (e.g. losing access to service), vulnerability reduced over time. 98 Table 2 - Paths away from a significant reduction in vulnerability for slightly vulnerable households (consistency threshold .65, α < .05) Pat h HH context Access to services Signific ant reducti on in vulnera -bility Covera ge Household Community Gainin g access to other service s Childre n (4+) Saving groups Direct cash transfe r Early childho od Pre￾Partum Parenti ng progra m SAGE progra m Entrep r. trainin g Volunt ary HIV testing 1 81% 2 54% 3 59% 4 33% 5 24% 6 25% 7 13% Notes: Outcome represents a 25% reduction in the HVAT vulnerability score. Table shows solution for the non-occurrence of this outcome. Solution coverage: 90%. Solution consistency: 80%. Directed assumptions: Saving groups (-), Direct cash transfer (-), Early childhood (-), Pre partum (-), Parenting program (-), SAGE program (-), Entrepreneurship training (-), Voluntary HIV testing (-), Other (-).