SUSTAINABLE OUTCOMES FOR YOUTH AND CHILDREN BASELINE REPORT March 15, 2017 This publication was produced for review by the United States Agency for International Development. It was submitted by the USAID/Uganda Monitoring, Evaluation and Learning Program, AID-617-C-13-00007 It was prepared by SoCha,llc under subcontract to QED Group,llc. SUSTAINABLE OUTCOMES FOR YOUTH AND CHILDREN BASELINE REPORT Submitted to: USAID/Uganda Prepared by: SoCha,llc, under subcontract to QED Group,llc Disclaimer: The authors’ views expressed in this document do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Acknowledgements We would like to thank USAID/Uganda, QED Group,LLC, CRS Uganda and MEEPP for the generous time and support they offered during the preparation for the baseline survey. First and foremost, we are grateful to Katharina Erxleben, Christine Muwanga, Phil Greene, and Douglas Blanton for endorsing the use of QCA as a rigorous impact assessment tool and inviting SoCha to participate. We also are grateful to the QED Group team – particularly Anant Singh, Johan Kleinhans, Gorretti Mbabazi, Augustine Wandera and Ruth Ankunda for their support throughout Phase 1 of our visit. Staff at CRS and MEEPP – especially Helena Dalton, Richard Ekodeu, Sasha Angelevski, Fred Werikhe, Olive D’Mello and Barbara Amuron – generously offered their time straight away, and have since been crucial to the development of our understanding of Sustainable Outcomes. We’d also like to recognize the insightful ethics training offered by Dr. Finally, we’d also like to thank Dr. Leyla Ismayilova and her team, Nathan Roter, Lu Yang, and Young Han, from the University of Chicago, School of Social Service Administration and Dr. Elias Dinas from Brasenose College, Oxford for their technical inputs and guidance on experimental design. -SoCha,LLC Acronyms ACODEV Action for Community Development AMELP Activity Monitoring, Evaluation and Learning Plan ANPPCAN African Network for Prevention & Protection Against Child Abuse & Neglect ART Antiretroviral therapy CAO Chief Administration Officers CDO Community Development Officers CRS Catholic Relief Services CSO Civil Society Organizations DCDO District Community Development Officers DID Difference in Difference Design DHMIS2 District Health Information System 2 DOVVC District OVC Coordination Committees ECD Early Childhood Development HIV Human Immunodeficiency Virus HVAT Household Vulnerability Assessment Tool HVPT Household Orphans and Vulnerable Children Vulnerability Prioritization Tool IPs Regional Implementing Partners MEEPP Monitoring and Evaluation of Emergency Plan Progress MER Monitoring, Evaluation and Reporting MGLSD Ministry of Gender Labor and Social Development ODK Open Data Kit OLS Ordinary Least Squares OVC Orphans and Vulnerable Children OVCMIS OVC Management Information Science PEPFAR President's Emergency Plan for AIDS Relief PSWs Para Social Workers QCA Qualitative Comparative Analysis SAGE Social Assistance Grants for Empowerment SCORE Sustainable Comprehensive Responses for OVC and their Families SDS Strengthening Decentralized Systems SILCs Savings and Internal Lending Communities SOCY Sustainable Outcomes for Children and Youth (aka Sustainable Outcomes) SOVCCs Subcounty OVC Coordination Teams SRMP Sub Recipient Management Policy TPO Transcultural Psychosocial Organization UNICEF The United Nations Children's Fund UPDF Uganda People’s Defense Force USAID United States Agency for International Development VHTs Village Health Teams VI Vulnerability Index WASH Water, Sanitation and Hygiene Table of Contents Acknowledgements............................................................................................................................................................. ii Acronyms............................................................................................................................................................................ iii Table of Contents............................................................................................................................................................... v Executive Summary.........................................................................................................................................................viii Background ..................................................................................................................................................................viii Baseline Goals and Design ........................................................................................................................................viii Baseline Implementation............................................................................................................................................... x Findings........................................................................................................................................................................... xi Conclusions and Recommendations........................................................................................................................xiv Introduction......................................................................................................................................................................... 1 Evaluation Purpose, Aims and Objectives................................................................................................................. 3 Research Strategy............................................................................................................................................................ 3 Research Design: Impact Evaluation......................................................................................................................... 4 Research Design: QCA ................................................................................................................................................ 6 Outcome: Defining Vulnerability ................................................................................................................................ 8 Power Analysis and Sample Size Calculations........................................................................................................... 9 Sampling Strategy .........................................................................................................................................................11 Survey Implementation ...............................................................................................................................................13 Baseline Survey Results....................................................................................................................................................15 Survey Data ...................................................................................................................................................................15 Group Comparisons of Household Socio-Demographic Characteristics...........................................................16 Group Comparisons of Secondary Outcomes........................................................................................................24 Vulnerability Assessment and HVAT Scoring ........................................................................................................27 HVAT Group Comparisons..................................................................................................................................28 HVAT Distribution.................................................................................................................................................29 Relative Biases..........................................................................................................................................................29 Sampling: Random Route vs. Sustainable Outcomes-Assisted........................................................................29 Cohort 1 vs. Cohort 2.............................................................................................................................................30 Structural Biases.......................................................................................................................................................31 Contradictions in scoring........................................................................................................................................32 Arbitrary Ordinal Spacing and Assignment.........................................................................................................32 Principle Components Analysis.............................................................................................................................33 CPA and Item Testing ............................................................................................................................................34 Geography.................................................................................................................................................................44 Mapping the QCA System..........................................................................................................................................44 Sustainable Outcomes MIS....................................................................................................................................45 Capturing Result 1 ...................................................................................................................................................46 Capturing Result 3 ...................................................................................................................................................47 Tracking Result 2 .....................................................................................................................................................48 Suggested Improvements to the System..............................................................................................................48 Conclusions and Recommendations................................................................................................................................ 0 Annex 1: Scope of Work ................................................................................................................................................... i Annex 2: Survey Instrument............................................................................................................................................ x Identifier Page................................................................................................................................................................. x Household Vulnerability Module................................................................................................................................ xi Child Inventory Page ................................................................................................................................................... xv Index Child and Kish Grid Page...............................................................................................................................xvi Child Questionnaire aged 10-17 years.....................................................................................................................xvii Child Questionnaire aged 0-9 years (for Caregiver)..............................................................................................xxv Annex 3: HVAT Scoring Method.............................................................................................................................xxxii CPA 1: Economic Strengthening .........................................................................................................................xxxii 1.1: Who is the main contributor to household income?..............................................................................xxxii 1.2: What is the main source of HH income?.................................................................................................xxxii 1.3: What is the current monthly HH income? (express in Uganda Shillings)...........................................xxxii 1.4: Do these statements apply to this HH? (Yes/No) .................................................................................xxxii 1.5: If the HH incurred any of the following expenses in the past three months, was it able to pay without difficulty?...............................................................................................................................................xxxiii CPA 2: Food Security and Nutrition....................................................................................................................xxxiii 2.1: Over the past month (mention month), what has been the MAIN source of food consumed by your HH?.......................................................................................................................................................................xxxiii 2.2: What does the family usually eat (at least 3 times a week), Yes/No...................................................xxxiii 2.3: How many meals does the HH have in a day?.......................................................................................xxxiii CPA 3: Health, WASH, and Shelter.....................................................................................................................xxxiii 3.1: Do the following apply to this HH? Indicate Yes/No/NA ................................................................xxxiii 3.2: Does the caregiver know the HIV status for all children in the household?.....................................xxxiv 3.3: Are all eligible children who are HIV+ and/ or have TB on treatment?...........................................xxxiv 3.4: Does the HH have a stable shelter that is adequate, safe and dry?.....................................................xxxiv 3.5: What is the type of a latrine/toilet facility used by members of your HH?.......................................xxxiv CPA 4: Education....................................................................................................................................................xxxiv 4.1: How many children aged 5-17 years in this HH are not going to school or miss school 3 or more times a week?.......................................................................................................................................................xxxiv CPA 5: Psychosocial Support and Basic Care .....................................................................................................xxxv 5.1: In the past 12 months, how often has someone in your HH felt so troubled that it was necessary to consult a spiritual, faith or traditional healer, counselor or health worker?...............................................xxxv 5.2: Are there any children in this HH who are withdrawn or consistently sad, unhappy or depressed, not able to participate in daily activities including playing with friends and family? Yes/ No ...............xxxv CPA: Child Protection.............................................................................................................................................xxxv 6.1: In the past month, have you or another adult in the household used the following method of discipline with any child in your household? (Please select all the methods that apply)..........................xxxv 6.2: In the past 12 months, has any child in the HH had the following happen to them, in or outside of the HH?................................................................................................................................................................xxxvi Annex 4: List of Sampled Parishes ................................................................................................................................... viii Executive Summary Background USAID/Uganda’s Orphans and Vulnerable Children (OVC) programming employs an integrated approach to reducing OVC and Youth vulnerability in Western, Southwestern and Central Uganda. Specifically, Sustainable Outcomes aims to achieve the following three Results:  Result 1: Orphaned and vulnerable children, youth and their caregivers are better economically empowered to access core services;  Result 2: Local government and CSOs and informal community structures increase and improve core services for orphaned and vulnerable children, youth and their caregivers;  Result 3: Improved coordination of community-based clinical and socio-economic services for efficiency and effectiveness along the continuum of care. These three results are designed to be mutually supporting, in which SILC households not only benefit from the expected increase in knowledge, access, and support, but also experience improved access to quality OVC and services across an improved body of coordinated service providers operating at local (CSO and informal), social service worker, local government and other service provider (other programs and NGOs) levels. Sustainable Outcomes has already entered its second year of implementation and have reached a significant number of households. System strengthening interventions under Result 2 have been rolled out in all target communities since October 2015. Under Results 1 and 3, households will be reached across three Cohorts: • Cohort 1 (since May 2016): 85% of communities • Cohort 2 (planned for January 2017): 5% of communities • Cohort 3 (planned for January 2018): 10% of communities (tbd. after review) This staggered implementation approach presents the opportunity to conduct an impact evaluation of how effective Sustainable Outcome’s is at reducing household, orphan and youth vulnerability. Specifically, the impact evaluation will systematically compare the difference in outcomes between Cohort 2 and Cohort 3. Baseline Goals and Design The purpose of this baseline is to help USAID/Uganda answer overarching questions regarding the effectiveness of Orphans and Vulnerable Children (OVC) programs; understand how different interventions or combinations of interventions of the program contribute to program outcomes; and generate evidence to inform decisions about future OVC programs. The aim is to assess and understand the impact of participating in programs that seek to improve household socioeconomic security through savings and strengthen institutional childcare service delivery on OVC caregivers, households and children over time (see Annex 1 for the full scope of work). The primary objectives of this baseline are to lay the foundation to answer two complementary and inter￾related questions:  Has caregiver participation in Sustainable Outcomes reduced household vulnerability and improved OVC well-being?  What combinations of factors (Including those within and outside of Sustainable Outcomes’ control) best identify how this may or may not have happened? Design ix The first question will be answered use a quasi-experimental, difference in difference, impact evaluation design in which the outcomes of Cohort 2 will be compared with a control group that will later be eligible to participate in Cohort 3. The second question will be answered using Qualitative Comparative Analysis (QCA). QCA is a comparative research approach that combines the diversity of implementation with data on relevant external factors into an analytical model that identifies how various pathways to the outcome can be achieved. This combined QCA/quasi-experimental design approach presents a unique opportunity to test Sustainable Outcomes’ underlying assumption that caregiver participation in a SILC group, combined with additional support services and an improved referral system, improves household economic status which in turn leads to improvements in child well-being. This baseline report is designed to lay the groundwork to address three different impact evaluation questions:  Question 1: Does caregiver participation in activities under Results 1 and 3 reduce household vulnerability and improve OVC well-being?  Question 2: Does caregiver participation in activities under Results 1 reduce household vulnerability and improve OVC well-being?  Question 3: Do the combined activities under Results 1, 2 and 3 reduce rates of HIV prevalence and incidence in participating subcounties? Moreover, it is also designed to address the following sets of evaluative questions using QCA: 1. What combinations of factors best identify how Sustainable Outcomes reduced household vulnerability and improved OVC/Youth outcomes? 2. What combinations of factors explain where Sustainable Outcomes did NOT reduce household vulnerability and improved OVC/Youth outcomes? 3. What external factors, especially additional services provided as the result of Sustainable Outcomes’ efforts to improve the referral system, are relevant for reducing household vulnerability and improving OVC/Youth outcomes? 4. Which activities implemented under Sustainable Outcomes may not be relevant for reducing household vulnerability and improving OVC/Youth outcomes? 5. How did households and OVC/Youth improve in subcounties where Sustainable Outcomes did not (yet) implement R1 and R3 activities? 6. What are the key factors of R2 that explain how HIV prevalence and incidence rates may have declined in Sustainable Outcomes subcounties relative to subcounties outside of the program? The primary impact indicator used for this evaluation is the vulnerability score assessed with the Household Vulnerability Assessment Tool (HVAT). Twenty-four secondary outcome measures are captured at the OVC and Youth wellbeing level for children aged 0-9 years (with questions directed to the primary caregiver) and youth aged 10-17 years. These include:  Percent of children whose primary caregiver knows the child’s HIV status  Percent of children < 5 years of age who are undernourished  Percent of youth too sick to participate in daily activities  Percent of children too sick to participate in daily activities  Percent of youth who have a Birth Certificate  Percent of children who have a Birth Certificate  Percent of youth enrolled in school  Percent of children enrolled in school  Percent of youth regularly attending school  Percent of children regularly attending school x  Percent of children <5 years with recent diarrhea  Percent of children <5 years with recent fever  Percent of youth >10 to 17 years reporting irregular food intake  Percent of children >2 to 9 years reporting irregular food intake  Percent of youth who progressed in school during the last year  Percent of children who progressed in school during the last year  Percent of children <5 years of age who recently engaged in stimulating activities with any household member over 15 years of age  Percent of caregivers who feel harsh physical punishment is appropriate means of discipline in the home or school  Percent of households able to access money to pay for unexpected expenses  Percent of households able to access money to pay for health expenses in past 3 months  Percent of households able to access money to pay for school expenses in past 3 months  Percent of households able to access money to pay for food expenses in past 3 months  Percent of children 1-5 years fully immunized  Percent of youth aged 10-17 years reporting basic support Two HIV-specific measures are captured at the sub county level over time. These are HIV prevalence and HIV incidence rates, and can be broken down into specific subgroups, such as girls ages 10-17. We will use data from the District Health Information System 2 (DHMIS2) database. Baseline Implementation The survey team conducted household surveys for 2,629 households across 82 parishes in 15 districts across the three regions. The survey is adequately powered (using the standard power of 80% at a significance level of 5%) to detect changes in each impact indicator that fall between 5%-10%. The control group included 834 households, Treatment group 1 (R3 group) assigned to receive the village Para-social work training included 863 households and the Treatment group 2 (R1R3) that will receive the combined SILC intervention included 932 households. Households are distributed across the three implementation regions as follows: HH Group Distribution by Region Region Control Group R3 group (Treatment 1) R1R3 group (Treatment 2) Total Central 834 460 0 1294 Western 0 317 0 317 Southwest 0 86 932 1018 Total 834 863 932 2629 Control and treatment groups were selected according to the Sustainable Outcomes implementation cycle. The R3 treatment group runs across all three regions, with a higher concentration in Central and the fewest in the Southwest. None of these households have received any R1 services or are slated to receive any services under Cohort 2. All households for the R1R3 group are from the Southwest Region, as this is the main region in which Sustainable Outcomes program staff will be providing R1 services (as well as receive R3 services). Households selected under the Control group are those which sit in districts that have not received any R3 or R1 services to date, but are eligible for participation in the program under the upcoming Cohort 3. These households therefore allow us to avoid any ethical issues associated with observing households in need but denying them any subsequent assistance (see Annex 4 for the complete list of Parishes sampled). xi Household sampling selection was aligned to Sustainable Outcomes’ enrollment target areas across Cohorts 2 and 3, and used a two stage cluster sampling approach to select households. In the first stage, 31 parishes were selected from each treatment and control group’s sampling frame. In the second stage, 27 to 30 households in the control group and R3 treatment were selected in each Parish using the “random route” technique. In the R1R3 treatment group, the survey team received lists of potential vulnerable households from Sustainable Outcomes local representatives and randomly selected a household from this list as the enumeration starting point. Households were first screen for consent and eligibility using the HVAT screening tool. The baseline survey was designed to collect sampling information at three levels of analysis: Household, OVC (0-9), and Youth (10-17). Information was collected across three dimensions: Vulnerability, OVC/Youth Status and Situational Analysis. Each dimension uses a questionnaire module that was already tested and IRB approved in Uganda. Data was collected using handheld tablets (with power banks) using Survey CTO. Survey instruments and consent forms were translated into Luganda, Runyankole, and Runyoro/Rutoro (see Annex 2 for the survey instrument). Consent forms were created for primary care givers, emancipated youth respondents (age 10-17), and child assent forms (signed by the primary care giver). During the baseline survey period, the study team also met with Sustainable Outcomes staff to work to establish a QCA-system within the program’s M&E. This entailed the following steps: 1. Socialized the Sustainable Outcomes teams on QCA: The study team gave presentations to Sustainable Outcome staff on how QCA functioned, it’s applicability to the program and next steps; 2. Confirmed the relevance of the QCA questions to be answered: The study also presented the QCA questions to the implementation team to discuss and confirm their relevance to management decision making. 3. Identified data inputs and modify, if needed and possible, the current MIS system: The study team held a number of meetings with the M&E team on the design and implementation of the program’s MIS. A number of changes were suggested (see below); 4. Understand the “human systems” behind the data: The team also met individually with program staff to identify data flows, reporting and roles/responsibilities associated with the collection of implementation data. Findings For the QCA design, the baseline survey team worked with Sustainable Outcomes staff to identify which sets of relevant implementation data will be used to answer the “how” questions of program effectiveness. Overall, Sustainable Outcomes has put into place a new system that sufficiently captures a significant amount of data to answer a wide array of implementation questions. However, the following limitations were observed:  There are no fields to capture GPS coordinates.  The current Household ID system holds the potential to generate duplicate numbers. Moreover, villages and Parishes that have the same name across the 16 districts cannot be distinguished from each other in the Household ID system. The implications are that users may incorrectly assign households to other districts.  The current system cannot capture multiple referrals. These concerns aside, the new system is comprehensive, user friendly and well designed. The only other major potential challenge involves capacity, as Sustainable Outcomes currently employs only a small number of M&E experts, none of which have database management expertise. xii The QCA system has strong potential to enable Sustainable Outcome implementer to use the results to provide ongoing program feedback, learning loops, mini-cost effectiveness analyses and midcourse program adaptations. These steps are not yet formally adopted in the program’s workplan. For the 24 secondary outcomes, five had no significant differences across the three groups. These all revolved around children under five on such matters as involvement in stimulating activities, irregular food intake, school attendance and progression, health and caregiver knowledge of HIV status. All remaining secondary indicators had significant differences. For the primary outcome indicator, the baseline survey found that household vulnerability using the HVAT tool was distributed according to the following (see Annex 3 for a breakdown of how the HVAT was calculated): Distribution of Vulnerability across Control and Treatment Groups HVAT Category Control R1 R1R3 Total Not Vulnerable 64 (8%) 60 (7%) 23 (2%) 147 (6%) Slightly Vulnerable 628 (75%) 667 (77%) 630 (68%) 1925 (73%) Moderately Vulnerable 142 (17%) 136 (16%) 279 (30%) 557 (21%) Critically Vulnerable 0 0 0 0 Total 834 863 932 2629 The differences between groups are significant (*** p<0.001) and can most likely be attributed to geography. Moreover, the HVAT’s six constituent “Core Program Areas” (CPAs, i.e. economic strengthening, food and nutrition, health and water, sanitation and hygiene (WASH), education, child protection and psycho-social) also held significant differences. Using data from a previous “Vulnerability Index,” analysis revealed that the Central region (which contains the control group) tends to have a lower rate of vulnerability compared to the Southwest and Western region (which mostly hosts R1 and R1R3 treatment groups). These differences are not an issue for the difference-in-difference design. The distribution of household vulnerability scores does not pose a threat to the impact evaluation design, and also improves the explanatory power of the QCA analysis by providing more evidence of the counter-factual within each treatment group. However, from a programmatic view, the seemingly low number of more vulnerable households as well as the percentage of “not vulnerable households” may be a cause for concern. This raised questions about the household selection approach and about the design of the HVAT tool; both of which required further investigation. In regards to household selection, households that were selected through pure random selection without the assistance of Sustainable Outcome stakeholders had no significant differences from those who were selected from lists generated by the Sustainable Outcomes team. However, two of the HVAT’s CPAs – psycho/social and child protection - most likely suffers from a positive social response bias (especially in regards to self-reporting of abuse and mental well-being), which may drive vulnerability scores downward (i.e. households appear less vulnerable than they actually are). Dropping these two CPAs pushes overall vulnerability upward (households are scored as more vulnerable). During the implementation, the survey team also discovered that Sustainable Outcomes program staff had also dropped these two CPAs, and were working from a leaner version known as the Household Assessment Tool (HAT) for Cohort 1. The survey team found that Cohort 1 has a higher percentage of “Not Vulnerable” and a slightly higher “Critically Vulnerable” representation, while Cohort 2 has a much higher percentage of “Moderately Vulnerable” and very similar “Slightly Vulnerable” distribution. As such, there do appear to be systematic differences between the two Cohorts in which Cohort 1 is thicker on both ends of the xiii vulnerability spectrum while Cohort 2 has a higher concentration of mid-level vulnerability. Of particular concern is the high number of “not vulnerable” households current receiving support under Cohort 1 (more than 14,000). In regards to the design of the HVAT tool, it was found that the way the tool is scaled has an impact on the resulting vulnerability scores and status. More precisely, a review of the HVAT tool scaling structure reveals the following:  Contradictions in scoring, due in part to inconsistent units of measurement;  Arbitrary spaces in the scoring of each item can bias household vulnerability scores downward (i.e. they are less vulnerable);  Many of the choices found under each question/item do not fall on the same spectrum (i.e. some represent different concepts than the one represented by the question);  The items on the HVAT are not equal in relevance and therefore should not be scored equally;  The items fall across different underlying dimensions and therefore are not easily aggregated using the current scoring method. Principle Component Analysis (PCA)1 was used to test the internal consistency of each item across the HVAT index and its six constituent “Core Program Areas” (CPAs, i.e. economic strengthening, food and nutrition, health and WASH, education, child protection and psycho-social). The results suggest a very poor fit in terms of how well all HVAT items combine to form a uni-dimensional scale. Less than 16% of all variance could be explained on a combined HVAT component, and 9 of the 17 items drop out with scores below a minimum .35 threshold. These results suggest that the HVAT cannot be easily reduced to a single dimension score. The implications are that HVAT scores can be a misleading proxy for comparing underlying vulnerability across households (for example, two households that hold the same overall vulnerability score but differ in their CPA distribution in fact may differ significantly). To improve the scale, items and options on the scale were separated and tested separately to generate total CPA scores instead of individual items. The results lead to the creation of two underlying dimensions associated with household vulnerability: “material vulnerability,” i.e. economic strengthening, food and nutrition, health and WASH and education; and “functional vulnerability,” i.e. child protection and psycho￾social support. Using this approach to assign weighted factor scores, new household vulnerability scores were generated and compared with the original raw scores. However, while these more accurate results revealed that the relative weights of each CPA shifted and were more accurately represented than using the original raw scoring method, no substantial differences in the distribution of household vulnerability between the two methods could be found. The survey team also explored how geography may help explain the seemingly lower level of household vulnerability across both Cohort 1 and the baseline. Using data from a previous “Vulnerability Index” that was reported in the Ministry of Gender Labor and Social Development (MGLSD)’s 2010 OVC Situational Awareness Report (p. 4), a quick comparison reveals that the Central region (which contains the control group) tends to have a lower rate of vulnerability compared to the Southwest and Western region (which mostly hosts R1 and R1R3 treatment groups): Critically Vulnerable Moderately Vulnerable Generally Vulnerable Total Vulnerability Central 7.8 33.6 52.7 94.1 1 PCA is a statistical technique used to identify a smaller number of uncorrelated variables, i.e. components, from a large set of variables. The goal is to explain the maximum amount of variance with the smallest number of components. xiv Eastern 7.5 45.5 43.8 96.8 North 9.3 53.6 35.9 98.8 Western 8.1 41.1 45.9 96.1 Average 8.1 42.9 45.1 96.1 However, the survey team was unable to obtain the primary data behind these numbers, as well as the scaling methods used to derive them. As such, one should be wary of making direct comparisons with this data and further research is most likely required. Conclusions and Recommendations These results lead to the following high level conclusions and recommendations:  The HVAT is an adequate tool to provide an overall snapshot of various dimensions of household vulnerability and is therefore useful for program decision making and addressing higher level evaluative questions of effectiveness. However, potential positive social response bias on the psycho-social and child protection CPAs suggest that results from these CPAs should be met with caution. Moreover, when comparing changes in vulnerability status using rigorous statistical approaches, the impact evaluation should break down the HVAT scores by CPA and item scores and search for statistical differences of each. Doing so will provide a more nuanced picture of what impacts have been achieved.  While the distribution of household vulnerability poses no threat to the design of the impact evaluation (and improves explanatory power of the QCA component), the lower seemingly low number of more vulnerable households as well as the percentage of “not vulnerable households” may be a cause for concern. While there are some scaling biases built into the way vulnerability is scored, once accounted for they do not reveal any substantial differences in the distribution of vulnerability across the sample. Rather, the differences may be better attributed to Ugandan economic geography, but this conclusion requires further investigation. Regardless, the high number of “not vulnerable” households enrolled in Cohort 1 does suggest that Sustainable Outcome staff review their selection procedures.  The embedded QCA system is comprehensive, user friendly and well designed. Sustainable Outcomes staff should consider improving the system to capture GPS, avoid duplicates and capture ongoing referrals by implementing partners. Moreover, the program may also wish to consider employing additional M&E staff to manage the wealth and quality of data that is being captured. The QCA system has strong potential to enable Sustainable Outcome implementer to use the results to provide ongoing program feedback, learning loops, mini-cost effectiveness analyses and midcourse program adaptations. The program should consider formally including these steps in its current workplan. 1 Introduction Uganda is currently the second youngest population in the world and the third fastest growing nation in Africa. However, deep and extensive vulnerabilities exist that especially affect children: 96% of children are considered vulnerable, and 62% of those living in poverty are children. Vulnerabilities take different forms, including poverty, malnutrition, and exposure to violence. Low levels of education and high prevalence of HIV/Aids among children themselves as well as within their families exacerbate these vulnerabilities. USAID/Uganda’s Orphans and Vulnerable Children (OVC) programming employs an integrated approach to address these factors. The Sustainable Outcomes for Children and Youth (Sustainable Outcomes) activity aims to economically empower children, youth, and their caregivers to access core services, strengthen systems to provide core services, and improve coordination of community-based clinical and socio-economic services for efficiency and effectiveness along the continuum of care. The activity is a five-year cooperative agreement implemented by a consortium led by Catholic Relief Services (CRS) across 17 districts in Central and Western Uganda. These districts were identified by the United States Agency for International Development (USAID) due to high prevalence rates of HIV and availability of trained community workers to deliver program services. Sustainable Outcomes’ goal is to help a minimum of 625,000 OVC, youth and 101,500 households to access core services for improving health, nutrition, education, and psychosocial well￾being to reducing abuse, exploitation and neglect. Sustainable Outcomes deploys an integrated approach to reduce OVC and Youth vulnerability that operates across multiple levels. Specifically, Sustainable Outcomes aims to achieve the following three Results:  Result 1: Orphaned and vulnerable children, youth and their caregivers are better economically empowered to access core services;  Result 2: Local government and CSOs and informal community structures increase and improve core services for orphaned and vulnerable children, youth and their caregivers;  Result 3: Improved coordination of community-based clinical and socio-economic services for efficiency and effectiveness along the continuum of care. Each Result corresponds to a different level of implementation, beneficiary and support package. For Result 1, OVCs and their care givers at the household level are the main beneficiaries of support, primarily through the formation of savings and internal lending communities (SILCs).2 SILCs are hypothesized to increase household income and savings, which will enable the household and OVC caregiver to better provide for OVCs as well as better access core OVC services outside of the home.3 Activities include increased access to temporary consumption support (by linking them to external resources such as SAGE, GiveDirectly, and WFP, as well as to groups within the community); enroll households in SILCs (in which 15-30 households pool savings on a weekly basis to create social funds and access loans; the returns from which are paid out to members after 12 months); train SILC members on a variety of topics, such as financial management, HIV 2 For a description of SILCs, see Vanmeenen G. (2010). Savings and Internal Lending Communities (SILC) Voices from Africa: The benefits of integrating SILC into development programming. Baltimore, MD: Catholic Relief Services. Drawing upon various descriptions in SILC evaluations, here SILCs can be summarized as model developed by CRS for user-owned, self-managed savings and credit groups that offer households a way to protect assets, smooth cash flow, and increase income. Community members are encouraged to self-select into groups of 15-30 people they know and trust. Compared to traditional accumulating savings and credit associations, SILC groups are more accessible, transparent, and flexible. Each group determines its meeting schedule, regular contributions to a loan fund, loan duration, interest, and maximum loan amount. Members can borrow from the fund at this predetermined interest rate and term. Interest and fines allow the common fund to grow. At the end of a predetermined time period, all or part of the common fund returns to the group members based on the total amount saved by each member. SILC groups are initially supported by project “field agents,” but the goal is institutional and financial independence (note: this was taken from the SILC Impact Evaluation of the STEPS OVC Project in Zambia Final Evaluation Report, found at www.silcevaluation.com). 3 The key assumption in this argument is that if SILCs do in fact increase household income and savings for members who are caregivers of OVCs, some of the benefits will also reach the OVC, e.g. caregivers will use these resources to benefit the child and not use them for something else. 2 prevention and treatment, skills building, and better parenting/caregiving; link girls to access to better education opportunities; and provide additional technical assistance, market access, and training to farmer￾focused SILCs to improve income. For Result 2, district government and CSOs are the main beneficiaries and here sustainable outcomes work with relevant district staff to create functioning OVC coordination mechanism, allocate resources for OVCs and youth, fill key OVC-support positions, conduct performance appraisals of relevant staff, use the OVCMIS for planning, create alternative care panels and transition OVCs from institutional to family based alternative care. The program will also build the capacity of CSOs to comply with national OVC and youth standards, conduct protection pilots and implement improved OVC service delivery. Finally, for Result 3, the primary beneficiary is the referral system, in which the Para-social workers (PSWs) and village health teams (VHTs) sit as the main actors who refer youth and OVCs to the appropriate provider in the service network. Here, activities include a mapping of district level services; training PSWs and VHTs on how to make referrals to multiple types of services, track those referrals, and assist vulnerable households with the development of household action plans; and conducting pilot projects on an Early Childhood Development (ECD) tracking component, ECD development spaces at health centers, and creating very young adolescent groups to discuss sexual reproductive health issues. Figure 1: Sustainable Outcomes Graphic Representation These three results are designed to be mutually supporting, in which SILC households not only benefit from the expected increase in knowledge, access, and support, but also experience improved access to quality OVC and services across an improved body of coordinated service providers operating at local (CSO and informal), social service worker, local government and other service provider (other programs and NGOs) levels. 3 Sustainable Outcomes will be considered successful when 100% of the 101,500 targeted households have left their status as vulnerable behind. System strengthening interventions under Result 2 have been rolled out in all target communities since October 2015. Under Results 1 and 3, households will be reached across three Cohorts: • Cohort 1 (since May 2016): 85% of communities • Cohort 2 (planned for January 2017): 5% of communities • Cohort 3 (planned for January 2018): 10% of communities (tbd. after review) Evaluation Purpose, Aims and Objectives The purpose of this evaluation is to help USAID/Uganda answer overarching questions regarding the effectiveness of Orphans and Vulnerable Children (OVC) programs; understand how different interventions or combinations of interventions of the program contribute to program outcomes; and generate evidence to inform decisions about future OVC programs (see Annex 1 for the Scope of Work). 4 The aim of this evaluation is to assess and understand the impact of participating in programs that seek to improve household socioeconomic security through savings and strengthen institutional childcare service delivery on OVC caregivers, households and children over time. The primary objectives of the evaluation are to answer two complementary and inter-related questions:  Has caregiver participation in Sustainable Outcomes reduced household vulnerability and improved OVC well-being?  What combinations of factors (including those within and outside of Sustainable Outcomes’ control) best identify how this may or may not have happened? Answering the first question suggests a counterfactual research design that compares changes in vulnerability status across various control and treatment groups to demonstrate “what would have been the change if households had NOT participated in Sustainable Outcomes.” This is commonly referred to as an impact evaluation. Answering the second question suggests a comparative research design that combines the diversity of implementation with data on relevant external factors into an analytical model that identifies how various pathways to the outcome can be achieved. Qualitative Comparative Analysis is the method commonly associated with this approach. In what follows below, we review the overall design strategy, the outcome(s) the Sustainable Outcomes will be evaluated against, the type of quasi-experimental design that has been used, the approach to QCA, and finish with a report on the baseline results. Research Strategy Combining a QCA-approach with an impact evaluation presents a unique opportunity to test Sustainable Outcomes’ underlying assumption that caregiver participation in a SILC group, combined with additional support services and an improved referral system, improves household economic status which in turn leads to improvements in child well-being. The combination of this approach yields a design strategy oftentimes referred to as “lumping and splitting”. For the sake of policy decision making, it is oftentimes necessary to “lump” a wide array of outcomes and pathways into two groups (control and treatment) represented with summary statistics based upon measures of central tendency (e.g. the mean, the regression line, etc.). This allows us to make clear, testable statements around program attribution and causal inference (aka internal validity). While useful, lumping seldom includes information on how the multitude of household pathways converged on these results across contexts, and so we are left with little information on how to reproduce the 4 An earlier version of this report was submitted in the form of a concept note. The creation of the note involved a document review of Sustainable Outcomes documents (including the ToR, the proposal, the workplan, the AMELP, regular reports, the baseline survey, and various tools), MEASURE Evaluation OVC tools and evaluations, a baseline study conducted by MEEPP, Government of Uganda OVC strategy, OVCMIS and Vulnerability Index documents, PEPFAR guidance documents, USAID/Uganda portfolio review of the OVC program, and literature on SILCs implemented elsewhere. The study team also spent one week in Kampala in mid-August visiting MEEPP, USAID/Uganda staff and CRS staff. The team leader was supported by the senior M&E education advisor and the education communications officer at the Learning Contract. 4 same results elsewhere. It is therefore necessary to “split” the control and treatment groups up into more refined subgroups to identify how local context matters and how various aspects of a program combine and interact in different ways. This then allows us to better understand how various pathways to the outcome occurred and can occur elsewhere (i.e. external validity). While also useful, splitting does run the risk of identifying pathways that may have been the result of random chance, and therefore our ability to attribute the outcome to these explanations is reduced. As such, combining the strengths of both approaches compensates for their respective weaknesses. This evaluation will use a longitudinal, quasi-experimental and QCA design using a multi-stage cluster sampling approach in which a representative group of households that participate in Sustainable Outcomes will be compared with households that have not yet participated (but will be eligible in the future). These groups will be selected from Cohorts 2 and 3, respectively.5 The evaluation will also incorporate a wide array of implementation and environmental data. Participants and households in both groups will take part in an annual interviewer-administered survey for three study years. Systematic follow up during and after Sustainable Outcomes implementation will allow for retrospective analysis of dose-response, sub-group analysis, and evaluation of the sustainability of both SILC and outcomes of participation. Research Design: Impact Evaluation This baseline report and dataset contains outcome and other data collected from two treatment and control groups to estimate the impact of Sustainable Outcomes on vulnerability and childhood outcomes. The difference between these groups will be the estimates of Sustainable Outcomes’ effects. Additional data collection waves are anticipated in midline (Fall 2017) and endline (Fall 2018).6 This baseline report is designed to lay the groundwork to address three different impact evaluation questions:  Question 1: Does caregiver participation in activities under Results 1 and 3 reduce household vulnerability and improve OVC well-being?  Question 2: Does caregiver participation in activities under Results 1 reduce household vulnerability and improve OVC well-being?  Question 3: Do the combined activities under Results 1, 2 and 3 reduce rates of HIV prevalence and incidence in participating subcounties? In designing the impact evaluation, the evaluation team considered a variety of experimental and quasi￾experimental designs. Many aspects of the Sustainable Outcomes program – such as the voluntary-based SILC group consisting of both direct and indirect beneficiaries – defy random assignment and thus we were unable to advance a randomized control trial design and instead turned to quasi-experimental designs. Originally, we had considered a regression discontinuity design based upon the program’s enrollment criteria, but further exploration revealed that these criteria served more as guidelines than as rigid standards that were enforced consistently across all areas. Finally, we had reviewed the option of using genetic matching techniques across the control and treatment groups, but ruled out this approach as the potential differences in geography separating the groups may undermine the matching assumptions of the model. This baseline is designed to support a difference-in-difference (DID) impact evaluation design. DID designs compare two types of changes: The changes in outcome of a treatment group (or groups) before and after an intervention has occurred with the changes in outcome of a control group (or groups) over the same period of time. This design puts in place two controls: The difference in the before-and-after outcomes for the group(s) enrolled in the program controls for factors that are constant over time in that group (as it is comparing the same group to itself over time); Then, this difference is compared with the difference in 5 In terms of household identification, we will use the same numbering system as CRS and merge our dataset with theirs. We will also use the same confidentiality protocols, with limited rights to the datasets in which household locaters are restricted, and keep this information on a secure server. Any analysis of the data will exclude this information. 6 Note: these times are subject to change according to the implementation cycle. 5 before-and-after outcomes for a group that was not enrolled in the program but was exposed to the same set of environmental conditions over the same time period to control outside time-varying factors. DID thus combines these two counterfactuals (before-and-after comparisons, and comparisons between those who were enrolled and those who were not enrolled) to produce a better estimate of the counterfactual to identify the impact of the program (see Figure 2 below). Figure 2: Difference in Difference Design Crucially, the DID design does not require that each group have statistically insignificant outcome differences at baseline, i.e. statistically significant differences across groups do not pose threats to the internal validity of the design (we find several statistically differences across groups, which is discussed below). Rather, the key assumption for DID is that the outcome across treatment and control groups will follow the same trend over time in absence of the treatment. For Question1, households and OVC/Youth in Cohort 2 subcounties that receive both R1 and R3 treatments (treatment group 1) will be compared to households and OVC/Youth in Cohort 3 subcounties (the control group) who do not receive any treatment until the end of 2017 and 2018. A simple representation of the measure of impact under the experimental design is the interaction effect of treatment and time, or the double-difference estimate, is shown in Equation (1): Estimate of impact of R1+R3 on Vulnerability = (YR1+R3,t2 – YR1+R3,t1) - (YC,t2 – YC,t1) where, Y = outcome of interest; T = treatment group; C = comparison group; t1 = baseline or beginning of study; and t2 = middle or end of study. As discussed above, this question will be addressed using a difference in difference design to compare differences between households enrolled in the treatment group of Cohort 2 and those who reside in the sub-counties associated with the planned implementation of Cohort 3. For Question 2, the effects of R1 will be estimated by making comparisons between households and OVC/Youth households that receive both R1 and R3 (treatment group 1) with those that only receive R3 (treatment group 2). The difference between the two groups will be the estimates for the R1 effect, as shown in Equation (2): Estimate of impact of R1 on Vulnerability = (YR1+R3,t2 – YR1+R3,t1) - (YR3,t2 – YR3,t1) For this question, the study also uses a difference in difference design to construct the control and treatment groups. To avoid contamination in the treatment group 2 (only R3), households were selected in Cohort 2 subcounties that did not contain parishes where R1 will be implemented (see below). For Question 3, the study will estimate the combined effects of R1+R2+R3 by comparing HIV Prevalence and Incidence rates across Sustainable Outcomes subcounties and various high risk groups, such as girls age 10-24, within those subcounties with rates in non-Sustainable Outcomes subcounties across the 17 districts, 6 as shown in Equation (3). This data will be collected from secondary sources; namely, the Ministry of Health DHMIS2 database. Estimate of impact of R1+R2+R3 on HIV Prevalence and Incidence = (YR1+R2+r3,t2 – YR1+R2+R3,t1) - (YC,t2 – YC,t1) For this design, the study also relies upon a difference in difference approach to construct the control and treatment group.7 Figure 3: Impact Evaluation Design All three equations will be estimated using a simple regression formula presented below: 𝑌𝑡2 − 𝑌𝑡1 = 𝛼 + 𝑇 𝛽 + 𝑋 𝛿 + 𝜀 where T is a dummy for assignment to treatment group (e.g. R1+R3) and X are characteristics of the unit of analysis (households, OVCs/Youth and HIV rates). This regression can be estimated using ordinary least squares (OLS). Research Design: QCA Whereas the DID design will confirm whether or not Sustainable Outcomes has worked, this study uses QCA to identify how it has worked. QCA is a data analysis technique for identifying which combinations of factors best explain how an outcome was (or was not) achieved. Factors here refer to a wide range of potential variables, e.g. continuous variables, categorical variables, qualitative changes in condition, M&E indicators, program outputs, relevant external variables outside of the program’s control, context variables, etc. These factors are built into a model that identifies how every possible combination (both observed and unobserved) can be associated with the outcome of interest. QCA then applies an algorithm to this model to identify sets of necessary and sufficient combinations of conditions that can be logically associated with explaining how the outcome was achieved. Crucially, QCA typically identifies several combinations of factors 7 The difference in difference designs are very straightforward comparisons of changes in treatment group with changes in a comparison group, but lacks random assignment. Instead, equivalence between the two groups will be found by adjusting for previous rates of change in HIV prevalence and incidence rates for control and treatment groups. 7 simultaneously, which can be used to account for various contexts and explain how there may be multiple pathways to achieving the same outcome. Whereas the impact evaluation is limited to testing the impact of only a few treatment arms at the Results level (i.e. R1, R1+R3, R1+R2+R3), QCA will be used to unpack how the various activities that fall underneath each result combine to achieve success. For this evaluation, there are several sets of factors that will be built in the model. For example, Result 1 consists of eight different activities. This study will use QCA to determine if all of these are necessary for success, or if a “leaner and meaner” essential package, or packages, can be identified. Moreover, using additional context variables, such as the presence of other donor-supported OVC programs, QCA will help identify under which conditions one essential package is more preferable to another. It should be noted that all of these factors will be modeled both in situations where the outcome has occurred, as well as in situations where it has not occurred to better understand why Sustainable Outcomes efforts may have been insufficient to be successful (see below in the results section). Finally, these outcomes (and lack thereof) are modeled across control and treatment groups, which may also reveal how households not (yet) enrolled in Sustainable Outcomes found alternative ways to reducing their vulnerability and improving OVC/Youth outcomes. Such insights may provide valuable lessons for adapting Sustainable Outcomes’ own package of services. Specifically, QCA will be able to address the following sets of evaluative questions: 7. What combinations of factors best identify how Sustainable Outcomes reduced household vulnerability and improved OVC/Youth outcomes? 8. What combinations of factors explain where Sustainable Outcomes did NOT reduce household vulnerability and improved OVC/Youth outcomes? 9. What external factors, especially additional services provided as the result of Sustainable Outcomes’ efforts to improve the referral system, are relevant for reducing household vulnerability and improving OVC/Youth outcomes? 10. Which activities implemented under Sustainable Outcomes may not be relevant for reducing household vulnerability and improving OVC/Youth outcomes? 11. How did households and OVC/Youth improve in subcounties where Sustainable Outcomes did not (yet) implement R1 and R3 activities? 12. What are the key factors of R2 that explain how HIV prevalence and incidence rates may have declined in Sustainable Outcomes subcounties relative to subcounties outside of the program? As with the impact evaluation, QCA will be able to address these questions at the household, OVC aged 0-9, and youth aged 10-17 and sub county levels. QCA will use the same outcome indicators as the impact evaluation. Yet unlike the impact evaluation, the QCA will incorporate a much larger number of factors to explain the outcome without requiring an increase in sample size. However, QCA is limited by the total number of relevant factors it can process, typically up to 10. To deal with this issue, we will run a series of multi-stage iterations of the model using various QCA parameters of fit (e.g. coverage and consistency) to reduce the total number of factors (approximately 50) to more manageable levels.8 QCA provides answers to the above listed evaluative questions in the form of solution sets that suggest further exploration, and invite Sustainable Outcomes staff to participate in the search for explanations. In doing so, QCA provides clues to improve the external validity of the findings so that they may be potentially generalized elsewhere. Specifically, QCA offers a number of techniques - such as identifying underlying 8 We use parameters of fit to understand if adding or dropping a given factor improves the overall explanatory power of the model. If there is no change to the parameters, the factor can be dropped as irrelevant. We also make analytical distinctions between context and proximate factors, and run them as separate models under what is known as two- or multi-stage QCA. The results of these two models are then combined into a meta-model that includes both levels of factors. 8 commonality, exploring irrelevance and failures, interrogating seemingly similar factors, and exploring contradictions in the model – that reveal how the underlying causal mechanisms of Sustainable Outcomes can be recreated elsewhere. Below in the baseline findings section we describe how the additional information required for the QCA analysis will be collected throughout the life of Sustainable Outcomes. This data will also be used to support the analysis of the impact evaluation, but QCA requires that we subject this data to an additional step of specification known as “thresholding,” in which qualitative anchors are placed on the factor data to make categorical distinctions between the presence and absence of change.9 Many of the relevant factors we have identified lack clear qualitative anchors. In such cases, we adopt a collaborative approach with technical experts and other stakeholders to identify where to draw substantive boundaries on the data to represent where meaningful change occurs. The advantage is that even if groups disagree on where a certain boundary is drawn, we can easily demonstrate the implications on the results of a wider variety of competing perspectives. What is important is that the investigator clearly identifies what the anchor is and how it was derived. Outcome: Defining Vulnerability The primary outcome measure is socioeconomic household vulnerability status. Household vulnerability is defined as the inability to cope and thrive with economic and other shocks. Operationally this is captured through the Household Vulnerability Assessment Tool (HVAT), which is administered at baseline and will be re￾administered to the same households across all three groups at the end of 2017 and 2018. The HVAT measures vulnerability across economic, food security, nutrition, health, WASH, shelter, education, psychosocial support, and child protection dimensions, and yields a composite score corresponding to “slightly”, “moderately”, and “critically” vulnerable categories .10 The HVAT contains question modules directed to the caregiver, children aged 0-9, and youth aged 10-17. The HVAT was chosen because it is the official vulnerability tool currently endorsed by the Ministry of Gender, Labor and Social Development. The HVAT is also the primary household vulnerability assessment tool used by the national OVCMIS, to which all OVC and youth-related programs are required to report.11 Finally, the HVAT is the primary outcome indicator used by the Sustainable Outcomes program to gauge their overall success, was captured under Cohort 1 and is regularly reported on as per the Sustainable Outcomes’ Monitoring, Evaluation and Learning Plan. Additionally, eight secondary outcome measures will be captured at the OVC and Youth wellbeing level for children aged 0-9 years (with questions directed to the primary caregiver) and youth aged 10-17 years. These measures are derived from PEPFAR’s Monitoring, Evaluation and Reporting (MER) Essential Survey Indicators for OVC Programs document (2014) and elaborated using the MEASURE Evaluation Guide (2015) for OVC program outcomes. These include:  Percent of children whose primary caregiver knows the child’s HIV status 9 One way of thinking about how anchors work is to compare it to the Celsius temperature scale. Although it is a continuous scale, the Celsius scale contains two “anchors” – 0 C and 100 C – that represent qualitative changes in the condition of water, i.e. to ice and vapor, respectively. QCA takes this same idea of qualitative anchors and applies it to data around social life. Socially-anchored data abounds in our industry, such as the “extreme poverty” ($1.90/day) threshold the World Bank imposes on daily income. QCA anchors M&E and other data much the same way as the World Bank and Celsius examples do by identifying “thresholds of social change” (we will use the terms threshold and anchor interchangeably). 10 It should be noted that the aforementioned Impact Evaluation that focused more narrowly on caregiver participation in SILCs adopted percentage of children aged 2-17 who have gone a whole day and night without eating in the last four weeks as its primary outcome measure, which was an item that comprised the Household Food Insecurity Access Scale (HFIAS) and similar to the HVAT. For this evaluation, we adopt the wider measure to account for the system-strengthening and improved service delivery aspects of Sustainable Outcomes, which include but go beyond the more narrow SILC focus. 11 The OVCMIS database can be found here: http://ovcmis.mglsd.go.ug/home.php?linkvar=Data%20Collection%20Tools&&action=Data%20Collection%20Tools 9  Percent of children < 5 years of age who are undernourished  Percent of children too sick to participate in daily activities  Percent of children who have a Birth Certificate  Percent of children regularly attending school  Percent of children who progressed in school during the last year  Percent of caregivers who agree that harsh physical punishment is an appropriate means of discipline or control in the home or school  Percent of households able to access money to pay for unexpected household expenses Finally, two HIV-specific measures are captured at the sub county level over time. These are HIV prevalence and HIV incidence rates, and can be broken down into specific subgroups, such as girls ages 10-17. We will use data from the DHMIS2 database. Power Analysis and Sample Size Calculations This study does not randomly assign households into control and treatment groups, but does rely upon quasi￾experimental methods that must be adequately powered to detect changes in the outcomes over time. Moreover, this study also randomly selects households from each group from various program population lists using a cluster sampling design. These two factors – adequate power and random selection – were used as the basis of the sample size calculation for each group. Sample size calculations for Impact Evaluation Questions 1 (R1+R3) and 2 (R1) are based upon the assumption that the main impact variable – household vulnerability – is a categorical variable with the following four potential values: “not”, “slightly”, “moderately”, and “critically” vulnerable. As such, we use the following formula to calculate the power of the design: 𝛽 = Φ ( |𝜏|√𝑛 2𝜎 − Φ−1 (1 − 𝑎 2 )), 𝑤ℎ𝑒𝑟𝑒:  𝜏 denotes the (expected) treatment effect (think of this as the difference in the mean level of the outcome between treated and control units);  Φ is the Cummulative Distribution Function of the Normal;  𝜎 is the (expected) standard deviation of the outcome(assumed to be the same for control and treated);  𝑎 is the significance level;  𝛽 stands for the estimate of statistical power, ranging from zero to one (probability of detecting a non-zero effect). We assume a significance level of 𝑎 =0.05. As the design seeks to test multiple treatment arms, it requires that we increase the sample size to reduce the probability of committing a type-I error, i.e. falsely concluding that a treatment with zero effect had a non-zero effect. In doing so, we adjust our significance level by the number of tests we will execute (2 treatments) using the Bonferroni adjustment rule, which translates to 𝑎 2 ⁄ . This adjustment rule tells us how to increase the sample size needed to remain at the conventional 0.8 level for β. When calculating the sample size needed to adequately power both treatment arms, we avoid the mechanistic procedure of running the exact formula because we cannot assume that the sampled vulnerable households will conform to parametric assumptions that follow fixed probability distributions. Instead, we derive the sample size by running 100 simulations to identify range of sample sizes that meet the conventional power 10 threshold of 0.8 level for 𝛽. Figure 4 below presents the simulation-based estimates on the number of observations needed for a given level of power to detect both effects and rank them in terms of magnitude. Put differently, the sample size is adequately powered to test both hypotheses against the null AND test if the differences between the two are also significant. Figure 4: Power Analysis Simulations The results of the power analysis reveal a sample size of 2,214 (again equivalent to an absolute difference of 10 percent one at alpha=0.05, two-sided, power=0.80).12 Here we also increase by 10 percent for nonresponse and sampling error, yielding a total of 2,436 observations (distributed evenly across each treatment group and the control group, i.e. 812 households per group). In terms of random sampling of each group, we obtain very similar sample size results using a cluster sample design following PEPFAR guidelines on OVC essential indicators. In estimating the sample size, we assumed that the true value of proportional indicates as (a) percent of children <5 years of age who are undernourished; (b) percent of children regularly attending school, etc. is 50%. Following Monitoring and 12 The simulations are based upon the following assumptions: a relatively conservative standard deviation of 1.2 (there is adequate room for households to vary in vulnerability), a mean of zero (i.e. the difference between the control and treatment groups is zero at baseline), and an Average Treatment Effect between .2 and .4 (this is because it is a categorical variable, which translates to 1/20th and 1/10th, respectively, aka powered to detect a minimal effect size that falls between 5%-10%). 11 Evaluation of Emergency Plan Progress (MEEPP) research, these assumptions can be based on data from Uganda District Health Survey 2011 (UBOS & ICF, 2012).13 Assuming a study design effect of 2 and a potential non-response of 5%, a confidence level of 95%, and a sampling error of 5%, we find: 810 0.95 2.0 0.05 (1 ) (1.96) 0.5 0.5 2 2 2 2 / 2         R Deff d Z P P n  where Z = 1.96 at 95% confidence level, p = 0.5 d = relative standard error i.e. (10%*0.5) = 0.05 DEFF = Design Effect of 2 R = response rate of 95% This corresponds to 810 households that will be sampled from 30 parishes in sampled subcounties from each of the OVC programs. On average, 27 households will be selected per each of the 30 parishes/clusters. As MEEPP has noted, this follows the “30 x n” design described in the MEASURE Evaluation Guide (2015). Overall, this yields a total of 2,430 households across the three groups (R1+R3, R1, and Control). Given the similarity of the power analysis and the cluster sampling design, we targeted a sample size of 2,436 households for this baseline. Sampling Strategy Household sampling selection was designed to be both representative of each control/treatment group area and sufficiently powered to detect changes in household vulnerability across each group. In defining each area, the survey team had to align household sampling selection with Sustainable Outcomes’ implementation process and variations across subcounties. 14 Although the sampling procedure was consistent across each group (systematic sampling with a random start), we used different procedures for the creation of each sampling frame. In doing so, we conducted a random selection of households based upon four programmatic lists provided by Sustainable Outcomes: 1. List A: Master Implementation List of all Districts, Subcounties and Parishes in the Sustainable Outcomes program area (all of which have received R2 activities); 2. List B: A list of locations in which all Cohort 1 households receive R1 and R3 activities (to the village level); 3. List C: A list of all Parishes which are expected to receive R3 services moving forward across all three regions (Central, West and Southwest) starting in January 2017. 4. List D: A list of target locations in which all Cohort 2 households are expected to be enrolled and receive R1 and R3 services (to the parish level). Note: this list only identified target areas; it did not identify actual households to be sampled. Note also that this list only identified parishes in the West and Southwest regions of Uganda. From these lists, we created three new lists: 5. R3 Master Sampling List: We created our master sampling frame for R3-only subcounties and parishes in List E. We modified List C by subtracting all of the subcounties from Lists B – i.e. those that were receiving R1 and R3 support under Cohort 1 – and List D – those parishes and 13 See MEEPP PEPFAR Uganda – OVC Outcomes Indicator Survey, Survey Protocol, May 24, 2016. 14 Originally, Sustainable Outcome program staff anticipated enrolling approximately 14,000 households into Cohort 2 with a strong focus on Result 1 activities at the household level. These households were to be selected in new subcounties contiguous to those already enrolled in the program, and would be identified by lists generated by local CSO and government partners. The survey team then planned to randomly select from these groups for the R1R3 treatment arm. However, due to implementation delays, staff turnover and other factors, the master sampling frame lists were not ready at the time of selection. 12 subcounties planned for R1 support under Cohort 2 – to create a master sampling frame of parishes for the R3 treatment. As such, this list contained the list of subcounties and parishes that will only receive benefits from R3 moving forward (controlling for R2), will not receive R1 services this year, and have not received R1 and R3 services previously. This list contained subcounties that cut across all three regions (Central, Southwest and West), shared two districts with the R1/R3 sampling frame (i.e. Kabale and Kamwenge, but no shared subcounties), and shared four districts with the control group (see below – Gomba, Hoima, Luweero, Mityana). 6. R1R3 Master Sampling List: We created this list from List B, but cross checked it with the list of areas in which SILCs under Cohort 1 were currently operating. We removed 2 areas that contained an overlap to ensure that there would be no “contamination” from Cohort 1 into Cohort 2. 7. Control Group List: We created our master sampling frame for control group subcounties and parishes using List A. However, we first modified List A by subtracting all the subcounties found on Lists B, C, and D to create a list of subcounties that are still in the Sustainable Outcomes area of interest, but as of yet have not received any benefits at the sub county level or lower and are not targeted to any kind of R1 or R3 support under Cohort 2. This list contained subcounties that were only located in the Central region of Uganda. As the survey team used the same household selection criteria as the program, senior management at Sustainable Outcomes agreed that the households sampled for the baseline would be eligible to enroll in Cohort 2, and that the households sampled for the control group would be eligible for enrolment under Cohort 3. It should be noted that subcounties from Wakiso and Kamwenge were dropped from the sampling frame. Wakiso was dropped as its close proximity to Kampala may suggest a different trend line of vulnerability than in the other districts, and we therefore used Wakiso as a pilot district to test the instruments. Kamwenge was dropped based upon feedback from the Sustainable Outcomes team, who suggested that they faced some challenges with their local partners in this district and implementation was unlikely to continue. We then used a two stage cluster sampling approach to select households. In the first stage, we selected 31 parishes from each group’s sampling frame. In doing so, we first randomized the order of each administrative category (e.g. the order of districts was randomized, then the order of subcounties, then the order of parishes and finally villages) to ensure there were no biases inherent in the list ordering. We then used systematic sampling with a random start to sample 31 parishes from each list (30 for the sample and one back up) from each list for a total of 93 Parishes for the entire sample. In the second stage, 27 to 30 households in the control group and R3 treatment were selected in each Parish using the “random route” technique.15 In the R1R3 treatment group, the survey team received lists of potential vulnerable households from Sustainable Outcomes local representatives – usually Para-social workers – and randomly selected a household from this list as the enumeration starting point. In most cases, these lists were incomplete and inaccurate, in which case the survey team relied upon the random route method to complete the survey quota for that parish area. Households were first screen for consent and eligibility (using the HVAT screening tool). Survey teams then met with the primary caregiver of each household (self-identified), as well as randomly select one youth respondent (age 10 to 17) and one child respondent (0-9, in which the caregiver provides the information) using the Kish Grid selection method, 15 Enumerators were instructed to start at the designated center of each Parish and to use random number generation to select with direction they faced. The starting point was the center of the parish and skip was of nth households. Nth house skips were determined by first taking an estimate of the total HH of the Parish (obtained when seeking permission from the LC1 representative) and dividing by 27. In areas with two enumerators, one would take different direction from the other with the same skip. They walked in the same direction on the same side of the street until they reached a crossroad or junction. They were then instructed to turn right and proceed toward the right-hand side of the road, following the new street until they reach the next road that goes off to the left side and call at the houses on the left￾hand street side. 13 which was built into the form and automatically applied once the enumerator had taken an initial household inventory. Survey Implementation Prior Approvals SoCha obtained Institutional Review Board (IRB) approval to conduct the survey from The AIDS Support Organisation (TASO) IRB at the beginning of November 2016 and obtained subsequent approval from the Uganda National Council for Science and Technology one week after. The survey team also obtained letters of support endorsed by USAID/Uganda, which were presented to district Chief Administration Officers (CAOs) as part of official requests to obtain permission to conduct the surveys in the target areas. Team Mobilization In November 2016 until middle of December 2016, the survey team was mobilized for implementation. Forty enumerators and eight field managers were mobilized and attended the one week training and field testing in Kampala and Wakiso. Out of these, thirty enumerators and six field managers were selected based upon their performance during the training, with a ratio of one field supervisor to five enumerators. Participants also received ethics, confidentiality, child protection and rights of the respondent training by a TASO Institutional Review Board representative. Finally, enumerators were trained on how to measure upper arm circumference (MUAC) on children under five years old using MUAC tape by a health officer from the Mulago National Referral Hospital. Survey Instruments The baseline survey was designed to collect sampling information at three levels of analysis: Household, OVC (0-9), and Youth (10-17). It will also collect information across three dimensions: Vulnerability, OVC/Youth Status and Situational Analysis (see Annex 2 for the survey instrument). Each dimension uses a questionnaire module that was already been tested and IRB approved in Uganda. They are as follows:  Vulnerability: As mentioned above, we used the HVAT as the primary tool for collecting information on household and youth/child vulnerability. Additionally, we incorporated questions from the Household Orphans and Vulnerable Children Vulnerability Prioritization Tool (HVPT), which contains additional items on household vulnerability. Combined, these tools contain items across six areas of inquiry.16  OVC/Youth Status: We also included questions to capture the eight OVC/Youth PEPFAR outcome indicators. The tools have already been tested in various African countries and recently MEEPP collected these indicators on a sample of Cohort 1 (they plan to run the same survey again in two years).  Situational Awareness: Finally, we incorporated a situational awareness module that captures current levels of social assistance provided from external sources. It contains questions about financial service, health facility, community based and government assistance access and use. The current situational awareness tool was applied by Sustainable Outcomes to Cohort 1, but they have not determined if/when they will apply the tool to Cohort 2. We incorporated additional questions regarding specific services provided by other activities and initiatives, which can be identified using the OVC MIS. The survey instrument was administered using handheld tablets (with power banks) using Survey CTO. Survey CTO is a dual offline/online survey platform based upon Open Data Kit (ODK), and is designed for field settings with little or no connectivity. Forms were programmed and designed to synchronize with the main server either in realtime or at the end of each workday when connectivity was limited. Survey 16 These are Economic Strengthening, Food Security and Nutrition, Health, WASH and Shelter, Psychosocial Support and Basic Care, Child Protection and Legal Support, Education. 14 instruments and consent forms were translated into Luganda, Runyankole, and Runyoro/Rutoro and programmed into the forms. Consent forms were created for primary care givers, emancipated youth respondents (age 10-17), and child assent forms (signed by the primary care giver). Each respondents was given a hard copy of the consent form and the enumerator kept an additional hard copy that was submitted to the field supervisor at the end of each day and was brought to Kampala at the end of each week. Quality Control For quality control purposes, we put into place several measures. First, field supervisors accompanied enumerators as observers on eight percent of the interviews. Second, field supervisors also revisited households and asked questions on a select number of items for 10% of the households surveyed. Third, the survey manager rechecked and observed field supervisor performance for five households per field supervisor. Fourth, the survey manager performed daily consistency checks on enumerator performance and was instructed to review enumerators who fell outside of two standard deviations of the norm of responses. One enumerator was replaced during the survey due to a high number of incomplete interviews. Originally, the survey manager was to also observe average interview times across enumerators; however, we later learned that the time stamp system used by Survey CTO re-wrote over previous time stamps when editing occurred. As such, we were unable to monitor this performance metric. Finally, enumerators were required to record the GPS coordinates of each household. Initially, the survey form did not contain any checks on GPS reading accuracy, but a best practice standard of within 10 meter accuracy was imposed during the course of the survey, requiring a resampling of 523 households that did not meet this standard. Issues Encountered The main issue encountered during the implementation of the survey involved an encounter with a rebel group in Kasese. A militia group known as Kirumiramutima attempted to take over the district government. This group engaged in a fire fight with the Uganda People’s Defense Force (UPDF) responded while the survey team was at the district center. Enumerators were instructed by Kirumiramutima representatives to leave the district immediately or face punishment. The survey team quickly complied and one of the assigned parishes was unable to be sampled. This parish was replaced by a replacement parish from Isingiro. The other main issue involved the practices of a small number of Para-social workers selected by Sustainable Outcomes to support implementation. Some PSWs informed the enumerator team that they required compensation for their assistance with household selection, as they were volunteers. The enumerator team was limited to only providing a transportation allowance, which involved a few thousand shillings to pay for Boda rides. A small handful of PSWs also attempted to direct enumerators to include their family households on the list of those sampled. In response, the enumerators ignored the list of households provided by these PSWs and selected houses using the random route method. This incident was limited to less than ten of the surveyed parish areas. Finally, a programming error in the Survey CTO form over wrote child inventory information in households that contained more than one child. This affected close to one thousand households, which required the enumerators to return to these households to retake child inventory information after the form was fixed. This issue extended the time of the survey by three days. 15 Baseline Survey Results Survey Data Group Distribution The survey team conducted household surveys for 2,629 households across 82 parishes in 15 districts across the three regions. The control group included 834 households, Treatment group 1 (R3 group) assigned to receive the village Para-social work training included 863 households and the Treatment group 2 (R1R3) that will receive the combined SILC intervention included 932 households. Overall, the baseline over-sampled by around 8.4% of households, in which the control group and R3 were over sampled by around 6-7% while the R1R3 treatment group was over sampled by around 15%. The reasons were for oversampling in the control and R3 treatment group were due to lag times in survey form reporting. Survey managers would typically receive notice that a given survey area had been completed only after daily uploading of completed survey forms to the master dataset had occurred. In some instances, this meant that a few higher-performing enumerators visited additional households after their parish quota was completed. The oversampling of R1R3 was also connected to a change in administrative boundaries that occurred after the initial parish selection had been made. Three parishes had been further split into six new parishes. Rather than re￾randomly select new parishes based upon these additions to the sampling frame, the survey team sampled all six parish areas. The result added close to 100 additional households to the R1R3 treatment group. Looking at the distribution of households across regions, we see the following: HH Group Distribution by Region Region Control Group R3 group (Treatment 1) R1R3 group (Treatment 2) Total Central 834 460 0 1294 Western 0 317 0 317 Southwest 0 86 932 1018 Total 834 863 932 2629 All households from the control group are selected from Central Region, while all households for the R1R3 group are from the Southwest, as this was the main region in which Sustainable Outcomes program staff will be providing R1 services. The R3 treatment group is spread across all three regions, with a higher concentration in Central and the fewest in the Southwest. Broken down by district, we find the following distribution: HH Group Distribution by District District Control Group R3 group (Treatment 1) R1R3 group (Treatment 2) Total Bushenyi 0 0 112 112 Gomba 54 148 0 202 Hoima 149 83 0 232 Isingiro 0 50 0 50 Kamwenge 0 83 0 83 Kanungu 0 0 237 237 Kasese 0 90 0 90 Kibaale 0 144 0 144 Luweero 229 90 0 319 16 Mityana 139 139 0 278 Ntugamo 0 0 266 266 Rakai 263 0 0 263 Rukungiri 0 0 84 84 Kabale 0 36 172 208 Kiruhura 0 0 61 61 Total 834 863 932 2629 Note that no districts overlap the control and R1R3 treatment group areas, but that R3 households overlap with the control group households in four districts and in Kabale with the R1R3 treatment group. A full list of all sampled parishes is found in Annex 4. Group Comparisons of Household Socio-Demographic Characteristics We ran statistical comparisons across the various means of each group to test for significant differences. For continuous variables, we ran F-tests. Otherwise, we used Chi-Squared (χ²) tests for categorical variables. The actual value of each test should not be directly interpreted, but instead should be understood in terms of the significance level of the result, represented as * p≤0.05, ** p≤0.01, *** p≤0.001. Results with none of these three designations should be understood as insignificant differences across the groups. The descriptive statistics for socio-demographic characteristics for the entire sample and by study group are presented in Table 1. Table 1: Household’s socio-demographic characteristics and economic vulnerability TOTAL CONTROL GROUP R3 Treatment Group 1 Village Para￾social workers (PSWs) R1R3 Treatment Group 2 Savings Intervention /SILCs + Village PSWs Comparison statistics F or χ² (N=2,629) (n=797) (n=771) (n=1,061) Frequency (Percentage, %) or Mean (SD) Socio-Demographic Characteristics of the Households Region 2992.66*** Central 1251 (47.6) 794 (100%) 454 (58.9) 0 Western 303 (11.5) 0 281 (36.4) 22 (2.1) South Western 1075 (40.9) 0 36 (4.7) 1039 (97.9) HH STRUCTURE AND SIZE Age of Household Head (in years), mean min/max 12-100 48.5 (SD=15.4) 49.29 (15.34) 47.14 (14.68) 48.91 (15.91) 4.44* Total number of people in HH, mean min/max 1-20 5.22 (SD=2.27) 5.72 (2.48) 5.55 (2.44) 4.61 (1.8) 69.12*** Number of children (0-17) in HH, mean mix/max 1-14 3.69 (SD=1.97) 4.06 (2.11) 3.99 (2.13) 3.19 (1.6) 59.57*** Child-headed HH [Youth (aged 10-17) head of HH] 14 (0.5%) 4 (0.5%) 3 (0.4%) 7 (0.7%) 0.64 Household has or cares for orphans 1957 (74.4%) 582 (73.0%) 569 (73.8%) 806 (76.0%) 2.31 Household head or caregiver has a 444 (16.9%) 104 99 241 43.03*** 17 severe disability (13.0%) (12.8%) (22.7%) HH ECONOMIC STATUS Household Income 116.45*** less than 50K 1,598 (60.8%) 396 (49.7%) 456 (59.1%) 746 (70.3%) between 50-99K 588 (22.4%) 196 (24.6%) 172 (22.3%) 220 (20.7%) between 100-149K 233 (8.9%) 107 (4.1%) 76 (2.9%) 50 (1.9%) between 150K-200K 105 (4.0%) 43 (1.6%) 36 (1.4%) 26 (1.0%) above 200K 105 (4.0%) 55 (2.1%) 31 (1.2%) 19 (0.7%) At least one HH member has any form of employment 1553 (59.1%) 581 (72.9%) 491 (63.7%) 481 (45.3%) 152.62*** Main Contributor to HH income 4.90 Parent 1743 (66.3%) 518 (65.0%) 526 (68.2%) 699 (65.9%) Child 58 (2.2%) 22 (2.8%) 12 (1.6%) 24 (2.3%) Grandparent / Non-relative 736 (28.0%) 226 (28.4%) 205 (26.6%) 305 (28.7%) Relatives 92 (3.5%) 31 (3.9%) 28 (3.6%) 33 (3.1%) Main source of HH income 178.99*** Casual labour 1126 (42.8%) 263 (33.0%) 301 (39.0%) 562 (53.0%) Formal job or business 586 (22.3%) 282 (35.4%) 175 (22.7%) 129 (12.2%) Informal job 634 (24.1%) 191 (24.0%) 216 (28.0%) 227 (21.4%) Remittance 283 (10.8%) 61 (7.7%) 79 (10.2%) 143 (13.5%) Asset ownership HH belongs to any financial savings or lending group 1034 (39.3%) 276 (34.6%) 256 (33.2%) 502 (47.3%) 47.85*** HH has access to land 2278 (86.6%) 751 (94.2%) 663 (86.0%) 864 (81.4%) 64.821*** HH has domestic animals 1654 (62.9%) 597 (79.5%) 498 (64.6%) 559 (52.7%) 97.621*** Ownership of other HH assets (cooking stove, beds, blanket, mattresses, mosquito net, shoes, cooking utensils, furniture) 114.411*** None or Almost none 211 (8.0%) 42 (5.3%) 77 (10.0%) 92 (8.7%) Yes, some of them 1513 (57.6%) 412 (51.7%) 377 (48.9%) 724 (68.2%) Yes, most of them 905 (34.4%) 343 (43.0%) 317 (41.1%) 245 (23.1%) The last time there was an unexpected urgent household expense, HH was able to access money to pay for that expense 895 (34.0%) 305 (38.3%) 319 (41.4%) 271 (25.5%) 58.94* FOOD SECURITY Over the past month, did anyone in the household ever go without food for a whole day because there wasn’t enough? 142.02*** Yes, more than 5 times 511 (19.4%) 103 (12.9%) 147 (19.1%) 261 (24.6%) Yes, 1-4 times 752 (29.0%) 202 (25.3%) 163 (21.1%) 397 (37.4%) No 1356 (51.6%) 492 (61.7%) 461 (59.8%) 403 (38.0 %) Number of meals per day 133.44** Not everyday 31 (1.2%) 4 (0.5%) 9 (1.2%) 17 (1.6%) 1 meal per day 604 (23.0%) 148 (18.6%) 161 (20.9%) 295 (27.8%) 2 meals per day 1558 (59.3%) 480 (60.2%) 406 (52.7%) 672 (63.3%) 3 meals or more 436 (16.6%) 165 (20.7%) 195 (25.3%) 76 (7.2%) Main source of food consumed by HH 144.37*** Home grown 1562 (59.6%) 461 (57.8%) 499 (64.7%) 602 (56.7%) 18 Bought 788 (30.0%) 304 (38.1%) 218 (28.3%) 266 (25.1%) Given for work 235 (8.9%) 23 (2.9%) 39 (5.1%) 173 (16.3%) Donated 44 (1.7%) 9 (1.1%) 15 (1.9%) 20 (1.9%) HEALTH, WATER, SANITATION AND SHELTER HH has access to safe drinking water within 30 min 1479 (56.3%) 462 (58.0%) 429 (55.6%) 588 (55.4%) 1.37 Main source of water 5.09 Unprotected water source (river, lake, pond, unprotected well) 1006 (38.3%) 314 (39.4%) 307 (39.8%) 385 (36.3%) Protected public water source (public taps, borehole, rainwater, protected spring/well) 1604 (61.0%) 479 (60.1%) 460 (59.7%) 665 (62.7%) Private water source (Private Connection) 19 (0.7%) 4 (0.5%) 4 (0.5%) 11 (1.0%) HH has access to a public health facility within 5 km 2119 (80.6%) 614 (77.0%) 588 (76.3%) 917 (86.4%) 38.78*** Caregiver knows HIV status for all children 1,295 (49.3%) 388 (48.7%) 394 (51.1%) 513 (48.4%) 1.51 All HH members sleep under a mosquito net 1322 (50.3%) 367 (46.0%) 339 (44.0%) 616 (58.1%) 50.87*** SOCIAL ASSISTANCE AND SERVICES Types of HH based services cash direct financial support 33 (1.3%) 6 (0.8%) 17 (2.2%) 10 (0.9%) 8.07* Loan Direct Financial Service 99 (3.8%) 23 (2.9%) 42 (5.4%) 34 (3.2%) 8.64* Parental Counseling 93 (3.5%) 29 (3.6%) 28 (3.6%) 36 (3.4%) 0.11 Types of community based services Savings groups 455 (17.3%) 126 (15.8%) 137 (17.8%) 192 (18.1%) 1.83 Parenting Programs 53 (2.0%) 15 (1.9%) 15 (1.9%) 23 (2.2%) 0.22 Government SAGE Program (pension program) 14 (0.5%) 6 (0.8%) 3 (0.4%) 5 (0.5%) 1.11 * p≤0.05, ** p≤0.01, *** p≤0.001 Household Structure and Size Households, on average, included five people (SD=2.27) and some households included as many as 20 people. An average household had three children (SD=1.97), with some household caring for as many as 14 children under the age of 17. Households from the control group had, on average, greater family size (χ2 =69.12, p<0.001). The average age of household head was 49 years of age (SD=15.4) and less than 1% of households had a youth head of household (under the age of 17). Household Economic Status Households in the sample differed significantly in income (F=116.45, p<0.001). Generally, households in the sample were in the lowest two income brackets. In the total sample (N=2,629), the majority of households (1,598, 60.8%) earned less than 50,000 UGX per year, while 22.4% (588) of the households earned between 50,000 and 99,000 per year. The remaining 16.9% of households earned over 100,000. The same pattern held true for the control group and treatment groups. Who Contributes to Household Income? 19 There were significant differences (χ²=152.62, p<0.001) between the groups as to whether or not there was one member of the household who had any form of employment. In the general sample (N=2,629), 1,553 (59.1%) households reported there being at least one member of the household who had employment, while 72.9% (581) of households in the control group answered similarly to this question. In the treatment group R3 there were 491 (63.7%) households who had a member working and slightly fewer (481, 45.3%) households with someone working in the R1R3 group. While there were no significant differences between the groups (χ²=4.90, p>0.001) as far as who was the main contributor to household income, parents and grandparents fulfilled this role in 66.3% (1,743) and 28.0% (736) of households, respectively. There were significant differences between the groups as to the nature of the work. In the total sample (N=2,629), the majority of households (1,126, 42.8%) performed casual labor. There were similar numbers of households who drew their main source of income from a formal job or business (586, 22.3%) and an informal job (634, 24.1%). In the total sample there were 283 households (10.8%) who took most of their income from remittances. In the control group (n=797), 263 (33.0%) households claimed casual labor, 282 (35.4%) claimed a formal job or business, and 191 (24.0%) claimed an informal job as the main source of household income. Far fewer households (61, 7.7%) took most of their income from remittances. In the R3 treatment group (n=771), the majority of households (301, 39.0%) took most of their income from casual labor. Fewer households (175, 22.7%) named formal job or business as their main source of income. There were 216 households (28.0%) that claimed informal jobs as their main source of income while only 79 households (10.2%) got most of their income from remittances. Meanwhile in the R1R3 (n=1,061) treatment group, 53.0% (562) of households got most of their income from casual labor. There were 227 households (21.4%) that took most of their income from informal jobs while 129 households (12.2%) had a formal job or business as their main source of income. Finally, 143 households (13.5%) drew most of their income from remittances. There were significant differences among the groups in terms of their non-financial assets, in particular land and animal ownership (χ²=64.821 and χ²=97.621, respectively). Most households (2,278 86.6%) in the total sample (N=2,629) had access to land. Similarly, 94.2% (751) of households in the control group (n=797); 86.0% (663) of households in the R3 treatment group (n=771); and 81.4% (864) of households in the R1R3 treatment group (n=1,061) had access to land. As for animal ownership, 2,278 (86.6%) households in the total sample (N=2,629) had animals. In similar proportions, 597 households (79.5%) in the control group (n=797); 498 households (64.6%) in the R3 treatment group (n=771); and 559 households (52.7%) in the R1R3 treatment group (n=1,061) owned animals. There were significant differences among the groups (χ²=114.411, p<0.001) in regards to their possession of other household assets (such as cooking stoves, beds, blankets, mattresses, mosquito nets, shoes, cooking utensils, furniture, etc.). Most households in all the groups reported having some or most of these other household assets. The breakdown for households in the R1R3 illustrates this point, as 77 households (10.0%) reports having none or almost none of the other assets; 377 (48.9%) reported having some of them and 317 (41.1%) reporting that they had most of the other household assets. There were significant differences (χ²=144.370, p<0.001) in the main sources of food that households reported in the past month. The households in all three groups showed similar breakdowns, with the majority (59.4%, 1,562) of households in the total sample (N=2,629) growing most of their food at home. There were 788 (30.0%) households who bought their own food; 235 households (8.9%) who were given food in exchange for work; and 44 households (1.7%) who received most of their food through donations. Education Significantly more children (1,478, 69.7%) in the survey (N=2,629) were enrolled in school (χ²=4.966). Interestingly, however, most families (1,506, 57.3%) reported having children who were not enrolled in 20 school. There was a significant difference (χ²=13.763, p<0.05) between groups as to whether or not there were children in the household who were not enrolled in school. Social Assistance and Services In terms of what kinds of social or governmental services households received, there were few significant differences between the groups. More households in both the R3 and R1R3 treatment groups received financial services than households in the control group (43.2%, 33.9%, and 22.9%, respectively), but these differences were not significant (χ²=1.826, p>0.05). For parenting services, roughly 2.0% of respondents in all groups indicated similar rates of receiving services, and any differences between groups were not significant (χ²=0.215, p>0.05). However, households in the R3 group did receive significantly more financial cash direct assistance services (χ²=8.071, p<0.05) than households in the R1R3 or the control groups, at 2.2% of families receiving these services compared to 0.9% and 0.8% for the R1R3 and control group, respectively. Also significant were differences in receiving loans assistance (χ²=8.642, p<0.05). Households in the R3 treatment group received loans assistance at a 5.4% rate compared to 3.2% and 2.9% for the R3 and control group, respectively. Food Security There were significant differences between the groups (χ²=142.02, p<0.001) in regards to the question of having to go a whole day hungry for lack of food. There were 492 households (61.7%) in the control group that responded no, whereas 59.8% (461) of families in the R3 treatment group and 403 families (38.0%) in the R1R3 treatment group never went a day without a meal. Also, most families (1,558, 59.3%) in the total sample (N=2,629) reported eating two meals per day. There were significant differences between groups in regards to meals per day (χ²=133.44, p<0.01). Importantly, 63.3% of families in the R1R3 treatment group (n=1,061) reported eating two meals a day while 52.7% of families (406) reported eating this many meals in the R3 group. Health, Water, Sanitation and Shelter On the question asking whether each household has access to safe drinking water within 30 minutes, 56.3% of the sample group answered they do (1,479). The control group had a little higher rate at 58% (462) while 55.6% of the R3 group (429) and 55.4% of the R1R3 group (588) had access to safe drinking water. The difference among groups is not statistically significant (χ²=1.37, p>0.05). Regarding the main source of water, majority of the respondents across the groups used protected public water source as 61% of the sample group (1,604), 60.1% of the control group (479), 59.7% of the R3 treatment group (460) and 62.7% of the R1R3 treatment group (665) use the protected public water source as their main source of water. The second most popular source of water is unprotected source of water such as river, lake, pond, etc. Approximately 38.6% of the sample group use this unprotected source (1,006) while 39.4% of the control group (314), 39.8% of the R3 treatment group (307) and 36.3% of the R1R3 group (385) used this source. The difference among groups in these categories is statistically insignificant (χ²=5.09, p>0.05) On the question asking whether the household has access to a public health facility within 5km, a much higher rate of people answered they do as 80.6% of the sample group said they have access (2,119). The control group has a slightly lower percentage of people who has access to a public health facility as 77.0% of the respondents answered they do (614). Approximately 76.3% of the R3 treatment group had this access (588) in comparison to 86.4% of the R1R3 group (917) who had access. The difference among groups is statistically significant (χ²=38.78, p<0.001) There were no significant differences between the groups in terms of caregivers knowing the HIV status of all children in the household. In the total sample (N=2,629), 49.3% (1,295) of the caregivers knew the HIV status of all children in the household. The control group showed a similar results as 48.7% (n=388) of 797 21 respondents claimed to know the HIV status of all children. In the R3, 51.1% (394) of the respondents knew the HIV status of all children while this number is a little lower (48.4%) among the R1R3 treatment group (n=1,061). For the questionnaire asking whether the household was able to access money the last time there was an unexpected urgent household expense, there were significant differences between the groups. Roughly one third (34.0%, 895) of households in the general sample (N=2,629) were able to access money to pay for that expense. In the control group, 38.3% of respondents (305) were able to. On the other hand, 41.4% of the R3 treatment group (319) and only 25.5% of the R1R3 group (271) had access to the money. It should be noted that for this question, enumerators were instructed to fill in “no” if there was no unexpected expense. In the future, breaking this question down into a two stage item might be preferable, e.g. first asking “Where there any unexpected expenses in the past three months; if so, was the household able to access funds to pay for them. Table 2: Child Inventory: Outcomes for all children (aged 0-17), as reported by caregiver TOTAL CONTROL GROUP R3 Treatment Group 1 Village Para-social workers (PSWs) R1R3 Treatment Group 2 SILCs + Village PSWs Comparison statistics F or χ² (N=2,629) (n=797) (n=771) (n=1,061) Frequency (Percentage, %) or Mean (SD) Age (in years), mean 7.03 (1.37) 6.96 (1.40) 7.07 (1.34) 7.07 (1.38) 1.15 HH has a HIV positive child 104 (4.0%) 47 (5.9%) 25 (3.2%) 32 (3.0%) 5.44 Antiretroviral therapy (ART) Treatment (n=2,445) 119 (4.9%) 59 (2.4%) 22 (0.9%) 38 (1.6%) 20.58*** HH has at least one child with disabilities 487 (19.9%) 187 (24.7%) 164 (24.4%) 136 (13.4%) 46.8*** At least one child has a Birth Certificate 1146 (46.9%) 471 (62.3%) 270 (40.2%) 405 (39.8%) 104.66*** All children in HH have a Birth Certificate 751 (30.7%) 347 (45.9%) 139 (20.7%) 265 (26.1%) 124.05*** Child has been immunized (n=2,445) 2,428 (99.3%) 751 (30.7%) 666 (27.2%) 1011 (41.3%) 0.56 At least one child in HH works (n=1,400) 1,211 (86.5%) 443 (31.6%) 393 (28.1%) 375 (26.8%) 134.8*** Attends School (n=2,445) 2,096 (85.7%) 668 (27.3%) 566 (23.1%) 862 (35.3%) 6.3* Child Protection issues experienced by any child (n=2,445) Emotional Abuse 100 (4.1%) 22 (0.9%) 40 (1.6%) 38 (1.6%) 8.95* Early Marriage 83 (3.4%) 18 (0.7%) 30 (1.2%) 35 (1.4%) 4.72 Any child is sexually active (n=2,445) 335 (13.7%) 186 (7.6%) 93 (3.8%) 56 (2.3%) 133.76*** Any child ever been pregnant (n=242, only among sexually active females) 25 (10.3%) 8 (3.3%) 10 (4.1%) 7 (2.9%) 7.40* Any child ever given birth (n=242, only among sexually active females) 23 (9.5%) 8 (3.3%) 10 (4.1%) 7 (2.9%) 5.24 Any child is married (n=335) 5 (1.5%) 2 (0.6%) 2 (0.6%) 1 (0.3%) 0.53 22 * p≤0.05, ** p≤0.01, *** p≤0.001 On the question asking whether there is any child with HIV in the household, there was no significant difference among groups (χ²=5.44, p>0.05), though 4% (104) of the total sample (N=2,629) claimed to have a child living with HIV. There was significant difference among groups as to whether there is at least one child with disabilities in the household (χ²=46.80, p<0.001). 19.9% (487) of the total sample (N=2,629) responded yes to this question, while 24.7% (187) of the control group responded that there is at least one child with disabilities in the household. Interestingly, in the R1R3 treatment group (=1,061) this number shrunk to 13.4% (136). There were also significant differences among groups on the question asking whether at least one child in the household has a birth certificate (χ²=104.66, p<0.001). There were 46.9% households (1,146) in the general sample (N=2,629) who had a child with a birth certificate while 62.3% (471) of the control group answered yes to the question. Interestingly, in the treatment groups, 40.2% (270) of the R3 group answered yes to the question while this number again decreased to 39.8% of R1R3 (405). On the questionnaire asking if all children in the household had a birth certificate, there was a significant difference among groups (χ²=124.05, p<0.001). Contrasting with the question of only having one child in the household with a birth certificate, 30.7% (751) of the total responses said all children in their household have a birth certificate. In control group, 45.9% of the group (347) answered that they have all a birth certificate for all children while only 20.7% (139) of the R3 had a birth certificate and 26.1% (265) of the R1R3 did. The mean age in this group was 4.25 (SD=2.56). Gender difference among groups is not statistically significant (χ²=1.047, p>0.05). Among children aged 0 to 9 in the entire sample (N=2,112), 51.0% are male (712) while 49% (684) are female. The gender distribution is similar in the control group (n=423) as 51.8% is male (219) and 48.2% (204) is female. Among the treatment groups, 48.9% (209) are male and 51.1% are female in the R3 group (n=427) while 52.0% (284) of the R1R3 group (n=546) is male and 48.0% (262) is female. The total number of caregivers with a child under the age of 5 years is 726, of which 81% (n=586) gave their consent to measure child’s arm circumferences. Malnutrition, measured by mid-upper arm circumference (MUAC), carried no statistically significant differences between the groups (χ²=4.98, p>0.05). The majority of the total sample (N=2,112) had no apparent risk of malnutrition (≥125 mm) while 90.9% (180) of the control group (n=423) also fell into this category. The treatment groups showed similar results. On the other hand, 3.2% (19) of the total sample showed moderate malnutrition (115-125 mm) in comparison to 3.5% of the control group (7), 4.0% of the R3 group (7), and 2.3% of the R1R3 group (5). Surprisingly, there were more children in the Severe Malnutrition (MUAC ≤115 mm) category. Seven percent of the total sample showed severe malnutrition (41), followed by 5.6% of the control group (11), 5.1% of the R3 treatment group (9), and 9.9% of the R1R3 (21). Among children aged 0-9 (n=2,112), 876 in the total sample have ever been tested for HIV (41.5%). Roughly half (49.1%) of caregivers (685) knew children’s HIV status in the general sample, while this was the case for 47.8% (202) of the control group (n=423). The distributions were roughly the same in the treatment groups. Interestingly, there were significant differences in children being too sick to participate in daily activities (χ²=21.22, p<0.001). These differences increased in the treatment groups as well. Compared the to the control group (36.5%, 244), 39.3% (244) of children in the R3 group were too sick to participate in daily activities and this increased to 47.8% (284) of children in the R1R3 treatment group. Education: Children Most school-aged (5-9 years old) children (96.7%, 1,045, n=1,373) in the sample had attended school. Of these, 74.1% (1,041) were currently enrolled in school. There were significant differences between the groups 23 (χ²=12.228, p<0.05) in this matter, with more children in the R1R3 treatment group being enrolled in school. Interestingly, children in the treatment groups were reported for missing school significantly more than in the control group (χ²=26.63, p<0.001). Those in treatment group R1R3 differed the most, with 43.8% of the group missing school compared to 30.3% in the control group. The main reason for missing school for this group was not having enough money for school fees, materials, or transport. About 28.4% of the group claimed this reason for missing school. In the domain of early childhood education, about half of respondents (50.8%) reported having children who attended any sort of organized early childhood education program. While there were differences between the groups, with the R1R3 having the lowest attendance rate here (61.2% compared to the 70.2% in the control group), these differences were not significant (χ²=4.88, p>0.05). Out of those whose children are not attending early childhood education (n=878), similar percentage (53.9%) of the sample group reported being engaged in stimulating activities with any household member over 15 years of age in the past 3 days (473). The rate drops significantly in the control group as only 15% of the control group (132) indicated that they engaged in stimulating activities in the past 3 days. Among the treatment groups, about 16% of the R3 treatment group (141) engaged in these activities while 22.8% of R1R3 (200) did so. However, the difference among groups in this questionnaire is not significant. Outcomes for Youth Aged 10-17 In the total sample of youth aged 10-17, there were 2,138 respondents. Of these, 1,098 (51.4%) were males. Youth in the sample on average were slightly older than 13 years old (13.22, SD=2.2). Most youth lived with other youth in the household. There were no significant differences among the groups in any of these categories (χ²=4.56, p>0.05). There were however significant differences in respondents’ having a birth certificate or not (χ²=16.95, p<0.05). While 11.6% of the R1R3 group reported having a birth certificate, only 6.7% in the R3 group had one and 7.7% of the control group. Education: Youth 10-17 Most of the youth in the sample (96.8%, 2,052) reported having attended school. Respondents in all groups reported similarly, with no significant differences between them (χ²=3.02, p<0.05). About 25% fewer youth in the sample were currently enrolled in school (1,478), however. Again, there were no significant differences between groups for this (χ²=4.97, p<0.05). Among those who reported not going to school, not having enough money was the main reason, with 25.4% of the R1R3 group, 19.0% of the R3 group, and 15.7% of the control group claiming this. The next most-cited reason for not attending school was being too sick to attend school. Among control group respondents, 3.9% claimed this. Similarly, 3.6% of the R3 group and 3.1% of the R1R3 group responded that being too sick was their main reason for skipping school. There were significant differences in respondents’ answers to this question (χ²=28.48, p<0.05). Health: Youth 10-17 Among respondents (n=2,019), nearly all were immunized (99.4%, 2,007). There were significant differences as to whether youth were getting ART or not (χ²=20.27, p<0.001). While 2.7% of respondents in the control group did get ART, only 1.5% and 1.0% of the R3 and R1R3 respondents get it, respectively. There were significant differences in youth’s school attendance (χ²=16.14, p<0.001). While only 29.4% of the control group responded that they attended school, 35.0% of the R1R3 group did. Similarly, when it came to working, 32.7% of youth in the control group reported working while 26.7% of the R1R3 treatment group did. Differences between groups were significant (χ²=126.16, p<0.001). Most youth in the survey were not sexually active, however there were significant differences between the groups (χ²=103.60, p<0.001). While 8.2% of youth in the control group reported being sexually active, 4.3% of the R3 treatment group and 2.7% of the R1R3 treatment group reported as such. Not many respondents 24 reported being subject to emotional abuse, however there were significant differences between groups (χ²=8.24, p<0.001). Youth in the sample reported significant differences in pregnancy (χ²=6.751, p<0.05). Notably, 4.6% of the R3 group reported being or having been pregnant, while this number was 3.7% for the control group. There is no significant difference in “Ever given a birth” questionnaire among groups. About 10.6% (23) of the total sample (n=218) answered to ever given a birth. Much smaller percentage (3.7%)of the control group has given a birth as only 8 people answered yes. On the other hand, approximately 4.6% of the people in the R3 (10) and 2.3% of the R1R3 (5) have given a Birth. Group Comparisons of Secondary Outcomes Twenty-four secondary outcome measures are captured at the OVC and Youth wellbeing level for children aged 0-9 years (with questions directed to the primary caregiver) and youth aged 10-17 years. These include:  Percent of children whose primary caregiver knows the child’s HIV status  Percent of children < 5 years of age who are undernourished  Percent of youth too sick to participate in daily activities  Percent of children too sick to participate in daily activities  Percent of youth who have a Birth Certificate  Percent of children who have a Birth Certificate  Percent of youth enrolled in school  Percent of children enrolled in school  Percent of youth regularly attending school  Percent of children regularly attending school  Percent of children <5 years with recent diarrhea  Percent of children <5 years with recent fever  Percent of youth >10 to 17 years reporting irregular food intake  Percent of children >2 to 9 years reporting irregular food intake  Percent of youth who progressed in school during the last year  Percent of children who progressed in school during the last year  Percent of children <5 years of age who recently engaged in stimulating activities with any household member over 15 years of age  Percent of caregivers who feel harsh physical punishment is appropriate means of discipline in the home or school  Percent of households able to access money to pay for unexpected expenses  Percent of households able to access money to pay for health expenses in past 3 months  Percent of households able to access money to pay for school expenses in past 3 months  Percent of households able to access money to pay for food expenses in past 3 months  Percent of children 1-5 years fully immunized  Percent of youth aged 10-17 years reporting basic support For the 24 secondary outcomes, five had no significant differences across the three groups. These all revolved around children under five on such matters as involvement in stimulating activities, irregular food intake, school attendance and progression, health and caregiver knowledge of HIV status. All remaining secondary indicators had significant differences. The results are presented below: Outcome Indicator 1: Percent of children whose primary caregiver knows the child’s HIV status; Chi-Sqd, Frequency and Percentage 25 Control Group (834) R3 Group (863) R1R3 Group (932) Total (2629) Chi-Sqd No 428 (51.32%) 425 (49.25%) 486 (52.15%) 1339 (50.93%) 1.58 Yes 406 (48.68%) 438 (50.75%) 446 (47.85%) 1290 (49.07%) Outcome Indicator 2: Percent of children < 5 years of age who are undernourished; Chi-Sqd, Frequency and Percentage Control Group (203) R3 Group (203) R1R3 Group (180) Total (586) Chi-Sqd No 192 (94.58%) 194 (95.57%) 160 (88.89%) 546 (93.17%) 7.656* Yes 11 (5.42%) 9 (4.43%) 20 (11.11%) 40 (6.83%) Outcome Indicator 3a: Percent of youth too sick to participate in daily activities; Chi-Sqd, Frequency and Percentage Control Group (696) R3 Group (718) R1R3 Group (713) Total (2127) Chi-Sqd No 487 (69.97%) 503 (70.06%) 442 (61.99%) 1432 (67.32%) 13.869*** Yes 209 (30.03%) 215 (29.94%) 271 (38.01%) 695 (32.68%) Outcome Indicator 3b: Percent of children too sick to participate in daily activities; Chi-Sqd, Frequency and Percentage Control Group (703) R3 Group (707) R1R3 Group (701) Total (2111) Chi-Sqd No 447 (63.58%) 436 (61.67%) 350 (49.93%) 1233 (58.41%) 31.597*** Yes 256 (36.42%) 271 (38.33%) 351 (50.07%) 878 (41.59%) Outcome Indicator 4a: Percent of youth who have a Birth Certificate; Chi-Sqd, Frequency and Percentage Control Group (696) R3 Group (718) R1R3 Group (713) Total (2127) Chi-Sqd No 549 (78.88%) 590 (82.17%) 528 (74.05%) 1667 (78.37%) 14.071*** Yes 147 (21.12%) 128 (17.83%) 185 (25.95%) 460 (21.63%) Outcome Indicator 4b: Percent of children who have a Birth Certificate; Chi-Sqd, Frequency and Percentage Control Group (703) R3 Group (707) R1R3 Group (701) Total (2111) Chi-Sqd No 545 (77.52%) 556 (78.64%) 492 (70.19%) 1593 (75.46%) 16.017*** Yes 158 (22.48%) 151 (21.36%) 209 (29.81%) 518 (24.54%) Outcome Indicator 5a: Percent of youth enrolled in school; Chi-Sqd, Frequency and Percentage Control Group (696) R3 Group (718) R1R3 Group (713) Total (2127) Chi-Sqd No 192 (27.59%) 243 (33.84%) 211 (29.59%) 646 (30.37%) 6.852* Yes 504 (72.41%) 475 (66.16%) 502 (70.41%) 1481 (69.63%) Outcome Indicator 5b: Percent of children enrolled in school; Chi-Sqd, Frequency and Percentage Control Group (446) R3 Group (462) R1R3 Group (463) Total (1371) Chi-Sqd No 105 (23.54%) 152 (32.90%) 97 (20.95%) 354 (25.82%) 19.032*** Yes 341 (76.46%) 310 (67.10%) 366 (79.05%) 1017 (74.18%) Outcome Indicator 6a: Percent of youth regularly attending school; Chi-Sqd, Frequency and Percentage Control Group (504) R3 Group (475) R1R3 Group (502) Total (1481) Chi-Sqd No 298 (59.13%) 248 (52.21%) 257 (51.20%) 803 (54.22%) 7.512* Yes 206 (40.87%) 227 (47.79%) 245 (48.80%) 678 (45.78%) Outcome Indicator 6b: Percent of children regularly attending school; Chi-Sqd, Frequency and Percentage Control Group (341) R3 Group (310) R1R3 Group (366) Total (1017) Chi-Sqd No 204 (59.82%) 162 (52.26%) 158 (43.17%) 524 (51.52%) 19.700*** Yes 137 (40.18%) 148 (47.74%) 208 (56.83%) 493 (48.48%) Outcome Indicator 7: Percent of children <5 years with recent diarrhea; Chi-Sqd, Frequency and Percentage Control Group (252) R3 Group (243) R1R3 Group (233) Total (728) Chi-Sqd 26 No 186 (73.81%) 189 (77.78%) 146 (62.66%) 521 (71.57%) 14.311*** Yes 66 (26.19%) 54 (22.22%) 87 (37.34%) 207 (28.43%) Outcome Indicator 8: Percent of children <5 years with recent fever; Chi-Sqd, Frequency and Percentage Control Group (252) R3 Group (243) R1R3 Group (233) Total (728) Chi-Sqd No 134 (53.17%) 132 (54.32%) 107 (45.92%) 373 (51.24%) 3.937 Yes 118 (46.83%) 111 (45.68%) 126 (54.08%) 355 (48.76%) Outcome Indicator 9a: Percent of youth reporting irregular food intake; Chi-Sqd, Frequency and Percentage Control Group (696) R3 Group (718) R1R3 Group (713) Total (2127) Chi-Sqd No 658 (94.54%) 652 (90.81%) 637 (89.34%) 1947 (91.54%) 13.036** Yes 38 (5.46%) 66 (9.19%) 76 (10.66%) 180 (8.46%) Outcome Indicator 10b: Percent of children reporting irregular food intake; Chi-Sqd, Frequency and Percentage Control Group (636) R3 Group (645) R1R3 Group (660) Total (1941) Chi-Sqd No 608 (95.60%) 599 (92.87%) 618 (93.64%) 1825 (94.02%) 4.512 Yes 28 (4.40%) 46 (7.13%) 42 (6.36%) 116 (5.98%) Outcome Indicator 11a: Percent of youth who progressed in school during the last year; Chi-Sqd, Frequency and Percentage Control Group (482) R3 Group (447) R1R3 Group (481) Total (1410) Chi-Sqd No 304 (63.07%) 276 (61.74%) 361 (75.05%) 941 (66.74%) 22.918*** Yes 178 (36.93%) 171 (38.26%) 120 (24.95%) 469 (33.26%) Outcome Indicator 11b: Percent of children (5 to 9) who progressed in school during the last year; Chi-Sqd, Frequency and Percentage Control Group (375) R3 Group (370) R1R3 Group (405) Total (1150) Chi-Sqd No 282 (75.20%) 279 (75.41%) 289 (71.36%) 850 (73.91%) 2.121 Yes 93 (24.80%) 91 (24.59%) 116 (28.64%) 300 (26.09%) Outcome Indicator 12: Percent of children <5 years of age who recently engaged in stimulating activities with any household member over 15 years of age; Chi-Sqd, Frequency and Percentage Control Group (834) R3 Group (863) R1R3 Group (932) Total (2629) Chi-Sqd No 625 (74.94%) 604 (69.99%) 670 (71.89%) 1899 (72.23%) 5.27 Yes 209 (25.06%) 259 (30.01%) 262 (28.11%) 730 (27.77%) Outcome Indicator 13: Percent of caregivers who feel harsh physical punishment is appropriate means of discipline in the home or school; Chi-Sqd, Frequency and Percentage Control Group (834) R3 Group (863) R1R3 Group (932) Total (2629) Chi-Sqd No 414 (49.64%) 385 (44.61%) 617 (66.20%) 1416 (53.86%) 92.802*** Yes 420 (50.36%) 478 (55.39%) 315 (33.80%) 1213 (46.14%) Outcome Indicator 14: Percent of households able to access money to pay for unexpected expenses; Chi-Sqd, Frequency and Percentage Control Group (834) R3 Group (863) R1R3 Group (932) Total (2629) Chi-Sqd No 517 (61.99%) 516 (59.79%) 705 (75.64%) 1738 (66.11%) 59.505*** Yes 317 (38.01%) 347 (40.21%) 227 (24.36%) 891 (33.89%) Outcome Indicator 15: Percent of households able to access money to pay for health expenses in past 3 months; Chi-Sqd, Frequency and Percentage Control Group (834) R3 Group (863) R1R3 Group (932) Total (2629) Chi-Sqd No 644 (77.22%) 666 (77.17%) 656 (70.39%) 1966 (74.78%) 14.789*** Yes 190 (22.78%) 197 (22.83%) 276 (29.61%) 663 (25.22%) 27 Outcome Indicator 16: Percent of households able to access money to pay for school expenses in past 3 months; Chi-Sqd, Frequency and Percentage Control Group (834) R3 Group (863) R1R3 Group (932) Total (2629) Chi-Sqd No 698 (83.69%) 729 (84.47%) 730 (78.33%) 2157 (82.05%) 13.741** Yes 136 (16.31%) 134 (15.53%) 202 (21.67%) 472 (17.95%) Outcome Indicator 17: Percent of households able to access money to pay for food expenses in past 3 months; Chi-Sqd, Frequency and Percentage Control Group (834) R3 Group (863) R1R3 Group (932) Total (2629) Chi-Sqd No 570 (68.35%) 651 (75.43%) 439 (47.10%) 1660 (63.14%) 168.755*** Yes 264 (31.65%) 212 (24.57%) 493 (52.90%) 969 (36.86%) Outcome Indicator 18: Percent of children 1-5 years fully immunized; Chi-Sqd, Frequency and Percentage Control Group (125) R3 Group (113) R1R3 Group (173) Total (411) Chi-Sqd No 5 (4.00%) 0 (0.00%) 44 (25.43%) 49 (11.92%) 52.843*** Yes 120 (96.00%) 113 (100.00%) 129 (74.57%) 362 (88.08%) Outcome Indicator 19: Percent of youth aged 10-17 years reporting basic support; Chi-Sqd, Frequency and Percentage Control Group (696) R3 Group (718) R1R3 Group (713) Total (2127) Chi-Sqd No 158 (22.70%) 184 (25.63%) 291 (40.81%) 633 (29.76%) 64.133*** Yes 538 (77.30%) 534 (74.37%) 422 (59.19%) 1494 (70.24%) * p≤0.05, ** p≤0.01, *** p≤0.001 Vulnerability Assessment and HVAT Scoring The primary impact indicator used for this evaluation is the Household Vulnerability Assessment Tool (HVAT). The HVAT is the official tool used by the Ministry of Gender Labor and Social Development (MGLSD) to obtain in-depth information about a household’s level of vulnerability. The HVAT is slated to be applied nationally and reported quarterly in the OVC MIS. A leaner variant of the HVAT is used by Sustainable Outcomes as the primary outcome indicator in the AMELP to gauge changes to household vulnerability over time (Result Area Indicator #2), as well as an implementation tool for household improvement and case management plans. As such, the HVAT is the logical choice for the impact evaluation, as it is aligned to Sustainable Outcomes program goals and performance metrics, is in line with the National MGLSD strategy and is accounted for in the national OVCMIS system. The HVAT is laid out according to six dimensions of vulnerability known as Core Programme Areas (CPAs). Each CPA is further divided according to a varying number of items; each of which is scaled across an ordinal spectrum of 0 to 4 in which 0 corresponds to some acceptable level of resilience while 4 corresponds to high levels of vulnerability. The national HVAT contains 18 items in total, but, due to our confidentiality protocols on HIV/AIDS status, the item seeking to capture this information (under the Health CPA) was dropped from the baseline. As a result, we used 17 items in total, which are distributed across the CPAs as follows:  CPA 1: Economic Strengthening (5 items)  CPA 2: Food Security and Nutrition (3 items)  CPA 3: Health, WASH and Shelter (3 items)17 17 For this baseline survey, we dropped one item from this CPA, i.e. the question: “Are all eligible children who are HIV+ and/or have TB on treatment?” as our IRB protocol stipulated that we would not directly ask caregivers and youth about HIV status. Asking this question would have violated this protocol. 28  CPA 4: Education (1 item)  CPA 5: Psychosocial Support and Basic Care (2 items)  CPA 6: Child Protection and Legal Support (2 items)  Total: 17 items that span from 0 to 4 for a potential total vulnerability score of 68 HVAT scores are given as a percentage (see Annex 3 for more information on how the HVAT was calculated). To compute overall vulnerability, scores for each item are tallied across all CPAs to form the numerator, and are divided by the highest possible vulnerability score (i.e. a score of 4 across all 17 items = 68). The result is a percentage score falling across 0 to 100%. Vulnerability is then assigned according to placement on the percentage score across potential scores across quartiles as follows:  Not Vulnerable: 0-24%  Slightly Vulnerable: 25-49%  Moderately Vulnerable: 50-74%  Critically Vulnerable: 75-100% HVAT Group Comparisons For this baseline, we found significant difference in the mean average vulnerability score and CPA ranking across all three groups: HVAT and CPA Group Comparisons (ANOVA) Mean (St. Dev.) F Comparison statistics F or χ² CPA_1_Economic 53.6% (3.7) 70.27*** CPA_5_Psycho_Social 15.8% (1.87) 15.23*** Control 55.0% (3.46) Control 16.8% (1.84) R1 Treatment 62.4% (2.99) R1 Treatment 62.4% (1.68) R1R3 Treatment 22.3% (2.02) R1R3 Treatment 18.1% (2.02) CPA_2_Nutrition 20.0% (1.81) 103.95*** CPA_6_Child_Protection 73.3% (2.14) 38.27*** Control 19.0% (1.75) Control 76.5% (2.22) R1 Treatment 28.9% (2.21) R1 Treatment 83.0% (2.15) R1R3 Treatment 49.8% (3.01) R1R3 Treatment 61.8% (1.96) CPA_3_Health 47.6% (3.04) 23.27*** HVAT_Score 41% (10%) 79.34*** Control 48.3% (2.93) Control 39% (10%) R1 Treatment 53.1% (2.98) R1 Treatment 39% (10%) R1R3 Treatment 38.8% (1.62) R1R3 Treatment 44% (10%) CPA_4_Education 34.8% (1.58) 16.2*** HVAT_Category¹ Slightly Vulnerable 84.54*** 29 Control 36.0% (1.57) Control Slightly Vulnerable R1 Treatment 44.8% (1.68) R1 Treatment Slightly Vulnerable R1R3 Treatment 53.6% (3.7) R1R3 Treatment Slightly Vulnerable *** p<0.001 ¹ HVATCategory reports using Chi-squared Test As with the other indicators discussed above, the significant differences among average vulnerability scores across groups is most likely a product of geographic variation. As stated before, this is not an issue for the difference-in-difference design as long as the parallel trends assumption still holds. HVAT Distribution Turning to the distribution of vulnerability, we find the following distribution using the HVAT: Distribution of Vulnerability across Control and Treatment Groups HVAT Category Control R1 R1R3 Total Not Vulnerable 64 (8%) 60 (7%) 23 (2%) 147 (6%) Slightly Vulnerable 628 (75%) 667 (77%) 630 (68%) 1925 (73%) Moderately Vulnerable 142 (17%) 136 (16%) 279 (30%) 557 (21%) Critically Vulnerable 0 0 0 0 Total 834 863 932 2629 Overall, we find a high concentration of “slightly vulnerable” across the three groups (73% of the total sample), followed by a smaller concentration of “moderately vulnerable” households, especially in the R1R3 group; a smaller percentage of “not vulnerable” households, and a complete lack of “critically vulnerable” households across the entire sample. Overall, the HVAT scores suggest that Sustainable Outcomes is supporting a higher number of “less vulnerable” households than those that score as “more vulnerable.” If we assume that Sustainable Outcomes is a program designed to assist the most critically vulnerable, a first blush interpretation of the baseline might suggest that the majority of the sampled households are not the priority groups Sustainable Outcomes should be targeting. The simplest explanation for this may be that the Sustainable Outcomes households selected are really not that vulnerable. While Sustainable Outcomes may be reaching the most vulnerable within their districts/subcounties, overall vulnerability in that area may be less than originally targeted. This opens the door to a number of interrelated questions, including those around targeting, such as “did the baseline target the right households?” or “is Sustainable Outcomes targeting the right households,”; or, alternatively, “is the HVAT properly calibrated to identify more vulnerable households?” We explore these questions below according to two lines of inquiry. First, we discuss potential relative biases in the sampling of respondents that explain a seemingly less vulnerable group of households. Second, we discuss potential structural biases in the way the HVAT is scaled that prevent more households from appearing more vulnerable (i.e. is the scale itself reflective of the reality of vulnerability)? Relative Biases Sampling: Random Route vs. Sustainable Outcomes-Assisted To better understand if and what kinds of relative biases may exist, it is useful to review and discuss the sampling strategy taken by the baseline survey, its sources and potential comparisons within the dataset and 30 with other data sources. As stated above, the baseline survey was based upon a two-stage sampling strategy in which parishes were randomly selected from master lists of parishes corresponding to the treatment and control groups. Master lists were based upon information provided by Sustainable Outcomes staff regarding planned intervention areas. The second stage household selection for the R1R3 treatment group was originally planned to occur in cooperation with Sustainable Outcomes staff, as they will be directly enrolling these households into the program. Survey enumerators were to first collect household lists from Sustainable Outcomes’ CSOs and PSWs in the field, after which time they would randomly select households for screening and surveying. However, at times Sustainable Outcomes’ partners were not available to assist the enumerators, at which time the enumerators would randomly select households using the random route method. This occurred in 21% of the sample households for the R1R3 treatment group. A comparison of the HVAT distribution of these two sampling approaches– assisted by Sustainable Outcomes and unassisted with Random Route – within the R1R3 treatment group is a useful step in identifying if there were any systematic biases in either sampling procedure. Put differently, are Sustainable Outcomes’ partners – i.e. local CSOs and PSWs who know the area and have a familiarity with the households – better able to identify highly vulnerable households than through random route selection? The comparison reveals the following: Comparison of Household Selection for R1R3 Treatment Random Route Random w/Sustainable Outcomes Assistance Not Vulnerable 1% 3% Slightly Vulnerable 71% 67% Moderately Vulnerable 28% 31% χ²=1.821, p=.402 The comparison reveals very few differences in terms of the distribution of results. Sustainable Outcomes￾assistance did yield a slightly higher number of “not vulnerable” households than through Random Route. Anecdotally, our survey team did report a few instances in which PSWs encouraged enumerators to sample their own families, and/or suggested they visit seemingly more well to do households than poor households (based upon the physical condition of the house), but these instances were few. The Random Route method, by contrast, seems to yield a slightly higher percentage of “slightly vulnerable” households than through PSW-assistance, but yields a lower percentage of “moderately vulnerable” households. More importantly, Chi-squared tests reveal no significant difference between the two approaches (χ²=1.821, p=.402), and thus we can confidently attribute any seeming differences to random error. Cohort 1 vs. Cohort 2 It is also useful to compare the household vulnerability distribution of Cohort 2 as a whole with that of Cohort 1. During the baseline data collection process we also discovered that Sustainable Outcomes staff had partially applied the HVAT (reworded as HAT) during Cohort 1. The main difference between the two tools is that the HAT dropped the Psycho-Social Support CPA and reduce the Child Protection CPA to one item. Indeed, this may be a useful modification as items under both CPA likely suffer from positive social response bias (e.g. do you find acceptable to hit your child?) and therefore push vulnerability downward (towards less vulnerable). We therefore reconfigured the Baseline HVAT scores into HAT scores to facilitate comparisons with Cohort 1. The Cohort 1 dataset contains 132,052 sampled households – a rather impressive effort presented in a clean, straight forward dataset. However, Sustainable Outcomes staff informed the survey team that, as of the time of the survey, they had not yet computed the vulnerability scores due to a lack of guidance on how to score each individual item. As such, vulnerability for Cohort 1 had not yet been computed. In response, we applied the same scoring method used on the HVAT to the HAT, as all of the items on both tools are nearly 31 identical, and dropped the additional items on the HVAT to ensure we had two comparable sets of vulnerability scores. We find the following vulnerability distribution across Cohort 1 and Cohort 2: Comparison of HAT Distribution Across Cohorts Category Cohort 2 Sample (2,629) Cohort 1 (132,052) Cohort Leader Not Vulnerable 188 (7%) 14296 (11%) Cohort 1 Slightly Vulnerable 1553 (59%) 75905 (57.3%) Cohort 2 Moderately Vulnerable 869 (33%) 39613 (30%) Cohort 2 Critically Vulnerable 19 (1%) 2237 (1.7%) Cohort 1 F=4.379, *p<0.05 Based upon the above, we find that significantly different (F=4.379, p<.05) vulnerability distributions across the two Cohorts.18 Cohort 1 has a higher percentage of “Not Vulnerable” and a slightly higher “Critically Vulnerable” representation, while Cohort 2 has a much higher percentage of “Moderately Vulnerable” and very similar “Slightly Vulnerable” distribution. As such, there do appear to be systematic differences between the two Cohorts in which Cohort 1 is thicker on both ends of the vulnerability spectrum while Cohort 2 has a higher concentration of mid-level vulnerability. Of particular concern is the high number of “not vulnerable” households current receiving support under Cohort 1 (more than 14,000). We’d therefore strongly recommend that Sustainable Outcomes staff review their selection procedures and take further steps to identify and reduce any potential biases. Although Sustainable Outcomes’ approach to using CSOs and PSWs does appear to have a slight bias towards less vulnerability across both Cohorts, we must caution against overstating these differences for two reasons. First, the sheer difference in sample size between Cohort 1 and Cohort 2 does limit our ability to make inferences about any significant differences between the two groups, as these large differences bias any differences as significant. Second, Sustainable Outcomes’ potential selection bias is limited to Cohort 1 and the R1R3 sample group – it doesn’t explain why the baseline survey team also found seemingly less vulnerable households in the R1 Treatment and Control group than what is to be expected. As discussed below, the HVAT scaling system may contain biases that pre-dispose it to score households as less vulnerable. Structural Biases We found that the way the HVAT is scaled has a large impact on the resulting vulnerability scores and status. More precisely, our review of the HVAT tool scaling structure reveals the following:  Contradictions in scoring, due in part to inconsistent units of measurement;  Arbitrary spaces in the scoring of each item can bias household vulnerability scores downward (i.e. they appear less vulnerable);  Many of the qualitative values found under each item either do not logically follow the structure of the data or do not have a relationship to other elements;  The items on the HVAT are not equal in relevance and therefore should not be scored equally  The items fall across different underlying dimensions and therefore are not easily aggregated using the current scoring method. We will visit each of these findings in turn. 18 Note that when using the HAT score (and potentially reducing the positive social response bias), baseline households appear to be more vulnerable, including the emergence of 19 “critically vulnerable” households. 32 Contradictions in scoring A close review of the HVAT scoring system reveals a number of inconsistencies and contradictions, mostly stemming from attempts incorporate two systems of measurement (e.g. relative percentages vs. absolute numbers) in the same item scoring system. We’ve summarized these below: Question 1.4 asks the respondent to select Yes or No on 7 items related to HH assets: This issue arises if more than 4 are Yes, which means that there are can be 0,1 or 2 "No"s. However, 1 and 2 "No"s are already assigned. Our solution was to assign "0" when all are Yes. Similarly, Question 3.1 Asks to select Yes or No on 8 items related to WASH assets: This question faces two issues. First, in a similar problem as above, when more than 4 items are Yes, it means that there are will either be 0,1, 2 and 3 "No"s. However, 1, 2 and 3 "No"s are already assigned. Second, when there are 4 NOs, it means there are also 4 YESs; i.e. the same answer gives two diametrically opposed scores. Our solution was to again assign "0" when all are Yes. Question 4.1 asks about number/percentage of children missing school: This question also faces two issues. First,” more than three children attending” and “none attends” are completely differently things - they shouldn't be scored the same way. Our solution was to assume this was a mistake and reinterpret as “More than 3 do NOT attend." Second, the scale should not combine a percentage system with an absolute number system (option 4 vs. the other options), because what if a household has 8 children and 4 attend? (this occurred on 18 occasions). This means 50% attend, which gives a score of 2, but this also means there are 3 NOT attending, a score of a 4. Our solution was remain consistent and assign a 4 in all cases where 3 or more eligible children do not attend, even if this means more than 50% are attending. Question 5.1 asks about the number of times someone in the HH has felt so troubled they need counseling: This question does not assign a score for those who have received counseling 5 times. Our solution was to assign it a 4. In all four cases, we chose solutions that biased vulnerability scores upward, i.e. applicable households received higher vulnerability scores due to our choices. However, we could have just as easily chose to bias the scores downward, as either choice has equal justification so long as it is consistently applied across questions. The implications of these decisions can shift the overall vulnerability distribution around 3% upwards or downwards. As such, we’d recommend stakeholders review these choices and revise the HVAT accordingly. Arbitrary Ordinal Spacing and Assignment As mentioned above, the modified HVAT used for the baseline is comprised of 17 items, each of which are scaled on a spectrum from 0 to 4 in which a 0 score corresponds to the absence of vulnerability while a 4 corresponds to the highest level of vulnerability. As a full ordinal scale, each item should contain 5 potential entries: 0,1,2,3 and 4. However, nine of the 17 items include gaps in the spectrum (i.e. they are not full five 33 point scales). Moreover, the gaps themselves are inconsistently applied (some lack a “2”, others lack a “3”, etc.). Specifically, items with gaps are: Item Number Missing Value 3.2; 3.3; 4.1; 5.2 1; scaled as 0,2,3,4 2.2; 2.3; 3.4 2; scaled as 0,1,3,4 1.5 3; scaled as 0,1,2,4 6.1 2 and 3; scaled as 0,1,4 We were unable to find a justification or rationale behind each gap, e.g. why skip a “2” in one item and then a “1” in another item.19 Moreover, the presence of these gaps does not appear to have any impact on the size of the denominator used to compute the score (recall that the HVAT is found by dividing the actual household score by 68, i.e. all 17 items multiplied by 4). Although every item is allocated an equal weight in the denominator of 0 to 4, the relative space within each item is unequally distributed. As a result, these gaps – where they appear (e.g. as a missing 2 or 3) and how often they appear (across 9 questions) - do have significant implications for the overall scoring. To illustrate this point, we rescored the HVAT through two steps: 1. We eliminated gaps in the scoring of each item and rescored accordingly (e.g. an item with options 0,2,3 and 4 was rescored as 0,1,2 and 3). 2. We reduced the size of the denominator accordingly (by eliminating the empty spaces above, we reduced the denominator from 68 to 52). These changes in scaling reveal the following changes to the vulnerability distribution: Comparison of HVAT Scoring Schemes: Arbitrary Spaces (Original) vs. Modified HVAT Category Original Modified Leader Not Vulnerable 147 (6%) 116 (4.4%) Original Slightly Vulnerable 1925 (73%) 1812 (69%) Original Moderately Vulnerable 557 (21%) 698 (26.5%) Modified Critically Vulnerable 0 3 (.1%) Modified F=25.91,***p<.0001 The results suggest that rescaling by eliminated arbitrary spaces in scoring does make a slight, but significant difference by pushing the vulnerability distribution upward (i.e. vulnerability score increases). Overall, the percentage of not vulnerable decreases, as do slightly vulnerable, even as the moderately and critically vulnerable categories increase. These differences are also highly significant, and reflect uneven gaps throughout the scoring system that is distributed unevenly. Principle Components Analysis The HVAT’s scoring system assumes that all items on the scale have equal weights, are equivalent and can therefore be aggregated. Moreover, the HVAT also assumes that the categorical options under each item (question) fall across the same ordinal scale and can be separated by whole number values. Yet this underlying assumption of “uni-dimensionality” has not been rigorously tested to understand to what extent the construct of “vulnerability” actually load on to all of these items (i.e. they all vary together). As such, these 19 We contacted the MGLSD representative in charge of the HVAT in this regard, and his response was “the HVAT may still require further refinement.” Email correspondence on March 4, 2017. 34 assumptions can be tested using Principle Components Analysis (PCA). Specifically, we used PCA to test the internal consistency of each item, of each CPA and of the entire HVAT index. We first tested the entire HVAT scale, including all 17 items at once, using PCA to understand if all could be reduced to one scale, or component that would explain an adequate amount of the total variance. In doing so, we imposed two best practice criteria: a. the shared variation that defined the first component must explain at least 40% of total variance, and b. each item’s option must have a correlation coefficient of at least .35. Total HVAT SCORE Component 1 Total Variance Explains 15.7% Main contributor Dropped Current monthly HH income 0.62 Main source of income 0.44 Assets 0.67 Expenses pay 0.51 Main source of food past month 0.38 Food groups 0.52 Meals per day Dropped HH conditions Dropped Children HIV status known Dropped Stable shelter 0.64 Type of a latrine 0.59 Education Dropped Faith healer Dropped Unhappy children Dropped Level of abuse Dropped Views on abuse Dropped The results suggest a very poor fit in terms of how well all HVAT items combine to form a uni-dimensional scale. Less than 16% of all variance could be explained on a combined HVAT component, and 9 of the 17 items drop out with scores below the .35 threshold. These results suggest that the HVAT cannot be easily reduced to a single dimension score. CPA and Item Testing We then explored to what extent items under each CPA held together as a component, i.e. if they fell across the same spectrum. We also ran similar tests for each individual item, and explored to what extent options under each item held together as a component. In doing so, we again imposed the same two criteria: a. the shared variation that defined the first component must explain at least 40% of total variance, and b. each item’s option must have a correlation coefficient of at least .35. For categorical variables that had no clear ordinal ranking, we recast options as dummy variables and loaded them jointly into each model. CPA 1: ECONOMIC STRENGTHENING Economic Strengthening Component 1 Total Variance Explains 37.3% 35 Main contributor Dropped Current monthly HH income 0.736 Main source of income 0.607 Assets 0.782 Ability to Pay Expenses 0.556 Not surprisingly, the Economic Strengthening CPA holds together better as an individual component than when it is combined with all other items on the CPA score. Total variance falls slightly below the 40% threshold. Interestingly, who the main household contributor seems to hold little relevance to the overall shared commonality that defines this CPA , even as levels and sources of income, as well as household assets and the ability to pay expenses, all can be reasonable scored across the same dimension. Who pays for MOST of the HH expenses? Who pays for MOST of the HH expenses? Component 1 Component 2 Total Variance Explained 46.7% 27.7% Parent -.991 Dropped Grand Parent .931 Dropped Relative Dropped .832 Child Dropped .487 The results suggest that, as a concept, household expenses are best explained by the Parent vs. Grand Parent Options, and that they work in opposition to each other (i.e. Parents reduce household vulnerability while Grand Parents increase it). However, Relatives and Children load better on a second component, which implies that they may be viewed as a separate measure of vulnerability and reflect different types of dynamics. As such, the analysis suggests that placing these four options on the same scale may be flawed, as they represent different conditions that fall across different spectrums. What is the main source of HH income? What is the main source of HH income? Component 1 Component 2 Component 3 Total Variance Explained 38.6% 32.7% 28.7% Formal .495 .817 Dropped Informal .532 -.799 Dropped Casual -.993 Dropped Dropped Remittances Dropped Dropped .984 Unfortunately, this item fails to come together as a consistent component and faces a number of uncertainties. Although the first three items do come together logically, informal labor appears to have a similar influence on vulnerability as formal labor, but moves in the opposite direction of Casual labor. This contracts the assumption on the HVAT, which scores these two options essentially the same. Moreover, remittances clearly reflects a very different kind of condition that has little to no relationship with the other options and thus does not consistently fit into the overall structure of this item. It would be more useful to break out questions on remittances as a separate item than to keep it in this scale. 36 What is the current monthly HH income? What is the current monthly HH income? Component 1 Component 2 Component 3 Total Variance Explained 34.9% 23.1% 21.2% Income under 50K -0.989 Dropped Dropped Income bzw 50-100k 0.777 -0.613 Dropped Income bzw 100k-150k 0.306 0.786 -0.537 Income bzw 150k-200k Dropped Dropped 0.604 Income over 200k Dropped Dropped 0.62 The results of the PCA on the currently monthly HH income reveal that the interval spacing between options is not internally consistent. Pushing past a monthly income of 150K per month suggests a different kind of condition than the lower income brackets, suggesting that the income bracket spacing used in the scale is inconsistent with the structure of responses. One potential solution is to reduce the number of options on this item scale in which the top three income categories are combined into one, i.e. 100,000 UGX+. What is the current monthly HH income? (re-configured) Component 1 Total Variance Explained 58.8% Income under 50K -0.997 Income bzw 50-100k 0.727 Income 100k+ 0.492 The results suggest a new scale that eliminates the previous arbitrary spaces, is more consistent and explains more variance. Household Assets Does the HH have any of the following?: Component 1 Component 2 Total Variance Explained 28.4% 14.6% Animals 0.624 0.369 transport 0.599 Dropped electronics 0.596 Dropped Skills 0.534 -0.492 Land 0.444 0.417 savings 0.412 Dropped Regular Employment 0.483 -0.609 Overall, the options under the Household Asset item loads quite well. Although the total variance explained is lower than the 40% threshold, this is in part due to a relatively higher number of options. Note, however, how “regular employment” is better explained as a separate component then when it is associated with the others (this second component may be better interpreted as agricultural assets than as household assets). Dropping this option slightly improves the overall variance explained (it increases to 31%). 37 Ability to Pay Expenses HH ability to pay following expenses in past 3 months Component 1 Total Variance Explained 61.40% Health expenses 0.794 Education expenses 0.78 Food expenses 0.776 The options on this item load quite well on only one component and, combined as a uni-dimensional construct, explain a high degree of variance. Interestingly, all three expense categories appear to make a relatively equal contribution to vulnerability. This item is the strongest on the overall HVAT, and could potentially be expanded to include other areas of expenses. CPA 2: FOOD AND NUTRITION Food and Nutrition Component 1 Total Variance Explains 42.2% Main source of food past month 0.76 Food groups 0.757 Meals per day Dropped Overall, the results from the food and nutrition CPA show that, while an adequate level of variance is explained as a whole, one of the three items is dropped from the dimension. Given the small number of items that comprise this CPA, the results suggest that the items on this scale should be serious reconsidered and reformulated. Below we identify two options associated with extreme vulnerability - food that is donated and consuming less than one meal a day – that drop out, which may suggest a qualitatively different experience that cannot easily be scaled with less vulnerable attributes. Main Source of Food What is the HH main source of food? Component 1 Component 2 Component 3 Total Variance Explained 45.6% 28.8% 25.6% Home Grown -0.985 Dropped Dropped Bought 0.889 -0.449 Dropped Given for Work Dropped 0.943 Dropped Donated Dropped Dropped 0.98 The options on this item load poorly on the first component, in that half of them drop out. There is, however, a strong contrast between obtaining food through purchase vs. home grown (which could be reflecting more of rural/urban split toward obtaining food than varying levels of vulnerability). The results also suggest that obtaining food through donation does not fit with the other options. Overall, sources of food, at least the way they are currently structured and formulated, may be a poor predictor of vulnerability and probably should not be bundled together. Types of Food 38 Type of Food Consumed by HH Component 1 Total Variance Explained 47.4% Body building foods 0.783 Protective/regulative foods 0.741 Energy foods 0.517 Overall, the options on types of food consumed by the household sufficiently load onto this item, and the overall variance explained is also sufficient. No change is needed for this item. Meals Per Day On Average, How Many Meals does a HH have in a day? Component 1 Component 2 Component 3 Total Variance Explained 43.7% 30.9% 25.4% 2 Meals/Day -0.997 Dropped Dropped 1 Meal/Day 0.71 -0.697 Dropped 3 Meals/Day 0.492 0.862 Dropped Less than 1 Meal/Day Dropped Dropped 0.996 Overall, the results here suggest that while most options are internally consistent and combine to explain an adequate level of variance, households who have “less than 1 meal/day” should not be scaled with the other options. We recoded this item, dropping the “less than 1 meal/day” option, and achieved better results: On Average, How Many Meals does a HH have in a day? Component 1 Total Variance Explained 58.1% 1 Meal/Day 0.72 2 Meals/Day -0.99 3 Meals/Day 0.49 The refined scale loads better in terms of overall variance explained. However, the seemingly counter intuitive influence of 2 meals per day vs. the other options is worth further investigation. CPA 3: Health, Water, Sanitation and Shelter Health, Water, Sanitation and Shelter Component 1 Total Variance Explains 36.7% Household conditions Dropped Children HIV status known Dropped Stable shelter 0.858 Type of a latrine 0.856 39 Similar to the case of CPA 2, the items under CPA 3 come close to combining to explain an adequate level of variance, but do so at the cost of dropping half of its items. This should be intuitive for the HIV status item, as it is difficult to find a rational for why this should have any kind of meaningful relationship to latrine type, HH conditions and shelter conditions. However, we would expect the household conditions item to have some commonality with latrine type and shelter conditions. Below we discuss the issues with the household conditions item. Household Conditions Household Conditions Component 1 Component 2 Component 3 Total Variance Explained 22.5% 14.3% 12.6% Hand washing facility 0.708 Dropped Dropped Garbage pit /dust bin 0.695 Dropped Dropped Drying rack for HH utensils 0.639 Dropped Dropped Separate house for animals 0.502 -0.365 Dropped A clean compound 0.378 Dropped -0.614 Access drinking water w/in 30 minutes of walking? Dropped 0.677 Dropped Access to a health facility within 5kms? Dropped 0.532 0.673 All HH members sleep under a mosquito net? Dropped 0.46 Dropped The explained variance of this item is low. This item clearly attempts to include too many options, many of which most likely cannot be placed on the same underlying scale. One potential issue revolves around the nature of each option – the first five options implicate household behaviors and habits related to hygiene, while the next two options implicate geographic location and access. The last option – sleeping under a mosquito net – do not intuitively fit under either of these two suggested concepts. We therefore recoded the data by dropping the mosquito net option and broke this item down into the two concepts suggested above – hygiene and access. They yield the following results: Separation of Hygiene and Access Items Hygiene Component 1 Access Component 1 Total Variance Explained 35.90% 54.60% Hand washing facility 0.371 Dropped Garbage pit /dust bin 0.64 Dropped Drying rack for HH utensils 0.696 Dropped Separate house for animals 0.514 Dropped A clean compound 0.708 Dropped Access drinking water w/in 30 minutes of walking? Dropped 0.739 Have access to a health facility within 5kms? Dropped 0.739 The results reveal much higher explained variance for both new items and a fairly consistent grouping of options. Based upon the above, we would recommend eliminating this item and replacing it with the two new items suggested above. Does the HH have a stable shelter? 40 Does the HH have a stable shelter? Component 1 Total Variance Explained 39.4% Safe Shelter -0.988 Shelter in need of Repair -0.643 Inadequate Shelter 0.422 Unsafe Shelter Dropped This item just falls below the cutoff for an acceptable level of explained variance. However, the option “unsafe shelter” – typically a dilapidated shack with direct exposure to rain – drops out, which suggests that this type of shelter is better associated with a different aspect of vulnerability than the others. Alternatively, less than 6% of households fell into this category, which may suggest that its absence is due to a smaller sample size. Type of Latrine Type of Latrine used by HH Component 1 Component 2 Component 3 Component 4 Total Variance Explained 33.9% 25.0% 20.9% 20.2% Private Latrine needs Repairs -0.958 Dropped Dropped Dropped Private Latrine 0.866 -0.485 Dropped Dropped Private Shared Latrine Dropped 0.954 Dropped Dropped Outside/Bush Dropped Dropped 0.964 Dropped Public Latrine Dropped Dropped 0.185 0.98 This item falls apart using PCA. No single component was able to adequately combine all options to acceptable levels of explained variance. One potential grouping that emerges involves different types of private latrine options and outside/public latrine use. However, when we recoded and retested various configurations of the above (e.g. dropping private shared, combining various options, etc.), we were still unable to find an acceptable grouping. As such, we recommend that stakeholders revisit this item and consider alternative formulations, as well as more in-depth exploration on the relationship of latrine-type/use to vulnerability. The remaining HVAT items were too few in number to test as sub-components of each CPA. In the case of CPA 4, only one item comprised this category, while CPAs 5 and 6 only relied upon two items. We therefore combined there analysis below. CPA 4: Education School Attendance Component 1 Component 2 Component 3 Total Variance Explained 37.1% 29.2% 27.4% All Attend -0.946 Dropped Dropped More than half 0.451 0.777 -0.371 Less than half 0.563 -0.745 Dropped No child attends Dropped Dropped 0.942 41 Note that this is the only item under the Education CPA. The school attendance item mostly loads, but explains households that do not send any of their eligible children to school appear to fall on a different dimension than those households that have some minimal level of attendance. Moreover, partial attendance households also have a reverse relationship to full attendance households, which may suggest more qualitative differences between partial and full attendance in relation to vulnerability. Given these potential qualitative differences, we also explored combining lack of attendance with birth registration (see below under Child Protection), as a lack of registration is oftentimes a barrier to school attendance in Uganda. The results are the following: Child Eligibility Component 1 Total Variance Explained 52.6% No child attends 0.725 Lack of Birth Registration 0.725 Given the better fit and closer conceptual relationship the PCA results suggest, we recommend using these options to create a new item under CPA 4. CPA 5: Psycho Social Wellbeing Psycho Social Wellbeing Component 1 Total Variance Explained 55.6% Depressed Children in HH 0.746 Consult a counselor 0.746 Both psycho-social well-being items were spread according to number of visits and children, i.e. they were coded as pseudo-continuous variables, so it made little sense to recode them as dummy variables to explore categorical differences and relationships. Instead, we combined them to test their consistency under the Psycho Social Well-being CPA. Overall, this CPA has a good fit, makes intuitive sense and holds together. The only recommended change would be to explore adding more items to provide a more robust measure. CPA 6: Child Abuse Has any child in the HH had the following happen Component 1 Component 2 Component 3 Total Variance Explained 16.5% 13.6% 11.8% Repeated Physical Abuse 0.361 Dropped Dropped Withheld a meal as discipline 0.386 -0.642 Dropped Involved in Child Labor 0.588 Dropped Family Separation 0.5 0.465 Dropped Sexually Abused Dropped Dropped 0.645 Stigmatized and/or Discriminated 0.39 Dropped Dropped Yelling & Screaming as discipline 0.58 Dropped Dropped Conflict with Law Dropped Dropped 0.667 42 Not All Registered Dropped 0.631 Dropped This item has several options that do not combine well and overall lacks consistency. Indeed, it is hard to see how events such as sexual abuse can be assumed to fall along the same spectrum of abuse as a lack of birth registration or withholding a meal as punishment, and the PCA results reinforce these common-sense notions. Nevertheless, some patterns still appear that allow us to make some recommended revisions to this item. First, as mentioned above, the “not all registered” option appears to fit better when bundled with questions under the education section regarding school attendance, and we recommend moving this option to that section of the HVAT. Overall, the instability of this item most likely reflects the very complicated, overlapping and contradictory structure of options and should be reworked. We tested several configurations of options under this item to create new items with corresponding options. These were the highest scoring results: Child Protection Recoded as Abuse, Trauma and Discipline Abuse Trauma Discipline Component 1 Component 1 Component 1 Total Variance Explained 37.4% 41.6% 51.7% Yelling & Screaming as discipline 0.691 Dropped Dropped Sexually Abused 0.542 Dropped Dropped Repeated Physical Abuse 0.594 Dropped Dropped Family Separation Dropped 0.726 Dropped Involved in Child Labor Dropped 0.694 Dropped Stigmatized and/or Discriminated Dropped 0.49 Dropped Conflict with Law Dropped Dropped 0.719 Withheld a meal as discipline Dropped Dropped -0.719 Although none of the new items score particularly high, the groups of the first two do appear to be intuitively bundled. Yelling/screaming, sexual abuse and physical abuse do share sufficient amounts of consistency and explained variance to be useful, although additional options under this item should be explored. Family separation, child labor and stigmatization may reflect various aspects of trauma, and hold together slightly better than abuse. Finally, conflict with the law and withholding meals sit in direct opposition to each other in terms of their contribution to vulnerability. This may reflect different approaches to discipline (or lack thereof) but we would not recommend grouping these options together until more convincing explanations are found. Regardless, this item captures a wide array of information that is useful, but further work should be done on improving how it is scaled. Given the results of the above analysis, it is clear that many of the underlying assumptions of the HVAT scoring system are violated and skew the results. The implications are that HVAT scores can be a misleading proxy for comparing underlying vulnerability across households. For example, two households that hold the same overall vulnerability score but differ in their CPA distribution in fact may differ significantly. Alternatively, one potential, although not comprehensive, solution would be to scale total CPA scores instead of individual items. The disadvantages to this approach involve a loss of more detailed information on the many items that contribute to vulnerability, but the advantages are that several CPAs can be scaled across a single component. Our results are as follows: 43 Vulnerability Across CPA Scores Material Vulnerability Functional Vulnerability Total Variance Explains 37.8% 51% Economic Strengthening 0.703 Dropped Food and Nutrition 0.622 Dropped Health and WASH 0.669 Dropped Education 0.431 Dropped Child Protection Dropped 0.714 Psycho-Social Support Dropped 0.714 Based upon the above, we find a general clustering of the first four CPAs under what we define as “material vulnerability,” i.e. economic strengthening, food and nutrition, health and WASH and education. Combined under one scale, these CPAs explain close to 40% of the variance of all items. We also find a second dimension of vulnerability in which child protection and psycho-social support CPAs combine to form what we interpret as “functional vulnerability,” which also explains a much higher amount of variance across those two CPAs. Using the Material Vulnerability component as the basis for weighting the relative importance of each CPA, we can then use least squares regression to generate factor scores that correspond to household vulnerability.20 After each household is scored, we apply the same categorical criteria to identify which type of weighted vulnerability corresponds to each household. In doing so, we generated the “maximum” (64 points of vulnerability) and “minimum” (0 points of vulnerability) weighted vulnerability score to determine the upper and lower threshold, respectively. These generated upper and lower boundary scores of -2.001 to 2.446 across the entire spectrum. The distribution was then divided into quartiles and vulnerability score was assigned according to the same system used by the un-weighted approach (i.e. 0-24% = Not Vulnerable, etc.) The implications are that, as components, the CPA scores do a much better job of capturing the underlying dimensionality of vulnerability than breaking the scale down further into individual items (there is too much “noise” at the item level). The two new “meta-components then appear to call across material vs. psychological well-being, in which child protection is a subset of the latter while issues related to health are a subset of the former. Using this approach to assign weighted factor scores, we find the following distribution: Comparison of Weighted vs. Un-weighted Vulnerability Scores Category Original Score Weighted Scores Not Vulnerable 6% 7% Slightly Vulnerable 73% 64% Moderately Vulnerable 21% 29% Critically Vulnerable 0% 0% 20 Factor scores are usually generated as linear combinations of observed variables which consider what is shared between the item and the factor (i.e., shared variance), and what is not measured (i.e., the uniqueness or error term variance). Generating factor scores from CPA typically involves three scoring techniques: Regression, Bartlett, and Anderson-Rubin. There are advantages to each, but we chose Regression scores based upon their higher levels of validity. See “Understanding and Using Factor Scores: Considerations for the Applied Research,” in Practical Assessment, Research & Evaluation, Vol. 14, No. 20, October 2009. Found here at: http://pareonline.net/pdf/v14n20.pdf 44 The results revealed that while the relative weights of each CPA shifted and were more accurately represented than using the original raw scoring method, no substantial differences in the distribution of household vulnerability could be found. Geography The survey team also explored how geography may help explain the seemingly lower level of household vulnerability across both Cohort 1 and the baseline. Using data from a previous “Vulnerability Index” that was reported in MGLSD’s 2010 OVC Situational Awareness Report (p. 4), a quick comparison reveals that the Central region (which contains the control group) tends to have a lower rate of vulnerability compared to the Southwest and Western region (which mostly hosts R1 and R1R3 treatment groups): Critically Vulnerable Moderately Vulnerable Generally Vulnerable Total Vulnerability Central 7.8 33.6 52.7 94.1 Eastern 7.5 45.5 43.8 96.8 North 9.3 53.6 35.9 98.8 Western 8.1 41.1 45.9 96.1 Average 8.1 42.9 45.1 96.1 However, the survey team was unable to obtain the primary data behind these numbers, as well as the scaling methods used to derive them. As such, one should be wary of making direct comparisons with this data and further research is most likely required. HVAT Summary Our analysis suggested that with the current scoring, the HVAT tool potentially understates child protection risks and may omit children who with greater psychosocial needs. For example, Section 6 on Child Protection: if child was sexually abused and his/her parent would talk about it was the only child protection issue that happened with him/her, then based on current scoring (scored 2 out of 8=0.25%, it would be placed him/her in slightly vulnerable group. Moreover, the sections measuring child protection risks and children’s psychosocial, emotional and legal needs are very limited and do not provide sufficient information to say much about these dimensions. Sections CPA 4-6 include about 1-2 items per section and the score of 20 is maximum possible score for all these section combined. For these CPAs, the analysis revealed that items measuring child protection risks (physical or sexual abuse, conflict with law) and psychosocial needs (being in school, children feeling unhappy and sad) fall on different domains and lead to separate scaling measurements that cannot be easily combined with the economic, nutritional and health dimensions. Thus, when we compare the PCA results of each CPA, we find that the HVAT tool is predominantly capturing economic aspects of household vulnerability. While poverty is a significant risk factor, it is not the only contributor to child protection outcomes. In the future, it would be helpful to strengthen the tool by revising the current scoring structure and creating more in-depth assessment of child’s risks to healthy psychosocial development and growth that go beyond economic risks. Mapping the QCA System For the QCA design, the baseline survey team worked with Sustainable Outcomes staff to identify which sets of relevant implementation data will be used to answer the “how” questions of program effectiveness. Once in place, the QCA system will enable Sustainable Outcome implementer to use the results to provide ongoing program feedback, learning loops, mini-cost effectiveness analyses and midcourse program adaptations. Building the system entailed the following steps: 45 1. Socialized the Sustainable Outcomes teams on QCA: The study team gave presentations to Sustainable Outcome staff on how QCA functioned, it’s applicability to the program and next steps; 2. Confirmed the relevance of the QCA questions to be answered: The study also presented the QCA questions to the implementation team to discuss and confirm their relevance to management decision making. 3. Identified data inputs and modify, if needed and possible, the current MIS system: The study team held a number of meetings with the M&E team on the design and implementation of the program’s MIS. A number of changes were suggested (see below); 4. Understand the “human systems” behind the data: The team also met individually with program staff to identify data flows, reporting and roles/responsibilities associated with the collection of implementation data. To the extent this system becomes operationalized depends both on decisions by Sustainable Outcomes staff to adopt the system (e.g. include it in their workplan) and additional ongoing technical support to provide guidance to staff to ensure it is used. For this baseline, the team was limited to completing the steps above and mapping the potential data that will feed into the QCA model. Below is a discussion of those data points and where they lie. Sustainable Outcomes MIS The study team met with the M&E team to discuss and reviewed the program’s new prototype M&E MIS. Previously, Sustainable Outcomes did not have a centralized MIS and compiled data in MS Excel. The new MIS system is expected to go live in March 2017 and will be the main reporting system used by CRS staff as well as their implementing partners in the field. The M&E team also shared the prototype system with the study team. Below is a high light of which fields and pages hold relevant data for the QCA model, and includes screen shots from the actual database. Figure 5 below presents the main implementation home page: Figure 5: Sustainable Outcomes MIS Home Page 46 In what follows below, we identified subsequent pages and fields which are relevant to the QCA analysis. Moreover, staff also indicated that an additional tab with specific information regarding households enrolled under the DREAMS program would also be included.21 Capturing Result 1 The activities under Result 1 are collected under the “Results Area 01” page, which compiles all resources and services provided to each household under Result 1 over time. Training: All households that receive training under Cohort 2 will be recorded under the training page. Training options include: Better Parenting, Financial Literacy Skills, Smart Skills, Family Life, Youth Entrepreneurial Skills and others related to SRH. Crucially, attendance at these trainings can be linked to each household. Combined, this information will not only allow the research team to understand the role of training in general, but will also help identify which types of training are more relevant for the outcomes the program seeks to achieve. The training page also captures names of individual trainers, which could be useful in explaining potential contradictions whereby the same training is associated to two opposing outcomes (i.e. it could be that the trainer’s implemented differently). Figure 6: Training Page Services: In addition to training, the Result 1 section captures a variety of services provided to each household, including economic strengthening, psychosocial support, child protection and legal support. These services will be modeled to identify optimal sets of “essential packages,” i.e. combinations of services, which can be associated with changes to the outcome. Figure 7: Services Page 21 DREAMS – with the goal of helping girls develop into Determined, Resilient, Empowered, AIDS-free, Mentored, and Safe women – seeks to reduce HIV infections among adolescent girls and young women in 10 sub-Saharan African countries. DREAMS includes the Rakai, Mityana and Gomba districts. 47 Additionally, the Results 1 section captures information on:  Apprenticeship Register: Participants, in particular Youth enrolled in Sustainable Outcomes, will be tracked over time using the Apprenticeship register page. One of the benefits of this page is that it captures information on the enterprise youth are affiliated.  Girl Child Education: This page will be limited to those households that qualify and receive education scholarships and benefits for girls. Although it does not track education outcomes, it does track education levels, which can be useful to answer various sub-questions such as what is the optimal target age for these scholarships to better reduce vulnerability.  Value Chain Support: Should this aspect of Result 1 take on a large role, the database is set up to capture a variety of input/output data, which may be useful to confirm if and draw linkages to changes in a household’s economic position in connection to the economic strengthening efforts of Sustainable Outcomes. Capturing Result 3 One of the more important pages in the MIS involves the Home Visit page. This page is designed to act as a log of household visits conducted by social workers and Para-social workers to provide a variety of services and gather information for referrals. This page also captures the presence of a household improvement plan as well as follow-on actions. It will be crucial for modeling variations and difference configurations of activities provided under Result 3. Figure 8: Social Worker and Para-Social Worker Home Visit Tracker 48 It should be noted that this data will be cross checked with the situational awareness data collected by the baseline and subsequent survey information. Tracking Result 2 The OVCMIS does capture various program inputs provided to support district referral systems under Result 2 – such as the Alternative Care Panel, staffing and appraisals at Community-Based Service Departments (CBSD), support to OVC coordination meetings and usage of the OVCMIS, and institutional care reports. These inputs/outputs cannot be connected to specific households, but they will be built into the QCA model as important context variables that may or may not be relevant to the outcomes. Suggested Improvements to the System The new MIS captures a significant amount of data that can be built into QCA models to answer a wide array of implementation questions. However, the prototype system does contain a number of limitations that can be improved. These are as follows:  There are no fields to capture GPS coordinates. This should be added and audited on a regular basis.  The household coding system works as follows: {First two letters of Implementing Partner}{First two letters of CSO}{HH number spanning from 0000 to 9999} While this numbering system is intuitive for the user who understands the implementation partners on sustainable outcomes, it does not contain geographic information. It is also linked to the sub county level and regenerated across different subcounties. The implications are that households in different subcounties that are serviced by the same CSO can have the exact same HH ID number. This could become increasingly problematic overtime as many CSOs operate across subcounties.  Currently, users can only assigned a PSW to one sub county, and the system does not allow PSWs to work across subcounties. The implications are that some PSW efforts may be under represented in the system, or some PSW records will have to be duplicated across subcounties.  Villages and Parishes that have the same name across the 16 districts cannot be distinguished from each other in the Household ID system. The implications are that users may incorrectly assign 49 households to other districts. This only affects a few areas (we only found 6 villages that shared the same names), but this should be noted to avoid user error.  Currently, the referral page and the household visit pages are only able to capture one referral and one visit per household. It cannot capture multiple referrals. If we assume that households that require more referrals are more vulnerable, this deficiency in the system implies we will under represent Sustainable Outcome’s efforts to reduce vulnerability through referrals and potentially generate misleading results. Apparently, multiple visits are still captured and recorded in the system, but stored in the back-end. If possible, it would be better to create a view that compiles all of these visits and referrals for the user to see. These concerns aside, the new system is comprehensive, user friendly and well designed. The only other major potential challenge involves capacity. Sustainable Outcome IPs and CSOs will be required to use the database to report all of their activities across household visits and social worker records. Given the volume of information that will be inputted into the system, it is highly likely that data quality issues will emerge on a regular basis and require constant management and data audits. However, Sustainable Outcomes currently employs only a small number of M&E experts, none of which have database management expertise (the design of the system was outsourced to a firm in South Africa). Therefore, the program should consider employing additional staff to ensure high quality data is maintained. The QCA system has strong potential to enable Sustainable Outcome implementer to use the results to provide ongoing program feedback, learning loops, mini-cost effectiveness analyses and midcourse program adaptations. These steps are not yet formally adopted in the program’s workplan. 0 Conclusions and Recommendations These results lead to the following high level conclusions and recommendations:  The HVAT is an adequate tool to provide an overall snapshot of various dimensions of household vulnerability and is therefore useful for program decision making and addressing higher level evaluative questions of effectiveness. However, potential positive social response bias on the psycho-social and child protection CPAs suggest that results from these CPAs should be met with caution. Moreover, when comparing changes in vulnerability status using rigorous statistical approaches, the impact evaluation should break down the HVAT scores by CPA and item scores and search for statistical differences of each. Doing so will provide a more nuanced picture of what impacts have been achieved.  While the distribution of household vulnerability poses no threat to the design of the impact evaluation (and improves explanatory power of the QCA component), the lower seemingly number of more vulnerable households as well as the percentage of “not vulnerable households” may be a cause for concern. While there are some scaling biases built into the way vulnerability is scored, once accounted for they do not reveal any substantial differences in the distribution of vulnerability across the sample. Rather, the differences may be better attributed to Ugandan economic geography, but this conclusion requires further investigation. Regardless, the high number of “not vulnerable” households enrolled in Cohort 1 does suggest that Sustainable Outcome staff review their selection procedures.  The embedded QCA system is comprehensive, user friendly and well designed. Sustainable Outcomes staff should consider improving the system to capture GPS, avoid duplicates and capture ongoing referrals by implementing partners. Moreover, the program may also wish to consider employing additional M&E staff to manage the wealth and quality of data that is being captured. The QCA system has strong potential to enable Sustainable Outcome implementer to use the results to provide ongoing program feedback, learning loops, mini-cost effectiveness analyses and midcourse program adaptations. The program should consider formally including these steps in its current workplan. i Annex 1: Scope of Work SCOPE OF WORK: BASELINE EVALUATION OF SUSTAINABLE OUTCOMES FOR CHILDREN AND YOUTH PURPOSE OF THE EVALUATION Uganda is currently the second youngest population in the world and the third fastest growing nation in Africa. However, deep and extensive vulnerabilities exist that especially affect children: 96% of children are considered vulnerable, and 62% of those living in poverty are children. Vulnerabilities take different forms, including poverty, malnutrition, and exposure to violence. Low levels of education and high prevalence of HIV/Aids among children themselves as well as within their families exacerbate these vulnerabilities. USAID/Uganda’s Orphans and Vulnerable Children (OVC) programming employs an integrated approach to address these factors. The Sustainable Outcomes for Children and Youth (Sustainable Outcomes) program aims to economically empower children, youth, and their caregivers to access core services, strengthen systems to provide core services, and improve coordination of community-based clinical and socio-economic services for efficiency and effectiveness along the continuum of care. USAID/Uganda wants to a) establish a baseline regarding key indicators and b) establish procedures that may be used for future monitoring and evaluation of OVC programs using Qualitative Comparative Analysis (QCA). BACKGROUND Program Summary Activity Name Sustainable Outcomes for Children and Youth (Sustainable Outcomes) Implementer Catholic Relief Services (CRS) Cooperative Agreement # AID-617-A-15-00005 Total Estimated Ceiling of the Evaluated Activity (TEC) $40,526,948 Life of Activity April 2015 – April 2020 Active Geographic Regions 17 districts in Central and Western Uganda USAID Office Education, Youth, and Childhood Development Sustainable Outcomes for Children and Youth is implemented by Catholic Relief Services (CRS) in partnership with a consortium of organizations, including Action for Community Development (ACODEV), African Network for Prevention and Protection against Child Abuse and Neglect (ANPPCAN), TPO Uganda, and Palladium (formerly Futures Group). ii Figure 1: Sustainable Outcomes Results Framework iii Development Hypothesis and Theory of Change Sustainable Outcomes for Children and Youth aims to address multiple forms of vulnerability among children and youth through an integrated, comprehensive approach (Figure 1). Specifically, it will economically empower orphaned and vulnerable children, youth, and their caregivers to access core services (Result 1), strengthen local government, CSOs, and informal community structures to increase and improve core services for orphaned and vulnerable children, youth, and their caregivers (Result 2), and improve coordination of community-based clinical and socio-economic services for efficiency and effectiveness along the continuum of care (Result 3). Sustainable Outcomes’ Development Hypothesis states that if local governments and CSOs increase and improve core services for OVC, youth and their caregivers, and community-based clinical and socio-economic services become more responsive and collaborate more effectively, then an enhanced, social support system will develop through which OVC, youth, and households will access core services, for improving health, nutrition, education, and psychosocial well-being and reducing abuse, exploitation and neglect. Overview of Interventions Households are enrolled into the program on the basis of certain criteria of vulnerability, which are assessed using the Ministry of Gender, Labour and Social Development-approved national tool: the Household Orphans and Vulnerable Children Vulnerability Prioritization Tool (HVPT). Alternatively, households are enrolled via referral from health facilities. Information from the vulnerability assessment then also serves as the basis for a tailored household action plan for referrals and interventions. Parasocial workers are trained to supervise the implementation of household action plans. Households are expected to graduate out of vulnerability within one or two years, but may be supported for up to three years. Depending on the circumstances of the household, participation in a savings and internal lending communities (SILC) savings group may be offered to them. SILC groups meet weekly and these meetings also provide a platform for financial education and training in parenting and communications within families, regarding for example gender roles and how to make joint decisions. Sustainable Outcomes also works with private sector to link households to markets and internship or apprenticeship opportunities. Sustainable Outcomes trains SILC agents for 12 – 18 months, after which SILC groups are expected to be self-sustaining. Households that don’t have the means to participate in a SILC group are linked to cash transfer opportunities. System strengthening interventions include support to district and sub-county OVC coordination committees (DOVCCs and SOVCCs) to develop operational plans, undertake support supervision, monitor and assess OVC program performance, carry out quality assessments and support supervision, data analysis, dissemination and advocacy. In addition, system strengthening interventions focus on optimal utilization of available resources for OVC services within local government, CSOs and informal community structures, tracking staffing levels and training Para-social workers, and support for the use of data. Interventions also aim to improve the functioning of the referral network, increase utilization of core services, and expand the case management approach to build long term sustainability. Through enhanced coordination, user-friendly directories, effective follow-up mechanisms and improved service provider capacities, OVC and caregivers who access one key core service will be linked to multiple core services that pave the way to sustained health outcomes and graduation from social assistance. Current Status of Implementation Sustainable Outcomes is intended to operate in 17 districts in Western and Central Uganda (Annex A). These districts were identified by the United States Agency for International Development (USAID) due to high prevalence rates of HIV and availability of trained community workers to deliver program services. System strengthening interventions under Result 2 have been rolled out in all target communities since October 2015. Household level interventions (Result 1 and Result 3) are being phased in according to the following plan: iv  Year 1 (since May 2016): 25% of communities  Year 2 (planned for October 2016): 40% of communities  Year 3 (planned for October 2017): 35% of communities Critical assumptions are that there is sufficient and quality of health and education services for the OVCs to access services and in addition, other Implementing Partners and USAID supported IPs to provide HIV/AIDS services to beneficiaries, and that local governments have adequate staff to support supervise and ensure quality of services to vulnerable children by service providers. Sustainable Outcomes Activity Monitoring, Evaluation and Learning Plan The evaluator will review information available through Sustainable Outcomes’ own data collection efforts, including vulnerability assessments and AMELP data, as well as other existing data sources, and utilize available data where possible. Sustainable Outcomes has undertaken a baseline survey in all three regions of the program (Central, South-Western and Western) in a random selection of the 17 project districts to establish pre-intervention conditions to inform the development of interventions and provide a basis for project monitoring and outcome monitoring indicators. The survey focuses on obtaining information on (1) characteristics of the households (economic status; clinical, community and social protection services received); (2) characteristics of the selected index child (caregiver’s knowledge of child’s HIV status, possession of a birth certificate for child, child’s school attendance, services currently receiving); and (3) attitudes of the caregiver towards physical punishment. Annex B shows a list of indicators on which the baseline has collected data. In addition, an external firm will conduct outcome monitoring of the Essential PEPFAR Indicators shown in Annex C at year one, three and year five of program implementation. EVALUATION QUESTIONS, DESIGN, AND METHODOLOGY With this Scope of Work (SOW), USAID/Uganda seeks to establish a baseline regarding key program indicators. Program outcome indicators of interest are linked to the three areas of interventions and are to be refined in collaboration between USAID/Uganda, Sustainable Outcomes, and the evaluator:  Result 1: financial self-efficacy in terms of a household's ability to meet basic needs of children in their care, including nutrition needs, school attendance, children's psychosocial wellbeing, as well as economic resilience to shocks, such as unexpected medical expenses and others.  Result 2: adequate functioning and/or efficiency of formal and informal systems and reporting, for example through case follow-up and capacity of probation officers.  Result 3: functioning of case management and referral systems. As stated above, the evaluator shall review and utilize existing data sources to the extent possible. However, the evaluator is also expected to develop survey instruments to address data needs not covered by existing sources. Data should be collected from two groups of beneficiaries: those who receive system strengthening only during year 2 and those who also receive household level interventions (see section 2). Data collection is expected to take place before program interventions begin in Cohort 2, so that a clean baseline for Cohort 2 can be established, but may include other cohorts to be determined in collaboration between USAID/Uganda, Sustainable Outcomes, and the evaluator. Data should be disaggregated by sex, disability status, single- vs. two-parent household, district, and possibly other beneficiary characteristics to be determined in collaboration between USAID/Uganda, Sustainable Outcomes, and the evaluator. Sample size calculations must take the disaggregation into account. Selection of indicators, instruments, and methods must be suitable to enable the use of Qualitative Comparative Analysis (QCA). QCA is a method to identify necessary and sufficient factors that can combine in various ways to produce an outcome. In short, QCA defines desired outcomes (such as for example school enrollment) and analyzes the presence or absence of various factors (such as specific v interventions) in conjunction with occurrence of the outcome. The method is superior to more traditional evaluation methods for evaluating a program with highly integrated interventions such as Sustainable Outcomes, because it allows identifies the contribution of each intervention as well as combinations of interventions. The evaluator will establish and document procedures that can be used for future monitoring and evaluation of Sustainable Outcomes and other OVC programs using QCA. The procedures developed under this SOW may be used to adapt Sustainable Outcomes’ own data collection activities as well as future evaluations of this and similar programs. This work will help USAID/Uganda in the long term to better answer overarching questions regarding the effectiveness of OVC programs, such as:  To what extent do program outcomes improve over time?  To what extent do household level interventions (as opposed to system strengthening alone) contribute to these changes?  How do program interventions affect change?  How cost effective are different pathways to change?  Sustainability: To what extent are processes being put in place to facilitate long-term success of program interventions beyond the program’s end? Which factors exist that may prevent long-term success? The ability to answer these questions will help USAID/Uganda understand to what extent the full set of program interventions has a positive effect on children and youth wellbeing, and how different interventions or combinations of interventions of the program contribute to program outcomes. This information will be crucial to inform decisions about future OVC programs. DELIVERABLES UNDER THIS SOW 1. Inception meeting: Within 1 week of award, the evaluation team will meet with USAID/Uganda to discuss the team’s understanding of the assignment, initial assumptions, evaluation questions, design, methodology, and work plan, and/or to adjust the Statement of Work (SOW), if necessary. 2. Concept note and work plan: Within 2 weeks of award but no later than July 29, 2016, the evaluation team must submit to the Agreement Officer’s Representative/Contracting Officer’s Representative (AOR/COR) a concept note and work plan, which will include: (1) a set of proposed indicators that will be measured, including indicator definitions and rationale for selecting them linking back to the evaluation purpose, (2) a detailed data collection plan that explains how indicators are going to be measured and how the data collection plan enables use of QCA; (2) draft questionnaires and other data collection instruments or their main features; (3) the list of potential interviewees and sites to be visited and proposed selection criteria and/or sampling plan (must include calculations and a justification of sample size, plans as to how the sampling frame will be developed, and the sampling methodology); (4) known limitations to the design, (5) the anticipated schedule and logistical arrangements; and (6) a list of the members of the evaluation team, delineated by roles and responsibilities, and (7) level of effort and cost information. USAID offices and relevant stakeholders are asked to take up to 5 business days to review and consolidate comments through the AOR/COR (by August 5, 2016). Once the evaluation team receives the consolidated comments on the initial concept note and work plan, they are expected to return a revised concept note and work plan within 5 business days (August 12, 2016). 3. Protocol for IRB submission: Within 4 weeks of award but no later than August 12, 2016, the evaluation team must submit to the Agreement Officer’s Representative/Contracting Officer’s vi Representative (AOR/COR) a data collection protocol ready for submission to the appropriate IRB and initiate the IRB process, expected to take up to one month. 4. Data collection: The evaluation team must complete all necessary preparations, such as enumerator recruitment and training and translations into local language, as well as the data collection according to the approved protocol no later than by the end of September 2016 given the anticipated enrollment of Cohort 2 into the program in October 2016. 5. Draft baseline report: The draft evaluation report should be consistent with the guidance provided in Section IX: Final report format. The submission date for the draft evaluation report will be determined in the evaluation work plan. Once the initial draft evaluation report is submitted, USAID/Uganda will have 10 business days in which to review and comment on the initial draft, after which point the AOR/COR will submit the consolidated comments to the evaluation team. The evaluation team will then be asked to submit a revised final draft report 10 business days hence, and again USAID/Uganda will review and send comments on this final draft report within 10 business days of its submission. 6. Final baseline report: The evaluation team will be asked to take no more than 10 business days to respond/incorporate the final comments from USAID/Uganda. The evaluation team leader will then submit the final report to the AOR/COR. All data and records will be submitted in full and should be in electronic form in easily readable format in line with all applicable provisions regarding privacy and protection of participants, organized and documented for use by those not fully familiar with the project or evaluation, and owned by USAID. 7. Final presentation: The evaluation team is expected to hold a final presentation in person/by virtual conferencing software to discuss the summary of findings and recommendations to USAID. EVALUATION TEAM COMPOSITION The evaluation team should consist of a team leader with expertise in evaluations. The evaluation team should further demonstrate experience in M&E of OVC programs, knowledge of the Ugandan context, as well as strong skills in qualitative and quantitative research, including survey design and analysis, and the ability to oversee any necessary translation of survey tools into local language. The evaluation team must include a specialist in the QCA method. A point of contact from USAID/Uganda will work closely with the evaluation team and may observe some of the data collection efforts. FINAL REPORT FORMAT The final baseline report should include an executive summary; introduction; background of the local context and the program being evaluated; the main evaluation purpose; the methodology or methodologies; the limitations to the design; findings, conclusions, and recommendations; and lessons learned (if applicable). The report should focus on a) presenting baseline findings, and b) explaining how the procedures used can inform the design of future monitoring and evaluation activities using QCA. The executive summary should be not more than 3 pages in length and summarize the purpose, background of the project being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations and lessons learned (if applicable). The baseline data collection methodology shall be explained in the report in detail. Limitations shall be disclosed in the report, with particular attention to the limitations associated with the methodology (e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.). The annexes to the report shall include: vii  The Evaluation SOW;  Any statements of difference regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team;  All tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides;  Sources of information, properly identified and listed; and  Disclosure of conflict of interest forms for all evaluation team members, either attesting to a lack of conflicts of interest or describing existing conflicts of. CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT Per the USAID Evaluation Policy and USAID ADS 203, draft and final evaluation reports will be evaluated against the following criteria, as applicable, to ensure the quality of the evaluation report. 22  The evaluation report should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate what worked in the project, what did not, and why.  Evaluation reports shall address all evaluation questions included in the SOW.  The evaluation report should include the SOW as an annex. All modifications to the SOW— whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline—need to be agreed upon in writing by the AOR/COR.  The evaluation methodology shall be explained in detail. All tools used in conducting the evaluation—such as questionnaires, checklists, and discussion guides—will be included in an annex in the final report.  Evaluation findings will assess outcomes and impact on males and females.  Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.).  Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or the compilation of people’s opinions. Findings should be specific, concise, and supported by strong quantitative or qualitative evidence.  Sources of information need to be properly identified and listed in an annex.  Recommendations need to be supported by a specific set of findings.  Recommendations should be action-oriented, practical, and specific, with defined responsibility for the action. OTHER REQUIREMENTS All quantitative data collected by the evaluation team must be provided in machine-readable, non-proprietary formats as required by USAID’s Open Data policy (see ADS 579). The data should be organized and fully documented for use by those not fully familiar with the project or the evaluation. USAID will retain ownership of the survey and all datasets developed. All modifications to the required elements of the SOW of the contract/agreement, whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline, need to be agreed upon in writing by the COR. Any revisions should be updated in the SOW that is included as an annex to the Evaluation Report. LIST OF ANNEXES  ANNEX A: SUSTAINABLE OUTCOMES DISTRICTS  ANNEX B: SUSTAINABLE OUTCOMES BASELINE INDICATORS 22 See Appendix I of the Evaluation Policy and the Evaluation Report Review Checklist from the Evaluation Toolkit for additional guidance. viii  ANNEX C: ESSENTIAL PEPFAR INDICATORS FOR OUTCOME MONITORING ANNEX A: SUSTAINABLE OUTCOMES DISTRICTS Bushenyi Gombe Hoima Isingiro Kabale Kampala Kamwenge Kanungu Kasese Kibaale Kiruhura Kyenjojo Luweero Mityana Ntungamo Rukungiri Wakiso ANNEX B: SUSTAINABLE OUTCOMES BASELINE INDICATORS 1 DSD OVC_SERV: Number of active beneficiaries receiving support from PEPFAR OVC programs to access HIV services 2 OVC_MONEY: Percent of households able to access money to pay for unexpected household expenses 3 Number of OVC and youth whose households were linked to social protection assistance 4 Number of project-supported savings groups 5 Number of girls supported with financial support for further education 6 Number of youth participating in agro-enterprise training 7 OVC_CP: Percent of caregivers who agree that harsh physical punishment is an appropriate means of discipline or control in the home or school 8 OVC_BCERT: Percent of children who have a birth certificate 9 Proportion of districts with functional district OVC coordination mechanisms 10 Proportion of annual local government resource allocation for children and youth from local revenues sources and unconditional grants 11 Proportion of districts with 80% of key positions substantively filled 12 Proportion of districts using OVCMIS for planning 13 Number of district alternative care panels supported 14 OVC_HIVST Proportion of beneficiary children whose primary caregiver knows the child’s HIV status 15 Proportion of referred OVC, youth and caregivers that complete a referral 16 Number of very young children reached with ECD services ix 17 DSD OVC_ACC: Number of active beneficiaries receiving support from PEPFAR OVC programs to access HIV services ANNEX C: ESSENTIAL PEPFAR INDICATORS FOR OUTCOME MONITORING x Annex 2: Survey Instrument Household Caregiver Oral Questionnaire Identifier Page DISTRICT: SUBCOUNTY: PARISH: VILLAGE: NAME OF INTERVIEWER: DATE OF INTERVIEW: NAME OF INTERVIEWEE: HOUSEHOLD CODE: GPS: Latitude: S __ __.__ __ __ __º Longitude: E __ __.__ __ __ __º Start Time: End Time: Questionnaire Serial Number: INTERVIEW LOG VISIT 1 VISIT 2 VISITI 3 DATE (day/month/year) INTERVIEWER COMMENTS Interview comment codes: Interview completed 1; Appointment made for later today 2; Appointment made for another day 3; Refused to continue and no appointment made 4; Other (Specify) 5 101 102 103 104 105 106 Line Please give the names of persons who usually live in household, starting with head of household. If (NAME of Child), what is the age? less than 1 year=0 If (NAME of child) what is the gender? (Female=1, Male=2) Does (NAME of Child) usually live here? (Y/N) Do you usually cares for/looks after (NAME of Child)? (Yes=1, Other=Give Name, Selfcare=0) If care giver, what is your relationship to (NAME of child)? (see codes below) A B C D E F G H I J K L M N CODES FOR Q105: RELATIONSHIP TO RESPONDENT 01 = BIOLOGICAL MOTHER 03 = NON-BIOLOGICAL PARENT 05 = AUNT/UNCLE 02 = BIOLOGICAL FATHER 04 = SIBLING 06 = GRANDPARENT xi Household Vulnerability Module Below is the main vulnerability module used for the baseline survey. The four leftmost columns identify from which tool the question was taken, and include MGLSD’s HVAT, Sustainable Outcomes’ HAT, MGLSD’s IPT, and the OVC indicator questionnaire used for PEPFAR indicators. Questions 44-47 were taken from the OVC situational awareness survey. Finally, the subsequent child inventory page and Index child modules were taken from the OVC indicator questionnaires. OVC IPT HAT HVAT ECONOMIC STRENGTHENING 1 1. Is this a child headed household? Yes No 4 2. Does the household head, spouse or guardian have any form of severe disability (e.g., physical, speech, visual, hearing, or mental handicap?)? Yes No X 3. Does the household have ANY member who has been very sick for at least three months during the past 12 months? (By very sick, I mean that the household head or any member was too sick to work or do normal activities around the house for at least three of the past 12 months) Yes No 2 1.4.4 4. Is there at least one member of the household who currently has formal or informal employment, is self-employed, has a business, or is engaged in an economically productive activity? Yes No 1 1.5 5. Who is the main contributor to household income? A Child Grandparent(s) Parent(s) Relatives/ Others 1.6 6. What is the current monthly HH income? (express in Uganda Shillings) <50k 50k – 99k 100k – 149k 150k￾200k >200k 2 1.2 7. What is the main source of household income? Formal Job/ Business Informal Job Casual Labour Remittance/ Others 1.5 8. The last time there was an unexpected urgent household expense (e.g. emergency medical expense or house repair), HH was able to pay that expense? Yes No 3 1.5.1 9. If the household incurred any HEALTH￾related expenses in the past three months, was it able to pay for these expenses? Yes Not always No 4 1.5.2 10. If the household incurred any SCHOOL￾related expenses in the past three months, was it able to pay for these expenses? Yes Not always No 5 1.5.3 11. If the household incurred any FOOD-related expenses in the past three months, was it able to pay for these expenses? Yes Not always No 6 1.4.5 12. Does anybody in the HH belong to any financial savings and lending group? Yes No 7 1.4.1 13. Any member of the HH owns an electronic gadget (radio, phone, TV) Yes No xii 1.4.2 14. Any member of the HH has a functional transport means (bicycle, motor cycle, boat) Yes No 14.3 15. Any member of the HH has vocational/apprenticeship/professional skills? Yes No 1.4.6 16. Household has domestic animals (cow(s), goat(s), Sheep, chicken, pig(s))? Yes No 1.4.7 17. HH has access to land for agriculture/hire? Yes No 7 7 18. Does the household own any of the following assets: cooking stove, beds, blanket, mattresses, mosquito net, shoes, cooking utensils, furniture. Yes, most of them Yes, some of them None or almost none OVC IPT HAT HVAT FOOD SECURITY AND NUTRITION 8 2.2 19. What does the family usually eat? (at least 3 times a week) Energy foods: (potatoes, banana, oils, posho, millet, rice, maize, bread, cassava) Yes No Body building foods: (beans, meat, soya, peas, milk, eggs, chicken, fish) Yes No Protective and regulative foods: (tomatoes, oranges, pawpaw, mangoes, pineapple) Yes No 2.1 20. Over the past month, what has been the MAIN source of food consumed by your household? Home grown Donated Bought Given for Work 9 2.3 21. How many meals does the household have in a day? Not everyday One meal per day Two meals per day Three or more X 22. Over the past month, did anyone in the household ever go without food for a whole day because there wasn’t enough? Yes, more than 5 Times a Month Yes, 1-4 times a month No 5 23. Has this household eaten at least 2 meals a day, every day, for the last month? OVC IPT HAT HVAT HEALTH, WATER, SANITATION AND SHELTER X 3.1.3 24. What is the distance (in Km) to the health care facility your household often uses? Kms Don’t Know X 25. When was the last time a member of the HH accessed a health facility? Weeks/Months Don’t Know 9 3.2 26. Does the caregiver know the HIV status for all children in the household? Yes No X 3.1.8 27. Do all HH members sleep under a mosquito net? Yes Some No X 8 28. Have all the children in the household been tested for HIV? Yes Some No 29. Are all eligible children who are HIV+ and/ or have TB on treatment? None of the Children Less than half of the Children More than half of the Children All of the Children 10 3.3 30. [If you don’t know of any HIV+ members of the household, ask this question]: Is there any member of the household who has a chronic disease? (HIV+, Cancer, TB, Sickle cells, diabetes etc.). If you know there is an HIV+ person in the household, you do not have to Yes No xiii ask this question, but check yes. X 31. What is the main source of water for members of your household? River, Lake, Pond, Unprotected well Public taps, Bore hole, Rainwater, Protected spring/well Private Connection 6 10 3.1.1 32. Does the household access drinking water from a safe source within 30 minutes? Yes No 11 33. Does the household have access to a latrine? Yes owned Yes shared No latrine 3.1.2, 3.1.4- 3.1.7 34. Observe the following: Has a clean compound Yes No Has a drying rack for HH utensils Yes No Has a garbage pit /dust bin Yes No Separate house for animals Yes No Hand washing facility Yes No 7 3.4 35. Observe: Does the HH have a stable shelter that is adequate, safe and dry? Not safe or stable shelter Inadequate, needs major repairs Needs some repairs but adequate Safe, adequate and dry 3.5 36. Observe: What type of a latrine/toilet facility is used by the HH? Bush/None Public Toilet for Pay Private, needs repair Private, adequate, but shared Private, safe, adequate, dry OVC IPT HAT HVAT EDUCATION X 11,12 4.1 37. How many of the children aged 5–17 years in this HH are not going to school or miss school 3 or more times a week [DO NOT INCLUDE 1-4 years] None attend Less than half attend regularly More than half attend regularly All attend regularly OVC IPT HAT HVAT PSYCHOSOCIAL SUPPORT AND BASIC CARE 5.1 38. In the past 12 months (STATE MONTH), how often has someone in your HH felt so troubled that it was necessary to consult a spiritual, faith or traditional healer, counselor or health worker? Enter number Not at all 5.2 39. Are there any children in this HH who are withdrawn or consistently sad, unhappy or depressed, not able to participate in daily activities including playing with friends and family? Enter number Not at all OVC IPT HAT HVAT CHILD PROTECTION AND LEGAL SUPPORT X 40. Do you think that hitting or beating a child is an appropriate means of discipline or control in the home? Yes No X 41. Do you think that hitting or beating a child is an appropriate means of discipline or control at school? Yes No 12 6.1 42. In the past month, have you or another adult in the household used the following method of discipline with any child in your household? (Please select all the methods that apply) Punched, kicked or hit as punishment Withheld a meal as punishment Yelling and screaming 6.2 43. In the past 12 months, has any child in the Repeated physical abuse xiv HH had the following happen to them, in or outside of the HH? [Note: If you see an obvious issue of abuse or you already know about it, then indicate yes]. Indicate Yes/ No Involved in Child Labour Family separation (ran away, chased) Sexually abused, defiled, raped, forced sex Stigmatised/ discriminated due to illness, disability or otherwise In contact/conflict with the law All Below Taken from Situational Awareness Survey HOUSEHOLD SERVICES 44. Has your household received services or participated in activities from a community based program in the last six months? By this I mean, in the last six months have you or someone in your household been visited by a community worker, or participated in any activities organized by the program such as a savings group or parenting program? Yes If No, then end module and go to next module 45. Are any or all of the services your household is receiving or participating in provided by Sustainable Outcomes Yes No Don’t Know 46. What type of household based services or activities (through a visit from a volunteer) has your household received or participated in the last six months? Circle all that apply Household visits from a volunteer Direct financial support Parenting counseling Early Childhood Development Health and hygiene HIV and GBV prevention Nutrition counseling Pre/post-partum counseling HIV testing Couples counseling Support obtaining a birth certificate Child protection Psychosocial support/basic needs Food security Other_______ None 47. What type of community based services or activities (outside of the home) has your household received or participated in in the last six months? Circle all that apply Savings groups Parenting program Government SAGE program Any other cash transfer Voluntary HIV testing/counseling Food security and Nutrition Skills and employment training Entrepreneurship training Other:_____ None xv Child Inventory Page You told me at the beginning of the interview that you are responsible for [refer to total # of children from Question 105 on the Identifier page] children (0-17 years). Starting with the oldest, please tell me the first names and ages of these children. Make sure the total number of children is the same as question 105 and Proceed to fill out chart HOUSEHOLD CODE: INTERVIEW DATE ID Name Y.O.B Sex B.Regist r Disable Immuni zed HIV status ART Works? Attends School Educati on Sexually Active? Protecti on Pregnan t Ever given birth Marital Status (M.F) (Y/N) (Y/N) or MD/PD (Y/N) (+/-/?) (Y/N) (S, W, UE, NA) (Y/N) Class complete d (Y/N) PA/EP/ EM,SA (Y/N) (Y/N) M or S 01 02 03 04 06 07 08 09 10 11 Assessor’s Comment: xvi Index Child and Kish Grid Page Information is required for only one child in each age group. If there is only one child in a given age group, ask the caregiver to reference that child in his/her responses to the questions pertinent to that age group. If there is more than one child in any age group, use the Kish grid to randomly select one child from each age group. Once you have selected the reference child in each age group remind the caregiver that his/her responses pertain to that child only. If there is no child of a given age group in the household, skip the module for that age group. Kish Grid In each age-group specific table below, list the names of all eligible children from oldest to youngest. Using the last digit of the serial number of the questionnaire, find that number along the top row of the table. Follow that number down to the last line where a child is listed. The number that you come to is the number of the child that should be surveyed (as indicated in the number column on the far left). Circle the index child selected from each age group or indicate in writing if there are no children in a given category. # Name of eligible individuals 0-9 years listed from oldest to youngest Age (0-9) Last digit of questionnaire serial number 1 2 3 4 5 6 7 8 9 0 1 1 1 1 1 1 1 1 1 1 1 2 1 1 2 2 1 1 2 2 1 1 3 3 2 1 3 2 1 3 2 1 3 4 2 3 4 1 2 3 4 1 2 3 5 5 4 3 2 1 5 4 3 2 1 1 2 3 4 5 # Name of eligible individuals 10-17 years listed from oldest to youngest Age (10-17) Last digit of questionnaire serial number 1 2 3 4 5 6 7 8 9 0 1 1 1 1 1 1 1 1 1 1 1 2 1 1 2 2 1 1 2 2 1 1 3 3 2 1 3 2 1 3 2 1 3 4 2 3 4 1 2 3 4 1 2 3 5 5 4 3 2 1 5 4 3 2 1 1 2 3 4 5 xvii Child Questionnaire aged 10-17 years SECTION 1: BACKGROUND INFORMATION Let’s start out by you telling me a little about yourself. No. Questions Coding Categories 101 Record / Confirm Child’s Name What is your name? 102 Record Child’s Line Letter from Household Schedule (Caregiver Questionnaire) 103* Record / Confirm Child’s Sex Female Male 1 2 104 In what month and year were you born? Month [____] Year [__|__|____] 105* How old were you at your last birthday? Confirm with 104 and adjust if necessary. Do not leave blank. If child does not know, ask caregiver to estimate age of child. [____] years 106 Who takes care of you? Do not read responses. Record one primary response only. Mother and/or father Sister and/or brother Aunt and/or uncle Grandmother and/or Grandfather Other relative Neighbor Friend No one/self Other: __________________ 1 2 3 4 5 6 7 8 66 --- END OF SECTION --- 18 SECTION 2: DIARY I would like you to talk to me about your day yesterday. If yesterday wasn’t a school day, ask about last school day. No. Questions Coding Categories SKIP 201 When did you get up – would you say, before the sun was up/it got light or after the sun was up/it got light? Before sunrise After sunrise 1 2 If After: 203 202 And what did you do after you got up, but before it got light? Anything else? Mark X in all applicable boxes in diary 203 Now, thinking about the time between when the sun came up/it got light and noon/the middle of the day, what did you do? Anything else? Mark X in all applicable boxes in diary 204 And around noon, what did you do? Anything else? Mark X in all applicable boxes in diary 205 Now, let’s think about the time between noon sundown/when it started to get dark, what did you do? Anything else? Mark X in all applicable boxes in diary 206 Now, let’s think about after it got dark. What did you do before you went to sleep? Anything else? Mark X in all applicable boxes in diary Instructions: Ask about the time frames one at a time; probe for additional activities before going on to the next time frame. Every column should have at least one activity box marked. Multiple activities permitted. Do not read response options. Activity Time 202 Before sun-up 203 Sun-up to noon 204 Noon 205 Noon to sun-down 206 After sun- down Sleep Meal Household chores Work on family / household farm Care for household member - child Care for household member - adult School attendance School work Work (excluding household chores) Informal recreation/leisure Organized recreation/club Other: specify ________________________________ SECTION 3: EDUCATION No. Question Coding Category SKIP 301* Are you currently enrolled in school? Yes (correct diary) No 1 2 If No: 19 306 302* During the last school week, did you miss any school days for any reason? Yes No 1 2 If No: 304 303 Why did you miss school days during the last school week? Do not read responses. Circle one primary response. No money for school materials, transport I am too sick to attend school School is too far away / no school I have to work I have to care for household members Parent/guardian does not want me to go to school I don’t like school School was not in session Other: ___________________________ 1 2 3 4 5 6 7 8 66 304* What grade/form/year are you in now? [____] All: 307 305 Why do you NOT go to school? Do not read responses. Circle one primary response. No money for school materials, transport I am too sick to attend school School is too far away / no school I have to work I have to care for household members Parent/guardian does not want me to go to school I don’t like school School was not in session Other: ___________________________ 1 2 3 4 5 6 7 8 66 306 Have you ever attended school? Yes No 1 2 If No: 401 307* Were you enrolled in school during the previous school year? Yes No 1 2 If No: 309 308* What grade/form/year were you in during the previous school year? [____] All: 401 309* What is the highest grade/form/year that you have completed? [____] SECTION 4: CHORES & WORK No. Questions Coding Categories SKIP 401 Check DIARY. Were the household chores and/or care for your family or household, mentioned? Yes No 1 2 If Yes: 403 402 Do you sometimes do household chores, or care for a member of your household? Yes (correct diary) No 1 2 If No: 405 20 No. Questions Coding Categories SKIP 403 What household chores do you usually do? Anything else? Multiple responses possible; circle all mentioned. Probe with response categories if necessary. Corroborate with diary. Prepare food 1 Fetch water 2 Clean toilets 3 Take care of children 4 Plant/tend to/harvest crops 6 Feed, care for animals 7 Wash clothes, blankets 8 Other: _____________________________ 66 404 About how much time do you spend per day doing household or farm chores for your family? Less than 1 hour 1-2 hours 3-4 hours More than 4 hours / most of the day It depends / it is different everyday 1 2 3 4 5 405 Check DIARY Was other work mentioned? Yes No 1 2 If Yes: 407 406 Apart from these chores, do you sometimes do other work outside your home? Yes (correct diary) No 1 2 If No: 411 21 No. Questions Coding Categories SKIP 407 What kinds of other work do you sometimes do? Anything else? Multiple responses possible; circle all mentioned. Probe with response categories if necessary. Corroborate with diary. Hawk goods 1 Sell food at market 2 Household / farm chores for other families 3 Work in a restaurant or bar 4 Help out in shop 5 Construction 6 Sewing 7 Mechanic 8 Clerk, Delivery, Administrative 9 Other: __________________________ 66 408 How often do you do other work? Would you say….? Read response categories Every day / most days 1 If Every day (1): 409 All others: 410 Several times a week 2 Once a week 3 Once in a while 4 409 About how much time do you spend per day doing this work? Less than 1 hour 1-2 hours 3-4 hours More than 4 hours It depends / it is different everyday 1 2 3 4 5 410 Have you ever received money for any of the work that you do? Yes No 1 2 411 What [else] do you do to get money? Nothing Begging Other: __________________________ 1 2 66 If work mentioned, return to 406-410. 412 What do you do with the money you get? Anything else? Multiple responses possible; circle all mentioned. Probe with response categories if necessary. Give to parents / guardians 1 Pay for my school expenses 2 Pay for school expenses of others 3 Buy food for myself 4 Buy food for others 5 Buy other things for myself 6 Save it 7 Other: __________________________ 66 SECTION 5: FOOD AND ALCOHOL CONSUMPTION Next I would like to ask you about what you eat and drink. No. Question Coding Category SKIP 501 In the past four weeks, did you have to eat a smaller meal than you felt you needed because there was not enough food? Yes No 1 2 If No: 503 22 502 If yes – How many times did this happen? Read out responses. Rarely (1-2 times in past 4 weeks) Sometimes (3-10 times in past 4 weeks) Often (more than 10 times in past 4 weeks) 1 2 3 503 In the past four weeks, did you have to skip a meal because there was not enough food? Yes No 1 2 If No: 505 504 If yes – How many times did this happen? Read out responses. Rarely (1-2 times in past 4 weeks) Sometimes (3-10 times in past 4 weeks) Often (more than 10 times in past 4 weeks) 1 2 3 505 In the past four weeks did you go to sleep at night hungry because there was not enough food to eat? Yes No 1 2 If No: 507 506 If yes – How many times did this happen? Read out responses. Rarely (1-2 times in past 4 weeks) Sometimes (3-10 times in past 4 weeks) Often (more than 10 times in past 4 weeks) 1 2 3 507* In the past four weeks did you go a whole day and night without eating anything because there was not enough food to eat? Yes No 1 2 If No: 509 508 If yes – How many times did this happen? Read out responses. Rarely (1-2 times in past 4 weeks) Sometimes (3-10 times in past 4 weeks) Often (more than 10 times in past 4 weeks) 1 2 3 509 Have you ever consumed a drink containing alcohol including beer, spirits – that is a whole glass or drink, not just a taste? Yes No 1 2 If No: 601 510 When was the last time you consumed a drink containing alcohol? Read out responses. Yesterday / a few days ago About a week ago More than a week ago 1 2 3 511 How often does it happen that you consume a drink containing alcohol? Read out responses. Only once in a while At least once a week 1 2 SECTION 6: HEALTH, SUPPORT AND PROTECTION Now I have a few questions about your health and wellbeing. No. Question Coding Category SKIP 601* Do you have a birth certificate? Yes No Don’t know 1 2 88 If No: 603 If DK: 603 602* Could you please show me your birth certificate? Seen / confirmed Not seen / not confirmed 1 2 603* At any point in the last 2 weeks, have you been too sick to participate in daily activities? Yes No 1 2 23 604 Do you have a disability that makes it difficult for you to participate in daily activities? Yes No 1 2 If No: 606 605 How would you describe your disability? Blind or partially blind Deaf or partially deaf I have difficulties learning Physical Other _________________________ 1 2 3 4 66 606* I’m going to ask you a few questions about people in your life. Please respond yes or no. Do you have someone in your life to turn to for suggestions about how to deal with a personal problem? Yes No 1 2 607* Do you have someone in your life to help with daily chores if you were sick? Yes No 1 2 608* Do you have someone in your life that shows you love and affection? Yes No 1 2 609* Do you have someone in your life to do something enjoyable with? Yes No 1 2 SECTION 7: HIV/AIDS KNOWLEDGE, ATTITUDES & SEXUAL BEHAVIOR Section may be restricted to ages 13-17 only We are nearly done. I have a few short questions on a disease called HIV/AIDS. No. Question Coding Categories SKIP 701 Have you ever heard of an illness called AIDS? Yes No 1 2 If No: 801 702 Can people reduce their chances of getting the AIDS virus by having just one uninfected sex partner who has no other sex partners? Yes No Don’t know / Not sure 1 2 88 703 Can people reduce their chance of getting the AIDS virus by using a condom every time they have sex? Yes No Don’t know / Not sure 1 2 88 704 Is it possible for a healthy-looking person to have the AIDS virus? Yes No Don’t know / Not sure 1 2 88 705 Can people get the AIDS virus from mosquito bites? Yes No Don’t know / Not sure 1 2 88 706 Can people get the AIDS virus by sharing food with someone who has AIDS? Yes No Don’t know / Not sure 1 2 88 707 Can the virus that causes AIDS be transmitted from a mother to her baby: a) During pregnancy? b) During delivery? Yes No DK a) During pregnancy 1 2 8 b) During delivery 1 2 8 24 c) By breastfeeding? c) By breastfeeding 1 2 8 708 I have a few more questions about HIV. If you don’t want to answer, that is all right. I don’t want to know the results, but have you ever been tested to see if you have the AIDS virus? Yes No Don’t know 1 2 88 If No: 710 If DK: 710 709 I don’t want to know the results but did you get the results of your test? Yes No 1 2 710 Do you know of a place where people can go to get tested for the AIDS virus? Yes No 1 2 SECTION 8: ACCESS TO HIV PREVENTION, CARE & SUPPORT We have arrived at the last section of the questionnaire. We are almost finished. Thank you very much for your participation so far. Instructions: Respondents should respond only for services that they personally have received. The caregiver or head of household will also be asked. Data may be cross-checked. OR, this question may be posed to either the adult or the child (instead of both). 801 I am going to read out a list of items and services. Please tell me if YOU have received or accessed any of these items of services in the last 6 months. Read out services. Confirm responses with caregiver. Circle final responses. [ADD / DELETE ITEMS AS RELEVANT TO PURPOSE] a) Health care from a health professional Yes No b) Home visit from a community worker or social worker Yes No c) Free school supplies or a school uniform Yes No d) Mosquito net Yes No Ages 13-17 e) Information on how to prevent HIV and other sexually transmitted infections Yes No f) Information on birth spacing Yes No g) Livelihood training Yes No Ages 15-17 h) Life skills training Yes No --- END OF SECTION --- Thank you! I have come to the end of my questions. Is there anything you would like to add or ask us? I very much appreciate your time today. If you have any further questions about the survey, please use the contact information on your consent form I am leaving with you. Thank you for participating in this interview! 013 END TIME [__|__|:[____] xxv Child Questionnaire aged 0-9 years (for Caregiver) SECTION 1: CHILD HEALTH & PROTECTION I am now going to ask you a few questions about [insert child’s name]. No. Question Coding Category SKIP 101 Record / Confirm Child’s Name 102 Record Child’s Line Letter from Household Schedule (Caregiver Questionnaire) 103* Record / Confirm Child’s Sex Female Male 1 2 104 In what month and year was [NAME] born? Month [____] Year [__|__|____ ] 105* Remind me, how old was [NAME] at their last birthday? Confirm with 104 and adjust if necessary. Do not leave blank. If unknown, ask caregiver to estimate. [____] years 106 Would you say that in general [NAME’s] health is……? Read out responses. Excellent 1 Very good 2 Good 3 Fair 4 Poor 5 107* In the last 2 weeks, has [NAME] been too sick to participate in daily activities? Yes No 1 2 108 Does [NAME] have a disability that makes it difficult for him/her to participate in daily activities? Yes No 1 2 If No: 110 109 How would you describe [NAME’s] disability? Blind or partially blind Deaf or partially deaf I have difficulties learning Physical Other________________ _____ 1 2 3 4 66 110* Does [NAME] have a birth certificate? Yes No Don’t know 1 2 88 If No: 112 If DK: 112 xxvi 111* Could you please show me [NAME’s] birth certificate? Seen / confirmed Not seen / not confirmed 1 2 No. Question Coding Category SKIP 112 FILTER. Age of child 5 years or older 0-4 years 1 2 If 5+ years: 128 113* Do you have a card where [NAME’s] vaccinations are written down? If yes, ask for card. Yes, seen Yes, not seen No Don’t know 1 2 3 88 If No: 113 If DK: 113 114* Check name on card to make sure card relates to child in question. Document the vaccinations recorded on the card. Only include documented vaccinations here. Yes, documented No a) BCG 1 2 b) OPV 0 1 2 c) OPV 1 1 2 d) OPV 2 1 2 e) OPV 3 1 2 f) DPT 1 1 2 g) DPT 2 1 2 h) DPT 3 1 2 i) Measles 1 2 If caregiver cannot produce a vaccination card for child, probe for vaccinations below. If you have documented the vaccinations from a card, but there are gaps in the vaccination record, probe with questions below. 115* Has [NAME] received a vaccine against tuberculosis, that is, an injection in the arm or shoulder, that usually causes a scar? (BCG) Yes No Don’t know 1 2 88 116* Has [NAME] received the polio vaccine, that is, drops in the mouth? Yes No Don’t know 1 2 88 If No: 121 If DK: 121 117* Has the child received OPV0, that is the first polio vaccine normally received in the first two weeks after birth? Yes No Don’t know 1 2 88 118* Has the child received OPV1, that is the second polio vaccine? Yes No Don’t know 1 2 88 119* Has the child received OPV2, that is the third polio vaccine? Yes No Don’t know 1 2 88 xxvii 120* Has the child received OPV3, that is the fourth polio vaccine? Yes No Don’t know 1 2 88 No. Question Coding Category SKIP 121* Has the child received the DPT vaccination, that is, an injection given in the thigh or buttocks, sometimes at the same time as polio drops? Yes No Don’t know 1 2 88 If No: 123 If DK: 123 122* How many times was the DPT vaccine received? Once Twice Three times Don’t know 1 2 3 88 123* Has the child received a measles injection, that is, a shot in the arm at the age of 9 months or older – to prevent him or her from getting measles? Yes No Don’t know 1 2 88 124* Has [NAME] had diarrhea in the last 2 weeks? Yes No 1 2 125* Has (NAME) been ill with a fever at any time in the last 2 weeks? Yes No 1 2 126 Sometimes adults taking care of children have to leave the house to go shopping, wash clothes, or for some other reasons, and have to leave young children. On how many days in the past week was [NAME] left alone for more than one hour? [____] days 127 On how many days in the past week was [NAME] left in the care of another child (that is, someone less than 10 years old) for more than an hour? [____] days 128 Did [NAME] sleep under a mosquito net last night? Yes No 1 2 129 I don’t want to know the results, but has [NAME] ever been tested to see if he/she has the AIDS virus? Yes No 1 2 If No: 201 130 I don’t want to know the results, but do you know the result of [NAME’s] test? Yes No 1 2 SECTION 2: CHILD EDUCATION AND WORK No. Question Coding Category SKIP 201 Filter: Age of child (Question 402) 5 years or older 3-4 years 0-2 years 1 2 3 If 3-4 years: 213 xxviii If 0-2 years: 301 I now have some questions for you about [NAME’s] schooling and chores. 202* Is [NAME] currently enrolled in school? Yes No 1 2 If No: 206 203* During the last school week, did [NAME] miss any school days for any reason? Yes No 1 2 If No: 205 204 Why did [NAME] miss school days during the last school week? Do not read responses. Circle one primary response. No money for school fees, materials, transport Child is too sick to attend school School is too far away / no school Child has to work to help family Child needs to care for sick household members Child does not like school Other: _______________________ 1 2 3 4 5 6 66 205* What grade/form/year is [NAME] in now? [____] All: 208 206 Why is [NAME] not enrolled in school? Do not read responses. Circle one primary response. No money for school fees, materials, transport Child is too sick to attend school School is too far away / no school Child has to work to help family Child needs to care for sick household members Child does not like school Child is too young to attend school Other: _______________________________ 1 2 3 4 5 6 7 66 207 Has [NAME] ever attended school? Yes No 1 2 If No: 211 208* Was [NAME] enrolled in school during the previous school year? Yes No 1 2 If No: 210 209* What grade/form/year was [NAME] in during the previous school year? [____] All: 211 210 What is the highest grade/form/year that [NAME] has completed? [____] xxix 211 In the past 6 months, has [NAME] worked for money or kind? Yes No 1 2 If No: 301 No. Question Coding Category SKIP 212 What did [NAME] do to earn these wages? Probe: Anything else? Multiple responses possible. Circle all mentioned. House chores, child care for other family 1 All: 301 Selling/Hawking goods 2 Labor, e.g., farm, construction 3 Other: ____________________ 66 213 Does [NAME] attend any organized or early childhood education program, such as a private or government facility, including kindergarten or community child care? Yes No 1 2 If Yes: 301 214 In the past 3 days, did you or any household member over 15 years of age engage in any of the following activities with [NAME]: Read out a through f one at a time. Yes No a) Read books to or looked a picture books with [NAME]? 1 2 b) Told stories to [NAME]? 1 2 c) Sang songs to [NAME] or with [NAME] including lullabies? 1 2 d) Took [NAME] outside of the home, compound, yard or enclosure? 1 2 e) Played with [NAME]? 1 2 f) Named, counted, or drew things with [NAME]? 1 2 SECTION 3: FOOD CONSUMPTION No. Question Coding Category SKIP 301 Filter: Age of child (Question 402) 2 years or older 0-1 years 1 2 If 0-1 years:401 Next I would like to ask you about what [Name} eats and drinks. 302 In the past four weeks, did [NAME] have to eat a smaller meal than you felt was needed because there was not enough food? Yes No 1 2 If No: 304 xxx 303 If yes – How many times did this happen? Read out responses. Rarely (1-2 times in past 4 weeks) Sometimes (3-10 times in past 4 weeks) Often (more than 10 times in past 4 weeks) 1 2 3 304 In the past four weeks, did [NAME] have to skip a meal because there was not enough food? Yes No 1 2 If No: 306 305 If yes – How many times did this happen? Read out responses. Rarely (1-2 times in past 4 weeks) Sometimes (3-10 times in past 4 weeks) Often (more than 10 times in past 4 weeks) 1 2 3 306 In the past four weeks did [NAME] go to sleep at night hungry because there was not enough food to eat? Yes No 1 2 If No: 308 307 If yes – How many times did this happen? Read out responses. Rarely (1-2 times in past 4 weeks) Sometimes (3-10 times in past 4 weeks) Often (more than 10 times in past 4 weeks) 1 2 3 308* In the past four weeks did [NAME] go a whole day and night without eating anything because there was not enough food to eat? Yes No 1 2 If No: 401 309 If yes – How many times did this happen? Read out responses. Rarely (1-2 times in past 4 weeks) Sometimes (3-10 times in past 4 weeks) Often (more than 10 times in past 4 weeks) 1 2 3 SECTION 4: ACCESS TO HIV PREVENTION, CARE AND SUPPORT No. Question Coding Category 401 I am going to read out a list of items and services. Please tell me if [child’s name] has received or accessed any of these items or services in the last 6 months. Read out services. Yes No i) (Psychosocial) counseling for a home visitor or social worker 1 2 j) Health care from a health professional 1 2 k) School fees paid for by organization 1 2 l) Free school supplies or a school uniform 1 2 m) Vitamin A supplement from an 1 2 xxxi organization n) Supplemental, emergency feeding 1 2 --- END OF SECTION --- Thank you! I have come to the end of my questions. Is there anything you would like to add or ask us? I very much appreciate your time today. If you have any further questions about the survey, please use the contact information on your consent form I am leaving with you. Thank you for participating in this interview! END TIME [__|__|:[____] xxxii Annex 3: HVAT Scoring Method CPA 1: Economic Strengthening 1.1: Who is the main contributor to household income? Scores:  Child = 4  Grandparent(s)/Others = 3  Relatives = 2  Parents = 0 1.2: What is the main source of HH income? Scores:  None = 0  Remittances/Other = 3  Causal Labor = 3  Informal Job = 2  Formal Job/Business/Self=Employed = 0 Note: No one reported “none” on the baseline 1.3: What is the current monthly HH income? (express in Uganda Shillings) Scores:  Less Than 50,000 UGX = 4  50,000=99,999 UGX = 3  100,000 = 149,999 UGX = 2  150,000 = 200,000 UGX = 1  Above 200,000 = 0 1.4: Do these statements apply to this HH? (Yes/No) 1) Any member of the HH owns an electronic gadget (radio, phone, TV) 2) Any member of the HH has a functional transport means (bicycle, motor cycle, boat) 3) At least one member of the HH has vocational/apprenticeship/professional skills 4) At least one member of the HH has formal employment, is self-employed or has a business 5) At least one member of the HH belongs to any financial savings and lending group 6) Household has domestic animals (cow(s), goat(s), Sheep, chicken, pig(s)) 7) HH has access to land for agriculture/hire Scores:  If 4 or more are NO = 4  If Three are NO = 3  If Two are NO = 2  If One is NO = 1  If more than 4 are Yes Score = 0 Note: This is an index of yes/no answers to 7 questions. One of the question is redundant, i.e. Does at least one member of the HH has formal employment, is self-employed or has a business? This was asked above as well and also inserted into this index here. xxxiii 1.5: If the HH incurred any of the following expenses in the past three months, was it able to pay without difficulty? 1) Health related expenses (Yes/No) 2) Education (school) related expenses (Yes/No) 3) Food related expenses (Yes/No) Scores:  If All are NO = 4  If Two are NO = 2  If One is NO = 1  If All are Yes = 0 Total Potential CPA Score: 20 CPA 2: Food Security and Nutrition 2.1: Over the past month (mention month), what has been the MAIN source of food consumed by your HH? Scores:  Donated = 4  Given for Work = 3  Bought = 2  Home grown = 0 2.2: What does the family usually eat (at least 3 times a week), Yes/No 1. Energy Food 2. Body Building Foods 3. Protective and Regulative Foods Scores:  None = 4  One food group = 3  Two food groups = 1  All food groups = 0 2.3: How many meals does the HH have in a day? Scores:  Some Days No Meal = 4  One Meal = 3  Two Meals per Day = 1  Three or more Meals per Day = 0 Total Potential CPA Score: 12 CPA 3: Health, WASH, and Shelter 3.1: Do the following apply to this HH? Indicate Yes/No/NA 1) Has access to safe water within 30 minutes (half an hour) or harvests rain water for domestic use 2) Has a clean compound 3) Has access to a public health facility within 5 kilometers 4) Has a drying rack for HH utensils xxxiv 5) Has a garbage pit /dust bin 6) Separate house for animals 7) Hand washing facility 8) All HH members sleep under a mosquito net Scores:  If 4 or more NOs = 4  If Three are NOs = 3  If Two are NOs = 2  If One is NO = 1  If four or more are YES = 0 3.2: Does the caregiver know the HIV status for all children in the household? Scores:  No = 4  Less than half of children = 3  Half or More of children = 2  Yes = 0 3.3: Are all eligible children who are HIV+ and/ or have TB on treatment? Scores:  None of the Children = 4  Less than half of children = 3  Half or More of children = 2  Yes = 0 3.4: Does the HH have a stable shelter that is adequate, safe and dry? Scores:  No stable shelter , adequate or safe place to live = 4  Shelter is not adequate, needs major repairs = 3  Shelter needs some repairs but is fairly adequate, safe, and dry = 1  Shelter is safe, adequate & dry = 0 3.5: What is the type of a latrine/toilet facility used by members of your HH? Scores:  Bush/ None = 4  Public toilet for pay = 3  Private needs some repair/ risky state = 2  Private but shared by more than one HH = 1  Safe, adequate & dry = 0 Total Potential CPA Score: 20 CPA 4: Education 4.1: How many children aged 5-17 years in this HH are not going to school or miss school 3 or more times a week? Scores: xxxv  All (or more than three=fourths) do NOT attend/DO miss 3 or more times a week: 4  50% or more do NOT attend and/or DO miss 3 or more times a week: 3  Less than half do attend and/or do NOT miss 3 or more times a week: 2  All Eligible Children Attend/don’t miss: 0 Total Potential CPA Score: 4 CPA 5: Psychosocial Support and Basic Care 5.1: In the past 12 months, how often has someone in your HH felt so troubled that it was necessary to consult a spiritual, faith or traditional healer, counselor or health worker? Scores:  5+ times = 4  3-4 times = 3  Twice = 2  Once = 1  Not at all = 0 5.2: Are there any children in this HH who are withdrawn or consistently sad, unhappy or depressed, not able to participate in daily activities including playing with friends and family? Yes/ No Scores:  All children =4  50% (more than half) or more = 3  Less than 50% (less than half) = 2  None = 0 Note: We asked about the number of children who fit these criteria per household. To compute the scores, we had to take this number into the total number of children in that household. There were 41 cases in which the HH misreported and we had more children missing school than living in the household. In such cases, we changed the number of children missing to match that of the number in the household. Total Potential CPA Score: 8 CPA: Child Protection 6.1: In the past month, have you or another adult in the household used the following method of discipline with any child in your household? (Please select all the methods that apply) 1. Punched, kicked or hit as punishment 2. Withheld a meal as punishment 3. Yelling and screaming Scores:  All three = 4  Two = 3  One = 2  None = 0 xxxvi 6.2: In the past 12 months, has any child in the HH had the following happen to them, in or outside of the HH? 1) Repeated physical abuse (Yes/No) 2) Withheld a meal to punish (Yes/No) 3) Involved in Child Labour (Yes/No) 4) Family separation (ran away, chased)/Neglected (Yes/No) 5) Sexually abused, defiled, raped, forced sex (Yes/No) 6) Stigmatised/ discriminated due to illness, disability or otherwise (Yes/No) 7) Using abusive words/language (Yes/No) 8) In contact/conflict with the law (Yes/No) Scores:  If four or more are YES = 4  If THREE are YES = 3  If TWO are YES = 2  If ONE is YES = 1  If All are NO = 0 Total Potential CPA Score: 8 Total Potential Combined CPA Scores: 64 Computation: Add the total actual scores across all items under each CPA and divide the number by 64. Vulnerability scores are then identified according to the following categories: Not Vulnerable = 0-24% Slightly Vulnerable = 25-49% Moderately Vulnerable = 50-74% Critically Vulnerable = 75-100 Annex 4: List of Sampled Parishes List of Sampled Parishes Region District Subcounty Parish CENTRAL REGION GOMBA KABULASOKE KISOZI CENTRAL REGION GOMBA KABULASOKE MAWUKI CENTRAL REGION GOMBA KANONI TOWN COUNCIL KANONI WARD CENTRAL REGION GOMBA KYEGONZA KOOME CENTRAL REGION GOMBA KYEGONZA SAALI CENTRAL REGION GOMBA MADDU NTALAGI CENTRAL REGION GOMBA MPENJA MPOGO CENTRAL REGION HOIMA BUGAMBE BUGAMBE CENTRAL REGION HOIMA BUHIMBA KYABATALYA CENTRAL REGION HOIMA BUSERUKA TOONYA CENTRAL REGION HOIMA BUSIISI DIVISION KIHUKYA WARD CENTRAL REGION HOIMA KAHOORA DIVISION CENTRAL WARD CENTRAL REGION HOIMA KIGOROBYA KAPAAPI CENTRAL REGION HOIMA KITOBA BULYANGO CENTRAL REGION HOIMA KIZIRANFUMBI BULIMYA CENTRAL REGION LUWEERO BAMUNANIKA KIBANYI CENTRAL REGION LUWEERO BAMUNANIKA SEKAMULI CENTRAL REGION LUWEERO BUTUNTUMULA BUKAMBAGGA CENTRAL REGION LUWEERO KALAGALA BUSOKE CENTRAL REGION LUWEERO KAMIRA MABUYE CENTRAL REGION LUWEERO KATIKAMU BUSULA-MUSAALE CENTRAL REGION LUWEERO KIKYUSA WABUSANA CENTRAL REGION LUWEERO MAKULUBITA KALASA CENTRAL REGION LUWEERO MAKULUBITA WALULEETA CENTRAL REGION LUWEERO NYIMBWA SSAMBWE CENTRAL REGION LUWEERO ZIROBWE NAKIGONZA CENTRAL REGION MITYANA BULERA KALAMA/KYAMUSISI CENTRAL REGION MITYANA BUSIMBI TTANDA CENTRAL REGION MITYANA BUTAYUNJA KITEBERE CENTRAL REGION MITYANA KAKINDU MWANDA CENTRAL REGION MITYANA MAANYI KASOTA CENTRAL REGION MITYANA MALANGALA MAGONGA CENTRAL REGION MITYANA MITYANA TOWN COUNCIL CENTRAL WARD CENTRAL REGION MITYANA MITYANA TOWN COUNCIL NORTH WARD CENTRAL REGION MITYANA SSEKANYONYI BULYANKUYEGE CENTRAL REGION MITYANA SSEKANYONYI NAMUNGO CENTRAL REGION RAKAI BYAKABANDA KITAASA CENTRAL REGION RAKAI KABIRA BWAMIJJA CENTRAL REGION RAKAI KALISIZO RURAL MATALE CENTRAL REGION RAKAI KASAALI KYAKONDA CENTRAL REGION RAKAI KIBANDA KYABIWA CENTRAL REGION RAKAI KYALULANGIRA KALUNGI CENTRAL REGION RAKAI LWAMAGGWA BUGONA CENTRAL REGION RAKAI NABIGASA KIJEJJA SOUTH WESTERN REGION BUSHENYI KYAMUHUNGA KIBAZI SOUTH WESTERN REGION BUSHENYI KYAMUHUNGA NSHUMI SOUTH WESTERN REGION BUSHENYI KYEIZOBA KITWE SOUTH WESTERN REGION BUSHENYI KYEIZOBA RUTOOMA SOUTH WESTERN REGION ISINGIRO ENDINZI KIKOBA SOUTH WESTERN REGION ISINGIRO NGARAMA BURUNGAMO SOUTH WESTERN REGION KABALE KAMUGANGUZI BURANGA SOUTH WESTERN REGION KABALE KAMUGANGUZI KASHEREGENYI SOUTH WESTERN REGION KABALE KAMUGANGUZI KICUMBI SOUTH WESTERN REGION KABALE KAMWEZI KASHEKYE SOUTH WESTERN REGION KABALE KAMWEZI KIGARA SOUTH WESTERN REGION KABALE KAMWEZI KYABUHANGWA SOUTH WESTERN REGION KABALE KASHAMBYA NYAKASHEBEYA SOUTH WESTERN REGION KANUNGU KAYONZA BUJENGWE SOUTH WESTERN REGION KANUNGU KAYONZA KARANGARA SOUTH WESTERN REGION KANUNGU KAYONZA MUKONO SOUTH WESTERN REGION KANUNGU KAYONZA RUTENDERE SOUTH WESTERN REGION KANUNGU NYAMIRAMA NYAKASHURE SOUTH WESTERN REGION KIRUHURA KAZO KAYANGA SOUTH WESTERN REGION KIRUHURA KENSHUNGA RUGONGI SOUTH WESTERN REGION NTUGAMO IHUNGA BUTANDA SOUTH WESTERN REGION NTUGAMO IHUNGA KAGAMBA SOUTH WESTERN REGION NTUGAMO IHUNGA KAGARAMA TOWN BOARD SOUTH WESTERN REGION NTUGAMO KIBATSI IBAARE SOUTH WESTERN REGION NTUGAMO KIBATSI KIBARUKO SOUTH WESTERN REGION RUKUNGIRI BUHUNGA BUHUNGA SOUTH WESTERN REGION RUKUNGIRI KEBISONI NYEIBINGO SOUTH WESTERN REGION RUKUNGIRI NYARUSHANJE IBANDA WESTERN REGION KAMWENGE BWIZI NTONWA WESTERN REGION KAMWENGE KICHECHE KANTOZI WESTERN REGION KAMWENGE NKOMA BISOZI WESTERN REGION KASESE KISINGA KAGANDO WESTERN REGION KASESE MUHOKYA KAHENDERO WESTERN REGION KASESE NYAMWAMBA DIVISION KISANGA WESTERN REGION KIBAALE BWIKARA MAIRIRWE WESTERN REGION KIBAALE BWIKARA NYAKARONGO WESTERN REGION KIBAALE BWIKARA NYAMAASA WESTERN REGION KIBAALE KAKUMIRO TOWN COUNCIL KABWOORO WESTERN REGION KIBAALE MPEEFU NYANTONZI