Photo: A social worker in front of a wall with pictures of children reintegrated from the child care institution. By Anna Nordenmark Severinsson, Maestral International. August 2018 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Maestral International in collaboration with Oxford Policy Management and Makerere University. End-line performance evaluation: Deinstitutionalization of Orphans and Vulnerable Children in Uganda (DOVCU) 2 ACKNOWLEDGEMENTS This baseline evaluation report was commissioned by John Snow International (JSI) and United States Agency for International Development (USAID) Displaced Children and Orphans Fund (DCOF). From these two organizations, Olga Cojocari (JSI), John Williamson and Martin Hayes (USAID/DCOF) oversaw the work of the consultancy assignment that resulted in this evaluation report. The evaluation consultancy was implemented in a three-partite partnership between Maestral International, US (lead agency), Oxford Policy Management (OPM), UK and Makerere University, Uganda. The evaluation and the evaluation team was led by Anna Nordenmark Severinsson, Senior Associate at Maestral International and International Advisor on Welfare and Protection System Reform. The different components of the qualitative methods that fed into this report led by Sope Otulana and Johanna Wallin (OPM). Johanna Wallin implemented the training for field researchers and oversaw the qualitative research for the evaluation; Michele Binci (OPM) provided technical assistance to the implementing partner ChildFund on the quantitative aspects of the evaluation methodology; Eddy Walakira and Stella Neema (Makerere), recruited and led the local research teams during the field phase of the evaluation; and Anna Nordenmark Severinsson and Beth Bradford (Maestral International), implemented the policy component of the evaluation methodology. Anna Nordenmark Severinsson authored the evaluation report with inputs from the rest of the evaluation team. The evaluation team would like to thank in particular Otai Moses, Asekenye Monica, Walter Okello and Fred Mutenyo from ChildFund Uganda for their collaboration and partnership throughout the evaluation. The team would also like to thank all the partners of the DOVCU project, namely ChildFund, TPO, Child’s I Foundation and Retrak and their staff for their support in the field, and for remaining available to share information with the evaluation team throughout the evaluation process. The evaluation team is also very appreciative of the local researchers (16 of them) from Makerere University, who are not named in person here, for their outstanding work in the field during the weeks of September 2015 when the data collection was undertaken. The team is also grateful to Chris Rayment (OPM), to Philip Goldman, Isabel de Bruin Cardoso and Joanna Ison (Maestral International) for their constant support to the process. Last but not least, the evaluation team is very grateful to all the women, men, and children in communities, child care institutions, remand homes, Child’s Restoration Outreach, and staff of these organizations, in the districts and of the Ministry of Gender, Labor and Social Development who participated in interviews and shared their experiences and views with the evaluation team. The report was prepared between October 2017 and August 2018. 3 ENDLINE PERFORMANCE EVALUATION OF THE PROJECT: DEINSTITUTIONALIZATION OF ORPHANS AND VULNERABLE CHILDREN IN UGANDA (DOVCU PROJECT) August, 6, 2018 DISCLAIMER The views expressed in this report do not necessarily reflect those of JSI or USAID/DCOF. The authors remain solely responsible for any errors that may remain in this report. 4 TABLE OF CONTENTS ACRONYMS ........................................................................................................................... 8 GLOSSARY ............................................................................................................................. 9 EXECUTIVE SUMMARY......................................................................................................... 11 EVALUATION PURPOSE & EVALUATION QUESTIONS ............................................................ 17 Evaluation Purpose................................................................................................................................................................17 Evaluation Questions ............................................................................................................................................................17 Impact ........................................................................................................................................................................................................... 17 Relevance..................................................................................................................................................................................................... 17 Effectiveness .............................................................................................................................................................................................. 18 Efficiency ..................................................................................................................................................................................................... 18 Sustainability ............................................................................................................................................................................................. 18 Human Rights ............................................................................................................................................................................................ 18 PROJECT BACKGROUND....................................................................................................... 20 Country context.......................................................................................................................................................................20 The DOVCU project ................................................................................................................................................................20 Conceptual framework and hypothesis ......................................................................................................................................... 20 Project approaches and strategies ................................................................................................................................................... 21 Project phases ........................................................................................................................................................................................... 23 EVALUATION METHODS & LIMITATIONS.............................................................................. 24 EVALUATION METHODS.......................................................................................................................................................24 LIMITATIONS ...........................................................................................................................................................................25 FINDINGS & CONCLUSIONS.................................................................................................. 28 IMPACT.......................................................................................................................................................................................28 Economic well-being ................................................................................................................................................................................. 28 Psychosocial well-being ............................................................................................................................................................................ 30 Food security and nutrition ................................................................................................................................................................... 32 Education Opportunities ......................................................................................................................................................................... 33 Psychosocial well-being, care and protection .................................................................................................................................. 34 Changes in district structures and services ...................................................................................................................................... 35 Changes in awareness, knowledge and technical skills amongst district, sub-county and community professionals .......................................................................................................................................................................................................................... 38 5 RELEVANCE...............................................................................................................................................................................40 EFFECTIVENESS.......................................................................................................................................................................46 EFFICIENCY ...............................................................................................................................................................................59 SUSTAINABILITY.....................................................................................................................................................................64 HUMAN RIGHTS.......................................................................................................................................................................71 CONCLUSIONS ..................................................................................................................... 78 ANNEXES................................................................................................................................ I ANNEX I: Lessons learned on project’s integrated package of support and strategies for prevention of separation ..................................................................................................................................................................................... i ANNEX II: Lessons learned on the project’s work with child care institutions ................................................xx ANNEX III: Lessons learned on the project’s work with different professionals.........................................xxiv ANNEX IV: Lessons learned on case management and follow up on reunification cases ....................... xxxii ANNEX V: Statistical Tables............................................................................................................................................xxxvi ANNEX VI: Evaluation Statement of Work................................................................................................................. xlvii ANNEX VII: Evaluation Methods and Limitations ..................................................................................................xlviii ANNEX VIII: Data Collection Instruments....................................................................................................................xlix ANNEX IX: Sources of Information ....................................................................................................................................... l 6 FIGURES & TABLES Figure 1: Percentage changes in the proportion of household economic vulnerability classification at baseline and end line. ...................................................................................................................................................................................................29 Figure 2: Proportion of reunified children who were reunified from CCIs, remand homes and from the street......31 Figure 3: Improvements made in CCIs on alternative care standards: baseline vs. endline ..............................................36 Figure 4: Distribution of the reunified children by place of reunification...............................................................................52 Figure 5: Number/proportion of households in destitute, struggling and growing categories of economic vulnerability in phase 1 and phase 2 districts................................................................................................................................. 59 Figure 6: Comparative analysis of wellbeing of children at-risk, reunified children from CCIs, RH and street...........65 Figure 7: Changes in the vulnerability level of children (endline compared to baseline) ...................................................74 Table 1: Shows number/proportion of households that belong to; destitute, struggling and growing level of economic vulnerability; data disaggregated by gender of household headship, all districts and phase 1 districts results........................................................................................................................................................................................................ iii Table 2: Changes in household’s economic vulnerability classification by districts.............................................................. iii Table 3: Number and % of households scoring below 49 points on FSVI CPA1 and 2; disaggregated by sex, age and all districts and phase 1 districts results........................................................................................................................................... iv Table 4: Household monetary income .............................................................................................................................................. v Table 5: Household capital and loans ................................................................................................................................................ v Table 6: Shows household economic livelihood security status (characteristics) at endline compared to baseline characteristics........................................................................................................................................................................................ vii Table 7: Number and % of supported households participating parenting group support sessions; disaggregated by vulnerability type (70-116 scores or less on FSVI key domains) ...............................................................................................ix Table 8: % of households reporting incidences of violence against children in their family in the last 12 months.......x Table 9: Participation in post treatment groups............................................................................................................................xi Table 10: Shows the significances in the mean differences in the vulnerability score of Children based on all CPA (EL - BL) – an indicator of changes in the wellbeing....................................................................................................................xii Table 11: Shows household access to basic needs status (characteristics) at endline compared to baseline characteristics.......................................................................................................................................................................................xiii Table 12: Shows household access to basic needs status (characteristics) at endline compared to baseline characteristics........................................................................................................................................................................................xv Table 13: Shows number/proportion of households that belong to; destitute, struggling and growing level of economic vulnerability; data disaggregated by gender of household headship, all districts and phase 1 districts results................................................................................................................................................................................................. xxxvi Table 14: Changes in household’s economic vulnerability classification by districts..................................................... xxxvi Table 15: % of remand homes showing reduced number of children their care and % changes in the number of children ............................................................................................................................................................................................. xxxvii Table 16: Number of children reunified from CCIs, remand homes and street (disaggregated by age) ................ xxxvii Table 17: % of CCIs showing reduced number of children in residential care and % changes in the number of children ............................................................................................................................................................................................ xxxviii Table 18: Shows the significances in the mean differences in the vulnerability score of Children based on all CPA (EL - BL) – an indicator of changes in the well-being........................................................................................................... xxxviii 7 Table 19: Shows household access to basic needs (baseline and end line values)....................................................... xxxviii Table 20: Shows household access to basic needs: education (baseline and endline values)......................................xxxix Table 21: Number of CCIs assessed at baseline and endline and their endline status........................................................xl Table 22: Shows CCI performance progress on AHR Standards as a result of the rollout of ACF endline compared to baseline...............................................................................................................................................................................................xli Table 23: Number of children reunified within and outside DOVCU-supported districts and sub-counties.............xlii Table 24: Shows the significances in the mean differences in the vulnerability score of Children based on all CPA (EL - BL) – an indicator of changes in the well-being..................................................................................................................xlii Table 25: Number of children reunified with families which were handled by TACs in each district ..........................xlii Table 26: Districts with lowest average score on child care provisions at BL and lowest change in number of children placed in residential care at end line..............................................................................................................................xliv Table 27: Number of reunified children with and without disability .....................................................................................xliv Table 28: Shows number and % of children engaged in exploitative unpaid employment ...............................................xliv Table 29: Shows household level status: ppsychosocial distress, child protection, knowledge on referral pathways ..................................................................................................................................................................................................................xlv Table 30: Shows changes in the wellbeing of children (combining prevention and reunified households) by age groups ....................................................................................................................................................................................................xlvi 8 ACRONYMS ACF Alternative Care Framework (of Uganda) ACP Alternative Care Panel (national) ACU Alternative Care Unit CAO Chief Administrative Officer CCI child care institution CPM child protection mechanisms CRO Child Restoration Outreach CSI Child Status Index CSS Community support structure DOVCC District Orphans and Vulnerable Children Committee DOVCU Deinstitutionalization of Orphans and Vulnerable Children Project in Uganda FBO faith-based organizations FGD focus group discussion FSVI Family Status Vulnerability Index HH Household IGA Income generating activities JSI John Snow International KII key informant interview MIS Management Information System MoGLSD Ministry of Gender, Labor and Social Development NGO Non-governmental organization OIC Officer-in-charge OVC Orphans and Vulnerable Children PSW para-social workers PSWO Probation and Social Welfare Officers SOP standard operating procedure TAC Teams Around the Child TOC theory of change TOT training of trainers UNICEF United Nations Children’s Fund USAID United States Agency for International Development VAC Violence Against Children VSLA Village Savings and Loan Association 9 GLOSSARY In this evaluation, distinction is made between the two different terms reunification and reintegration of children. Reunification refers to the process of physically returning a child to the family of origin for the purpose of establishing or re-establishing long-term care in this family. Reintegration, on the other hand, is a process that ideally is the end-result of a reunification, but is a longer process involving the child and his/her family and community and where a safe, nurturing, life-long family care has been established for a child. External actors can support this process but reintegration is not an automatic result of a reunification, nor is it an automatic result of provision of services to support a reintegration process. The end result of a reintegration process, as referred to in this report, is that a child has been reintegrated. In the case of the DOVCU project, the objective was to reunify approximately 2,000 children living in institutional care or in the streets with birth families or into alternative family-based care and that these children have improved positive outcomes in the twelve months following reintegration. This objective then implies that the project establishes, though case-monitoring, the point in time when a reunified child is to be considered as reintegrated. It furthermore implies that monitoring continues 12 months after a child is considered as reintegrated and that positive outcomes are observed in the wellbeing of the child as a result of reintegration. It should be noted that in DOVCU-documents (training documents, SOPs etc.) the two different terms reunification and reintegration are in use, but they are not defined. Nor are the criteria to be used by the project to determine if a child is to be considered as reintegrated, following reunification, defined or discussed in DOVCU documents. Project monitoring has focused on measuring reintegration through the proxi-indicator that children have stayed in the family where they were reunified between 9 and 12 months following the reunification. This, however, is not considered by this evaluation as a sufficient measure of whether a “safe, nurturing, life-long family care has been established for a child”. Therefore, to be able to state whether reunification has resulted in reintegration, the evaluation has captured stories of reunified children’s subjective wellbeing which have informed the findings of this evaluation report. As a result of this, throughout this report, the most commonly used term is reunification. This term is also used to describe when a child has stayed in the family for some time. The term reintegration is in use specifically when the evaluation has been able to capture through the qualitative data the extent to which reunification has resulted in a child staying in the family where he/she was reunified between 9 and 12 months following the reunification and where the subjective wellbeing of the child clearly indicates that the child feels safe, nurtured and stably living in that family. In addition to these two terms, another term, namely resettlement is sometimes used in popular language in Uganda. This term is also sometimes used in DOVCU documentation. While at international level, resettlement usually refers to a process involving placement of refugees in a third country, in Ugandan context, resettlement is used as equivalent to reunification. It should furthermore be noted that in popular language in Uganda, there is often no difference made between reunification and reintegration. The terms resettlement, reunification and reintegration are often used interchangeably and simply means returning a child to his/her family of origin. In quotes, throughout the report, these words have been captured as used by the respondent. 10 In this report, the term resettlement has sometimes been used in quotes or when specific project documents or policy documents used this term. In this report resettlement means returning a child to his/her family of origin. 11 EXECUTIVE SUMMARY EVALUATION PURPOSE AND EVALUATION QUESTIONS The objective of the evaluation is to assess the performance of the “Deinstitutionalization of Orphans and Vulnerable Children Project in Uganda” (DOVCU) with regards to the creation of sustainable changes in the lives of two beneficiary groups, namely 43,000 vulnerable children living in targeted households and 2,000 children at risk as a result of an integrated package of support. A second objective was to assess how and if these results came about from systems changes and identify which strategies and approaches were the most effective for achieving the change in children’s lives. The evaluation responded to 25 specific evaluation questions regarding the projects impact, relevance, effectiveness, efficiency and sustainability and also assessed the projects results from the point of view of its contribution to human rights. PROJECT BACKGROUND The “Deinstitutionalization of Orphans and Vulnerable Children Project in Uganda” was a 36-month child care reform project supported by John Snow International (JSI) and United States Agency for International Development (USAID) implemented in 12 districts of Uganda. To achieve its two objectives, the project included two components. The first component targeted destitute at-risk families in 2 parishes, in 2 sub-counties in each of the project’s 12 districts with prevention activities. This component included strategies for economic strengthening, building parenting skills, addressing social problems, and mobilized and built capacity of para social workers (PSW) and community child protection mechanisms (CPMs) to be involved in identification and referrals of vulnerable families and children at risk of separation. The second component of the projects worked to reintegrate children from child care institutions (CCIs), remand homes and from the street. The targeting strategies of the project were distinct for each component. Children at risk and to be reintegrated from CCIs, remand homes and from the street were already separated from their families and were targeted at the location where they were living/staying when the project started. The children supported in vulnerable households were identified and targeted by the project based on a targeting strategy that had multiple stages of both qualitative and quantitative methods to identify households vulnerable to separation. These households were selected from the 4 project parishes within the 12 project districts. Children to be reintegrated, on the other hand were reunified with families in their districts of origin, which sometimes meant outside of the 12 project districts and the parishes where the project focused its prevention work. EVALUATION DESIGN, METHODS AND LIMITATIONS The evaluation relied on a mixed-methods approach with a strong qualitative component. The methodology consists of three distinct methodological pillars: policy analysis, qualitative component and quantitative data analysis. While the evaluation team collected the data in 6 sample districts out of the 12 project districts for primary data collection. This included Gulu, Kabarole, Kasese, Kamuli, Luwero and Mbale. The quantitative data analysis presented as part of this evaluation is relying on project monitoring data. The limitations of the evaluation are mainly related to the quantitative data. This data had been collected for project monitoring purposes and as a result, no impact evaluation design or related sampling strategy informed the selection of the sample or the determination of the sample size. As a consequence, the findings of the quantitative analysis may not be representative of the population from which the sample was drawn. Furthermore, due to logistical and financial limitations, project monitoring assessments sampled a smaller number of households at endline. This had implications for the children who were 12 reunified. Of all 1,743 reunified children reported by the project, endline project monitoring data only covers the 698 children within reach, i.e. those reunified within the 12 districts where DOVCU focused its prevention activities. FINDINGS AND CONCLUSIONS Impact: The project had impact in addressing causes to separation at three different levels: immediate, underlying and root causes. Interviews at community level indicate that interventions are perceived to match identified community needs. Findings based on a comparison of baseline and endline data on vulnerable households show a reduction in the targeted households’ economic vulnerability. The number of households classified at destitute level reduced from 56% at baseline 30% by endline. On the other hand, most households seem to have shifted into the struggling category, from 44% at baseline to 69% at endline, while only about 1% of households graduated to the growing category. It seems that the time during which vulnerable households received support, was not enough to make the significant shift to the growing category. Furthermore, in the context of the volatile environment in which the respondents live, a shift from destitute to growing category may not be enough to keep families out of situations that make them vulnerable to separation. There is no aggregate national or district data that can evidence project impact in improving child separation-trends overall. This said, DOVCU supported the reunification of a total of 1,743 children to families. Of these 84% were reunified from different CCI, 4% were reunified from remand homes and 12% were reunified from the street. The districts that were involved in the project showed an average of 26% reduction in the number of children in care. Between baseline and endline, monitoring data indicate that overall, there has been a reduction in child vulnerability scores for children living in targeted households. The most significant changes can be seen in health, shelter and care. This is followed by food and nutrition and education and skills training. Psychosocial support and protection also show an improvement, though slightly lower. Findings indicate that the economic strengthening activities have helped vulnerable households to better regulate consumption and increase food security and nutrition. However, respondents do not consider that these activities have fully brought them out of food insecurity for the longer term. For children a number of changes were documented: • Majority of children in the qualitative case studies of at risk households perceive that they now eat better now than two years ago • For children who have been reunified, there is a significant variance in children’s responses around subjective well-being and as a result, the extent to which they are to be considered as having been reintegrated. Children who have been reunified from the street or remand homes, consider their situation with regards to food security and nutrition to have improved. Children who have been reunified from CCIs perceive their access to services and food to be more precarious following reunification, and this adds to stress and reduces psychosocial well-being. This should be seen in the context that some CCIs are very well-resourced, as compared to the families where children have been returned. • There has been an improvement in household’s ability to send children to school and in terms of psychosocial well-being, care and protection, respondents including for children, perceive the family situation to have improved. • The majority of children at risk and children reunified, all cited violence against children to have been something they experienced, but that this was not something they worried about now. 13 The DOVCU project contributed to strengthening the knowledgebase of MoGLSD on the number of CCIs and the achievements of standards in CCIs in the 12 project districts. At district level, the project piloted a new gatekeeping structure – Teams Around the Child (TAC) – in 11 districts. Across districts there is evidence indicating that the DOVCU project inspired many changes in the availability of services provided by CCIs and built their capacity to strengthen the way they operate today as compared to the situation at baseline. For example, the DOVCU project has contributed to the fact that: • There are fewer CCIs in the DOVCU-districts at endline, as compared to baseline. • On average improvements have been made in all operational quality standards for CCIs covered by national standards. • Between the baseline and endline evaluation, DOVCU has contributed to significant changes in the awareness and attitudes of CCIs around the national alternative care framework and their role in the alternative care system. This mindset change has mitigated the risks of resistance to the change amongst the child care institutions that was observed at baseline. In spite of these improvements that can be credited as contributions of DOVCU project to the implementation of the national alternative care framework, there are still more improvements needed in the way CCIs operate as service providers in the alternative care system. The changes made in the practices of the CCI varied significantly from one CCI to the other. Some CCIs engaged more intensely with the project and made improvements in several operational quality standards for CCIs, while other CCIs only made some improvements in one or a few areas of these standards. As a result of DOVCU, awareness raising, skills building and training activities, the evaluation documented a number of changes in awareness, technical skills and professional practices across districts. The project has contributed to a greater knowledge amongst professionals at all levels, of risks, response options and of the roles of various professionals in the response, and also produced important changes in professional practices. District staff are now more regularly and systematically involved in providing oversight and applying stricter gatekeeping procedures for children to CCIs. One area where the project has had mixed results is in the area of case management. Relevance of the Project The project’s integrated model, which linked family strengthening, child protection and economic strengthening measures, was relevant in Ugandan context. The combination of the tools used, FSVI and CSI indexes, and the process designed to identify households for the project’s prevention component, as well as the step-by-step sequencing of different types of support to destitute households were as perceived as relevant and useful to consider in a scale-up of the project. Furthermore, the integrated approach is assessed as relevant to address different categories of risk that are present in the communities where the project operated and it was perceived by respondents to address the primary risk areas for child separation. For the households that were targeted by the project for the prevention component, findings suggest that the assistance provided by the project through the integrated package of services was appropriate. However, amounts of assistance , e.g. the size and frequency of cash transfer, length of trainings etc., were seen by many households as being insufficient to meet their needs. 1 For the children returning to family care and families receiving children back from CCIs and remand homes, stakeholders provided more mixed reviews regarding the relevance of the support provided. The project was overall assessed as very relevant, both in terms of its alignment with policy environment, community values in which it operated, and in terms of its timing. It relevantly addressed 1 Size depended on the households needs and family size, frequency depended on the progress overtime, length of training was standard by design 14 attitudes and resistances that prevailed amongst some CCIs at the inception of the project and it successfully leveraged existing formal system stakeholders to get further involved in alternative care. The project’s interventions within remand homes were felt as less relevant to address all the issues that are leading children to being placed in remand. Effectiveness: It is not possible to conclude that the project achieved its first objective, namely to support 43,000 children in vulnerable households2, however the 2,234 households that were directly supported through the project’s prevention component families remained intact and the interventions received by these families are perceived as having matched expressed needs. This said, evidence suggests that DOVCU support to prevent separation helped during the lifetime of the project to prevent separation but that this effect might not yet have resulted in sufficient and sustainable improvements in households to make prevention last. The second objective: to improve wellbeing of children through supporting the reunification of approx. 2,000 children was only partially achieved. The project reunified 1,743 children. Of these most children are believed to be living in families but for 1,045 children who were reunified outside of the 12 DOVCU districts documentation on their status could not be supplied to this evaluation given that the project did not include them for monitoring data collection at endline. Furthermore, among the children who were reported as reunified within the 12 DOVCU districts some children have stayed with the same service provider (CCI) that transformed into a boarding school, or the child moved to boarding school facility provided by another service provider. Furthermore, the wellbeing and quality in the reintegration process of the reunified children seems to vary. There are some specific factors which seem to have influenced the reintegration process and children’s well-being. The quality in the reintegration process depended on where the child was reunified from: remand homes, CRO, or which CCI the child came from. Furthermore, the ability of CCIs to support a reintegration processes depended on a number of factors. TACs got involved in quality assuring the reunification when many children that the project counts as reunified had already been returned home and this seems to have influenced the quality of the process. A total of 20% of children were reunified through a process that was quality assured by TACs. There was difference in the “readiness” of communities, and in the support provided by the project to the communities and families where children were reunified and this seems to have influenced reintegration. This said, the majority of caregivers and children indicate they are happy about being reunified, as long as DOVCU and/or the CCI continue to support them. Efficiency There is evidence suggesting that the overall impact of the project’s strategies to reduce vulnerability was affected by the late start of the project, and especially so in phase 2 districts. If there is one specific component of support that was often singled out as quickly having produced an important effect and a multiplier effect in the lives of children and vulnerable families, it is the cash transfer. Another strategy that was perceived as efficient and effective in the communities was the various community groups that were created. The strategies employed by the project to change mindsets, priorities and professional practices around alternative care, were by and large efficient. Both for CCIs and remand homes, an area where more support would have been needed is in strengthening case management to ensure equal quality of the follow-up and documentation of monitoring visits. 2 At the stage of project design, the figure of 43,000 children was defined based on the assumption that there is on average of 5 children per targeted household. 15 Sustainability The impact in the lives of beneficiaries of the project seems vulnerable over the long run. Respondents continuously highlight the precarious situation that they find themselves in and express a dependence on the continuous support by DOVCU in order to avoid separation or re-separation. Findings indicate that the psychosocial and community-based elements of the project are more likely to sustain efforts supporting the reintegration of children after the project has ended Teams Around the Child (TACs) are assessed as having good prospects to become sustainable, but under the name of district Alternative Care Panels. The sustainability of case management and support to reunified children and children at risk is more questionable. In a few districts, the district leadership express commitment to mobilize sustained funding for priorities related to alternative care reform. Human Rights The prevention component of the project managed to target some of the most vulnerable families but the children who were reunified from CCIs into families were not necessarily coming from the CCIs with the poorest quality of care. The project seems to have contributed to preventing child protection issues and human rights violations in communities for a broader number of children than those that were particularly targeted by the project’s prevention and reintegration components. The project furthermore mitigated risks for creating greater vulnerabilities for children who were already living within the targeted households. For children who were reunified from CCIs, it is unclear if the project has prevented these children from becoming more vulnerable as a result of the DOVCU reunification activities and support towards reintegration. Across all districts, the project has contributed significantly to strengthen duty bearers’ knowledge about their obligations and roles in alternative care. Stakeholders across all respondent groups seem to have understood the message that family care is in the best interests of the child. However, stakeholders have not understood that child rights are absolute and that there is no hierarchy of rights whereby one right is more important than another. In the context of the DOVCU project, stakeholders seem to have understood that family care is more important than any other right . Furthermore, the project did not put in place a clear system of accountability for the case management follow-up of children who were reunified. Conclusions The DOVCU project was a complex and ambitious project. The evaluation concludes that the project to a large extent has demonstrated an effective and relevant package, methods and strategy to prevent separation of children. The project faced some challenges in the design when it comes to the reintegration component of the project. At the time when the project selected the areas where it focused its prevention work, the places of origin of the children to be reintegrated was not known. As a result, the project interventions for prevention, which could have been beneficial to the households that received children for reintegration, only benefited a few of the children who were reunified with their families. This said the project has contributed to a significant change in mindsets around the importance of family care for children, it has managed to break the trend of an increasing number of CCIs opening up in the 12 focus districts, and the project furthermore significantly strengthened knowledge, skills and professional practices of district, sub-county and community stakeholders involved in alternative care. 16 Overall, there is great enthusiasm created around the DOVCU project which most stakeholders consider as a success. 17 EVALUATION PURPOSE & EVALUATION QUESTIONS EVALUATION PURPOSE The objective of the evaluation is to assess the performance of the “Deinstitutionalization of Orphans and Vulnerable Children Project in Uganda” (DOVCU) with regards to the creation of sustainable changes in the lives of two beneficiary groups, namely 43,000 vulnerable households and 2,000 children at risk as a result of an integrated package of support. A second objective was to assess how and if these results came about from systems changes and identify which strategies and approaches were the most effective for achieving the change in children’s lives. Lastly, the evaluation also captures lessons learned around the project’s work with child care institutions (CIIs) and the project’s integrated package of support and on the work with different professionals involved in gatekeeping. With this in mind, the purpose of the evaluation is to establish and ensure accountability to the donor, government stakeholders and the beneficiaries of the support; contributing to learning, including documenting and generating lessons learned on how current approaches can be strengthened to support vulnerable groups and prevent family separation and reintegrate children back to nurturing family environments. This learning will be used to inform current programming both in Uganda and elsewhere and help the implementing partners and the donor support policy debates on child care system reform. In order to achieve this purpose, the evaluation took both participatory and formative approaches. EVALUATION QUESTIONS The evaluation covered the following evaluation questions: Impact 1. To what extent and how did this project identify/test successful interventions that a) prevent child separation and b) support reintegration of children at risk or in residential care? Has their well-being improved? For which children was impact most visible? 2. To what extent and how did the project contribute to the reduction in the number of children living in institutions and on the street in the targeted districts? 3. To what extent and how have prevention methods employed by the project increased nutrition, education opportunities, care, and protection for children in targeted households? 4. To what extent, and which systemic changes did the project bring about at community, regional, and national levels that are enabling children to live in family care and preventing inappropriate placements in institutional care? 5. To what extent and how did the project contribute to implementation of national care reform measures? Relevance 6. To what extent and how did the project’s integrated model, which links family strengthening, child protection and economic strengthening measures, offer examples and approaches for expansion, adaptation, and/or replication? What combination of interventions within this model could be adapted or expanded? 7. How did the political environment and context of the project influence operations and impact, and to what extent was the project able to mitigate the risks and leverage new circumstances/opportunities that arose? 8. To what extent and how was the integrated model, which links family strengthening, child protection and economic strengthening measures to maximize prevention for children at risk of separation from family and support those returning to family-based care, relevant and sufficiently adapted to the communities where the project operated? 18 9. Based on what is found to be the most common risk factors to separation, to what extent and how did the project manage to address these in an appropriate manner, including for boys and girls? Effectiveness 10. To what extent have the targets of the project to support approx. 43,000 children in vulnerable households and approx. 2,000 children to be reintegrated been reached and how were they achieved? 11. To what extent did the beneficiaries of the project, including children, families and communities, appreciate the services provided? Did the project incorporate sufficient participatory feedback processes to ensure the target reach was involved and informed of the intervention and received services most appropriate for their needs? 12. What were the most significant changes in the lives of the two target groups achieved by the project, and what factors seem to have influenced these changes? 13. How effective was the project in engaging with various structures and professionals involved in the project, including district authorities, child care institutions, probation social work officers, community development officers, alternative care panels, remand homes, community child protection structures, para-social workers (PSWs), etc.? 14. To what extent did the project manage to disseminate information regarding lessons learned from its activities and strategies? How did the project adjust its own strategies to take into account any learning that derived from the implementing process? Efficiency 15. To what extent were the project strategies, to reduce vulnerability of households and to reintegrate children, efficiently implemented? Were there any implementation gaps? 16. Which strategies / activities used by the project proved to be the most efficient in influencing the lives of children and vulnerable families? Were there any strategies / activities which were not at all efficient? 17. Which strategies used by the project proved to be the most efficient in influencing the system reform? Were there any strategies / activities which were less efficient? 18. To what extent were the human resources, coordination mechanisms, time and financial resources invested in the implementation of the project efficiently used and were they sufficient to achieve the targets of the project? Sustainability 19. For the children who have been reunified with their families or placed in alternative family-based care, what reintegration methods have supported stable and sustained placement of children and to what extent? Are children still in the homes and in school 6 months (and more) after placement? What makes these reintegration methods sustainable? If there has been relapse in reintegration, why has this been the case? 20. By project end, to what extent have structures been established or strengthened that can continue to provide, on an ongoing basis, adequate case-management and support services for children at risk? Are these sustainable and what makes them sustainable? 21. For the different changes observed in the capacity and functioning of professionals, child care structures and coordination mechanisms (e.g. probation and social welfare officers (PSWOs), PSWs, alternative care panels), to what extent are they likely to sustain the work, to continue meeting and to continue to apply what they have learned after the end of the project? Human Rights 19 22. To what extent did the project manage to target those most at risk (e.g. children in institutions and children on the street), including tailoring activities to accommodate needs of the most vulnerable children (children under three, girls, children with disabilities, street children)? 23. To what extent was the project able to prevent child protection issues and human rights violations (e.g. child labor, violence against children, neglect, early marriage, etc.) and mitigate risks for creating greater vulnerability (e.g. child labor and any kind of harm)? 24. How did the project address child rights violations that required urgent attention? 25. To what extent did the project contribute to strengthening duty bearers’ knowledge of their obligations and their capacity to meet those obligations? 20 PROJECT BACKGROUND COUNTRY CONTEXT Uganda has an overall population of 37.7 million, with close to 60% being children below the age of 18 [United Nations Children’s Fund (UNICEF), 2015]. Rich in natural resources, the country has seen steady economic growth in recent years, but still remains one of the poorest countries in the world. Social indicators are generally providing a picture of a struggling health and education system and high levels of vulnerability of children. In the absence of better alternatives, placements of children outside of the family, either in kinship care, or in residential care has been a coping strategy for many families living in poverty, lacking access to health care or quality education, but also to children who have been orphaned by war or by the HIV crisis. For example, Uganda has a higher prevalence of children living in related (kinship) care than most other countries in the East Africa region (Better Care Network, 2015). Prior to the government adopting a policy on alternative care, the Alternative Care Framework (ACF), in 2011, an increasing number of children were being placed in residential care institutions that were reported to be “mushrooming” in Uganda. Around the time of adopting the ACF, the Ministry of Gender Labor and Social Development (MoGLSD) estimated that there were around 57,000 children living in residential care across Uganda (ChildFund, 2013). At the same time, Uganda was among the top 20 countries of origin for inter-country adoptions with several organizations raising their concerns regarding the lack if safeguards and oversight of procedures for inter-country adoption (UNICEF, 2014). THE DOVCU PROJECT The “Deinstitutionalization of Orphans and Vulnerable Children Project in Uganda” was a 36-month child care reform project supported by John Snow International (JSI) and United States Agency for International Development (USAID) implemented in 12 districts of Uganda set out to develop and test a package of services at community level in order to 1) prevent the separation of children for their families, and 2) facilitate the reintegration of children from child care institution (CCIs), the street and remand homes, back into family-based care.3 At the time when the project was approved, 2014, several other projects had already started to strengthen Uganda’s child protection and social welfare systems. Reforms in the development and provision of alternative care started in 2011. Capacity of the central MoGLSD had already been strengthened through the establishment of an Alternative Care Unit (ACU) that became responsible for the oversight and regulation the increasing number of CCIs that were operating without registration in the country, and outside the oversight of the Government system. The DOVCU project was introduced as a contribution to these alternative care reforms a few years into the process. The overall objective was to “measurably improve the safety, wellbeing and development of highly vulnerable children, particularly those who are living without adequate family care” (ChildFund, 2013). The project aimed to reach approx. 43,000 vulnerable children living in households facing high risk of separation and to reintegrate approx. 2,000 children currently living in institutional care or in the streets. Conceptual framework and hypothesis 3 Northern districts: Gulu, Lira; Western districts: Kasese, Kabarole, Kabale; Eastern districts: Mbale, Kamuli, Jinja, Iganga; Central districts: Luwero, Kampala, Wakiso 21 The project’s conceptual framework defined layers of protection and components of an alternative child care system. It aimed to address the “root causes” of child separation at the family level and strengthen formal and informal systems for service delivery at the community level (ChildFund, 2013). According to the conceptual framework the DOVCU project aimed to contribute to the strengthening the continuum of care defined by the Ugandan Alternative Care Framework (ACF) (MoGLSD, 2011), including preventive support services, emergency and permanent care, and gatekeeping and through this contribute to deinstitutionalization, placing emphasis on the de￾institutionalization of children through reintegration back into families. The project was designed based on a hypothesis that the placement of children who are outside of family care, living in residential care, into stable family environments through tracing of families, family reunification and placement in alternative family-based care would generate improved child wellbeing (ChildFund, 2013). Project approaches and strategies The DOVCU project took a bottom-up, top-down participatory approach, involving formal and informal stakeholders from all levels of governance in skills building and training activities to activate their roles and the functions they have to carry out in the operationalization of the ACF. In order to support the overall project objectives, to reduce unnecessary separation of children from their families, and to ensure nurturing family care for children, the project targeted to reach approximately 43,000 vulnerable children in vulnerable households and approximately 2,000 children living in CCIs or in the streets. The project made a number of assumptions related to the type of support needed to change life-outcomes of these beneficiaries. For families and children at risk of separation, the project assumed that a reduction in unnecessary separation of children from households which are considered as “at risk of family separation”, can be achieved through strengthened and stabilized economic conditions in the households; through providing vulnerable households with labor market skills; strengthening parenting skills and an improved understanding by vulnerable households and their communities of the benefit of family-based care over institutional care. The assumption was that the project support would create sustainable changes in the lives of these two beneficiary groups as a result of an integrated package of support. For children living outside of nurturing family care (in child care institutions, remand homes, or on the street), the project was designed based on the assumption that placements of children who are outside of family care into stable family environments can be attained through tracing of families, family re-unification activities or placements in alternative family-based care. As such the project was designed to have two distinct components: one prevention component and one reintegration component which were implemented following distinct targeting processes for each: • Prevention component: DOVCU selected the districts (12 districts), sub-counties (2 sub￾counties / district) and parishes (2 parishes / sub-county) in which it focused its activities related to the prevention component of the project. To inform interventions with vulnerable families 22 and children, the project collected baseline and monitoring data using tools such as Child Status Index (CSI), Family Status Vulnerability Index (FSVI), to inform targeting of interventions and project design. During implementation targeted households were provided with an integrated package of support, and communities and para social workers were mobilized and skilled in parenting practices, and economic strengthening skills. • Reintegration component: CCIs, remand homes and a service provider working with children on the street were involved or targeted with a number of activities. In the 12 project districts, CCIs were mapped out and assessed per the national guidelines and standards for operation. CCIs were supported to develop improvement plans, were involved in awareness raising on family￾based care and national standards, received training for administrators and social workers (including in alternative care), and mentoring and support for the reintegration case management process (including family tracing). The reunification process and the support provided to enable reintegration were implemented collaboratively between district officials, child care institutions, remand homes, TACs, PSWOs, CDOs, para-social workers and community child protection structures. It was assumed that such activities would generate reintegration and an improved wellbeing for the concerned children. To support the reintegration of children from the street, DOVCU partnered with a Non-Governmental Organization (NGO) called Child Restoration Outreach (CRO) in Mbale district that has significant experience in reintegrating children from the street back in families. Within the DOVCU project, CRO focused its attention on improving its professional practices to reach girls who have been living and working on the streets, because they often have specific and different needs as compared to boys. Activities included the development and promotion of the reintegration SOPs and tools (same as for reintegration of children from remand homes). However, the children to be reintegrated originated from many more districts and parishes than those supported through the project’s prevention component. Children were therefore reunified to a large number of communities some of which were supported through the projects prevention component, others not. All children who were reunified, within or outside of the 12 focus districts, where supported with a minimum package of supplies and services.4 Furthermore, the project aimed to support national alternative care reform efforts through deinstitutionalization and strengthen gatekeeping practices at district and institutional level. To increase functionality of various structures and professionals involved in alternative care, the project relied heavily on producing guidance documents, tools, various training and skills building activities to show how to use new procedures, and to produce changes in mindsets, professional practices and systemic changes at the level of national and district structures, CCIs, CRO and remand homes. These stakeholders were furthermore involving in and included in training, coaching activities and a wide range of stakeholders, including national, district, sub-country and community professionals and structures were mobilized as a key strategy of the project to achieve institutional changes and changes in professional practices that could help sustain the beneficiary level achievements in the longer term. 4 According to the project, all the 1045 children reunified in the 44 districts outside the 12 DOVCU districts received post placement follow up services either through DOVCU or respective CCIs. Some children benefited from both partners (CCIs and DOVCU project) in the given packages as per the case assessments and case plans. For cases that were handled directly by CCIs, DOVCU ensured that the geo-location and status of all these reunified children and their households were ascertained through tracing, upon which responsibilities were shared with respective CCIs to undertake follow up as per the case management system for children placed back to family care. According to the project, DOVCU district social workers provided technical support to the respective CCIs social workers to facilitate conformity in the reintegration process. While all the case files of those reunified by CCIs were held by the CCIs themselves, according to DOVCU these were continuously reviewed by DOVCU social worker. Selected sample cases were jointly followed up by both DOVCU and CCI social workers. DOVCU provided monitoring and guidance to CCIs for the children they reunified with families. 23 Project phases Although the project was formally approved in July 2014, the start was delayed for political reasons. Activities formally started in November after the official launch. To manage the significant geographical scope of the project, covering 12 districts, the project was implemented in a phased manner: ● Between July and December 2014: A preparation phase staff hire allowed for adaptation of tools, mobilization of stakeholders and putting in place the structure of the project (e.g. recruitments). ● January 2015-June 2015: Start-up in the first 6 districts (phase 1 districts) focused on assessments of CCIs and CCI-improvement planning in the first 6 districts, service mapping, household and family identification and assessment, market assessment, baseline study, and training of trainers (TOT), development of the case management system, training and other capacity building activities. ● June 2015 – June 2017: Implementation in phase 1 districts included activities such as coaching and support work, finalization of case management tools, forming Village Savings and Loan Association (VSLA) groups, cash transfers and training in financial management for families, work on parenting training, case audits, on-going capacity building, awareness building, reunifications and follow ups, work in Remand Homes and with street children, promotion of standard operating procedures (SOPs), baseline evaluation and community trainings. ● September 2015 – March 2016: Start up in the remaining 6 districts (phase 2 districts) allowed to replicate the same assessments as previously carried out in phase 1 districts. ● March 2016 – June 2017: Implementation in phase 2 districts with same activities are implemented in phase 1 districts. ● June – December 2017: A closeout phase was common for both phase 1 and phase 2 districts in spite of the significant difference in implementation period. It included activities such as continued parenting training, financial management/VSLA groups, on-going awareness raising and training in SOPs in addition to reflection meetings of project partners in districts and regions, endline assessments of CCIs and of status of well-being of beneficiaries of the project and gradual handover of activities to the partner CCIs and government officials. 24 EVALUATION METHODS & LIMITATIONS EVALUATION METHODS A common methodology was developed for the evaluation both baseline and endline. It is based on participatory and formative approaches to evaluate the project’s overall theory of change (TOC). The project consortium and USAID were involved in brainstorming sessions to finalize the evaluation questions and feed into the evaluation design. Both for the baseline and the endline the project consortium partners were consulted in the data collection process and involved in dialogue on preliminary observations and findings. The evaluation places a strong emphasis on learning and maintaining high ethical standards in research. As such the design of the evaluation and the tools were reviewed and approved by an ethical review committee prior to data collection. The evaluation relied on a mixed-methods approach (USAID, 2013) with a strong qualitative component. The methodology consists of three distinct methodological pillars used to triangulate and consolidate findings, and to answer the different evaluation questions: policy analysis, qualitative component and quantitative data analysis. A policy-based analysis focused on the project’s intent to bring about systemic change in child care structures and institutions at national, district and local levels. This part of the methodology consisted of reviewing relevant policy and child care system regulatory documents, case files and seeking information through key informant interviews (KII) with national and district level policy makers, civil servants, professionals, management and social workers of service providers to form a view on the set up of system, and to assess changes in services, professional practices and the alternative care system more generally. The field work started with a one-day workshop with the DOVCU consortium to identify lessons learned and debrief on the project consortium’s own assessment of results, strengths and challenges in the implementation of the project. This was followed by individual meetings with each of the consortium members to discuss the different contributions to the project. Followed by KII with stakeholders working for the MoGLSD, supporting or overseeing the implementation of Uganda’s implementation of the ACF: district authorities and Orphans and Vulnerable Children (OVC) committees (9 representatives); district community development officers (DCDOs) (5); and gatekeeping structures and professionals at district level, including Probation and Social Welfare Officers (5 permanent and one acting PSWO) and Community Development Officers (23 CDOs in total); members of TACs (19 members); representatives from three different remand homes located in in Gulu, Kabarole and Mbale districts; representatives from reintegration services for children on the street (Child Restoration Outreach in Mbale district); and directors, managers and social workers in 21 CCIs spread across the six research districts. Finally, the policy-based analysis was also informed from a number of quantitative indicators for which data was collected through the projects’ own monitoring process. A qualitative component of the evaluation focused on the perceptions of children and household members, community-based structures and PSWs regarding their changing vulnerability over the life of the project, general wellbeing, and prospects for the future. Through qualitative methods, including KII and focus group discussions (FGD), information was collected on the mechanisms through which different interventions of the project had been implemented and perceived by recipients. Together with the monitoring data collected by the project implementers, the qualitative assessment has made attempts to unpack beneficiaries’ self-assessment of their personal welfare and the extent to which the project’s interventions contributed to any changes in this. Data was collected in 6 districts, with a total of 12 communities visited. In each community teams visited four households where children had been reunified with their families and four at-risk households where caregivers and children were interviewed separately. In addition, the teams conducted FGDs in each community with 1) PSWs, 2) CPMs, 3) women in the community, and 4) men in the community. 25 The quantitative data analysis presented as part of this evaluation is relying on project monitoring data. Monitoring data is used to provide a summary picture of the baseline and endline samples of beneficiaries from which qualitative data was collected, while also descriptively investigating the changes observed between the baseline and endline for the target groups of the project. The analysis helps shed light on changes in vulnerability. At the same time, the quantitative data formed the basis for a descriptive analysis disaggregated across categories of interest, including gender, age groups and geographic location. Quantitative monitoring data allows measurement and analysis of changes over time on key project outcome indicators but cannot be used to make claims of causality or impact attributable to be project. This point is further elaborated in the section on limitations that follows. The evaluation sampled 6 of the 12 project districts for primary data collection. This included Gulu, Kabarole, Kasese, Kamuli, Luwero and Mbale. LIMITATIONS Since the monitoring data was not collected by the evaluation team itself a process was outlined to ensure quality control of data before its use in the evaluation. During the training of research teams in Kampala, the qualitative research team created an abridged version of both the Family Support Vulnerability Index (FSVI) and the Child Support Index (CSI), to be administered during household case studies taking place as part of the qualitative research. The aim was to control the quality of responses provided to these tools, in order to verify accuracy of the data to be used in the evaluation. At the conclusion of fieldwork in all six districts, these survey forms were provided to ChildFund for comparison with the original survey tools and there were no indications of discrepancies found between the data from the short-form instruments and the original instruments. Consequently, the data from the DOVCU project’s monitoring has been used for the purposes of this evaluation. There are limitations affecting the quantitative analysis, which need to be taken into account when interpreting its findings. This evaluation does not include a quantitative impact estimation component. The quantitative data discussed in the report was not collected through the use of a household survey designed and developed for the specific purpose of evaluating the impact of the project. Project beneficiaries, including households, caregivers and children, were not selected using any probability sampling. There can be no causal inference or external validity5 claim derived from the quantitative data analysis presented. Because end line quantitative data was collected as part of the regular project monitoring no impact evaluation design or related sampling strategy informed the selection of the sample or the determination of the sample size. As a consequence, the findings of the quantitative analysis may not be representative of the population from which the sample was drawn and cannot form the basis for econometrically robust impact estimate. The investigation of quantitative data should therefore be treated strictly as a descriptive means to find patterns and intertemporal trends from baseline to end line. Any descriptive associations detected across categories of households or children should not be read as statistically significant correlations. As already described in the baseline evaluation, the resulting tabulations derived from the monitoring data still allow us to draw an interesting descriptive picture of the changing characteristics of project beneficiaries, which can be integrated with qualitative findings to build a narrative around the project evolution and its effects over time. 5 External validity is the degree to which the results emerging from the analysis of the study sample would hold for other relevant populations in other contexts and at other times. 26 It is important to consider that the end line project monitoring sample of households and children, surveyed represents a sub-sample of those surveyed at baseline: 3,297 households and 6,754 children were surveyed and assessed at baseline; while end line sample covered a sub-set of 2,235 households and 3,498 children. The baseline covered a larger sample due to the initial targeting needs of the project, and the end line focused on households the project worked with over the course of the project. While the figures reported in the Project Monitoring Plan (PMP) reflect these two separate samples, in line with the format agreed at inception, other descriptive trend analyses of differences between baseline and end line included in the report focus on changes that occurred within the same sample of households that were interviewed at the two points in time. In technical terms, the trend analysis focuses on a panel of households.6 This ensures a greater degree of comparability between baseline and end line and makes the analysis of the related intertemporal changes in indicators of interest more robust and informative. Important to note is that of the sample selected for project monitoring purposes at end line, reported logistical and financial limitations of the project did not permit a rigorous end line assessment in households outside of DOVCU supported districts. This had implications for the children who were reintegrated. Of all 1,743 reintegrated children reported by the project, end line project monitoring data only covers the 698 children within reach, i.e. those reintegrated with the 12 districts where DOVCU focused its prevention activities. However, the conditions for the reintegration process varied significantly for the children who were reintegrated within the project supported districts and parishes and those who were reintegrated outside of these districts and parishes and therefore the “sample” may not be entirely representative. Further, the qualitative and policy analysis sampled six districts all phase 1 districts where the project had a longer programmatic engagement. The findings might be slightly influenced by this. Though the project had slightly differing packages for the two objectives (prevent child separation and support reintegration), the qualitative sample was drawn out of communities where the prevention component was to be implemented,7 as the children to be reintegrated had not yet been identified at the time of baseline. Since the evaluation design relied on a change-over-time approach in understanding the impacts, this sampling could not be changed for end line. This led to a situation where the majority of sampled households were located in the communities where the project had implemented the prevention component. Furthermore, the evaluation sampled households from within the 12 districts where it had focused its project work. Therefore, the sample of reintegrated children did not include children who had been reintegrated outside of the 12 project districts. The project interventions evaluated under the prevention component are from the perspective of households at risk, as well as households who have received children for reintegration. The sections that follow represent a synthesis of diverse perceptions, and changes to factors that stakeholders identified during the baseline as influencing separation and reintegration. Discussion focuses primarily on perceived changes to these factors, and the extent to which respondents perceive change as related to DOVCU interventions. The monitoring data is considered throughout to triangulate and further unpack the qualitative findings. Because of the evaluation design and its limitations, impact findings focus primarily on changes, and perceived impact around relevance, effectiveness, efficiency, sustainability and human rights. 6 Within the same households the surveyed children are not necessarily the same as they were selected in specific age groups and the distribution of children across these groups changed over time. 7 There are two cases of caregivers visited where the household falls out with the prevention component target areas, however, this data is not strong enough to draw any conclusions regarding the differing contributions of the project where the prevention package was supplied, versus where only the reintegration package was provided. 27 28 FINDINGS & CONCLUSIONS IMPACT The following section presents the results and impact achieved by the DOVCU project. Evaluation question 1: To what extent and how did this project identify/test successful interventions that prevent and support reintegration of children at risk? The project addressed causes to separation at three different levels: immediate, underlying and root causes, and interviews at community level indicate that interventions are perceived to match identified community needs. The baseline evaluation identified three different layers of causes to separation: immediate causes, underlying causes and root causes. “Immediate causes are those causes which seem to be more directly associated with risks that lead to the decision of a child to leave, or the decision of a parent, community member or a professional to refer a child to a child care institution or report him/her to the police.” 8 The baseline evaluation identified immediate causes to include high levels of food insecurity, negative household dynamics, violence and child labor, drug and alcohol abuse in the family, and lack of access to basic social services. “Underlying causes are those causes which contribute to a specific vulnerability that make such a situation more likely to happen,” such as poverty, psychosocial vulnerability and barriers to good parenting. 9 “Root causes are hierarchically the underpinning societal situation contributing to a general overall vulnerability”10, such as lack of financial opportunities, lack of land and assets, social and cultural norms, and political instability and displacement. The sections below illustrate how the project interventions have contributed to addressing some of these issues. A more detailed analysis is provided in “Annex I: Lessons learned on project’s integrated package of support”. Economic well-being Findings based on a comparison of baseline and endline data on vulnerable households show a reduction in the household economic vulnerability. The changes found show that: • The number of households classified at destitute level reduced from 1,252 (56%) at baseline, to 670 (30%) by endline. • On the other hand, there are increasing percentages of households that score in the struggling category from 974 (44%) at baseline to 1,534 (69%) at end line. This indicates that the project has helped shift the supported households mainly from destitute to struggling. This shift has occurred to a similar level in both male- and female-headed households. • There are two districts, namely Kabale and Kampala, which show a negative shift. In Kabale, there were 122 (52%) of households that scored as destitute at baseline, while there were 153 (66%) at end line. In Kampala, 63 households (69%) scored as destitute as baseline, while this had increased to 71 (78%) of the supported households at end line. More detailed figures are provided in “Annex V: Statistical Tables”, Table 13 and 14. 8 DOVCU baseline report 9 Ibid. 10 Ibid 29 Figure 1: Percentage changes in the proportion of household economic vulnerability classification: Baseline Value (BLV) and Endline Value (ELV). Only about 1% of households graduated to the growing category, i.e. in relative terms the less poor category of household vulnerability. In the context of the volatile environment in which the respondents live, a shift from destitute to growing category may not be enough to keep families out of situations that make them vulnerable to separation. As one caregiver in Luwero explained: “…it [the financial support] is working but you don’t prosper because even though we farm sometimes seasons fail and rearing might also not go well meaning that we are still down because we don’t have jobs that quickly boost someone. Sometimes things are good sometimes bad.” Findings indicate that there have been some improvements in the households’ economic livelihood security, with access to land and livestock improving as well as access to credit sources. Project beneficiaries feel that this has contributed to prevent separation. As identified in the baseline, and continuously so at end line, poverty is a key risk factor to separation as well as being a barrier to reintegration with lack of funds making it difficult for families to meet children’s basic needs. It is important to note that financial vulnerability and poverty are understood not only as cash poor, but in terms of a lack of assets, being labor constrained, having a shortage of economic opportunities, and not having a high resilience against unexpected economic shocks. In this context, in terms of prevention, the qualitative analysis confirms that: • DOVCU’s financial support has been supporting families in many ways. On average, households that were targeted for cash transfers received two rounds. In total cash transfers were distributed to 720 out of the 739 destitute households in the phase 1 districts. With the added support, families reported being able to pay school fees, could more readily afford food, clothing, or inputs for farming, and emphasized that the financial support has helped keep children in school. Respondents perceive this to reduce the risk of separation, and to help families reduce stress. As a man from Gulu stated: “Before this support, I had no food for children, couldn’t afford to send children to the hospital, no money for fees for their education, no clothing, no beddings. Now with the support, I have also started changing the diet for the family. Part of the money I used it for buying cloths and change diet.” • Respondents in all households emphasize the VSLA training as impactful. 96% of the targeted households received business skills training, 79% were provided with financial literacy training and 83% with VSLA training. As one caregiver in Kabarole mentioned: “They took us and taught us how to save money and invest it so that it can keep on earning profit. We then formed a VSLA where we are 30 members and we are saving together…We save money every week. This has 30 helped me to work hard because minimum saving is two thousand shillings a week. But if you can afford more it is okay because it goes to your account. This can help you to get more money when you are looking for a loan.”.” Several respondents stressed how they ended up using the cash transfer for immediate needs such as medicine or food, and that they therefore ran out before they could invest it towards longer-term stability. Though respondents are appreciative with regards to the business skills and financial literacy training, they highlight the challenges they face in applying the skills learned from these trainings. Key challenges highlighted are perceived insufficient funds and the volatile environment. As stated by one caregiver: “That training can be helpful if you are earning and can budget for what you earn, because they trained us that if you get money you should save to start up an income generating activity. But if you don’t have the capital then you can’t start up the business. Among the things they taught was how to save and how create income, but it is very good information which I will implement when I get money.” Psychosocial well-being Training reportedly made parents more aware of their duties as parents and how to approach their children. The baseline evaluation identified that neglect or lack of awareness about how to provide proper care, supervision and attention to children in the household further increases vulnerability to separation. The training on psychosocial issues, parenting and home visits were very particularly appreciated by community members, PSWs, and CPM members who cited changes within the community as a result. As one woman explained: “What I normally see with these children is that the moment they come back into their homes, there is always a warm welcome for them (…) We have been trained and now that is what we do to such children: talking and advice is the best they can receive when they get back home.” Training of parents and the training for PSWs were perceived to increase chances of success in creating the change needed in communities to prevent separation. It is important to acknowledge that at this stage, it is difficult to assess norms and behavioral change due to the short time frame of the projects. But clearly, parents/caregivers are more aware of alternative norms and attitudes towards child care. This is illustrated by one caregiver: “What they teach us knows that when a child reaches a certain age he/she changes. Knowing that if a parent works together with a child, a child becomes free to share his/her problem with the parent. We also learnt to sit with children, not to just beat them, but also make children your friends. This has changed the lives of our children and they are free to talk to us.” Findings from interviews with children indicate that parents engage with this in practice. The majority of children interviewed do not mention violence within the home at end line.11 Evaluation question 2: To what extent and how did the project contribute to the reduction in the number of children living in institutions and on the street in the targeted districts? There is no aggregate national or district data that can evidence impact in improving child separation-trends overall. It is not known how many new children separated from their families and came to the street or live in extended or other families. It is not known how many children separated from their families and migrated within the country, ending up in CCIs or on the streets in other districts. Interviews with policy level stakeholders in several of the districts suggest that there are even more children separating and coming into street-situations than before because of natural disasters (drought, famine, landslides), urbanization and difficult situations for children at home (violence, alcoholism, child neglect). The scope of the DOVCU project was too small to have a measurable impact on this nationwide situation. 11 It is important to note that contrary to baseline, children were this time interviewed in their homes. Although parents were not present during the interviews, the fact that children were interviewed at home could make it less likely for children to disclose certain information. Additionally, what is perceived as ‘violence’ is often very different in differing contexts, where harsher discipline can easily become normalized. 31 This said, DOVCU supported the reunification of a total of 1,743 children to families. As illustrated in Figure 2 below, of these 1,743 children: • 1,465 children (84%) were reunified from different CCIs; • 74 children (4%) were reunified from remand homes; and • 204 children (12%) were reunified from the street. More details on the numbers of reunified children can be found in “Annex V: Statistical Tables”, Tables 15, 16 and 17. Figure 2: Proportion of reunified children who were reunified from CCIs, remand homes and from the street The districts that were involved in DOVCU showed a significant reduction in the number of children in CCIs. This is illustrated in the project’s monitoring data, which indicates that the districts involved with the DOVCU project have an average 26% reduction in the number of children in care. Luwero and Kamuli districts have seen a greater than 300% reduction in the number. The only district with no reduction is Kampala (see also: “Annex V: Statistical Tables”, Table 17). Across all districts, interviews with policy level stakeholders confirm this trend and attribute it to changing priorities in terms of the services provided by CCIs, fewer admissions, greater outflow of children and that children stay for shorter time-periods. For example, in one district the Probation and Social Welfare Officers (PSWO) reported: “Actually, almost all of them (CCIs in the district) have reduced numbers. Almost all of them. I would say that 100% of the CCIs that we have in (district name) have reduced numbers. I was telling you that there are some that became schools. We have one in (name) sub-county (…) it was turned into a primary and secondary school and the children that were there were resettled.” In another district, a representative of a CCI reported: “Previously we were admitting 30-40 per year. Last two years we have had 15 new children” and in yet another district another CCI, illustrated this trend with the following words: “The other achievement is the reduction in the number children, and regulation of those who come in. Before the project we had 22, currently we have 11, not because we have been forced, but because we appreciate the alternative care framework. We have resolved on the number we can manage.” There have also been some changes in the overall numbers of children in remand homes, however these changes, whether increases or decreases, cannot be attributed to results of the DOVCU project. This is because the entry and exit of children 84% 4% 12% % of children reunified from CCIs, remand homes & street Reunified from CCIs Reunified from remand homes Reunified from the street 32 into remand homes, to a large extent, is dependent on children’s court sentencing, something that DOVCU did not try to influence through its activities (see also: Annex V: Statistical Tables”, Table 15). Evaluation question 3: To what extent and how have prevention methods employed by the project increased nutrition, education opportunities, care, and protection for children in targeted households. This section focuses on the extent to which prevention methods employed by the project increased food security and nutrition, education opportunities, care and protection of children in targeted households. Between baseline and end line, monitoring data indicate that overall, there has been a reduction in child vulnerability scores. The most significant changes can be seen in health, shelter and care. This is followed by food and nutrition and education and skills training. Psychosocial support and Protection also show an improvement, though slightly lower (see also: “Annex V: Statistical Tables”, Table 18). Food security and nutrition Findings indicate that the financial support package has helped vulnerable households to better regulate consumption and increase food security and nutrition. However, respondents do not consider the financial support has fully brought them out of food insecurity for the longer term. This is illustrated by quantitative findings from the Family Status Vulnerability Index (FSVI) tool that indicate households have increased the number of meals per day: at baseline 15% of households stated they rarely had food to eat, and were going to bed hungry most nights, while this had reduced to 3% by endline. At baseline 55% of vulnerable households stated they frequently had less food to eat than needed, complaining of hunger, compared with 32% at end line. The majority of households (57%) now feel that they have enough to eat , depending on season or food supply, while at baseline only 26% had enough to eat (see also “Annex V: Statistical Tables”, Table 7). Qualitative data confirm that respondents still perceive food security as precarious. Across districts, respondents continuously refer to the volatile environment in which they live, and how this impacts on food security. A caregiver reflected on this with the following words, a quote that is indicative of multiple households in the qualitative sample: “I can say our welfare is not stable, it fluctuates. Sometimes we are doing well and sometimes we are economically badly off. My husband is a trader of bananas. There is scarcity of bananas now. It is rare to find and we end up eating/using part of the business money. Our incomes are now miserable, there is scarcity of food, school fees for children in school is difficult to come by now. We have encroached and used money for business and spent it on home consumption.” The majority of respondents are wary of stating that their food security status is better, instead stating that it is “better right now,” as one caregiver put it. With regards to children’s subjective well-being on food security, the majority of children in the qualitative case studies of at risk households perceive that they now eat better than before (two years ago). Children highlight diversification in the type of food consumed: “We eat good food these days and it’s always enough food at home. There is a very big change in the way we eat; these days we eat more nice foods like meat,” said an at-risk child in Kabarole. Still, children are acutely aware of the precariousness of the food situation, referring to “it depends on if mom has money” and similar statements to explain that while the food situation is slightly better now than before, this is not the consistent day-to-day. Moreover, several children in households that received cash transfers highlight how there was food “for a while” but not anymore. In many cases, the same respondents emphasize that their parents now have a small plot of land and that, such as one child put it, “we planted, but it is still in the garden, it is not yet ready.”.” 33 For children who have been reunified12, there is a significant variance in children’s responses around subjective well-being. Some of this appears to relate to household vulnerability status, with children returning to destitute households more likely to state that they now “eat less than when I was at the home” (reunified child, Mbale). Children in this situation, similar to children in at risk households, highlighted how the cash transfer provided by DOVCU helped their families buy food, though in several cases “this food is no longer” (reunified child, Luwero). • Children who have been reunified from the street or remand homes, consider their situation with regards to food security and nutrition to have improved in the majority of cases, though still be precarious: “There was a time when we completely did not have anything to eat, we had potatoes that we had planted but wasn’t ready for eating yet. But (DOVCU) came and bought for us some posho and beans” (reunified child, remand home, Gulu). Children reunified from these situations are more likely to indicate their improvement as relative, i.e. “it was bad there [at the home] but here also we go to bed hungry, but not always” (reunified child, remand home, Gulu). Importantly, children emphasize that while they may eat less regularly than when they were at the remand homes, they eat more of a balanced diet. This is similar to perceptions of children reunified from CCIs. The data around children previously living on the street is more in line with children from destitute households, where the situation was bad and it is still not secure, but better. • When children previously at CCIs assessed their subjective well-being, they emphasize the regularity of meals prior to reunification. The majority of children reunified from CCIs perceive their food security and nutrition to have been better prior to reunification (while in the CCI) but consider the situation at home to be better ‘now’ than when they first came back home. This thus indicates that the prevention strategies of the project, especially the financial support package has, in the majority of cases, been able to contribute to improved food security and nutrition situation within family-based care. Education Opportunities There has been an improvement in households’ ability to send children to school. As highlighted in the baseline, lack of access to basic social services such as access to good quality education is strongly associated by respondents with children separating from their families, in particular to go live in a CCI, where parents think that they will receive education. This “pull effect” of CCIs is important, as end line findings around admission state school/education as one of the main reasons (together with poverty) for children to be admitted. Baseline quantitative analysis showed how children in destitute and struggling households were more likely to miss school. This is consistent with qualitative end line findings which see respondents highlighting ‘ability to pay for school fees’ as a major change in their wellbeing. The FSVI assessment shows 49% of households now have all children (of school age) attending school, as compared to 33% at baseline (see also “Annex V: Statistical Tables”, Table 20). This said, while the majority of reunified children state they are now able to attend school following reunification13, several children in at risk households stress that they are still sent home from school due to unpaid school fees: “It is school that is a challenge because I am always sent back home from school because of school fees” (at risk child, Gulu). 12 For children who have been reunified, it is important to note that they reference two points in time in changes to their food consumption and perceived food security and nutrition. Children who have been reunified refer 1) to when they were in the previous CCI/remand home/on the street, 2) to when they were reunified, and 3) to when the end line was conducted (‘now’). 13 Several CCIs have continued to support reunified children’s education. DOVCU’ cash transfer has reportedly also contributed to support children’s continued education. 34 Psychosocial well-being, care and protection In terms of psychosocial well-being, care and protection, including for children, respondents perceive the family situation to have improved. Similar to the baseline, negative household dynamics, such as domestic violence or spousal quarrel which were often quoted as a situation that could lead to separation, where prevalent across districts also at end line. During baseline, the qualitative component noted the particular widespread absence of mental health services. This is highlighted at end line as somewhat addressed through DOVCU, in particular with reference to PSWs. Caregivers refer to PSWs as “listening when I need” (caregiver, Mbale), and “they even guided me to a group” (caregiver, Kabarole). Furthermore, at baseline child respondents across districts widely cited corporal punishment, or ‘overbeating’ as an accepted norm that can lead children to leave home. This was not highly prevalent in the end line findings amongst children in all respondent groups. The majority of children at risk and children reunified, all cited violence against children to have been something they experienced previously, in particular the reunified children, but that this was not something they worried about now. When children did refer to violence, this was mainly in reference to fears of violence such as “if you get robbed” (reunified child, CCI, Kamuli), “some older people in the community may beat on you” (at risk child, Luwero), and “there are stories of people that take children” (reunified child, CCI, Mbale). Respondents continuously highlighted these project interventions as essential in improving the overall well-being of the household. Notably, very few children in the sample raised alcohol consumption as an issue, and there were low reports of violence in all households. Findings indicate that children who have been reunified from CCIs perceive their access to services and food to be more precarious following reunification and this adds to stress and reduces psychosocial well-being. Children in all categories highlight that they feel less stressed than two years ago or prior to reunification. Children at risk are, the majority of times, linking this to “no longer being chased from school” (child at risk, Kamuli), and “we have some money for food” (child at risk, Mbale). While children who have been reunified feel less stress with regards to experiencing violence, or maltreatment, they still perceive the situation in family-based care as stressful. In particular, children who have been reunified from CCIs emphasize that the stress is less at home now than at the point of reunification, but that they feel stressed about money running out, or accessing services, such as education, something that they did not worry about as much while at the CCI. Evaluation question 4: To what extent, and which, systemic changes did the project bring about at community, regional, and national levels that are enabling children to live in family care and preventing inappropriate placements in institutional care? The project influenced many systemic changes which are summarized here. More details are provided in Annex II: Lessons learned on the projects work with Child Care Institutions, Annex III: Lessons learned on the project’s work with various professionals and Annex IV: Lessons learned on case management. The DOVCU project contributed to strengthening the knowledge base of MoGLSD on the number and standards in CCIs in the 12 project districts. The project had a strong data collection strategy and generated a significant amount of evidence on CCIs and their standards. Between 2015 and 2016, through a process that systematically supported the MoGLSD in identifying and assessing CCI which were previously not known or registered by the ministry, an overall 147 CCIs were assessed between March 2015 and June 2016 with MoGLSD involvement at baseline and 91 CCIs were assessed 35 with MoGLSD involvement between October and November 2017 across the 12 project districts (see also: “Annex V: Statistical Tables”, Table 21).14 Changes in district structures and services At district level, the project piloted a new gatekeeping structures – Teams Around the Child (TAC) – in 11 districts. Informed from the experience from a national Alternative Care Panel (ACP) that was established under another project, the DOVCU project had planned to establish pilot district￾ACPs to oversee alternative care placements and adoption, and a second structure called a Team Around the Child (TACs) to oversee reintegration processes. In 2015, DOVCU developed TOR both for district ACPs and TACs. Similar to the national ACP, these structures were supposed to be multi￾disciplinary decision-making bodies. However, in the absence of a national agreement on the scope of these two different teams, and especially whether alternative care and adoption should be a district or national function, only TACs, to be in charge of overseeing reintegration processes, were set up in 11 out of the 12 project districts (all but Wakiso district). The first TAC to be set up was in Mbale district in September 2015, and the last one in Iganga, in February 2017 (see also: “Annex V: Statistical Tables”, Table 25). It took TACs between 2 and 5 months between their formation (appointment and training of members) and their first formal meeting. Across districts there is evidence indicating that the project contributed directly to many changes in the availability of services provided by CCIs and in the way they operate today as compared to the situation at baseline. In total, the project worked with 126 CCIs across the 12 districts. The following quote, from one district reflects well the full spectrum of changes that can be observed in many CCIs across DOVCU-districts: “There are many changes: the biggest is that they have be able to register: they are officially known, they have reorganized themselves, they have employed qualified staff, like matrons, nurses; improved infrastructure. Before the project some of the institutions were managed by the director and his wife. Most of them now have a child care policy, they have also started reporting to the district about their operations, they provided information about the children. The quality of care for the children has also improved, they are well fed and go to school. They report about what they are doing, the core requirements of the child are being attended to. They are far better than before.”.” Stakeholders furthermore attribute the fact that there are fewer CCIs as compared to baseline to the DOVCU project. Since the beginning of the project: • A total of 21 CCIs have closed; and • Another 14 had ended providing residential care and were operating either as a boarding school or a non-residential school or another kind of community-based organization (CBO). In four of the six study districts for the evaluation, district officials confirmed that some of the CCIs that had closed or stopped operating as CCIs, had done so as a matter of urgency, a process that was often initiated by the CCI itself following the initial assessment facilitated through DOVCU, and not always accompanied by the DOVCU project. 14 At the end of the project a number of child care institutions that were assessed at baseline had closed (21 CCIs) or transformed into another type of service (11 had transitioned into becoming a school and 3 had transitioned into a community-based organization). A number of CCIs had also not been involved in activities supporting reintegration of children (21 CCIs across all districts were not involved in DOVCU). At end-line, the DOVCU project therefore chose to only assess the CCIs they had worked with, and furthermore did not reassess transformed or closed CCIs. This explains the difference in the number of CCIs assessed at baseline as compared to end-line. 36 The project has directly contributed to the fact that CCIs have made improvements in all areas covered by national standards. The areas where average scores have improved the most are in practices related to “child resettlement and alternative care provisions” which is also the area where CCIs scored the lowest at baseline. This data is captured in the project’s monitoring data from the assessment that scores CCIs across seven alternative care standards. The improvement is made from an average for 1.6 scores (out of the maximum 4) to 2.87 scores at endline. The area that comes as the second-best improvement area is in “post placement support.” This was the second poorest performance area at baseline and is now the best performance area at endline (average score across all assessed CCIs moving from 1.84 at baseline to 2.99 (out of a maximum of 4 scores) at endline (see also “Annex V: Statistical Tables”, Table 22) Figure 3: Improvements made in CCIs on alternative care standards: baseline vs. endline KII with policy level stakeholders, review of policy documents developed by CCIs, and case file reviews, could further confirm the changes in practices and operations in CCIs which are summarized here: Between the baseline and end line evaluation, there are significant changes in the awareness and attitudes of CCI around the national alternative care framework and their role in the alternative care system. This mindset change has mitigated the risks of resistance to the change amongst the child care institutions that was observed at baseline. The following quote from the Chief Administrative Officer (CAO) in one of the study districts reflects well a sentiment that was expressed by various respondent groups across all districts: “Some child care institutions were hesitant to this project in the beginning but later they realized that it was for the good of children and in the best interest of children. They understood that they need to comply with alternative care standards. CCIs attend now most of our meetings and are working to improve their standards of work.” Furthermore, CCIs across all the study districts have made 37 significant progress in improving their policies. The evaluation team had access to copies of human resources policies, child protection policies, reunification policies, and emergency procedures, which had been developed since the beginning of the projects (although not in all CCIs). Some CCIs have established management committees, while other CCIs had made improvements to their facilities to better comply with policies. Across all the districts, many of the CCIs visited have better human resource capacity as a result of hiring social workers and/or recruiting more qualified staff. Some CCIs had changed their management. Improvements in human resource capacity was also to a large extent attributed by stakeholders to the skills building activities of the DOVCU project involving management and staff. Across all study districts, the evaluation was able to confirm that many CCIs are implementing better gatekeeping procedures, as a direct contribution of DOVCU’s capacity building activities targeting CCIs. Interviews with different respondent groups (PSWOs, CDOs, CCI directors and social workers, TAC members) confirm that these improved “gatekeeping procedures” often involve significant changes to admission and reintegration practices, and to what many CCIs now establish, through case planning, as the objectives of children’s stay. The evaluation also documented that not all CCIs changed their views around their role in the alternative care system, and the level of engagement with the project, and the changes made in practices of the CCI varied significantly from one CCI to the other. The project’s own monitoring data from the end line assessment of CCIs, confirmed that in all CCIs visited, there is room for improvement in at least one, and in many cases in several standard-areas and most CCIs are yet to receive their license to operate. For example, even though many CCIs have strengthened their human resource capacity as compared to baseline, there are still significant differences between CCIs. If at baseline there were many CCIs that did not have any social workers at all, now there are more CCIs that employ staff to perform social work duties, such as being involved in tracing, reunification reintegration activities, and in providing support for the reintegration of the child, and in developing, reviewing and updating case files. However, qualification levels of the social workers now available in CCIs, vary a lot. From case file reviews and discussions with CCI-social workers on the follow-up they provide to specific cases, it was evident that many social workers who are available in CCIs do not have advanced skills in, for example recognizing delays in child development, or signs of stress in children who have been reunified. The attitude of the director and of founders/funders of the CCIs seems to have been of significant importance to the extent to which each CCI embraced the project and the change it proposed. An attitude to this change also seems to be heavily influenced by the funding available to support changes in practices, much of which comes from outside Uganda. As a result of DOVCU project activities all targeted remand homes and the national rehabilitation center have reunified children as per SOPs, and there are some improvements in professional practices (also confirmed by district stakeholders), for example in the use of individual case files based on SOP tools, and self-reported skills of staff to do tracing and engage in community dialogue on the issues of children in contact with the law. However, many of these changes are perceived as unsustainable (see also sustainability chapter), and for example case files observed still have a large room for improvement. In two of the remand homes visited SOPs were no longer in use because there was no budget for paper to print forms. The staff or volunteers that DOVCU put in place in these two remand homes were no longer there. In the third remand home most of the case files inspected were incomplete with empty or only partially filled forms. SOPs. 38 The service provider working with children on the street increased their knowledge and skills to provide services to reunify and support the reintegration of street connected girls. All in all, DOVCU trained 90 CRO staff on SOPs and data collection tools relevant for their reintegration activities and through the project and through this, supported CRO to reunify 204 girls from the street. Girls identified in street situations by CRO, were provided with case management services, skills training and income generating activities (IGA), literacy and numeracy classes, counselling, nutritional and health services, and support to re-enter school. The evaluation documented knowledge and skills amongst staff to provide services on trauma counseling & child preparation for reintegration, life skills for girls, psychosocial support and counselling, monitoring, evaluation and reporting of activities (e.g. to the OVC Management Information System (MIS)). The evaluation also documented a greater use of evidence in their case management practices when admitting and reintegrating girls from the street. A review of case files indicates that CRO is using the case management tools from SOPs and are particularly performing well on identifying child needs and risks through documentation captured in these tools. Changes in awareness, knowledge and technical skills amongst district, sub-county and community professionals As a result of DOVCU, awareness raising, skills building and training activities, the evaluation documented a number of changes in awareness, technical skills and professional practices across districts. These results are summarized below, and further detailed in Annex III: Lessons learned on the project’s work with different professionals: The project has contributed to a greater knowledge amongst professionals at all levels, of risks, response options and of the roles of various professionals in the response. A member of a District OVC Committee illustrated this finding with the following reflection: “The responsibility of all actors is clearer. Child protection is not just about being good Samaritan. Now we know to ask about the PSWs and what they can do to help in child protection issues.” The evaluation furthermore found that the project contributed to strengthen the technical skills of staff in MoGLSD and district staff (mainly PSWOs and CDOs) to implement standard procedures for inspecting CCIs and in various professional’s skills to facilitate processes for reintegration of children from CCIs, from the street and from remand homes. Stakeholders across various respondents’ groups and districts also converge around a view that the project’s training and skills building activities have improved professional skills of PSWOs, CDOs and PSWs to specific provide services to families and children. According to the CAO, the highest officer in rank in one of the districts, “PSWOs and CDOs have become more serious” as a result of the DOVCU project. This view, and the importance of DOVCU for “skilling up Uganda” was echoed also in other districts. All of this has furthermore produced important changes in professional practices. District staffs are now more regularly and systematically involved in providing oversight and applying stricter gatekeeping procedures for children to CCIs: “before when we were receiving almost every child for whom orphanhood was claimed, we just sent to CCIs without establishing the facts on ground; without social inquiry.” The evaluation evidenced that DOVCU has contributed to better identification of vulnerability and risk and a better functioning referral system between the four parishes, two sub counties in each district. Across all six study districts for the evaluation, a wide range of professionals and stakeholders express the sentiment that this improvement is linked to changes generated by DOVCU in the awareness, skills and knowledge on child protection risks of community structures and PSWs, and of sub county professionals and PSWOs at district level. This is exemplified by one DCDO who noted: “Yes, especially now I will look at the para social workers. They can identify child rights violations and report them.” PSWs the other hand, described how the process of visiting, interacting, and speaking to families altered as a result of their training, and discussed this as an impactful shift as a result of DOVCU. 39 One area where the project has had mixed results is in the area of case management. The evaluation noted a number of challenges related to the case management, which have persisted since the baseline evaluation was undertaken. While many improvements have been made to community level identification and referral processes, the evaluation team tried to get access to review case files of reunified children and found that the process seemed poorly documented and as one professional noted: “There were a lot of tools – so many tools – it was time consuming and there is some duplication.” More details are provided in “Annex IV: Lessons learned on case management”. Evaluation question 5: To what extent and how did the project contribute to implementation of national care reform measures? DOVCU has contributed to sustained awareness, visibility and priority to alternative care at national level but not to significantly strengthening it. At the time when the project started in 2014, visibility of alternative care was already high on the MoGLSD agenda. A policy framework had been adopted in 2011, and an alternative care unit had been established within the MoGLSD. At central level, several stakeholders interviewed for the evaluation are suggesting that the project did not exploit lessons learned to the extent that it would have been possible and useful, to elevate the alternative care agenda further as a national priority. This quote from a national stakeholder illustrates: “DOVCU was a big project but I do not think that the Government has spoken about the project in an equivalent size as the project was.” DOVCU has contributed to some extent to changes in district policies and priorities. For example, it is felt that visibility and priority of alternative care in the 12 project districts as compared to the baseline situation has increased. The ability of higher-level officials in the six districts to speak about the priorities of national policy (the ACF) is evidence of this and they testify about the learning the project has generated for them. As one high-level district official reflected: “It’s really been a big learning for our district – for all those who work with children.” The project has contributed to a significant change in mindsets, at all levels, around the role of child care institutions and the importance of family care for children. This is evidenced in interviews with a wide range of district stakeholders who testify of the fact that, at the start of the project, many stakeholders saw CCIs as a primary source of relief in dealing with vulnerable children. Today they rather speak in favor supporting families so that children can stay at home. The following quote from the former director of a CCIs that closed in one of the districts, illustrates: “I got some training from DOVCU, they visited together with the Elders, they trained me on child protection, filing and documentation. What touched me most was the need to take back children to home so that they can grow up appreciating their cultures, in their communities.” Stakeholders see that it is not necessarily in the best interests to place children in CCIs and may be looking at the services provided with more critical eyes. A senior official in one district reflected on this: “Although those institutions do help, they had their intentions like maybe they are business oriented and also, they are biased with their religion (…) For me I think it is of importance that we take back these children to where they come from.” In the same district another high￾level district official agreed that: “Instead of helping children in the care institution, it is better to strengthen the family so that they are able to look after their children as well and mitigating those factors which normally cause children to run away from their families.” In yet another district, another high-level district official reflected on how the change in mindsets had changed behaviors of stakeholders at all levels: “In institutions there are schedule and programs, in the community you must adapt and learn from the society and become a well-working citizen. Mindsets are changed now. DOVCU gave support and training that benefit the success in the community. Also, families not supported by DOVCU have learned from DOVCU-families good practice, they adopt good examples (...) Our mind-sets have changed. We have now started to go slow on allowing institutions to 40 institutionalizing children (…) In the past it was an immediate and easy solution. Now community structures are first investigated and prepared for the child to hopefully be able to move back after a short-term stay.” There is now greater involvement of higher-level stakeholders at district level in activities related alternative care and child protection. At the beginning of the project, higher-level district stakeholders were rarely involved in any activities related to alternative care. Today they are reported to provide oversight of activities related to alternative care, and also to be directly involved, for example on the TAC or in CCI inspection visits. For example, in three of the six study districts for the evaluation, it is the DCDO who takes on the role of chairperson of the TAC. In another district, the assistant CAO had been involved in visiting CCIs for the first time through the DOVCU project. In yet another district, district stakeholders clearly associated the work of DOVCU with the district policy improvements. A new district Child Protection Policy has been adopted too in this district to, as the CAO put it: “complete the gaps in national policy, especially for example around children who are out of school and child labor.” The policy provides direction for how alternative care for children should be implemented in the district and that CCIs will be a last resort. RELEVANCE The relevance section aims to unpack whether specific needs of beneficiaries’ match what the project offered, to assess whether there were gaps in interventions gaps in interventions and whether there is any evidence of potential adaptations needed to strengthen the project and if these were headed. The evaluation included four relevance questions. Evaluation question 6: To what extent and how did the project’s integrated model, which links family strengthening, child protection and economic strengthening measures, offer examples and approaches for expansion, adaptation, and/or replication? What combination of interventions within this model could be adapted expanded? The project’s integrated model, which linked family strengthening, child protection and economic strengthening measures was relevant in Ugandan context. It can be usefully be replicated in other districts of the country. However, lack of access to quality education and health services, as noted in the baseline evaluation of DOVCU, are key drivers to separation for many children, and an expanded model of the project could usefully include stronger elements, especially of the financial support component to bridge the gaps that remained for beneficiaries of support. For example, cash support could be provided for a longer time period, with larger amounts. In addition to each of the components within the integrated package for support, there are some examples of approaches used by the project that the evaluation documented as particularly relevant for expansion. For example: • The combination of the tools used (FSVI and CSI indexes) and the process designed by the project to identify households for the project’s prevention component was perceived by many stakeholders across districts as very relevant and useful. This process, respondents felt, allowed for a fair and transparent identification process of the most vulnerable families. The process combined community assessments, the use of FSVI to objectively categorize vulnerable households and to identify programmatic areas, and qualitative information gathering through community discussions to triangulate information and identify households. A CDO in one district reflected on the relevance of this process with the following words: “We did community assessments and the community participated and identified the most vulnerable. The project also had their own indicators, so I think that the most vulnerable household were chosen,” and in another district the CDO praised the tools used by the DOVCU project to support these assessments: “In the project we had the right tools and they were used properly this contributed to success” (CDO). 41 Some stakeholders went as far as suggesting that this process had helped communities accept the final selection of households to be supported in such a way that it prevented jealously between community members: “Community meeting was also about showing how they (households) were selected because everyone wanted to be supported and there would have been problems” (CDO). The limitations of this approach were that the project, because of its own funding restrictions, was only able to support a selection of the families categorized in the destitute category with cash transfers while the difference between a destitute household and struggling households in many cases was very small. In addition, vulnerability in a country like Uganda can be fluctuating with time and therefore, as one respondent stressed, “there are those that fell into vulnerability after the start of the project” (CDO). • Another example of an approach perceived relevant for replication in other districts is the step by step sequencing in of different types of support to destitute households. Many stakeholders described how this approach had allowed families to progressively address the root, underlying and immediate causes to their vulnerability. Stakeholders expressed how it had been relevant to first provide counselling and advise, for example to address alcohol abuse, followed by skills building activities to strengthen financial literacy and planning. Cash transfers were provided after this to stabilize the household economically. If necessary this was followed by a second and third transfer to allow the household to plan economically for their future. Many stakeholders reflected on the fact that, had cash transfers been provided too early in the process, families would not have been able to make good use of them. Once provided, cash transfers combined with the training on how to grow family income, and the setting up of VSLAs, gradually sequenced in support to families in a way that is perceived as having allowed households to gradually and incrementally address the factors that contributed to their destitution status. As a CDO and a PSWO reflected: “The integration of all activities is key, including the communities. It was a very good design of the program” and “It is very important that families are first trained how to use the money before the money is received.” Stakeholders made similar reflections regarding the reintegration of children. First the project addressed family attitudes and then, if needed, they were provided with cash transfers: A CDO explained: “The most important were the packages that were targeting to changing the attitudes of the care givers to enable the children to stay and be reunified. The cash came second, so parents have understood that there are small things that can make children run away. Skilling parents on parenting better were the most important (…) You can give packages, but if you haven’t changed the mind-set and got some new skills the package will not do much difference in the long-run.” The elements of the integrated package, the process and approach taken to its implementation allowed for flexibility and attention to the individual circumstances prevailing in each family, or as one PSWO explained: “When these households were assessed, they were linked to para social workers. Each para social worker was allocated a number of households to work with. They had household plans for each of these households, they identified the problems within each household and how they would support households. They gave them cash transfers to help them address the immediate needs in the households (...) you find that somebody had received 200,000. Now they are in position to buy some food, to buy a mattress, to buy a blanket and to start up some, maybe like charcoal selling. (…) and they were not given only one cash transfer, some of them have received three cash transfers. Then also they were sensitized on parenting, on how to they would also get to other service providers (...) so what they would not get under DOVCU, they would do under (another project).” 42 Evaluation question 7: To what extent and how was the integrated approach model, which links family strengthening, child protection and economic strengthening measures to maximize prevention for children at risk of separation from family, and support those returning to family-based care, relevant and sufficiently adapted to the communities where the project operates? The integrated approach is assessed as relevant to address different categories of risk that are present in the communities where the project operated. The financial package, parental trainings, community groups, and home visits focused on actively reducing risk for separation. The financial package increased the chances of children attending school, was perceived to improve nutrition of children, and reduced family stress caused from a lack of income. Parental trainings helped parents know effective ways to speak to their children, diffuse conflict, encourage children to stay at homes, as well as taught them about community structures they could call on for assistance. Community groups were seen by beneficiaries and community stakeholders to create changes across the community, and specifically decrease alcohol consumption, the rate of domestic violence in households, and increase chances for savings. Furthermore, the integrated model was perceived by respondents to address the primary risk areas for child separation. Respondents revealed an appreciation for the parental trainings and an active utilization of the information gained in trainings. Interviews revealed participation in community groups and a resulting decrease in alcohol consumption and conflict within families; and both para social workers, caregivers and community members perceived that home visits helped support family cohesion in a number of areas. Even though the evaluation cannot assess directly if the project prevented the separation of children in the targeted households, respondents believe that the package did help families in their communities to support children to stay in families. Furthermore, there is no evidence from this evaluation suggesting that any child from the targeted prevention households separated from his/her family during the lifetime of the project. This result was attributed to increased school attendance, better nutritional opportunities, and decreased violence in the household. People also believed that the vocational trainings gave children the tools to start making money. One women described the shifts taking place in the community where she lived and were DOVCU focused its support: ”There used to be many children at home without going to school and also domestic violence was at its peak but now those cases are rare, and many children have been enrolled in schools and they are in good health. I just urge them [DOVCU] to gather more energy and resources and they continue with helping us and also we the community members will have to put more effort in utilizing the funds they have provided us by making use of the land and supporting the children with food and other requirements that may be needed.”.” Findings from the qualitative component of the evaluation suggest that the integrated package did have some gaps in services provided, as perceived by the beneficiaries and community stakeholders in the areas where the project operated. The most reoccurring comment about gaps in services was about housing and the fact that people felt that housing was an issue that went unaddressed by the project. A caregiver in Luwero exemplified: “The way we see it, we are still very poor…we are still poor because if you don’t have your own place you remain low.” In addition to housing, respondents said they would appreciate specific support such as livestock or seeds for planting. When asking for farm animals and seeds, people made these requests because they felt it would lead to more sustainable ends: “give us cows, goats, so that we are able sell and be able to pay with fees, without having to beg or wait for someone like World Vision or (DOVCU) to support us.”.” Most perceived gaps in services do not relate to new components that need to be added but expanded ones. Respondents highlight the precarious nature of their environment that continues to 43 put households at risk of separation. They stress that it will take longer to move away from this situation than the current project’s time frame: “I think they need to support us until we are stable enough to start an income generating activity. They also need to train us on IGAs first, and only then give us money when we know how best to use it in business” (Caregiver, Gulu). This quote is important as it stresses a key point highlighted by respondents: they received training, and feel they learned from training, but the cash transfer was in many cases spent on immediate needs. Respondents also remained concerned about the vulnerability of households where one or more parent was disabled. While those with disability expressed appreciation for the financial support provided by DOVCU, they also expressed that it did not meet all their needs. For the children returning to family care and families receiving children back from CCIs and remand homes, stakeholders provide mixed reviews regarding the relevance of the support provided. For children returning from CCIs, specifically management and social workers of CCIs, CDOs and PSWO sometime felt that the preparation, both in terms of dialogue with families and specific material support to families could have been better to build a solid foundation for the continued reintegration process. This was especially the case for children returning to parishes or districts where DOVCU was not operational. Such preparation also varied a lot depending on where the child was reunified from. In terms of the specific support provided by DOVCU to families receiving children for reintegration, the material supply items included in the reintegration package were very appreciated, especially the mattresses. Several policy level stakeholders also felt that DOVCU very relevantly extended support to other children who were already in the family in the households where children were reunified. However, while the project’s interventions very relevantly addressed several of the perceived push factors to separation in the parishes where it focused its prevention work, families who lived in parishes or districts outside of the DOVCU focus areas did not benefit from similar interventions. Furthermore, while the project overall addressed push factors in families that lead them to abandon children, it did not address directly the pull factors, especially unavailability of quality of education close to where the child lives and special health needs which lead families to bring children to CCIs. These needs were left to be addressed by CCIs which in many cases they did, but also in many cases did not. Specific gaps were perceived when the CCI could not continue supporting the child’s education and health needs. As one policy stakeholder explained: “For the reunified children that will continue getting support especially in terms of school dues from the CCIs that kept them, I think it is promising and there will be permanency. For those that have to depend on UPE for education, and also place in homes that are not well of socio-economically, the risk of separation is still a reality.” Furthermore, for the children who were reunified from remand homes the reintegration package and support were perceived as insufficient: shelter, school fees, specific support such as catch up-classes, was felt to be missing to bring these children back to school. Remand home staff expressed a certain level of frustration since the project did not “meet expectations or address the reasons why children end up in remand homes (…) Our cases are a manifestation of the lack of community programs.” Finally, for children returning to family care, needs are not only linked to socio-economic situations, but in some cases, children need legal support. This, was identified by some stakeholders as a gap in the DOVCU project. Evaluation question 8: Based on what are found to be the most common risk factors to separation, to what extent and how did the project manage to address these in an appropriate manner, including for boys and girls? The evaluation assesses that the various project activities align with, and arguably successfully identify community needs, though may not able to comprehensively address these. The project focused addressing financial and other vulnerabilities of households, some of which was tackled through the financial support package, and there are some positive findings in terms of the financial components potential to help recipients cope with financial vulnerability. Yet, the main question is with regards to 44 tackling long-term financial vulnerability issues, and the extent to which current indicated improvements in access to basic services may be affected following project end. Project recipients live in highly volatile contexts, and though VSLA groups are an important step towards insurance against shock, in small and close-knit communities where multiple respondents are likely to experience shocks simultaneously, VSLAs might have difficulties to be sustained if multiple respondents are affected at the same time by challenges that limit their ability to pay back. These types of root causes to separation such as an economic climate that offers limited financial opportunities, or highly volatile contexts, may be beyond the project’s control, however, it is important to consider the findings with this in mind. Parenting skills training appears to have provided caregivers with enhanced ability to cope with stress and overcome some of the psychosocial barriers to being good parents. Yet, respondents highlight the need for refresher trainings. This was particularly the case in communities where parent groups are not as active. Respondents also link the reduced need for parental groups and support with improvement in financial vulnerability but highlight that if the financial situation gets worse again, stress in the household might increase making it more difficult to “continue to be a good parent” without support. There are furthermore some indications that the project did not disaggregate or differentiate between the risk factors for boys and girls in being separated. This was particularly the case for children placed in CCIs. TAC members in one district reflected on this with the following words: “When we look at families where children are received, families are more keen to receive children who are older and are more keen to take girls than boys. Because boys should have land.” However, project monitoring statistics do not indicate that this affected the project’s ability to reintegrate both boys and girls to an equal extent. Evaluation question 9: How did the political environment and context of the project influence operation and impact, and to what extent was the project able to mitigate risks and leverage new circumstances/opportunities which arose? The project is assessed as very relevant, both in terms of its alignment with policy environment, community values in which it operated, and in terms of its timing. It is aligned and pilots some of the provisions of the national policy on alternative care, and many stakeholders also referred to the fact that family care, and thus preventing separation and supporting the reintegration of children, aligns very strongly with traditional community values which are very favorable of children growing up in family environment. There was furthermore overwhelming agreement that “the project was very timely”, and in some districts even addressing an acute problem: One PSWO reflected on this from the perspective of how the system before the project had failed to get alternative cares priorities right: “This project came at a time when we needed it most. In Uganda, the issues of institutionalization of children were very rampant. This had become a way of surviving, homes for children were coming up and many children were going there. Most children even went to these CCIs without knowledge of PSWO. It was timely because there was some kind of disconnect between CCIs, PSWO and other stakeholders at the time.” in another district, another PSWO reflected both on the project’s timeliness in terms of the acute needs because of increasing abandonment rates, and because of the inadequacy of CCIs in that context: “There was a time when they abandoned babies, there is even one month we got about five newborn babies abandoned. It was that month of April, and then we were like “Oh, God, what is this?” (DOVCU) came in at a time when there were so many children’s homes that were not meeting the standards. And when they came in there were so much needed. When they did these assessments in these homes, you would go to a home and it was so horrible you would not even want to hear anything (about what is going on there), you just say close now.” (TAC member) DOVCU was relevant to address the attitudes and types of resistance that prevailed amongst some CCIs at the inception of the project. In some of the project districts (e.g. Kampala, Kamuli and 45 Luwero) there has been a greater increase in the numbers of child care institutions prior to the project’s start, than in other districts (e.g. Kasese and Kabarole). Nevertheless, in all districts, the project addressed and has managed to break a negative dynamic around the expansion of CCIs and also brought the public eye to the bad practices of some of the CCIs. A PSWO from a district with previously high prevalence of CCIs reflected on this: “At the inception, of course one of the challenges was that, the whole attitude was that they are going to close our institutions and the issue of child care institutions is more personal than them thinking they are complementing the efforts of government - This is my business, this is what I want to do, these are my children, I took these children from the community, you are not supposed to take them away, their parents have given them to me, these are poor children - There was that kind of resistance.” Over the life of the project there were some contextual factors that may have influenced the impact and the effectiveness of the project and its approaches and support. For example, food insecurity is an important push factor leaving children to separate from their families, or parents leaving their children to child care institutions. Drought may have contributed to some families not being able to progress from its destitution level as much as the project had hoped for. There were several stakeholders who referred to the absence of strategies in DOVCU to mitigate the effect of drought on families, as an important weakness of the project. In one of the drought affected districts the CDOs reflected on this situation with the following words: “All the people who benefited appreciated but want more. If there is drought, the household need to learn some water saving techniques to allow them to irrigate the crops also during drought..” The project very relevantly and successfully leveraged existing formal system stakeholders to get further involved in alternative care. This was a specific strategy employed by the project, which was highly appreciated and referred to as very successful across districts. Stakeholders were mobilized from all levels, national, district, sub-county down to community level, formal and informal structures, public and non-public actors (such as CCIs). As already reported in the impact section, formal district stakeholders referred to DOVCU having “activated their mandates”, community child protection structures and para social workers were connected with formal district structures in referral trajectories and felt more connected as a result of the DOVCU strategy, resulting in a significant capacity strengthening of the stakeholders to be involved in the alternative care systems. The step by step approach of involving CCIs in supporting the implementation of the project’s second goal, namely the reintegration of children, was often referred to as very relevant. Identifying and assessing CCIs in the district helped illuminate the conditions under which children were living. The training and coaching of staff of CCIs progressively helped mobilize these stakeholders for involvement in the project, and also to mitigate some of the fears about being “closed down”, or about a too rapid reunification of children. If there is anything in this strategy that several stakeholders felt should be re-considered for any future similar project, it is the “stick approach” that especially CCIs in some districts felt in some districts. With this they meant that some assessment teams, it was felt, came with a “threat” of closing the CCI if they didn’t comply. This put the CCIs on the defensive, and in some cases made CCIs feel pressured to start reintegrating children before safeguards had been put into place. One national stakeholder made the following reflection, in relation to this: “What should be taken into account and changed in all this, would be to reduce the stick approach, or to have stronger national and district capacity ready to facilitate closure of really poorly functioning facilities, be clear about the accountability framework and ensure formal stakeholders (national and districts) are strong enough to quality assure the process of reintegration, and again monitor and act upon cases that did not work well. A plan needs to be in place for that.” The project’s interventions within remand homes were felt as less relevant to address all the issues that are leading children to being placed in remand, often for petty offences and often over-staying in the system. Even though the project mobilized many more stakeholders than originally 46 planned (incl. magistrates and police), across more districts than the DOVCU-coverage, the reunification and support to reintegration of children from remand homes should be more relevantly addressed in the context of broader justice system reforms and support the development of understanding for, development and implementation of strategies for diversion in the first place. EFFECTIVENESS This section will discuss the effectiveness of the DOVCU project. Effectiveness is understood as whether the right activities were implemented with the quality required for the project to achieve its goals. The evaluation was guided by five effectiveness-questions, which are reported here: Evaluation question 10: To what extent have the targets of the project to support 43,000 children in vulnerable households and 2,000 children to be reintegrated been reached and how were they achieved? Regarding the project’s objective 1: To prevent separation through supporting 43,000 children in vulnerable families, the project’s monitoring data does not allow for a conclusion on whether this target was fully reached. The target to reach 43,000 children in vulnerable families included an assumption that the project would reach children both directly, through supporting specific families, assessed as “at risk households”, and indirectly reaching children from many more households through various community group sessions and volunteers which were trained to do outreach work. This component of the project was implemented in 4 parishes per district15 and included for example VSLA, parenting groups and post treatment groups which were focused in the parishes selected for the prevention component. The project reports that: • 3,878 households were trained in business skills, savings and credit investment; • 5,812 households were visited annually per parish by CDOs or PSWOs; • 3,617 households participated in parent’s groups support sessions. It is likely that several families benefited from more than one of these activities and are therefore counted in more than one of the output indicators. It is not known how many children were reached as an extension of these activities. More importantly, the project supported directly a total of 2,234 households to reduce the specific risk identified in these households that was identified when the project at baseline assessed a total of 3,297 households using FSVI. Of the directly supported households: • 1,252 were assessed and categorized as destitute households at end line; • 974 were categorized as struggling households; and • 8 were categorized as growing households at end line. It is not possible to establish directly if the risk that the family would disintegrate reduced in the 2,234 directly supported households, as a result of project activities. However, as reported in the impact section, the support provided by the project was matching most of the identified needs and the causes of separation. Many stakeholders and beneficiaries believed that the direct support provided to families, especially the cash transfer, was of critical importance even though the amount and length of the project was perceived, in many cases, as insufficient for families to stabilize economically, and to change behaviors toward significant and sustainable coming out of household poverty and improving upon family disintegration. Testimonies from some supported families and district stakeholders suggested that in some cases, poor uptake of project support could be linked to size of families, suggesting that families with many children took longer time to stabilize or that supportive or non￾supportive community environments could influence to what extent families could thrive as a result of 15 The 4 parishes/district were located in 2 different sub-counties within the district. 47 the support provided by the project. This suggests that DOVCU support to prevent separation served as a trigger but did not yet or alone prevent separation. Regarding the project’s objective 2: To improve the wellbeing of children through supporting the reintegration of approximately 2,000 children, the project reunified 1,743 children who have been found to stay in the place they were reunified to at least 6 months following reunification. Of these the qualitative component of the evaluation as well as triangulation of different methods established that: • most children are believed to be living in families; • 1045 children who were reunified outside of the 12 DOVCU districts were not reassessed or visited by the project at end line and the extent to which they are to be considered as reintegrated is not known; • some children (un-known number) have stayed with the same service provider (CCI) that transformed into a boarding school, or the child moved to boarding school facility provided by another service provider. The director from one CCI explains in the quote below how his CCI transformed into a boarding school where children could stay on for education, while going home for holidays: “(name) first was an orphanage home at the same time having a primary school. As such all the beneficiaries were boarding scholars including our secondary students. Since our involvement with (DOVCU) and the probation officers making sure that the CCIs follow the Ugandan law which changed, I think in 2012 to make orphanages just to be a short￾term thing, whereas most of us were operating like a long-term home. But that was not the Ugandan government’s aim (…) and then officially from the beginning of last year, we were only a primary school, we are a boarding primary school, we are registered and we are no longer registered as a CCI.” In interviews with reunified children, some children described how they had not actually left the CCI while the corresponding caregivers said that they had been instructed that children would now live with them, but the CCI had let them know the child could stay in the facility for education but the caregivers would now have to pay school fees. In these cases, as explained by some caregivers and stakeholders at the CCI, children had previously spent holidays with their families, even when admitted to the CCI and as such children perceived very little to have changed in their lives. Caregivers on the other hand felt confused that they now had to pay for children to attend what had previously been free. The wellbeing of the 1,743 children that the project has reunified seems to vary and while most have remained in the family where they were reunified, reintegration is for many children not yet fully achieved. As already illustrated in the impact chapter reunified children perceive their overall care and support to have increased following reunification, and many reunified children say that they are happy to be at home. Yet, children who have been reunified from the street or remand homes, mostly consider their food security to have improved while the majority of children who were reunified from CCIs perceive their food security to have been better prior while in CCIs. Furthermore, while the majority of reunified children state they regularly attend school, children from CCIs perceive their access to services as more precarious following reunification, and for this to add stress. The process towards reintegration of children from CCIs and the transition to family-based care was challenging for some children and the documentation on the follow-up provided to children who experienced stress during the process of reintegration has not been documented and completed properly to allow for a systematic review of this process.16 Respondents state how 16 As per DOVCU information, case plans existed for all reunified children and case files for placement follow up existed at the project’s satellite offices. However, in spite of requesting and looking for such files, the evaluation team could not gain access to a sufficient enough sample to be able to state that follow-up had been systematically documented. The evaluation team reviewed a number of casefiles at each CCIs, PSWO office visited and discussed the documentation in FGDs with TACs, CDOs 48 children are used to certain norms and behavior in the CCI, and in particular with regards to responsibilities around the house. A caregiver in Luwero explained: “His situations changed cause he was used to the conditions of “a home”, in “a home” a child grows up without knowing that he can be told to wash dishes alone, cause he is used to doing things in a group, he expects food to be just cooked for him, also he expects not to work such as digging and welcoming visitors since he stayed in “a home” he didn’t know but now he learnt and takes responsibility when left home alone” This sentiment is reflected across respondents, with the idea that children who have previously been living in CCIs often are perceived as “spoiled” and are not used to carry out their chores. This is reflected in some of the interviews with children but is largely dependent on the conditions in the CCI from where children were reunified, and whether children were used to carrying out chores from their time in the institution. Some children bring up ‘digging’ at home as something they do not like about being back. Also when it comes to nutrition, children’s subjective well-being is largely dependent on where they previously were. Children who were previously in CCIs with regular meals and fairly diverse foods complain that, although they get enough food at home and do not go hungry, they are not happy that it is always “the same food.” Caretakers acknowledge this, with one caretake in Luwero stating: “he first felt bad [when coming back home]. They used to take care of him well in the ‘home’. When he came back he would sometimes now take tea with no sugar because we couldn’t afford sugar. Also, he didn’t find the things he used to eat in a ‘home’ so it hurt him at first and he ended up sick, but later he got used to it.” It is important to acknowledge that while this subjective well-being is real, and important, and can affect children’s emotional well-being, it does not necessarily mean that children are worse off after reunification.17 What is important to note, however, is that children and their caregivers may need to be counselled, may need referrals, and may need other forms of help to help children get over stress or negative feelings that they may experience as a result of the changes they’ve lived. From case file reviews it was very difficult to assess to what extent they had been given such counselling and referrals. While professionals across all districts explained that follow-up with reunified children was done, case records provided poor evidence of this statement. Furthermore, it was also clear that within the sample of reunified children interviewed, the team met with and interviewed several children who had been back home for a long time (more than 6 months) and were still having difficulties in adapting to their home environments. Some parents describe the reunification process as rushed and complain about limited support to have been provided to support the process of reintegration of children with their families. The following comment from the caregiver of a child who was reunified illustrates: “I am not sure about the kind of support, but I think they were giving money and even food but am not sure of the details because it was the mother who was receiving everything and I was always away from here. The program is only targeting (name of the child) and not the other children in the home” (Gulu district). Coordination for emergency situations and follow up was a concern to some families due to transportation and communication limitations. Staff in remand homes felt that the process towards reintegration of children from remand homes was a difficult, to a large extent due to persisting stigma in communities against children who have committed a crime. This was confirmed also by community members who stated having difficulties accepting children back if the child had been accused of murder or rape. For example, parents and para social workers. The statement regarding documentation therefore is informed from the case files reviewed, observations on storing practices of case files, and the process and difficulties in the process of obtaining access to case files. 17 This evaluation was only able to capture children’s own summative experiences of their wellbeing status. Project monitoring data measuring quantitative changes are based on changes at family-based care level, and as such measure changes following reunification rather than a comparative analysis of child well-being prior to, and following, reunification. 49 would not want their children to play with them because they feel their ‘bad’ tendencies will impact their own children’s behavior. As explained by women during a FGD in Kabarole: “It depends on what crime the child has committed. If the child has stolen or such things you may want him to go away. But if he has raped a girl, I cannot allow my daughter to be near this boy because I will still think he can be having the habit hidden there. He might do it again. So, I cannot allow my child to be near him. But let’s say if he stole a sugarcane or maize and got scared and ran away, when he comes back I can allow my children to play with him. It will depend on what crime the child has committed.” In was felt by remand home staff that the project could not do enough to help overcome this challenge. The reintegration process and its effect on children’s well-being Across all districts, the selection of the children to be reunified into family care was mainly done by the service provider, namely CCIs, remand homes, CRO. While DOVCU trained CCIs, CRO and remand homes on approved home regulations guidelines which included criteria for identification of children for reunification, CCIs report that they applied the criteria they found the most useful, for example: children who wanted to go home or children of specific ages; some reunified only “older” children on the basis that they could be consulted in the process; and other cases CCIs chose to reunify “children”, who were turning or had turned18. In one children 10-15 years of age were selected, and in yet another only younger children were reunified because younger children were found easier to reintegrate. One CCI reunified children who had maintained contact with their families or children who had expired care orders. Generally, children with disabilities were said to be excluded from reunification processes, according to CCI directors and social workers because these were considered as more difficult cases to reintegrate. For children in remand homes and children coming from the street the selection was more targeted. It was the most vulnerable amongst the children who had completed their sentences and were to be released or those who were found to be not guilty but rather incarcerated for essentially being vulnerable and on the street, who were supported through the project. CRO specifically targeted girls to be reintegrated with the project’s support. The quality in the support provided to facilitate reintegration of children depended on where the child was returned from: remand homes, CRO, or which CCI the child came from. The reason for that was that each institution had varying capacity to support a reintegration process, monitoring and follow up of children, and DOVCU’s main support to the process was training, coaching and skills building activities. The project could not and did not, with the exceptions of volunteers recruited to remand homes, bridge any of the institutions’ gaps in social work staff capacity or funding for transport to support tracing and follow-up monitoring visits to children who had been reunified. The ability of CCIs to support a process to facilitate the reintegration process depended on a number of factors. This included for example the perceived urgency of the CCI to proceed with reunification. Some CCIs proceeded very quickly with a rushed process that jumped some steps. For example, in one district a CCI closed and all children were sent home from one day to the other. Other CCIs took their time and embarked very progressively and carefully in the process to reunify children with their families. Quality of the process also depended on the willingness of founders and funders to financially support the recruitment of social workers to support the process and the turn-around of social workers. Some of the visited CCIs reported high turn-over of social workers something which reduced the sustainability of skills gained through training provided by DOVCU. According to CCIs, it also depended on whether funding was available to the CCI to support the preparation of families before the child was sent back, and on whether funding was available in the CCI to continuing support 50 the child’s education or special health-needs once back in the family. A PSWO in one district reflected on this with the following words: “There was one case when we started the resettlement and the conditions in the Household (HH) you could realize that these people are very poor. But of course, you are bringing this child and the child has been used to this kind of life in the CCI, and the child is now back to this HH… I started to question how this child is going to cope up. But thank God that some of our CCIs became so cooperative that during the process of reintegration of children that were in the CCIs, we involved their donors so what has happened is that most of the children resettled have been placed on an outreach program from CCIs and have been receiving school fees.” Stakeholders refer to a good quality reunification process, preparing for the reintegration of children, as a step by step process, starting with tracing and preparing families to take children back already at the time of admitting the child to the CCI. A TAC member in one districts described the process with following words: “We begin when the child is admitted to the CCI. All Children going into CCIs should go through PSWOs, LC1 and district officials. Immediately after admitting the child the social worker starts making the resettlement plan. They should immediately start in the tracing of the child – where did the child come from (…) The resettlement process is that after identifying the relative and guardian: you visit the home and you assess the status of that home, you identify the circumstances for resettling the child. Then there is the development the resettlement plan. If resettlement is not safe, you agree between TAC and the CCI on what to do with the child.” However, not all cases follow this process. A national level stakeholder involved in alternative care reflected on this with the following words: “The caregivers said that they were told one day that the child would be reintegrated. The next day the child came back. There was no preparation. They were told that the Government had decided that all children should be taken back home. There was no question whether she (the caregiver) felt it was possible or not.” This statement was confirmed by some caregivers and children in HH receiving children for reintegration, many of whom were unable to explain the reunification and reintegration process. Parents and caregivers state that their child was brought back since “she was too young and our home was near” (Caregiver, Kabarole), without being able to expand on this explanation. Similar explanations were given by other caregivers and children interviewed who could not explain why they had been brought back home: “…the Aunty (caregiver of the child care institution) came and told us that we were going. I remember it was very early in the morning and we were from digging. She mentioned our names and asked us to go shower, after bathing, we didn’t know what we were going to do, when it came to evening, we started moving to our homes, some children reached their homes so late. When I reached home I was welcomed warmly. In the morning some men came on a motorcycle and asked me if I didn’t have any challenges coming back home and I replied “yes.” So that’s when I started living home” (Reunified child, CCI, Luwero). Though the TACs became involved after many children had already been returned home, their involvement seems to have influenced the quality of the reunification and reintegration process. Of the 1,743 children who were reunified from CCIs into families, a total of 356 (20%) were reunified following a process that was quality assured by TACs. While there is no quantitative project data available to compare the well-being of children reunified through a process that was overseen by TACs and those who were reunified without the process quality assured by TACs, qualitative data from 51 various respondent groups suggests that the quality of the process differed significantly before TACs were involved and after TACs got involved. One TAC member shared: “Yes there is a difference. Those (cases) that did not go through TACs, there was no prior assessment of families where they were reintegrated. Some of these were so vulnerable so we had to recommend them for welfare support (…) There was a case where two children were defiled by their father and the CCI reintegrated the children in the community and the father was still in the community. The community was not prepared, and the children were not prepared, so abuse continued. The TAC recommended that the CCI took children back. Initially there were not this kind of checks and balances. Those who were reintegrated through the involvement of TAC, the assessment was done first.” In another district, the PSWO confirmed that: “There have been no failures in the reintegration cases since TAC started.” A national stakeholder reflected on the quality that TACs provided to the whole reintegration process: “Where TACs do well is that in complicated cases TACs went to families and triangulated information given in the initial assessment. They left no stone unturned.” There was significant difference in the “readiness” of communities, and in the support provided by the project to the communities and families where children were reunified and this seems to have influenced the quality in the reintegration processes. While families of 1,743 reunified children benefited from the same reintegration package18 there was significant difference in the level of mobilization and training of the community structures and para-professionals between prevention parishes and non-prevention parishes. Figure 4 below provides details on the distribution of children by place of reunification. In total 1,439 of the 1,743 children who were reunified with families through DOVCU, were reunified in households and communities which were less prepared to receive children back, as compared to the 304 children who were reunified with families in the parishes where DOVCU focused its prevention work. “Annex V: Statistical Tables”, Table 23 provides further details. 18 Types of items included in the reintegration package were: Mattresses, Blankets, Sugar, Soap, Beans, Sanitary pads, Knickers, Counter books, Pens, Clothes 52 Figure 4: Distribution of the reunified children by place of reunification For the children who were reunified in families to parishes that had also been supported by the project’s prevention component, the mobilization and training of PSWs and CPMs served as protective factors. In these parishes community members in FGDs expressed understanding and empathy for the conditions leading children to leave their homes, and they felt a responsibility for accepting these children, expressing ‘mercy’ towards them, disciplining them, and letting their children play with them. Many people expressed that it is the community’s responsibility to look out for these children and ensure they stay on a productive path. Some parents described how they were sitting down with their children and reunified children to discuss the importance of staying within the family structure and behaving in positive ways. Additionally, community members described taking in children that were at risk and wanting to leave their families and supporting their wellbeing and education if financially possible until they could be reunited with their parents. This is corroborated in interviews with caregivers and children, with no one raising issues of the community not supporting the reintegration process. In terms of direct visible support to families that received children for reintegration, respondents highlight the financial component in particular as important in influencing the reintegration of children. The cash transfer is continuously cited by caregivers and children in these households as an important contribution to facilitating reintegration, and even stated by para social workers as a key factor in families agreeing to reunification: “There is a family who are destitute. The daughter ran away to work in town. When this project of (DOVCU) came, one of her aunties met her and told her about the project and how they were going to assist the girl’s father. She told the girl to go back home and go back to school. At 60% 23% 17% Reunified children in other districts than DOVCU's 12 focus districts Reunified children within DOVCU's 12 focus district but outside sub-counties supported through prevention-component Reunified children within DOVCU's 12 focus district and within sub-counties supported through prevention-component 53 first the girl refused. So, when the father got cash transfer, he went to look for the girl (…) she came back home and is now in Primary School.” Beyond the support for schooling which was supported by many CCIs, respondents appreciated the reintegration package provided by DOVCU as important. For the 259 reunified children who benefited from vocational training, respondents also highlight this support as important in contributing to the reintegration process. Evaluation question 11: To what extent did the beneficiaries of the project, including children, families and communities, appreciate the services provided? E.g. did the project incorporate sufficient participatory feedback processes to ensure that the interventions and services were the most appropriate for the needs of beneficiaries? Families and communities appreciated the services provided and attributed them to an increase in school attendance, improved nutrition, decreased alcohol consumption, and decreased violence within the household. Aside from some specific comments on additional resources needed, the feedback from communities on the project was overwhelmingly positive and appreciative. People perceived that child protection mechanisms in the community were more effective now and PSWs and CPMs alike also perceived their work as creating more positive results within the community. Community members perceived the financial package as crucial to the changes in wellbeing brought about by the program. While most respondents perceived the amount given by DOVCU as helpful, many respondents asked for the amount to increase to further support to education, nutrition, health, farming, and business opportunities. Parental trainings were often cited as helpful. Parents, CPMs, and PSWs said the trainings both helped keep their personal families together as well as aided their fellow community members in successfully staying together as a unit. Community groups were perceived as integral to creating changes within the community and there was energy around the groups with members making promises to continue beyond the DOVCU project. People particularly perceived them as helpful to fostering supportive household environments and decreasing alcohol consumption and family violence. Home-visits by PSWs and the CPMs were appreciated by community members. However, some community actors felt that there was room for more communication between the different actors, and that phone credit or a cellular phone would be helpful for that. The majority of caregivers and children indicate they are happy about being reunified, as long as DOVCU and/or the CCI continues to support them. As exemplified by one caregiver: “The steps, they came and told me that the Government (…)wants them to care for children in their homes and it made me happy cause I was fearful that they will no longer look after him here at home but they assured me that they will continue with their responsibilities and they still do and this makes me happy…they talked to him (the child) and told him that at the end of the term you will go to your mother and study from there. He called me and told me and I asked him if it made him happy and he responded yes.” This said, people interviewed felt that there was a lack of immediate support for children upon reunification. 54 Evaluation question 12: What were the most significant changes in the lives of the two target groups, which were achieved by the project, and what factors seem to have influenced these changes? There appears to have been a greater shift in access to basic needs and services as compared to psychosocial well-being and care. As discussed in the impact-section, the mean differences in the vulnerability score of children based on all Core Project Areas (CPAs) indicate an overall shift in vulnerability, indicating an improvement in children’s well-being19. However, this cannot be distinguished between households at risk, and households in the reintegration category. Instead data is based on two points in time. Within this, the most significant shifts can be seen in CPA3: health, shelter and care, and CPA2 (food and nutrition) followed by education and skills training (CPA4). Reduced vulnerability can also be seen with regards CPA5 (psychosocial support) and CPA6 (protection) but is less than for the other CPAs (see also “Annex V: Statistical Tables”, Table 24). The data from the FSVI tool around economic vulnerability shows changes in livelihood security and access to basic needs, with supported households shifting out of the destitute category. While this does not apply to all households, the monitoring data is in line with qualitative findings of respondents emphasizing the financial support component as significant in impacting changes in their lives. As stated by one caregiver in Gulu: “It has been support from this organization in form of the money they gave us, and also the food that they provided. The training on how to manage finances is also important.” This said, it is continuously seen in the findings that the integrated aspect of the project has produced the most significant change in their lives: “the most significant change was the knowledge (DOVCU) gave us about savings and doing business. That one opened my eyes because before I was doing nothing apart from farming and doing casual labor. But now I do farming and business” (Caregiver, Kabarole). Yet, the financial component is credited more for changes, as this is seen as opening up opportunities for respondents in terms of how to allocate resources, by the cash transfer covering school fees for example, as well as reducing stress in the household. Respondents also perceive improvement in access to basic services, such as education and health, which are attributed to DOVCU. Qualitative findings indicate that the support provided by DOVCU enables children to go to school and experience improved nutrition in households that were supported by DOVCU. 20 Evaluation question 13: How effective was the project in engaging with various structures and professionals involved in the project, including district authorities, child care institutions, probation social work officers, community development officers, alternative care panels, remand homes, CPMs, PSWs? 19 Negative means indicate improvement, considering the vulnerability score is less at endline than at baseline. 20 As stressed throughout this report, the high emphasis on the cash transfer as a reason for changes needs to be carefully considered. Respondents attribute a lot of changes to be linked to receiving the cash transfer. For example, children emphasize a change in their subjective well-being as due to now being able to attend school. This cash transfer is also linked to the significant change in CPA3: health, shelter and care, and CPA2: food and nutrition. Respondents state that they are able to prioritize these needs once school fees are covered. Consequently, the most significant change seems to have been a change in access to basic needs and services, with qualitative findings associating this change largely with receiving the cash transfer. For reunified children and households, it is important to consider this in the context of the findings for EQ3 on children’s subjective well-being. Whilst the wider data does not fully speak to children’s well-being before and after reunification, these summative findings indicate that in the majority of cases children’s access to basic goods and services were more prevalent, and perceived as less precarious, when children were in a CCI as compared to in family-based care. As such, though vulnerability may have reduced at family level where the child was taken to be reintegrated, children reunified from CCIs indicate a greater change in the psychosocial support and care (based on the summative subjective well-being interviews) as compared to the access to basic goods and services. In this area, reunified children indicate that they are worse off. 55 The project’s engagement with district stakeholders (CAO, DCDO, PSWO, CDOs, members of TAC) was very effective. These stakeholders, across districts express a great appreciation for the project design, noting their involvement from the beginning and throughout implementation. Stakeholders such as CAO, DCDO, PSWO, CDOs testify how the project involved them in the process identifying the parishes to focus on for the prevention component. Professionals, such as PSWOs and CDOs, across all districts also had satisfaction about their involvement in the assessments of CCIs. Their involvement in the mapping and assessment exercise has facilitated their understanding of the significance of the problem and activated them in having oversight of the CCIs. District stakeholders stayed involved and informed during the entire implementation of the project. A PSWO in one of the study districts, illustrated what was heard by many other stakeholders across districts: “First of all, I should say that the involvement of my office was right away from the start. When the project started we were involved in providing information to the project on the sub-counties that are most affected by children separating from parents (…) We agreed on where to focus the project (…) I have been part and parcel of this project, issues of resettlement, issues of assessing CCIs, issues of having discussions with management of the CCIs.” In another district the Assistant CAO shared how DOVCU was unique in how it involved district stakeholders: “(…) there is no project that has involved the district like this project.” In yet another district, the Assistant CAO expressed his satisfaction: “We are very happy for (DOVCU) work (…) What we got from (DOVCU) is a unique project that should be adopted by more organizations” The strategy to rely on and build the capacity of formal district stakeholders is perceived as effective as it has contributed clarifying to district stakeholders their respective roles in the alternative care system, to activate these stakeholders and to increase the legitimacy of the project. A CDO explained how she now better understands the complementarity between NGOs and government: “NGOs come in to help us fill the gaps, it’s up on us now to follow up – we were trained, we practiced it, it’s up on us now” (CDO). CDOs in another district expressed how the project had boosted their confidence when it comes to guiding the CCIs in their work or working with vulnerable families and children: “They have professionally improved our ability, we have been empowered.” In one district the involvement of district officials ensured backing from law enforcement when the assessment team encountered problems with CCIs that were not collaborative. Political district stakeholders and Local Council members have been less directly involved but are aware of the project. The engagement of the project with community level stakeholders was appraised as very effective and appreciated strategy. To a large extent the project’s success is to be attributed to how community structures and especially the PSWs facilitated implementation at the grassroots. PSWs were seen by beneficiaries and district stakeholders alike to be effectively and actively working in the community and with the families. FGDs in communities reveal that men and women appreciated their input and sought out their help with family situations. Men and women seemed to trust, listen to, and alter their behavior according to the advice of PSWs. They were regarded by community members and CPMs to hold knowledge on child protection issues and specifically how to support children vulnerable to separation. CPMs and PSWs described increased collaboration and engagement with referrals, responses to emergency situations, and follow-ups with at risk families. The engagement of the project with CCIs in many cases started off as a difficult experience for CCIs, but the quality of the engagement improved with time. Reflecting on this process of engagement between DOVCU and CCIs, an alternative care expert explained how the “quality of engagement with CCIs, especially at the beginning of the process when assessments started” differed a lot as compared to later in the project. Baseline assessments of CCIs that were collaboratively undertaken by DOVCU and the government were used to make recommendations for closures, improvement and licensing of CCIs and as a national stakeholder explained: “there were many people involved who got power and they used it differently: some well, others not so well.” Engagement with CCIs started off on shaky 56 grounds. In many cases, CCIs expressed that the initially felt “threatened to be closed down” and “controlled.” There are many stakeholders who were later involved in the project who felt that “the name ‘deinstitutionalization” was very problematic – and targeted, first impression leaves a lasting impact.” In several cases, this resulted in CCIs rushing to send children home in order to comply with what they felt was the message of the project and government. Other CCIs simply closed and sent children home without preparation and without due process. With time the project’s investments in continuous coaching, training and dialogue with CCIs aided building the relationship into a constructive partnership. This, many CCIs expressed, helped CCIs to appreciate that what they were doing previously was not in line with government policy. A few different CCIs reflected on the effectiveness of training: “Trainings were good to prepare the minds of what is coming. Our minds were prepared”; and in another district: “Training made us change our minds – the project was not well thought of at the beginning”; and in the same district but in another CCI: “Our eyes were opened that families are the best, and that we need to be considering other forms of alternative care.” In yet another district, the manager of CCI reflected on the quality of this training: “The people who trained us. They knew well what they were talking about.” The training approach, as well as the involvement and buy-in of formal government officials in the processes supported by the project, has been referred to by many as very central in reducing the friction between the project and the CCIs, something that helped many CCIs overcome their initial resistance and suspicion of the DOVCU project. Ultimately, with only very few exceptions, the CCIs visited by the evaluation team felt that the implementation of the project as a partnership approach was effective. They felt listened too and referred to their engagement with DOVCU as a constructive and an effective collaboration leading to impacts on how they operate their CCIs and how they supported reintegration processes and where they will focus in the future. The management from one CCI that during the baseline evaluation expressed negative feelings about DOVCU, at the endline summarized the experience as follows: “The project was backed from the government. This was important (...) At the beginning the feeling was fear and (name of the CCI) was concerned about the speed for the work (reunification of children) to be completed. But the fear and uncertainty disappeared after about 6 months. The timeline (of the project) was tough but OK. The requirements were OK, with room for dialogue (…) When the timeline was too tight, it could easily be adjusted.” In spite of this partnership there is evidence suggesting that CCIs to a large extent felt that they were “left alone” in the process of reunifying and supporting the reintegration of children. Even though TACs, once they had been established, got involved in providing oversight of the process leading to the reintegration of children, TACs and CCIs expressed that: “a lot depended on the CCI” who were the sole responsible for implementing reintegration decisions and following up cases, and as a TAC member reflected: “not everyone is able to do good work.” Up until the end of the project there is no agreement amongst PSWOs, CDOs, CCIs, nor amongst DOVCU consortium partners who is the ultimate accountable for ensuring quality in the reintegration of children over the long run. The quotes below from two CCIs in two different districts illustrate: “No, we worked with the LCs, so we went district by district, we are the ones who drove, not me myself, but the workers from this place they are the ones that escorted them to their homes, got the papers signed, checked their… you know, to see if the home situation had changed in some cases, and that is how they were resettled. But these the community development officer, child protection officers, those other organizations, they did not assist in that. All they did, they checked that we were doing it;” 57 “No support was given by government, only by NGOs. Government tells what needs to be done, but they do not give any funds to support to manage. The probation officer has often no vehicle. The (CCI) need to providing a vehicle. It is time for government to support more.” Furthermore, there is still low awareness amongst CCIs, on how to graduate from the status “recommended for improvement” to being approved to operate under license by MoGLSD. Several of the visited CCIs were yet to receive the formal approval to operate and in several cases the CCIs expressed the feeling that it is up to them to go and submit the papers to MoGLSD to get approval status, rather than the district authorities responsible to do this for all the CCIs in the district. Although remand homes appreciate the tools, training and the support received from DOVCU, remand home staff express frustration with the fact that this support was insufficient to the extent that it became irrelevant to help solve the challenges faced by children placed in remand. There is a general feeling was that the DOVCU project did not help with meeting psychosocial needs of children, with urgent essential needs, or to help moving cases through court, something which is needed before children can go through a reintegration process. Children still get stuck in the justice system and while, as one remand home expressed: “Expectations were high for this project (…) it was too rigid and did not listen to us. It was about deinstitutionalization but we needed help to move cases through court before we could look at reintegration” and a similar sentiment was expressed in another district where the in-charge of the remand home explained that: “Our major objective is the court process and work, and after that our relationship ends with the child.” If remand home staff and management, who are close to and suffer daily from all of the challenges that children who enter the justice system face, and felt that DOVCU couldn’t help solve this, some district level stakeholders felt grateful for the little that the project did. In one district, the CAO who at baseline expressed concerned about the poor conditions for children in remand in his district, at endline felt that DOVCU had given some relief and, as he expressed: “responded to our plea for help with the remand home” by supporting both children and the staff for at least a short time. Similarly, CRO, working with children coming from the street in Mbale district, expressed a sentiment of having felt listened to: “We were always being consulted and we were a part of the constant revising” and reflected on the capacity building the project had achieved for them: “We were working on this before but this project helped us to standardize the process and include more stakeholders in our work.” There is evidence suggesting that DOVCU could have been more effective in its engagement and capacity building of national MoGLSD. For example, in spite of DOVCU’s significant efforts to collect data and strengthen the knowledgebase on alternative care in the 12 project districts, the MoGLSD database on CCIs remains weak. The CCI directory is not systematically updated. One challenge is the fact that the system remains manual and this challenges data-flow from district to national level. Another challenge is staffing in the ACU that is hosting the database. There are no dedicated personnel with solid data analysis skills and such skills were not transferred through the DOVCU project. None of the sophisticated data storage and analysis capacity that was available in the DOVCU consortium seems to have been transferred to Government. A MoGLSD-official expressed frustration with this situation: “We still have the database over institution, anybody can easily access it. When we go and do the assessment, when a PSWO does an assessment we want them to enter into a system and that the system is updated, like for the OVC MIS. We want the same. We now have to do the updating manually.” Another stakeholder involved in alternative care reflected on the fact that: “DOVCU has possibly supplied data, but did not support the ACU to enter and use the data (…) Capacity of MoGLSD to manage and use data has not been sustained or built” There are national level stakeholders who feel that DOVCU could have been more effective in exploiting the experience from the pilot in 12 districts to strengthen the national capacity to provide oversight, enforcement and replication of best practices. A stakeholder in MoGLSD 58 expressed this: “When you look at the regulations, the ministry should work with CCIs to help them close. We have draft guidelines for closures (…) that we need to finalize, but the DOVCU project did not help finalize this.” Some stakeholders perceive this as a missed opportunity: “We make those recommendations (of CCIs to close) but we do not have the guidelines to close CCIs in a way so as to safeguard the best interests of the child (…) So CCIs were left to self-close, or the CCIs in the pipeline for closure that didn’t self-close are still around.” Evaluation question 14: To what extent did the project manage to disseminate information regarding lessons learned from its activities and strategies? How did the project adjust its own strategies to take into account any learning that derived from the implementing process? The project organized a number of events to promote the project and share lessons learned and although not commented on separately stakeholders interviewed for the evaluation these are likely to have contributed in an important way to the overall change in mindsets, which is reported to have happened in the impact section of this evaluation. The project worked closely with PSWs to cascade information on child protection, parenting to other community members through organized community events such as dance, drama performances at traditional ceremonies and community events, during church services and at market. Radio talk shows on child protection, substance abuse, and parental responsibility were implemented. Community groups like mother’s and women’s groups were engaged to learn and share messages. The project also supported the “Ugandan’s Adopt” web page and promotional materials. The project had children’s sensitization activities and children’s groups on child rights, placed stickers on bicycles and motorcycles promoted the project. The project produced a short video on “My Father’s House” about how the CCI was transforming the way they care for children. There was an awareness campaign with faith-based actors and networks on preventing separation and promoting family care. District professionals and CCIs generally express appreciation for the project’s community awareness and sensitization activities. This, they report, has contributed to a change in attitudes, not only around the importance of family care, but also around importance of parenting skills and psycho￾social issues which, they feel, has contributed to reducing domestic violence that is an important factor contributing to children leaving home. One CCI exemplified: “The project has really brought about big community sensitization. People in the community are beginning to see their responsibility in caring for children”, a message that is reinforced and confirmed by a CDO in the same district: “The district is slowly phasing out CCIs – this has started with the participation in this project. The CCIs understand that taking children home is better for children.” In another district, the PSWO reflected on the effects of DOVCUs community sensitization and made the following conclusion: “Violence Against Children (VAC) is lower in the two parishes where the project concentrated. There is better handling of situations at the PSW level and also increased community awareness” The project included many learning elements, which it used to inform interventions. For example, the preparatory process which assessed communities, households, CCIs and children was again and again referred to by stakeholders as a very useful strategy to inform the project’s targeting. Several stakeholders testified positively about the project’s ability to flexibly adapt the timing and sequencing of the activities in order to allow for quality in interventions. The impact chapter and effectiveness chapter provide evidence of the fact that resources to mobilize the involvement of stakeholders who were crucial for the successful implementation of the project were well invested. One district-level stakeholder noted with satisfaction that: “DOVCU brought the entire force: government, NGOs, child care homes, community, behind the effort. The evaluation team furthermore could document at endline that he project had adjusted its strategy with remand homes significantly, and involved many more stakeholders, and also recruited volunteers to support children psychosocial wellbeing and support reintegration efforts, as a direct follow up to the baseline evaluation recommendations. 59 EFFICIENCY This section will discuss the efficiency of the project, defined as how well the project was implemented and whether the implementation was optimal in terms of time and resources. The evaluation was guided by four efficiency questions. Evaluation question 15: To what extent were the project strategies, to reduce vulnerability of households and to reintegrate children, efficiently implemented? Were there any implementation gaps? There is evidence suggesting that the overall impact of the project’s strategies to reduce vulnerability was affected by the late start of the project and especially so in phase 2 districts. The project’ start was initially delayed by a few months, and then saw a relatively long preparatory process was undertaken to target activities. While this was perceived as a thorough and necessary process, it also meant that direct implementation with beneficiaries did not start immediately. Furthermore, because of the significant scope of the project, covering 12 districts, districts were divided up in two groups: phase 1 and phase 2 districts. Stakeholders and beneficiaries in phase 1 districts benefited from a longer implementation (2 full years). Phase 2 districts had a similar start-up phase as phase 1 districts, replicating the assessments that were used for targeting of activities but because of starting later direct implementation with stakeholders and beneficiaries took place only for one year and two months. The direct implementation phase, in both phase 1 and phase 2 districts was the phase when district, sub-county and community stakeholders benefited from coaching and training activities and when communities and targeted vulnerable beneficiaries received cash transfers, training in financial management, training on parenting, awareness building and reunifications. As a result of the short implementation period, also in phase 1 districts (where this evaluation focused its qualitative data collection) several stakeholders referred to the implementation period being “too short”, or “not enough” to have a lasting and significant effect in the lives of beneficiaries especially (see also sustainability section for more discussion around this). The effect of this on project results is evidenced also in the project’s monitoring data that shows an important difference in improvement in household destitution between phase 1 and phase 2 districts: • In phase 1 districts 60% of households were assessed as destitute at baseline and this had reduced to 21% at endline, • In phase 2 districts 51% were assessed as destitute at baseline and 41% had the same status at endline. • Both in phase 1 and phase 2 districts most targeted households transitioned into the struggling category and only a few households made it to the growing category. Figure 5 below illustrates and “Annex X: Lessons learned on the project’s integrated package of support” provide further illustration in the difference in results between phase 1 and phase 2 districts. Figure 5: Number/proportion of households in destitute, struggling and growing categories of economic vulnerability in phase 1 and phase 2 districts Baseline Endline Vulnerability level All Districts Phase 1 District Phase 2 District All District Phase 1 District Phase 2 District Destitute 1252 (56%) 739 (60%) 513 (51%) 670 (30%) 261 (21%) 409 (41%) Struggling 974 (44%) 484 (39%) 490 (49%) 1534 (69%) 949 (77%) 585 (58%) Growing 8 6 2 30 19 11 60 (0%) (1%) (0%) (1%) (2%) (1%) Once the direct implementation started it was found to have been implemented efficiently for most parts with only few exceptions. In spite of a short direct implementation phase, the changes achieved in community and stakeholder-perceptions around the importance of family care and the changes documented in professional practices and services provided by CCIs and CRO, all of which are documented in the impact section of this report were very good. This said, some aspects of support were perceived to having arrived late, in particular the stipend to volunteers who operated in support of remand homes, and of PSWs who sometimes complained that their stipend was often delayed due to reporting. While this does not seem to have had any effect on delaying their work, they felt that this incentivized them to undertake the challenging work they were responsible for under DOVCU. Moreover, PSWs also expressed that budget constraints affected attendance in trainings. One PSW illustrated this: “It is really difficult because many times we invite people to come for the meeting and many do come expecting something that we don’t have but we have been open and always tell them we have nothing.” Another example of support coming too late was the reintegration package. Given that this package originally was not planned, the late delivery to families was not a problem according to some stakeholders: “When children were resettled first, we didn’t promise that there was any kit. The issue was focusing on the process. So, when they received the kits this was taken as a surprise and some households took that as a thank you – a gift.” (PSWO). This was not a shared view and some stakeholders were critical of the fact that families initially were without much support after receiving the child. One CDO stated: “… of course, there were some households that were struggling to take on an extra child and for them this kit would have been helpful earlier. The impact on the child in these households would have been better if the reintegration kit was provided at the beginning of the process. For these children they came with the experience of sleeping on the mattress, they had to adjust to a very different situation at home (…) in terms of transitioning from the welfare in the institution to the welfare at home, this package could have made the transition process smoother.” And another: “They did not get the reintegration package immediately, it took about 8 months. By the time when the follow-up was provided the children was gone from home (…) The children disappearing had gone to look for jobs (…) none of them left their homes permanently.” The strategy foreseen by the project for safeguarding that reunification of children was done in the best interests of the children was only partially efficiently implemented and this is felt to have impacted on the quality of the process of reintegration of many children. TACs were formed and trained between September 2015 and February 2017 and came on board progressively to review the cases of children proposed for reintegration. In all districts except Iganga, reunification of children started before TACs had been formed and were operational. In Luwero, Kampala and Wakiso none of the cases of reunified children were reviewed by a TAC prior to sending children home. In total, only 20% of all the children who were reunified through DOVCU had their cases reviewed and approved for reunification by a TAC (see also “Annex V: Statistical Tables”, Table 25). Once on-board, TACs provided a multi-disciplinary review of cases prior to reunification, and, as discussed in other sections of this report, became very appreciated for their role in quality assuring reintegration processes. Evaluation question 16: Which activities/strategies used by the project proved to be the most efficient in influencing the lives of children and vulnerable families? Were there any activities/strategies which were not at all efficient? If there is one specific component of support that was often singled out as quickly having produced an important effect and a multiplier effect in the lives of children and vulnerable 61 families, it is the cash transfer. This component of the support was perceived as having a multiplier effect in that it allowed beneficiaries to access services, and to save money and start a business. The efficiency of this support also depended on the preparation that had been done prior to its delivery to vulnerable households. Across the districts, the integrated approach was referred to as having maximized the effect of cash transfer in the lives of the beneficiaries that received it. In one district, the PSWO reflected on how the project had made sure that beneficiaries, after receiving the transfer, made good use of it through monitoring families: “There was a cash transfer, I was fully involved in handing over, in other places I witnessed, and then I was monitoring to see that the money was put to good use, sometimes people are given money and people just eat it. That is why NGOs score much better than Mr. Government. Mr. Government gives you and people just eat, never monitors. For this project, even if you were to go to the ground you would see something tangible from the DOVCU project.” But at the same time as this support was perceived as of critical importance, several respondents in communities also expressed needs for larger amounts and more regular transfers to support education, nutrition, and income opportunities for their families. Another strategy that was perceived as efficient and effective in the communities was the various community groups that were created. These were regarded by community members as having maximized their potential within communities and are also perceived as important for sustainability as they can easily continue operating after the funding package ceases within communities. As much as all of the individual activities and the support package as a whole were appreciated and considered as important, many stakeholders also felt that the strategic package of support was insufficient. In one district, the DCDO reflected on the insufficiency of activities to reduce vulnerability with the following words: “The support, like the loans and savings associations and the cash support… This is something, but when you look at the human being if you want to assist this person, there are many, many aspects that you need to do. The program gave the training, cash… but the program could have done a little bit more. For example, the trainings: if you provide a training on agriculture, and this was not provided, this would have allowed them (the families) to develop better livelihood (…) If you give a little more money say 200,000, of this you invest 50% and then you go for treatment, then you can have a greater effect. The program gave little money. Maybe focus on fewer families, districts, but make more significant impact.” Evaluation question 17: Which strategies used by the project proved to be the most efficient in influencing the system reform? Were there any strategies / activities which were less efficient? The strategies employed by the project and activities implemented with district policy and professional stakeholders to change mindsets, priorities and professional practices around alternative care, were by and large efficient. According to the project’s own monitoring data, between the project’s start and its end, 499 policy stakeholders and professionals were involved in different types of training activities across the 12 districts and nationally. These strategies and activities are perceived as having been implemented in a timely manner and with enough resources to produce change in the professional activities of the targeted stakeholders. This finding is on one hand evidenced in the impact chapter and is also confirmed directly by respondents from these stakeholder groups who converged around the view that the training received on alternative care that combined parenting, psychosocial support, financial literacy, child protection, micro-projects etc., coaching and skills building activities had been efficient for them to change their views around the importance of family care, and to change professional practices. However, some respondents also felt that these training activities had 62 been too short to produce a deeper change in how they do their work. They expressed a need for follow-up training allowing for deeper exchanges on the practical challenges in applying new practices and techniques in their everyday work. A staff from a remand home illustrated this sentiment “Trainings are very short. Two days is not enough. It would be more useful if the training was repeated and run with reminders instead of in one shot.” Several CDOs repeated a similar need: ““There was too little training (...) It would have been useful to get more training on parenting.” The strategies and activities to change practices of CCIs towards deinstitutionalization were implemented over sufficient time, with an intensity to support many child care institutions to make significant changes to their admission, care and reintegration practices. Across the 12 districts, DOVCU assessed 147 child care institutions at baseline and worked with 91 of them over the projects life-time while these CCIs made important changes to their practices as evidenced in the impact section of the report, the significance of changes varied from CCI to CCI. Furthermore, a significant number of CCIs that are operating have still not received their license. For more equal effect across CCIs, and for CCIs to be fully compliant with govern standards, more time and resources would have been needed. In that sense, DOVCU can be seen as having triggered change but not as having completed the work. The project expanded and over-delivered on its strategy with the four remand homes, the national rehabilitation center. The original plans for training and provision of technical assistance and supported supervision to the four remand homes and the national rehabilitation center, were expanded with exchange learning visits between remand homes. Furthermore, they were provided with additional support to improve counselling services through recruitment of volunteer staff, and the project expanded its work to include more district stakeholders, such as police and magistrates, in training activities around the prevention and reintegration of children from remand homes. Because children in remand homes and the national rehabilitation center originated from all over Uganda the project saw the need for involving stakeholders from many more than the 12 DOVCU districts in training activities. However, in spite of this very relevant expansion of the strategy it was not enough to influence more systemic reform in remand homes. As observed in the baseline and endline, remand homes, which are part of the broader justice system face many challenges which influence practices for the entry of children into the justice system and the reintegrating of children once they are released. The project did not have enough resources to address all these challenges. Both for CCIs and remand homes, an area where more support would have been needed is in strengthening case management to fully ensure equal quality of the follow-up, effective monitoring, case plan implementation, and documentation of monitoring visits. In total, 237 policy and professional stakeholders from remand homes, the national rehabilitation center and CRO were trained on the use of SOPs for the reintegration of children, and 348 stakeholders from CCIs and district government were trained on alternative care. Even though the case files for children in CCIs more generally speaking have improved since baseline, the reunified children’s case files which the evaluation team reviewed either on-site in the CCIs and remand homes, and sometimes also in DOVCU-district offices, had many weaknesses. For example, the evaluation team reviewed several case files and met children who had been reunified and who displayed physical signs of potential trauma or stress, such as frequent unexplainable stomach aches, anxieties and fears which had not been followed up sufficiently to find causes and solutions to the problems expressed by the children themselves. Across districts, follow-up visits to reunified households implemented by CCIs, and the documentation they hold on these visits have focused more on education and health issues, less on psychosocial issues of children. Furthermore, it is not clear to what extent CCIs and others doing follow-up visits have consulted with children systematically during follow-up visits. In some case files, consulting with children is reported, in other cases not. The evaluation team came across several reunified children who stated 63 that they were not consulted during follow-up visits and several of these had problems that they raised with the evaluation team. The regularity of follow-up of reunified children furthermore seems to have varied between CCIs and depending on which stakeholder did the follow-up. In some cases, this follow up was reported to be very regular (weekly, monthly, once every three months or for as long as needed), while in other cases follow-up by the CCIs was not done according to a specific schedule, or at all. The quality of the information collected and documented in case files, as well as the actions taken on any concerns observed during visits, seems to have varied importantly depending on who was involved in the follow-up, and how well-informed and well-resourced this person was. Evaluation question 18: To what extent were the human resources, coordination mechanisms, time and financial resources invested in the implementation of the project efficiently used and were they sufficient to achieve the targets of the project? The evaluation establishes that the project has been efficient in its use of human resources, coordination mechanisms, time and financial resources to achieve its goals. DOVCU was a complex project, with ambitious goals to reintegrate children and prevent separation, involving many stakeholders and beneficiaries at national, district, community and household levels in 12 districts where mindsets of the stakeholders that the project had to rely on, were not yet prepared. However, there were some challenges that surfaced during the implementation of the project as a result of the project’s design, and resources available were not sufficient to fully achieve the targets of the project. This said, it is felt that, of the overall time available for the project to be implemented, the time that the project was actively working in communities and providing support to families was not enough for the project to have a maximum and lasting effect. Indeed, the project had a long preparatory phase, and the second phase districts had a relatively shorter period of active implementation in communities than the phase one districts had. Illustrating the perceived insufficiency of time of the project, a CDO stated: “By the time the community was feeling the benefits the project was over,” a sentiment that was echoed across districts, such as by the CAO in another district who stated: “Three years is not long enough for this kind of project but it seemed to have been well implemented.” As the project was coming to an end when data collection took place for the endline evaluation, several stakeholders were still not aware of the fact that the project was about to end and stated: “This program was just a pilot and we learned a lot, but now we need the actual program to really roll out what worked” (CDO) All of this has influenced prospects for sustainability which are further discussed in the sustainability chapter. The evaluation team documented some challenges with regards to the sufficiency in the number of project social workers, their level of training and retention in the project in order for these to be able to quality assure the case monitoring of reunified children. Given the weak social work capacity available through the Government structures in Uganda and the unequal level of commitment and capacity of CCIs to do regular follow-up with reunified and reintegrated children which have been evidenced in other sections of this report, the evaluation concludes that the project did not have enough complementary social work capacity to be able to train and support, and quality assure the work of other structures (such as CCIs, remand homes) and professionals (such as PSWOs, CDOs and para￾social workers) to guarantee quality in the follow-up and case monitoring with the children who were reunified with families. Overall, the project employed three senior social workers at HQ level, 10 social workers that were based in the districts, and four regional social workers that each covered one region21. 21 North, south, east, west. 64 SUSTAINABILITY This section will discuss the sustainability of the results of the project, of reintegration efforts, structures established and professional changes achieved. The evaluation was guided by three questions, which are reported below: Evaluation question 19: For the children who have been reunified with their families or placed in alternative family-based care, what reintegration methods have supported stable and sustained placement of children and to what extent? Are children still in the homes and in school 6 months (and more) after placement? What makes these reintegration methods sustainable? If there has been relapse in reintegration, why has this been the case? It is not fully known for how many children’s reunifications with their families have resulted in stable and sustained placements. This evaluation relied on the project’s monitoring data for quantitative data and triangulated this data with qualitative data collected as part of the evaluation’s primary data collection methods. While the monitoring data of the project indicates that 100% of reunified children reassessed at endline by the project have been in the placement for at least 6 months, and 82% have been in family care for at least 24 months, this contrasts with information received from the evaluation’s qualitative sample of households and the policy component of the evaluation. Through these different methods a few cases of households were identified where children were said to no longer be at home and in one interview with an ex-director from a closed CCI it was reported that a child had gone missing after the CCI was closed. The case was known to several stakeholders but had not been counted in the project’s data. In addition, as elaborated in evaluation question 10, some families testified that their children had in fact never returned home, but had remained in the “CCI” that had transitioned into a boarding school. To this should be added that the project’s monitoring data captured the endline status only of the 698 (40%) children who were reunified within DOVCU’s 12 focus districts whom were visited by the project’s endline monitoring assessment. For children who have been reunified outside of the 12 DOVCU districts, the evaluation does not have data. For the children who are at home, a variety of factors, such as ability to pay for school fees, food security, and quality of care, appear to influence whether a child remains within a household. For the households that were geographically located in the sub-counties where DOVCU focused its prevention work, the financial package was cited by respondents as the most significant support contributing to families being able to keep children at home. The financial package is used primarily for school fees, allowing households to spend more on food and other needs from their sources of income. Furthermore, the financial package also allowed family members to support farming inputs or their small businesses, even if they also wished for this package to be bigger. Parental trainings and community groups were also seen to directly reduce child separation risk factors. In terms of community mechanisms, the most utilized and effective mechanisms to reduce the chances of falling back into risk seemed to be home visits by PSWs and the community groups supported by the project. Respondents also attributed parental trainings with this opportunity to form supportive communities. In addition, caregivers and members of the community particularly highlighted a better understanding of how community protection structures operate, and for this understanding of how community protection services utilize each other for referrals and coordination to better support households and children. In FGDs, community members, PSWs, and CPMs alike expressed increased knowledge about child protection as a result of the trainings. Each group expressed that the trainings affected how they treated their own children and the community’s children, and they all described passing on this knowledge to their fellow community members. Community members and PSWOs said that the knowledge about child protection went beyond targeted households, as behavior modelling fostered better interactions between parents and children in non-targeted households as well. 65 All of these factors, which were addressed in the sub-counties where DOVCU focused its prevention work, thus contributed to reduce the risk for secondary separation, and to support the sustainability in placements and reintegration process of 304 out of 1,743 reunified children. For the remaining 1,439 children who were reunified outside of DOVCU’s 12 focus districts and sub￾counties, the prospects for sustainability of placement are more precarious, given that the project did not support families and communities in these sub-counties and districts to address the factors which otherwise are reported as factors contributing to separation, and in most cases, there was no follow up. For the children who were reunified within the DOVCU supported parishes from CCIs, while the findings around project interventions and improvement in well-being do highlight elements of sustainability while the project was still operational, there are indications that sustainability beyond the project end is vulnerable. For example, amongst the households and children who were assessed at endline, monitoring data indicates that there has been consistent improvement in well-being for all categories of children over a period of 6-18 months follow-up, with positive shifts to less critical vulnerability categories. However, children who are in the households targeted by the prevention components of the project (“at risk children”) show better improvement in wellbeing than reunified children. This is illustrated in Figure 6 below22. Between all categories of children who have been reunified, children who have been reunified from CCIs show less relative improvement in well-being as compared to children who have been reunified from remand homes or the street. The importance of this finding from the point of view of prospect for full reintegration and sustainability of such placements, especially for this category of children, is also illustrated in the qualitative findings regarding overall subjective well-being. Children who have been reunified from CCIs, while perceiving overall care and psychosocial well-being to have improved, stress the perceived precariousness of access to basic services. They still feel vulnerable something which affects their subjective feelings of well-being. While part of this may be that children who have been reunified from CCIs experience transitional stress that can affect overall well-being it is still important to note that children who have been reunified from CCIs perceive their access to services as more precarious than prior to reunification. Figure 6: Comparative analysis of wellbeing of children at-risk, reunified children from CCIs, RH and street. Risk Overall Child at risk (prevention) Reintegrated CCI Reintegrated RH Reintegrated Street Baseline Phase 1 district Critical 49 (3%) 41 (3%) 1 (0%) (0%) 7 (6%) Medium 1623 (86%) 1315 (88%) 190 (75%) 22 (100%) 96 (82%) Low 218 (12%) 143 (10%) 61 (24%) (0%) 14 (12%) Endline Phase 1 district Critical 6 (0%) 6 (0%) (0%) (0%) (0%) Low 696 (37%) 504 (34%) 124 (49%) 8 (36%) 60 (51%) Medium 1178 (62%) 979 (65%) 128 (51%) 14 (64%) 57 (49%) Respondents continuously highlight the highly precarious situation that they find themselves in and express a dependence on the continuous support by DOVCU in order to avoid separation or re-separation. This particularly speaks to households who have received cash transfers. While policy 22 As indicated previously, the monitoring data on well-being is from the point of reunification and then endline. Data was not collected whilst children were in CCIs/remand homes/on the streets. The data comparable points are thus not before and after reunification, but at the start of reunification and later on ('now'). We thus can't compare well-being before and after reunification. 66 level stakeholders appear quite positive about the sustainability of the financial support and training given to beneficiaries, beneficiaries themselves are more worried, and in all districts question what will happen if DOVCU no longer supports them. While the financial component appears to have contributed to some households being able to diversify their livelihood, and more respondents are accessing VSLAs, factors that are beyond project control, i.e. the root causes for separation such as a volatile environment and limited economic opportunities create a context in which respondents are highly vulnerable to fall back into a situation where separation occurs. This is particularly referenced with regards to access to basic services such as education. Moreover, while VSLAs provide a more sustainable function than cash transfers, the evaluation questions to what extent these mechanisms are resilient to shock which may affect the majority of households. For example, it is not clear whether the VSLAs are able to sustain if a large number of its members were to experience shock/be unable to pay back. Consequently, though households appear to have shifted gradually into less vulnerable situations, this shift appears precarious. As testified by one caregiver, falling back into vulnerability would increase risk for separation or re-separation: “not being able to get the basic needs leads to stress for both the parents and the children themselves which leads to separation of the child so as to find a better living” (caregiver, reunified household, Mbale). Findings indicate that the psychosocial and community-based elements of the project are more likely to sustain reintegration processes after the project has ended. Respondents continuously cite ‘knowledge’ as something that will remain when the project ends. However, PSWs do emphasize the need for refresher trainings. Important to note is that potential effects of parent trainings at this stage are mainly noticeable in terms of ‘knowledge’ as compared to norms or behavioral change. Caregivers are able to explain what they learned in training but are less able to provide examples of how this has translated into practice. While parents may currently be able to explain what they have learned, this learning may not have been sufficient to change attitudes around parenting more deeply, and any direct behavioral changes following the training may thus not be strong enough to sustain without continuous reminders. Consequently, it may be too early in the change process to assess behavioral change and sustainability in behavioral change. However, continued change in norms and behavioral is likely to be reliant on the sustained operation of community groups such as fathers and mothers’ unions, post￾treatment support groups etc. Similarly, PSWs are perceived as key in supporting the changes on community level. It is not clear to what extent PSWs will continue to receive sufficient support to be able to sustain their activities within the community. Consequently, findings indicate that the project has sustained support for the reintegration of children in a number of cases throughout the life of the project, though the findings cannot show this is sustainable in the long term. The volatile environment in which families and children live makes sustainability a challenge and as stressed by policy level stakeholders, full reintegration of children and sustainability in family placements will also heavily depend on CCI’s continued support to the reunified children to access education and health services. While most of the CCIs interviewed by the evaluation team that had contributed with such support during the project’s implementation time also expressed commitment to sustain this support in the future, there were also policy level several stakeholders expressing worry about what would happen after the project’s end. This was particularly the case for the children who were reunified from CCIs that could not sustain the support for the child’s quality education. There were several CCIs that did not have this capacity also during the project’s life time: “For the reunified children that will continue getting support especially in terms of school dues from the CCIs that kept them, I think it is promising and there will be permanency. For those that have to depend on UPE for education, and also place in homes that are not well of socio-economically, the risk of separation is still a reality.” (PSWO) 67 Evaluation question 20: By project end, to what extent have structures been established or strengthened that can continue to provide, on an ongoing basis, adequate case management and support services for children at risk? Are these sustainable and what makes them sustainable? The project established TACs in 1123 out of 12 project districts, all of which the evaluation assesses as having good prospects to become sustainable, but under the name of district Alternative Care Panels. A number of factors contribute to this potential sustainability. Firstly, the national ACPF foresees establishment of ACPs in each district of Uganda. At the time of the endline evaluation, the MoGLSD was in the process of documenting the experience of TACs in DOVCU districts to feed lessons learned from this pilot into national guidelines for district ACPs. Furthermore, TACs were set up as a multi-disciplinary team, as per the TORs. However, in some of the districts the district leadership took the decision to go beyond this requirement and embed the functions of TACs in existing district community department functions. In these cases, given that several of the representatives on TACs are paid district staff, this will facilitate the sustainability of TACs in the future. In a couple of the districts, the district leadership furthermore has plans to integrate TAC functions with the duties of the existing District Orphans and Vulnerable Children Committee (DOVCC). The following quote from a CAO in one of the districts provides illustration: “At the end of the project we need to integrate the TACs in the DOVCCs (District OVC Committees). TAC is the structure of a project. The integration of TAC activities with DOVCCs activities will help sustain the work beyond the projects timeframe since departments already have budget for their activities” (CAO). While the integration of the tasks of TACs with the functions of district community departments and DOVCCs can facilitate the use of district budget for TAC activities, such earmarked funding is not yet available in any of the districts. Nevertheless, TAC members in several districts have been sufficiently empowered through DOVCU and express a strong commitment to sustain their work of these committees beyond the project’s life￾time, regardless of the funding situation. In one of the districts, TAC members made the following declaration at the end of a focus group discussion with the evaluation team: “Come back! The way you found us is the way we shall be” (Joint declaration of TAC-members in one district). In another district, TAC members expressed their commitment to sustaining the activities with the following words: “ (Through DOVCU the TAC has received…) so much facilitation that even right now, even if there is no facilitation I can be very sure that this team will continue to do its work (…) these people now, are very interested to really do their work and that shows that they will continue doing their work whether we are there, or the project has left” (TAC member) The sustainability of case management and support to reunified children and children at risk is more questionable. The case management, follow up and support provided to reunified children to facilitate their reintegration have, during the project’s lifetime, been facilitated by a number of stakeholders and actors, ranging from project staff, PSWOs, CDOs, community child protection structures and PSWs depending on location. The project also relied very much on CCIs, remand homes and CRO to continue monitor the well-being and provide support to reunified children. As discussed in previous chapters, the capacity of CCIs to monitor and follow-up on children who have been reunified, has varies between CCIs and so has the capacity to continue supporting the child and his/her family to facilitate the reintegration process. It has depended on the availability of funds within the CCI to recruit social workers and to undertake monitoring visits. It has also depended on the geographical proximity of the CCI to the children who were reunified (many reunification cases were indeed in non-DOVCU districts). In the situations where CCIs during the project’s life-time had limited capacity to follow-up on and support the children they reunified, this capacity challenge is likely to continue also after the project’s life-time. For the CCIs that did demonstrate capacity to follow-up on children who have been 23 TACs were created in 2015 in Mbale 09-2015; Kamuli 12-2015; in 2016 in Kasese 04-2016; Kabarole 04-2016; Gulu 04-2016; Luwero 07-2016; Jinja 08-2016; Lira 10-2016; Kampala 10-2016; Kabale 11-2016; and in 2017 Iganga 02-2017. 68 reunified, the evaluation documented that some of them have had a rapid turn-over of social workers during the project’s life-time. Thus, it is not sure that all CCIs can sustain the skills sets in case management and follow-up for reunified children that were built through the project’s training and coaching activities, nor is it sure they can continue supporting and following up on the children until they are fully reintegrated. Given that the case management and follow up to the 1,045 children who were reunified outside of the 12 focus districts was questionable even during the project’s life-time, there are concerns for these children in particular. A national level stakeholder summarized the challenges related to case management, monitoring of reunified children, and support to facilitate the reintegration of children, with the following words: “My worry is that we do not have enough social workers. The scope of DOVCU was huge. There were big numbers to reintegrate and almost no additional social work capacity.” For the children who were reunified from remand homes, the capacity of remand homes to continue being involved in monitoring and following up on cases until these are fully reintegrated is not likely to be sustained. Remand homes fulfill certain government functions as part of the overall justice system and they do not perceive it as their role to continue managing cases and monitor the well-being of children who have been released, once these have served their sentence. Furthermore, remand homes lack funding to undertake such activities. In one district, the responsible person for the remand home was explaining how the staff of the remand home indeed had been trained on how to use SOPs in the reunification of children, but that they already saw challenges in applying these skills: “The project did give us social workers and psychologists and we did do some reintegration. This was a help at that time.” Thus, staff in at least a couple of the remand homes were reporting to no longer use the SOPs and explain the reason for that with the following words: “Reintegration has to be funded – and we don’t have funding so we don’t do proper reintegration” [Officer-in-charge (OIC) Remand Home] And in another district, the OIC for the remand home expressed that “We know how to do it but we do not have the resources to continue the process.” Another national level stakeholder, expressed similar concerns: “There was a verbal declaration from MoGLSD to remand homes during the launch that they should use the SOPs. But remand homes need the facilitation and resources to implement these SOPs. So, the issue may be that MoGLSD may fail to give the facilitation resources to remand homes.” (National level alternative care expert). CRO, on the other hand, demonstrated good capacity to do follow up and case management for reunified children during the project’s life-time. They have made the use of the FVSI / CSI for assessments of families and children prior to reunification, and to monitor well-being post reunification, as their standard tool also in other projects. Given the track record of this NGO in mobilizing funds for reintegration activities targeting children in street situations, their follow-up is more likely to be sustained. The project has not built enough capacity of PSWOs and CDOs to mobilize them in having a role in monitoring and quality assuring the case management provided by CCIs. As discussed in other sections of the report, capacity gaps of PSWOs and CDOs are many and not necessarily only related to skills. Indeed, they face a number of constraints, such as lack of transport and funding for monitoring activities to be able to be actively involved in the continued follow-up of children who have been reunified. In order for PSWOs and CDOs to be actively involved in quality assuring the case management and follow up provided by CCIs, they would have needed more capacity built than what the project could provide through training. Given that the project’s capacity building strategies focused on skills building, this was not enough to fully bridge the gaps in capacity of some of the CCIs and remand homes in their follow-up and support to children who have been reunified. 69 Because of their proximity to communities, para social workers, community child protection structures and CDOs demonstrate the greatest potential for sustained follow-up and support to children at risk and children who have been reunified. The evaluation documented (as reported in previous sections) that these personnel have had their capacities and skills strengthened significantly. The training received by para social workers and child protection mechanisms have for example increased coordination during referrals and follow-ups, and these strengthened relationships are perceived as likely to continue after the project has ended. Furthermore, the community groups created through the project seem to provide one of the most sustainable mechanisms for change within communities. The savings groups, alcohol support groups, and parental trainings have all created shifts in community knowledge and appear to have buy in from community members. These groups are reported by community structures and para social workers as having become self-sustaining and will continue beyond the project. Parental training has changed the dynamics between parents and children, but it is unclear if these effects will hold once refresher courses end with the program. Evaluation question 21: For the different changes observed in the capacity and functioning of professionals, child care structures and coordination mechanisms (e.g. probation social work officers, para professionals, alternative care panels), to what extent are they likely to sustain the work, to continue meeting and to continue to apply what they have learned after the end of the project? DOVCU built skills at many levels, including of MoGLSD to have a stronger role in national oversight of the alternative care system, of district officials and professionals and of community structures and para social workers to increase their involvement in alternative care priorities. This has resulted, across the board, in a strong commitment to sustain activities initiated by DOVCU. The sustainability of all these skills and of the activities that put such skills to practice, such as meetings, assessments of CCIs, meetings of TACs, and also of undertaking refresher trainings depend to a large extent on funding. At the level of MoGLSD, the knowledge and skills to undertake assessments of CCIs as per the national standards are perceived as sustainable. However, there is only a small team available in MoGLSD to undertake and sustain the CCI-assessment process launched by DOVCU at a larger scale. Furthermore, MoGLSD capacity to enforce recommendations for closure of CCIs that are below standard, or to proceed with criminal procedures against CCIs suspected to be involved in irregular activities was reported as a challenge already during DOVCU. In spite of this, a new comprehensive national child policy with a costed implementation plan was in the making during data collection for the endline evaluation. It includes, amongst other priorities, a sustained government priority to alternative care reform efforts. As such, it also provides an important framework within which additional resources can be mobilized to support this reform. A national government stakeholder expressed very good prospects for some additional capacity and funds to come MoGLSD’s way in forthcoming budgets: “Next year’s national budget will invest some money in the alternative care unit: money for inspections and making sure that reintegration is done (…) The budget is for training and to implement processes around inspections. Before the government did not have any such funds. The department gets the funds, and a lot of the money of the department goes to feeding children in remand home, but now it is the first time that the government will have some of its own funds to carry on with inspections and the processes around reintegration, like training and facilitation like fuel.” In a few of the researched districts, the district leadership express commitment to mobilize sustained funding for priorities related to alternative care reform and for some of the activities that were initiated by DOVCU. In one district the CAO made the following reflection: “Now that the project is ending, we ask ourselves what to do as government. NGOs are supplementing our work. Now we 70 should be prepared to consolidate and sustain these activities. This is why we need to integrate these activities in our plans and budgets and make proposals to other partners” (CAO). In the same district, the PSWO had already made concrete steps to explore opportunities to get national government funds, and funds from other sources to support district priorities in alternative care. “I am happy that in the meeting I attended in Kampala yesterday I heard that there should be grant for children. That can facilitate TACs. This would be a national government grant. But also district local government have some funds from local revenue and some of these funds should be allocated for child care support.” Also, in other districts do professionals involved in district alternative care work, see the potential for sustainability, through government being more involved in planning and budgeting for alternative care priorities: “If the district can take up this work, with planning as well as with budgeting, then we will be able to do big things for children” (PSWO). But there were also voices of skepticism heard amongst district officials and professionals. In one district, the Assistant CAO expressed the following concern: “The probation has a department. It is still having other constraints outside the project. For example, if they do not have a running vehicle, how are they, will they be able to follow cases? If the CDO at sub county does not have fuel to see where these children are; it means they will be in office they will not be able to supervise and get the real picture of what is happening on the ground (…) You see, our social workers have been trained but they cannot work without papers, pen, and fuel.” In another district, the PSWO reflected on government’s commitment to address social problems and poverty at all: “Poverty and all that underlies it is still a challenge (…) There’s always concern that government programs won’t be there” (PSWO) Mainly amongst front-line professionals, such as TAC members, CDOs and PSWs, the potential for sustainability is not necessarily associated with prospects to get funding for activities to be sustained. Instead, several of these professionals speak about how the project has contributed to behavioral change and changes in professional practices that can be sustained through passion and commitment. In one district, a TAC-members expressed this with the following words: “I have learned that we need to be passionate and there needs to be love for the children (…) This is what has inspired the TAC. We sat together case by case and decided together so we are now colleagues. When it comes to children you sacrifice your time, because without time you cannot do good work with the child. You are putting in your own money and time for that child (…) Our commitment will just continue even if the project ends…” District professionals put a lot of hope in PSWs for sustaining the work, since they live in the community, close to where problems occur: “The beauty of the PSW is that they live in the community and the community feels good about them” (PSWO). While it is unclear if the PSW will sustain their work in the same manner, indeed, they expressed a number of challenges related to monetary compensation and transport which were challenging them in their work already during project implementation, it is clear from FGDs that there has been a fundamental shift in PSWs understanding of child protection as a result of DOVCU training. This knowledge has reportedly strengthened their own families as well as the families of community members. Therefore, there is the possibility that this knowledge base will continue to support positive changes and the protection of children within the community. PSWs and CPMs discussed each other’s work and their coordination at length in interviews. DOVCU training made the different actors aware of each other’s work and how they could work more effectively together within the community. Many interviews highlighted their joint work as 71 ‘colleagues’ and showcased their knowledge of when and how to work together. For example, as one PSW noted: “Working together with the office of the CDO has made us realize that we are also part of the CDO structure in the community. The CDO also seek advice from us and they allocate work to us, they can make a call that “please make follow up of this case that is in your community.” The linkages are also very good and has made work very easy, plus it has improved policing.” All of this said, across all the districts, there seemed to be relative agreement across stakeholders that three years was not long enough to ensure the sustainability of project impacts. “Three years is too short. What can be the sustainability and how can we be assured that the work will not fall back?” (CAO). And several stakeholders expressed the need for the pilot to go to scale: “This has been a pilot – one we have learned so much from. We have come really far. Some people said that we should have been in all the sub-districts, but this gave us the opportunity to learn and to become more active as we take things to scale” (PSWO); “This program was just a pilot and we learned a lot, but now we need the actual program to really role out what worked” (CDO) HUMAN RIGHTS Given that the project under evaluation was working with the most vulnerable groups in society and that it aimed to address some of the human rights violations of children who were beneficiaries of the project the evaluation included a specific evaluation criterion on human rights guided by four questions. Evaluation question 22: To what extent did the project manage to target those most at risk (e.g. children in institutions and children on the street), including tailoring activities to accommodate needs of the most vulnerable children (children under three, girls, children with disabilities, street children)? The prevention component of the project managed to target some of the most vulnerable families. This was confirmed by a number of stakeholders throughout districts, both among policy-level respondents and community level stakeholders who expressed their appreciation, both of the tools and the process through which vulnerable, destitute families were selected and assessed to be beneficiaries of the project’s prevention component. This was felt to be a useful and transparent process, even though, as some stakeholders noted, the limited coverage of the project meant that not all those who were in similar destitute situation could be included in the project activities. Amongst all children who were reunified from CCIs, remand homes and from the street, the project partially included those most at risk. For children in residential care, there are some children who can be considered as “more at risk” than other children. For example, international research has shown that young children below the age of three, have greater needs for individual stimulation and opportunities to bond with a stable care giver, needs which are often difficult to meet for young children in residential care. The project-design did not convey any specific messages regarding the vulnerability of institutionalized children under the age of 3 years and as a result no specific attention was given to that consideration within the project. In fact, in most cases it was much older children who were reunified. Consequently, the monitoring data from the DOVCU project indicates that of all reintegrated children from CCIs, only approx. 9% of the total of 1.465 reunified children, were below the age of 5. Of all the children who were reunified from CCIs, 53% were boys and 47% were girls, most children (73%) were in the age group 6-14 years old, followed by (18%) children who were above 15 years of age. The children who were reunified from CCIs through DOVCU were not necessarily coming from the CCIs with the poorest quality of care. Children who are living in CCIs with poor quality of care, can be considered more vulnerable or more at risk than children living in CCIs with at least high quality 72 of basic needs, health care and education. While the project monitoring data does not allow for a detailed breakdown on which individual institutions the reunified children came from, aggregate data per district suggests that, of the five districts that had the lowest average score for all CCIs on child care provisions (Wakiso, Kasese, Kamuli, Luwero and Kampala), three of these district (Wakiso, Kasese and Kampala) were also amongst the five districts that reunified the least children (see also “Annex V: Statistical Tables”, Table 26). Instead, qualitative data collected through interviews with policy level stakeholders and stakeholders in CCI confirmed that CCIs “self-selected” and volunteered to start reintegration activities. Furthermore, within each CCI, the selection of children to be reintegrated was not based on set criteria or an assessment of who was most at risk. Stakeholders in CCIs and at district level indicate that the selection was driven by which cases seemed to be the “easiest” to reintegrate. This was, for example, children who had relatives ready to receive them, or whether the child was ready to go home. In one CCI, the management described how they mainly reunified children who were older and almost “aged out”, leaving small children, under age five and below, stay in the CCI. Several other stakeholders expressed a similar priority, and a preference to reunify older children, because according to these stakeholders, older children were considered by families as easier to take back in family care because they are relatively more autonomous than younger children who have greater care needs. Monitoring data from the DOVCU project indicates that of all reunified children, 3% were children with disabilities (see also “Annex V: Statistical Tables”, Table 27). This is in line with international experience, which has often shown that children with disabilities are often the last ones to benefit from de-institutionalization efforts, in spite of the fact that they have similar needs for family care as children in general. In one of the districts, CDOs felt that the inclusion of children with special needs in reintegration efforts was a challenge to them because the project’s training opportunities did not, according to them, equip stakeholders enough to address the issues of families to children with special needs: “We have been having a challenge of children with special needs. The trainings do not consider children with special needs. When you have a child in the sub-county like that, it is so difficult to know what to do.” (CDO). Stigma and attitudes to children with special needs were reported as barriers to their inclusion in reintegration processes. The project upheld gender parity between boys and girls who were reunified from CCIs, with 53% boys being reunified from CCIs, and 47% girls. This is in line with international child rights standards, which establish that boys and girls have equal rights. With regards to children in remand homes, the project’s targeting of those children who had served their sentence seemed appropriate. In Uganda the minimum age of criminal responsibility is 12 years of age and there was no indication of the fact that any child below that age was held in any of the remand homes that the project worked with. Most reunified children from remand homes, 95% were boys and this can be explained by the fact that boys are disproportionately represented in the remand home population. As has already been discussed in previous sections of the report, findings suggest that the project’s package of support was not enough to address the specific needs of children from remand homes during the reintegration process. Remand home stakeholders often referred to stigmatizing attitudes in communities where children were reunified, that they felt were challenging to overcome. For children on the street, the project purposefully selected and reunified girls, as they had been identified as underserved by existing services and with particular risks, which the project wanted to develop experience around how to address. Consequently 100% of all the children who were reunified from the street were girls. Of these, 27% were in the age range 6-14 years old, and 73% were above 15 years old. 73 Evaluation question 23: To what extent was the project able to prevent child protection issues and human rights violations (e.g. child labor, violence against children, neglect, early marriage, etc.) and mitigate risks for creating greater vulnerability (e.g. child labor and any kind of harm)? The project seems to have contributed to preventing child protection issues and human rights violations in communities for a broader number of children than those that were particularly targeted by the project’s prevention and reintegration components. This is evidenced in that the project has achieved a mindset change around child protection violations amongst beneficiaries. The better functioning of community-based referrals systems, improved connectedness between community structures and districts structures and professionals as well as changes in practices, especially in many CCIs, are amongst some of the factors that have been facilitated by the project and will contribute to preventing child protection violations in the future. A CDO explained how this work had effect beyond the targeted households with the following words: “The truth is the DOVCU has carried out training to parents, and the leaders like parish chiefs, councilors, religious leaders, PSWs, so they now know that issues like children spending time in sugar cane plantation is a violation of their rights. And they should work against it.” For children who were living in supported households, interventions contributed to preventing perpetration of violations of several child protection issues within the targeted households. Monitoring data indicates that 13% of children in the age category of 14-17 years who were living in targeted households at baseline were engaged in full time exploitative unpaid work and this had reduced to 8% at endline. The proportion of households that reported incidences of various forms of violence against children in their family in the last 12 months reduced in all categories. For example, households that reported that a child was abused (sexually, physically or emotionally) and/or being subjected to child labor or otherwise exploited reduced from 6% at baseline to 1% at endline; households that reported that a child is “neglected, given inappropriate work for his or her age, or is clearly not treated well in household or institution” reduced from 17% at baseline to 4% at endline (see also “Annex V: Statistical Tables”, Table 28 and 29). Children interviewed in household case studies by the evaluation team confirmed that violence against them has been something they experienced in the past (in particular children now reunified had experienced this more when they were just reunified), but that this was not something they worried about now. When children did refer to violence during household case studies at endline, this was mainly in reference to fears of violence such as “if you get robbed” (reunified child, CCI, Kamuli), “some older people in the community may beat on you” (at risk child, Luwero), “there are stories of people that take children” (reunified child, CCI, Mbale). Similarly, none of the children spoken with reported experiencing harmful traditional practices. Some children mention that these are present in the community, but to not have experienced these. Similarly, the project mitigated some risks for creating greater vulnerabilities for children who were already living within the targeted households. Monitoring data captures that • HH that were reporting to have had psychosocial distress most of the time in the last 12 months reduced from 18% of households at baseline to 8% at endline; • The proportion of targeted households that seemed hopeless and sad (emotional distress) in the last 12 months reduced from 18% at baseline to 3% at endline; • The proportion of targeted households that were reporting frequent or periodic signs of aggressive behaviors, domestic violence, child abuse or child neglect, reduced from 14% at baseline to 0% at endline; and • The proportion of households where the caregivers in the past 12 months used discipline approaches that included punching, kicking, hitting, withholding a meal or basic needs, use abusive words/ language reduced from 60% at baseline to 55% at endline. See also “Annex V: Statistical Tables”, Table 28 and 29 74 For children who were reunified from CCIs with families in view of being reintegrated, it is unclear if the project has prevented these children from becoming more vulnerable as a result of the DOVCU reunification activities. An important human rights principle to consider when working with vulnerable children and children at risk is the all child rights are equal and no right is superior to any other. This means that children have equal rights to protection, development, survival and participation. At any given time, when considering interventions for children, these should aim to maximize the potential for children to access all of their rights or as many of their rights as possible. Another important guiding principle in programming addressing human rights, including on child protection, is to “Do No Harm.” This means that it is the responsibility of all relevant stakeholders involved in the project addressing child protection concerns to ensure that vulnerable children and children at risk who are involved in a project do not suffer any additional harm from their involvement in the project. When project interventions will have the effect of changing the circumstances under which children will access their rights, such as changing the care arrangement for children, a best interests determination24 needs to inform decisions for each individual child. In the context of this, it is important to assess separately the project’s ability to prevent child protection issues for children who were already living in targeted households, and for children who were not living in households prior to the start of the project. As illustrated in the figure below, monitoring data shows that between the three groups of children who were reunified (from CCIs, remand homes and street), and the forth group of children who were already in family care and whose households were targeted for prevent support, children coming from CCIs were the ones who showed the greatest well-being when assessed at baseline. They are also the group of children who have seen the smallest shift between baseline and endline into the “low vulnerability” category. Even though there is an increase in the proportion of children reunified from CCIs that are classified in the low vulnerability category (from 24% at baseline to 39% at endline), the shift was the smallest for this group of children. An important point to note regarding these figures is that while the baseline assessed all children who were reunified, the endline assessment undertaken by the project only covered a sample of children. This sample was selected from the 12 districts where DOVCU was focusing its project activities. Given the difference in support given to community structures in DOVCU districts, as compared to non-DOVCU districts, there is a possibility that vulnerability levels are higher in non-DOVCU supported districts and this is not captured in the monitoring data. In fact, very little is known about the children reunified outside of DOVCU districts and this is of concern. If this is considered in the light of findings presented in previous sections of the report regarding households’ continued vulnerability, concerns expressed, in particular by children reunified from CCIs regarding their continued access to services, and the child rights principles introduced at the beginning of this section, the evaluation cannot exclude that some children have become more vulnerable as a result of the DOVCU reintegration activities. This statement is furthermore to be considered in in light of very limited documentation available on the case management follow up provided to these children. Figure 7: Changes in the vulnerability level of children (endline compared to baseline) Risk Overall Child at risk Reunified CCI Reunified RH Reunified Street Baseline Critical vulnerability 92 (3%) 84 (3%) 1 (0%) 0 (0%) 7 (6%) Medium vulnerability 3003 (86%) 2459 (89%) 426 (76%) 22 (100%) 96 (82%) Low 381 (11%) 235 (8%) 132 (24%) 0 (0%) 14 (12%) 24 This is needed because the circumstances for all children are different and it is not possible to determine that one set of interventions is best for all children who are in the same situation. Best interests’ determination should look at the child’s individual circumstances where he she currently is accessing some of all of his/her rights, the considered new arrangement, and assess how the specific child can access all or as many as possible of his/her rights. A best interests determination should also assess risks of any harm which may be caused by the intervention. 75 vulnerability Endline Critical vulnerability 14 (0%) 14 (1%) 0 (0%) 0 (0%) 0 (0%) Medium vulnerability 2482 (71%) 2071 (75%) 340 (61%) 14 (64%) 57 (49%) Low vulnerability 970 (28%) 683 (25%) 219 (39%) 8 (36%) 60 (51%) Evaluation question 24: How did the project address child rights violations that required urgent attention? The project put in place some measures to address concerns expressed in the baseline evaluation for children in remand homes. Given the precarious situation of these children, the project expanded its work with remand homes, supported the recruitment of volunteers who could help remand home staff address some of the psychosocial concerns identified for this group of children in particular. For children in CCIs, the baseline evaluation particularly noted the risks to children during too rapid reunification, something that the project only partially managed to mitigate. As noted in previous sections of the report, many children were reunified through the project without a prior best interest determination of a TAC, which had been foreseen as a safeguarding mechanism. Furthermore, in four of the six study districts for the evaluation, district officials confirmed that some of the CCIs that had closed or stopped operating as CCIs, had done so as a matter of urgency, a process that was often initiated by the CCI itself following the initial assessment facilitated through DOVCU, and not always accompanied by the DOVCU project. It is not known how many children were sent home from these CCIs, without assessment, reintegration support or follow up. In one district several stakeholders referred to one particular case of a child who had gone missing after such rapid closure. There was no follow up provided to this case. The above said, the project did expand its strategy to support the 1,743 children that the project reunified through the project. All these children were supported with a reintegration package, as also noted in other sections of the report. Evaluation question 25: To what extent did the project contribute to strengthening duty bearers’ knowledge of their obligations and their capacity to meet those obligations? Across all districts, the project has contributed significantly to strengthen duty bearers’ knowledge about their obligations and roles in alternative care. Community members in FGDs confirmed that the parenting training provided by the project taught them about child rights, which in turn affected how they treated their children and other children in the community. As discussed in the impact section and also further detailed in annexes, district level stakeholders and professionals are now, to a much greater extent than at baseline, aware of the alternative care framework, the importance of family care, and what are the various roles and obligations they have to implement this policy framework. Professionals, CCI and community-based volunteers and structures have also changed their practices to work better together to ensure that rights of children to family and community life is upheld. If stakeholders had a better understanding in their roles and obligations, some of their capacity is unfortunately still constrained by limited resources to carry out their responsibilities. However, as documented in the sustainability section, there are some plans being made at district level, and at national level to mobilize resources for continuing to give alternative care priority in district activities. Stakeholders across all respondent groups seem to have taken the message of family care and the primordial right, overriding other rights. This was expressed across districts and by many different stakeholders. In one districts, a CDO expressed this with the following words: “DOVCU, is 76 about ensuring that children in CCI go back too families because it’s the best place to grow and anything you do it must be for the best interest of the child. It is intended to ensure that children are returned to home care, according to what we said it is the best place, ACF tries to solve the problems afflicting children, to ensure that they get the best care in the best place, for a child to live well, they must be with the family or the next of kin..” A PSWO, assessed the success of the project by the number of children who were reunified: “For me I see this, they (DOVCU) have scored 90%. Many children were resettled, a reduction in the number (…) an achievement.” The project did not put in place a clear system of accountability for the case management follow-up of children who were reunified. Child rights standards call for clear accountability frameworks to be established between all stakeholders, including donors and international partners who are involved in work with children. Even though the project invested resources in the development, promotion and training on SOPs in order to resource different professionals to have a role in the follow up of the children who were reunified from remand homes, CCIs and from the street, this was not enough to ensure an accountability system for the best interests of the children who were reunified. The project relied on other stakeholders to carry the main responsibility for the regular monitoring and follow up on reintegration cases. This was most often done by the CCIs themselves and by community structures and volunteers. If CCIs to a large extent were responsible for implementing the reunification process and were often also committed to monitoring the extent to which this resulted in a safe reintegration, the feeling of accountability to ensure safe reintegration differed significantly between different CCIs. When CCIs did not develop this accountability, either because of a lack of commitment, because lack of resources, or when resources constrained them to take on this accountability because of distance to the reunified child, it is not clear who took on the accountability for ensuring close follow-up on these cases and for monitoring regularly that the reunified child was doing well. As discussed in other sections of the report, community structures and volunteers trained by DOVCU were only present in the parishes where the project implemented the prevention work and did not cover all areas where children were reunified. Furthermore, across all districts where a TAC was established, these had a clear role and responsibility in multi-disciplinary review of, and decision on cases prior to the child returning to family care. Their role in the follow-up of reviewing these cases after the child had been reunified, and on deciding on case closure because the child was considered as reintegrated was not as clear. In some cases where TAC members stated that they were involved in monitoring the well-being of reunified children, this function was often blurred with the understanding of individual professional roles. In some cases, for example, a CDO or a PSWO who was a member of the TAC had been involved in a follow-up visit in his/her individual professional capacity, but there was no systematic evidence found of TACs being regularly involved as a multi-disciplinary group in monitoring and reviewing the success of reintegration processes. Finally, the investment of DOVCU in training, coaching of various district and sub-county stakeholders on the use of SOPs, was no enough to result in PSWOs or CDOs becoming involved in quality assuring the reintegration process facilitated by CCIs. For example, across the study districts, PSWOs and CDOs did not have easy access to copies of case files of children reunified from CCIs, and it was not clear when was the last time they did any quality assurance of CCI’s follow-up on reintegration cases. In interviews, it was often stated that these case files were kept at PSWO office something which upon verification turned out to be untrue. In terms of the project’s own follow up, resources did not allow for enough follow-up to fill the gaps in other stakeholder’s follow-up. The project’s own follow-up to reintegration cases first took place when the reintegration package was distributed and when child status data was collected mid-term and at endline. However, the timing of this was not regular enough to detect and address challenges in the reintegration process. Furthermore, due to logistical and financial constraints the project only covered 40% of the households with reunified children in the end-line assessment of reunified children25, thus it did not take 25 The reason for this low coverage was that the majority of the reunified children were outside of the 12 DOVCU-supported districts. The reported logistical and financial limitations of the project did not permit a rigorous end-line assessment in households outside of DOVCU 77 accountability for reassessing the wellbeing of children reunified outside of the 12 project districts. It is not clear who was accountable for assuring the regular monitoring and follow up of the wellbeing of these reunified children. supported districts and therefore the 698 children within reach were taken by DOVCU as a representative number for the 1743 reported cases of children reunified with families. However, the conditions for the reintegration process varied significantly for the children who were reunified within the project supported districts and parishes and those who were reunified outside of these districts and parishes and therefore it is questionable to what extent this “sample” can be considered as representative. 78 CONCLUSIONS The DOVCU project was a complex and ambitious project that aimed to demonstrate how separation of children can be prevented in a highly vulnerable context and how children can be reintegrated into families in a safe and sustainable manner. As such it addressed very relevant and urgent needs in the Ugandan context. The evaluation concludes that the project to a large extent has demonstrated an effective and relevant package, methods and strategy to prevent separation of children. This strategy included a very thoughtful process of targeting vulnerable communities and the most vulnerable families, a relevant package of support which addressed many of the needs that communities have expressed as necessary to prevent separation. The project furthermore sequenced activities in a way that these created a positive effect in the up-take of the support. If targeted households did not graduate to the growing category to the extent that what had been expected, the cause for this is likely related to the highly vulnerable environment in which beneficiaries live, which necessitated support over a longer timeframe, and for some families, larger amount of the support. The project faced some challenges in the design when it came to combining the prevention and the reintegration components of the project. In practice, it was not possible to fully make sure that the children who were reunified or placed in family care benefited from the full package of support and community mobilization that DOVCU generated in the communities targeted by the prevention component. At the time when the project selected the areas where it focused its prevention work, the origin of the children to be reintegrated was not known, and the project interventions for prevention, which could have contributed to prevent secondary separation of reintegrated children only benefited a few of the children who were reunified. This created some risk for the quality and sustainability of the reintegration process for a number of children which this evaluation documented indirectly. This said, some of the greatest successes of the project have been that it has contributed to a significant mindsets change around the importance of family care for children. It has contributed to breaking the trend of an increasing number of CCIs opening up in the 12 focus districts, the number of children in CCIs have reduced in these districts, and many, many CCIs have significantly improved their practices and work with children as direct result of the project. This is a great achievement. The project furthermore significantly strengthened knowledge, skills and professional practices of district, sub-county and community stakeholders involved in alternative care, and overall has contributed to create a very favorable environment for continued work on alternative care in Uganda. In spite of some of the challenges which have been documented in this evaluation report, especially around the reintegration of children from CCIs and remand homes, the project is by most of the stakeholders consulted, perceived to be a success. The below quote from a TAC member illustrates well the enthusiasm created around the DOVCU project: “I think that it is a success. To me, I have seen this project as a success (...) I am actually proud of it like I was implementing it, to the extent that when we had the child forum we were requested as the district to present a model that has worked well for the district and in fact it was me who said: “I think DOVCU has worked“ (…) because you know, you go to (region), you find that because of the project the children that were resettled have stayed in their homes.” i ANNEXES ANNEX I: LESSONS LEARNED ON PROJECT’S INTEGRATED PACKAGE OF SUPPORT AND STRATEGIES FOR PREVENTION OF SEPARATION Background The DOVCU project aimed at testing interventions to prevent separation of children and the institutionalization of children. Prevention activities focused on identifying destitute and struggling households that might be at-risk for separation. The process included raising awareness about family￾based care and the alternative care framework and preparing community support structures (CSS) to identify households. Vulnerability and needs assessments used the CSI and FSVI tools, adapted for Uganda based on international best practice models. A focus of the prevention work was on providing cash and asset transfers and linkages to other supports, establishing and training VSLA groups, cash management and income generation training, vocational training for youth, parenting education and parent groups, PSS, substance use support groups, and monitoring family improvement through home visiting (using the same tools as above). Linkage to other services began with a service mapping looking at health, nutrition centers, schools, law enforcement, disabilities unions, social protection, livelihoods programs and others. According to reports the need for mental health and substance abuse services became apparent and activities were added. Initially the project intended to use mobile money for cash transfers, but most households were without mobile phones so a separate process for cash transfers was established. The integrated package of support consisted of a several measures aiming to strengthen households economically, given that poverty is identified as a major contributing factor to separation. However, not all poor families send a child to live in residential care. It is therefore key to further identify what the additional triggering factors are, that combined with poverty, can lead a child to become separated from his/her family. To support this process, the baseline evaluation of DOVCU identified three different layers of causes to separation, namely immediate causes, underlying causes and root causes that are contributing to the separation of children from their families. “Immediate causes are those causes which seem to be more directly associated with risks that lead to the decision of a child to leave, or the decision of a parent, community member or a professional to refer a child to a child care institution or report him/her to the police” 26. The baseline evaluation identified such causes to include high levels of food insecurity; negative household dynamics, violence and child labour; drug and alcohol abuse in the family, and lack of access to basic social services. “Underlying causes are those causes which contribute to a specific vulnerability that make such a situation more likely to happen” 27, such as poverty; psychosocial vulnerability and barriers to good parenting. “Root causes are hierarchically the underpinning societal situation contributing to a general overall vulnerability”28, such as lack of financial opportunities; lack of land and assets; social and cultural norms; political instability and displacement. 26 DOVCU baseline report 27 Ibid. 28 Ibid ii The integrated package of support at work The components of the integrated package addressed many of these identified causes in a sequenced manner. Counselling and advise, for example to address alcohol abuse and to build parenting skills were provided first and were meant to help address some of the more immediate causes to separation, such as negative household dynamic, violence and child labor, drug and alcohol abuse. Addressing eventual drug and alcohol abuse was furthermore necessary early in the process to address some of the root causes to the poverty situation the family was living in. This was then followed by skills building activities to strengthen financial literacy and planning. The first cash transfers to “at-risk” families were meant to stabilize the household economically. This allowed families to address immediate causes such as acute food insecurity which, as an extension of poverty, can lead to the separation of a child. If necessary, depending on the destitution level of families, the first cash transfer was then followed by a second and a third transfer to allow the household to plan economically for their future, addressing poverty, and as such an important underlying cause leading to the immediate cases for separation of a child. In parallel, families and communities were receiving training on how to grow family income, and the setting up of VSLAs. This was meant to support families to grow economically, sufficiently for them to become, not just less poor, but to become more resilient, addressing the root causes: such as creating financial opportunities, opportunities for generating some assets and make families more resilient to the shocks that may plunge them into poverty again. The gradually sequenced support to destitute families allowed households to incrementally address the causes that can lead to separation. In parallel, the DOVCU project also had awareness raising activities in communities, addressing underlying social norms that can lead to separation, and successfully changed community attitudes around child care institutions and the importance of family care. Furthermore, linkages were created to other services and as such, addressing an immediate cause to separation, namely lack of access to basic social services. The package was holistic and as a CDO and a PSWO reflected: “The integration of all activities is key”, and the strategy designed to sequence activities to incrementally address first immediate, then underlying and root causes was felt to be “very important that families are first trained how to use the money before the money is received.” (PSWO). Detailed analysis of the results of the integrated package of support on causes for separation Economic well-being Financial vulnerability is caused by several situations, such as unemployment, limited access to land (including conflicts around land ownership), lack of productive assets and labor constrained households. Some regions have continued to be more vulnerable to poverty due to natural disasters, political instability and conflict. In the baseline findings, the data collection illustrated a high prevalence of destitute households in the six study districts for this evaluation. The findings from an analysis of quantitative monitoring data show an overall shift in households from destitute (baseline: 60%) to struggling (endline: 77%) categories. This shift seems to have occurred to a similar level in both male and female headed households. The change in destitution status of households in all districts shows a similar pattern. However, changes in destitution status are smaller in phase 2 districts as compared to phase 1 districts. Table I below details changes. iii Table 1: Shows number/proportion of households that belong to; destitute, struggling and growing level of economic vulnerability; data disaggregated by gender of household headship, all districts and phase 1 districts results BLV ELV Household headship Vulnerability type All Districts Phase 1 District All District Phase 1 District Total Destitute 1252 (56%) 739 (60%) 670 (30%) 261 (21%) Struggling 974 (44%) 484 (39%) 1534 (69%) 949 (77%) Growing 8 (0%) 6 (1%) 30 (1%) 19 (2%) Male headed Destitute 476 (21%) 292 (24%) 246 (11%) 94 (8%) Struggling 543 (24%) 248 (20%) 758 (34%) 437 (36%) Growing 8 (0%) 6 (0%) 23 (1%) 15 (1%) Female headed Destitute 776 (35%) 447 (36%) 424 (19%) 167 (14%) Struggling 431 (19%) 236 (19%) 776 (35%) 512 (42%) Growing 0 (0%) 0 (0%) 7 (0%) 4 (0%) Struggling households thus represent the great majority of households (77% in phase 1 districts, 69% in phase I and phase II districts combined) at endline while at baseline the great majority of households were classified as destitute households. Similarly, although female headed households remain worse off than male headed households on average, the percentage difference in households at the destitute level has decreased by 6 percentage-points and is thus half what it was at baseline. In absolute terms, more female headed households have thus shifted out of the destitute category. This may be explained by the cash transfer provision, as nearly twice as many female headed households (532) to male headed households (289) received cash transfers in phase 1 districts. On average, households that were targeted for cash transfers received two rounds of cash transfers. In total cash transfers were distributed to 720 out of the 739 destitute households in the phase 1 districts. Most growing households (less poor in relative terms) are located in Gulu and Luwero districts, however the numbers remain low. Gulu also sees the biggest shift in households out of destitute status (in relative terms), with 60% of households shifting from destitute to struggling categories, with Mbale seeing the biggest shift in absolute terms. Two phase 2 districts (Kabale and Kampala) show a negative shift from struggling to destitution levels. Table 2: Changes in household’s economic vulnerability classification by districts Baseline Endline District Destitute Struggling Growing Destitute Struggling Growing Gulu 96 (68%) 46 (32%) (0%) 12 (8%) 127 (89%) 3 (2%) Iganga 76 (41%) 108 (58%) 1 (1%) 37 (20%) 146 (79%) 2 (1%) Jinja 110 (77%) 33 (23%) (0%) 61 (43%) 80 (56%) 2 (1%) Kabale 122 (52%) 111 (48%) (0%) 153 (66%) 80 (34%) (0%) Kabarole 67 (50%) 66 (50%) (0%) 46 (35%) 86 (65%) 1 (1%) Kampala 63 (69%) 28 (31%) (0%) 71 (78%) 20 (22%) (0%) Kamuli 87 (52%) 77 (46%) 2 (1%) 19 (11%) 145 (87%) 2 (1%) Kasese 160 (57%) 120 (43%) 1 (0%) 34 (12%) 243 (86%) 4 (1%) Lira 114 (36%) 198 (63%) 2 (1%) 66 (21%) 241 (77%) 7 (2%) Luwero 89 (56%) 68 (43%) 1 (1%) 35 (22%) 119 (75%) 4 (3%) Mbale 239 (69%) 107 (31%) 2 (1%) 114 (33%) 229 (66%) 5 (1%) iv Wakiso 29 (71%) 12 (29%) (0%) 22 (54%) 19 (46%) (0%) Total 1252 (56%) 974 (44%) 9 (0%) 670 (30%) 1535 (69%) 30 (1%) To highlight these changes, all 6 phase 1 districts had over 50% of households in the destitute category at the baseline stage of the project, with the endline results showing over 60% in the struggling category. Three out of the six districts have over 80% of households now in the struggling category. The majority of households thus graduated to struggling level, and the findings show that this shift was significant. Yet it is important to note that only about 1% of households graduated to the growing level, (i.e. in relative terms the less poor category of household vulnerability). There were no significant differences between the phase 1 and phase 2 districts. Very few households have thus been able to progress from struggling to growing. At baseline 6 households were in the growing category and at endline 19 households were in the growing category. As such, the grand majority of households have remained destitute or shifted to struggling. However, this may partially be explained by the fact that the struggling category was fairly broad. Monitoring data does not reflect the shifts within this category explicitly. Moreover, the qualitative findings highlight the continuous vulnerability of households with regards to the volatile environment in which respondents live. As one caregiver in Luwero explained: “…it [the financial support] is working but you don’t prosper cause even though we farm sometimes seasons fail and rearing might also not go well meaning that we are still down cause we don’t have jobs that quickly boosts someone. Sometimes things are good sometimes bad.” Consequently, while the financial support may contribute to pulling households out of destitute economic vulnerability, the growing category may be too far off for the majority of households, in particularly considering the often highly volatile context and the short time frame of the project. Table 3: Number and % of households scoring below 49 points on FSVI CPA1 and 2; disaggregated by sex, age and all districts and phase 1 districts results Sex / Ag Characteristics of the house disaggregated by age. No. of households; all districts No. of households; phase 1 districts # and % Score <=49 baseline all districts # and % Score <=49 baseline phase 1 districts # and % Score <=49 endline all districts # and % Score <=49 endline phase 1 districts Total 2235 1228 982 (44%) 490 (40%) 1565 (70%) 968 (79%) Male 1027 546 551 (52%) 254 (35%) 781 (70%) 452 (86%) Female 1208 682 431 (32%) 236 (31%) 784 (66%) 516 (72%) Male 50+ 334 212 174 (52%) 100 (47%) 254 (76%) 174 (82%) Female 50+ 492 300 198 (40%) 119 (40%) 337 (68%) 235 (78%) Still, the majority of respondents attribute a change in their economic vulnerability to the financial support provided by DOVCU. As seen in table 3, there has been a reduction in phase 1 households where the targeted caregiver has no income from 30% (371 households) to 5% (60 households). Similarly, a drop can be seen in households where caregivers have poor income. Households thus seem to have largely shifted upwards with a larger percentage (60%) of targeted caregivers now having a slight income. Moreover, the project reports that 96% of the targeted households received business skills training, 79% were provided with financial literacy training and 83% with VSLA training. As would be expected, the cash transfer is highlighted as having contributed to changes in economic well-being by all respondents in v households that were targeted for this. In addition, respondents in all households emphasise the VSLA training as impactful. As one caregiver in Kabarole mentioned: “They took us and taught us how to save money and invest it so that it can keep on earning profit. We then formed a VSLA where we are 30 members and we are saving together…We save money every week. This has helped me to work hard because minimum saving is two thousand shillings a week. But if you can afford more it is okay because it goes to your account. This can help you to get more money when you are looking for a loan.” Respondents also highlight that the training has motivated them in their everyday lives. While there has been no change in bank account ownership in phase 1 districts, households still state that they are now able to save, largely through the VSLAs with 44% of endline phase 1 households now having access to credit services (see table 4 below). Several respondents mention that the training has showed them the collective responsibilities associated with the VSLA and stress the importance of paying back on time so that the VSLA can keep functioning. This is marginally reflected in the monitoring data where fewer households have failed to pay, and a slightly larger number of respondents have paid back in full. Table 4: Household monetary income All districts BL Phase 1 districts BL All districts EL Phase 1 districts EL Household monetary income by status Family, especially the targeted caregiver has no income that supports family and children in the household 574 (26%) 371 (30%) 118 (5%) 60 (5%) Family, especially the targeted caregiver has poor income that does not sufficiently meet the needs of the family and children in the household 1265 (57%) 623 (51%) 852 (38%) 366 (30%) Family, especially the targeted caregiver has slight income however it is not sustainable enough to meet the needs of the household 366 (16%) 218 (18%) 1179 (53%) 733 (60%) Family, especially the targeted caregiver has a sustainable income that supports family 28 (1%) 16 (1%) 86 (4%) 69 (6%) Table 5: Household capital and loans All districts BL Phase 1 districts BL All districts EL Phase 1 districts EL Household headship ownership of bank account Yes 203 (9%) 121 (10%) 285 (13%) 122 (10%) No Account 2027 (91%) 1107 (90%) 1950 (87%) 1106 (90%) Household access to credit source and capital SACCOs 105 (5%) 61 (5%) 89 (4%) 64 (5%) VSLA, or any saving group 527 (24%) 293 (24%) 1042 (47%) 537 (44%) Private Individual 161 (7%) 98 (8%) 215 (10%) 165 (13%) Household loan borrowers with good loan repayment status Failed to pay 111 (14%) 58 (13%) 95 (7%) 28 (4%) Not yet repaid due to some reasons 232 118 278 142 (20%) vi (30%) (27%) (22%) Partly repaid (remittances going on well) 262 (33%) 163 (37%) 472 (37%) 264 (38%) Fully paid 100 (13%) 55 (12%) 217 (17%) 137 (20%) Not yet due 80 (10%) 47 (11%) 216 (17%) 124 (18%) With regards to the business skills and financial literacy training, though respondents are appreciative, they highlight the challenges they face in applying the skills learned from these trainings. Key challenges highlighted are perceived insufficient funds and the volatile environment. Respondents argue that though training is useful, it can only be impactful if one has sufficient funds to start a business, as stated by one caregiver: “That training can be helpful if you are earning and can budget for what you earn, because they trained us that if you get money you should save to start up an income generating activity. But if you don’t have the capital then you can’t start up the business. Among the things they taught was how to save and how create income, but it is very good information which I will implement when I get money” To this extent, respondents instead emphasize the immediate use for the cash transfer and how it has helped them in purchasing goods that they were lacking. Several respondents stress how they ended up using the cash transfer for these immediate needs such as medicine or food, and that they therefore ran out before they could invest it. However, importantly, respondents feel they have learned the skills and know what they would do with the money were they to receive another cash transfer, “I would have bought chicken and goats that I would rear at home. I would also use the money to improve on my roof as well because the grass is old and it is leaking. Then I would use the rest to take my children to school.” Consequently, respondents perceive the financial package to have put them on the path towards decreased economic vulnerability, but for more support to be needed in order for them to continue to progress. Elements of this may be due to respondent bias (where respondents wish to receive continued support). However, as seen in the monitoring data: though there is an upward mobility, there is a limitation to the pace of this shift. Households are shifting out of the destitute category and appear to be shifting upwards within the struggling category (though this cannot be teased out from the monitoring data), yet few households have been able to shift into the growing category at the time of endline. Yet, there are some indications that the financial package has helped households diversify their livelihood options. With poverty as a key factor in making households more vulnerable to separation, poverty is closely linked to households facing limited economic opportunities. Table 6 shows a variety of characteristics used to unpack economic livelihood security status. In particular, a change can be seen in the number of households engaging in petty business such as vending, road side selling and kiosks, moving from 13% of households (155) in phase 1 baseline to 21% (264) in endline. The investment in this type of livelihood is consistent with qualitative findings where respondents report engaging in selling second hand clothes, sewing, having small kiosks amongst other things. However, in line with the abovementioned concern raised by respondents, there is a question to the initial investment needed to both start and continue this business. Respondents state that it is difficult as “many people do the same things” (Caregiver, Mbale). This reality that many respondents invest in similar small businesses within small communities, was also observed by the teams during data collection, and is reflected in the wider findings. This raises questions around market saturation, as for example the findings show 4 caregivers within the same community investing in the selling of second hand clothes. vii Table 6: Shows household economic livelihood security status (characteristics) at endline compared to baseline characteristics BLV ELV All districts BL Phase 1 districts BL All districts EL Phase 1 districts EL Household main source of household income by sources None 118 (5%) 75 (6%) 27 (1%) 14 (1%) Remittances 85 (4%) 48 (4%) 70 (3%) 29 (2%) Casual Labor 440 (20%) 193 (16%) 360 (16%) 149 (12%) Informal/ Self – employment 240 (11%) 119 (10%) 201 (9%) 102 (8%) Peasantry / hiring out labor on other farms 1014 (45%) 588 (48%) 1009 (45%) 620 (50%) Petty Business e.g. vending, road side selling, kiosk 255 (11%) 155 (13%) 466 (21%) 264 (21%) Formal Business e.g. any licensed business 17 (1%) 14 (1%) 25 (1%) 17 (1%) Commercial Farmer 37 (2%) 17 (1%) 42 (2%) 15 (1%) Formal employment 27 (1%) 19 (2%) 35 (2%) 18 (1%) Household engagement in land cultivation by Size of land. 0 – acres 678 (30%) 308 (25%) 426 (19%) 166 (14%) 0.1 - 0.24 acres 691 (31%) 428 (35%) 668 (30%) 313 (26%) 0.5 - 1acre 510 (23%) 327 (27%) 675 (30%) 442 (36%) More than 1 acre 348 (16%) 163 (13%) 465 (21%) 306 (25%) Household Livestock Ownership by type of livestock Cattle 284 (13%) 121 (10%) 300 (13%) 134 (11%) Goats or sheep or Pigs 617 (28%) 370 (30%) 968 (43%) 650 (53%) Small ruminants (Rabbits etc.) 55 (2%) 32 (3%) 100 (4%) 75 (6%) Chicken and Other Birds 754 (34%) 426 (35%) 1112 (50%) 696 (57%) Ox-traction (Oxen, Ox-plough) 74 (3%) 21 (2%) 67 (3%) 10 (1%) In addition to investing in petty business, there is a slight shift in the size of land cultivation by households, with more households now cultivating land sizes above 0.5 acres (748 or 61%) as compared to baseline (490 or 50%). Yet, households still refer to not having access to land as one of the key challenges faced29. Respondent feel that though cultivating land can help them with basic needs in terms of food, the land they have access to is not sufficient to sell beyond the household usage and as such 29 Food security is discussed under EQ3. viii cannot support wider financial needs. Though the quantitative data shows that more households have access to land, and that a majority of these own the land they farm, the qualitative data indicates lack of land to cultivate as a major issue still faced by respondents, in particular amongst destitute households. The baseline findings showed that destitute households were among the most vulnerable due to not owning or having access to land, whereas 80% of households classified as medium and low risk did own and have access to their land. Endline qualitative findings indicate that respondents perceive the financial package by DOVCU as instrumental in allowing households to access to land, both through the cash transfer as well as through VSLA. As stated by one respondent: “It was last year, they taught us before giving us the money. They told us that this money is for feeding our families and for any other income generating activity or start saving in a VSLA or buy birds and animals for rearing. We also asked them that what if we use part of this money for farming, they said that as long as we think that it is a profitable venture they don’t have any problem with it, but the major aim is to stabilize our households. However, they cautioned us that agriculture is prone to change in weather so they wouldn’t advise us to use that little money for farming” (Caregiver, Gulu). This element of agriculture as highly prone to weather adversity is highlighted by respondents in all districts, and many state instead that they try to invest in livestock to be able to also generate some disposable income as well as meet basic food needs. As seen in table 6, 1,565 of phase 1 households now own some type of livestock, as compared to 970 during baseline. The most substantial increase can be seen in goats or sheep or pigs, rabbits and chicken and other birds, as compared to cattle or ox￾traction. Respondents state that they would want to invest in larger livestock such as cattle or ox￾traction but find that their available funds are not sufficient for these purchases as it is perceived as a large and risky investment: “not much has changed because the cow we bought instead took us backwards. I am still the same as before I got the cash transfer because my savings were invested in a cow which was taken away” (Caregiver, Kabarole). The respondent explained that they had purchased a cow as an investment for milk and daily income through selling this milk, but that the person selling it was a fraud and as such the actual owner of the cow came and reclaimed it. There are several stories similar to this from the sample, and in each case, respondents have chosen to invest in smaller, perceived to be less risky, livestock since such as goats, chicken or rabbits. Overall, respondents positively connect the grants they receive from DOVCU with being able to invest in livestock and cultivation. As stated by one caregiver in Gulu: “I think we are still struggling, but there is a slight difference because of that money. I bought hens which I am rearing. I also have crops in the garden which we cultivated using that money.” To this extent respondents in all districts feel that the cash transfer has helped them diversify their livelihood to the extent that they are a little less dependent on one source either for income, or for basic needs. Respondents clearly link these decisions to invest to the training they received: “the trainings I received enabled me to do business. At the time I got the money the business was a bit more profitable than it is today. I bought one small goat at that time, and not it’s a big goat” (caregiver, Kabarole). Psychosocial vulnerability and barriers to good parenting As highlighted in the baseline evaluation, neglect or lack of awareness about how to provide proper care, supervision and attention to children in the household further increases vulnerability to separation. Baseline data indicates that poor psychosocial well-being of caregivers can moreover make a household more vulnerable to separation, and for these internal, psychological processes (e.g. feelings, emotions, perceptions) to be linked to the external, social contexts of caregivers30. 30 See further discussion in DOVCU: baseline report ix Second to the financial support, respondents highlight parental training and support as key for preventing separation, “they have been teaching how parents and children should behave to ensure a stable environment for all to exist. They said that if a child does something bad, find time and talk to him so they understand. Those are the things (DOVCU) has been teaching us, plus business” (Caregiver, Gulu). Table 7: Number and % of supported households participating parenting group support sessions; disaggregated by vulnerability type (70-116 scores or less on FSVI key domains) Sex / Age Total household assessed # and % participating in PG; All districts # and % participating in PG; Phase 1 districts All <70 70-116 <70 70-116 Total 2235 1396 (62%) 885 (40%) 511 (23%) 1228 (60%) 423 (19%) 309 (14%) Male 50+ 334 202 (60%) 135 (40%) 67 (20%) 212 (60%) 76 (23%) 51 (15%) Female 50+ 492 306 (62%) 190 (39%) 116 (24%) 300 (61%) 103 (21%) 80 (16%) Male 25-49 621 407 (66%) 280 (45%) 127 (20%) 305 (58%) 107 (17%) 70 (11%) Female 25- 49 670 408 (61%) 236 (35%) 172 (26%) 357 (60%) 118 (18%) 95 (14%) Male 15-24 70 41 (59%) 27 (39%) 14 (20%) 28 (54%) 10 (14%) 5 (7%) Female 15- 24 42 29 (69%) 14 (33%) 15 (36%) 21 (71%) 7 (17%) 8 (19%) Male 6-14 2 1 (50%) 1 (50%) 0% 1 (0%) 0% 0% Female 6-14 4 2 (50%) 2 (50%) 0% 4 (50%) 2 (50%) 0% Caregivers perceive the parental training to make them more aware of their responsibilities and duties as parents. For example, they felt they knew a more appropriate way to respond to their children during conflict, and perceived these new strategies as decreasing the chances of their children leaving. One respondent in the child protection mechanisms focus groups discussion, who is also themselves a parent, elaborated on this: “When (DOVCU) came I have been included as a member of a parenting group and I train other parents on how they should parent their children, (DOVCU) has changed my life. My husband divorced me and left me behind with many children but I have managed to single handedly look after the children and cater for them.” This quote highlights the sentiment across districts, where the parental training is seen to have provided caregivers with the tools to be ‘good parents.’ Contrary to baseline findings, where respondents perceived ones’ fundamental character and upbringing to significantly impact what someone was like as a parent, respondents now highlight that they themselves might have thought they knew how to parent, but training has taught them to view their relationship to their children in a different light. Caregivers state that while training has helped them in realizing this, the parents group aids in continuing to develop this and manage it in practice. One caregiver from Mbale stresses the communal support that parents groups provide: “This group is so good because it has helped us in developing the way we think and care for the children. It has done a lot in reuniting families [such as] counselling, advising other families to feed the children, stopping parents from mistreating their children. The main purpose of the group is to support families in avoiding child separation and reuniting children back to their families. During baseline, the emphasis was very much on a parents’ ability to provide and meet children’s basic needs, in addition to care and guidance. Interestingly, care and guidance were primarily highlighted by respondents in the endline findings, with ability to provide as second. At endline respondents stressed that one could be a good parent even if one was poor. Yet, caregivers also linked their ability to be better parents to the financial support, as exemplified by this quote from Mbale: “There are things that I x have done differently after the training I have received which include; I am able to feed my children twice in a day, buy clothes, provide books, provide treatment when they fall sick whereby buy drugs for them, I love my children because I am not rude to them and we work together with, I know how to save my money, I can buy for them meat and rice when they want to change some diet, I am able to work with my children well” All interviewees highlighted parental behaviour and abuse as a cause for separation, and neglect was particularly mentioned as a risk factor for step children within families. As during baseline, social norms and practices towards child protection were brought up as factors that may limit the harm to children that may result in separation through children running away or being removed. Parents treating their children harshly, make them work too much, abusing them, and neglecting them lead children to leave their homes. Parental training and home visits were viewed as decreasing violence in households and promoting the rights of children. Domestic violence was consistently identified as a risk factor for separating families, with family discord between parents and/or parent and children to lead to separation. Descriptive findings from the endline shows a slight reduction in the frequency of psychosocial and emotional distress in supported households, with the frequency or periodic signs of aggressive behaviours, domestic violence, child abuse and child neglect having reduced from 15% to 1% in phase 1 households. Additionally, the proportion of caregivers who use harsh disciplining approaches for children has slightly fallen, from 66% in baseline to 55% in endline (phase 1 districts). This is consistent with findings from the qualitative cases, where children and parents do report to still use harsher forms of discipline, though stating awareness that this is not always the best approach. Overall, respondents were able to explain what they had learned in training provided by DOVCU, even though they were not able of accounting for situations when they had used this in practice. Thus, it is not sure that although training may have changed parents’/caregivers’ awareness, that it had changed norms and behaviours regarding child care. Table 8: % of households reporting incidences of violence against children in their family in the last 12 months Type of abuse BL (P1) EL (P1) Targeted child is abused, sexually or physically, emotionally and/or being subjected to child labor or otherwise exploited 89 (7%) 8 (1%) Targeted child is neglected, given inappropriate work for his or her age, or is clearly not treated well in household or institution 193 (16%) 44 (4%) There is some suspicion that the targeted child may be neglected, over￾worked, not treated well, or otherwise maltreated 454 (37%) 401 (33%) Targeted child does not seem to be abused, neglected, do inappropriate work, or be exploited in other ways 485 (40%) 774 (63%) % of caregivers who uses humiliating or rudimentary approach (Punched, Kicked or hit, withheld a meal or basic needs, use abusive words/ language) to discipline a child in the past 12 months 802 (66%) 670 (55%) At baseline, monitoring data showed that the association between alcohol and drug issues and household risk status did not appear correlated, yet, respondents continuously highlight alcohol as a factor. In line with baseline findings, alcohol use was constantly mentioned as a risk factor to separation because this is perceived as linked with family conflict, abuse, and lack of financial income. Monitoring data show that 28% (349) of caregivers in phase 1 districts are enrolled in a psychosocial support group and are participating in peer support post treatment groups. Respondents across districts highlighted post-treatment groups and support as significant in improving household situations and preventing separation/enabling reunification. One CPM focus group discussed the effect the training had xi on his life as well as the lives of those he interacted with: “I am glad that (DOVCU) included me in this group. I used to drink a lot, but because of them [DOVCU] I have a good family. I am sending my children to school now and I am also ensuring that other families send children to school, that they stop drinking, they stop roaming around and I have done this because of what (DOVCU) has done in my life and for other youth of this community. Right now, I am a very smart man, I have also been able to educate other youth to stop drinking, to work hard and to provide for their families. (DOVCU) thank you for supporting mothers who are now able to feed children, to provide clothing, bedding and to educate them.” Indeed, monitoring data show that 1228 caregivers in phase 1 districts are taking part in post treatment groups, with 30% (373) of these caregivers showing reduced substance abuse. While direct causality cannot be assessed, findings from the qualitative data indicate that the ‘AA-groups’, as referred to by respondents, are highly significant in reducing alcohol consumption, and stressed as a key impact by the project. Table 9: Participation in post treatment groups Total number of households supported assessed at endline; % and # of caregivers (at risk and reunified households) participating in post treatment group and showing reduced substance use All districts Phase 1 districts All districts Phase 1 districts Total 2235 1228 672 (30%) 373 (30%) Male 1027 546 334 (37%) 177 (36%) Female 1208 682 338 (28%) 196 (26%) Detailed analysis of the effect of the integrated package of support on the nutrition, education opportunities, care, and protection for children in targeted households While the previous section focused on the perceived impact of interventions, this section focuses on the extent to which the integrated package of support and the prevention methods employed by the project increased food security and nutrition, education opportunities, care and protection of children in targeted households and as such, increased their well-being. The evaluation was designed to allow for comparison over time, in particular with regards to subjective wellbeing of vulnerable households, and of children in every group targeted by the DOVCU project. The available quantitative data assesses changes in children’s well-being following reunification, or in the case of children at risk: since baseline, i.e. changes to well-being while in family-based care. Thus, this section combines changes in well-being for children at risk, and reunified children, In at risk households, increased wellbeing can contribute to prevent separation, while in reunified households, improvements in well-being may be necessary to ensure that the child is fully reintegrated and is not put through secondary separation (e.g. going back to the CCI, leaving for the street, for another placement etc.) At the time of the baseline evaluation, children to be reintegrated had not yet been identified, there is a limitation in the study’s ability to speak to the wellbeing of these children specifically. Yet, the participatory exercise of assessing wellbeing across a number of dimensions over time (past, present and future) allows for children to give a summative perspective on the changes they have encountered before (while in a CCI, remand home or on the street), now (when reunified/reintegrated) and in the future. This allows for children themselves to assess contribution of reunification and reintegration on their wellbeing. Some of this will be discussed below. xii Table 10 shows the mean differences between baseline and endline in the vulnerability score of children based on all Core Project Areas (CPAs). The results indicate that overall, there has been a reduction in child vulnerability scores indicating an improvement in children’s wellbeing31. The most significant changes can be seen in Health, Shelter and Care (CPA3), followed by food and nutrition (CPA2) and education and skills training (CPA4). Psychosocial support (CPA5) and Protection (CPA6) also show an improvement, though slightly lesser so. Each of these core project areas are discussed in the relevant sections below. Table 10: Shows the significances in the mean differences in the vulnerability score of Children based on all CPA (EL - BL) – an indicator of changes in the wellbeing Diff in CPA scores EL – BL Mean diff Std. Deviatio n Std. Error Mean 95% CI of the Difference t dof Sig. (2- tailed ) Lowe r Uppe r EL - BL CPA 2 -1.336 3.754 0.064 -1.462 -1.211 -20.893 344 4 0.00 EL - BL CPA 3 -1.578 2.831 0.048 -1.673 -1.484 -32.721 344 4 0.00 EL - BL CPA 4 -1.293 4.33 0.074 -1.438 -1.148 -17.525 344 4 0.00 EL - BL CPA 5 -0.947 2.782 0.047 -1.04 -0.854 -19.982 344 4 0.00 EL - BL CPA 6 -0.694 1.743 0.03 -0.753 -0.636 -23.388 344 4 0.00 EL - BL CPA 2,3,4,6 -5.849 10.801 0.184 -6.21 -5.488 -31.783 344 4 0.00 Food security and nutrition The findings indicate that the financial support package has aided in households’ ability to smoothen consumption. Respondents perceive saving groups and diversification of livelihood as having contributed to enabling households to cope with shortages for longer periods of time. This appears to have translated into increased food security and nutrition in vulnerable households, with FSVI findings indicating that households have improved access to food and nutrition, with an increased number of meals households have a day. The great majority of households (77%) in phase 1 districts fall within the 2-3 meals a day category, as compared to a lesser majority 52% of households in this category at baseline. Importantly, the number of households having 0-1 meals a day has decreased significantly, from 560 households in baseline to 210 at endline. Yet, as with findings around economic vulnerability, shifts into the relative lesser vulnerability category (more than 3 meals a day) had seen the smallest shift, with a percental change of 4% (2 to 6). Importantly, these findings show access to meals per day during the worst month of the year. Yet, as with economic vulnerability respondents still perceive food security as precarious. Respondents across districts continuously refer to the volatile environment in which they live, and how this impacts on food security. There appears to be a fear of relying too much on agriculture due to weather adversities and as a consequence, households aim to invest in petty business or other livelihood 31 Negative means indicate improvement, considering the vulnerability score is less at endline than at baseline. xiii opportunities that are perceived as less precarious: “I can say our welfare is not stable, it fluctuates. Sometimes we are doing well and sometimes we are economically badly off. My husband is a trader of bananas. There is scarcity of bananas now. It is rare to find and we end up eating/using part of the business money. Our incomes are now miserable, there is scarcity of food, school fees for children in school is difficult to come by now. We have encroached and used money for business and spent it on home consumption” (Caregiver, Kabarole). The above quote is indicative of multiple households in the qualitative sample. Households still state there to be a scarcity of food, despite monitoring data showing households to be relatively better off. Yet, it is important to acknowledge that households do highlight that they have better access to food that before, in spite of perceiving there to be scarcity. There is an indication that respondents do not consider the financial support to have necessarily brought them out of food insecurity, but rather has aided in coping in the now. Respondents emphasize how the cash transfer helped them buy food and cater to basic needs, but not necessarily allowing them to invest in businesses or livelihood opportunities (or that they were able to invest, but the investment did not work out). As such, most respondents are wary of stating that their food security status is better, instead stating that it is “better right now” (caregiver, Mbale). Still, as seen in table 11 below, household’s access to food and nutrition appears to have significantly improved in phase 1 districts since baseline. The majority of households at baseline stated they frequently had less food to eat than needed, complaining of hunger (53%), and around 15% of households stated they rarely had food to eat, going to bed hungry most nights. On the contrary, endline findings show the majority of households (67%) to now have enough to eat some of the time, depending on season or food supply. Hence, while there is still a high presence of food insecurity perceived, and likely experienced, by households there are indication of a positive shift that respondents to attribute to the financial support component of the project. Table 11: Shows household access to basic needs status (characteristics) at endline compared to baseline characteristics Baseline Endline All districts BL Phase 1 districts BL All districts EL Phase 1 districts EL Household access to food and nutrition Rarely has food to eat and goes to bed hungry most nights 335 (15%) 182 (15%) 82 (4%) 35 (3%) Frequently has less food to eat than needed, complains of hunger 1218 (55%) 656 (53%) 706 (32%) 254 (21%) Has enough to eat some of the time, depending on season or food supply 585 (26%) 324 (26%) 1280 (57%) 818 (67%) Well fed, eats regularly 95 (4%) 65 (5%) 165 (7%) 121 (10%) Household # of meals per day in worst months of the year Had 0 - 1 meals 1167 (52%) 560 (46%) 641 (29%) 210 (17%) Had 2 - 3 meals 1028 (46%) 638 (52%) 1495 (67%) 942 (77%) Had More than 3 meals 36 (2%) 27 (2%) 99 (4%) 76 (6%) Household resilient source on food consumed in the households Donated 103 (5%) 49 (4%) 17 (1%) 6 (0%) Given in return for work 296 (13%) 153 (13%) 68 (3%) 34 (3%) Bought from the market 1118 (50%) 539 (44%) 1315 (59%) 668 (54%) Home grown 702 (32%) 477 (39%) 833 (37%) 520 (42%) xiv Household months of food shortage in the last 12months 6 or more Months 668 (30%) 372 (31%) 321 (15%) 144 (12%) 3 - 5 Months 1120 (51%) 629 (52%) 1274 (58%) 712 (58%) 0 - 2 Months 423 (19%) 214 (18%) 594 (27%) 370 (30%) Household shelter No stable, adequate, or safe place to live 469 (21%) 272 (22%) 115 (5%) 49 (4%) Place needs major repairs, crowded, inadequate, not weather protective 831 (37%) 472 (38%) 591 (26%) 251 (20%) Place needs some repairs but is fairly adequate, dry, and safe 678 (30%) 344 (28%) 1105 (49%) 646 (53%) Place that is adequate, dry, and safe 255 (11%) 139 (11%) 423 (19%) 281 (23%) As for children’s subjective well-being on food security, the majority of children in the qualitative case studies of at risk households perceive that they now eat better than before (two years ago). Children highlight diversification in the type of food consumed: “we eat good food these days and it’s always enough food at home. There is a very big change in the way we feed; these days we eat more nice foods like meat” (child at risk, Kabarole). Still, children are acutely aware of the precariousness of the food situation, referring to “it depends on if mom has money” and similar statements to explain that while the food situation is slightly better now than before, this is not always the case. Moreover, several children in households that received cash transfers highlight how there was food “for a while” (child at risk, Gulu) but not anymore. In many cases, the same respondents emphasize that their parents now have a small plot of land and that “we planted, but it is still in the garden, it is not yet ready” (child at risk, Gulu). Importantly, the cash transfer here appears significant in influencing children perceptions of food security and nutrition. Households in the sample that have received cash transfers are more likely to have children mention an improvement in food consumption, but equally to have children mentioning that this has not lasted. On the contrary, in households that did not receive cash transfers, children perceive food consumption to either have improved, or not changed. This raises questions regarding the ability of the financial support package to support the most vulnerable households, that may have spent a large amount of their cash transfer on meeting immediate needs and been less able to smoothen continuous consumption through the investment. For children who have been reunified, it is important to note that they reference two points in time in changes to their food consumption and perceived food security and nutrition. Children who have been reunified refer 1) to when they were in the previous CCI/remand home/on the street, 2) to when they were reunified, and 3) to when the endline assessment was conducted (‘now’). There is a significant variance in children’s responses around subjective well-being. Some of this appears related to household vulnerability status, with children returning to destitute households more likely to state that they now “eat less than when I was at the home” (reunified child, Mbale). However, children in this situation, similar to children in at risk households, highlight how the cash transfer provided by DOVCU helped their families buy food, in several cases “this food is no longer” (reunified child, Luwero). Children who have been reunified from the street or remand homes, consider their food security and access to nutritious food to have improved in the majority of cases, though for this to still be precarious: “there was a time when we completely did not have anything to eat, we had potatoes that we had planted but wasn’t ready for eating yet. But (DOVCU) came and bought for us some posho and beans” (reunified child, remand home, Gulu). Children reunified from these situations are more likely to indicate their improvement as relative, i.e. “it was bad there [at the home] but here also we go to bed hungry, but not always” (reunified child, remand home, Gulu). Importantly, children emphasise that while they may eat less regularly than when they were at the remand homes, they eat more of a balanced diet. xv This is similar to perceptions of children reunified from child care institutions. The data around children previously living on the street is more in line with children from destitute households, where the situation was bad and it is still not secure, but better. Importantly, when children previously living in child care institutions assess their subjective well-being they emphasise the regularity of meals prior to reunification. The majority of children who have been reunified from child care institutions perceive their food security and nutrition to have been better prior to reintegration but consider the relative situation at home to be better ‘now’ than at the point of reunification. This thus still indicates that the prevention strategies of the project with regards to the financial support package has, in the vast majority of cases, been able to improve the food security and nutrition situation within family-based care, though this however does not necessarily mean that this is overall better for children as compared to prior to reintegration. Education Opportunities As highlighted in the baseline evaluation, lack of access to basic social services such as access to good quality education is strongly associated by respondents to children separating from their families, in particular to go live in child care institutions. This ‘pull effect’ of child care institutions is important, as endline findings around admission to CCIs state school/education as one of the main reasons (together with poverty) for children being admitted into residential care. Baseline quantitative analysis showed how children in destitute and struggling households were more likely to frequently miss school. This is consistent with qualitative endline findings which see respondents highlighting ‘ability to pay for school fees’ as a major change in their wellbeing. Findings from the FSVI assessment show what 50% of households in phase 1 districts now have all children (of school age) attending school, as compared to 34% at baseline. Table 12: Shows household access to basic needs status (characteristics) at endline compared to baseline characteristics Baseline Endline All districts BL Phase 1 districts BL All districts EL Phase 1 districts EL Child access to age appropriate education; Pre-School / Basic Educ. None of the children is attending school 192 (9%) 100 (8%) 65 (3%) 36 (3%) Some children not attending school 1078 (49%) 601 (49%) 895 (40%) 507 (41%) All children attending school 738 (33%) 408 (34%) 1087 (49%) 612 (50%) Children not of school going age 196 (9%) 106 (9%) 183 (8%) 69 (6%) As discussed above, the financial package is perceived to have stabilized families, with caretakers able to purchase animals, invest in cultivation, or support small businesses. As highlighted by one woman in Gulu: “Since I am weak I can’t go digging in people’s garden so when I later got some little money I begun doing some small business and that is where I began paying them in school bit by bit. And later (DOVCU) came and supported me. . . I struggled with these children like that but the support (DOVCU) gave was the boost to me.” To this extent, the financial package is considered to have expanded options and opportunities within households. Primarily, respondents in the endline evaluation described a real increase in their ability to send their children to school, “I received this support one time but before this, I didn’t have food, sending children to school was a challenge, clothing them was difficult, but now with this support I am happy and two of my children have joined a secondary school with the support I received two times” (Caregiver, Luwero). All children interviewed (at risk and reunified) highlight education as a main aspect of their well-being, and this was similarly raised at baseline as a reason for separation. Several of the reunified children state that xvi the reason they left their parents/caregiver to begin with was so that “I could attend school” (reunified child, Mbale). However, while the majority of reunified children state they are now able to attend school following reunification due to the support from DOVCU, several children in at risk households stress that they are still sent home from school due to unpaid school fees: “it is school that is a challenge because I am always sent back home from school because of school fees” (at risk child, Gulu). Though not conclusive, there is some indicative evidence that this is more prevalent in households that have not received the cash transfer component of the financial package. Monitoring data for phase 1 districts shows that 65% of targeted households have children attending school regularly (4 or more days per week), as compared to 44% during baseline. While this is a positive shift, there is still a significant number of children who do not attend school on a regular basis. With the money for school fees, parents state they are able put an increased amount of money towards household spending on food, putting less strain on their food budgets and increasing the food intake of their children. Having to prioritise the cost of school fees and supplies at the expense of other household spending is stressed by respondents as a challenge, and thus all respondents who have received the cash transfer emphasis the effect this has had on their sense of wellbeing by no longer having to worry about school fees. This is further linked to the project with respondents perceiving a lack of education opportunities as a risk factor for children leaving. Importantly, as with food and nutrition, the sustainability of the contribution of the project on enhancing the number of children able to attend school is questionable through this reliance that respondents’ emphasis on the cash transfer, and comments by children regarding how they view their future schooling as dependent on continued DOVCU support. To this extent, for children who were separated due to lack of access to education, there is a question regarding whether the reunification will be sustainable if households are no longer able to cover school fees. Psycho social wellbeing, care and protection Respondents in all districts raised that the behavioral responses to situations of poverty and negative household dynamics are often impacting negatively on children. Negative coping mechanisms to the “stress” associated with poverty and negative household dynamics was highlighted by respondents to include alcohol and drug consumption, as well as sending children away. As with baseline, these factors where prevalent across districts, and respondents. However, while at baseline, women were more likely to see alcohol as a family issue causing separation, and men more likely to view it as an individual problem, male caregivers continuously emphasized how they had realized that their alcohol consumption affected the whole household “the drinking, it made me a bad parent, it made me a bad husband” (caregiver, Kamuli). Alcohol consumption has now reduced slightly amongst recipients who take part in post-treatment support groups. Respondents continuously highlighted these project interventions as essential in improving the overall well-being of the household. Notably, very few children in the sample raised alcohol consumption as an issue at endline, and there were low reports of violence in the household. However, this may be partially due to children feeling less comfortable to discuss these issues while at home (with their caregiver next door). Yet, child respondents across districts during baseline widely cited corporal punishment, or ‘overbeating’ as an accepted norm that can lead children to leave home. This was not highly prevalent in the endline findings amongst children in any of the respondent groups. The majority of children at risk and children now reunified all cited violence against children to have been something they experienced in the past (in particular children now reunified), but that this was not something they worried about now. When children did refer to violence, this was mainly in reference to fears of violence such as “if you get robbed” (reunified child, CCI, Kamuli), “some older people in the community may beat on you” (at risk child, Luwero), “there are stories of people that take children” (reunified child, CCI, Mbale). However, children mentioning these fears of violence said they did not know anyone who had experienced this. Similarly, none of the children xvii spoken with report experiencing harmful traditional practices. Some children mention that these are present in the community, but to not have experienced these. Yet, this is something that some of the children who have been reunified now state to fear, which they did not while at the CCIs: “I fear some relatives because they believe in witchcraft and they can be harmful” (Reunified child, CCI, Gulu). During baseline, the qualitative component of the evaluation noted the particular widespread absence of mental health services. This is highlighted at endline as somewhat addressed through DOVCU, in particular through help from para social workers. Caregivers refer to para social workers as “listening when I need to” (caregiver, Mbale), and “they even guided me to a group” (caregiver, Kabarole). This perception of respondents of para social workers as fundamental in helping them cope with stress was further highlighted through a vast majority of caregivers stating that they would turn to a para social worker if they were feeling emotionally distressed. However, para social workers still emphasize, as in the baseline evaluation, that there are challenges with living in a hard to reach areas where accessing services becomes difficult both for caregivers and community members themselves, but also for para social workers to connect people to appropriate services. Some para social workers indicated that they have to “listen a lot” and that this can at times be overwhelming (para social worker, Mbale). Respondents highlighted how the support they receive in dealing with emotional distress helps them be a better parent. As with the violence above, children perceived their experiences of stress to be either the same, or better at the stage of endline. Some of the qualitative findings indicate that children who have been reunified in many cases are less responsive than children at risk who have remained with their families. The data is unable to unpack this fully, but it is important to keep this in mind when considering the findings presented. Respondents furthermore consider the parental training was instrumental in improving child protection both within the home, and within the wider community. Caregivers highlight how they “spend more time” with their children, and this is echoed in the responses from children themselves. Most children spoken to now list their mother, father, grandmother or grandfather as their key important relationship. In some cases, children who have been ‘reunified’ are still attending boarding school. These children thus report similar challenges as children in institutions at baseline, feeling they are not always able to speak with the person closest to them “as they are not here” (‘reunified’ child, CCI, Mbale). Most children stated that their parents/caregivers provide adequate care for them and perceived their overall care and support to be better or the same. Overall, monitoring data show that 52% of supported children (both at risk and reunified) in phase 1 districts show reduced psychosocial distress based on the child status index of core project component 5 (psychosocial support and basic care) and CPA6 (child protection). However, findings indicate that psychosocial distress is slightly more prevalent amongst children with disabilities. This was not evident through the qualitative sample at endline, but disability was emphasized in the baseline research as a factor increasing stigma and affecting overall care and support for children. Children in all categories highlight that they feel less stressed than two years ago/prior to reintegration. Children in “at-risk” households the majority of times link this to “no longer being chased from school” (child at risk, Kamuli), and “we have some money for food” (child at risk, Mbale). With children who have been reunified, while they emphasize feeling less stress with regards to experiencing violence, or maltreatment, as compared to prior reintegration, still perceive the situation in family-based care as stressful. In particular, children who have been reunified from CCIs emphasize that the stress is less at home now than at the point of reunification, but that they feel stressed about money running out, or accessing services, such as education, something that they did not worry about as much while at the CCI. Findings indicate that children who have been reunified from CCIs perceive their access to services and food to be more precarious following reintegration, and for this to add stress. xviii Lastly, due the work of para social workers and CPMs, community members perceive an increase in the level of protection of children within communities. CPMs largely credited DOVCU’s parenting training with reducing children’s dropout from school and separation from families due to increased knowledge on child protection: “I received training from (DOVCU) and it was so good that they added me more knowledge on child protection because I have worked with the community for a very long time now, this was important and because of the training, I have been able to reduce on the number and rate of school dropout by the children, some parents even when their children go wrong they want to beat the children and hurt them but I have been able to sensitize them too with the knowledge I got and as a Child Protection person in the community and children plus parents have started behaving well because they were able to listen to me” (CPM, Gulu). Summary comments The nature and prevalence of vulnerabilities identified in the baseline evaluation, pointed towards a need for services that support vulnerable households in overcoming challenges and becoming more resilient to risks which may lead to separation. Overall, respondents perceived the combination of the financial support and training as helping to produce both immediate changes in household financial health as well as continued changes in the psychosocial wellbeing of the household due to trainings and resulting changes in household dynamics. Respondents furthermore felt that the financial support and parental trainings resulted in more supportive household environments that enabled children to stay with their families. The project seems to have achieved some smoothing of income through the financial support, with households now being able to save some income to access in case of shocks. Yet, the financial support is perceived as insufficient for sustained income-generating activities at this stage. Respondents explained how they have spent most of their support on addressing immediate needs and been less able to invest in sustained activities. Psychosocial and parental support is equally perceived to have targeted needs in the community, with highly positive responses from caregivers and the wider community. Parents/caregivers are clearly more aware of alternative norms and attitudes towards child care. Yet it is important to acknowledge that at this stage, it is difficult to assess norms and behavioural change due to the short time-frame of the projects. In addition to this, the financial support component appears to have enhanced household’s abilities to cope with some aspects of vulnerability that increases the risk of separation. It seems to have contributed to improvements in the access to basic services, and specifically, as illustrated by the findings from the qualitative component of the evaluation, that the support enabled children to go to school and to have an improved nutritional status in households that were supported by DOVCU. Similarly, findings indicate that respondents (including children) largely perceive the family situation to have improved in terms of psycho social well-being, child care and protection. Yet, there are some differentiations in between categories of children. Children in “at-risk” households show the greatest shift in wellbeing, as compared to children who have been reunified. Moreover, children reunified from remand homes, or from the street show a greater shift in well-being than children who have been reunified from child care institutions. Children who were reunified from child care institutions stress the perceived precariousness of access to basic services. All in all, a key conclusion is that if the integrated package did not have full effect, it is felt that this was not a problem of design of the intervention, rather an issue of sufficiency in time and money to allow families to graduate to a status where the family income was growing sufficiently to create resilience against shock and unforeseen events. Some stakeholders also suggested that the package could have xix included a few more elements for which poor communities have a need. This furthermore leads to some questions of sustainability around changes in financial vulnerability, in particular. It is unclear if the positive developments created as a result of the financial package will cease or begin to regress when the financial support ends. xx ANNEX II: LESSONS LEARNED ON THE PROJECT’S WORK WITH CHILD CARE INSTITUTIONS Background When DOVCU started, most CCIs were operating according to their own rules and priorities and were disconnected from the formal system of oversight. There was little awareness amongst CCIs about the national government policy, the alternative care framework, and little accountability to follow it. When the process of assessing CCIs started at the start-up of the DOVCU project, and the project was introduced to CCIs, some CCIs immediately embraced the project’s goals, others reacted with fear and suspicion. Overall, at baseline, there were an important number of CCIs that expressed concern and reluctance to be involved. CCIs for example expressed fears that reintegration processes would be forced upon them and the children. Some CCIs were suspicious of the DOVCU NGOs-consortium and were questioning the fact that other NGOs “were coming to impose decisions on the CCIs” on matters which were not of their concern. Overall results of collaboration with CCIs To a significant extent the project managed to raise awareness of CCIs and contributed to a change in attitudes amongst them in favor of supporting children in their families first and foremost. The project has also managed to create positive attitudes amongst this important stakeholder group towards engaging not just as an alternative care service provider receiving children, but to collaborate with other stakeholders within a larger alternative care system to first and foremost prevent the entry of children into care. As has been detailed in other sections of this report, child care institutions that were involved in the project, have to an important extent changed their admission, care and reintegration practices, something which has contributed to fewer children being placed in residential care in the DOVCU districts, as compared to baseline. Facilitating factors One of the things which made CCIs change their minds, was the fact that the trainings provided by the project opened their eyes to the fact that they were breaching government laws and policies. A former CCI director in one district reflected on this eye-opening experience with the following words: “I received visits from DOVCU and the office of the probation, they talked about the situation of the children in the orphanage and convinced me to resettle them. There were issues because the beddings were not enough for all the children, and also by that time the dormitory was not ready, I used to have all of them stay in the family house. My orphanage was not up-to-date, I had not yet registered, and the process that I went through to get these children was not proper.” Another factor that has facilitated CCIs in embracing the change proposed by the project has been the readiness of funders and founders to allow the CCI to use funds for supporting reunified children in their families instead. A significant number of CCIs that engaged in supporting the reintegration of children, with more successful results, had provided support to children’s education, health and other needs, once children were reunified. It was not all CCIs, however, that had this capacity. One CCI-director reflected on this: “Although it’s not yet implemented, we are also thinking about changing the approach, it is still my idea that I am yet to sell to the donors of identifying children and supporting them from their communities, this somehow reduces on the expenses of the home, for example money to keep one child in a month can be used to support three children in the community. We have registered as an NGO but we have not yet gotten our approvals.” Changes in practices of child care institutions: xxi With few exceptions, CCIs across all the researched districts have made significant progress in improving their policies. The evaluation team had access to copies of human resources policies, child protection policies, resettlement policies, emergency procedures which had been developed since the beginning of the DOVCU project with support from coaching activities implemented by the DOVCU project consortium partners. In some cases, the CCIs were explaining how staff and children had also been involved into the development of these policies (although this was not the most common practice). Policies did not just exist on paper, but a number of changes in CCI practices and processes were also documented during the visits to CCIs selected in the evaluation sample. The evaluation team documented, for example, that some CCIs has established management committees, other CCIs had made improvements to their facilities to better comply with policies. E.g. in one CCIs visited changes had been done to the dormitories and a new building had been constructed to create separate dormitories for girls one boys. In the same CCI, the space was increased for each child and privacy of each child improved as compared to baseline. Across all the districts, many of the CCIs visited have better human resource capacity as a result of hiring social workers and/or recruiting more qualified staff compared to prior practice. Some CCIs had changed their management. Improvements in human resource capacity was also to a large extent attributed by stakeholders to the skills building activities of DOVCU that had involved management and staff. Amongst CCIs there was overwhelming appreciation of the fact that this training was the first time they were involved in a purposeful and thoughtful training on alternative care, making them see their role in an overall system, but also learning about their specific accountabilities in case planning and alternative care of children. The following quote for a director of a CCI provides a good illustration: “It (DOVCU) emphasized organizations to employ qualified workers, this has happened and such a person plays their role very well, not like the former director. The project followed up on this. The reduction in the numbers, the child situation has changed seriously, the issue of resettlement of some children from this home is attributed to the project. Through the trainings and sharing ideas with others, I came to appreciate the need for resettling children. Using the (DOVCU) training, I was able to convince the donors to take up resettlement, so now they are aware that the children are now here for only a short period and they will have to be reunited with their families.” Even though many CCIs have strengthened their HR capacity as compared to baseline, there is still significant differences between CCIs. If at baseline there were many CCIs that did not have any social workers at all, now there are more CCIs that employ staff to perform social work duties, such as being involved in tracing and reintegration activities, in developing, reviewing and updating case files. However, qualification levels of the social workers now available in CCIs, vary a lot. From casefile reviews and discussions with CCI-social workers on the follow-up they provide to specific cases, it was evident that many social workers who are available in CCIs do not have advanced skills in, for example recognizing delays in child development, or signs of stress in children who have been reunified. For example, in one of the CCI, perceived as the “best practice” in that district, casefiles for several reunified children documented “ulcers” which, upon probing from the evaluator, was defined further by the social worker as “stomach aches.” In none of these cases did this condition trigger any suspicion or follow up of eventual psychosocial issues linked to the reintegration experience, even though the medical check-ups were not able to confirm health or nutrition issues as a cause. Across all study districts, the evaluation was able to confirm that many CCIs are implementing better gatekeeping procedures. Interviews with different respondent groups (PSWOs, CDOs, CCI directors and social workers, TAC members) confirm that these improved “gatekeeping procedures” often involve significant changes to admission practices and reintegration practices, and to what the CCIs establish as the objectives of children’s stay. For example, across all districts, CCIs describe how xxii they first try to counsel parents who bring their children to a CCI, that home is the best place for children and advise what they can do to take responsibilities as parents. Furthermore, as compared to the situation at baseline when many CCIs did not have any admission criteria and some were recruiting children from communities, various respondents’ groups now confirm that CCIs have stricter admission criteria and are only receiving children who are assessed as the most vulnerable. The following quote from a responsible person from a CCI in one of the districts illustrates what was echoed by many CCIs across all districts: “Gatekeeping has improved – maybe we have had 20 come for admission but we have admitted only 5 of them – before we would have accepted all (…) Before DOVCU the first question was do we have the money to care for this child? Now it is “does this child really need to come here”?” Furthermore, stakeholders across various respondent groups, and case files for children placed in CCIs, confirm that many CCIs now make more systematic use of the FSVI tool to evaluate which children meet the criteria to be admitted to the CCI. Several CCIs across districts are describing the admission of children, not as a one off event, but as a process involving several steps and including care planning with an objective of reintegration established already at the stage of admission and initial case assessment: One CCI director described this process as follows: “Now each time a child is admitted, from day one we should do resettlement plan so we already have a vision for the child (…). The child passes through a process. The PSWO recommends him to come here. Then the child should also have a letter referring the child here. This can be the LC1 chair recommending the placement, or a church-worker or a government worker at sub-county level. We ask for a death report, a birth certificate (…) Once a child is received we take the history of the child, like circumstances of the child and who is the next of kin. This information is for the resettlement plan decision. If the child can be cared for by a relative, we enroll him for the outreach program that can last for 6 months. Then the caregiver gets training here on nutrition and health.” CCIs refer to their residential care for children to “be more temporary”, and to “children staying for shorter periods of time.” This is confirmed by other respondents too. In one district the PSWO, for example expressed what was felt by many other stakeholders in other districts, that “CCIs understand that they are a temporary solution for children.” If CCIs had experience of children leaving their CCI also in the past, many CCIs have speeded up their reintegration activities and are more actively involved in preparing the families to receive children back. DOVCU has thus contributed to making CCIs being involved in reintegration of children at a larger scale today as compared to before the project started. In one CCI, a social worker described how he was involved in training and preparing households for reintegration of children: “Training of the homes was done by me, social worker. To prepare them I used a tool to identify what is missing, as well as what to eat and what to grow (…) My estimation is that takes about 2 years to prepare a family to be fit to receive a child.” It was difficult for the evaluation to objectively confirm to what extent and how DOVCU has contributed to improvements in care practices of CCIs, but several CCIs referred to such improvements. One CCIs, for example, felt that they now pay more specific attention to individual needs of child: “There are so many changes that have occurred and continue as result of DOVCU, one of them is about the screening of the children. Before the project, we did not mind about categorization of cases. You could find that an orphaned child was together with one with multiple disability. During the training we appreciated the categorization and individualizing of the children and what are the specific needs of a child. For example, (name) has a lot of disabilities, not standing, no feeding, does not feed with physical food. Many of the children used to miss special attention (to their specific needs).”.” Many CCIs referred to their children now having better connections with families because of bonding visits and because of children going home for holidays. Even though the quantitative endline assessment established that improvements in care practices had been made in many cases, casefiles available for review in CCIs are still not at a level where it is possible to evaluate to what extent children have their individual needs met. xxiii All of the above said, the evaluation also documented that not all CCIs changed their views around their role in the alternative care system, and the level of engagement with the project varied. As also confirmed by the DOVCU endline-assessment of CCIs against alternative care standards, the evaluation confirmed that in all CCIs visited as part of the evaluation sample, there is room for improvement in at least one, and in many cases in several standard-areas and most CCIs are yet to receive their license to operate. The attitude of the director and of founders/funders of the CCIs seem to have been of significant importance to the extent to which each CCI embraced the project and the change it proposed. Attitudes to this change also seems to be heavily influenced by the funding available to support changes in practices. xxiv ANNEX III: LESSONS LEARNED ON THE PROJECT’S WORK WITH DIFFERENT PROFESSIONALS Background In order to be able to facilitate the reintegration of children and to prevent separation of children from families the project was to contribute to “systemic changes” in the 12 project districts. Projects documents are identifying the priority to support child care institutions to “re-conceptualize their role”, to strengthen gatekeeping of child care institutions and other agencies, and to support the development of a continuum of care services as a contribution to implementation of national alternative care policy (ChildFund, 2013). This was to be achieved by raising general community awareness on child protection and building the capacity of community volunteers such as para social workers and community-based child protection mechanisms. In addition, the use of vulnerability assessment tools the CSI and FSVI, aimed to strengthen identification and response to vulnerable families and children. Targeted household economic strengthening activities such as cash transfers, financial literacy training, micro enterprise, VSLA groups were used to support destitute households. A number of manuals were developed to guide the work, including a VSLA field officer training guide, a manual for financial education, a household economic strengthening guide and a manual for parenting learning sessions. DOVCU developed the guide, Working with Child Care Institutions: A strategy for the engagement with child care institutions to improve practice as a way to encourage consistency in oversight of CCIs. The strategy relied heavily on training as a method to build capacity and involved professionals at all levels: Staff at the MoGLSD and the Alternative Care Unit, PSWOs, members of TACS, CDOs, para social workers and members of community. As one district level stakeholder appreciated: “DOVCU brought the entire force (government, NGOs, child care homes, community) behind the effort.” Experience of establishing “Teams Around the Child” TACs were established at different times in different districts between 2015 and 2017 and played a role in reviewing casefiles from the CCIs to decide on which children would be reintegrated. The idea was to diversify this responsibility which previously was mainly held by the PSWO or CCIs. Given that these teams were established some time into the project implementation, not all cases of children who were reunified as a result of the project, were reviewed by a TAC. TAC was set up as a multi-disciplinary team, as per the TORs. In some of the districts the members were appointed from existing DOVCC. Chairmanship varied between districts. In some cases, chairmanship was taken by the DCDO, in other by the PSWO. Other members most often included representatives from education and health departments of the district community service; the police in charge of child and family protection unit and a representative of a CCIs. Scope and representation The scope of TACs varied slightly between districts. Although TACs were supposed to mainly be in charge in reviewing and approving reintegration cases, in some districts TACs were also involved in assessing child care institutions and providing recommendations for improvement and approval. Furthermore, in at least one district did the TAC also, at its own initiative, get involved in approving cases of children to be placed in foster care and adoption. Overall, it was felt that 7-10 members was a good number, and that the current representation included most of the relevant disciplines. However, some stakeholders have suggested that TACs would have become more effective if they had included some expertise in child psychology and legal affairs. xxv Results of TACs It is felt by many stakeholders that the TACs established in DOVCU districts served to pilot the design and operations of gatekeeping procedures for cases of reintegration. To-date, there is no plan in MoGLSD to sustain TACs under their current name, but possibly to absorb these as district ACPs once the (draft) national Guidelines on Establishment of Alternative Care Panels in Uganda (supported by DOVCU in 2017) have been finalized and formally adopted. In an important way the pilot experience with TACs is feeding into these national guidelines and will allow for replication of these structures across Uganda. In summary, even though TACs are not recognized as a formal structure by government they are seen (as one national stakeholder put it) as a: “fantastic contribution” to “pilot gatekeeping through TACs.”.” Furthermore, TACs served an important convening role and facilitated inter-sectorial collaboration. In the past, coordination between departments in community services at district level has not been common practice. Each department is used to work independently. With the establishment of TACs different sectors within the district community service realized that all their work feeds into the wellbeing of children. In several districts, TAC members for the first time made home-visits as a team to families who were supposed to receive children for reintegration or had received children for reintegration. In one district, a representative of the TAC expressed his view on the team-spirit that had emerged as a result of the TAC, with the following words: “The TAC really is a team – all of us offer a bit different services and so we can see how the child can be best served.”.” TACs facilitated a better understanding of roles, responsibilities and mutual professional respect of various professionals and as a result, facilitated referrals to health and education departments: In one district, a CCIs representative on the TAC reflected on this ability of TACs to mobilize different sectors: “TAC is able to mobilize people to go to families if more is needed like health reports, family situation, community support and community organizations, education plan. In another district, another CCI representative appreciated how TACs had helped shed light on the difficult aspects of alternative care work, something which generated mutual professional respect: “TAC helped people to appreciate some of our challenges – through the project people did some tracing and realized that our work is not so easy.”.” Many policy level stakeholders and members of TACs themselves feel that TACs helped ensure that reintegration cases were better managed and went through due process. Unfortunately, there is no monitoring data available to compare if there are any differences in wellbeing and reintegration status of children who were reunified through a process that was quality assured by TACs and those who were reunified without TACs being involved. However, many stakeholders across districts reflected on the fact that there had been a difference, and that quality of reintegration improved with the involvement of TACs. As one district authority puts it: “TAC has helped to regulate reintegration in the district”, and in another district, the PSWO reflected on the fact that TACs had made the process more thoughtful: “TAC allowed us to say “hold on, let’s take another look; let’s make sure this can work.”.” We looked at why the child went into the child care institution and how they will be safe coming out. Since TAC there have been no failed reintegration.”.” Challenges The effectiveness of TACs has varied between districts. All seem to have met regularly, but in some TACs the team spirit and team work has developed into something very strong, while in other districts, specific members are taking a great role than other members. A factor that seems to have affected the effectiveness of TACs is the membership, who chaired it, how rapid the turn-over of members has been xxvi and the overall commitment that was built amongst the members over time. If, for example turn-over of members was high, the team did not function so well as a team but was rather driven by activities of specific members. In other cases, even though turn-over of members was not high, specific members were dominating and this seems to also have influenced the overall direction taken by TACs. In some district there seems to have been confusion around the specific role of the TAC vis-à-vis the role of specific individual members of the TAC. Beyond the specific meetings when TACs got together to review cases that were being suggested for reintegration, TAC members in several districts have not been distinguishing between their specific professional role and their role as a member of the TAC. For example, PSWO and CDOs in particular often describe their role in following up on the CCIs on the CCI assessment recommendations as something they have done as members of TAC, as well as the follow up they have done in specific cases. But it seems to not always have been clear if/how this follow should be fed back to the TAC as a team, and whether the specific activity was undertaken by the PSWO and CDO as individual professionals. Also, to other members it has not always been clear what should they do as a team, what individual members should do as part of their job, and how to report on that professional role to the TAC or vice-versa. Even though some stakeholders stated that case documentation had improved as a result of TACs the evaluation could not find any evidence of TACs having played a role in improving case documentation. This “improvement” seems to refer mainly to the meeting files which contained papers on the cases that were prepared for TAC members ahead of each meeting. However, these meeting files are not easily available (people talk about them but only in a few cases was the evaluation team able to see the files) and they do not seem to be in use for post-reintegration review and monitoring of cases. There is not a shared understanding amongst stakeholders, and even within the DOVCU project consortium, on what is the ultimate measure of impact or success of a TAC. Several stakeholders, including TAC members themselves, seem to understand their impact and success by the simple fact that children are reunified. Their explicit purpose is understood as quality assuring the steps in the process of reunification until the time when the child is placed within a family. The role of TACs post￾placement is less clear and their involvement in monitoring how children are doing after reunification and in deciding on case closure following reintegration, is negligible, as compared to their involvement in the preceding steps prior to the reunification of the child. Nevertheless, TACs’ seem to have been an important structure to demonstrate “the process to follow to safely reintegrate children into families”, as one stakeholder put it. Experiences from the work with district, sub-county and community professionals Something that summarizes the experience of the project’s work with different professionals is that the project has made district, sub-county and community professional’s work on alternative care more systematic and connected. In one district, the PSWO summarized well a sentiment that was echoed across districts by different professionals, with the following words: “I used to be involved in alternative care work before. Like before the project, if there was child who was found in the villages abandoned we used to take care of the child. But the project has made this work more systematic. Now it is not just me alone doing this work, there are many others involved. There are the CCIs, the communities, the families and also the children.” Professionals furthermore felt respected and are proud, both of the project and their own role in it. In one districts the PSWO described this with the following words: “One lesson has to do with the issue of coordination that is very important. In this project it was demonstrated that through coordination, people with xxvii specialized skills, if they bring it to the table, everyone benefits from these skills. (TPO, RETRAK, Child’s I Foundation, ChildFund), each played a specific role. We had the best dancers in the hall and the project moved very well with that. If you discuss with leadership at sub-country, you may think that all of us have been project officers… The issue of the involvement of all formal stakeholders is very, very important. So, if I am talking about these things it is because some of us did not experience these things before. All of us are proud of the output and we are yet to be very proud of the outcomes of this project." Results of the project on professional skills and practices As a result of DOVCU, awareness raising, skills building and training activities, the evaluation documented the following changes: There is a greater knowledge amongst professionals of risks, response options and roles in response. This is evidenced in interviews with various respondents across the study districts who converge around the view that professionals, such as district officials, DCDO, PSWOs, CDOs, staff in CCIs, community child protection mechanisms and community volunteers have better knowledge the risks families face which can lead to separation of children. As such the project has generated an agreement amongst stakeholders on what the priorities in the response to these risks are (prevention over institutionalization of children) and a greater understanding of the respective professional roles in the referral and response. A member of a District OVC Committee illustrated this finding with the following reflection: “The responsibility of all actors is clearer. Child protection is not just about being good Samaritan. Now we know to ask about the para social workers and what they can do to help in child protection issues”). A CDO in the same district expressed similar views: “Now everyone is better at knowing who does what cases, and we are getting the more pressing issues at our level and have more time for them.”.” In another district another district OVC Committee member explained how the role for CDOs and para social workers had not been clear before the project, nor had the working relationship between the two been clear. But the training provided by the project in referral, child protection and reporting mechanisms, it is felt, has helped to clarify their respective roles. Even within the LC1, knowledge of their roles in child protection and alternative care is strengthened: My knowledge on child protection and on referral pathways increased” (DOVCC member) The project contributed to strengthen the technical skills of staff in MoGLSD and district staff (mainly PSWOs and CDOs) to implement standard procedures for inspecting CCIs. While Approved Child Homes Rules and the Assessment toolkit were already in place prior to the project’s start, DOVCU has trained and provided technical assistance to the MoGLSD team and the professionals charged with overseeing CCIs in their districts, in the use of the toolkit. For assessments of CCIs. Consequently, the MoGLSD staff could be heavily involved in the DOVCU baseline and endline assessments of CCIs. Each PSWO office in the researched districts for the evaluation has been involved in such assessments and have forms available in their office from these assessments. The following words from a high-level MoGLSD official illustrate the satisfaction of MoGLSD and district professionals with how DOVCU have skilled them in this area: “The technical skills in the government to accompany and lead such processes related to inspections, accompanying CCIs to improve and to support reintegration of children, is there now.”.” The project has generated improvements in professional skills to facilitate processes for the reintegration of children from CCIs, from the street and from remand homes. This finding is valid for several different professional groups, including CDOs, PSWOs, DCDOs, social workers in remand homes, CCIs, social workers in the CRO in Mbale. A testimony from a CDO illustrates: “The project built the capacity of the CDOs to follow up the resettled households including the children from CCI and CRO” (CDO). xxviii Stakeholders across various respondents’ groups and districts also converge around a view that the project’s training and skills building activities have improved professional skills of PSWOs, CDOs and para social workers to provide specific services to families and children. According to the District Chief Administrative Officer, the highest officer in rank in one of the districts, “PSWOs and CDOs have become more serious” as a result of DOVCU. This view, and the importance of DOVCU for “skilling up Uganda” (Assistant CAO) was echoed also in other districts. The skills that have been quoted by various stakeholders, including these professionals themselves, as the skills that have professionalized these workers, include different skills to support vulnerable families and families at risk. Professionals for example feel empowered with skills to educate families and community members on parenting, microenterprise development, financial management, child protection (training rolled out to para social workers, CPMs and parents/caregivers); on the importance of family care for children, and in providing psycho-social support to vulnerable families, to help with problem solving within the communities, and to help set up VSLAs. A para social worker noted: “From the parenting training we even learned what are our own gaps – from this we can counsel parents – parenting does not separate between rich and poor.”.” The person in charge in a remand home noted: “I learned how to handle children. Very interesting to work with them, the problem solving in the community, with complete collaboration, school, community, church, everyone. Different families have different perceptions, we need to work with changing their attitudes in order to not push their children away.”.” Changes in professional practices of district and sub-county professionals As a direct contribution of DOVCU’s involvement of district staff in assessing CCIs, there is now more regular and systematic oversight of child care institutions by district authorities. In the researched districts, district professionals have carried out at least two comprehensive assessments of CCIs, and in several cases have done follow-up visits and partial assessments of CCIs to follow up on previous recommendations from assessments. However, even if this assessment frequency is an improvement as compared to district’s involvement in providing oversight of CCIS prior to the project’s start, the national standards call for an assessment of CCIs every 6 months. While CCIs in the researched districts reported continuous and regular interactions with the district officials especially the PSWOs, there was no evidence that this interaction was about formal assessments or a reassessment of CCIs. Furthermore, it seems to not be fully clear, neither to district professionals nor to CCIs, whom amongst various stakeholders are responsible ensuring that CCIs receive the “approved status” once they have complied with national standards. Furthermore, districts and national MoGLSD are not systematically involved across districts in ensuring that CCIs are closed or investigated if they are found non-compliant or involved in irregular activities. The evaluation came across examples where CCIs scored poorly during the baseline assessment and did not initiate any improvement work, and this did not trigger any targeted actions by the district authorities or the MoGLSD towards these CCIs. Across districts the relevant professionals seem to be using the government approved tool for assessments, however the quality of documentation and record keeping on CCIs varies between districts. In one of the districts, for example, the PSWO holds files for all CCI assessments. Forms are signed by various people from the inspection team and files contain “improvement plans.”.” But there are also examples of other districts where there is divergence between verbal accounts on the number of assessments conducted, and the number of documented assessments and reassessments to be found in record keeping. In none of the districts are professionals themselves doing any systematic analysis on the progress made against various improvement areas that have been changes in professional’s involvement in supervising CCIs has generated evidence on the strengths and weaknesses of the CCIs operating in each district, even if they are not yet fully equipped to act on these findings. As one PSWO put it: “Before we were not doing the routine supervision and that is why the conditions were very bad. I at least now know that every CCI has some policies in place.” In another district, the DCDO noted that: “We were able xxix to ensure that we get to know which the child care institutions in the district are because we mapped them all (…) so now at least we are consistent in what we are doing when it comes to the child care institutions. We are able to monitor them and supervise their activities from a known point of view. We know what to do.” District professionals apply stricter gatekeeping procedures for referrals of children to CCIs. In one district, the PSWO noted that: “before when we were receiving almost every child for whom orphanhood was claimed, we just sent to CCIs without establishing the facts on ground; without social inquiry.”.” In several districts, stakeholders note that, to a larger extent now than prior to the project’s start, referrals to CCIs go through the PSWO. In a couple of the districts, PSWOs and CCIs now hold regular coordination meetings to discuss referrals. Stakeholders from the majority of researched districts also note that the cases that are admitted to CCIs are now the more “difficult” cases of child abandonment, such as for example when parents cannot be traced. The project has contributed to better identification of vulnerability and risk and a better functioning referral system between the four parishes, two sub counties in each district where it has implemented its prevention work. Across all 6 research districts, a wide range of professionals and stakeholders (CDOs, PSWOs, DCDOs, DOVCC-members), express the sentiment that this improvement is linked to changes generated by DOVCU in the awareness, skills and knowledge on child protection risks of community structures and para social workers, and of sub county professionals and PSWOs at district level. Community structures and volunteers are reportedly now better able to solve some of their own problems and know better when to refer cases up to the sub-county or district level. A CDO in one district expressed this with the following words: “We have structures who are our eyes on the ground (para social workers). We use those structures now and we have been doing more work to strengthen these structures to better understand our role and their role. They are then better able to support the family in the community” (CDO). In another district the DCDO noted: “Yes, especially now I will look at the para social workers. They can identify child rights violations and report them. That has been very good, they are giving information. When it comes to the Community Development Officers at least they are able to go and make follow ups; they are able to go and monitor; they are able to give direction on how to handle some of these issues in their sub counties because of the training that they have received.” In yet another district the PSWO felt that DOVCU had contributed to a “strong partnership between government and non-government organizations to transform the community.” And that “they (DOVCU) really bridged the gap between government, district, local, community and NGO” (PSWO). Furthermore, the availability of new tools for systematic assessment of vulnerability and more regular and higher quality home visits to families have led professionals and community stakeholders to being more systematically involved in alternative care cases and processes related to the reintegration of children in communities. A CDO in one district reflected that: “Our work has changed because we are in the community much more.”.” As a result of all these changes several stakeholders across districts have expressed a feeling that cases of abuse and neglect being received, registered and monitored in a more timelier manner. A PSWO in one district was of the opinion that in the sub-counties where DOVCU did not work, there was a lower level of child protection response, fewer people who know how to do referrals, and that people are less willing to report child abuse. A CDO in another district expressed similar views: “Our work is different compared to the other parishes (non-DOVCU), In terms of child protection we are more able to access service providers, know them better, and we know who to call. The linkages between us are better. Our work was simplified because the community works better” (CDO) Changes in practices of para social workers and CPMs Also, at community level practices related to identification and interactions with vulnerable families changed as a result of the project. In FGDs, para social workers described how the process of visiting, interacting, and speaking to families altered as a result of their training, and discussed this as xxx an impactful shift. Para social workers and CPMs that had already been operating in the community described the effect that their training and credited it with changing the process of their work, further developing their skills at supporting at risk children and families, and with implementing a more formal follow-up system into their work. One para social worker described the shift in family interaction and support as a result of trainings with the following words: ‘We started the work immediately after training and we would go to churches where there is a gathering and later when (DOVCU) came and trained us again. We then begun moving to households and the community in general.’ This shift to home visits and follow up visits allowed para social workers to more readily observe interactions in the home and identify at risk behaviors and situations. It allowed them to look at the environment, how clean it was, how parents spoke to their children, and also allowed the opportunity to view children’s behaviors and talk to them one-on-one in their home environment. Para social workers and CPMs described a shift in the effectiveness of their process after implementing lessons from their training: “Let me begin from when we started, the community was in a mess, people would drink anyhow, the rate of defilement was high, fights were a common occurrence. But when (DOVCU) asked the district that give us your Para social workers so we can work with them. That was in April 2015, so when we started working, we begun with community sensitization.”.” Parental training was particularly effective at improving para social workers and CPM’s identification of children at risk of separation and providing support to families. As a para social worker from Gulu described: “They trained us on parenting skills, then they sent us stop pretest the skills that we learnt during the training. They taught us processes and also gave us a tool kit for parenting skills, so when we go to a household we can observe what is happening and then ask that what does this picture show, is someone doing this in the community. So, if they are doing it, they may admit or simply laugh. So, they taught us to assess parenting skills with help of the manual, however we have it off head.” Para social workers stated that eventually, after home visits were established, some parents would approach them to visit their homes; they credited this to the success of the program and stated that this was a sign the community has internalized the importance of training and the para social workers’ knowledge. The frequency of contact with communities appeared to have increased as a result of the training and refresher course. Para social workers knew how often they followed up with families and stated that they would make two formal visits as well as informal visits. Interviews with community members and professionals clearly illustrate a change in the process of identifying at risk situations, an increased skill level and comfort in para social workers and CPMs work, and more structured follow ups with families. Findings indicate that a process for referrals is in place and community professionals know each other and when to make referrals; however, while some improvement on referrals is mentioned in interviews there is no documented evidence of such. Some community professionals cite an improvement and credit it to the training they received. A CPM respondent described the effects of training on his ability to understand the different community partners and make appropriate referrals: “I also got training from (DOVCU) as it was an eye opener to me. Right now, I am an area leader for LC 1 of this place. Because of this training, I have been able to refer children who have been abused to CPCs, Para social workers, police, (DOVCU) and other organizations…This is all because of what (DOVCU) has done in this area whereby even us the leaders, it has reduced on our work and that is a great relief.”.” In interviews, para social workers and professionals discuss referring to the CDO, LC, para social workers, police, the hospital, and other actors for the relevant needs. It is clear that training providing knowledge of who to refer to for what concern, and para social workers and CPMs said that the training and resulting changes have strengthened coordination. It is unclear if training increased the frequency of referrals, but the introduced forms did formalize the documentation of referrals. Some xxxi interviewees highlighted the need for increased communication between community professionals, pointing to a need to develop communication capacity among the various actors. xxxii ANNEX IV: LESSONS LEARNED ON CASE MANAGEMENT AND FOLLOW UP ON REUNIFICATION CASES Background DOVCU contributed to the development of standard tools to support a more consistent implementation for case management procedures for children to be reintegrated from the 12 project districts. For example, SOPs were adopted by the government for the reintegration of children from remand homes. The SOPs described the reintegration process to be followed and included tools to be used in the case management process. Furthermore, the project developed a Case Management Training Handbook (no date) and the DOVCU Case Management Tools (2015 draft), they trained, mentored and supervised CCI, remand and Child Restoration Outreach (CRO) in the use of these tools, and reported that case auditing was done to check if case management procedures were being followed. According to the project’s quarterly reports, reunified children were followed up at least every three months in 75% of cases. Furthermore, according to project reports, DOVCU piloted the ChildLinks program, a mobile phone system that allows PSWOs to update wellbeing forms and referral forms that have been adapted from DOVCU tools directly on the phone capturing real time data as well as issuing alerts and referrals quicker. Community level identification and referral by para social workers and CPMs to prevent separation. Results The evaluation captured many views that converged around the view that “case management in child protection” functions better in the “DOVCU parishes” and “DOVCU districts” as compared to non-project sites. To a large extent this seems to be valid statements for the community identification and referrals of cases of vulnerable families and children. For example, CPMs perceived para social workers as operating more consistently within communities and for them to coordinate their work better. Men and women described para social workers coming to their house, reinforcing parental trainings, and supporting them in cultivating positive relationships with their children and in following up on cases of domestic violence, alcohol abuse, and treatment of children. As one CPM described the work of para social workers: “They go and report these cases to police (…) and they make follow up. Now they use us to find out how such families are coping up. They target children who are very vulnerable and are unable to support themselves and homes with domestic violence.” This said, the one area that parents and community members described as needing more support is with children directly after they have reunified with families. When children return home, people do not always come to see how they are doing. Therefore, community members perceived the prevention measures as more consistently circulating in their communities then visits to follow up on reintegration cases to prevent secondary separation. Training, it was felt helped generate a better understanding of the roles of various players and for what purpose they could turn to each other for support and coordination. Para social workers and CPMs each highlight their work with each other and confirm that coordination for referrals and follow ups are in place. As one respondent described: “After making my referral, I go back to them and find out what steps are taken about the referral I made, I need to get feedback from them and I don’t sit and wait for another problem or sometimes even they call me themselves to tell me what transpired after my referral.”.” Training also generated a better understanding of when cases should be escalated and reported. One CPM discussed this with the following explanation: “(DOVCU) trained me that such cases should not xxxiii be resolved from home and I make follow ups with police for those other cases that need immediate intervention, when our bosses call us for a meeting, I attend and I give in my input and how people should live in the community, there should be no violence in families and I create awareness with what I learnt from (DOVCU).”.” Challenges Para social workers cite transport, mobilization, and lack of incentive as barriers to their work. Para social workers testify of the fact that it is difficult to reach homes for follow up visits without transportation help, and also feel that without any financial compensation they had little incentive to do home visits. As one para social workers stated: ‘The other challenge is during monitoring, this kind of work requires a lot of motivation and (DOVCU) should really motivate us to work hard. They should give us money to move around and make follow ups.’ The lack of transport for para social workers also makes it a challenge to respond to emergencies. As one para social worker described: ‘Transport may not be at hand if something happens, the various institutions may not be in position to respond in time.’ Community members confirmed this with stressing the effects that delay in responding can have in successfully supporting families in times of emergency. One women illustrated this with the following words: “They need to be supported with transport, I am a single mother and HIV positive and there was a time I was really ill and this lady was visiting me and she kept advising me to take courage because at some point I even never wanted to go to the hospital and death was on my mind but because of them I am still alive. This lady would cover long distance on foot to check on the other people and me as well, so even if a bicycle can help with their movements.” Moreover, transport and communication were often cited as affecting effective coordination between community partners. As once CPM respondent stated: ‘Since (DOVCU) has put different groups of people handling children’s rights we need a mobile phone for easy communication.’ Para social workers and CPMs acknowledged the air time credit given to them by (DOVCU) but cited an increase in credit and possibly even a mobile phone to use would streamline communication between different child protection volunteers and professionals. CPMs would express that now that para social workers were integrated into services provided to the community, they need to be available to help more readily. Transportation and phones would help with this. Case management of reunification cases The case management for children who are reunified follow a different route. In these cases, DOVCU supported various institutions (CCIs, remand homes and CRO) to strengthen their documentation in case files, making sure these were up to date and that children (in the cases where the child was placed in a CCI) had valid care orders. Results There is more systematic involvement of various professionals in reintegration processes: According to qualitative data, as a result of DOVCU, various professionals’ involvement in alternative care and reintegration work has now become more systematic: A PSWO in one district noted: “I used to be involved in alternative care work before. Like before the project if there was child who was found in the villages abandoned we used to take care of the child. But the project has made this work more systematic. Now it is not just me alone doing this work but there are many others involved. There are the CCIs, the communities, the families and also the children.” There is greater inter-disciplinary way of working amongst district professionals: This is accounted for by many stakeholders, district authorities and professionals themselves. The evaluators also xxxiv observed this during focus group discussions with TAC members whom in several districts were very at ease in each other’s company and had common stories to tell about cases they had been working on together. At district level, most influence is seen in the inter-disciplinary work between the PSWOs, the education department, the health department and Police of the district community services. The facilitating factor seems to have been the TACs which, according to CCIs, CDOs, CAOs played a major role in referring and mobilizing support for cases within these sectors. A CCI-director expressed his experience and satisfaction with this multi-disciplinary team work as follows: “We bonded with these institutions and they expanded our world.”.” In another district, the CAO felt that the project “saw a role for every stakeholder,” and in yet another district, the TAC team, in the FGD reflected jointly on their team work with the following words: “We moved as a team, police focusing on child protection, health looking at the medical aspects.”.” In yet another district the collaboration between district professional colleagues, (police, judge, courts etc.) through the work on reintegrating children from remand homes had allowed professionals from different sectors to work together. Having been trained together allowed all professionals to better understand the respective professional roles and areas of expertise. This, according to the OIC of one remand home resulted in “respect of everyone’s work.”.” The concrete result of this team work was, in one district, the pooling of resources. TAC members reflected in the FGD that “If one department does not have resource to support a successful reintegration, other members or stakeholders of the TAC can take on and support by providing vehicle or similar. In the past the work and responsibility was only on one person, the probation officer. Today the team has one voice.” Challenges The project had limited impact on strengthening the overall district case management system for children at risk. Each stakeholder seems to have continued doing their own case management (with varied levels of quality), and still today there is no shared understanding of the role of the PSWO and CDOs in the quality assurance of the case management and follow-up to reintegration cases managed by each entity. One national level stakeholder involved in alternative care illustrated this with the following words: “DOVCU came at a time when Uganda does not have a case management system (…) there is an issue of social service workforce that is a challenge. Regulations are rarely implemented because they (PSWOs) are unable to do that. PSWOs have so many roles (…) In the end there is still no capacity in MoGLSD to help in reintegration processes. It was beyond the project to integrate this process in the Government.”.” An example of the limited effect on the overall case management system is the fact that across districts, record-keeping for cases followed by PSWOs is still very weak. Even though the evaluation team was told that PSWOs in some districts kept copies of case records for reunified children, the evaluation team could not find evidence of that. Overall, for the cases that are followed by PSWOs, the filing system is poor, casefiles are often incomplete and a variety of tools are in use. Furthermore, the evaluation did not find any indications or references on the results of the pilot “ChildLinks” mobile phone system that was introduced for PSWOs. Another example of the same issue is that the involvement of PSWOs and CDOs in case management varies depending on the type of case and on their individual interpretations of their professional role. In Mbale, where the project worked with all three child-target groups (children found to be on the street, children in CCIs and children in remand homes), the PSWO knowledge, involvement and documentation of cases differed for each target group. For example, all case files for children reunified from CCIs were kept by CCIs and the DOVCU project. For children reunified from remand homes (in both Mbale and Kabarole), case files were transferred to the PSWO in the district where the child had been placed in remand. Discussions had taken place on whether the casefiles should follow the reunified child and be transferred to the PSWO in the district where the child was reunified, but no agreement was reached on this point. For children on the street, casefiles were kept by the NGO without xxxv oversight or knowledge of PSWO or CDOs. The working relationship with district and sub-county professionals on the reintegration of children from the street was described as a “difficult working relationship.”.” There was an assumption made in the project that follow up to reintegration cases would be done by the service provider and community structures but it is unclear to what extent this has happened. Verbally, many CCIs state that they are following up on children regularly, at least those who live within the district. One CCI social worker testified: “We visit the children – we talk to them every week – we were like a family – they are like my children”, and another CCI stated: “Monitoring of the homes is done on quarterly basis by the program manager and program team. The visits are surprise visits at home and at school. We also review the family home environment. Does the toilet work, kitchen, light? Working together with the district probation office. Discussions are also done with the children. The visits are documented with internal documentation tools/forms.” However, this is not a consistent finding and documentation of follow up is poor. Even though record keeping on children in CCIs and remand homes more generally has improved since baseline, it was difficult to get access to documentation that could substantiate the follow up provided to children who have been reunified. The case files for reunified children that were reviewed by the evaluation team were often incomplete. Where follow up visits were documented, these often focused on education and health issues, less on the psychosocial wellbeing of the child. Some professionals who were involved in follow up expressed frustration over the fact that the documentation process was very complex. As one professional put it: “There were a lot of tools. So many tools. It was time consuming and there is some duplication.”.” xxxvi ANNEX V: STATISTICAL TABLES IMPACT Evaluation question 1: To what extent and how did this project identify/test successful interventions that prevent and support reintegration of children at risk? Table 13: Shows number/proportion of households that belong to; destitute, struggling and growing level of economic vulnerability; data disaggregated by gender of household headship, all districts and phase 1 districts results Baseline value Endline value Household headship Vulnerability type All Districts Phase 1 District All District Phase 1 District Total Destitute 1252 (56%) 739 (60%) 670 (30%) 261 (21%) Struggling 974 (44%) 484 (39%) 1534 (69%) 949 (77%) Growing 8 (0%) 6 (1%) 30 (1%) 19 (2%) Male headed Destitute 476 (21%) 292 (24%) 246 (11%) 94 (8%) Struggling 543 (24%) 248 (20%) 758 (34%) 437 (36%) Growing 8 (0%) 6 (0%) 23 (1%) 15 (1%) Female headed Destitute 776 (35%) 447 (36%) 424 (19%) 167 (14%) Struggling 431 (19%) 236 (19%) 776 (35%) 512 (42%) Growing 0 (0%) 0 (0%) 7 (0%) 4 (0%) Table 14: Changes in household’s economic vulnerability classification by districts Baseline value End line value District Destitute Struggling Growing Destitute Struggling Growing Gulu 96 (68%) 46 (32%) (0%) 12 (8%) 127 (89%) 3 (2%) Iganga 76 (41%) 108 (58%) 1 (1%) 37 (20%) 146 (79%) 2 (1%) Jinja 110 (77%) 33 (23%) (0%) 61 (43%) 80 (56%) 2 (1%) Kabale 122 (52%) 111 (48%) (0%) 153 (66%) 80 (34%) (0%) Kabarole 67 (50%) 66 (50%) (0%) 46 (35%) 86 (65%) 1 (1%) Kampala 63 (69%) 28 (31%) (0%) 71 (78%) 20 (22%) (0%) Kamuli 87 (52%) 77 (46%) 2 (1%) 19 (11%) 145 (87%) 2 (1%) Kasese 160 (57%) 120 (43%) 1 (0%) 34 (12%) 243 (86%) 4 (1%) Lira 114 (36%) 198 (63%) 2 (1%) 66 (21%) 241 (77%) 7 (2%) Luwero 89 (56%) 68 (43%) 1 (1%) 35 (22%) 119 (75%) 4 (3%) Mbale 239 (69%) 107 (31%) 2 (1%) 114 (33%) 229 (66%) 5 (1%) Wakiso 29 (71%) 12 (29%) (0%) 22 (54%) 19 (46%) (0%) Total 1252 (56%) 974 (44%) 9 (0%) 670 (30%) 1535 (69%) 30 (1%) Evaluation question 2: To what extent and how did the project contribute to the reduction in the number of children living in institutions and on the street in the targeted districts? xxxvii Table 15: % of remand homes showing reduced number of children their care and % changes in the number of children Total # of remand Homes # of Remand Homes % remand homes showing reduced # of children placed # of children % change on the # of children in Remand Home BL EL BL EL Total 5 5 5 40% (2) 338 292 -14% Naguru 1 1 1 0 104 106 2% Gulu 1 1 1 0 38 56 32% Fort portal 1 1 1 0 43 51 15% Mbale 1 1 1 1 41 11 -73% Kampiringisa National Rehabilitation Center (KNRC) 1 1 1 1 112 68 -39% Table 16: Number of children reunified from CCIs, remand homes and street (disaggregated by age) Male All Districts Female All Districts # of children reunified directly / indirectly under DOVCU Total 1743 841 902 1743 Child Care Institution (CCI) Age <5 125 98 27 125 Age 6-14 1072 567 505 1072 Age >15 268 105 163 268 CCI Total 1465 770 695 1465 Remand Home (RH) Age <5 0 0 0 0 Age 6-14 19 19 0 19 Age >15 55 52 3 55 RH Total 74 71 3 74 Street Connected (SC) Age <5 0 0 0 0 Age 6-14 55 0 55 55 Age >15 149 0 149 149 SC Total 204 0 204 204 xxxviii Table 17: % of CCIs showing reduced number of children in residential care and % changes in the number of children Total # of CCIs # of CCIs % CCIs showing reduced # of children placed in residential care # of children % change in the # of children in BL EL BL EL the CCI Total 147 147 91 58 (64%) 6331 5016 -26% Luwero 23 23 6 5 (83%) 347 82 -323% Kamuli 10 10 4 4 (100%) 536 129 -316% Iganga 4 4 2 2 (100%) 133 54 -146% Mbale 6 6 4 2 (50%) 264 164 -61% Lira 5 5 4 3 (75%) 161 103 -56% Kabarole 5 5 4 1 (25%) 290 202 -44% Jinja 16 16 14 10 (71%) 500 351 -42% Kabale 11 11 9 3 (33%) 280 237 -18% Kasese 5 5 5 2 (40%) 196 169 -16% Gulu 9 9 5 3 (60%) 1000 870 -15% Wakiso 16 16 14 11 (79%) 1773 1701 -4% Kampala 37 37 20 12 (60%) 851 954 11% Evaluation question 3: To what extent and how have prevention methods employed by the project increased nutrition, education opportunities, care, and protection for children in targeted households. Table 18: Shows the significances in the mean differences in the vulnerability score of Children based on all CPA (EL - BL) – an indicator of changes in the well-being Diff in CPA scores EL – BL Mean diff Std. Deviation Std. Error Mean 95% CI of the Difference t dof Sig. (2- tailed) Lower Upper EL - BL CPA 2 -1.336 3.754 0.064 -1.462 -1.211 -20.893 3444 0.00 EL - BL CPA 3 -1.578 2.831 0.048 -1.673 -1.484 -32.721 3444 0.00 EL - BL CPA 4 -1.293 4.33 0.074 -1.438 -1.148 -17.525 3444 0.00 EL - BL CPA 5 -0.947 2.782 0.047 -1.04 -0.854 -19.982 3444 0.00 EL - BL CPA 6 -0.694 1.743 0.03 -0.753 -0.636 -23.388 3444 0.00 EL - BL CPA 2,3,4,6 -5.849 10.801 0.184 -6.21 -5.488 -31.783 3444 0.00 Table 19: Shows household access to basic needs (baseline and end line values) Baseline value Endline value All districts BL Phase 1 districts BL All districts EL Phase 1 districts EL Household access to food and nutrition Rarely has food to eat and goes to bed hungry most nights 335 (15%) 182 (15%) 82 (4%) 35 (3%) xxxix Frequently has less food to eat than needed, complains of hunger 1218 (55%) 656 (53%) 706 (32%) 254 (21%) Has enough to eat some of the time, depending on season or food supply 585 (26%) 324 (26%) 1280 (57%) 818 (67%) Well fed, eats regularly 95 (4%) 65 (5%) 165 (7%) 121 (10%) Household # of meals per day in worst months of the year Had 0 - 1 meals 1167 (52%) 560 (46%) 641 (29%) 210 (17%) Had 2 - 3 meals 1028 (46%) 638 (52%) 1495 (67%) 942 (77%) Had more than 3 meals 36 (2%) 27 (2%) 99 (4%) 76 (6%) Household resilient source on food consumed in the households Donated 103 (5%) 49 (4%) 17 (1%) 6 (0%) Given in return for work 296 (13%) 153 (13%) 68 (3%) 34 (3%) Bought from the market 1118 (50%) 539 (44%) 1315 (59%) 668 (54%) Home grown 702 (32%) 477 (39%) 833 (37%) 520 (42%) Household months of food shortage in the last 12months 6 or more months 668 (30%) 372 (31%) 321 (15%) 144 (12%) 3 - 5 months 1120 (51%) 629 (52%) 1274 (58%) 712 (58%) 0 - 2 months 423 (19%) 214 (18%) 594 (27%) 370 (30%) Household shelter No stable, adequate, or safe place to live 469 (21%) 272 (22%) 115 (5%) 49 (4%) Place needs major repairs, crowded, inadequate, not weather protective 831 (37%) 472 (38%) 591 (26%) 251 (20%) Place needs some repairs but is fairly adequate, dry, and safe 678 (30%) 344 (28%) 1105 (49%) 646 (53%) Place that is adequate, dry, and safe 255 (11%) 139 (11%) 423 (19%) 281 (23%) Table 20: Shows household access to basic needs: education (baseline and endline values) Baseline Endline All districts BL Phase 1 districts BL All districts EL Phase 1 districts EL Child access to age appropriate education: pre-school / basic education None of the children is attending school 192 (9%) 100 (8%) 65 (3%) 36 (3%) Some children not attending school 1078 (49%) 601 (49%) 895 (40%) 507 (41%) All children attending school 738 (33%) 408 (34%) 1087 (49%) 612 (50%) Children not of school going age 196 (9%) 106 (9%) 183 (8%) 69 (6%) Evaluation question 4: To what extent, and which systemic changes did the project bring about at community, regional, and national levels that are enabling children to live in family care and preventing inappropriate placements in institutional care? xl Table 21: Number of CCIs assessed at baseline and endline and their endline status Districts # of CCIs Assessed # of CCIs - Not assessed at EL # of CCIs # of CCIs transitioned to School by EL # of CCIs transitioned to CBO by EL CCIs not worked with Baseline Endline Closed by EL Total 147 91 56 (38%) 21 (14%) 11 (7%) 3 (2%) 21 (14%) Gulu 9 5 4 (44%) 3 (33%) 0% 0% 1 (11%) Iganga 4 2 2 (50%) 2 (50%) 0% 0% 0% Jinja 16 14 2 (13%) 0% 2 (13%) 0% 0% Kabale 11 9 2 (18%) 0% 2 (18%) 0% 0% Kabarole 5 4 1 (20%) 0% 1 (20%) 0% 0% Kampala 37 20 17 (46%) 0% 0% 0% 17 (46%) Kamuli 10 4 6 (60%) 4 (40%) 1 (10%) 1 (10%) 0% Kasese 5 5 0 (0%) 0% 0% 0% 0% Lira 5 4 1 (20%) 0% 1 (20%) 0% 0% Luwero 23 6 17 (74%) 11 (48%) 2 (9%) 2 (9%) 2 (9%) Mbale 6 4 2 (33%) 1 (17%) 1 (17%) 0% 0% Wakiso 16 14 2 (13%) 3 (19%) 1 (6%) 0% 1 (6%) xli Table 22: Shows CCI performance progress on AHR Standards as a result of the rollout of ACF endline compared to baseline CCI assessment average score32 out of 4 at baseline CCI assessment average score out of 4 at endline Districts GMS FM HR CCP RK CRAC PPS Overall average score GMS FM HR CCP RK CRAC PPS Overall average score % change in the average scores Total 2.18 2.13 2.3 2.56 2.19 1.6 1.84 2.14 2.6 2.88 2.84 3.11 2.93 2.87 2.99 2.89 26% Gulu 2.63 2.36 2.78 3.26 2.6 1.76 1.86 2.54 3.36 2.9 2.83 3.52 3.7 3.54 3.04 3.27 22% Iganga 3.2 2.15 2.3 2.4 2.25 1.55 2 2.32 3 3.8 2.05 3.25 3.25 2 2.5 2.84 18% Jinja 2.82 3.21 2.91 3.2 3.22 2.3 2.83 2.93 3.06 3.29 3.14 3.57 3.56 3.47 3.79 3.41 14% Kabale 1.52 1.37 1.84 2.4 1.51 1.27 1.58 1.62 2.79 3.17 3.07 3.28 2.89 2.98 3.42 3.09 47% Kabarole 2.03 2.57 2.28 2.53 2.03 0.55 1.55 2.02 2.78 2.95 2.33 2.68 2.58 2.38 1.75 2.49 19% Kampala 1.73 2.02 2.11 2.37 1.85 1.75 1.86 1.92 2.03 2.81 3 2.87 2.42 2.54 2.33 2.57 25% Kamuli 1.4 0.5 1.46 2.09 1.23 0.27 0.3 1.18 2.35 0.65 1.89 2.93 2.63 3.03 3.3 2.4 51% Kasese 1.91 1.64 1.61 2.08 1.65 0.74 1.28 1.66 2.64 2.99 2.84 3.15 3.38 2.91 3.68 3.08 46% Lira 2.5 3 3 3.25 2.75 3.25 1.75 2.88 4 4 4 4 4 4 4 4 28% Luwero 2.03 1.92 2.5 2.24 2.01 1.09 1.5 1.9 2.1 2.17 2.25 1.89 1.92 1.81 1.83 1.99 5% Mbale 3.59 2.69 3.11 3.34 2.96 1.99 2.35 2.94 3.56 3.59 3.5 3.79 3.74 3.83 3.95 3.71 21% Wakiso 2.24 1.84 2.03 2.07 2.16 1.51 1.73 1.95 2.05 2.61 2.46 2.93 2.67 2.49 2.87 2.58 25% 32 Score 4: Fully implemented / Complies fully with legislation, regulations and best practice / Well documented and reviewed frequently Score 3: Implemented but undefined / undocumented, Needs some additional work / modification Score 2: Partially implemented, Informal or Undocumented / Lacking Score 1: Considered / planned but not implemented, Severely deficient Score 0: Not in place or not considered, Completely inadequate, Noncompliance with legislation, regulations and best practice xlii EFFECTIVENESS Evaluation question 10: To what extent have the targets of the project to support 43,000 children in vulnerable households and 2,000 children to be reintegrated been reached and how were they achieved? Table 23: Number of children reunified within and outside DOVCU-supported districts and sub-counties Number of children reunified All districts Phase 1 districts Phase 2 districts Total 1743 1101 642 Outside DOVCU-supported districts 1045 710 335 Within DOVCU-supported districts 698 391 307 Within DOVCU-supported district but outside supported Sub Counties 394 227 167 Within DOVCU-supported districts and sub-counties 304 164 140 Evaluation question 12: What were the most significant changes in the lives of the two target groups which were achieved by the project, and what factors seem to have influenced these changes? Table 24: shows the significances in the mean differences in the vulnerability score of Children based on all CPA (EL - BL) – an indicator of changes in the well-being Diff in CPA scores EL – BL Mean diff Std. Deviation Std. Error Mean 95% CI of the Difference t dof Sig. (2- tailed) Lower Upper EL - BL CPA 2 -1.336 3.754 0.064 -1.462 -1.211 -20.893 3444 0.00 EL - BL CPA 3 -1.578 2.831 0.048 -1.673 -1.484 -32.721 3444 0.00 EL - BL CPA 4 -1.293 4.33 0.074 -1.438 -1.148 -17.525 3444 0.00 EL - BL CPA 5 -0.947 2.782 0.047 -1.04 -0.854 -19.982 3444 0.00 EL - BL CPA 6 -0.694 1.743 0.03 -0.753 -0.636 -23.388 3444 0.00 EL - BL CPA 2,3,4,6 -5.849 10.801 0.184 -6.21 -5.488 -31.783 3444 0.00 EFFICIENCY CHAPTER Evaluation question 15: To what extent were the project strategies, to reduce vulnerability of households and to reintegrate children, efficiently implemented? Were there any implementation gaps? Table 25: Number of children reunified with families which were handled by TACs in each district # of children reunified from 2015 -2017 under DOVCU Project Month and year established # of reunified cases handled by TAC % of reunified children handled by TAC Total 1743 11 356 20% Mbale 464 09-2015 103 22% Kamuli 113 12-2015 71 63% Gulu 112 04-2016 16 14% Kasese 123 04-2016 20 16% xliii Kabarole 148 04-2016 32 22% Luwero 141 07-2016 0 0% Jinja 105 08-2016 37 35% Lira 157 10-2016 14 9% Kampala 59 10-2016 0 0% Kabale 114 11-2016 21 18% Iganga 47 02-2017 47 100% Wakiso 160 N/A 0 0% HUMAN RIGHTS CHAPTER Evaluation question 22: To what extent did the project manage to target those most at risk (e.g. children in institutions and children on the street), including tailoring activities to accommodate needs of the most vulnerable children (children under three, girls, children with disabilities, street children)? xliv Table 26: Districts with lowest average score on child care provisions at BL and lowest change in number of children placed in residential care at end line Districts Average scores in child care provisions at baseline District with poorest average score on child care provisions at baseline % of CCIs showing reduced # children placed in residential care at end line % change in the # of children in CCI between baseline and end lines District with lowest % change in # of children in CCI at end line Waksio 2.07 12 11 (79%) -4% 11 Kasese 2.08 11 2 (40%) -16% 9 Kamuli 2.09 10 4 (100%) -316% 2 Luwero 2.24 9 5 (83%) -323% 1 Kampala 2.37 8 12 (60%) 11% 12 Kabale 2.4 7 3 (33%) -18% 8 Iganga 2.4 6 2 (100%) -146% 3 Kabarole 2.53 5 1 (25%) -44% 6 Jinja 3.2 4 10 (71%) -42% 7 Lira 3.25 3 3 (75%) -56% 5 Gulu 3.26 2 3 (60%) -15% 10 Mbale 3.34 1 2 (50%) -61% 4 Table 27: Number of reunified children with and without disability All districts Phase 1 M F M F Total number of children reunified; directly under DOVCU and indirectly Total number of children reunified 841 902 562 761 1743 Not disabled 808 875 530 738 1683 Disabled 33 27 32 23 60 % of children reunified who disabled 4% 3% 6% 3% 3% Evaluation question 23: To what extent was the project able to prevent child protection issues and human rights violations (e.g. child labor, violence against children, neglect, early marriage, etc.) and mitigate risks for creating greater vulnerability (e.g. child labor and any kind of harm)? Table 28: shows number and % of children engaged in exploitative unpaid employment Baseline value Endline value Response option 14-17 18-25 14-17 18-25 Child is engaged in full time exploitative unpaid employment 40 (13%) 9 (21%) 29 (8%) 3 (7%) Child is engaged in full time poorly paid employment 41 (13%) 11 (26%) 47 (13%) 0 (0%) Child is engaged in part time paid employment 83 (27%) 12 (28%) 85 (23%) 9 (20%) Child is engaged in part time paid employment that allows them to study and develop skills 149 (48%) 11 (26%) 214 (57%) 34 (74%) xlv Table 29: Shows household level status: ppsychosocial distress, child protection, knowledge on referral pathways % of household heads who report having had psychosocial distress in the last 1 year BLV All districts ELV All districts Most of the times 402 (18%) 174 (8%) Sometimes 1487 (67%) 1746 (78%) Never 341 (15%) 315 (14%) % of household heads reporting emotional distress in the last 12 months Family seems hopeless, sad, withdrawn, a member wishes could die, or wants to be left alone. Targeted child may refuse to eat, sleep poorly, or cry a lot. 398 (18%) 66 (3%) Family is often withdrawn, irritable, anxious, unhappy, or sad. Targeted child may cry frequently or often be in active. 864 (39%) 470 (21%) Family is mostly happy but occasionally a member is anxious or withdrawn. Targeted child may be crying, irritable, or not sleeping well some of the time 718 (32%) 1205 (54%) Family seems happy, hopeful, and content 244 (11%) 494 (22%) % of households reporting violent (common social problems) behavioral and mental pattern within their households in the last 12 months. There are frequent or periodic signs of aggressive behaviors, domestic violence, child abuse, child neglect 313 (14%) 51 (0%) The household is known for alcohol or drug over use, alcohol addiction 220 (10%) 246 (2%) Family conflict, conflict with mate, child problems is frequent 218 (10%) 366 (3%) The family is frequently or periodically faced with community conflict 229 (10%) 1272 (12%) Some of the above signs but a bit mild 607 (27%) 4520 (42%) None of the above 644 (29%) 4302 (40%) % of caregivers and parents knowledgeable on the most correct referral pathway for CP in case of VAC. 1207 (54%) 1372 (61%) % of caregivers who uses humiliating or rudimentary approach (Punched, Kicked or hit, withheld a meal or basic needs, use abusive words/ language) to discipline a child in the past 12 months 1338 (60%) 1234 (55%) % of households reporting incidences of VAC in their family in the last 12 months Targeted child is abused, sexually or physically, emotionally and/or being subjected to child labor or otherwise exploited 140 (6%) 14 (1%) Targeted child is neglected, given inappropriate work for his or her age, or is clearly not treated well in household or institution 375 (17%) 88 (4%) There is some suspicion that the targeted child may be neglected, over-worked, not treated well, or otherwise maltreated 851 (38%) 808 (36%) Targeted child does not seem to be abused, neglected, do inappropriate work, or be exploited in other ways 858 (39%) 1318 (59%) % of households with children or any child (0-17 years) currently not living in the household in the past 3 to 12 months Yes 781 (35%) 561 (25%) No 1432 (65%) 1673 (75%) Reasons for not living in the household (in percentages) Child left home for job elsewhere 119 (15%) 74 (13%) Don't know where the child has gone 48 (6%) 6 (1%) Child does not stay in this home – reasons unknown 109 (14%) 79 (14%) Child living with relative because family cannot support them 336 (42%) 195 (35%) xlvi Table 30: Shows changes in the wellbeing of children (combining prevention and reunified households) by age groups Baseline End line All districts Phase 1 districts All districts Phase 1 districts Age Critical Medium Low Critical Medium Low Critical Medium Low Critical Medium Low Total 92 (3%) 3003 (86%) 381 (11%) 49 (3%) 1623 (86%) 218 (12%) 14 (0%) 2482 (72%) 970 (28%) 6 (0%) 1178 (63%) 696 (37%) Age <5 17 (2%) 742 (86%) 105 (12%) 7 (2%) 369 (85%) 56 (13%) 2 (0%) 659 (76%) 204 (24%) 1 (0%) 290 (68%) 136 (32%) Age 6-14 24 (3%) 759 (84%) 121 (13%) 15 (3%) 436 (84%) 65 (13%) 6 (1%) 625 (69%) 273 (30%) 1 (0%) 315 (61%) 199 (39%) Age >15 51 (3%) 1502 (88%) 155 (9%) 27 (3%) 818 (87%) 97 (10%) 6 (0%) 1198 (71%) 493 (29%) 4 (0%) 573 (61%) 361 (38%) Male Total 52 (3%) 1565 (88%) 157 (9%) 27 (3%) 846 (88%) 87 (9%) 8 (0%) 1291 (73%) 470 (27%) 5 (1%) 619 (65%) 330 (35%) Male Age <5 8 (2%) 393 (87%) 52 (11%) 3 (1%) 198 (86%) 30 (13%) 0 (0%) 347 (77%) 105 (23%) (0%) 156 (68%) 72 (32%) Male Age 6-14 15 (3%) 406 (89%) 36 (8%) 8 (3%) 227 (91%) 14 (6%) 3 (1%) 331 (72%) 123 (27%) 1 (0%) 167 (67%) 81 (33%) Male Age >15 29 (3%) 766 (89%) 69 (8%) 16 (3%) 421 (88%) 43 (9%) 5 (1%) 613 (71%) 242 (28%) 4 (1%) 296 (62%) 177 (37%) Female Total 40 (2%) 1438 (84%) 224 (13%) 22 (2%) 777 (84%) 131 (14%) 6 (0%) 1191 (70%) 500 (29%) 1 (0%) 559 (60%) 366 (40%) Female Age <5 9 (2%) 349 (85%) 53 (13%) 4 (2%) 171 (85%) 26 (13%) 2 (0%) 312 (76%) 99 (24%) 1 (1%) 134 (67%) 64 (32%) Female Age 6-14 9 (2%) 353 (79%) 85 (19%) 7 (3%) 209 (78%) 51 (19%) 3 (1%) 294 (66%) 150 (34%) (0%) 148 (56%) 118 (44%) Female Age >15 22 (3%) 736 (87%) 86 (10%) 11 (2%) 397 (86%) 54 (12%) 1 (0%) 585 (70%) 251 (30%) (0%) 277 (60%) 184 (40%) xlvii ANNEX VI: EVALUATION STATEMENT OF WORK (see below) APC Request for Proposals – Performance Evaluation of Child Care Reform Project in Uganda i REQUEST FOR PROPOSALS: Performance Evaluation of Child Care Reform Project in Uganda Issuance Date: October 1, 2014 Deadline for Questions: October 7, 2014 at 5PM EDT Closing Date: October 31, 2014 at 5PM EDT The Advancing Partners & Communities (APC) project invites qualified U.S.-based Private Voluntary Organizations, international or national non-governmental organizations, faith-based organizations, and universities, to submit proposals for an independent performance evaluation of a child care reform project that will be implemented in Uganda by a consortium of non￾governmental organizations led by ChildFund International (hereinafter referred to as “Implementing Partner”). One Fixed Price Contract is anticipated to be awarded under this Request for Proposals (RFP). It will be issued by JSI Research & Training Institute, Inc. (JSI) under the APC project. Proposals, including reference documents, must be submitted by October 31, 2014 at 5PM EDT. Proposals must be submitted online. One (1) electronic copy of the proposal should be sent to grants-APC@jsi.com. RFP Questions and Answers Questions about this RFP should be submitted to APC at grants-APC@jsi.com, in English, no later than October 7, 2014 at 5PM EDT. A list of all questions and answers will be posted on the APC website by October 14, 2014. The RFP will consist of this announcement and the following: Section A – Proposal Description Section B – Proposal Instructions Section C – Eligibility and Selection Criteria Attachment 1 – Past Performance Information Table Attachment 2 – Summary and Detailed Budget Templates Issuance of this RFP, or the submission of a proposal, does not constitute an award commitment on the part of the United States Government (USG), JSI, and/or FHI 360, nor does it commit the USG, JSI and/or FHI 360 to pay for costs incurred in the preparation and submission of a proposal. JSI reserves the right not to fund any of the proposals received. APC Request for Proposals – Performance Evaluation of Child Care Reform Project in Uganda i REQUEST FOR PROPOSALS Performance Evaluation of Child Care Reform Project in Uganda Table of Contents SECTION A: PROPOSAL DESCRIPTION ............................................................................................... 1 A1. Introduction........................................................................................................................................ 1 A2. Project to be Evaluated ...................................................................................................................... 1 A3. Background ........................................................................................................................................ 2 A4. Purpose and Use of the Evaluation .................................................................................................... 3 A5. Evaluation Questions ......................................................................................................................... 3 A6. Evaluation Methods ........................................................................................................................... 4 A7. Deliverables and Timeline ................................................................................................................. 5 A8. Expected Deliverables........................................................................................................................ 5 A9. Composition of the Evaluation Team ................................................................................................ 7 Team Leader ............................................................................................................................................. 8 A11. Scheduling, Logistics and Other Support......................................................................................... 8 A12. Requirements for Reporting............................................................................................................. 8 SECTION B: PROPOSAL INSTRUCTIONS............................................................................................. 9 B1. Proposal Guidelines............................................................................................................................ 9 B2. Proposal............................................................................................................................................10 SECTION C: ELIGIBILITY AND SELECTION CRITERIA...................................................................12 C1. Introduction ......................................................................................................................................12 C2. Eligibility..........................................................................................................................................12 C3. Selection Process..............................................................................................................................12 C4. Evaluation Criteria ...........................................................................................................................12 C5. Weighing of Evaluation Criteria ......................................................................................................13 Attachment 1: Past Performance Information Table...................................................................................14 Attachment 2: Summary and Detailed Budget Template ...........................................................................15 APC Request for Proposals – Performance Evaluation of Child Care Reform Project in Uganda 1 SECTION A: PROPOSAL DESCRIPTION A1. Introduction This solicitation is for an independent performance evaluation of a child care reform project that will be implemented in Uganda by a consortium of non-governmental organizations led by ChildFund International. The project that will be evaluated is 36 months in duration and will be implemented in 12 districts in Uganda. The independent performance evaluation will be designed to measure the performance and sustainability of the project activities and will identify broader lessons related to preventing unnecessary separation of children from families and facilitating family reintegration. The evaluation will include baseline, mid-term, and end-line data collection and reports. The organization awarded the contract (hereinafter referred to as “Evaluating Agency”) will be expected to work closely with the Implementing Partner to develop and operationalize indicators that will be used to measure the project’s performance and to produce a detailed research design. A research design document will be produced by the Evaluating Agency in consultation with USAID Displaced Children and Orphans Fund (DCOF), JSI/APC, and the Implementing Partner. This design document will provide details on the methods and plans for carrying out the baseline, mid-term, and end-line studies. A2. Project to be Evaluated The project that will be evaluated is responding to the situation of children outside of family care in Uganda. This project has two main objectives: 1) To reduce unnecessary separation of children from their families, and 2) To place children outside of family care in nurturing families. By the end of the project, the implementers expect to achieve the following outcomes: • Approximately 43,000 vulnerable children living in target households show an improvement in selected child- and household-level outcomes in the two years following their enrollment in the project. • Approximately 2,000 children living in institutional care or in the streets are reunified with birth families or placed into alternative family-based care and have improved positive outcomes in the twelve months following reintegration. The project will pilot an integrated package of interventions that will: 1. Create opportunities for existing care institutions to re-conceptualize their role and acquire relevant knowledge and skills to support the aims and implementation of the Alternative Care Framework and to ensure family based care for all children. This process will be undertaken in close collaboration with government agencies with APC Request for Proposals – Performance Evaluation of Child Care Reform Project in Uganda 2 oversight authority over child care institutions, and will build on a model of social work best practices. 2. Reintegrate girls living on the street. 3. Strengthen Uganda’s Ministry of Gender, Labor and Social Development and the capacity of district governments to monitor institutions and effectively implement the country’s new Alternative Care Framework. 4. Strengthen informal child protection mechanisms and mobilize communities to monitor children’s wellbeing. 5. Strengthen household economy and parenting skills. The project will use proven community mobilization strategies– using role models, training community leaders, and mobilizing faith-based organizations to support family-based care. The project will work in the 12 districts that concentrate 50% of the institutions in Uganda and that are located across four regions: East, West, North, and Central region. Activities at the community level will be concentrated in the four most vulnerable parishes in each district. The independent evaluation for this project should collect baseline data and produce a baseline report at the beginning of the project, produce a midterm review near the middle of the project, and collect end-line data and produce a final evaluation report at the end of the project. The Evaluating Agency will be expected to work closely with the Implementing Partner during project design and roll-out to revise a performance monitoring plan (PMP) that will provide guidance for project M&E and will facilitate the collection of baseline, midterm, and end-line data. The project M&E team will work with the Evaluating Agency to refine the logical framework and household data collection tools and agree on data that will be collected, stored, and shared with the Evaluating Agency. A3. Background In Uganda, an estimated 57,000 children are living in child care institutions, an increase from less than 1,500 in 1997. Residential care facilities have multiplied from approximately 35 in the mid- 1990s to over 500 in 2012. If unaddressed, these conditions risk causing serious delays in the social, behavioral, and cognitive development of these children, as well as attachment disorders. The major factors thought to be causing children to leave home to live on the streets or in institutions are: violence and abuse within the home, family breakdown, poverty, high prevalence of alcoholism and substance abuse, psychosocial distress, gender-based violence, and harmful cultural practices that precipitate child abandonment and pre-mature exit of children from the family unit. These conditions are more severe in war-affected Northern Uganda where trauma and depression are common and poverty is widespread. APC Request for Proposals – Performance Evaluation of Child Care Reform Project in Uganda 3 Uganda’s social services system is overwhelmed by a very young and rapidly growing society and the HIV/AIDS crisis that has resulted in over 1 million deaths and produced an estimated 1.1 million orphans. As a result of the increased number of orphans, combined with fewer caregivers, the extended family is no longer the safety net that it once was. Faith-based and women’s groups such as voluntary associations, faith-based groups, and self-help groups have been the most effective actors to respond to and support orphans using local resources. The Government’s legislative, policy, and programmatic initiatives to protect orphans and vulnerable children include Chapter 59 of the Children’s Act, the National Orphans and Vulnerable Children policy (2004), and the 2nd National Strategic Program Plan of Interventions for Orphans and other Vulnerable Children (NSPPI-2). The National Framework for Alternative Care (2012) is a clear call to action. The project that will be evaluated aims to prevent family separation, strengthen local capacities to care for and protect children, strengthen the institutional and policy environment, and strengthen child protection mechanisms. This project is intended to directly contribute to Objectives 2 and 4 in the USG Action Plan on Children in Adversity. It will also contribute to USAID’s Country Development Cooperation Strategy for Uganda 2011-2015, particularly with programs targeting vulnerable groups, persons with disabilities, and gender dynamics. A4. Purpose and Use of the Evaluation USAID/ DCOF will use the results of the performance evaluation to inform future programming. The evaluation should effectively measure the performance and sustainability of activities associated with prevention of separation and reintegration of children who have been reunified or placed in family-based alternative care. A5. Evaluation Questions The research questions will be finalized by the Evaluating Agency during an in-country consultation workshop with the Implementing Partner, USAID, and other stakeholders, and approved by USAID/DCOF. Illustrative questions include: 1. Have reintegration methods resulted in stable and sustained placements for children? 2. Did the project measurably improve outcomes related to the safety, wellbeing, and development of children in highly vulnerable households or for those who were living without adequate family care? 3. Have prevention methods employed by the project increased nutrition, education opportunities, care, and protection for children in targeted households? APC Request for Proposals – Performance Evaluation of Child Care Reform Project in Uganda 4 4. Did the project bring about systemic changes at the community, regional, and national levels that are enabling children to live in family care and preventing inappropriate placements in institutional care? 5. By project end, to what extent have structures been established that can continue to provide, on an ongoing basis, adequate case-management services for children at risk? 6. Did the project offer models and approaches for expansion, adaptation, and/or replication? 7. Did the project influence national care reform measures? 8. By what percentage did the number of children living in institutions and on the streets in the targeted districts decrease? A6. Evaluation Methods Proposals must include a detailed description of the proposed research design and methodology. The methods and evaluation design must be as rigorous as reasonably possible. Research subjects will include children in residential care, children who have been reunified or placed in family-based care, and children in households where they are vulnerable to separation. The performance evaluation should track the situations of a sample of children in each of these circumstances to understand the results of their involvement with the project. Given the fact that the Implementing Partner will be identifying beneficiaries gradually for inclusion project activities, the evaluation sampling will have to take place on a rolling basis. Exact details and timing for evaluation sampling will have to be coordinated with the Implementing Partner. The evaluation must include baseline, mid-term, and end-line data collection and reports. Methods of data collection may include: • Review of relevant national documents and technical resource material • Group interviews • Focus group discussions • Surveys • Key informant interviews • Direct observation techniques • Participatory appraisal methods Since this evaluation potentially includes collecting sensitive data from human subjects, particularly children, the proposals must adequately describe how ethical review requirements will be met and estimates on how long approvals will take. APC Request for Proposals – Performance Evaluation of Child Care Reform Project in Uganda 5 A7. Deliverables and Timeline The Evaluating Agency is expected to provide a detailed timeline with the research design document. This timeline should be aligned with the Implementing Partner’s work-plan. The Evaluating Agency is expected to work with the Implementing Partner to develop a plan for data collection. Specific due dates for deliverables will be subject to approval by USAID/ DCOF. Activities 6-MONTH SEMESTERS Details 1 2 3 4 5 6 1. Develop research design Completed in close collaboration with USAID/DCOF and the Implementing Partner before starting the baseline data collection. 2. Contribute to PMP Completed with the research design document, also in close collaboration with the Implementing Partner. 3. Produce baseline report Data collection completed within first three months of implementation 4. Produce mid-term review report Review including FGDs, Direct Observation, and KI interviews 5. Produce final evaluation report Data collected during the last three months of implementation. 6. Provide on-going remote technical support to Implementing Partner. The Evaluating Agency will be expected to be on call for periodic technical follow-up with USAID/DCOF and Implementing Partner. A8. Expected Deliverables The assignment will be comprised of the following steps: • Research Design: The Evaluating Agency will be expected to work remotely with the Implementing Partner and USAID/DCOF to develop a research design document. This document will include a detailed description of research methods and plans and clearly defined and operationalized indicators that will be used in the evaluation. The document will also include a division of responsibilities regarding data collection for the evaluation and a detailed timeline for all activities associated with the evaluation. A draft of the Research Design document will be submitted to USAID/ DCOF for approval before finalization. • Performance Monitoring Plan: The Evaluating Agency will work closely with the Implementing Partner to develop a PMP that is aligned with the evaluation’s research design. • Data Collection Tools: The Evaluating Agency will develop or adapt existing data collection tools (e.g. questionnaires, FGD guides, observation checklists) that will be submitted to USAID/DCOF for approval. APC Request for Proposals – Performance Evaluation of Child Care Reform Project in Uganda 6 • Training of Enumerators: The Evaluating Agency will recruit, hire, train, and manage interviewers for data collection. Evaluation team members who are key personnel will be expected to be in Uganda to carry out this training and ensure adequate management of data gathering. • Baseline Data Collection: Depending on what is agreed upon for the research design, some data may be collected by the Implementing Partner and some will be collected by the Evaluating Agency. The Evaluating Agency will be expected to work with the Implementing Partner and interviewers to ensure the quality of data collection. This will entail periodic in-country training and supervision. Evaluation team members who are key personnel will be expected to be in Uganda for relevant data collection activities. • Produce Baseline Report: The Evaluating Agency will provide a draft of the report for USAID/DCOF’s review and approval before finalization. • Presentation of Baseline Results: The Evaluating Agency will present the results of the baseline study to USAID/DCOF and the Implementing Partner in person or via a live webinar. • Midterm Review Data Collection: The Evaluating Agency will collect qualitative data for the mid-term review. Evaluation team members who are key personnel will be expected to be in Uganda for data collection. • Produce Midterm Report: The Evaluating Agency will provide a draft of the report for USAID/DCOF’s review and approval before finalization. • Presentation of Mid-term Results: The Evaluating Agency will present the results of the mid-term review to USAID/DCOF and the Implementing Partner in person or via a live webinar. • End-line Data Collection: Depending on what is agreed upon through the research design, some data will be collected by the Implementing Partner and some will be collected by the Evaluating Agency. The Evaluating Agency will be expected to work with the Implementing Partner to ensure the quality of data collection. This will entail in-country training and supervision. Evaluation team members who are key personnel will be expected to be in Uganda for data collection. • Produce Final Evaluation report: The Evaluating Agency will provide a draft Final Evaluation Report that will incorporate analysis of the findings of the baseline, midterm, and end-line studies. The draft will be submitted for USAID/DCOF’s review and approval before finalization. • Presentation of Final Results: The Evaluating Agency will present the results of the end￾line study and final performance evaluation to USAID/DCOF and the Implementing Partner in person or via a live webinar. • On-going Technical Support: The Evaluating Agency will be expected to be on-call for remote technical input and support. APC Request for Proposals – Performance Evaluation of Child Care Reform Project in Uganda 7 USAID will own all data and each work product produced through this evaluation. A9. Composition of the Evaluation Team Resumes of the proposed Evaluation Team members must be submitted with the Proposal. At least one team member must have research credentials and experience in evaluation design and methods with data collection and analysis expertise. The team must also have a subject matter expert in the area of child care reform with professional experience in Uganda. The team leader must have strong team management skills, and sufficient experience with evaluation standards and practices to ensure a credible product. The team leader is expected to have a working partnership with the funder as the team moves through the evaluation research design and planning process. He/she is also a person who must be able to deal effectively with senior U.S. and host country officials and other leaders. A10. Required experience and skills of key personnel: Research Specialist • At least five years of experience supervising monitoring, evaluation, and research efforts, preferably with respect to child protection, capacity building, and service delivery. • Strong quantitative or mixed-method and analytical skills. • Ability to articulate technical information clearly and effectively to both technical and non-technical audiences. • Master’s Degree or higher in public health, demography, health management, social science, biostatistics, statistics, or a related field is preferred. Subject Matter Expert • At least 8 years of experience in social welfare particularly relating to child welfare. • High level of understanding of current issues relevant to vulnerable children, including child protection systems, family strengthening, processes for family (re)integration of children, and international development. • High level of understanding of care and care reform, including application of the UN Guidelines on Alternative Care. • Understanding of current research and research questions on children’s placement and care. • Working knowledge of programming in international contexts (preferably in sub-Saharan Africa) with family strengthening, alternative care, family reunification, and adoption APC Request for Proposals – Performance Evaluation of Child Care Reform Project in Uganda 8 and solid work experience in an international environment, including experience in sub￾Saharan Africa and preferably Uganda. • Master’s Degree or higher in social science, social policy, or related field. Team Leader • The team leader may also serve another function on the team, but should have 5 or more years of experience managing a project in an African country. • Excellent communication and diplomatic skills and ability to work with a diverse group of professionals from NGO, government, and donors. • Ability to manage priorities and to accomplish time sensitive goals with a small team. • Prior experience managing an evaluation. A11. Scheduling, Logistics and Other Support Neither USAID or JSI nor the Implementing Partner will be expected to provide assistance with logistics or administrative support for the evaluation. However, the Implementing Partner will be collecting potentially relevant information through their own M&E systems. The availability of this data will be discussed and negotiated during the three-day evaluation planning workshop in Uganda. USAID/DCOF will make arrangements for this workshop and the Evaluating Agency will be responsible for costs of sending its own representatives. The Evaluating Agency is also responsible for hiring, training, and supervising enumerators and any other staff hired for the evaluation. The Evaluating Agency must specify their sources for administrative support in Uganda, including scheduling of appointments, finding translators and interpreters, and arranging vehicles. The proposal must provide a description of how this support will be provided. The budget should appropriately reflect this local support. A12. Requirements for Reporting The evaluation team will provide oral briefings when mid-term and final drafts are submitted. USAID/DCOF will be provided with two weeks to review submitted drafts and to provide comments before finalization of the reports. The reports must clearly differentiate between findings, conclusions, and recommendations. The evaluation reports should cover the following: • Executive Summary; • Purpose, scope, and audience of the evaluation; • Evaluation questions; APC Request for Proposals – Performance Evaluation of Child Care Reform Project in Uganda 9 • The context (relevant history, demography, socio-economic status, and basic political arrangements of the community, country, or region) in which the project or policy intervention was designed and implemented and aspects of the social, economic, and political/administrative context relevant to the separation of children from families and for reintegration into family care; • Description of the project, its objectives, scope and activities; • The conceptual framework underlying the project; • Data collection: the unit of analysis, sample, type of data collected, quality of data, and analytic techniques used; • Evaluation findings: Findings are the empirical facts gathered by an evaluation team about the performance, cost effectiveness, relevance, sustainability or impacts of an intervention, strategy, policy or organization; • Conclusions: These represent the evaluators’ interpretations and judgments based on the findings and empirical data gathered and analyzed, and these should include attention to the potential wider influence within Uganda of the project on care reform, as well as its specific results; • Recommendations: These refer to the actions which the evaluation team recommends for the project, USAID, and other actors with current or potential roles related to preventing unnecessary separation of children from families and supporting the (re)integration of children into families. These should be logically derived from findings and conclusions; • Lessons learned: These are broader implications. SECTION B: PROPOSAL INSTRUCTIONS B1. Proposal Guidelines Applicants are encouraged to read the RFP in its entirety, and ensure that a proposal addresses all the items cited in the proposal instructions and meets the eligibility criteria. Proposals may be submitted by U.S. Private Voluntary Organizations, international or national non-governmental organizations (NGOs), faith-based organizations, and universities. All proposals must be submitted by October 31, 2014 at 5PM EDT. Proposals submitted after the closing date and time will not be considered. The proposal must be accompanied by a cover letter typed on official organizational letterhead and signed by a person who has signatory authority for the Applicant. Submit the complete proposal package (electronic format) on or before the due date and time to grants-APC@jsi.com. RFP Questions and Answers Questions about this RFP may be submitted to APC at grants-APC@jsi.com, in English, by October 7, 2014 at 5PM EDT. A list of all questions and answers will be posted on the APC website by October 14, 2014. APC Request for Proposals – Performance Evaluation of Child Care Reform Project in Uganda 10 Receipt of a proposal to this request does not constitute an award commitment on behalf of the USG, JSI, and/or or FHI 360, nor does it commit the USG, JSI, and/or FHI 360 to reimburse any costs incurred in the preparation and submission of a proposal. JSI, FHI 360, and the USG reserve the right not to fund any and/or all of the proposals received. All proposals received on or before the due date and time will be reviewed by APC staff for responsiveness to the specifications of the RFP. Proposals determined to be non-responsive will be eliminated as not being eligible for review by the technical evaluation committee. Eligible proposals will then be evaluated according to the criteria provided below. The committee will determine which organization will be funded based on the submitted proposals. The award of a contract is contingent on the availability of funds and the approval of JSI/APC. The successful Applicant will be awarded a contract from JSI on behalf of USAID. All Applicants will be notified in writing whether or not their proposal is recommended for funding. B2. Proposal The Proposal describes how the Applicant intends to carry out the Proposal Description (Section A). It should be concise, specific, and complete, and demonstrate a clear understanding of the work to be undertaken and the responsibilities of all parties involved. It must demonstrate the Applicant’s eligibility, as well as its capabilities and expertise. The proposal must be submitted in English. Proposals may be no more than ten (10) pages in length (including logical framework). The following are NOT included in the 10-page proposal limit: cover page, table of contents, acronym list, executive summary, budget documents, up to five Past Performance Information Tables and resumes of key personnel. All proposals should be prepared on standard letter-sized pages (8 ½ x 11”) with 1” margins, using a font size of 12 Times New Roman, except within tables and in the budget section where the font size may be smaller but must be easily readable. At this initial stage, Applicants should NOT submit any additional documents with their proposal. JSI/APC will confirm receipt via email of all proposals within ten (10) calendar days of the submission deadline. Applicants should submit all documents as email attachments. It is not necessary to send a hard copy of the Technical Proposal via mail or fax. The Technical Proposal and the corresponding Budget Documents (budget narrative and budget summary) should be submitted as separate attachments. Applicants should not include photographs as part of their proposals. Applicants must use the following format to outline their proposal. This outline corresponds to the evaluation criteria (Section C). Cover Page (1 page) Include the name of the organization submitting the proposal, evaluation title, proposed evaluation dates, requested amount (in USD) of USAID funding, and name, office address, phone, fax, email of the primary individual responsible for the proposal and one alternate. APC Request for Proposals – Performance Evaluation of Child Care Reform Project in Uganda 11 Executive Summary (1 page) The Executive Summary should provide an overview of key features of the proposed evaluation. The summary should include a brief description of the overall evaluation, objectives, methods, intended results, and level of funding requested. Core Proposal (10 pages) • Technical Approach to Evaluation • Methods used: Specifics on research methods that will be used • Data collection: Detailed description of how data will be collected, by whom and timeframe • Child protection and ethical considerations for data collection and use • Evaluation limitations • Evaluation timeline and deliverables • Staffing structure • Logical framework Summary and Detailed Budget (See Attachment 2) A detailed line-item and a summary budget should be prepared based on the responsibilities described above and presented in the format specified in Attachment 2. All budget estimates must be in U.S. Dollars. The Budget should include the following categories of cost (exclude any do that do not apply): Personnel, Staff Benefits, Consultants, Travel, and other Direct Costs. The category Overhead or Indirect Costs may be used only if the organization has a Negotiated Indirect Cost Rate Agreement (NICRA) from a US federal agency or an Indirect Rate calculated and certified by an acceptably recognized public accounting firm. Otherwise please include all overhead costs (share of rent, utilities, management costs, etc.) within the other categories. No profit or fee may be charged. A detailed budget narrative/notes that justifies the costs as appropriate and necessary for the successful completion of proposed activities should be attached to the budget. The budget narrative/notes should clearly describe the project and cost assumptions. All proposed costs and estimates must be reasonable and allowable in accordance with the US government’s Cost Principles. In summary, all proposed costs must be directly applicable to performing the work under the award; and budgeted amounts should not exceed the market cost/value of an item or service. The proposed budget should not exceed $400,000. Supporting Information • Past performance tables (see Attachment 1) • Resumes of evaluation team APC Request for Proposals – Performance Evaluation of Child Care Reform Project in Uganda 12 SECTION C: ELIGIBILITY AND SELECTION CRITERIA C1. Introduction Applicants should note that the selection criteria presented in Section C4 serve: (1) to identify the significant matters which Applicants should address in their proposals; and (2) as the standard against which all proposals will be evaluated. Applicants must organize the narrative sections of their proposals in the same order as the selection criteria. Proposals found to be ineligible and proposals that deviate significantly from this structure will not be reviewed. The Technical Proposal will be evaluated in accordance with the criteria set forth below. The contract shall be made to the Applicant whose submission, based on indicated evaluation criteria, represents the most solid technical approach and addresses the identified results. Proposals will undergo a preliminary review by APC project staff for completeness and responsiveness. Incomplete proposals, proposals judged to be unresponsive to the solicitation, and proposals submitted after the due date and time will be considered to be ineligible and will NOT be considered for an award. C2. Eligibility Proposals may be submitted by U.S. Private Voluntary Organizations, international or national non-governmental organizations (NGOs), faith-based organizations, and universities. The Applicant must have adequate financial controls and expect to meet requirements defined by 22 CFR 228 (see http://www.gpo.gov/fdsys/pkg/FR-2012-01-10/pdf/2011-33240.pdf). Applicants should demonstrate that they have significant and relevant experience concerning the issues, populations and/or interventions that they are proposing to address in the evaluation. Familiarity with USAID policies and procedures is valued. C3. Selection Process Eligible proposals will be reviewed and evaluated by Technical Evaluation Committee against the selection criteria described below. The APC project Finance Director will use the conclusions and recommendations of the Technical Evaluation Committee together with an evaluation of costs to determine if negotiations with an Applicant will be required. The costs involved in preparing the Proposal shall not be directly reimbursed regardless of the outcome of the negotiations. C4. Evaluation Criteria The Proposal will be evaluated in accordance with the criteria set forth below. Thereafter, the proposed Budget of an acceptable proposal will be reviewed for general reasonableness, allowability, and allocability of costs. A contract will be offered to the Applicant whose proposal offers the greatest value, and will be made based on the ranking of proposals according to the technical selection criteria identified below. APC Request for Proposals – Performance Evaluation of Child Care Reform Project in Uganda 13 The APC project reserves the right to request further information from all organizations deemed to have met initial selection criteria, as needed to make a final selection of a contractor. The proposal will be evaluated against the following criteria: A. Technical Approach to Evaluation [40 points] • Is there a clear explanation of what the proposed evaluation will do, how, and by whom? • Are the specific approaches and methods to be used clearly described, technically sound, appropriate, and likely to be effective? • Is the proposed approach to project evaluation realistic, technically sound, and likely to produce meaningful results? • Are viable indicators specified for proposed outcomes? • Are the roles of and relationships with the Implementing Partner clear, viable and appropriate? • Does the logical framework lay out a clear, credible process for achieving intended evaluation results? • Are there plans to ensure that protective and ethical procedures are in place with regard to the collection and use of data? B. Management, Implementation, and Staffing [30 points] • Is there a clear overview of the timeline for the major stages of the proposed evaluation? • Do proposed evaluation personnel appear well-qualified and have relevant experience in relation to their proposed positions? • Are there policies or procedures in place to mitigate the risk of child abuse, exploitation, violence, or neglect by evaluation personnel? • Is the proposed evaluation plan cost-effective with regard to the use of resources to produce results? C. Institutional Capabilities and Past Performance [30 points] • Does the organization have a strong track record for evaluating this type of project? • Does it have the necessary technical expertise? • In the experience of the reviewers, has the organization demonstrated success with similar evaluations in the past? C5. Weighing of Evaluation Criteria A. Technical Approach to Evaluation 40 points B. Management, Implementation, and Staffing 30 points C. Institutional Capabilities and Past Performance 30 points TOTAL 100 Points APC Request for Proposals – Performance Evaluation of Child Care Reform Project in Uganda 14 Attachment 1: Past Performance Information Table Organization Name Evaluation Objectives Description of evaluation Activities undertaken Geographic Location (district, sub-county, village, parish) Funding Agency’s Contact Information (name and address of the Agency and name and phone number/email address of project contact) Period of Implementation Total Budget APC Request for Proposals – Performance Evaluation of Child Care Reform Project in Uganda 15 Attachment 2: Summary and Detailed Budget Template ORGANIZATION NAME Summary Budget Line Item Year 1 Year 2 Year 3 Total 1 TOTAL STAFF/PERSONNEL 0 0 0 0 2 TOTAL STAFF BENEFITS 0 0 0 0 3 TOTAL CONSULTANT COSTS 0 0 0 0 4 TOTAL TRAVEL COSTS 0 0 0 0 5 TOTAL OTHER DIRECT COSTS 0 0 0 0 6 TOTAL INDIRECT COSTS 0 0 0 0 TOTAL 0 0 0 0 APC Request for Proposals – Performance Evaluation of Child Care Reform Project in Uganda 16 Total by Line Item Exchange Rate1 USD = 0 Annual Inflation 0% Annual Staff Salary Increase (if applicable) 0% Line Items Unit Cost Units Amount Units Amount Units Amount Units Amount I. STAFF/PERSONNEL A. Project Full-Time Staff (Level of Effort = 100%) 1. Name Title $ - /mo 0.00 $ - 0.00 $ - 0.00 $ - 0.00 $ - 2. Name Title $ - /mo 0.00 $ - 0.00 $ - 0.00 $ - 0.00 $ - 3. Name Title $ - /mo 0.00 $ - 0.00 $ - 0.00 $ - 0.00 $ - 4. Name Title $ - /mo 0.00 $ - 0.00 $ - 0.00 $ - 0.00 $ - Subtotal Project Full-Time Staff $ - $ - $ - $ - B. Project Support Staff (Level of Effort < 100%) Level of Effort 1. Name Title 0% $ - /mo 0.00 $ - 0.00 $ - 0.00 $ - 0.00 $ - 2. Name Title 0% $ - /mo 0.00 $ - 0.00 $ - 0.00 $ - 0.00 $ - 3. Name Title 0% $ - /mo 0.00 $ - 0.00 $ - 0.00 $ - 0.00 $ - Subtotal Project Support Staff $ - $ - $ - $ - TOTAL STAFF/PERSONNEL $ - $ - $ - $ - II. STAFF BENEFITS A. Benefits - Project Full-Time Staff 1. Name & Title of Staff Member or Type of Benefit $ - /mo 0.00 $ - 0.00 $ - 0.00 $ - 0.00 $ - 2. Name & Title of Staff Member or Type of Benefit $ - /mo 0.00 $ - 0.00 $ - 0.00 $ - 0.00 $ - 3. Name & Title of Staff Member or Type of Benefit $ - /mo 0.00 $ - 0.00 $ - 0.00 $ - 0.00 $ - 4. Name & Title of Staff Member or Type of Benefit $ - /mo 0.00 $ - 0.00 $ - 0.00 $ - 0.00 $ - Subtotal Allowances/Benefits - Project Full-Time Staff $ - $ - $ - $ - B. Benefits - Project Support Staff 1. Name & Title of Staff Member or Type of Benefit $ - /mo 0.00 $ - 0.00 $ - 0.00 $ - 0.00 $ - 2. Name & Title of Staff Member or Type of Benefit $ - /mo 0.00 $ - 0.00 $ - 0.00 $ - 0.00 $ - 3. Name & Title of Staff Member or Type of Benefit $ - /mo 0.00 $ - 0.00 $ - 0.00 $ - 0.00 $ - Subtotal Allowances/Benefits - Project Support Staff $ - $ - $ - $ - TOTAL ALLOWANCES/STAFF BENEFITS $ - $ - $ - $ - III. CONSULTANT COSTS A. International Consultants 1. Name or Purpose of Contract $ - /day 0.00 $ - 0.00 $ - 0.00 $ - 0.00 $ - TOTAL CONSULTANT COSTS $ - $ - $ - $ - IV. TRAVEL COSTS PROGRAM TOTAL YEAR 1 YEAR 2 YEAR 3 APC Request for Proposals – Performance Evaluation of Child Care Reform Project in Uganda 17 A. International Travel 1. Staff Travel From: To: Origin Destination $ - /roundtrip 0.00 $ - 0.00 $ - 0.00 $ - 0.00 $ - 2. International Consultant Travel From: To: Origin Destination $ - /roundtrip 0.00 $ - 0.00 $ - 0.00 $ - 0.00 $ - 3. Airport transportation $ - /trip 0.00 $ - 0.00 $ - 0.00 $ - 0.00 $ - (for Staff and International Consultant Travel) Subtotal International Travel $ - $ - $ - $ - B. In-Country Travel & Ground Transportation 1. Air Travel $ - /trip 0.00 $ - 0.00 $ - 0.00 $ - 0.00 $ - 2. Ground Transportation $ - /day or tri 0.00 $ - 0.00 $ - 0.00 $ - 0.00 $ - 3. Specify Other In-Country Travel Cost $ - / 0.00 $ - 0.00 $ - 0.00 $ - 0.00 $ - 4. Specify Other In-Country Travel Cost $ - / 0.00 $ - 0.00 $ - 0.00 $ - 0.00 $ - Subtotal In-Country Travel & Ground Transportation $ - $ - $ - $ - C. Per Diem 1. Staff - International Travel Destination $ - /day 0.00 $ - 0.00 $ - 0.00 $ - 0.00 $ - Destination $ - /day 0.00 $ - 0.00 $ - 0.00 $ - 0.00 $ - 2. Consultant Travel Consultant Name or PurposDestination $ - /day 0.00 $ - 0.00 $ - 0.00 $ - 0.00 $ - Subtotal Per Diem $ - $ - $ - $ - TOTAL TRAVEL COSTS $ - $ - $ - $ - VI. OTHER DIRECT COSTS A. Office Costs 11 Specify Other Direct Cost $ - / 0.00 $ - 0.00 $ - 0.00 $ - 0.00 $ - 12 Specify Other Direct Cost $ - / 0.00 $ - 0.00 $ - 0.00 $ - 0.00 $ - 13 Specify Other Direct Cost $ - / 0.00 $ - 0.00 $ - 0.00 $ - 0.00 $ - TOTAL OTHER DIRECT COSTS $ - $ - $ - $ - IX. INDIRECT COSTS (IF APPLICABLE) A. Indirect Costs 0.00% $ - TOTAL INDIRECT COSTS $ - $ - $ - $ - TOTAL BUDGET #REF! #REF! #REF! #REF! xlviii ANNEX VII: EVALUATION METHODS AND LIMITATIONS (see below) 1 Maestral International, L.L.C. Evaluation design for the Performance Evaluation of Care Reform Project in Uganda Evaluation dates: 1 January, 2015 – 31 August, 2017 Maestral International, L.L.C. Partnering with: Oxford Policy Management and Makerere University Name of contracted organization: Maestral International, L.L.C. Organization address: 150 South 5th Street, Suite 3250 Minneapolis, MN 55402 USA Organization phone: + 612.354.3042 Organization fax: + 612.605.0156 Primary contact name: Ms. Anna Nordenmark Severinsson Primary contact phone: +41-79-452.07.56 Primary contact email: anordenmark@maestral.org Secondary contact name: Ms. Nicole Williams Secondary contact phone: +612.354.3042 Secondary contact email: nwilliams@maestral.org 27 March, 2015 2 Table of Contents I. BACKGROUND ......................................................................................................................................... 4 II. PROJECT HYPOTHESIS AND THEORY OF CHANGE ......................................................................... 4 III. OBJECTIVE AND PURPOSE OF THE EVALUATION ............................................................................ 5 IV. EVALUATION QUESTIONS ..................................................................................................................... 6 Impact ......................................................................................................................................................... 6 Relevance .................................................................................................................................................. 7 Effectiveness .............................................................................................................................................. 7 Efficiency .................................................................................................................................................... 8 Sustainability .............................................................................................................................................. 8 Rights Based .............................................................................................................................................. 8 Lessons learned ......................................................................................................................................... 9 V. EVALUATION DESIGN AND METHODS ................................................................................................ 9 Policy based analysis ................................................................................................................................. 9 Qualitative component of the evaluation ................................................................................................. 10 Quantitative component of the evaluation ...............................................................................................11 VI. TOOLS AND INSTRUMENTS.................................................................................................................12 VII. SAMPLING STRATEGY ..........................................................................................................................13 District sampling .......................................................................................................................................14 Selection of child care institutions and remand homes ..........................................................................14 Selection of households and children .....................................................................................................16 VIII. ANALYSIS PLAN AND VALIDATION OF FINDINGS ............................................................................16 Step 1: Analysis of the data derived from each method separately .......................................................16 Step 2: Triangulation of individual findings..............................................................................................18 Step 3: Synthesis of findings ...................................................................................................................18 IX. LIMITATIONS OF THE EVALUATION ...................................................................................................18 3 X. CHILD PROTECTION AND ETHICAL CONSIDERATIONS FOR DATA COLLECTION AND USE ...19 XI. ANNEX 1 – RESEARCH MATRIX – BASELINE, MIDLINE AND END-LINE EVALUATIONS ............21 Baseline research matrix .........................................................................................................................21 Midline research matrix ............................................................................................................................34 End-line evaluation design matrix ...........................................................................................................43 XII. ANNEX 2 – EVALUATION TIMELINE ....................................................................................................59 4 I. Background The evaluation is an independent performance evaluation of a 36-month JSI/USAID-supported child care reform project that will be implemented in 12 districts of Uganda by a consortium of non￾governmental organizations. The project to be evaluated has two main objectives: 1) To reduce unnecessary separation of children from their families, and 2) To place children outside of family care in nurturing families. Each of these objectives have quantified targets. To achieve these objectives and ensure sustainability of the outcomes in the lives of children, the project also aims at supporting a reform and strengthening of the institutional and policy environment of the child care system in Uganda. The performance evaluation to be undertaken by Maestral international, in partnership with Oxford Policy Management and Makerere University, will measure the performance and sustainability of the project activities and will also identify broader lessons related to preventing unnecessary separation of children from families and facilitating family reintegration. The evaluation will be implemented in three phases, namely baseline, midline and end-line data collection phases which will happen in parallel with the implementation of the project. II. Project hypothesis and theory of change The evaluation will establish the performance and lessons learned on a project hypothesis that will test the effectiveness of an integrated package of interventions for families and children at risk and a theory of change that takes a point of departure in a series of assumptions on how these activities and methodologies will generate sustainable outcomes in the lives of beneficiaries and in the operations of the formal child care system. The overall assumptions of the project as articulated in the project documents are that: 1. A reduction in unnecessary separation of children from households which are considered “at risk of family separation”, can be achieved through strengthened and stabilized economic conditions in the households; providing vulnerable households with labor market skills; strengthening parenting skills and an improved understanding by vulnerable households and their communities of the benefit of family based care over institutional care. 2. It is furthermore assumed that if vulnerable families receive this kind of support, children living in the supported households will show reduced psychological distress. 3. Placements of children who are outside of family care (in residential care or in street situations1 ) into stable family environments can be attained through tracing of families, family re-unification activities or placements in alternative family based care. Implementation will be done collaboratively between district officials, selected child care institutions, remand homes, alternative care panels, probation social work officers, para-social workers and community child protection structures. This includes formal structures such as Village Child Protection Committees and Village Council members, as well as informal structures like faith-based organizations and district inter-faith committees. 4. It is furthermore assumed that such reintegration activities will generate an improved wellbeing for the concerned children. 1 To define and identify children living in the street, the evaluation will use a definition of that has been widely adopted by UNICEF and other organisations working with vulnerable children. This is, “any girl or boy who has not reached adulthood, for whom the street (in the broadest sense of the word, including unoccupied dwellings, wasteland, etc.) has become her or his habitual abode and/or sources of livelihood, and who is inadequately protected, supervised or directed by responsible adults” (Inter-NGO, 1985). This definition was formulated by Inter-NGOs in Switzerland in 1983. “Children of the street” are homeless children who live and sleep on the streets in urban areas. They are totally on their own, living with other street children or homeless adult street people. 5 5. It is also assumed that, through involving the various stakeholders of the child care system in the testing of this integrated package, using a participatory approach and implementing a range of capacity building strategies, this will not only support the implementation of national policy but will also produce sustainable institutional changes and changes in professional practices. These changes include among others, strengthening enforcement of care standards, gatekeeping practices, development of preventative family supports and shifting the reliance on residential care towards family based alternative care services. It is also expected to increase the government’s commitment of resources to the sector which will ensure sustainability of the care reform efforts. As such, data collection will take place in the various stages of the evaluation, from three levels of a theory of change at input-, outcome and impact as illustrated below. III. Objective and purpose of the evaluation The objective of the evaluation is to: 1. Assess the performance of the project with regards to creating sustainable changes in the lives of two beneficiary groups, namely 43,000 vulnerable households and 2,000 children at risk as a result of an integrated package of support provided through the project; and 6 2. Assess how these results were made possible through systems changes and identify which strategies and approaches were the most effective for achieving the change in children’s lives. With this in mind, the purpose of the evaluation is to: 1. Establish and ensure accountability to the donor, government stakeholders and the beneficiaries of the support; 2. Contribute to learning, including documenting and generating lessons learned on how current approaches can be strengthened to support vulnerable groups and prevent family separation, and reintegrate children back to nurturing family environments. This learning will be used to inform current programming both in Uganda and elsewhere, and help the implementing partners and the donor nurture current policy debates on child care system reform. In order to achieve this purpose, the evaluation will take a participatory and formative approach and ensure close liaison with stakeholders, locally in country and at global level, including USAID Displaced Children and Orphans Fund (DCOF), JSI/APC and the Implementing Partners. IV. Evaluation questions During a brainstorming session with the implementing partners and the donor, a number of evaluation questions were identified and will guide the evaluation throughout all three phases. The questions were based on a prior identification of the stakeholders to be included in the project’s activities and an analysis of what would be the reasonable influence that the project could have on these stakeholders.2 The below evaluation questions are organized according to OECD/DAC evaluation criteria on the impact, relevance, effectiveness, efficiency and sustainability of the project. In addition to these criteria, the evaluation will respond to questions regarding the human rights based approach taken by the project and questions have been articulated to capture issues of gender sensitivity of the project3 . While the evaluation will seek to document lessons learned especially in the areas identified below, it will also identify and document any other pertinent lessons which can be derived from the project’s implementation. The below questions will be adapted for the baseline and midline phases of the evaluation: Impact 1. To what extent and how did this project identify/test successful interventions that a) prevent child separation and b) support reintegration of children at risk or in residential care? Has their well-being improved? For which children was impact most visible? 2. To what extent and how did the programme contribute to the reduction in the number of children living in institutions and on the street in the targeted districts? (Disaggregation by children reintegrated from institutions and from the street) 3. To what extent and how have prevention methods employed by the project increased nutrition, education opportunities, care, and protection for children in targeted households? 2 Use has been made of stakeholder analysis guidance such as: http://www.odi.org/publications/5257-stakeholder￾analysis and https://openknowledge.worldbank.org/bitstream/handle/10986/6652/390220Tools0fo101OFFICIAL0USE0ONLY1.pdf? sequence=1 3 Use was made of: http://www.usaid.gov/sites/default/files/documents/1865/201sae.pdf 7 4. To what extent, and which systemic changes did the project bring about at community, regional, and national levels that are enabling children to live in family care and preventing inappropriate placements in institutional care? In particular, this refers to:  District authorities and OVC committees  Alternative care panels  Probation Social Work Officers  Community Development Officers  Remand homes  Child Care Institutions  Para-social workers  Community child protection structures  National bodies overseeing Alternative Care Framework implementation 5. To what extent and how did the project contribute to implementation of national care reform measures, for example:  Functioning and capacity of probation social work officers  Functioning of alternative care panels and as such contributing to implementation of Alternative Care Framework (de-I)  Capacity of Ministry of Gender, Labour and Social Development to provide an oversight gatekeeping role (data collection and monitoring of the reform, inspection and enforcement of the law and policy)  Visibility of the need for a reform of the child care system (and inclusion in sector reform measures)  Promotion of preventive and family support services  Promotion of system for family and community based alternative care (e.g. foster care, kinship care, supported independent living) and national adoption. Relevance 6. To what extent and how did the project’s integrated model, which links family strengthening, child protection and economic strengthening measures, offer examples and approaches for expansion, adaptation, and/or replication? What combination of interventions within this model could be adapted, expanded? 7. How did the political environment and context of the project influence operation and impact, and to what extent was the project able to mitigate risks and leverage new circumstances/opportunities which arose? 8. To what extent and how was the integrated approach model, which links family strengthening, child protection and economic strengthening measures to maximize prevention for children at risk of separation from family, and support those returning to family based care, relevant and sufficiently adapted to the communities where the project operates.? 9. Based on what is found to be the most common risk factors to separation, to what extent and how did the project manage to address these in an appropriate manner, including for boys and girls? Effectiveness 10. To what extent have the targets of the project to support 43,000 households and 2,000 children to be reintegrated been reached and how were they achieved? 11. To what extent did the beneficiaries of the project, including children, families and communities, appreciate the services provided? Did the project incorporate sufficient participatory feedback 8 processes to ensure the target reach was involved and informed of the intervention and received services most appropriate for their needs? 12. What were the most significant changes in the lives of the two target groups which were achieved by the project, and what factors seem to have influenced these changes? 13. How effective was the project in engaging with various structures and professionals involved in the project, including district authorities, child care institutions, probation social work officers, community development officers, alternative care panels, remand homes, community child protection structures, para-social workers etc.? 14. To what extent did the project manage to disseminate information regarding lessons learned from its activities and strategies? How did the project adjust its own strategies to take into account any learning that derived from the implementing process? Efficiency4 15. To what extent were the project strategies, to reduce vulnerability of households and to reintegrate children, efficiently implemented? Were there any implementation gaps? 16. Which strategies / activities used by the project proved to be the most efficient in influencing the lives of children and vulnerable families? Were there any strategies / activities which were not at all efficient? 17. Which strategies used by the project proved to be the most efficient in influencing the system reform? Were there any strategies / activities which were less efficient? 18. To what extent were the human resources, coordination mechanisms, time and financial resources invested in the implementation of the project efficiently used and were they sufficient to achieve the targets of the project? Sustainability 19. For the children who have been reunified with their families or placed in alternative family based care, what reintegration methods have supported stable and sustained placement of children and to what extent? Are children still in the homes and in school 6 months (and more) after placement? What makes these reintegration methods sustainable? If there has been relapse in reintegration, why has this been the case? 20. By project end, to what extent have structures been established or strengthened that can continue to provide, on an ongoing basis, adequate case-management and support services for children at risk? Are these sustainable and what makes them sustainable? 21. For the different changes observed in the capacity and functioning of professionals, child care structures and coordination mechanisms (e.g. probation social work officers, para professionals, alternative care panels), to what extent are they likely to sustain the work, to continue meeting and to continue to apply what they have learned after the end of the project? In addition to the DAC criteria, we will assess the programme on two additional domains which we believe are pertinent to successful implementation of the programme. Rights Based 22. To what extent did the project manage to target those most at risk of separation from their families, including tailoring activities to accommodate needs of the most vulnerable children? 4 In the context of this evaluation, as we are not conducting a value for money assessment, efficiency will be measured through subjectively looking at how well-organised, competent, non-wasteful the project and its activities were. 9 23. To what extent was the project able to prevent child protection issues and human rights violations (e.g. child labour, violence against children, neglect, early marriage, etc.) and mitigate risks for creating greater vulnerability (e.g. child labour and any kind of harm)? 24. How did the project address child rights violations that required urgent attention? 25. To what extent did the project contribute to strengthening duty bearers’ knowledge of their obligations and their capacity to meet those obligations? Lessons learned 26. What lessons can be learned from the project’s work with child care institutions? 27. What lessons can be learned from the project’s integrated package of support? 28. What lessons can be drawn from work with different professionals involved in gatekeeping and outreach to vulnerable families (e.g. para professionals, how are they working together with probation social work officers and what is motivating them in their work)? V. Evaluation design and methods As further elaborated below, the evaluation will rely on a mixed-methods approach5 with a strong qualitative focus of the primary data collection to be done by the evaluation team. Given that the focus of the evaluation is on the overall performance of the project rather than on quantitative impact estimates of the interventions, the overall evaluation design will be non-experimental. This means that the data collection will be done only on the ‘treatment’ group of children and households that benefit from the project, without sampling a ‘control’ group. As such, it will be possible to establish contribution effects of the project in the lives of beneficiaries and towards systemic reform, but it will not be possible to determine what would have happened in the absence of the intervention. We have accordingly designed the evaluation questions to reflect this; focusing on the change among participants, rather than trying to establish causality, which would require a comparison group. Since the evaluation places a strong emphasis on learning and generating lessons learned it will use a semi-fixed design. It is proposed that the evaluation team relies on the current design for the research, but remains flexible to adjust interview lists, data collection tools and analytical approaches as it deems necessary in order to capture significant changes, side-effects and interesting lessons learned which the project may have influenced. The evaluation uses a rigorous qualitative approach, including a range of qualitative evaluation tools, to meet the multiple objectives of the project. Qualitative data collection and analysis would be undertaken at different points during the evaluation’s three different stages (base-line, mid-term review and end-line data collection) and will cover different samples of individuals and groups associated with the complementary goals of the project. In particular, qualitative information will be derived from household case studies, key informant interviews and focus group discussions, which will be undertaken using a range of participatory tools. A parallel policy based analysis of the child care system will attempt to monitor the systemic changes brought about by the implementation of the project. The analysis of data collected from each of these samples will be used to either triangulate findings, or to answer the different evaluation questions. Policy based analysis One objective of the project is to bring about systemic change in child care structures and institutions at national, district and local level. We will review relevant policy and child care system regulatory documents, case files and have key informant interviews with policy makers, civil servants and 5 http://usaidprojectstarter.org/sites/default/files/resources/pdfs/Mixed_Methods_Evaluations_Technical_Note.pdf 10 management of service providers to form a view on the current set up of the system and how the project contributes to its reform. This set of interviews will be held with a group of stakeholders who represent the core of the current child care system at the district and national levels, including - National bodies overseeing Alternative Care Framework implementation - District authorities and OVC committees - Alternative care panels - Probation Social Work Officers (PSWOs) - Community Development Officers (CDOs) - Remand homes - Child Care Institutions (CCIs) The evaluation will collect and analyze quantitative data on systemic indicators such as the number of residential care facilities inspected, the number of children in various forms of alternative care (residential care and family based alternative care), the number of children deinstitutionalized or reintegrated in families from the street and institutions, and the number of institutions closed or transformed. A survey is envisaged with PSWOs at district level to complement the qualitative picture established through key informant interviews and analysis of statistics as described above, and to assess the capacity of the professionals working under the Ministry of Gender, Labor and Social Development’s supervision to implement the country’s Alternative Care Framework. This survey will be administered with the support of the Ministry’s mailing system and is proposed to be self￾administered to PSWOs in all districts of Uganda. Such survey will provide insights on any differences between project districts and non-project districts in the PSWO’s skills, awareness, oversight, supervision and reporting, referrals, coordination and follow-up with other service providers. The feasibility of the survey was discussed with project partners and a representative of the alternative care unit in the Ministry of Gender, Labor and Social Development during the evaluations planning phase. A formal agreement to conduct the survey was not possible at the time of the planning mission, however it was agreed that the process would be formalized during the baseline phase when the survey data would be collected for the first time. The evaluation team will fully rely on the Ministry of Gender, Labor and Social Development for the dissemination of the survey and compilation of responses. Qualitative component of the evaluation A core component of the evaluation is represented by a qualitative analysis of the performance of the project, which aims to provide a clear idea of how the project is affecting the wellbeing of vulnerable children and their households (particularly around domestic violence, substance abuse, gender-based violence and cultural practices). In particular, this analysis will focus on two distinct areas related to the project. On the one hand, information will be collected on the perceptions of children and household members regarding their changing conditions of vulnerability, general wellbeing, and prospects for the future. On the other hand, information will be collected on the mechanisms through which the different interventions have unfolded and on the channels through which they have produced their effects at both the household and the community levels—how does change happen? This involves looking at operational aspects of the programme, but also testing the assumptions underpinning the programme’s Theory of Change, whether or not they hold, and what the implications are for the design and implementation of the programme. Together with the monitoring data collected through the PMP, this wide-ranging qualitative assessment will form the basis of our understanding of beneficiaries’ self-assessment of their personal welfare as a result of the interventions and will help us assess changes in parenting skills within the household, changes in child care systems and attitudes in the local community. This will inform our assessment of whether the planned activities are appropriate and adequate to achieve the desired objectives. 11 Two distinct samples of children will be selected from the two groups that are expected to benefit from the project. The first sample of children will be derived from the 43,000 vulnerable children who are expected to show some improvements in wellbeing outcomes as a result of support interventions targeting the households in which they are living. The second sample of children will be derived from the 2,000 children whose outcome indicators are expected to improve as a result of them being reintegrated with their families of origin or placed into alternative family-based care. Both samples of children will be interviewed twice, at the beginning and at the end of the evaluation; however, whilst the qualitative data from the first sample will be collected at baseline and at end￾line, the data from the second sample will be collected at mid-line and end-line. This sequencing reflects the fact that the reintegration of children into family-based care will take time and any improvements in their welfare indicators are expected to be observable after twelve months. Quantitative component of the evaluation Monitoring data represents a wealth of information that will form the basis for a quantitative component of the evaluation analysis centred on the indicators for which the data will be collected. The quantitative component of the evaluation will aim to analyze the influence of the project interventions on the two target groups, namely the 43,000 vulnerable children who are expected to show some improvements in wellbeing outcomes and the 2,000 children whose outcome indicators are expected to improve as a result of them being reintegrated with their families of origin or placed into alternative family-based care. The evaluation will seek to establish the proportion of vulnerable children and children at risk who are showing improvements and attempt to establish trends in these improvements, noting any specific areas which show consistent improvements or which show little or no improvements as a result of the project interventions. The evaluation will also seek to analyze any specific factors which seem to influence improvements in wellbeing more than others. For this analysis, the evaluation will be relying on monitoring data from the project. Quantitative monitoring data will be collected before the project is implemented, with the aim of gathering relevant information on the wellbeing status of households and children that will be targeted by the support interventions. This baseline analysis will provide us with an idea of the number and proportion of households and children that appear to be particularly vulnerable, as measured by the assessment tools used during the data collection process. This could also inform the identification of project targets in terms of expected proportional and absolute improvements, on the basis of the indications emerging from relevant baseline values. At the same time, the data collected at baseline will form the basis for a descriptive analysis disaggregated along gender lines, age groups and geographic location. Data collected at baseline will pave the way for an assessment of the changes measured over time and at the end-line on key outcome indicators. Additionally, the data will help draw a broad picture of the pre-intervention issues affecting households and children. The quality of the monitoring data will be assessed through random checks carried out during the course of the qualitative study. Field teams undertaking qualitative data collection will randomly select a limited number of 24 households (4 per district in 6 districts) that have allegedly participated in the monitoring exercise. The respondents will be asked whether data was collected by the implementing partner as part of the monitoring plan and which data collection tools were used. A reduced version of the data collection tools (FSVI and CSI) will be administered by the field teams to determine whether the data collected during the monitoring exercise is accurate and consistent. The datasets created as part of the monitoring exercise will also be reviewed to assess the quality of the data entry process and ensure that the data files (Excel or SPSS) meet the basic standards needed to perform data analysis. 12 VI. Tools and instruments For the primary data collection to be undertaken by the evaluation, a range of qualitative tools will be employed to collect information. These are detailed in annex 1 and include: A paper-based Survey with probation social work officers (PSWOs) will focus on areas in their capacity which the project has a potential to influence. This includes their knowledge and awareness of the alternative care framework, their involvement and working methods with alternative care panels, their capacity to perform core gatekeeping functions, as well as their supervisory skills and interconnectedness with other professionals such as those working in health facilities, or at community level, including community development officers (CDOs) and para-social workers. Household Case Studies will be conducted in households with “reintegrated children” as well as households with vulnerable children. In particular, we envisage the use of a life history approach, which will enable us to obtain a better understanding of household context, intra-household dynamics and adults’ attitudes towards children. The household case studies will be based on semi￾structured questionnaires and will utilize participatory tools in order to ensure that the interviewees have the freedom to elaborate on their answers. These case studies will provide us with a wealth of information on children’s wellbeing and how their welfare and social status has changed over time as a result of the project. Information on nutrition, educational attainment, health conditions and child labour patterns will all contribute to obtain a broad picture of how the project is performing. Focus Group Discussions (FGDs). In addition to individual interviews, focus group discussions will be conducted to encourage a collective discussion of the circumstances in which the project is operating at the local level. In this case, the emphasis will be placed on changes at the community level that the different interventions are likely to generate and on the reactions that these changes will have produced. We propose to conduct three sets of FGDs per parish: one for women from the community, one for men from the community, and one for those involved in community child protection structures, such as religious leaders, school teachers, health care workers and members of community organizations. The participants to these focus groups will be representatives of relevant local institutions that could have an impact on child care in the communities. Key Informant Interviews (KIIs). We also propose to conduct key informant interviews (KIIs) with relevant stakeholders at the district, sub-county, parish and community level. These may include, for example, desk officers for the Ministry of Gender, Labour and Social Development, district OVC￾committees, representatives of alternative care panels, staff of NGOs, and various parish leaders. This will also feed into the policy based analysis. The qualitative research will also make use of various social research tools to analyze processes, stakeholders and institutions. These could include, for example:  Gender analysis. This tool can generate empirical data and analyse the differential impact of the project on men, women, girls and boys with a focus on the broader social rules that govern gender roles and relations, especially around care. This is particularly relevant, as care responsibilities are often prescribed along gender roles, and we would want to be able to assess the programme’s ability to reach and influence both male and female caregivers. It may also help identify disparities in outcomes between boys and girls, as well as address key issues around the relevance of the programme by highlighting the roles and status of males and females, their different needs and priorities (both children and caregivers) and the possible different effects. Gender analysis will also guide the development of data collection tools to track gender-related project impacts.6 6 USAID (2011) Tips for Conducting a Gender Analysis at the Activity or Project Level. http://www.usaid.gov/sites/default/files/documents/1865/201sae.pdf accessed 5 March 2015. 13  Institutional mapping. A method of identifying and representing perceptions of key formal and informal institutions, individuals and their relationships. Enables understanding of underlying structural causes of care related issues.  Causal flow diagram. A participatory method of showing diagrammatically the relationships between variables associated with poverty, exclusion or social change. It traces differences in cause-effect relationships by different groups including children and different caretakers.  Stakeholder analysis matrices. These can be used to plot the stakeholders—duty bearers and rights holders—and analyse their roles and relationships.7 Quantitative data will be drawn from results of the tools which the implementing partner has proposed to use, the OVC Vulnerability Index and Child Status Index. The tools (attached in annex) will be used as follows:  At baseline and end-line, Household Case Studies with Vulnerable Households: 4 case studies in 2 parishes for each of the 6 districts to be visited. Total per round: 48 (24 boys, 24 girls)  At baseline, FGDs with Children Living in the Street: 4 FGDs in the district where this intervention is being carried out. Total: 4 FGDs (2 with girls, 2 with boys, with 12 individuals participating in each FG)  At baseline, FGDs with children in remand homes: 2 FGDs in 2 of the districts where this work is taking place. Total: 4 FGDs (2 with girls, 2 with boys)  At baseline, FGDs with Children Living in Residential Care: 2 FGDs in 2 child care institutions for each of the 6 districts to be visited. Total: 24 FGDs (12 with girls, 12 with boys)  At midline and end-line, Household Case Studies in homes with “Reintegrated Children”: 4 FGDs in 2 parishes for each of the 6 districts to be visited: Total per round: 48 FGDs (24 with boys, 24 with girls)  At baseline, midline, and end-line, 3 FGDs (1 with women, 1 with men, 1 with community child protection structures) in 2 parishes for each of the 6 districts to be visited. Total per round: 36  At baseline, midline, and end-line at least 6 Key Informant Interviews with relevant national child care system stakeholders, 6 Key Informant Interviews and 1 FGD with district level child care system stakeholders, including staff in child care institutions and remand homes in each of the 6 districts to be visited. Total per round: 42 KII and 6 FGDs  At midline and end-line: checks of programme monitoring data for each of the vulnerable households and households with children reintegrated from institutions and the street in at least 48 households across the 6 districts to be visited. A sound ethical research design will ensure the safeguarding of participating children and their caregivers. Data collection tools involving children will be participatory, and take into account children’s age and understanding of the situation. More on ethical considerations can be found under section X. VII. Sampling strategy The evaluation will collect data from 6 representative districts, purposively selected from the 12 targeted by the project and will aim to cover both of the target groups. We recognize that the administrative units of Uganda include sub-county and parish levels, but we have chosen not to 7 See also: http://www.odi.org/publications/5257-stakeholder-analysis and https://openknowledge.worldbank.org/bitstream/handle/10986/6652/390220Tools0fo101OFFICIAL0USE0ONLY1.pdf ?sequence=1 14 structure our geographic sampling strictly around these divisions, as targeted households and institutions may not be evenly distributed across counties and parishes. Therefore, rather than establishing sampling criteria for sub-county and parish level, below we outline how we will sample the study districts and the institutions, households, and communities that will be part of the evaluation. District sampling Six (6) study districts will be selected purposively from the 12 programme districts based on criteria that are closely linked to the project’s objectives and in order to represent the large variety of operational realities. The six study districts will be selected based on the following criteria: 1. High numbers of children in care 2. Low numbers of children in care 3. High levels of social vulnerability (deriving from cultural and social factors such as potential risks due to polygamy, forced early marriages, domestic violence) 4. Relatively lower levels of social vulnerability (deriving from cultural and social factors such as potential risks due to polygamy, forced early marriages, domestic violence) 5. High levels of economic vulnerability affecting the level of poverty of households (deriving from factors such as livelihoods and development, infrastructure, exposure to conflict, border district) 6. Relatively lower levels of economic vulnerability affecting the level of poverty of households (deriving from factors such as livelihoods and development, infrastructure, exposure to conflict, border district) When districts are selected based on these 6 criteria it will also be ensured that:  The one district that will have a convergence of all project elements will be represented among the 6 sampled districts.  A geographical coverage of north, south, central, eastern and western regions of the country are represented to the extent that this is possible. Selection of child care institutions and remand homes The second step in the sampling strategy will be to select a sample of child care institutions and remand homes that are included in the project. Given that the project will work to reintegrate children from child care institutions in all six of the study districts, while working with children in remand homes only in one district, the sampling strategy for these two types of institutions will be different. For child care institutions, a purposeful sampling will be done across all six study districts to ensure the best representation possible of the characteristics of institutions that can be found in Uganda. To the extent possible, this will include for example:  Child care institutions of different sizes, including small, medium and large institutions  Child care institutions which include a representative range of age groups, such as facilities mainly being home for children under 3 years of age, mostly children over 15, etc.  Child care institutions with and without children with special needs (including physical or mental disabilities, and HIV/AIDS, residential special needs schools)  Child care institutions with strong links to other modes of alternative care or adoption, for example institutions with strong links to kinship communities or local community, institutions with group homes, institutions with adoption (international and domestic) programmes etc. 15 At the time of selecting the child care institutions, the evaluation team will also be informed by the results from the assessment of institutions done as part of the project’s strategy for selecting child care institutions to work with. Any significant characteristics of these child care institutions which stem out from this assessment and are found to be important, and that are not covered in any of the categories mentioned above, will be included. Supporting evidence for an institution’s inclusion or exclusion from the purposive sample will be drawn from the implementing partner’s profiling and data on targeted institutions. The below list represents the known sample frame to date, to be completed once the project has selected the institutions to be included based on the process of assessment. District Northern districts Sample frame (all CCIs) Number and characteristics of institutions included in the project (to be completed based on assessments of CCIs, due to be finished end of May 2015) Gulu (district included in first round of assessments of CCIs) 12 institutions Lira 7 institutions Western districts Kasese (district included in first round of assessments of CCIs) 7 institutions Kabarole (district included in first round of assessments of CCIs) 26 institutions Kabale 6 institutions Eastern districts Mbale (district included in first round of assessments of CCIs) 7 institutions Kamuli (district included in first round of assessments of CCIs) 6 institutions Jinja 40 institutions Iganga 8 institutions Central districts Luwero (district included in first round of assessments of CCIs) 19 institutions Kampala (district included in first round of assessments of CCIs) 47 institutions Wakiso 38 institutions For remand homes, a purposeful sampling of two remand homes will be done to ensure that the project elements of convergence are captured and to ensure a geographical spread of the facilities. Geographical region District and name of remand home Convergence of project elements Eastern region Mbale remand home (closed facility) District where all project elements will converge Central region Naguru remand home (closed facility) No convergence of project elements 16 Western region Fortportal remand home (closed facility) No convergence of project elements Northern region Gulu remand home (closed facility) District where work in remand homes converge with work in CCIs and in communities. National coverage Kampiringisa National rehabilitation centre (semi-open facility), placed in Mpigi district No convergence of project elements Selection of households and children At the level of individual children, we will select an even number of boys and girls. The Household Case Studies will be selected purposively, to include households representing a range of livelihoods strategies (if possible), and demographic diversity such as household size and composition. Community focus groups will comprise of homogenous groups of 8-12 women or men; respondents will be selected to provide opinions and views across a range of age, social status, and caregivers/non-caregivers. Focus groups with community child protection structures will comprise of groups of 8-12 individuals, male and female, who can speak to the vulnerabilities in their communities, as well as the roles of formal and informal support structures which affect the wellbeing of children in their community. We assume that not all children in an institution will be from families native to the parish where the facility is located. Therefore, where possible, communities will be sampled based on the communities where the highest proportion of children come from that particular community or parish, within a reasonable distance from the institution. VIII. Analysis plan and validation of findings The evaluation analysis plan includes three distinct steps, taking into account and benefitting from the mixed method approach of the evaluation. Step 1: Analysis of the data derived from each method separately Policy-analysis will be done using information from both qualitative and quantitative methods. Descriptive statistics will be generated on a number of systemic indicators included in the PMP which focus on the performance of the system to implement the country’s alternative care framework. Some of this data will be compared with data collected through the government’s management information systems in order to explore whether the work of the project on inspections of CCIs, reintegration of children is also contributing to improved statistics at the level of the Ministry of Gender, Labour and Social Development. A qualitative analysis from key informant interviews with system stakeholders will focus on key messages and story lines on the capacity of various stakeholders and changes in child care system reform in the three main areas of prevention, gatekeeping and provision of family based alternative care, including deinstitutionalization. In addition to this, descriptive statistics will be developed from the survey with PSWOs and will provide a more comprehensive picture of status and changes in awareness, skills, referrals, coordination and oversight/supervision of social work professionals and para-social workers resulting from the project. Descriptive statistics for this component of the evaluation will be generated from Excel. 17 More complex quantitative analysis will also be done for the outcome indicators in the PMP referring to the household- and individual-level analysis (from the two tools CSI, FSVI). In this respect, the analysis plan is divided into two phases: data entry and initial cleaning with EPIDATA software and data analysis with SPSS and STATA. The quantitative data emerging from the PMP from these two tools will serve three separate but interlinked analytical purposes: (1) obtaining process routine data that will assist the implementing partners to monitor the progress achieved in the course of the programme roll-out; (2) collecting information on households and children in order to assign them into pre-defined categories associated with different programme interventions; and (3) gathering a wealth of household- and child-level micro-data that will enable the evaluation team to carry out a descriptive analysis of the programme impact on a range of wellbeing domains of interest. The latter will also serve as a parallel line of analysis aimed at supplementing the findings of the qualitative study undertaken by the evaluation team. The differentiation between the first and the second analytical objectives is broadly defined by the use of output indicators for process routine data and outcome indicators for household- and child￾level data. The PMP reports the nature of the indicator for which the information is gathered, distinguishing between output and outcome indicators. In addition, a number of output indicators have been selected with the aim of linking the analysis on changes pertaining to outcome indicators with the implementation of specific programme interventions that are likely to affect them. The PMP refers to this cross-analysis in the sections regarding the plan for data analysis of the specific output indicators that will be used to link programme interventions with households’ and children’s wellbeing indicators. In order to make this cross-analysis possible, the data collection tools concerning household- and child-level outcome indicators have been enhanced with the inclusion of questions directly relating to the relevant project outputs. In fact, the FSVI and CSI now contain an initial section named ‘Identification of the Household with Project Interventions’ and ‘Identification of the Child with Project Interventions’ respectively, which include a range of output indicators related to the respondent’s status. The latter will be identified through the use of indicators relying on a ‘score’ system based on crucial wellbeing domains, which will in turn form the basis for a more in-depth descriptive analysis. This multi-layered analysis will be aimed at understanding whether the variables measuring the wellbeing of the households and the children are changing/improving over the course of the project and whether the implementation of specific programme interventions are producing any noticeable impact for the target recipients. The analysis of the qualitative data collected from children, households and communities will commence during daily debriefing sessions run by the respective research team leaders, in which common and unusual themes will be identified and analysed, alongside any differences found between social groups. The daily debriefings will also enable the researchers to clarify issues and strengthen understanding by incorporating additional key questions into the next day of research. Each research team will produce a district-level research report for internal use by the evaluation team, which provides a summary of the research process and a summary of contextual issues and key preliminary findings. The data from each FGD and KII will then be transcribed and, after a process of review and clarifications, imported into NVIVO8 . We will develop a data coding framework, focusing on the key impact areas and emerging themes associated with the Theory of Change, as well as constraining 8 Qualitative data analysis software. 18 and facilitating factors related to effectiveness, relevance, and sustainability. Using NVIVO, data under each research theme can be disaggregated and analysed by gender, district, community, and respondent type (child, caregiver, opinion leader, institutional staff, etc.) in order to identify differences in experiences and perceptions. Step 2: Triangulation of individual findings As a second step, we will look at the qualitative, quantitative and policy based analysis components of the evaluation in parallel, identifying trends, and outlining where findings from the three lines of inquiry support or contradict each other. For example, the quantitative descriptive analysis will be enhanced by social workers’ reports that may provide greater insights into the causes of emerging trends across beneficiary households and children. Furthermore, while the overall evaluation aims at drawing some lessons on the project’s assumptions and theory of change, through the quantitative analysis it will be possible to establish links between outputs and outcome indicators of the project and this will be particularly interesting to support qualitative findings on the performance of the project’s theory of change. For example, if, and how certain interventions with children and households produce a change in wellbeing, as well as if, and how certain project inputs produce a change at outcome level of the system and contribute to impact of the project on the overall child care system. Step 3: Synthesis of findings In the third step, all methods, including the qualitative and quantitative components of the evaluation, together with the policy based analysis of the overall status of the child care system, progress and the direction of its reform, will provide the basis for answering the different research and evaluation questions to draw conclusions and lessons learned on the overall theory of change. These methods and analytical steps will provide us with insights into the collective dynamics influencing or being influenced by the project and will enable us to not only describe the performance of the project but also to identify and share best practices and lessons learned for this multi-layered project on child care reform. The interpretation of the findings will thereafter be integrated in the reports (baseline, midline and end-line evaluation reports). Validation of findings will take place after each phase of the evaluation, in the form of a presentation of the findings to implementing partners and the donor, and a subsequent discussion and reaction. IX. Limitations of the evaluation In the course of designing the evaluation framework, some limitations of the evaluation have been identified and are further described here. In this section we also discuss our approach to addressing these challenges. First, we realize that children interviewed in institutions may not be from the parish, or even the district, where the institution is located. As such, interviews with child protection mechanisms, men and women in the communities surrounding the institution may not reflect the exact situations which led the child to be placed in a care facility or the context into which they are being reintegrated. To mitigate the risk that this poses to the qualitative research, as described in section VIII above, we propose that, where possible, we will select communities based on the proportion of children in the facility that are from a particular parish within a reasonable distance of the facility. Second, it will not be possible for the evaluation team to do follow-up data collection with children remaining in the child care institutions or on the street. As the project aims at working towards the reintegration of these children, the midline and end-line data collection will be with the children from this beneficiary group who have been reintegrated with families. On the other hand, however, the 19 policy based part of this evaluation will assess the extent to which structures have been established, capacities, skills and work processes have improved in a way that continues to provide, on an ongoing basis, adequate case-management services for children at risk. For the children who remain in institutions or on the street, the data collection will therefore focus, in the midline and end￾line evaluation on the extent to which the project has contributed to systemic changes, and is able to provide follow-up with the children who remain on the street or in institutions. Finally, there is a risk that some children who have been reintegrated from institutions or from the street relapse because the reintegration efforts failed. To the extent that it is possible, the evaluation team will then try to identify those children as they can provide important information on what contributed to the failure of the reintegration effort. It is possible however, that the children may not be possible to track down. In that case, interviews, household case studies and focus groups with families, community members and child protection structures will attempt to capture information about what may have caused the reintegration effort to fail. X. Child protection and ethical considerations for data collection and use Conducting qualitative fieldwork of this nature requires high ethical standards to ensure that expectations are not raised, confidentiality is maintained, and respondents are never forced to participate or encouraged to speak about subjects that may be traumatizing or feel uncomfortable in any way. It is also important that all activities are age appropriate. We will draw on our experience of qualitative fieldwork to ensure that these considerations are met. Ethical considerations will influence the entire evaluation process, including the current evaluation design and data collection tools, composition, recruitment and management of the evaluation team; consultations and interviews with key informants; data storage and use9 . Since the evaluation will be directly collecting data from human subjects, some of these being children, the evaluation design and data collection tools will go through a formal approval process with OPM’s Ethical Review Committee. In addition, all members of the evaluation team will be briefed on guidelines for ethical research involving children (drawing, for example, on guidance developed by UNICEF) and held accountable for their behavior as per Maestral’s and OPM’s codes of conduct. In the planning phase of the evaluation the possibility to pass the evaluation design through a Ugandan based IRB was considered as well, and discussed both within the evaluation team and with the project partners. It was then agreed that the project partners would ensure agreement to implement the evaluation as per the current evaluation framework, and within the context of the DOVCU-project, would be sought from the Child Protection Working Group within the Ministry of Gender, Labour and Social Development, since the current evaluation is not an academic research of similar type as the research which usually goes through such ERB. All children, adults, households, and communities will be protected against any form of harm, manipulation and malpractice following established ethical guidelines on the subject. Informed consent on the objectives of our evaluation will be sought from all respondents. Regarding children to be involved in the data collection, an informed consent will be acquired from a parent or guardian and informed consent will be sought from the children themselves. Since most of the children to be involved in the data collection are from vulnerable populations, there may be cases where obtaining informed consent from a parent or guardian is difficult. In those cases, special care will be taken to develop appropriate strategies for communicating the implications of involvement in the research. The children will also be asked if there is any person of confidence that the child-respondent would 9 UNEG Guidance on Integrating Human Rights and Gender Equality in Evaluation accessed on: http://www.uneval.org/document/download/1294 20 like to accompany him/her in the interview. Special care will be taken to ensure that vulnerable children do not feel pressed or obliged to take part in the interview. Strict confidentiality will be maintained throughout the interviewing process as appropriate. Ethical and data protection issues will be addressed for our data collection on all the different samples. The evaluation will avoid any kind of harm to interviewees and interviewers. Since the evaluation will be collecting data from highly vulnerable households and children at risk it is possible that the evaluation team will encounter cases of children who are found to be in abusive or otherwise unacceptable situations within their families or communities. In such situations the evaluation team will follow what the legislation in Uganda foresees as the legally appropriate action and report such incidences to a professional social worker (PSWOs) for follow-up. The evaluation team will also inform the project partner about the situation so that project partners can check that appropriate follow-up was indeed provided by a professional. A similar strategy will be used to refer a child who is extremely vulnerable or in the case the interview brought up trauma. In all interview situations, children will be given the space to transparently and confidently bring up his/her concerns with the situation the child is in or with the interview itself. Data collection tools will include a referral form to be used for cases that require referral and the field researchers will be trained on how to use the same. Although guidelines on research with child respondents often do not specify a minimum age, we propose that children targeted for interviews in this evaluation should not be below the age of 10 years and will not be older than 18 years. Methodology and instruments for the evaluation are designed to take into account age of interviewees, and their suitability will be reviewed as part of the independent ethical review prior to the start of data collection. For the qualitative data collection with the two beneficiary samples, each phase of the evaluation foresees a testing of the data collection tools prior to a full roll-out of the data collection. This process will ensure full adaptation of the tools to the situations found on the ground. We note here also that research is likely to take place in Gulu district, and other parts of Northern Uganda which are considered post-conflict areas. These are areas where past displacement, major disruptions to livelihoods, death of caretakers, and the use of children as soldiers will potentially have significant impact on the wellbeing of children, and families’ ability to care for their children. To address this, we will also ensure a review of the evaluation framework by OPM’s Conflict Sensitivity Analysis team, to ensure that the evaluation is relevant for a post conflict context, and avoids negative impacts of the evaluation on potential conflict dynamics. 21 XI. Annex 1 – Research matrix – baseline, midline and end-line evaluations Baseline research matrix Level of data collection in theory of change Explanation Research questions Indicators / qualifiers of improvements Data collection methods / tools Level 1 – Impact on national child care system reform Data will be collected to define status and opportunities for child care reform towards ensuring that unnecessary separation of children from their families is reduced, namely expanding family support services, strengthening gatekeeping and ensuring reduced reliance on institutional care and expansion of family based options. Impact: In what way is the project contributing to the implementation of national care reform measures? Quantitative indicators: Numbers of residential care services inspected (CCI directory) Numbers of residential care services under transformation / closed (CCI directory) Numbers of residential care services provided with license (CCI directory) Number of districts with a plan for transformation / closure / upgrading of institutions and for expanding preventative family and child support services and foster care (Ministry of Local Government) Number and types of family based alternative care service providers Areas for qualitative inquiry Awareness within MoGLSD of core gatekeeping functions. Key Informant Interviews (KII) with national level Government stakeholders KII with national level child care reform stakeholders (beyond government structures) Desk review of policy documents and records from Central Government CCI directory, OVC-MIS and services directory 22 Relevance: How does the political environment and context of the project enable/inhibit the project operation and what seem to be the main risks and opportunities to be leveraged? Current status and regularity in the updating of various MIS (CCI directory, OVC-MIS, service directory of MoGLSD. Status of agreement in MoGLSD on core national indicators (and inclusion into the sectoral performance measurement plan) to monitor progress of child care reform (incl. the gatekeeping function of the Ministry) Status and regularity of inspections of CIIs, licensing CCIs as per Alternative Care Framework by MoGLSD. Status of agreement within MoGLSD on how the process and work-plan for how these inspections will be carried out Status on actions taken by MoGLSD for closure of CCIs Status on actions taken by MoGLSD to address irregular and unethical practices in child care homes Status of actions taken by MoGLSD and Ministry of Local Government to support the development of family based alternative care services as well as family support services Position of child care reform targets in OVC strategy document (2011-2015). Level 2a – Direct influence of the outcomes of the project at policy and system levels Data will be collected to define status of the capacity of District Governments and sub￾districts structures to monitor institutions and effectively implement the Impact: In what way is the project contributing to implementation of national care reform measures at district level and below? Quantitative indicators: % of child care institutions in the district that have been inspected, licenced, have an improvement plan, are acting on recommendations from the assessment Project Monitoring data using quantitative indicators for system change 23 country’s new Alternative Care Framework. Data will also be collected to define status of existing child care institutions, if there is agreement on the need to expand family based care and social work best practices. Data will also be collected to define the capacity of informal child protection mechanisms to do basic preventative work, facilitate identification of risk and ensuring referrals and their current level of interactions with the formal child protection system. Which systemic changes does the project have the potential to influence at community, district, and national levels that are enabling children to live in family care and preventing inappropriate placements in institutional care? % of children placed in a child care institution who have a care plan % of children placed in a child care institution who have a care plan that has been reviewed in the last 3 months % of targeted CCI and remand homes with reduced number of children living in care centers. Number of districts with alternative care panels established Number of cases discussed by alternative care panels monthly % of children in alternative care in the district (targeted by the project) who are in: child care institutions, foster care, biological family, kinship family, adopted domestically, adopted internationally Number of children recorded to be on the street in the district, disaggregated by boys and girls Proportion of new cases (inflow) of child care placements in the district by number of cases leaving a child care placement (outflow), disaggregated by outflow for inter-country adoption, domestic adoption, foster care, family reintegration % of children outside of family care who are registered for re-integration who have a permanency plan, who have a permanency plan that has been implemented, and where the child is still in placement after 6 months KII with district level officials, PSWOs, CDOs, members of alternative care panels Review of care plans for children selected for reintegration Review of records on assessments of institutions and structured field observations from visits of institutions, KII with staff in institutions. Survey of PSWO, administered through Ministry of Gender, Labour and Social Development. FGDs with community based child protection structures, para social workers Sustainability: What is the current status of child care structures that provide, on an ongoing basis, adequate case￾management and support services for children at risk? Are these sustainable and what makes them sustainable or what are the challenges to their sustainability? Regarding the current capacity and functioning of professionals, child care structures and coordination mechanisms (e.g. probation social work officers, para professionals, alternative care panels), to what extent do these exist, are they capable of provide quality case management and social work procedures, ensuring a regulated process for placements of 24 children outside of family care? Number of family visits per year per parish, disaggregated by CDO and para-social workers Number of cases handled by CDOs and para￾social workers per year per parish, disaggregated by CDO and para-social workers Areas of qualitative inquiry District council: Status of interactions (started or not started) with project coordinators and briefings received on the project; Awareness and interest of ACF; Level of involvement (actions taken) by district authorities in moving the ACF forward (e.g. bylaws passed); Existence of plans for oversight and reporting on CCIs, and for expanding family support and family based care services. District OVC committee: Status of interactions (started or not started) with project coordinators and briefings received on the project; Existence of plans and vision for the number of OVC-children to be reduced in the district and the number of preventative services to be increased (district level planning in line with the ACF). Alternative care panels: Existence of an alternative care panel in the district; Level of activity (status of selection of members, regularity of meetings; Awareness and compliance with core functions; Level of participation of core members; Knowledge and use of foster care and adoption guidelines; Coverage of various types of cases; Regularity Human Rights: What is the current knowledge of duty bearers on their obligations? 25 of reviews of placements and movements of children). PSWOs: Status of interactions (started or not started) with project coordinators and briefings received on the project; Regularity and quality of social inquiry with families at risk, quality and use of standardized processes and forms for case assessments, quality of case files, regularity and quality of after placement follow-up, regularity and quality of inspections and reporting on CCIs and children placed in CCIs to central Ministry, use of standardized referral protocols and clarity on the trajectory of cases; level of in awareness and skills (parenting skills, communicating with children, basic counselling skills; Regularity in interactions and supervisory activities of para￾social workers. Current awareness of PSWOs and CDOs of National Alternative Care Framework and understanding of their role in its implementation. Current awareness of PSWOs and CDOs, remand homes of Retrak’s SOPs for reunification, understanding of what is required to put them into practice. Current level of engagement and collaboration between PSWOs, CDOs and psychosocial workers with community-based partners in view of improving identification of children outside of family care, establishing secure placements in family based care, monitoring children's wellbeing. Child Care institutions: Status of interactions (started or not started) with project coordinators and briefings received on the project; Awareness of rules and processes to be followed for admission of children; Improvements on standards listed in the child 26 care inspection toolkit; attitude and compliance towards decisions taken by Ministry on registration, improvements or closure: Compliance and collaboration with district authorities in the implementation of decisions and reintegration of children; Strategies in place to ensure that, if children are re-integrated from CCIs these places are not filled up by new children; Strategies in place (and dialogue between CCIs and PSWOs) on how to ensure continuity of services provided in CCIs, (e.g. if children are enrolled for educational purposes); appreciation of CCIs of de-I processes. Remand homes: Status of interactions (started or not started) with project coordinators and briefings received on the project; Knowledge, skills and utilization by staff of guidelines and tools (standard operating procedures) which project trained them on; Level of compliance with decisions on children (e.g. if children who have served their sentence are allowed to go home); Frequency and quality of contacts with PSWOs. Community Development Officers (CDOs): Status of interactions (started or not started) with project coordinators and briefings received on the project; Existence of these community workers (measured by % of existing posts that have been filled); Connectedness between community workers and formal system; Frequency and quality of supervision of, and reporting on CCIs. Para-social workers: 27 Status of interactions (started or not started) with project coordinators and briefings received on the project; Awareness of parenting skills (following training received), and ability to put it into practice through interactions with parents; Regularity of interactions and connectedness with CDOs and PSWOs. Community Child Protection Structures: Status of interactions (started or not started) with project coordinators and briefings received on the project, Current status of availability, quality and regularity of training for people performing these functions; Frequency of referrals coming from these structures to PSWOs, Appreciation by these structures of a necessity to take action to solve problems of families and children at risk at their level; Current status of these structure’s involvement in child care issues, e.g. to make people aware of the existence of services (facilitating access for people in need); Current status of their regularity and quality of interactions with PSWOs and CDOs; Current level of awareness and skills to identify at risk children. Current availability of, and level of engagement by volunteers (incl. district inter-religious committees) to identify and refer children living outside of family care. Integration between parts of the system: Appreciation and understanding of all actors (above) on referral lines, linkages/methodologies to work through Community Support Structures, FBOs, and District Inter-Faith Committees to improve tracing of families, and ensure standardized 28 trajectory of cases to place children in family based alternative care. Current availability of mental health services in target district(s). Level 2b – Direct influences of the project in the lives of vulnerable children and children at risk Data collection will be done to define current wellbeing status of one of the two specific target groups, namely of 43,000 children living in vulnerable households considered “at risk” for separation; Data will be collected at the household level to assess their condition of vulnerability in terms of economic status, access to basic needs and other key domains of household welfare; Data will also be collected on the profile and needs of this target group (why they consider themselves to be at risk) to inform the relevance of the integrated package of support services which will be promoted by the project; Data will be collected on the profile and needs of this children in institutions, Impact: What is the current status of vulnerable households with children at risk of separation? What are the welfare domains that show the most severe vulnerabilities? What is the current status of risk and vulnerability of the children living in vulnerable households? What are the needs in terms of support services to prevent child separation? How can their wellbeing be improved? What is the current status of risk and vulnerability of children in institutions, remand homes and on the street? What are their needs in terms of support services to support reintegration of these children? How can their wellbeing be improved? Quantitative baseline indicators: Households: Proportion of households that belong to the categories based on the Family Status Vulnerability Index for CPA 1 (household economic livelihood security) and CP 2 (Access to basic needs): ‘destitute’ (scored over 50 points in the Family Status Vulnerability Index), ‘struggling’ (scored 25 to 49 points) and ‘growing’ (scored at most 24 points or lower). Average score of households at baseline, based on the destitution analysis in the Family Status Vulnerability Index for CPA1 and CPA2 Number and proportion of destitute households located in the targeted areas that are reached by the program intervention and are provided with cash and assets transfers. (to be collected once the intervention has started and compared at end-line to the proportion of households that will have graduated to growing households). Number and proportion of struggling households located in targeted areas that are supported by the project (to be collected once the intervention has started and will be compared at end-line to the proportion of households that will have graduated to growing households. Quantitative data collection tools as defined in the PMP: - Family Status Vulnerability Index (FSVI) - Child Status Index (CSI) - Check lists and reports produced by social workers 29 remand homes and on the street (why/if they consider themselves to be at risk) to inform the relevance of the integrated package of support services which will be promoted by the project. What is the current known number of children living in institutions and on the street in the targeted districts? Average households’ score in key domains of economic livelihood security, access to basic needs, health and care, psychosocial support and basic care, child protection and legal support, of the Family Status Index - vulnerability and risk assessment, with geographical, gender (sex of household head) and age group (age of household head) data disaggregation for subset analysis. Number of children who have been identified for placements in alternative family based care Caregivers: Proportion of caregivers and parents that belong to the critical vulnerability category based on the CSI - family based child care and status Vulnerable children: Number of Older Highly Vulnerable Children (boys and girls) identified using CIS tool in the targeted areas. Proportion of children (boys and girls) identified as Highly Vulnerable according to the CSI score system, amongst all surveyed children Proportion of children showing psychosocial distress issues based on CSI domains of Psychosocial support & Basic care, Protection Number and relative proportion of disabled children amongst all surveyed children Average score of children in targeted households, in terms of child status and wellbeing across key domains of CSI, with Relevance: Do these specific needs match what the project is offering through its integrated model which links family strengthening, child protection and economic strengthening measures? Are there any interventions which seem to be missing? Does the integrated approach model, which links family strengthening, child protection and economic strengthening measures to maximize prevention for children at risk of separation from family and support those returning to family based care, need specific adaptation in the different communities where the project will be operating? What are the most common risk factors to separation, including for boys and girls, and does the project address these in an appropriate manner? 30 Sustainability For the children who will be reunified with their families or placed in alternative family based care, what specific needs do they have that reintegration methods need to take into account to ensure stable and sustained placement of children? average scores calculated for each Core Program Area (CPA) and data disaggregation performed along gender lines, age group and geographic location. Average score of reunified children, in terms of child status and wellbeing across key domains of CSI, with average scores calculated for each Core Program Area (CPA) and data disaggregation performed along gender lines, age group and geographic location (to be collected once the reunification process has started). Areas for qualitative inquiry: Households at risk (caregivers): Perceived needs by the families to be included among project beneficiaries, of their vulnerability, risk and needs for services and support (incl. amounts of cash-, asset transfers to overcome risk and vulnerability, needs for trainings on household's cash management, savings and credit, and financial literacy); appreciation of the Highly Vulnerable Households of importance of parenting skills and needs for parenting education; appreciation of participants in Parents Groups of importance of peer support to reinforce home visits and peer support; Appreciation by caregivers of importance and need for psychosocial support from community support structures. Community-based child protection structures: Perceived needs by community child protection structures for capacity building and current Rights Based Who are the children at most risk of separation from their families and how will the project respond to their needs? Are there any immediate child protection issues encountered (e.g. child labour, violence against children, neglect, early marriage etc.) which the project needs to take into account in the tailoring of their activities? Are there any child rights violations that required urgent attention? 31 understanding of how to do home visit and psychosocial support. Observed current capacity of, and understanding by community child protection structures on risks that can lead to family separation, how to assess and respond to these risks. Current frequency of home-visits and monitoring of households at risk by community structures (including youth and children's groups). Observed quality of content of communication messages on family-based child care and positive child protection practices. Profile (according to community CP structures) of beneficiary families and the match between most urgent needs and the supply of support from the integrated package of interventions. Children in CCIs Appreciation by children in CCIs of their current wellbeing, risk and needs for services and the extent to which they think the mainstream support package developed by the project can address these needs. Indicators of distress in children, not captured during the project monitoring. How children rate the care/services they are receiving (in light of their own circumstances) and whether they are encouraged and given the opportunity to provide participatory feedback; 32 Levels at which children experience or participate in actions violating their rights (child labour, violence against children, early marriage, etc.); Particular vulnerabilities of girl children Profile (according to case files) of children in CCIs (reasons and key contributory causes for why they ended up in CCIs) and the match the most urgent needs and the supply of support from the integrated package of interventions. Appreciation by professionals (PSWOs, CDOs, staff in CCIs or remand homes) who are in contact with child and family beneficiaries of what services are needed to strengthen wellbeing and reduce risk. Older HVCs (incl. substance users, children in Remand Homes and children on the street): Appreciation by beneficiaries (Older HVC) of their needs for vocational training to strengthen their skills and opportunities for entering the local labour market. Appreciation by recovering substance users of needs for support (incl. through peer support and post-treatment groups). Appreciation by children to be reintegrated of their current wellbeing, risk and needs for services and the extent to which they think the mainstream support package developed by the project can address these needs. Indicators of distress in children, not captured during the project monitoring. 33 Profile (according to case files) of child beneficiaries (reasons and key contributory causes for why they ended up on the street or in CCIs) and the match the most urgent needs and the supply of support from the integrated package of interventions. Appreciation by professionals (PSWOs, CDOs, staff in CCIs or remand homes) who are in contact with child and family beneficiaries of what services are needed to strengthen wellbeing and reduce risk. Women and men in the community: Community-wide perceptions of the needs of families that are vulnerable to separation Community-wide knowledge of and attitudes towards child protection Social norms around caregiving (particularly gendered) Contextual information on risks, vulnerabilities and need for support across the community Contextual information on challenges to caregiving, especially for men and women of different age cohorts Nature and prevalence of harmful traditional practices that do not promote child protection Awareness of and attitudes toward sources of psychosocial and practical community-based support structures Interaction between formal and informal child protection structures at community level 34 Midline research matrix Level of data collection in theory of change Explanation Evaluation questions Indicators / qualifiers of improvements Data collection methods / tools Level 1 – Impact on national child care system reform Data will be collected to define status and if any recent changes in the child care reform have influenced the implementation pace of the project in a way that necessitates revisions in the activities or implementation schedule of the project. Impact: In what way is the project contributing to the implementation of national care reform measures? Quantitative indicators: Numbers of residential care services inspected (CCI directory) Numbers of residential care services under transformation / closed (CCI directory) Numbers of residential care services provided with license (CCI directory) Number of districts with a plan for transformation / closure / upgrading of institutions and for expanding preventative family and child support services and foster care (Ministry of Local Government) Areas of qualitative inquiry: Current knowledge and appreciation by key reform stakeholders at national level of the project, the effectiveness of its activities and strategies and any eventual lessons learned shared by the project so far. Awareness within MoGLSD of core gatekeeping functions. Key Informant Interviews (KII) with national level Government stakeholders KII with national level child care reform stakeholders (beyond government structures) Desk review of policy documents and records from Central Government CCI directory, OVC-MIS and services directory Relevance: How does the political environment and context of the project enable/inhibit the project operation and are there any risks that need to be mitigated or opportunities to be leveraged to ensure smooth operation of the project and maximized impact? Effectiveness To what extent did the project manage to disseminating lessons learned from its activities and strategies and to adjust its own strategies to take into account any learning that derived from the implementing process? 35 Current status and regularity in the updating of various MIS (CCI directory, OVC-MIS, service directory of MoGLSD) Status of agreement within MoG on core national indicators (and inclusion into the sectorial performance measurement plan) to monitor progress of child care reform (incl. the gatekeeping function of the Ministry) Status and regularity of inspections of CIIs, licensing CCIs as per Alternative Care Framework by MoGLSD Status of agreement within MoGLSD on how the process and work-plan for how these inspections will be carried out Status of actions to be taken by MoGLSD for closure of CCIs Status of actions to be taken by MoGLSD to address irregular and unethical practices in child care homes Inclusion of child care reform targets in OVC strategy document (2011-2015) Sustainability: What is the current status of child care structures that provide, on an ongoing basis, adequate case￾management and support services for children at risk? Are these sustainable and what makes them sustainable or what are the challenges to their sustainability? Regarding the current capacity and functioning of professionals, child care structures and coordination mechanisms (e.g. probation 36 social work officers, para professionals, alternative care panels), to what extent do these exist, are they capable of providing quality case management and social work procedures, ensuring a regulated process for placements of children outside of family care? Human Rights: What is the current knowledge of duty bearers on their obligations? 37 Level 2b – Direct influences of the project in the lives of vulnerable children and children at risk Baseline data collection will be done for the target group “children to be reintegrated” to define current wellbeing status of the 2,000 children living in institutional care or on the street who will be reunified with birth families or placed in alternative family based care. Impact: What specific needs do the targeted children have and the families where reintegration will happen, including nutrition, education opportunities, care, and protection needs? Vulnerable children: % Older Highly Vulnerable Children (boys and girls) Age 14-25 employed at waged or self￾employed % children showing reduced psychosocial distress based on child status index domains of psychosocial support & Basic care, Protection % of targeted disabled children showing positive emotional health and social behaviour; happy and content and play with peers and participate in family group activities based % of reunified children showing consistent improvement in child status/wellbeing over a period of 6 - 18 months of follow up Number of children who have been identified for reunification or placement in alternative family based care Number of children who have been reunified or placed in alternative family based care % of children who have been reunified or placed in alternative family based care that remain in their placements % of children who have been reunified or placed in alternative family based care who have returned to institutions or to the streets Areas of qualitative inquiry: Older HVCs (incl. substance users): Quantitative project Monitoring data FGDs with children at risk and the families where they have been reintegrated FGDs with community child protection structures Review of case management files and care files/plans of children in CCIs and to be reintegrated from the street. KIIs PSWOs, staff CCIs, CDOs, staff in remand homes. Relevance: Do these specific needs match what the project is offering through its integrated model which links family strengthening, child protection and economic strengthening measures? Are there any interventions which seem to be missing? 38 Sustainability For the children who will be reunified with their families or placed in alternative family based care, what specific needs do they have that reintegration methods need to take into account to ensure stable and sustained placement of children? Appreciation by beneficiaries (Older HVC) of their needs for vocational training to strengthen their skills and opportunities for entering the local labour market. Appreciation by recovering substance users of needs for support (incl. through peer support and post-treatment groups). Appreciation by children to be reintegrated of their current wellbeing, risk and needs for services and the extent to which they think the mainstream support package developed by the project can address these needs. Indicators of distress in children, not captured during the project monitoring. Profile (according to case files) of child beneficiaries (reasons and key contributory causes for why they ended up on the street or in CCIs) and the match the most urgent needs and the supply of support from the integrated package of interventions. Appreciation by professionals (PSWOs, CDOs, staff in CCIs or remand homes) who are in contact with child and family beneficiaries of what services are needed to strengthen wellbeing and reduce risk. Women and men in the community Community-wide perceptions of the needs of families that are vulnerable to separation and families with reintegrated children Community-wide knowledge of and attitudes towards child protection Rights Based Of the children identified to be at most risk of separation from their families, how is the project able to target them with their activities? Are there any immediate child protection issues (e.g. child labour, violence against children, neglect, early marriage etc.) which the project needs to take into account in the tailoring of their activities? Are there any child rights violations that required urgent attention? 39 Social norms around caregiving (particularly gendered) Contextual information on risks, vulnerabilities and need for support across the community Contextual information on challenges to caregiving, especially for men and women of different age cohorts Nature and prevalence of harmful traditional practices that do not promote child protection Awareness of and attitudes toward sources of psychosocial and practical community-based support structures Interaction between formal and informal child protection structures at community level Level 3a – Effectiveness and efficiency of project inputs / activities towards building capacity of the child care system Data will be collected to establish if the inputs and activities of the project are happening effectively and efficiently and if the project overall is likely to achieve its objectives (namely to build the capacity of formal and informal actors / structures to effectively implement the new ACF and of CCIs to change their Effectiveness: What type of engagement has the project managed to establish with various structures and professionals involved in the project, including district authorities, child care institutions, probation social work officers, alternative care panels, remand homes, community child protection Quantitative indicators: Number of district Alternative Care Panels, trained / supported to start activities, alternative care directories and child care directories to be maintained/updated. Numbers of institutions trained. Number of PSWOs and CDOs trained on how to implement National Alternative Care Framework Project Monitoring data (quantitative and qualitative) KII with district level officials, PSWOs, CDOs, members of alternative care panels Review of care plans for children selected for reintegration 40 role), within the planned timeframe. structures, para-social workers etc.? Number of remand homes trained on Retrak’s SOPs for reunification. Number of PSWOs, CDOs and psychosocial workers trained, engaged to collaborate with community-based partners in view of improving identification of children outside of family care, establishing secure placements in family based care, monitoring children's wellbeing. Number of volunteers (incl. district inter￾religious committees) trained to identify and refer children living outside of family care. Areas of qualitative inquiry: Current status and quality of alternative care directories and child care directories. Current capacity (measured by availability of dedicated and trained staff, type of active family tracing and reintegration activities, contacts with PSWOs(CDOs) of targeted CCI’s to do family tracing and reintegration of separated children Current awareness of PSWOs and CDOs of National Alternative Care Framework and understanding of their role in its implementation. Current awareness of PSWOs and CDOs, remand homes of Retrak’s SOPs for reunification, understanding of what is required to put them into practice. Current level of engagement and collaboration between PSWOs, CDOs and psychosocial workers with community-based partners in view of improving identification of children outside of Review of records on assessments of institutions and structured field observations from visits of institutions, KII with staff in institutions. KIIs with staff in remand homes. Survey of PSWO, administered through Ministry of Gender, Labour and Social Development. FGDs with community based child protection structures, para social workers Efficiency: What strategies and activities are supported by the project to influence the capacity of these structures / professionals? Have the planned investments been made in putting in place the necessary human resources, coordination mechanisms, time and financial resources to efficiently implement the project activities? 41 family care, establishing secure placements in family based care, monitoring children's wellbeing. Current availability of, and level of engagement by volunteers (incl. district inter-religious committees) to identify and refer children living outside of family care. Current availability of mental health services in target district(s). Level 3b - Effectiveness and efficiency of project inputs / activities towards changing the situation of vulnerable households/parents and children at risk Data will be collected to establish if the inputs and activities of the project are happening effectively and efficiently and if the project overall is likely to achieve its objectives (strengthening parenting skills, mobilizing communities, community leaders, faith-based organizations to support family based care, strengthen household economy and activities for reintegration of girls on the street.) Effectiveness What type of engagement has the project managed to establish with the 43,000 households and the 2000 children to be reintegrated? Is the project seeking feedback from the beneficiaries to ensure the target reach is involved and informed of the intervention and that services are informed about their needs? Quantitative indicators: Number of street girls reunified with families. Number of destitute households receiving cash, asset transfers and/or other sources of support. Number of struggling households trained in cash management and income generation skills Number of youth (Older HVC) receiving vocational training. Number of Vulnerable Households (including adoptive and reintegrated households) receiving parenting education as per expanded Essential Package. Number of households participating in group parent support sessions. Number of community support structures introduced to CSS Model to screen households in psychosocial distress. Number of recovering substance users included in peer support post-treatment groups. Quantitative and qualitative project monitoring data Household Case Studies in homes with Reintegrated Children FGDs with women, men and community child protection structures in target communities KIIs district officials. 42 Number of caregivers and community members sensitized on substance abuse using peer counselling, follow up, checking on relapsed members, coping and managing. Number of health facilities in the target district(s) mobilized and trained to provide PSS for patients referred by community support structures. Number of community support structures provided with capacity building plans. Number of community child protection structures trained on assessment and response procedures for risks that can lead to family separation. Number of community members (including youth and children's groups) trained on how to do home-visits and monitoring of households at risk. Number of dissemination events organized by community structures including youth and children's groups on family-based child care and positive child protection practices. Areas for qualitative inquiry: Households at risk (caregivers): Appreciation of the Highly Vulnerable Households of importance of parenting skills and needs for parenting education. Appreciation of participants in Parents Groups of importance of peer support to reinforce home visits and peer support. 43 Appreciation by caregivers of importance and need for psychosocial support from community support structures. Older HVCs (incl. substance users): Appreciation by beneficiaries (Older HVC) of their needs for vocational training to strengthen their skills and opportunities for entering the local labour market. Appreciation by recovering substance users of needs for support (incl. through peer support and post-treatment groups). Community-based child protection systems: Appreciation by community child protection structures of capacity building activities and current understanding of how to do home visit and psychosocial support. Observed capacity of, and understanding by community child protection structures on risks that can lead to family separation, how to assess and respond to these risks. Frequency of home-visits and monitoring of households at risk by community structures (including youth and children's groups). Observed quality of content of communication messages on family-based child care and positive child protection practices. End-line evaluation design matrix 44 Level of data collection in theory of change Explanation Evaluation questions Indicators / qualifiers of improvements Data collection methods / tools Level 1 – Impact on national child care system reform Data will be collected to define if the child care system has been reformed (is reforming) in a way that ensures that unnecessary separation of children from their families is reduced, namely if services are expanding, gatekeeping is improved and if there is reduced reliance on institutional care and expansion of family based options. Impact: To what extent did the project contribute to implementation of national care reform measures, such as for example: - Capacity of Ministry of Gender, Labour and Social Development to provide an oversight gatekeeping role (data collection and monitoring of the reform, inspection and enforcement of the law and policy) - Visibility of the need for a reform of the child care system (and inclusion in sector reform measures) - Promotion of system for national adoption and foster care - Promotion of preventive and family support services - Promotion of system for family and community based alternative care (e.g. foster care, kinship care, supported independent living) and national adoption Quantitative indicators: Numbers of residential care services inspected (CCI directory) Numbers of residential care services under transformation / closed (CCI directory) Numbers of residential care services provided with license (CCI directory) Number of districts with a plan for transformation / closure / upgrading of institutions and for expanding preventative family and child support services and foster care (Ministry of Local Government) Qualifiers of functioning (which project has ability to influence): Agreement in MoG on core national indicators (and inclusion into the sectoral performance measurement plan) to monitor progress of child care reform (incl. the gatekeeping function of the Ministry) Continuous updating of data of the CCI directory by MoG Continuous updating of data on service directory by MoG Regular inspections of CIIs, licensing CCIs as per Alternative Care Framework by MoG Key Informant Interviews (KII) with national level Government stakeholders KII with national level child care reform stakeholders (beyond government structures) Desk review of policy documents and records from Central Government CCI directory, OVC-MIS and services directory 45 Relevance: How did the political environment and context of the project influence the project operation and the impact, and to what extent was the project able to mitigate risks and leverage new circumstances /opportunities which arose? Agreement within MoG on how the process and work-plan for how these inspections will be carried out Actions taken by MoG for closure of CCIs Actions taken by MoG to address irregular and unethical practices in child care homes Integration of child care reform targets into the next OVC strategy document (post-2011-2015 OVC strategy) Level 2a – Direct influence of the outcomes of the project at policy and system levels Data will be collected to define if the project has influenced changes in the capacity of District Governments and sub￾districts structures to monitor institutions and effectively implement the country’s new Alternative Care Framework. Data will also be collected to define changes in existing child care institutions, if family based care is gaining importance and if social work best practices is being respected. Data will also be collected to establish if the project is contributing to strengthen informal child protection mechanisms and their Impact: To what extent did the project contribute to implementation of national care reform measures, for example: - Functioning and capacity of probation social work officers - Functioning and quality of alternative care panels (key component of impl. of Alternative Care Framework) To what extent, and which systemic changes did the project bring about at community, regional, and national levels that are enabling children to live in family care and preventing inappropriate placements in institutional care? Quantitative indicators: % of child care institutions in the district that have been inspected, licenced, have an improvement plan, are acting on recommendations from the assessment % of children placed in a child care institution who have a care plan and who have a care plan that has been reviewed in the last 3 months % of targeted CCI and remand homes with reduced number of children living in care centres. Number of districts with alternative care panels established Number of cases discussed by alternative care panels monthly % of children in alternative care in the district (targeted by the project) who are in: child care institutions, foster care, biological family, Project Monitoring data using quantitative indicators for system change KII with district level officials, PSWOs, CDOs, members of alternative care panels Review of care plans for children selected for reintegration Review of records on assessments of institutions and structured field observations from visits of institutions, KII with staff in institutions. 46 interaction with the formal child protection system.  District authorities and OVC committees  Alternative care panels  Probation Social Work Officers  Community Development Officers  Remand homes  Child Care Institutions  Para-social workers  Community child protection structures kinship family, adopted domestically, adopted internationally Number of children recorded to be on the street in the district, disaggregated by boys and girls Proportion of new cases (inflow) of child care placements in the district by number of cases leaving a child care placement (outflow), disaggregated by outflow for inter-country adoption, domestic adoption, foster care, family reintegration % of children outside of family care who are registered for re-integration who have permanency plan, who have a permanency plan that has been implemented, and where the child is still in placement after 6 months. % of the reintegrated children who are still in placement after six months who attend school, have access to sufficient nutrition (disaggregated by age, sex and disability). Number of family visits per year per parish, disaggregated by CDO and para-social workers Number of cases handled by CDOs and para￾social workers per year per parish, disaggregated by CDO and para-social workers Qualifiers of functioning (which project has ability to influence): District council: Appreciation of interactions with project coordinators and briefings received through the project; Awareness and interest of ACF; Level of Quantitative survey of PSWO, administered through Ministry of Gender, Labour and Social Development. FGDs with community based child protection structures, para social workers Sustainability: By project end, to what extent have structures been established or strengthened that can continue to provide, on an ongoing basis, adequate case￾management and support services for children at risk? Are these sustainable and what makes them sustainable? For the different changes observed in the capacity and functioning of professionals, child care structures and coordination mechanisms (e.g. probation social work officers, para professionals, alternative care panels), to what extent are they likely to sustain the work, to continue meeting and to continue to apply 47 what they have learned after the end of the project? involvement (actions taken) by district authorities in moving the ACF forward (e.g. bylaws passed); Existence of plans for oversight and reporting on CCIs, and for expanding family support and family based care services. District OVC committee: Appreciation of interactions with project coordinators and briefings received through the project; Existence of plans and vision for the number of OVC-children to be reduced in the district and the number of preventative services to be increased (district level planning in line with the ACF). Alternative care panels: Existence of an alternative care panel in the district; Appreciation of training received through the project; Quality of the members; Regularity of meetings; Compliance with core functions; Level of participation of core members; Knowledge and use of foster care and adoption guidelines; Coverage of various types of cases; Regularity of reviews of placements and movements of children PSWOs: Appreciation of training received through the project; Regularity and quality of social inquiry with families at risk, quality and use of standardized processes and forms for case assessments, quality of case files, regularity and quality of after placement follow-up, regularity and quality of inspections and reporting on CCIs and children placed in CCIs to central Ministry, use of standardized referral protocols and clarity on the trajectory of cases; level of in awareness Human Rights: To what extent did the project contribute to strengthen the knowledge of obligations of the duty bearers and the capacity of those who are the duty bearers? Lessons learned: What lessons can be learned from the project’s work with child care institutions? What lessons can be drawn from the work with different professionals involved in gatekeeping and outreach to vulnerable families (e.g. para professionals how are they working together with probation social work officers and what is motivating them in their work)? 48 and skills (parenting skills, communicating with children, basic counselling skills; Regularity in interactions and supervisory activities of para￾social workers Child Care institutions: Appreciation of contacts with project staff and trainings received through the project; Awareness of rules and processes to be followed for admission of children; Improvements on standards listed in the child care inspection toolkit; attitude and compliance towards decisions taken by Ministry on registration, improvements or closure: Compliance and collaboration with district authorities in the implementation of decisions and reintegration of children; Strategies in place to ensure that, if children are re-integrated from CCIs these places are not filled up by new children; Strategies in place (and dialogue between CCIs and PSWOs) on how to ensure continuity of services provided in CCIs, (e.g. if children are enrolled for educational purposes); appreciation of CCIs of de-I processes. Remand homes: Appreciation of training received through the project; Knowledge, skills and utilization by staff of guidelines and tools (standard operating procedures) which project trained them on; Level of compliance with decisions on children (e.g. if children who have served their sentence are allowed to go home); Frequency and quality of contacts with PSWOs. Community Development Officers (CDOs): 49 Appreciation of training received through the project; Existence of these community workers (measured by % of existing posts that have been filled); Connectedness between community workers and formal system; Frequency and quality of supervision of, and reporting on CCIs. Para -social workers: Appreciation of training received through the project; Awareness of parenting skills (following training received), and ability to put it into practice through interactions with parents; Regularity of interactions and connectedness with CDOs and PSWOs. Community Child Protection Mechanisms: Availability, quality and regularity of training for people performing these functions; Appreciation of training received through the project; Changes in the frequency of referrals coming from these structures to PSWOs, actions taken by these structures to solve the problems at their levels; Frequency and activities to make people aware of the existence of services (facilitating access for people in need); Regularity and quality in interactions with PSWOs and CDOs; Awareness and skills to identify at risk children Integration between parts of the system: Appreciation and understanding of all actors (above) on referral lines, linkages/methodologies to work through Community Support Structures, FBOs, and District Inter -Faith Committees to improve tracing of families, and ensure standardized trajectory of cases to place children in family based alternative care. Impact: Quantitative indicators: 50 Level 2b – Direct influences of the project in the lives of vulnerable children and children at risk Data collection will be done on influences of the project on improvements in wellbeing in the lives of two specific target groups, namely of 43,000 children living in vulnerable households considered “at risk” for separation; and 2,000 children living in institutional care or on the street that have been reunified with birth families or placed in alternative family based care. These are to show improved wellbeing outcomes 12 months following reintegration. Based on the observed changes, findings will be drawn on the impact, and relevance and sustainability of the integrated package of support tested and promoted by the project. To what extent did this project identify/test successful interventions that a) prevent child separation and b) support reintegration of children at risk or in residential care? Has their well-being improved? For which children was impact most visible? To what extent and how did the programme contribute to the reduction in the number of children living in institutions and on the street in the targeted districts? (Disaggregation by children reintegrated from institutions and from the street) To what extent have prevention methods employed by the project increased nutrition, education opportunities, care, and protection for children in targeted households? Households: Average % of destitute households provided with cash and assets transfers that have graduated to growing households and have remained in that cohort for a period of at-least 12- 24 months over the project duration % Households that score at-most 49 point and lower in Family Status Vulnerability Index for CPA 1 ( household economic livelihood security) and CP 2 (Access to basic needs) in Household OVC Vulnerability assessment % of struggling households supported by the project with overall score of 0-24 in Family Status Vulnerability Index for CPA 1 ( household economic livelihood security) and CP 2 (Access to basic needs) in Household OVC Vulnerability assessment (graduated to growing households) Average % of households that score at-most 34 points in multi-key domains on economic livelihood security, access to basic needs, health and care, psychosocial support and basic care, child protection and legal support, of the Family Status Index - vulnerability and risk assessment Caregivers: % of individuals participating in post treatment groups including identified caregivers in psychosocial distress showing reduction in substance abuse % of supported caregivers and parents that have moved out of critical vulnerability (Score 90-116) and show reduced vulnerability score (Score Quantitative project Monitoring data Household Case Studies of vulnerable households FGDs with women, men and community child protection structures in target communities Household Case Studies in homes with “Reintegrated Children” Relevance: To what extent did the project’s integrated model which links family strengthening, child protection and economic 51 strengthening measures offer models and approaches for expansion, adaptation, and/or replication? What combination of interventions within this model could be adapted, expanded? To what extent was the integrated approach model, which links family strengthening, child protection and economic strengthening measures to maximize prevention for children at risk of separation from family and support those returning to family based care, relevant and sufficiently adapted to the communities that the project was operating in will be working in? Based on what is found to be the most common risk factors to separation, to what extent did the project manage to address these in an appropriate manner including for both boys and girls? below 89) across key domains of CSI - family based child care and status Vulnerable children: % Older Highly Vulnerable Children (boys and girls) Age 14-25 employed at waged or self￾employed % children showing reduced psychosocial distress based on child status index domains of psychosocial support & Basic care, Protection % of targeted disabled children showing positive emotional health and social behaviour; happy and content and play with peers; participate in family group activities; and attend school. % of reunified children showing consistent improvement in child status/wellbeing over a period of 6 - 18 months of follow up Number of children who have been reunified or placed in alternative family based care % of children who have been reunified or placed in alternative family based care that remain in their placements % of children who have been reunified or placed in alternative family based care who have returned to institutions or to the streets Areas for qualitative inquiry: Level of appreciation of the families receiving services and the extent to which they think these services have reduced their vulnerability. Sustainability 52 For the children who have been reunified with their families or placed in alternative family based care, what reintegration methods have supported stable and sustained placement of children and to what extent? Are children still in the homes, and in school 6 months (and more) after placement? What makes these reintegration methods sustainable? If there has been relapse in reintegration why has this been the case? Level of appreciation of reintegrated children of their current placement, quality of it, sufficiency of services provided (types and intensity) and the current wellbeing of children. Profile of children and families and the match the most urgent needs and the supply of support from the integrated package of interventions. Appreciation by beneficiaries of the combined effect of the package and if anything was missing that should have been included in the support package. Appreciation by the beneficiaries of what interventions had the most effect for them. Feeling/appreciation of whether the current status (wellbeing and non-risk) is sustainable, and the extent to which it has supported them to put in place mitigating procedures for falling back into vulnerability/risk. Of those most at risk (children with disabilities, on the street), feeling of appropriateness of the support provided, and if there was anything they might have needed other than the mainstream support package developed by the project. Indicators of distress in children, not captured during the project monitoring. Women and men in the community Community-wide perceptions of the needs of families that are vulnerable to separation, and how they have been met by the programme Rights Based To what extent did the project manage to target those most at risk of separation from their families? To what extent was the project able to prevent child protection issues (e.g. child labour, violence against children, neglect, early marriage etc.) and mitigate risks for creating greater vulnerability (e.g. child labour and any kind of harm)? How did the project address urgent child rights violations that required urgent attention? 53 Community-wide knowledge of and attitudes towards child protection, and changes in child protection since the programme began Any changes in social norms around caregiving (particularly gendered) Contextual information on risks, vulnerabilities and need for support across the community Contextual information on challenges to caregiving, especially for men and women of different age cohorts The nature and prevalence of harmful traditional practices that do not promote child protection Awareness of and attitudes toward sources of psychosocial and practical community-based support structures Interaction between formal and informal child protection structures at community level Lessons Learned: What lessons can be learned from the project’s integrated package of support? Level 3a – Effectiveness and efficiency of project inputs / activities towards building capacity of the child care system Data collection on the level of implementation of activities as per work plans, and observations on the perceived sufficiency of the planned activities to achieve the outcomes, e.g in terms of building the capacity of formal and informal actors / structures to effectively implement the new ACF and of CCIs to change their role. Effectiveness: How effective was the engagement of the project with various structures and professionals involved in the project, including district authorities, child care institutions, probation social work officers, community development officers, alternative care panels, remand homes, community child protection structures, para-social workers etc.? Quantitative indicators: Number of district Alternative Care Panels, trained / supported to start activities, alternative care directories and child care directories maintained/updated. Numbers of institutions trained and with capacity in family tracing and reintegration of separated children Number of PSWOs and CDOs trained and with understanding of how to implement National Alternative Care Framework Project Monitoring data (quantitative and qualitative) KII with district level officials, PSWOs, CDOs, members of alternative care panels Review of care plans for children selected for reintegration 54 Efficiency: What strategies that were used by the project proved to be the most efficient in influencing the system reform? Were there any strategies / activities which were less efficient? To what extent were the human resources, coordination mechanisms, time and financial resources invested in the implementation of the project efficiently used and were they sufficient to achieve the targets of the project? Number of remand homes trained and using Retrak’s SOPs for reunification. Number of PSWOs, CDOs and psychosocial workers trained, engaging and collaborating with community-based partners in view of improving identification of children outside of family care, establishing secure placements in family based care, monitoring children's wellbeing. Number of volunteers (incl. district inter￾religious committees) trained to identify and refer children living outside of family care. Qualifiers: Quality of alternative care directories and child care directories. Improvements in targeted CCI’s capacity to do family tracing and reintegration of separated children Improvements in targeted PSWOs and CDOs understanding of how to implement National Alternative Care Framework Uptake in the use of Retrak’s SOPs for reunification. Level of engagement and collaboration between PSWOs, CDOs and psychosocial workers with community-based partners to view of improving identification of children outside of family care, establishing secure placements in family based care, monitoring children's wellbeing. Review of records on assessments of institutions and structured field observations from visits of institutions, KII with staff in institutions. Quantitative survey of PSWO, administered through Ministry of Gender, Labour and Social Development. FGDs with community based child protection structures, para social workers 55 Quality and intensity in engagements by volunteers (incl. district inter-religious committees) to identify and refer children living outside of family care. Level 3b - Effectiveness and efficiency of project inputs / activities towards changing the situation of vulnerable households/parents and children at risk Data collection on the completion of implementation of activities as per work plans to strengthen parenting skills, to mobilize communities, community leaders, faith￾based organizations to support family based care, activities to strengthen household economy and activities for reintegration of girls on the street. Observations on the sufficiency, appreciation and appropriateness of these activities to achieve improvements in the situation of vulnerability and risk of the beneficiaries. Effectiveness To what extent has the target of the project to support the 43,000 households and the 2000 children to be reintegrated been reached and how were these changes achieved? To what extent did the beneficiaries of the project, including children, families and communities, appreciate the services provided? Did the project incorporate sufficient participatory feedback processes to ensure the target reach was involved and informed of the intervention and received the services most appropriate for their needs? Which were the most significant changes in the lives of the two target groups which were achieved by the project, and what factors seem to have influenced these changes? Quantitative indicators: Number of street children placed in alternative family based care or in independent living arrangements (disaggregated by sex). Number of destitute households that received cash-, asset transfers and/or other sources of support. Number of struggling households trained in cash management and income generation skills Number of youth (Older HVC) who received vocational training and graduated from programme. Number of Vulnerable Households (including adoptive and reintegrated households) that received parenting education as per expanded Essential Package. Number of households that have participated in group parent support sessions. Number of community support structures using CSS Model to screen households in psychosocial distress. Number of recovering substance users who participate in peer support post-treatment groups. Number of caregivers and community members sensitized on substance abuse using peer Quantitative and qualitative project monitoring data FGDs with community child protection structures, families at risk and children at risk Household Case Studies of vulnerable households FGDs with women, men and community child protection structures in target communities Household Case Studies in homes with “Reintegrated Children” KIIs district officials. Efficiency 56 To what extent were the project strategies used to reduce vulnerability of households and to reintegrate children efficiently implemented? Were there any implementation gaps? What strategies / activities which were used by the project proved to be the most efficient in influencing the lives of children and vulnerable families? Were there any strategies / activities which were not at all efficient? To what extent were the human resources, coordination mechanisms, time and financial resources invested in the implementation of the project efficiently used and were they sufficient to achieve the targets of the project? counselling, follow up, checking on relapsed members, coping and managing. Number of health facilities in the target district(s) providing PSS for patients referred by CSS. Number of community support structures with capacity building plans. Number of community child protection structures trained to assess and respond to risks that can lead to family separation. Number of community members (including youth and children's groups) trained to do home-visits and monitoring of households at risk. Number of dissemination events organized by community structures including youth and children's groups on family-based child care and positive child protection practices. Areas for qualitative inquiry: Households at risk (caregivers): Sufficiency of cash-, asset transfers and/or other sources of support to overcome risk and vulnerability. Appreciation by beneficiaries of the trainings of whether these changed their knowledge and behaviours in terms of household's cash management, savings and credit, and financial literacy. Appreciation of the Highly Vulnerable Households that received parenting education 57 as per expanded Essential Package of whether this programme improved their parenting skills, behaviours and practices. Appreciation of participants in Parents Groups of whether peer support in group sessions has reinforced home visits and peer support. Appreciation by caregivers of psychosocial support received from community support structures. Perceived and observed changes in level of distress among caregivers in psychosocial distress who have received peer counselling, follow up by community members. Older HVCs (incl. substance users): Appreciation by beneficiaries (Older HVC) of vocational training of whether these programmes improved the relevance of their skills for the local labor market and changed their livelihood options. Appreciation by, and observed changes in behaviours of recovering substance users who have participated in peer support post￾treatment groups. Perceived and observed improvements in the prioritization and availability of mental health services in target districts. Community-based child protection systems: Quality of capacity building plans for community support structures. 58 Improvements in understanding of project home visit and psychosocial support guidance as evidenced in training post-test Observed and perceived changes / improvements in in the capacity of community child protection structures to assess and respond to risks that can lead to family separation. Observed and perceived changes / improvements in the frequency and quality of community structures (including youth and children's groups) home-visits and monitoring of households at risk. Observed and perceived appreciation and efficiency of dissemination events and messages on family-based child care and positive child protection practices. 59 XII. Annex 2 – Evaluation timeline EVALUATION: PLANNING PHASE Activities Deliverables Timing 1. Desk-review of project documents and documents on child care system and the reform in Uganda 2. Consultations with USAID and the Implementing Partner to detail the theory of change for the project, components of the evaluation approach, agree on evaluation questions and expectations on performance monitoring plan 3. Preparation of a detailed evaluation design, including the theory of change for the project 4. Review of methodology by OPM-ethical review committee 5. Drafting of performance monitoring plan 6. Feedback from project partners 7. Finalization of Research design and PMP 8. Drafting of data collection tools 9. Feedback by project partner 10. Finalization of data collection tools Deliverable 1: Research Design Deliverable 2: Performance Monitoring Plan (PMP) Deliverable 3: Data Collection Tools By 27 March 2015 draft evaluation framework to JSI/USAID and ERB for approval By 1 April evaluation tools to JSI/USAID and ERB for approval 22 April 2015 receipt of comments from JSI/USAID and ERB 6 May 2015 finalization of evaluation design and submission to JSI/USAID EVALUATION: BASE-LINE DATA COLLECTION 11. Selection and training of local researchers 12. Finalization of interview schedule Data collection for policy level analysis 13. Qualitative data collection on different samples 14. Debriefing with project implementing partners and donor 15. Drafting of Baseline Report 16. Presentation of Baseline Results to project partner and donor Deliverable 4: Training of local researchers Deliverable 5: Baseline Data Collection Deliverable 6: Baseline Report Deliverable 7: Presentation of Baseline Results Training of local researchers 24-28 August 2015 Baseline field work: 24 August – 11 September 2015 Submission of draft Baseline report to ChildFund JSI/USAID, 9 October 2015 Presentation of findings from baseline (to take place any time after 9 October) 23 October – deadline for feedback for ChildFund, JSI/USAID Submission of final version of baseline report 30 October 2015 EVALUATION: MID-TERM DATA COLLECTION 11. Updating and revision of data collection tools 12. Finalization of interview schedule 13. Refresher training / workshop with local partner 14. Data collection on performance of the project Deliverable 8: Mid-term Review Data Collection Deliverable 9: Mid-term Report Deliverable 10: Presentation of Mid-term Results 1 September – 30 October, 2016 60 EVALUATION: PLANNING PHASE 15. Qualitative data on children to be reintegrated(for inclusion in baseline report) 16. Debriefing with project implementing partners and donor 17. Drafting of mid-term report 18. Presentation of mid-term findings and results to project partner and donor EVALUATION: END-LINE DATA COLLECTION 19. Updating and revision of data collection tools 20. Finalization of interview schedule 21. As needed, recruitment and training of new members of local research teams, refresher training/workshop for old local research teams. 22. Data collection on systemic components of the evaluation 23. Qualitative data collection on different samples 24. Debriefing with project implementing partners and donor 25. Drafting of End-line Report 26. Presentation of End-line Results to project partner and donor Deliverable 11: End-line Data Collection Deliverable 12: Final Evaluation Report Deliverable 13: Presentation of Final Results Between 1 April – 15 August 2017 (to be synchronized with end of project activities). EVALUATION: TECHNICAL SUPPORT (ONGOING) 27. On-going technical support Deliverable 14: On-going Technical Support 26 days in total (11 days by team leader and 15 days by quantitative data collection expert to be implemented throughout the duration of the project) xlix ANNEX VIII: DATA COLLECTION INSTRUMENTS (see below) Endline evaluation Data collection tools for the Performance Evaluation of DOVCU-project Evaluation dates: 1 January, 2015 – 31 January, 2018 Maestral International, L.L.C. Partnering with: Oxford Policy Management and Makerere University Name of contracted organization: Maestral International, L.L.C. Organization address: 150 South 5th Street, Suite 3250 Minneapolis, MN 55402 USA Organization phone: + 612.354.3042 Organization fax: + 612.605.0156 Primary contact name: Ms. Anna Nordenmark Severinsson Primary contact phone: +41-79-452.07.56 Primary contact email: anordenmark@maestral.org Secondary contact name: Ms. Nicole Williams Secondary contact phone: +612.354.3042 Secondary contact email: nwilliams@maestral.org Table of Contents I. INTRODUCTION TO THE DATA COLLECTION TOOLS.............................................................1 II. OVERVIEW OF DATA COLLECTION PLAN – NATIONAL, DISTRICT AND SUB-DISTRICT LEVELS 1 III. INTERVIEWS TO BE CONDUCTED AT NATIONAL LEVEL .......................................................2 Tool 1 –Questionnaire with PSWOs (self-administered) ............................................................................... 2 Tool 2 – Guide for Discussion with DOVCU project partners ....................................................................... 8 Tool 3 – Guide for Key Informant Interviews with key child protection stakeholders not involved in the DOVCU project: ..........................................................................................................................................................17 Tool 4 – Guide for Key Informant Interviews at Ministry of Gender Labour and Social Development: ......26 Tool 5 – Guide for Key Informant Interview with Ministry of Local Government: ........................................36 IV. INTERVIEWS TO BE CONDUCTED AT DISTRICT LEVEL:......................................................38 Tool 6 – Checklist for Chief Administrative Officer District (Courtesy meeting) / Chairperson from District Council .........................................................................................................................................................38 Tool 7 – Guide for Key Informant Interviews with District OVC committee:................................................43 Tool 8 - Guide for Key Informant Interviews with District Community Development Officer ......................46 Tool 9 – Guide for Key Informant Interviews with PSWOs: ........................................................................51 Tool 10 – Guide for Focus Group Discussions with Community Development Officers (CDOs) (1 FGD / district) ..........................................................................................................................................................62 Tool 11 – Guide for Focus Group Discussions with members of Teams Around the Child / Alternative Care Panels (1 FGD / district) ...............................................................................................................................69 V. INTERVIEWS TO BE CONDUCTED AT SUB-DISTRICT LEVEL..............................................77 Tool 12 – Guide for Key Informant Interviews with staff in Child Care institutions (1 KII / institution x 2 institutions / district): .....................................................................................................................................77 Tool 13 – Guide for Key Informant Interviews with staff in Remand homes (1 KII / institution in 2 remand homes): ........................................................................................................................................................83 Tool 14 – Guide for Key Informant Interviews with CBO working with children on the street (1 KII / institution in Mbale district) ...............................................................................................................................................87 VI. INTERVIEWS TO BE CONDUCTED AT COMMUNITY LEVEL.................................................92 Tool 15 – Guide for Focus Group Discussions with Para-social workers (1 FGD / parish) ........................92 Tool 16 – Guide for Focus Group Discussions with Community Child Protection Mechanisms (1 FGD / parish) ......................................................................................................................................................................95 Tool 17 – Guide for conducting Case Studies with Households (caregivers): (8 household case studies / parish) ..........................................................................................................................................................99 Tool 18 – Tool for activity for reintegrated children ...................................................................................102 Tool 19 – Guide for Focus Group Discussions with women and men in the community (1 FGD / parish per gender) .......................................................................................................................................................105 VII. REFERRAL TOOL FOR FIELD RESEARCHERS.....................................................................110 1 I. Introduction to the data collection tools The below tools are to be used as indicative guides for key informant interviews and focus group discussions. The interviewer should be free to request for clarifications, add additional questions and elaborate on the questions as the interview / discussion proceeds. Any important modifications to the interview guides made during the field work will be discussed in the evaluation team to ensure that adjustments are synchronized between different interviewers. The tools will be adapted prior to the mid-line and end-line phases of the data collection. II. Overview of data collection plan – national, district and sub-district levels1 NATIONAL LEVEL DATA COLLECTION Questionnaire with PSWO from 6 districts KII with key stakeholders supporting child care reform initiatives / activities (not working in Government) - DOVCU project partners - UNICEF - Members of National Child Protection Working Group - Experts supporting child care reform efforts in Uganda (non-DOVCU project experts) KII with key national government child care system stakeholders - Assistant commissioner/Commissioner Youth and Children, Ministry of Gender, Labour and Social Development - Representative from Alternative Care Unit, Ministry of Gender, Labour and Social Development - Representative from unit responsible for inspections of CCIs, Ministry of Gender, Labour and Social Development Representative of OVC national implementation unit, Ministry of Gender, Labour and Social Development - Person in charge of OVC MIS, Ministry of Gender, Labour and Social Development - Person in charge of CCI directory, Ministry of Gender, Labour and Social Development - Representative of Ministry for Local Government - Representative of National Alternative Care Panel DISTRICT LEVEL DATA COLLECTION (X 6) Interviews to be conducted at district head-quarter level Interviews to be conducted at Parish level / City level Interviews to be conducted at Community and Household level (Parish 1) Interviews to be conducted at Community and Household level (Parish 2) DOVCU Project Officer & Social Worker (Briefing meeting) Chief Administrative Officer District (Courtesy meeting) / Chairperson District council (1 KII/district) Staff in Child Care institutions (1 KII / institution x 2 institutions / district) Para-social workers (1 FGD / parish) Para-social workers (1 FGD / parish) Chairperson District OVC committee (1KII / district) Children in institutions (2 FGDs / institution x 2 institutions / district) Community Child Protection Mechanisms (1 FGD / parish) Community Child Protection Mechanisms (1 FGD / parish) District Community Development Officer 1 KII / District Staff Remand homes (1 KII / institution in 2 remand homes) Households at risk (caregivers): (4 household case studies / parish) Households at risk (caregivers): (4 household case studies / parish) 1 Yellow coloured fields include primary data to be collected as part of policy based analysis. Blue coloured fields include primary data to be collected through qualitative component of the methodology. 2 PSWOs (1 KII / district) Children in remand homes (2 FGDs / remand home x 2 remand homes) Women in the community (1 FGD / parish) Women in the community (1 FGD / parish) Community Development Officers (CDOs) (1 FGD / district) Children on the street (2 FGD with children on the street) Men in the community (1 FGD / parish) Men in the community (1 FGD / parish) Members of Alternative Care Panels (1 FGD / district) Staff CBO working with children on the street (1 KII in Mbale district) 2 III. Interviews to be conducted at national level Tool 1 –Questionnaire with PSWOs (self-administered) The current questionnaire is carried out to inform the end-line evaluation of the USAID/DCOF and JSI supported project: Deinstitutionalization of Vulnerable Children in Uganda (DOVCU), implemented by ChildFund and partners between 2014 and 2017. The project has worked with a large number of stakeholders, including district level officials, teams around the child, alternative care panels, probation social welfare officers, community development officers, para-social workers, community child protection structures, health facilities, child care institutions, vulnerable families and children, and with children in institutions, remand homes and children found to be living in street situations. The project was implemented in 12 districts of Uganda. This questionnaire is to be filled out by Probation Social Welfare Officers (PSWOs) in the 6 districts sampled for the endline evaluation. If there is any question you feel you cannot answer, please leave it blank and go to the next question. Your participation is anonymous. The analysis from this questionnaire will be presented in a report that will include data from a large number of stakeholders at national and district levels. No respondents will be identified by name in the report. We appreciate very much your time and participation. Demographics: 1. Name of district: 2. Gender: Male: Female: 3. Professional function: 4. Years in this function: Information about trainings, skills and awareness 5. What is your highest educational degree: 6. Can you name some of the main national policy documents and/or legislation, which guide you in your work? a. b. c. d. 7. Can you name some of the guidelines and/or operating procedures which guide you in your work: a. b. 3 c. d. 8. Since you started working in this position, when was the last time that you had formal training to facilitate your current job functions? Less than 1 year ago Between 1-3 years ago Between 3-6 Years ago More that 6 years ago 9. What were the themes of the trainings you have received (check all that apply): Parenting skills Communicating with children Counselling skills Case management skills Other: Information about general work operations: 10. Which of the job functions below describe well the work you currently do every month (check all that apply): Social inquiries with families and case management (including preparation of case files, going to court, referral of cases for support and follow-up) Information collection, inspections of child care institutions and entering data into the information systems Supervision and support to social work professionals and community volunteers (including child development officers and para-social workers) Advocacy for populations in critical need of support from services to access the services they need Provision of services (e.g. counselling, give parental advice etc.) and support for populations in critical need of support from services Coordination of social services and resources for vulnerable populations Awareness raising and education to promote wellbeing of populations (including on parenting skills, on how to access services and support) Other: 4 11. Is the separation of children from their families or caregivers a problem that requires the priority attention of policy makers and professionals working with families and children in Uganda? YES: NO: Information about case load: 12. Please rank the types of cases from 1-5 in order of how much time you spend in your work. 1 being types of cases you spend the MOST time on. 5 being types of cases you spend the LEAST time on. Cases of families who are very poor or are in critical need of support from services Cases of children who are to be / have been placed in a child care institution Cases of children who have become victims of any form of violence Cases of children who have committed an offence Cases of children who lack supervision of a permanent care giver (e.g children in street situations). 13. Currently, how many open cases (in numbers) do you have?: 14. For the last 12 months (counting from the day of responding to the survey), how many cases (in numbers) have been closed: Information about work with Child Care Institutions: 15. Is follow up on cases of children who have been placed in child care institutions part of your job? Yes, it is my job No, it is the job of somebody else 16. How many children residential care (in numbers) are you currently monitoring? 17. In your district, how many child care institutions for children without parental care are currently registered and accredited according to the children’s home regulations of the Ministry of Gender, Labour and Social Development? 18. In your district, how many child care institutions for children without parental care do you estimate operate but are NOT registered according to the children’s home regulations of the Ministry of Gender, Labour and Social Development? 19. How many child care institutions have you visited in the last 12 months? 20. Have you been trained on the Ministry of Gender, Labour and Social Development’s “Children (Approved Home) Regulations Assessment Toolkit”: YES: NO: 5 21. If yes, do you think this training was enough for you to be able to participate in an inspection of a child care institution? YES: NO: 22. How many formal inspections of residential care child institutions did you do in the last 12 months based on the Ministry of Gender, Labour and Social Development Assessment Toolkit? Information about work with Alternative Care Panels / Teams Around the Child 23. In your district, is there currently anything called an “Alternative Care Panel” or “Teams Around the Child”? YES: NO: 24. If yes, how often do you participate in the meetings of the “Alternative Care Panel” or “Teams Around the Child”? At least 1 time per week At least 1 time per month At least 1 time in 3 months Less than 1 time in 3 months Never or almost never 25. If yes to question 23, do you consider the Alternative Care Panel / “Teams Around the Child” an EFFICIENT decision making body to prevent children from family separation or to place children in good quality alternative care? YES: NO: 26. If yes to question 25, why is it efficient? Because of the quality of its members Because of the useful information shared in the meetings Because it can refer children to services of good quality Other: 27. If no to question 25, why is it NOT efficient? Because of the quality of its members Because of the information shared in the meetings 6 Because it cannot refer children to services of good quality Other: Information about referrals and collaboration with other services: 28. From where/who do you get referrals of cases of children who are identified as being at risk of being separated from their families or are at risk of violence, abuse, neglect or other harm? (check all that apply) Community structures (e.g. youth clubs, faith based organizations etc.): Para-social workers CDOs Providers of health services Providers of education services Other: 29. If a case of a family in critical need of support or a child at risk is referred to you, is there a standard procedure that you must follow to respond to the case? Yes, I use a standard procedure I have been trained on No, I improvise and use my professional judgement 30. How often do you discuss cases of vulnerable families or children or children at risk with the following structures / people (cross the one option that applies to each person / structure)? Every day Once a week Once per month Less than once per month Almost never or Never Supervisor Peers Community structures (e.g. youth clubs, faith based organizations etc.) CDOs Para-social workers Providers of 7 health services Providers of education services Alternative Care Panels Courts Child Care Institutions Remand homes THANK YOU FOR YOUR TIME! 8 Tool 2 – Guide for Discussion with DOVCU project partners Project implementation and monitoring 1. Please give an overall introduction of the project implementation, the types of activities undertaken, materials developed, what went well and what were challenges in the project implementation and monitoring. Examples of information the evaluation seeks to explore: - Type of learning processes and materials developed by the project - If there were any issues in the external environment and context of the project that influenced the project operation and the impact (e.g. political environment). - Mitigation of risks and new circumstances / opportunities for leveraging which arose. 2. Please describe how you monitored the project, how often and for what purposes? Examples of information the evaluation seeks to explore: - How the project integrated lessons learned and adjusted implementation plans - The type of issues identified during project monitoring that required adjustments in project implementation - What type of adjustments/changes that were made in the project during implementation in case, for example, there were unintended consequences identified of the implemented activities, Impact on system reform 3. In your opinion, to what extent did the project contribute to implementation of national care reform measures? Examples of information the evaluation seeks to explore: - Did the capacity of Ministry of Gender, Labour and Social Development to provide an oversight role of the child care system improve as a result of the project (e.g. their capacity to collect data and monitor the child care reform, to do inspection and enforcement of the law and policy) - If it is felt that the project increased the visibility of the need for a reform of the child care system - If the project contributed to better awareness of alternatives to institutions, such as for example national adoption, foster care, kinship care, supported independent living - If the project contributed to better awareness on the need to promote preventive and family support services 4. In your opinion, to what extent, and which systemic changes did the project bring about at community and district levels that are enabling children to live in family care and preventing inappropriate placements in institutional care? Examples of information the evaluation seeks to explore: - If capacity increased among district authorities and OVC committees to plan for services that can support child care reform efforts, and provide support to families and children - Capacity strengthening of Probation and Social Welfare Officers (PSWOs) and CDOs - Creation of gatekeeping mechanisms such as Alternative Care Panels (ACPs) or Teams Around the Child (TAC). - Changes in practices of Remand homes 9 - Changes in practices of Child Care Institutions - Strengthened capacity of Para-social workers to address alternative care and issues of separation and risk of separation for children - Strengthened capacity of Community child protection structures to work on alternative care issues of separation and risk of separation for children 5. In how many districts were Alternative Care Panels and Teams Around the Child established and how have these functioned: Examples of information the evaluation seeks to explore: - How were member selected - Who are the members - Regularity of meetings; - Level of participation of core members; - What were the core functions - Compliance with core functions; - The extent to which members knew and used foster care and adoption guidelines; - Coverage of various types of cases; - Regularity of reviews of placements and movements of children. - Any challenges and lessons learned from the work of ACPs and TACs. 6. In your opinion, what are the changes in child care institutions (CCIs) as a result of DOVCU￾project activities: Examples of information the evaluation seeks to explore: - How many child care institutions in the DOVCU-districts district have been inspected? - For those CCIs that received recommendations for improvements to be made, how did the project follow-up with these CCIs? - How many developed an improvement plan? - What was the experience and lessons learned regarding the implementation of improvement plans? - Who followed up with CCIs on the improvements that were needed, and was there any re￾inspection of CCIs after some time? Who did that re-inspection? - Of the CCIs that were inspected by MoGLSD and their team, how many received a license to operate? - What happened after CCIs received a license to operate, is there for example any obligation to regularly report on activities in the CCI to MoGLSD, as a requirement to maintain the license? 7. Were there any CCIs that closed completely? What do you think led to their closure? Examples of information the evaluation seeks to explore: - In which districts did this happen and what were the names of these CCIs? - How were such closures planned and monitored? - What happened to children in the CCIs that closed? - Who was involved in the oversight and monitoring of such closures? - Were there any CCIs that closed spontaneously without oversight by the project or by MoGLSD? - In which districts did this happen and what were the names of these CCIs? - Is it known what happened to children in the CCIs that closed? - What did the project do, and what did MoGLSD do when they learned about these self-closures? - What attempts did the project make, or MoGLSD and district staff make to trace and check on the children from the CCIs that self-closed? 10 8. In the DOVCU-districts how many of all the CCIs that were assessed and that the project worked with can now report to have a reduced number of children as compared to baseline? Examples of information the evaluation seeks to explore: - The extent to which the DOVCU-project has influences the numbers of children who are placed in CCIs. The effects can take different forms as below, and we are interested to understand what kind of dynamics that the DOVCU project has contributed to:: - Are there less children overall in CCI now as compared to three years ago? - Do fewer children enter CCIs now as compared to three years ago? - Do more children leave a child care placement now as compared to three years ago? - Do children on average stay shorter periods in CCIs as compared to three years ago? 9. Regarding the practices of child care institutions, in your opinion, were there any observed and documented changes in how they operate as a result of the DOVCU-project? Examples of information the evaluation seeks to explore: - Changes in governance and management culture (e.g. operating with formal registration, policies on having employed qualified social workers, overall human resource policies, and minimum qualifications for staff working in the CCI, policies for child protection, adequate descriptions of the types of care facility and services provided there etc.) - Changes in funding culture (e.g. fundraising for provision of residential care to individual children only or fundraising also to support children in their community) - Referral and admission practices (e.g. availability of care orders for all children staying at the CCI, criteria for admission, whether there are more streamlined referral and admission processes going through care planning and placement reviews with involvement of PSWOs and multi-disciplinary panels at district level) - Practices, policies and guidelines for in case review and reintegration - Policies for family visits and staying in touch with culture and community of children who are placed - Existence of standards of services provided to children in CCIs Changes in remand home practices 10. In your opinion, What are the changes in remand homes as a result of DOVCU-project activities?: Examples of information the evaluation seeks to explore: - The extent to which DOVCU has contributed to improvements in knowledge, skills and utilization by remand home staff of guidelines and tools (standard operating procedures) which project trained them on; - Improvements in the level of compliance with decisions on children (e.g. if children who have served their sentence are allowed to go home); - Availability of strategic partnerships and alliances for follow-up on cases of reintegration from remand homes. Beneficiaries: overall 11. The DOVCU project aimed to support 43,000 vulnerable households to prevent the separation of children from their families, and to reintegrate 2,000 children from child care institutions, from the 11 street and from remand homes. What is your perception of the success of the project in reaching those targets? Examples of information the evaluation seeks to explore: - Did the DOVCU project support the implementation of the conditions for such ambitious targets to be reached? - What were the strengths and weaknesses of the changes introduced by the DOVCU project to reach those targets? Beneficiaries: vulnerable families targeted for prevention 12. Please describe how families receiving prevention support were targeted. Examples of information the evaluation seeks to explore: - Was the targeting of the households that received support from the DOVCU project appropriate? - Did the project reach those families where children were the most at risk of separation? - If yes, what specifically made the project successful in reaching those most at risk? If no, what were the challenges? - Were activities to support families and children who were reintegrated tailored to the specific needs of each family / child? If yes, what were the strengths? If no, what were the challenges? 13. In your opinion, what has been the effect of the DOVCU-project’s interventions in the lives of the families that were who received support from the project? Examples of information the evaluation seeks to explore: - What kind of support packages did families receive in the end? - Who was targeted? - How did the project ensure to target the most vulnerable families? Was targeting successful or were there any cases, for example, were the wrong beneficiary was selected? - How successful were the various interventions in preventing separation of children at risk from their families? According to your own assessment, did the wellbeing of vulnerable families improve after DOVCU interventions? - What were the indicators of such success? - Were there any cases (households) when prevention methods of the project did not improve the situation for children in the household? Why was this? How did the project address this situation? - For which families were prevention efforts the most successful? What do you think were the circumstances that made these cases successful? Beneficiaries: reintegrated children 14. How were the children to be reintegrated from child care institutions selected? Examples of information the evaluation seeks to explore: - What profile did the children have, who ultimately got reintegrated? - Who decided which children should be reintegrated? - How was the decision made? - Were these children the most vulnerable or those most in need of being reintegrated? 12 15. Overall, how many children have been reintegrated or placed in alternative family based care from CCIs and what has been the overall experience of these reintegration cases? Please talk both about the positive experience and the challenges that were faced. Examples of information the evaluation seeks to explore: - If in addition to the quoted number, how many cases of children leaving CCIs the project is thought to have influenced, even if not all these cases were followed by the project? - If the project reached its targets, what were the facilitating factors ensuring the target were reached? - If targets were not reached, what were the challenges in reaching targets? 16. Of the children that the DOVCU project reintegrated, do all children have a permanency plan? Examples of information the evaluation seeks to explore: - Contents of permanency plans; - If any child did not have a permanency plan, why not? 17. Of the children that the DOVCU project reintegrated, what has been the experience of implementing the permanency plan? Examples of information the evaluation seeks to explore: - What were successes and challenges? - In how many cases was it not possible to implement the permanency plan, what were the challenges? - Now that the project is ending, for how long will children continue to be followed up by ChildFund and the project partners, by PSWOs, CDOs or other government staff? - What will allow for such follow-up after the project has ended (e.g. availability of sustained mobility of PSWOs and CDOs, institutionalized government guidelines for case management, reintegration and permanency planning etc.) 18. What has been the impact of the reintegration of children on the children and the families they were reintegrated in to? Examples of information the evaluation seeks to explore: - What kind of support did children receive (pre-, during and post reintegration activities)? - What kind of support did families receive (pre-, during and post reintegration activities)? - How successful were the reintegration activities overall? According to your assessment, did the wellbeing of reintegrated children improve after DOVCU interventions? - In what way did their wellbeing improve? Was there any area of wellbeing that showed greater improvements than other? - Did all reintegrated children remain in their placements? - Do you know of any case where the child did not remain in his/her placement? What happen? - What kind of actions did the project take in the cases where children did not show improved well￾being after they had been reintegrated - Do you know of any case where the reintegration of the child did not show improvements in the well-being status of the child? What happen? How did the project address this situation? - Do you know of any case where a reintegrated child returned to institutions or to the streets? What happen? How did the project address this situation? 13 19. For the children who have been successfully reintegrated with their families or placed in alternative family based care, what made the reintegration method successful and sustainable? Examples of information the evaluation seeks to explore: - What makes these reintegration methods sustainable? - For which children was impact most visible? - Why do you think reintegration efforts were more successful for some children? - What were the circumstances that influenced this? 20. For the children for whom reintegration was not successful, what were the challenges and how were those mitigated? Examples of information the evaluation seeks to explore: - How were risks assessed and monitored - If risk was identified, how was it reduced 21. To what extent was the project able to prevent child protection issues and human rights violations (e.g. child labour, violence against children, neglect, early marriage, etc.) and mitigate risks for creating greater vulnerability (e.g. child labour and any kind of harm)? Examples of information the evaluation seeks to explore: - If risks for any particular human rights violations were observed; - The reasons for those risks Reintegration of children from remand homes 22. Overall, how many children have been reintegrated or placed in alternative family based care from remand homes as a result of the project? Examples of information the evaluation seeks to explore: - Are there less children in remand homes now as compared to three years ago? - Do fewer children enter remand homes now as compared to three years ago? - Have the reasons for why children are placed in remand homes changed (e.g. now only for heavier crimes, or also petty crimes, behavioral or statutory offences) - Do more children leave a child care placement (outflow) now as compared to three years ago? - Do children more now than three years ago, leave remand homes quicker and easier or does it still happen that children miss their trials because of fuel shortages or inability to get to the court room for hearings? 23. What has been the experience of these cases of reintegration? Please talk both about the positive experience and the challenges that were faced in the process. Examples of information the evaluation seeks to explore: - For the children who are going home from remand homes, do they have a plan for how reintegration should be done to ensure safe and permanent reintegration? - What was the experience in implementing this plan? Reintegration of children from the street: 14 24. Overall, how many children have been reintegrated or placed in alternative family based care from the street as a result of the project? Examples of information the evaluation seeks to explore: - If there is an overall reduction in the phenomenon of children recorded to be on the street in the districts where DOVCU operated 25. What has been the experience of these cases of reintegration? Please talk both about the positive experience and the challenges that were faced in the process. Examples of information the evaluation seeks to explore: - What was the experience of implementing the permanency plan – successes and challenges? - In how many cases was it not possible to implement the permanency plan, what were the challenges? - Now that the project is ending, for how long will children continue to be followed up by ChildFund and the project partners, by PSWOs, CDOs or other government staff? - What will allow for such follow-up after the project has ended (e.g. availability of sustained mobility of PSWOs and CDOs, institutionalized government guidelines for case management, reintegration and permanency planning etc.) Professional practices of PSWOs, CDOs and para-social workers 26. In your opinion, what changes in professional practices of PSWOs, CDOs and para-social workers can be attributed to the project? Examples of information the evaluation seeks to explore: - If PSWOs and CDOs are more regularly involved in alternative care cases, cases of children to be reintegrated from the street or from remand homes - Changes in the quality and use of standardized processes and forms for alternative care case assessments and reintegration - Regular follow-up to cases of risk for children (in CCIs, on the street, in remand homes, in families) - Improvements in the quality of case files for alternative care cases - Better regularity in follow-up in alternative care placements - Better regularity in PSWO and CDO monitoring of cases until reintegration - Better regularity and quality of inspections and reporting on CCIs and children placed in CCIs to central Ministry - Regular use of standardized referral protocols; - Better clarity among professionals on the trajectory of cases (e.g. referrals and admission practices). - Collaborative spirit between all professionals involved at district, community and service delivery level to first and foremost ensure prevention of separation, and as a last resort quality of placements of children into care. Community child protection mechanisms and para-social workers 27. In your opinion, what changes in community child protection mechanisms and para-social workers can be attributed to the project? Examples of information the evaluation seeks to explore: 15 - Improvements in understanding by community based child protection structures and para social workers of their key roles and contributions to prevent institutionalization and follow-up on child protection cases and cases of children who have been reintegrated. - Changes in awareness and skills to identify at risk children; - Changes in the frequency of referrals coming from these structures to PSWOs; - More proactive actions taken by these structures to solve the problems at their levels; - Proactive activities of these structures to make people aware of the existence of services that can help prevent separation pf children from their families. Services and supports provided through DOVCU 28. To what extent do you think the DOVCU project has responded to most common risk factors for separation (including for boys and girls) Examples of information the evaluation seeks to explore: - Was the support from the integrated package of interventions meeting the most urgent needs of children and families that the project targeted? - Which services/support offered by DOVCU have been the most useful and had the most effect for prevention? - What combinations of supports have been most useful to ensure child protection, and safe reintegration? - Were there, in your opinion, any missing elements in the support provided that should have been included in the support package to vulnerable families, in order to prevent the separation of children from these families? - To what extent did support and services offered to families need adaptation to the local context? Please give examples of how such adaptation was made. 29. In your opinion, would the services and supports offered in DOVCU-districts would have the same effect if replicated / expanded in other parishes and/or districts in Uganda or are there any elements that would need better adaptation? Examples of information the evaluation seeks to explore: - What kind of adaptation is suggested 30. Which structures and processes and procedures have the DOVCU-project has contributed to establish? Examples of information the evaluation seeks to explore: - E.g. if alternative care panels, teams around the child were established, if processes and procedures related to supervision/monitoring of CCIs and of reintegrated children, case management for children in need of alternative care or at risk changed or improved during the project; - If these structures, processes and procedures are perceived as sustainable; - What makes these changes sustainable? - What would be needed to make them sustainable? Lessons learned: 31. What are the specific lessons learned from this project that would be useful to document and share? 16 Examples of information the evaluation seeks to explore: - Reflections on the overall design of the project and if (in retrospect) selection of intervention areas and activities were sound. - Things that worked well, or were a challenge in the project’s work with child care institutions. - Suggestions for how to overcome such challenges in any future project with similar objectives. - Things that worked well, or were a challenge in the project’s work with different professionals involved in gatekeeping and outreach to vulnerable families (e.g. para professionals how are they working together with probation and social welfare officers and what is motivating them in their work) - Things that worked well, or were a challenge in the project’s integrated package of support? - Things that worked well, or were a challenge in the project’s cash-benefit and income generating approaches. 17 Tool 3 – Guide for Key Informant Interviews with key child protection stakeholders not involved in the DOVCU project: Introduction: Interviewer to start with an introduction, which organization he/she is representing and in which context the interview is taking place: The interview is being carried out in the context of an end-line evaluation of a project (DOVCU) that has been implemented in the area of child care and de-institutionalization by MoGLSD and a consortium led by ChildFund (RETRAK, TPO and Child’s I Foundation). The project started in 2014 and was implemented for 3 years across 12 districts of Uganda. It had for overall objective to prevent family separation and reintegrate children without appropriate family care back into nurturing family environments. The project worked with a large number of stakeholders, including district level officials, alternative care panels, probation officers, community development officers, para-social workers, community child protection structures, health facilities, child care institutions, vulnerable families and children and with children in institutions, remand homes and children found to be living in street situations. With this interview, we want to establish your involvement into the project, what and how you think it has contributed to changes in the lives of the beneficiaries. What you will tell us in the interview will remain confidential. The information you share with us will be compiled in a report that will have more than 100 respondents. We may quote you, but it will not be possible to link quotes or information discussed in the report with any specific respondent. Notes and recordings from the discussion will remain with the evaluator and will not be shared with anybody else. The interview is expected to take 1 hour. Ask for permission to start the interview, and to record the interview. Project implementation and monitoring - Please give an overall introduction of what you know about the DOVCU-project: Examples of information the evaluation seeks to explore: - Types of activities of the project that the person witnessed - Reflections on challenges and lessons learned Impact on system reform - In your opinion to what extent did the project contribute to implementation of national care reform measures? Examples of information the evaluation seeks to explore: - Did the capacity of Ministry of Gender, Labour and Social Development to provide an oversight role of the child care system improve as a result of the project (e.g. their capacity to collect data and monitor the child care reform, to do inspection and enforcement of the law and policy) - If it is felt that the project increased the visibility of the need for a reform of the child care system - If the project contributed to better awareness of alternatives to institutions, such as for example national adoption, foster care, kinship care, supported independent living - If the project contributed to better awareness on the need to promote preventive and family support services 18 - In your opinion, to what extent, and which systemic changes did the project bring about at community and district levels that are enabling children to live in family care and preventing inappropriate placements in institutional care? Examples of information the evaluation seeks to explore: - If capacity increased among district authorities and OVC committees to plan for services that can support child care reform efforts, and provide support to families and children - Capacity strengthening of Probation and Social Welfare Officers (PSWOs) and CDOs - Creation of gatekeeping mechanisms such as Alternative Care Panels (ACPs) or Teams Around the Child (TAC). - Changes in practices of Remand homes - Changes in practices of Child Care Institutions - Strengthened capacity of Para-social workers to address alternative care and issues of separation and risk of separation for children - Strengthened capacity of Community child protection structures to work on alternative care issues of separation and risk of separation for children - In how many districts were Alternative Care Panels and Teams Around the Child established and how have these functioned? Examples of information the evaluation seeks to explore: - How were member selected - Who are the members - Regularity of meetings; - Level of participation of core members; - What were the core functions - Compliance with core functions; - The extent to which members knew and used foster care and adoption guidelines; - Coverage of various types of cases; - Regularity of reviews of placements and movements of children. - Any challenges and lessons learned from the work of ACPs and TACs. - In your opinion, what are the changes in child care institutions (CCIs) as a result of DOVCU￾project activities? Examples of information the evaluation seeks to explore: - How many child care institutions in the DOVCU-districts district have been inspected? - For those CCIs that received recommendations for improvements to be made, how did the project follow-up with these CCIs? - How many developed an improvement plan? - What was the experience and lessons learned regarding the implementation of improvement plans? - Who followed up with CCIs on the improvements that were needed, and was there any re￾inspection of CCIs after some time? Who did that re-inspection? - Of the CCIs that were inspected by MoGLSD and their team, how many received a licence to operate? - What happened after CCIs received a license to operate, is there for example any obligation to regularly report on activities in the CCI to MoGLSD, as a requirement to maintain the licence? - Were there any CCIs that closed completely? What do you think led to their closure? Examples of information the evaluation seeks to explore: 19 - In which districts did this happen and what were the names of these CCIs? - How were such closures planned and monitored? - What happened to children in the CCIs that closed? - Who was involved in the oversight and monitoring of such closures? - Were there any CCIs that closed spontaneously without oversight by the project or by MoGLSD? - In which districts did this happen and what were the names of these CCIs? - Is it known what happened to children in the CCIs that closed? - What did the project do, and what did MoGLSD do when they learned about these self-closures? - What attempts did the project make, or MoGLSD and district staff make to trace and check on the children from the CCIs that self-closed? - In the DOVCU-districts how many of all the CCIs that were assessed and that the project worked with can now report to have a reduced number of children as compared to baseline? Examples of information the evaluation seeks to explore: - The extent to which the DOVCU-project has influences the numbers of children who are placed in CCIs. The effects can take different forms as below, and we are interested to understand what kind of dynamics that the DOVCU project has contributed to:: - Are there less children overall in CCI now as compared to three years ago? - Do fewer children enter CCIs now as compared to three years ago? - Do more children leave a child care placement now as compared to three years ago? - Do children on average stay shorter periods in CCIs as compared to three years ago? - Regarding the practices of child care institutions, were there any observed and documented changes in how they operate as a result of the DOVCU-project? Examples of information the evaluation seeks to explore: - Changes in governance and management culture (e.g. operating with formal registration, policies on having employed qualified social workers, overall human resource policies, and minimum qualifications for staff working in the CCI, policies for child protection, adequate descriptions of the types of care facility and services provided there etc.) - Changes in funding culture (e.g. fundraising for provision of residential care to individual children only or fundraising also to support children in their community) - Referral and admission practices (e.g. availability of care orders for all children staying at the CCI, criteria for admission, whether there are more streamlined referral and admission processes going through care planning and placement reviews with involvement of PSWOs and multi-disciplinary panels at district level) - Practices, policies and guidelines for in case review and reintegration - Policies for family visits and staying in touch with culture and community of children who are placed - Existence of standards of services provided to children in CCIs Changes in remand home practices - In your opinion, what are the changes in remand homes as a result of DOVCU-project activities: Examples of information the evaluation seeks to explore: - The extent to which DOVCU has contributed to improvements in knowledge, skills and utilization by remand home staff of guidelines and tools (standard operating procedures) which project trained them on; 20 - Improvements in the level of compliance with decisions on children (e.g. if children who have served their sentence are allowed to go home); - Availability of strategic partnerships and alliances for follow-up on cases of reintegration from remand homes. Beneficiaries: overall - The DOVCU project aimed at supporting 43,000 vulnerable households to prevent the separation of children from their families, and to reintegrate 2,000 children from child care institutions, from the street and from remand homes. What is your perception of the success of the project to reach those targets? Example of information we are interested to receive: - Did the DOVCU project support the implementation of the conditions for such ambitious targets to be reached? - What were the strengths and weaknesses of the changes introduced by the DOVCU project to reach those targets? Beneficiaries: vulnerable families targeted for prevention - Please describe how families to be targeted for prevention support were targeted. Examples of information the evaluation seeks to explore: - Was the targeting of the households that received support from the DOVCU project appropriate? - Did the project reach those families where children were the most at risk of separation? - If yes, what specifically made the project successful in reaching those most at risk? If no, what were the challenges? - Were activities to support families and children who were reintegrated tailored to the specific needs of each family / child? If yes, what were the strengths? If no, what were the challenges? - What has been the effect of the DOVCU-project’s interventions in the lives of the families that were targeted for prevention activities and who received support from the project? Examples of information the evaluation seeks to explore: - What kind of support packages did families receive in the end? - Who was targeted? - How did the project ensure to target the most vulnerable families? Was targeting successful or were there any cases, for example, were the wrong beneficiary was selected? - How successful were the various interventions in preventing separation of children at risk from their families? According to your own assessment, did the wellbeing of vulnerable families improve after DOVCU interventions? - What were the indicators of such success? - Were there any cases (households) when prevention methods of the project did not improve the situation for children in the household? Why was this? How did the project address this situation? - For which families were prevention efforts the most successful? What do you think were the circumstances that made these cases successful? Beneficiaries: reintegrated children 21 - For children to be reintegrated from child care institutions, how were children to be reintegrated selected? Examples of information the evaluation seeks to explore: - What profile did the children have, who ultimately got reintegrated? - Who decided which children should be reintegrated? - How was the decision made? - Were these children the most vulnerable or those most in need of being reintegrated? - Overall, how many children have been reintegrated or placed in alternative family based care from CCIs and what has been the overall experience of these reintegration cases? Please talk both about the positive experience and the challenges that were faced. Examples of information the evaluation seeks to explore: - If in addition to the quoted number, how many cases of children leaving CCIs the project is thought to have influenced, even if not all these cases were followed by the project? - If the project reached its targets, what were the facilitating factors ensuring the target were reached? - If targets were not reached, what were the challenges in reaching targets? 32. Of the children that the DOVCU project reintegrated, do all children have a permanency plan? Examples of information the evaluation seeks to explore: - Contents of permanency plans; - If any child did not have a permanency plan, why not? 33. Of the children that the DOVCU project reintegrated, what has been the experience of implementing the permanency plan? Examples of information the evaluation seeks to explore: - What were successes and challenges? - In how many cases was it not possible to implement the permanency plan, what were the challenges? - Now that the project is ending, for how long will children continue to be followed up by ChildFund and the project partners, by PSWOs, CDOs or other government staff? - What will allow for such follow-up after the project has ended (e.g. availability of sustained mobility of PSWOs and CDOs, institutionalized government guidelines for case management, reintegration and permanency planning etc.) - What has been the effect of the reintegration of children on the children and the families where children were received? Examples of information the evaluation seeks to explore: - What kind of support did children receive (pre-, during and post reintegration activities)? - What kind of support did families receive (pre-, during and post reintegration activities)? - How successful were the reintegration activities overall? According to your assessment, did the wellbeing of reintegrated children improve after DOVCU interventions? 22 - In what way did their wellbeing improve? Was there any area of wellbeing that showed greater improvements than other? - Did all reintegrated children remain in their placements? - Do you know of any case where the child did not remain in his/her placement? What happen? - What kind of actions did the project take in the cases where children did not show improved well￾being after they had been reintegrated - Do you know of any case where the reintegration of the child did not show improvements in the well-being status of the child? What happen? How did the project address this situation? - Do you know of any case where a reintegrated child returned to institutions or to the streets? What happen? How did the project address this situation? - For the children who have been successfully reintegrated with their families or placed in alternative family based care, what made the reintegration method successful and sustainable? Examples of information the evaluation seeks to explore: - What makes these reintegration methods sustainable? - For which children was impact most visible? - Why do you think reintegration efforts were more successful for some children? - What were the circumstances that influenced this? - For the children for whom reintegration was not successful, what were the challenges and how were those mitigated? Examples of information the evaluation seeks to explore: - How were risks assessed and monitored - If risk was identified, how was it reduced - To what extent was the project able to prevent child protection issues and human rights violations (e.g. child labour, violence against children, neglect, early marriage, etc.) and mitigate risks for creating greater vulnerability (e.g. child labour and any kind of harm)? Examples of information the evaluation seeks to explore: - If risks for any particular human rights violations were observed; - The reasons for those risks Reintegration of children from remand homes 34. Overall, how many children have been reintegrated or placed in alternative family based care from remand homes as a result of the project? Examples of information the evaluation seeks to explore: - Are there less children in remand homes now as compared to three years ago? - Do fewer children enter remand homes now as compared to three years ago? - Have the reasons for why children are placed in remand homes changed (e.g. now only for heavier crimes, or also petty crimes, behavioral or statutory offences) - Do more children leave a child care placement (outflow) now as compared to three years ago? - Do children more now than three years ago, leave remand homes quicker and easier or does it still happen that children miss their trials because of fuel shortages or inability to get to the court room for hearings? 23 35. What has been the experience of these cases of reintegration? Please talk both about the positive experience and the challenges that were faced in the process. Examples of information the evaluation seeks to explore: - For the children who are going home from remand homes, do they have a plan for how reintegration should be done to ensure safe and permanent reintegration? - What was the experience in implementing this plan? Reintegration of children from the street: 36. Overall, how many children have been reintegrated or placed in alternative family based care from the street as a result of the project? Examples of information the evaluation seeks to explore: - If there is an overall reduction in the phenomenon of children recorded to be on the street in the districts where DOVCU operated 37. What has been the experience of these cases of reintegration? Please talk both about the positive experience and the challenges that were faced in the process. Examples of information the evaluation seeks to explore: - What was the experience of implementing the permanency plan – successes and challenges? - In how many cases was it not possible to implement the permanency plan, what were the challenges? - Now that the project is ending, for how long will children continue to be followed up by ChildFund and the project partners, by PSWOs, CDOs or other government staff? - What will allow for such follow-up after the project has ended (e.g. availability of sustained mobility of PSWOs and CDOs, institutionalized government guidelines for case management, reintegration and permanency planning etc.) Professional practices of PSWOs, CDOs and para-social workers - What changes in professional practices of PSWOs, CDOs and para-social workers do you think the DOVCU - project activities influenced: Examples of information the evaluation seeks to explore: - If PSWOs and CDOs are more regularly involved in alternative care cases, cases of children to be reintegrated from the street or from remand homes - Changes in the quality and use of standardized processes and forms for alternative care case assessments and reintegration - Regular follow-up to cases of risk for children (in CCIs, on the street, in remand homes, in families) - Improvements in the quality of case files for alternative care cases - Better regularity in follow-up in alternative care placements - Better regularity in PSWO and CDO monitoring of cases until reintegration - Better regularity and quality of inspections and reporting on CCIs and children placed in CCIs to central Ministry - Regular use of standardized referral protocols; - Better clarity among professionals on the trajectory of cases (e.g. referrals and admission practices). 24 - Collaborative spirit between all professionals involved at district, community and service delivery level to first and foremost ensure prevention of separation, and as a last resort quality of placements of children into care. Community child protection mechanisms and para-social workers - What changes in community child protection mechanisms and para-social workers do you think the DOVCU - project activities influenced: Examples of information the evaluation seeks to explore: - Improvements in understanding by community based child protection structures and para social workers of their key roles and contributions to prevent institutionalization and follow-up on child protection cases and cases of children who have been reintegrated. - Changes in awareness and skills to identify at risk children; - Changes in the frequency of referrals coming from these structures to PSWOs; - More proactive actions taken by these structures to solve the problems at their levels; - Proactive activities of these structures to make people aware of the existence of services that can help prevent separation pf children from their families. Services and supports provided through DOVCU - To what extent do you think the DOVCU project has responded to most common risk factors for separation (including for boys and girls) Examples of information the evaluation seeks to explore: - Was the support from the integrated package of interventions meeting the most urgent needs of children and families that the project targeted? - Which services/support offered by DOVCU have been the most useful and had the most effect for prevention? - What combinations of supports have been most useful to ensure child protection, and safe reintegration? - Were there, in your opinion, any missing elements in the support provided that should have been included in the support package to vulnerable families, in order to prevent the separation of children from these families? - To what extent did support and services offered to families need adaptation to the local context? Please give examples of how such adaptation was made. - Do you services and supports offered in DOVCU-districts would have the same effect if replicated / expanded in other parishes and/or districts in Uganda or are there any elements that would need better adaptation? Example of information we are interested to receive: - What kind of adaptation is suggested - To what extent do you feel that the DOVCU-project has contributed to put in place structures and processes that are sustainable beyond the project’s timeframe? Examples of information the evaluation seeks to explore: 25 - What structures have been established / strengthened? - What processes and procedures have been changed ((e.g. supervision/monitoring of CCIs and of reintegrated children, case management for children in need of alternative care or at risk) - What makes these changes sustainable? - What would be needed to make them sustainable? Lessons learned: - What are the specific lessons learned from this project that would be useful to document and share: Examples of information the evaluation seeks to explore: - Reflections on the overall design of the project and if (in retrospect) selection of intervention areas and activities were sound. - Things that worked well, or were a challenge in the project’s work with child care institutions. - Suggestions for how to overcome such challenges in any future project with similar objectives. - Things that worked well, or were a challenge in the project’s work with different professionals involved in gatekeeping and outreach to vulnerable families (e.g. para professionals how are they working together with probation and social welfare officers and what is motivating them in their work) - Things that worked well, or were a challenge in the project’s integrated package of support? - Things that worked well, or were a challenge in the project’s cash-benefit and income generating approaches. 26 Tool 4 – Guide for Key Informant Interviews at Ministry of Gender Labour and Social Development: Interviewer to start with an introduction, which organization he/she is representing and in which context the interview is taking place: The interview is being carried out in the context of an end-line evaluation of a project (DOVCU) that has been implemented in the area of child care and de-institutionalization by MoGLSD and a consortium led by ChildFund (RETRAK, TPO and Child’s I Foundation). The project started in 2014 and was implemented for 3 years across 12 districts of Uganda. It had for overall objective to prevent family separation and reintegrate children without appropriate family care back into nurturing family environments. The project worked with a large number of stakeholders, including district level officials, alternative care panels, probation officers, community development officers, para-social workers, community child protection structures, health facilities, child care institutions, vulnerable families and children and with children in institutions, remand homes and children found to be living in street situations. With this interview, we want to establish your involvement into the project, what and how you think it has contributed to changes in the lives of the beneficiaries. What you will tell us in the interview will remain confidential. The information you share with us will be compiled in a report that will have more than 100 respondents. We may quote you, but it will not be possible to link quotes or information discussed in the report with any specific respondent. Notes and recordings from the discussion will remain with the evaluator and will not be shared with anybody else. The interview is expected to take 1 hour. Ask for permission to start the interview, and to record the interview. Project implementation and monitoring 1. Please tell us how the MoGLSD was involved in the DOVCU-project implementation, the Ministry’s role, the types of activities MoGLSD was involved in? Examples of information the evaluation seeks to explore: - What briefings, trainings and other activities that they were involved in, through the DOVCU project, what were the strengths and what were the weaknesses of these? - How many staff in this office were trained? On what topics were the trainings? - Particularly, what new did you learn from trainings? Have you changed any of your processes / practices as a result of these trainings? What changes did you introduce? - What were the strengths and what were the weaknesses of these trainings? - What activities and strategies used by the project have you found to be the most useful, to for example change the way you work (assessments, coaching, training, technical assistance activities etc.) 2. How would you describe your overall experience in the project? Examples of information the evaluation seeks to explore: - What went well and what were challenges in the project implementation and monitoring. - Were the human resources (e.g. social work capacity, staff providing training and coaching activities), and other resources invested in the project such as coordination mechanisms (e.g. TAC and ACPs, social workers and project coordinators), time and other resources sufficient for there to be a lasting change in professional practices & were these resources complementary to what you can mobilize themselves. - Is there any activity that you think should have been included that was not included? Did you provide this feedback to the DOVCU-project staff? What did they do? 27 Impact on system reform 3. In your opinion, to what extent did the project contribute to implementation of national care reform measures? Examples of information the evaluation seeks to explore: - Did the capacity of Ministry of Gender, Labour and Social Development to provide an oversight role of the child care system improve as a result of the project, please provide examples - If it is felt that the project increased the visibility of the need for a reform of the child care system - If the project contributed to better awareness of alternatives to institutions, such as for example national adoption, foster care, kinship care, supported independent living - If the project contributed to better awareness on the need to promote preventive and family support services 4. Please describe if and how you were involved in monitoring of the project and if and how data collected through this project informed any information systems or databases at the national level: Examples of information the evaluation seeks to explore / see: - Sample of the alternative care directories and child care directories – established through the project - Changes in these directories since the beginning of the project - How up-to-date these directories are - How often they are updated - Strengths and weaknesses of these directories? - How data is used them to analyse the situation? - Whether these directories be sustained? - Whether MoGLSD is now regularly monitoring any national child care indicators and which indicators they monitor 5. Has the MoGLSD been involved in inspection of child care institutions through this project? If yes, please describe that experience. Examples of information the evaluation seeks to explore / see: - Who and how many people were involved in the inspections? - When did these happen and how often were they repeated - What follow-up was provided after the first assessment was done with CCIs identified for improvement – was re-inspection done and CCIs given a license? - What follow-up was provided after the first assessment with CCIs identified for further investigations and closures? 6. Have there been any changes in staffing or procedures for MoGLSD to provide oversight of CCIs, since the start of the DOVCU project? Examples of information the evaluation seeks to explore / see: - If expertise has been recruited on facilitating closures - If there are any guidelines adopted on how to perform closures - If any MoUs or collaboration was initiated with police or prosecutors to enable enforcement of the law and policy in cases where CCIs were refusing to comply, or in violation with the law 28 7. In your opinion, did the project influence any systemic changes at community and district levels that are enabling children to live in family care and preventing inappropriate placements in institutional care? To what extent can the changes be attributed to the project? Examples of information the evaluation seeks to explore: - If capacity increased among district authorities and OVC committees to plan for services that can support child care reform efforts, and provide support to families and children - Capacity strengthening of Probation and Social Welfare Officers (PSWOs) and CDOs - Creation of gatekeeping mechanisms such as Alternative Care Panels (ACPs) or Teams Around the Child (TAC). - Changes in practices of Remand homes - Changes in practices of Child Care Institutions - Strengthened capacity of Para-social workers to address alternative care and issues of separation and risk of separation for children - Strengthened capacity of Community child protection structures to work on alternative care issues of separation and risk of separation for children 8. What has been the experience of the Ministry with Alternative Care Panels and Teams Around the Child? How many districts Alternative Care Panels and Teams Around the Child were established? How have these functioned? Would you expect them to continue functioning? (if no, why not?) Examples of information the evaluation seeks to explore: - How were member selected - Who are the members - Regularity of meetings; - Level of participation of core members; - What were the core functions - Compliance with core functions; - The extent to which members knew and used foster care and adoption guidelines; - Coverage of various types of cases; - Regularity of reviews of placements and movements of children. - Any challenges and lessons learned from the work of ACPs and TACs. 9. What are the changes in child care institutions (CCIs) as a result of DOVCU-project activities?: Examples of information the evaluation seeks to explore: - How many child care institutions in the DOVCU-districts district have been inspected? - For those CCIs that received recommendations for improvements to be made, how did the project follow-up with these CCIs? - How many developed an improvement plan? - What was the experience and lessons learned regarding the implementation of improvement plans? - Who followed up with CCIs on the improvements that were needed, and was there any re￾inspection of CCIs after some time? Who did that re-inspection? - Of the CCIs that were inspected by MoGLSD and their team, how many received a licence to operate? - What happened after CCIs received a license to operate, is there for example any obligation to regularly report on activities in the CCI to MoGLSD, as a requirement to maintain the licence? 10. Were there any CCIs that closed completely? 29 Examples of information the evaluation seeks to explore: - In which districts did this happen and what were the names of these CCIs? - How were such closures planned and monitored? - What happened to children in the CCIs that closed? - Who was involved in the oversight and monitoring of such closures? - Were there any CCIs that closed spontaneously without oversight by the project or by MoGLSD? - In which districts did this happen and what were the names of these CCIs? - Is it known what happened to children in the CCIs that closed? - What did the project do, and what did MoGLSD do when they learned about these self-closures? - What attempts did the project make, or MoGLSD and district staff make to trace and check on the children from the CCIs that self-closed? 11. In the DOVCU-districts how many CCIs did the project assess and work with and how many of these CCIs can now report to have a reduced number of children as compared to baseline? Examples of information the evaluation seeks to explore: - The extent to which the DOVCU-project has influences the numbers of children who are placed in CCIs. The effects can take different forms as below, and we are interested to understand what kind of dynamics that the DOVCU project has contributed to:: - Are there less children overall in CCI now as compared to three years ago? - Do fewer children enter CCIs now as compared to three years ago? - Do more children leave a child care placement now as compared to three years ago? - Do children on average stay shorter periods in CCIs as compared to three years ago? 12. Regarding the practices of child care institutions, were there any observed and documented changes in how they operate as a result of the DOVCU-project? Examples of information the evaluation seeks to explore: - Changes in governance and management culture (e.g. operating with formal registration, policies on having employed qualified social workers, overall human resource policies, and minimum qualifications for staff working in the CCI, policies for child protection, adequate descriptions of the types of care facility and services provided there etc.) - Changes in funding culture (e.g. fundraising for provision of residential care to individual children only or fundraising also to support children in their community) - Referral and admission practices (e.g. availability of care orders for all children staying at the CCI, criteria for admission, whether there are more streamlined referral and admission processes going through care planning and placement reviews with involvement of PSWOs and multi-disciplinary panels at district level) - Practices, policies and guidelines for in case review and reintegration - Policies for family visits and staying in touch with culture and community of children who are placed - Existence of standards of services provided to children in CCIs Changes in remand home practices 13. We are also interested to know what are the changes in remand homes as a result of DOVCU￾project activities: Examples of information the evaluation seeks to explore: 30 - The extent to which DOVCU has contributed to improvements in knowledge, skills and utilization by remand home staff of guidelines and tools (standard operating procedures) which project trained them on; - Improvements in the level of compliance with decisions on children (e.g. if children who have served their sentence are allowed to go home); - Availability of strategic partnerships and alliances for follow-up on cases of reintegration from remand homes. Beneficiaries: overall 14. The DOVCU project aimed at supporting 43,000 vulnerable households to prevent the separation of children from their families, and to reintegrate 2,000 children from child care institutions, from the street and from remand homes. What is your perception of the success of the project to reach those targets? Example of information we are interested to receive: - Did the DOVCU project support the implementation of the conditions for such ambitious targets to be reached? - What were the strengths and weaknesses of the changes introduced by the DOVCU project to reach those targets? Beneficiaries: vulnerable families targeted for prevention 15. Please describe how families to be targeted for prevention support were targeted. Examples of information the evaluation seeks to explore: - Was the targeting of the households that received support from the DOVCU project appropriate? - Did the project reach those families where children were the most at risk of separation? - If yes, what specifically made the project successful in reaching those most at risk? If no, what were the challenges? - Were activities to support families and children who were reintegrated tailored to the specific needs of each family / child? If yes, what were the strengths? If no, what were the challenges? 16. What has been the effect of the DOVCU-project’s interventions in the lives of the families that were targeted for prevention activities and who received support from the project? Examples of information the evaluation seeks to explore: - What kind of support packages did families receive in the end? - Who was targeted? - How did the project ensure to target the most vulnerable families? Was targeting successful or were there any cases, for example, were the wrong beneficiary was selected? - How successful were the various interventions in preventing separation of children at risk from their families? According to your own assessment, did the wellbeing of vulnerable families improve after DOVCU interventions? - What were the indicators of such success? - Were there any cases (households) when prevention methods of the project did not improve the situation for children in the household? Why was this? How did the project address this situation? - For which families were prevention efforts the most successful? What do you think were the circumstances that made these cases successful? 31 Beneficiaries: reintegrated children 17. For children to be reintegrated from child care institutions, how were children to be reintegrated selected? Examples of information the evaluation seeks to explore: - What profile did the children have, who ultimately got reintegrated? - Who decided which children should be reintegrated? - How was the decision made? - Were these children the most vulnerable or those most in need of being reintegrated? 18. Overall, how many children have been reintegrated or placed in alternative family based care from CCIs and what has been the overall experience of these reintegration cases? Please talk both about the positive experience and the challenges that were faced. Examples of information the evaluation seeks to explore: - If in addition to the quoted number, how many cases of children leaving CCIs the project is thought to have influenced, even if not all these cases were followed by the project? - If the project reached its targets, what were the facilitating factors ensuring the target were reached? - If targets were not reached, what were the challenges in reaching targets? 19. Of the children that the DOVCU project reintegrated do all children have a permanency plan and what has been the experience of implementing the permanency plan? Examples of information the evaluation seeks to explore: - What were successes and challenges? - In how many cases was it not possible to implement the permanency plan, what were the challenges? - Now that the project is ending, for how long will children continue to be followed up by ChildFund and the project partners, by PSWOs, CDOs or other government staff? - What will allow for such follow-up after the project has ended (e.g. availability of sustained mobility of PSWOs and CDOs, institutionalized government guidelines for case management, reintegration and permanency planning etc.) 20. What has been the effect of the reintegration of children on the children and the families where children were received? Examples of information the evaluation seeks to explore: - What kind of support did children receive (pre-, during and post reintegration activities)? - What kind of support did families receive (pre-, during and post reintegration activities)? - How successful were the reintegration activities overall? According to your assessment, did the wellbeing of reintegrated children improve after DOVCU interventions? - In what way did their wellbeing improve? Was there any area of wellbeing that showed greater improvements than other? - Did all reintegrated children remain in their placements? - Do you know of any case where the child did not remain in his/her placement? What happen? 32 - What kind of actions did the project take in the cases where children did not show improved well￾being after they had been reintegrated - Do you know of any case where the reintegration of the child did not show improvements in the well-being status of the child? What happen? How did the project address this situation? - Do you know of any case where a reintegrated child returned to institutions or to the streets? What happen? How did the project address this situation? 21. For the children who have been successfully reintegrated with their families or placed in alternative family based care, what made the reintegration method successful and sustainable? Examples of information the evaluation seeks to explore: - What makes these reintegration methods sustainable? - For which children was impact most visible? - Why do you think reintegration efforts were more successful for some children? - What were the circumstances that influenced this? 38. For the children for whom reintegration was not successful, what were the challenges and how were those mitigated? Examples of information the evaluation seeks to explore: - How were risks assessed and monitored - If risk was identified, how was it reduced 39. To what extent was the project able to prevent child protection issues and human rights violations (e.g. child labour, violence against children, neglect, early marriage, etc.) and mitigate risks for creating greater vulnerability (e.g. child labour and any kind of harm)? Examples of information the evaluation seeks to explore: - If risks for any particular human rights violations were observed; - The reasons for those risks Reintegration of children from remand homes 40. Overall, how many children have been reintegrated or placed in alternative family based care from remand homes as a result of the project? Examples of information the evaluation seeks to explore: - Are there less children in remand homes now as compared to three years ago? - Do fewer children enter remand homes now as compared to three years ago? - Have the reasons for why children are placed in remand homes changed (e.g. now only for heavier crimes, or also petty crimes, behavioral or statutory offences) - Do more children leave a child care placement (outflow) now as compared to three years ago? - Do children more now than three years ago, leave remand homes quicker and easier or does it still happen that children miss their trials because of fuel shortages or inability to get to the court room for hearings? 41. What has been the experience of these cases of reintegration? Please talk both about the positive experience and the challenges that were faced in the process. 33 Examples of information the evaluation seeks to explore: - For the children who are going home from remand homes, do they have a plan for how reintegration should be done to ensure safe and permanent reintegration? - What was the experience in implementing this plan? Reintegration of children from the street: 42. Overall, how many children have been reintegrated or placed in alternative family based care from the street as a result of the project? Examples of information the evaluation seeks to explore: - If there is an overall reduction in the phenomenon of children recorded to be on the street in the districts where DOVCU operated 43. What has been the experience of these cases of reintegration? Please talk both about the positive experience and the challenges that were faced in the process. Examples of information the evaluation seeks to explore: - What was the experience of implementing the permanency plan – successes and challenges? - In how many cases was it not possible to implement the permanency plan, what were the challenges? - Now that the project is ending, for how long will children continue to be followed up by ChildFund and the project partners, by PSWOs, CDOs or other government staff? - What will allow for such follow-up after the project has ended (e.g. availability of sustained mobility of PSWOs and CDOs, institutionalized government guidelines for case management, reintegration and permanency planning etc.) Professional practices of PSWOs, CDOs and para-social workers 22. What changes in professional practices of PSWOs, CDOs and para-social workers do you think the DOVCU - project activities influenced: Examples of information the evaluation seeks to explore: - If PSWOs and CDOs are more regularly involved in alternative care cases, cases of children to be reintegrated from the street or from remand homes - Changes in the quality and use of standardized processes and forms for alternative care case assessments and reintegration - Regular follow-up to cases of risk for children (in CCIs, on the street, in remand homes, in families) - Improvements in the quality of case files for alternative care cases - Better regularity in follow-up in alternative care placements - Better regularity in PSWO and CDO monitoring of cases until reintegration - Better regularity and quality of inspections and reporting on CCIs and children placed in CCIs to central Ministry - Regular use of standardized referral protocols; - Better clarity among professionals on the trajectory of cases (e.g. referrals and admission practices). - Collaborative spirit between all professionals involved at district, community and service delivery level to first and foremost ensure prevention of separation, and as a last resort quality of placements of children into care. 34 Community child protection mechanisms and para-social workers 23. What changes in community child protection mechanisms and para-social workers do you think the DOVCU - project activities influenced: Examples of information the evaluation seeks to explore: - Improvements in understanding by community based child protection structures and para social workers of their key roles and contributions to prevent institutionalization and follow-up on child protection cases and cases of children who have been reintegrated. - Changes in awareness and skills to identify at risk children; - Changes in the frequency of referrals coming from these structures to PSWOs; - More proactive actions taken by these structures to solve the problems at their levels; - Proactive activities of these structures to make people aware of the existence of services that can help prevent separation pf children from their families. Services and supports provided through DOVCU - To what extent do you think the DOVCU project has responded to most common risk factors for separation (including for boys and girls) Examples of information the evaluation seeks to explore: - Was the support from the integrated package of interventions meeting the most urgent needs of children and families that the project targeted? - Which services/support offered by DOVCU have been the most useful and had the most effect for prevention? - What combinations of supports have been most useful to ensure child protection, and safe reintegration? - Were there, in your opinion, any missing elements in the support provided that should have been included in the support package to vulnerable families, in order to prevent the separation of children from these families? - To what extent did support and services offered to families need adaptation to the local context? Please give examples of how such adaptation was made. 24. Do you services and supports offered in DOVCU-districts would have the same effect if replicated / expanded in other parishes and/or districts in Uganda or are there any elements that would need better adaptation? Example of information we are interested to receive: - What kind of adaptation is suggested 25. To what extent do you feel that the DOVCU-project has contributed to put in place structures and processes that are sustainable beyond the project’s timeframe? Examples of information the evaluation seeks to explore: - What structures have been established / strengthened? - What processes and procedures have been changed ((e.g. supervision/monitoring of CCIs and of reintegrated children, case management for children in need of alternative care or at risk) - What makes these changes sustainable? 35 - What would be needed to make them sustainable? Lessons learned: 26. What are the specific lessons learned from this project that would be useful to document and share: Examples of information the evaluation seeks to explore: - Reflections on the overall design of the project and if (in retrospect) selection of intervention areas and activities were sound. - Things that worked well, or were a challenge in the project’s work with child care institutions. - Suggestions for how to overcome such challenges in any future project with similar objectives. - Things that worked well, or were a challenge in the project’s work with different professionals involved in gatekeeping and outreach to vulnerable families (e.g. para professionals how are they working together with probation and social welfare officers and what is motivating them in their work) - Things that worked well, or were a challenge in the project’s integrated package of support? - Things that worked well, or were a challenge in the project’s cash-benefit and income generating approaches. 36 Tool 5 – Guide for Key Informant Interview with Ministry of Local Government: Introduction: Interviewer to start with an introduction, which organization he/she is representing and in which context the interview is taking place: The interview is being carried out in the context of an end-line evaluation of a project (DOVCU) that has been implemented in the area of child care and de-institutionalization by MoGLSD and a consortium led by ChildFund (RETRAK, TPO and Child’s I Foundation). The project started in 2014 and was implemented for 3 years across 12 districts of Uganda. It had for overall objective to prevent family separation and reintegrate children without appropriate family care back into nurturing family environments. The project worked with a large number of stakeholders, including district level officials, alternative care panels, probation officers, community development officers, para-social workers, community child protection structures, health facilities, child care institutions, vulnerable families and children and with children in institutions, remand homes and children found to be living in street situations. With this interview, we want to establish your involvement into the project, what and how you think it has contributed to changes in the lives of the beneficiaries. What you will tell us in the interview will remain confidential. The information you share with us will be compiled in a report that will have more than 100 respondents. We may quote you, but it will not be possible to link quotes or information discussed in the report with any specific respondent. Notes and recordings from the discussion will remain with the evaluator and will not be shared with anybody else. The interview is expected to take 1 hour. Ask for permission to start the interview, and to record the interview. General questions: 1. In your opinion, what is the current level of visibility in Uganda of child care reform issues, need for de￾institutionalization, and need to address the situation of child homes? Examples of information we are interested in receiving: - Are the needs of children who are without a primary care giver a priority to the government? To the MoLG? - Not a priority? What are competing priorities? - How does the stakeholder ensure visibility and priority of child care reform and deinstitutionalization in the implementation of their own mandate? 2. Do you know about a project called DOVCU? If yes, what do you think about the relevance of it in the current context in Uganda: Examples of information we are interested in receiving: - Does the respondent know about this project, what its priorities are and what results it will contribute towards? - Was the respondent involved in any discussions related to the project? What was it about? - What do you think may be the contribution of this project to the child care reform in Uganda? 3. Are you aware of the Alternative Care framework? What are some of its priorities? Examples of information we are interested in receiving: - Ask the respondent to describe the priorities of the ACF 37 4. Within your mandate to monitor the compliance of districts with national policy, have there been any changes in the indicators you use to monitor compliance of districts with the ACF over the last three years? Examples of information we are interested in receiving: - Which indicators - Inspection forms 5. The project has been implemented in 12 districts across Uganda and has tested support and service package to prevent that children are separated from their families. In your opinion, have there been any changes in the capacity of various stakeholders at community and district levels that are enabling children to live in family care and preventing placements in institutional care? Examples of information we are interested in receiving: - Since the start of the DOVCU project, what actions have been taken, such as for example passing bylaws, budgeting for new services for vulnerable families, or children at risk in district plans? - Are there, in the district, any plans for how professionals, such as the PSWO or the CDOs can continue to provide oversight and reporting on CCIs after the end of the DOVCU project? - Are there, in the district, any plans for how this district should expand family support and family based care services that have proven to be successful for vulnerable families and children at risk? 6. To what extent do you think districts have the capacity to sustain efforts initiated by DOVCU now that the project is ending? If you think it unlikely, why? 7. Do district meet regularly with all organizations working on child protection (as started by the SUNRISE project)? Do they discuss alternative care and deinstitutionalization priorities? 8. Do the DOVCC and SOVCC committees discuss how to prevent the separation of children from their families, and how to support the reintegration of children with their families? 38 IV. Interviews to be conducted at district level: Tool 6 – Checklist for Chief Administrative Officer District (Courtesy meeting) / Chairperson from District Council Introduction: Interviewer to start with an introduction, which organization he/she is representing and in which context the interview is taking place: The interview is being carried out in the context of an end-line evaluation of a project (DOVCU) that has been implemented in the area of child care and de-institutionalization by MoGLSD and a consortium led by ChildFund (RETRAK, TPO and Child’s I Foundation). The project started in 2014 and was implemented for 3 years across 12 districts of Uganda. It had for overall objective to prevent family separation and reintegrate children without appropriate family care back into nurturing family environments. The project worked with a large number of stakeholders, including district level officials, alternative care panels, probation officers, community development officers, para-social workers, community child protection structures, health facilities, child care institutions, vulnerable families and children and with children in institutions, remand homes and children found to be living in street situations. With this interview, we want to establish your involvement into the project, what and how you think it has contributed to changes in the lives of the beneficiaries. What you will tell us in the interview will remain confidential. The information you share with us will be compiled in a report that will have more than 100 respondents. We may quote you, but it will not be possible to link quotes or information discussed in the report with any specific respondent. Notes and recordings from the discussion will remain with the evaluator and will not be shared with anybody else. The interview is expected to take 1 hour. Ask for permission to start the interview, and to record the interview. 1. Have you ever heard about the Alternative Care Framework? What are its priorities? 2. What do you think about these priorities and the DOVCU objectives: prevent family separation and reintegrate children without appropriate family care back into nurturing family environments? Are these important to be addressed in this district? 3. How has the district been involved in the DOVCU-project implementation? What activities was the district involved in? What went well and what were challenges in the project implementation and monitoring? Examples of information the evaluation seeks to explore: - What briefings, trainings and other activities that they were involved in, through the DOVCU project, what were the strengths and what were the weaknesses of these? - How many staff in this office were trained? On what topics were the trainings? - Particularly, what new did you learn from trainings? Have you changed any of your processes / practices as a result of these trainings? What changes did you introduce? - What were the strengths and what were the weaknesses of these trainings? - What activities and strategies used by the project have you found to be the most useful, to for example change the way you work (assessments, coaching, training, technical assistance activities etc.) 4. How would you describe your overall experience of being involved in the project? Examples of information the evaluation seeks to explore: 39 - Were the human resources (e.g. social work capacity, staff providing training and coaching activities), and other resources invested in the project such as coordination mechanisms (e.g. TAC and ACPs, social workers and project coordinators), time and other resources sufficient for there to be a lasting change in professional practices & were these resources complementary to what you can mobilize themselves. - Is there any activity that you think should have been included that was not included? Did you provide this feedback to the DOVCU-project staff? What did they do? 5. How would you assess the capacity in this district to prevent institutionalization and follow-up on child protection cases and cases of children who have been reintegrated? Examples of information the evaluation seeks to explore: - Capacity and interest to plan and budget for new services - Capacity of PSWOs and CDOs to monitor CCIs and follow up with reintegrated children 6. In your opinion, to what extent, and which systemic changes did the project bring about at district level that are enabling children to live in family care and preventing placements in institutional care? Examples of information the evaluation seeks to explore: - If capacity increased among district authorities and OVC committees to plan for services that can support child care reform efforts, and provide support to families and children - Capacity strengthening of Probation and Social Welfare Officers (PSWOs) and CDOs - Creation of gatekeeping mechanisms such as Alternative Care Panels (ACPs) or Teams Around the Child (TAC). - Changes in practices of Remand homes - Changes in practices of Child Care Institutions - Strengthened capacity of Para-social workers to address alternative care and issues of separation and risk of separation for children - Strengthened capacity of Community child protection structures to work on alternative care issues of separation and risk of separation for children 7. Did this district establish an Alternative Care Panel and/or a Team Around the Child? If yes, how have these functioned? Examples of information the evaluation seeks to explore: - How were member selected - Who are the members - Regularity of meetings; - Level of participation of core members; - What were the core functions - Compliance with core functions; - The extent to which members knew and used foster care and adoption guidelines; - Coverage of various types of cases; - Regularity of reviews of placements and movements of children. - Any challenges and lessons learned from the work of ACPs and TACs. 8. In your opinion, what are the changes in child care institutions (CCIs) in this district as a result of DOVCU-project activities: Examples of information the evaluation seeks to explore: 40 - How many child care institutions in the DOVCU-districts district have been inspected? - For those CCIs that received recommendations for improvements to be made, how did the project follow-up with these CCIs? - How many developed an improvement plan? - What was the experience and lessons learned regarding the implementation of improvement plans? - Who followed up with CCIs on the improvements that were needed, and was there any re￾inspection of CCIs after some time? Who did that re-inspection? - Of the CCIs that were inspected by MoGLSD and their team, how many received a licence to operate? - What happened after CCIs received a license to operate, is there for example any obligation to regularly report on activities in the CCI to MoGLSD, as a requirement to maintain the licence? 9. Were there any CCIs that closed completely? What do you think led to their closure? Examples of information the evaluation seeks to explore: - In which districts did this happen and what were the names of these CCIs? - How were such closures planned and monitored? - What happened to children in the CCIs that closed? - Who was involved in the oversight and monitoring of such closures? - Were there any CCIs that closed spontaneously without oversight by the project or by MoGLSD? - In which districts did this happen and what were the names of these CCIs? - Is it known what happened to children in the CCIs that closed? - What did the project do, and what did MoGLSD do when they learned about these self-closures? - What attempts did the project make, or MoGLSD and district staff make to trace and check on the children from the CCIs that self-closed? 10. In your opinion, what are the changes in remand homes as a result of DOVCU-project activities?: Examples of information the evaluation seeks to explore: - The extent to which DOVCU has contributed to improvements in knowledge, skills and utilization by remand home staff of guidelines and tools (standard operating procedures) which project trained them on; - Improvements in the level of compliance with decisions on children (e.g. if children who have served their sentence are allowed to go home); - Availability of strategic partnerships and alliances for follow-up on cases of reintegration from remand homes. 11. In your opinion, what changes in professional practices of PSWOs, CDOs and para-social workers do you think the DOVCU - project activities influenced: Examples of information the evaluation seeks to explore: - If PSWOs and CDOs are more regularly involved in alternative care cases, cases of children to be reintegrated from the street or from remand homes - Changes in the quality and use of standardized processes and forms for alternative care case assessments and reintegration - Regular follow-up to cases of risk for children (in CCIs, on the street, in remand homes, in families) - Improvements in the quality of case files for alternative care cases - Better regularity in follow-up in alternative care placements - Better regularity in PSWO and CDO monitoring of cases until reintegration 41 - Better regularity and quality of inspections and reporting on CCIs and children placed in CCIs to central Ministry - Regular use of standardized referral protocols; - Better clarity among professionals on the trajectory of cases (e.g. referrals and admission practices). - Collaborative spirit between all professionals involved at district, community and service delivery level to first and foremost ensure prevention of separation, and as a last resort quality of placements of children into care. 12. What changes in community child protection mechanisms and para-social workers do you think the DOVCU - project activities influenced: Examples of information the evaluation seeks to explore: - Improvements in understanding by community based child protection structures and para social workers of their key roles and contributions to prevent institutionalization and follow-up on child protection cases and cases of children who have been reintegrated. - Changes in awareness and skills to identify at risk children; - Changes in the frequency of referrals coming from these structures to PSWOs; - More proactive actions taken by these structures to solve the problems at their levels; - Proactive activities of these structures to make people aware of the existence of services that can help prevent separation pf children from their families. 13. Since the start of the DOVCU project, what actions have you taken to put in place structures and processes that can ensure that prevention of separation and reintegration of children into families remains a priority also in the future? Examples of information the evaluation seeks to explore: - What structures have been established / strengthened? - Have any bylaws been passed - Has the proportion of the district budget that is allocated for services for vulnerable families, or children at risk in increased? What is the current proportion and what was the proportion in 2014? - Are there, in the district, any plans for how professionals, such as the PSWO or the CDOs can continue to provide oversight and reporting on CCIs after the end of the DOVCU project? - Are there, in the district, any plans for how this district should expand family support and family based care services that have proven to be successful for vulnerable families and children at risk? - To what extent do you think this district has capacity to sustain efforts initiated by DOVCU beyond the project’s timeframe? - Do you, in this district continue to meet regularly with all organizations working on child protection (as started by SUNRISE)? Are alternative care and CCI issues also discussed there? - DOVCC and SOVCC – do these committees now also discuss how to prevent the separation of children from their families, and how to support the reintegration of children with their families? 14. What are the specific lessons learned from this project that would be useful to document and share: Examples of information the evaluation seeks to explore: - Reflections on the overall design of the project and if (in retrospect) selection of intervention areas and activities were sound. - Things that worked well, or were a challenge in the project’s work with child care institutions. - Suggestions for how to overcome such challenges in any future project with similar objectives. 42 - Things that worked well, or were a challenge in the project’s work with different professionals involved in gatekeeping and outreach to vulnerable families (e.g. para professionals how are they working together with probation and social welfare officers and what is motivating them in their work) - Things that worked well, or were a challenge in the project’s integrated package of support? - Things that worked well, or were a challenge in the project’s cash-benefit and income generating approaches. 43 Tool 7 – Guide for Key Informant Interviews with District OVC committee: Introduction: Interviewer to start with an introduction, which organization he/she is representing and in which context the interview is taking place: The interview is being carried out in the context of an end-line evaluation of a project (DOVCU) that has been implemented in the area of child care and de-institutionalization by MoGLSD and a consortium led by ChildFund (RETRAK, TPO and Child’s I Foundation). The project started in 2014 and was implemented for 3 years across 12 districts of Uganda. It had for overall objective to prevent family separation and reintegrate children without appropriate family care back into nurturing family environments. The project worked with a large number of stakeholders, including district level officials, alternative care panels, probation officers, community development officers, para-social workers, community child protection structures, health facilities, child care institutions, vulnerable families and children and with children in institutions, remand homes and children found to be living in street situations. With this interview, we want to establish your involvement into the project, what and how you think it has contributed to changes in the lives of the beneficiaries. What you will tell us in the interview will remain confidential. The information you share with us will be compiled in a report that will have more than 100 respondents. We may quote you, but it will not be possible to link quotes or information discussed in the report with any specific respondent. Notes and recordings from the discussion will remain with the evaluator and will not be shared with anybody else. The interview is expected to take 1 hour. Ask for permission to start the interview, and to record the interview. 1. Have you ever heard about the Alternative Care Framework and its priorities? 2. What do you think about these priorities and the DOVCU objectives: prevent family separation and reintegrate children without appropriate family care back into nurturing family environments? Are these important to be addressed in this district? 3. Please give an introduction on the involvement of the district in the DOVCU-project implementation, the types of activities was the district involved in, what went well and what were challenges in the project implementation and monitoring. Examples of information the evaluation seeks to explore: - What briefings, trainings and other activities that they were involved in, through the DOVCU project, what were the strengths and what were the weaknesses of these? - How many staff in this office were trained? On what topics were the trainings? - Particularly, what new did you learn from trainings? Have you changed any of your processes / practices as a result of these trainings? What changes did you introduce? - What were the strengths and what were the weaknesses of these trainings? - What activities and strategies used by the project have you found to be the most useful, to for example change the way you work (assessments, coaching, training, technical assistance activities etc.) 4. How would you describe your overall experience of being involved in the project? Examples of information the evaluation seeks to explore: - Were the human resources (e.g. social work capacity, staff providing training and coaching activities), and other resources invested in the project such as coordination mechanisms (e.g. TAC 44 and ACPs, social workers and project coordinators), time and other resources sufficient for there to be a lasting change in professional practices & were these resources complementary to what you can mobilize themselves. - Is there any activity that you think should have been included that was not included? Did you provide this feedback to the DOVCU-project staff? What did they do? 5. How would you assess the capacity in this district to prevent institutionalization and follow-up on child protection cases and cases of children who have been reintegrated? Examples of information the evaluation seeks to explore: - Capacity and interest to plan and budget for new services - Capacity of PSWOs and CDOs to monitor CCIs and follow up with reintegrated children 6. According to you, to what extent, and which systemic changes did the project bring about at district levels that are enabling children to live in family care and preventing inappropriate placements in institutional care? Examples of information the evaluation seeks to explore: - If capacity increased among district authorities and OVC committees to plan for services that can support child care reform efforts, and provide support to families and children - Capacity strengthening of Probation and Social Welfare Officers (PSWOs) and CDOs - Creation of gatekeeping mechanisms such as Alternative Care Panels (ACPs) or Teams Around the Child (TAC). - Changes in practices of Remand homes - Changes in practices of Child Care Institutions - Strengthened capacity of Para-social workers to address alternative care and issues of separation and risk of separation for children - Strengthened capacity of Community child protection structures to work on alternative care issues of separation and risk of separation for children 7. What changes in community child protection mechanisms and para-social workers do you think the DOVCU - project activities influenced: Examples of information the evaluation seeks to explore: - Improvements in understanding by community based child protection structures and para social workers of their key roles and contributions to prevent institutionalization and follow-up on child protection cases and cases of children who have been reintegrated. - Changes in awareness and skills to identify at risk children; - Changes in the frequency of referrals coming from these structures to PSWOs; - More proactive actions taken by these structures to solve the problems at their levels; - Proactive activities of these structures to make people aware of the existence of services that can help prevent separation pf children from their families. 8. Since the start of the DOVCU project, what actions have you taken to put in place structures and processes that can ensure that prevention of separation and reintegration of children into families remains a priority also in the future? Examples of information the evaluation seeks to explore: - What structures have been established / strengthened? 45 - Have any bylaws been passed - Has the proportion of the district budget that is allocated for services for vulnerable families, or children at risk in increased? What is the current proportion and what was the proportion in 2014? - Are there, in the district, any plans for how professionals, such as the PSWO or the CDOs can continue to provide oversight and reporting on CCIs after the end of the DOVCU project? - Are there, in the district, any plans for how this district should expand family support and family based care services that have proven to be successful for vulnerable families and children at risk? - To what extent do you think this district has capacity to sustain efforts initiated by DOVCU beyond the project’s timeframe? - Do you, in this district continue to meet regularly with all organizations working on child protection (as started by SUNRISE)? Are alternative care and CCI issues also discussed there? - DOVCC and SOVCC – do these committees now also discuss how to prevent the separation of children from their families, and how to support the reintegration of children with their families? 46 Tool 8 - Guide for Key Informant Interviews with District Community Development Officer Introduction: Interviewer to start with an introduction, which organization he/she is representing and in which context the interview is taking place: The interview is being carried out in the context of an end-line evaluation of a project (DOVCU) that has been implemented in the area of child care and de-institutionalization by MoGLSD and a consortium led by ChildFund (RETRAK, TPO and Child’s I Foundation). The project started in 2014 and was implemented for 3 years across 12 districts of Uganda. It had for overall objective to prevent family separation and reintegrate children without appropriate family care back into nurturing family environments. The project worked with a large number of stakeholders, including district level officials, alternative care panels, probation officers, community development officers, para-social workers, community child protection structures, health facilities, child care institutions, vulnerable families and children and with children in institutions, remand homes and children found to be living in street situations. With this interview, we want to establish your involvement into the project, what and how you think it has contributed to changes in the lives of the beneficiaries. What you will tell us in the interview will remain confidential. The information you share with us will be compiled in a report that will have more than 100 respondents. We may quote you, but it will not be possible to link quotes or information discussed in the report with any specific respondent. Notes and recordings from the discussion will remain with the evaluator and will not be shared with anybody else. The interview is expected to take 1 hour. Ask for permission to start the interview, and to record the interview. Project implementation and monitoring 1. Please give an introduction of your involvement in the DOVCU-project implementation, the types of activities you were involved in, what went well and what were challenges in the project implementation and monitoring. Examples of information the evaluation seeks to explore: - What briefings, trainings and other activities that they were involved in, through the DOVCU project, what were the strengths and what were the weaknesses of these? - How many staff in this office were trained? On what topics were the trainings? - Particularly, what new did you learn from trainings? Have you changed any of your processes / practices as a result of these trainings? What changes did you introduce? - What were the strengths and what were the weaknesses of these trainings? - What activities and strategies used by the project have you found to be the most useful, to for example change the way you work (assessments, coaching, training, technical assistance activities etc.) 2. From this project, what have you understood as being your role in supporting the implementing the Alternative Care Framework? Examples of information the evaluation seeks to explore: - Their described role in identification and referral of cases of children in need of care and protection - Challenges or opportunities in their role to implement this policy 47 3. According to you, to what extent, and which systemic changes did the project bring about at community and district levels that are enabling children to live in family care and preventing inappropriate placements in institutional care? Examples of information the evaluation seeks to explore: - If capacity increased among district authorities and OVC committees to plan for services that can support child care reform efforts, and provide support to families and children - Capacity strengthening of Probation and Social Welfare Officers (PSWOs) and CDOs - Creation of gatekeeping mechanisms such as Alternative Care Panels (ACPs) or Teams Around the Child (TAC). - Changes in practices of Remand homes - Changes in practices of Child Care Institutions - Strengthened capacity of Para-social workers to address alternative care and issues of separation and risk of separation for children - Strengthened capacity of Community child protection structures to work on alternative care issues of separation and risk of separation for children 4. We are now interested to learn more about the Alternative Care Panels and Teams Around the Child, please let us know in how many districts Alternative Care Panels and Teams Around the Child were established and how these have functioned: Examples of information the evaluation seeks to explore: - How were member selected - Who are the members - Regularity of meetings; - Level of participation of core members; - What were the core functions - Compliance with core functions; - The extent to which members knew and used foster care and adoption guidelines; - Coverage of various types of cases; - Regularity of reviews of placements and movements of children. - Any challenges and lessons learned from the work of ACPs and TACs. 5. In your opinion, what are the changes in child care institutions (CCIs) as a result of DOVCU￾project activities: Examples of information the evaluation seeks to explore: - How many child care institutions in the DOVCU-districts district have been inspected? - For those CCIs that received recommendations for improvements to be made, how did the project follow-up with these CCIs? - How many developed an improvement plan? - What was the experience and lessons learned regarding the implementation of improvement plans? - Who followed up with CCIs on the improvements that were needed, and was there any re￾inspection of CCIs after some time? Who did that re-inspection? - Of the CCIs that were inspected by MoGLSD and their team, how many received a licence to operate? - What happened after CCIs received a license to operate, is there for example any obligation to regularly report on activities in the CCI to MoGLSD, as a requirement to maintain the licence? 48 6. Were there any CCIs that closed completely? What do you think led to their closure? Examples of information the evaluation seeks to explore: - In which districts did this happen and what were the names of these CCIs? - How were such closures planned and monitored? - What happened to children in the CCIs that closed? - Who was involved in the oversight and monitoring of such closures? - Were there any CCIs that closed spontaneously without oversight by the project or by MoGLSD? - In which districts did this happen and what were the names of these CCIs? - Is it known what happened to children in the CCIs that closed? - What did the project do, and what did MoGLSD do when they learned about these self-closures? - What attempts did the project make, or MoGLSD and district staff make to trace and check on the children from the CCIs that self-closed? 7. In the DOVCU-districts how many of all the CCIs that were assessed and that the project worked with can now report to have a reduced number of children as compared to baseline? Examples of information the evaluation seeks to explore: - The extent to which the DOVCU-project has influences the numbers of children who are placed in CCIs. The effects can take different forms as below, and we are interested to understand what kind of dynamics that the DOVCU project has contributed to:: - Are there less children overall in CCI now as compared to three years ago? - Do fewer children enter CCIs now as compared to three years ago? - Do more children leave a child care placement now as compared to three years ago? - Do children on average stay shorter periods in CCIs as compared to three years ago? 8. Regarding the practices of child care institutions, were there any observed and documented changes in how they operate as a result of the DOVCU-project? Examples of information the evaluation seeks to explore: - Changes in governance and management culture (e.g. operating with formal registration, policies on having employed qualified social workers, overall human resource policies, and minimum qualifications for staff working in the CCI, policies for child protection, adequate descriptions of the types of care facility and services provided there etc.) - Changes in funding culture (e.g. fundraising for provision of residential care to individual children only or fundraising also to support children in their community) - Referral and admission practices (e.g. availability of care orders for all children staying at the CCI, criteria for admission, whether there are more streamlined referral and admission processes going through care planning and placement reviews with involvement of PSWOs and multi-disciplinary panels at district level) - Practices, policies and guidelines for in case review and reintegration - Policies for family visits and staying in touch with culture and community of children who are placed - Existence of standards of services provided to children in CCIs Changes in remand home practices (question valid only in Mbale and Gulu districts) 9. In your opinion, what are the changes in remand homes as a result of DOVCU-project activities: Examples of information the evaluation seeks to explore: 49 - The extent to which DOVCU has contributed to improvements in knowledge, skills and utilization by remand home staff of guidelines and tools (standard operating procedures) which project trained them on; - Improvements in the level of compliance with decisions on children (e.g. if children who have served their sentence are allowed to go home); - Availability of strategic partnerships and alliances for follow-up on cases of reintegration from remand homes. Professional practices of PSWOs, CDOs and para-social workers 10. What changes in professional practices of PSWOs and CDOs do you think the DOVCU - project activities influenced: Examples of information the evaluation seeks to explore: - If PSWOs and CDOs are more regularly involved in alternative care cases, cases of children to be reintegrated from the street or from remand homes - Changes in the quality and use of standardized processes and forms for alternative care case assessments and reintegration - Regular follow-up to cases of risk for children (in CCIs, on the street, in remand homes, in families) - Improvements in the quality of case files for alternative care cases - Better regularity in follow-up in alternative care placements - Better regularity in PSWO and CDO monitoring of cases until reintegration - Better regularity and quality of inspections and reporting on CCIs and children placed in CCIs to central Ministry - Regular use of standardized referral protocols; - Better clarity among professionals on the trajectory of cases (e.g. referrals and admission practices). - Collaborative spirit between all professionals involved at district, community and service delivery level to first and foremost ensure prevention of separation, and as a last resort quality of placements of children into care. Community child protection mechanisms and para-social workers 11. What changes in community child protection mechanisms and para-social workers do you think the DOVCU - project activities influenced: Examples of information the evaluation seeks to explore: - Improvements in understanding by community based child protection structures and para social workers of their key roles and contributions to prevent institutionalization and follow-up on child protection cases and cases of children who have been reintegrated. - Changes in awareness and skills to identify at risk children; - Changes in the frequency of referrals coming from these structures to PSWOs; - More proactive actions taken by these structures to solve the problems at their levels; - Proactive activities of these structures to make people aware of the existence of services that can help prevent separation pf children from their families. 12. What changes in the collaboration between different professionals has the DOVCU-project influenced? 50 Examples of information the evaluation seeks to explore: - If the PSWO, as part of the DOVCU-activities, collaborated more or differently with CDOs and psychosocial workers with community-based partners to improve identification of children outside of family care - How they worked together with these professionals towards securing placements in family based care, and how they jointly monitored children's wellbeing? - How often did such collaboration take place and were you, through this collaboration, able to better identify and refer children living outside of family care? - What were the strengths in this collaborative work and in what way did DOVCU change how you do such work in the district - What are the challenges in collaboration with other professionals? - How did DOVCU project help you to address these challenges? 13. How do you assess the sustainability of changes that the project has influenced, or of the project structures that have been created (such as ACPs and TACs) Examples of information the evaluation seeks to explore: - Is there capacity to sustain efforts (e.g. supervision/monitoring of CCIs and of reintegrated children, case management for children in need of alternative care or at risk) beyond the project’s timeframe? - What capacity, in particular, do you feel that the DOVCU-project contributed to strengthening? - What additional needs do you have in order to sustain efforts, that DOVCU-project did not address? Lessons learned: 14. What are the specific lessons learned from this project that would be useful to document and share: Examples of information the evaluation seeks to explore: - Reflections on the overall design of the project and if (in retrospect) selection of intervention areas and activities were sound. - Things that worked well, or were a challenge in the project’s work with child care institutions. - Suggestions for how to overcome such challenges in any future project with similar objectives. - Things that worked well, or were a challenge in the project’s work with different professionals involved in gatekeeping and outreach to vulnerable families (e.g. para professionals how are they working together with probation and social welfare officers and what is motivating them in their work) - Things that worked well, or were a challenge in the project’s integrated package of support? - Things that worked well, or were a challenge in the project’s cash-benefit and income generating approaches. 51 Tool 9 – Guide for Key Informant Interviews with PSWOs: Introduction: Interviewer to start with an introduction, which organization he/she is representing and in which context the interview is taking place: The interview is being carried out in the context of an end-line evaluation of a project (DOVCU) that has been implemented in the area of child care and de-institutionalization by MoGLSD and a consortium led by ChildFund (RETRAK, TPO and Child’s I Foundation). The project started in 2014 and was implemented for 3 years across 12 districts of Uganda. It had for overall objective to prevent family separation and reintegrate children without appropriate family care back into nurturing family environments. The project worked with a large number of stakeholders, including district level officials, alternative care panels, probation officers, community development officers, para-social workers, community child protection structures, health facilities, child care institutions, vulnerable families and children and with children in institutions, remand homes and children found to be living in street situations. With this interview, we want to establish your involvement into the project, what and how you think it has contributed to changes in the lives of the beneficiaries. What you will tell us in the interview will remain confidential. The information you share with us will be compiled in a report that will have more than 100 respondents. We may quote you, but it will not be possible to link quotes or information discussed in the report with any specific respondent. Notes and recordings from the discussion will remain with the evaluator and will not be shared with anybody else. The interview is expected to take 1 hour. Ask for permission to start the interview, and to record the interview. Project implementation and monitoring 15. Please give an introduction of your involvement in the DOVCU-project implementation, the types of activities you were involved in, what went well and what were challenges in the project implementation and monitoring. Examples of information the evaluation seeks to explore: - What briefings, trainings and other activities that they were involved in, through the DOVCU project, what were the strengths and what were the weaknesses of these? - How many staff in this office were trained? On what topics were the trainings? - Particularly, what new did you learn from trainings? Have you changed any of your processes / practices as a result of these trainings? What changes did you introduce? - What were the strengths and what were the weaknesses of these trainings? - What activities and strategies used by the project have you found to be the most useful, to for example change the way you work (assessments, coaching, training, technical assistance activities etc.) 16. From this project, what have you understood as being your role in supporting the implementing the Alternative Care Framework? Examples of information the evaluation seeks to explore: - Their described role in identification and referral of cases of children in need of care and protection - Challenges or opportunities in their role to implement this policy 52 Involvement and experience of work with child care institutions and reintegration of children from CCIs 17. Who was a member of the team that carries out monitoring of CCIs? Were you involved and in what way were you involved? Examples of information the evaluation seeks to explore: - How often have they been monitoring CCIs. - What specifically were they monitoring in the CCIs - How did they document any observations from that monitoring - What were the main issues, strengths and challenges that have come out from that monitoring 18. In your opinion, what are the changes in child care institutions (CCIs) as a result of DOVCU￾project activities: Examples of information the evaluation seeks to explore: - How many child care institutions in the DOVCU-districts district have been inspected? - For those CCIs that received recommendations for improvements to be made, how did the project follow-up with these CCIs? - How many developed an improvement plan? - What was the experience and lessons learned regarding the implementation of improvement plans? - Who followed up with CCIs on the improvements that were needed, and was there any re￾inspection of CCIs after some time? Who did that re-inspection? - Of the CCIs that were inspected by MoGLSD and their team, how many received a licence to operate? - What happened after CCIs received a license to operate, is there for example any obligation to regularly report on activities in the CCI to MoGLSD, as a requirement to maintain the licence? 19. Were there any CCIs that closed completely? What do you think led to their closure? Examples of information the evaluation seeks to explore: - In which districts did this happen and what were the names of these CCIs? - How were such closures planned and monitored? - What happened to children in the CCIs that closed? - Who was involved in the oversight and monitoring of such closures? - Were there any CCIs that closed spontaneously without oversight by the project or by MoGLSD? - In which districts did this happen and what were the names of these CCIs? - Is it known what happened to children in the CCIs that closed? - What did the project do, and what did MoGLSD do when they learned about these self-closures? - What attempts did the project make, or MoGLSD and district staff make to trace and check on the children from the CCIs that self-closed? 20. In the DOVCU-districts how many of all the CCIs that were assessed and that the project worked with can now report to have a reduced number of children as compared to baseline? Examples of information the evaluation seeks to explore: - The extent to which the DOVCU-project has influences the numbers of children who are placed in CCIs. The effects can take different forms as below, and we are interested to understand what kind of dynamics that the DOVCU project has contributed to:: 53 - Are there less children overall in CCI now as compared to three years ago? - Do fewer children enter CCIs now as compared to three years ago? - Do more children leave a child care placement now as compared to three years ago? - Do children on average stay shorter periods in CCIs as compared to three years ago? 21. Regarding the practices of child care institutions, were there any observed and documented changes in how they operate as a result of the DOVCU-project? Examples of information the evaluation seeks to explore: - Changes in governance and management culture (e.g. operating with formal registration, policies on having employed qualified social workers, overall human resource policies, and minimum qualifications for staff working in the CCI, policies for child protection, adequate descriptions of the types of care facility and services provided there etc.) - Changes in funding culture (e.g. fundraising for provision of residential care to individual children only or fundraising also to support children in their community) - Referral and admission practices (e.g. availability of care orders for all children staying at the CCI, criteria for admission, whether there are more streamlined referral and admission processes going through care planning and placement reviews with involvement of PSWOs and multi-disciplinary panels at district level) - Practices, policies and guidelines for in case review and reintegration - Policies for family visits and staying in touch with culture and community of children who are placed - Existence of standards of services provided to children in CCIs 22. Have you been involved in activities related to reintegration of children from child care institutions into families? Examples of information the evaluation seeks to explore: - How they did family tracing activities - Preparations for reintegration that they were involvement in (preparations with the child and preparations with the family) - Strengths in the way these processes were implemented - Weaknesses and challenges in the way these processes were implemented - As compared to pre-2014, how their professional practices and involvement in reintegration activities has changed, been strengthened 23. For children to be reintegrated from child care institutions, how were children to be reintegrated selected? Examples of information the evaluation seeks to explore: - What profile did the children have, who ultimately got reintegrated? - Who decided which children should be reintegrated? - How was the decision made? - Were these children the most vulnerable or those most in need of being reintegrated? 24. Overall, how many children have been reintegrated or placed in alternative family based care from CCIs and what has been the overall experience of these reintegration cases? Please talk both about the positive experience and the challenges that were faced. Examples of information the evaluation seeks to explore: 54 - If in addition to the quoted number, how many cases of children leaving CCIs the project is thought to have influenced, even if not all these cases were followed by the project? - If the project reached its targets, what were the facilitating factors ensuring the target were reached? - If targets were not reached, what were the challenges in reaching targets? 25. Of the children that the DOVCU project reintegrated do all children have a permanency plan and what has been the experience of implementing the permanency plan? Examples of information the evaluation seeks to explore: - What were successes and challenges? - In how many cases was it not possible to implement the permanency plan, what were the challenges? - Now that the project is ending, for how long will children continue to be followed up by ChildFund and the project partners, by PSWOs, CDOs or other government staff? - What will allow for such follow-up after the project has ended (e.g. availability of sustained mobility of PSWOs and CDOs, institutionalized government guidelines for case management, reintegration and permanency planning etc.) 26. What has been the effect of the reintegration of children on the children and the families where children were received? Examples of information the evaluation seeks to explore: - What kind of support did children receive (pre-, during and post reintegration activities)? - What kind of support did families receive (pre-, during and post reintegration activities)? - How successful were the reintegration activities overall? According to your assessment, did the wellbeing of reintegrated children improve after DOVCU interventions? - In what way did their wellbeing improve? Was there any area of wellbeing that showed greater improvements than other? - Did all reintegrated children remain in their placements? - Do you know of any case where the child did not remain in his/her placement? What happen? - What kind of actions did the project take in the cases where children did not show improved well￾being after they had been reintegrated - Do you know of any case where the reintegration of the child did not show improvements in the well-being status of the child? What happen? How did the project address this situation? - Do you know of any case where a reintegrated child returned to institutions or to the streets? What happen? How did the project address this situation? 27. For the children who have been successfully reintegrated with their families or placed in alternative family based care, what made the reintegration method successful and sustainable? Examples of information the evaluation seeks to explore: - What makes these reintegration methods sustainable? - For which children was impact most visible? - Why do you think reintegration efforts were more successful for some children? - What were the circumstances that influenced this? 28. For the children for whom reintegration was not successful, what were the challenges and how were those mitigated? 55 Examples of information the evaluation seeks to explore: - How were risks assessed and monitored - If risk was identified, how was it reduced 29. To what extent was the project able to prevent child protection issues and human rights violations (e.g. child labour, violence against children, neglect, early marriage, etc.) and mitigate risks for creating greater vulnerability (e.g. child labour and any kind of harm)? Examples of information the evaluation seeks to explore: - If risks for any particular human rights violations were observed; - The reasons for those risks Involvement and experience of work with children to be reintegrated from remand homes 30. Were you also involved in working with remand homes and children from remand homes to support reintegration activities of these children? Examples of information the evaluation seeks to explore: - What activities the stakeholder was involved in - How they did family tracing activities - Preparations for reintegration that they were involvement in (preparations with the child and preparations with the family) - Strengths in the way these processes were implemented - Weaknesses and challenges in the way these processes were implemented - As compared to pre-2014, how their professional practices and involvement in reintegration activities has changed, been strengthened 31. Overall, how many children have been reintegrated or placed in alternative family based care from remand homes as a result of DOVCU activities and what has been the experience of these cases of reintegration? Please talk both about the positive experience and the challenges that were faced in the process. Examples of information we are interested in: - Are there less children in remand homes now as compared to three years ago? - Do fewer children enter remand homes now as compared to three years ago? - Have the reasons for why children are placed in remand homes changed (e.g. now only for heavier crimes, or also petty crimes, behavioural or statutory offences) - Do more children leave a child care placement (outflow) now as compared to three years ago? - Do children more now than three years ago, leave remand homes quicker and easier or does it still happen that children miss their trials because of fuel shortages or inability to get to the court room for hearings? - For the children who are going home from remand homes, do they have a plan for how reintegration should be done to ensure safe and permanent reintegration? 32. We are also interested to know what are the changes in remand homes as a result of DOVCU￾project activities: Examples of information the evaluation seeks to explore: 56 - The extent to which DOVCU has contributed to improvements in knowledge, skills and utilization by remand home staff of guidelines and tools (standard operating procedures) which project trained them on; - Improvements in the level of compliance with decisions on children (e.g. if children who have served their sentence are allowed to go home); - Availability of strategic partnerships and alliances for follow-up on cases of reintegration from remand homes. Reintegration of children from the street: 33. Were you also involved in working with remand homes and children from remand homes to support reintegration activities of these children? This question is only for Mbale district Examples of information the evaluation seeks to explore: - In what way was the stakeholder involved - How they did family tracing activities - Preparations for reintegration that they were involvement in (preparations with the child and preparations with the family) - Strengths in the way these processes were implemented - Weaknesses and challenges in the way these processes were implemented - As compared to pre-2014, how their professional practices and involvement in reintegration activities has changed, been strengthened 34. Overall, how many children have been reintegrated or placed in alternative family based care from the street as a result of DOVCU activities and what has been the experience of these cases of reintegration? Please talk both about the positive experience and the challenges that were faced in the process. This question is only for Mbale district Examples of information that we are interested to receive: - If there is an overall reduction in the phenomenon of children recorded to be on the street in the districts where DOVCU operated - What was the experience of implementing the permanency plan – successes and challenges? - In how many cases was it not possible to implement the permanency plan, what were the challenges? - Now that the project is ending, for how long will children continue to be followed up by ChildFund and the project partners, by PSWOs, CDOs or other government staff? - What will allow for such follow-up after the project has ended (e.g. availability of sustained mobility of PSWOs and CDOs, institutionalized government guidelines for case management, reintegration and permanency planning etc.) Beneficiaries: vulnerable families targeted for prevention 35. Let’s talk about prevention activities supported by the DOVCU project to support vulnerable families in order to prevent the separation of children from families, in what way were you involved in those activities and what was the experience: Examples of information the evaluation seeks to explore: - The types of activities the stakeholder was involved in - Overall experience and assessment of whether these activities were useful and successful 57 36. Please describe how families to be targeted for prevention support were targeted. Examples of information the evaluation seeks to explore: - Was the targeting of the households that received support from the DOVCU project appropriate? - Did the project reach those families where children were the most at risk of separation? - If yes, what specifically made the project successful in reaching those most at risk? If no, what were the challenges? - Were activities to support families and children who were reintegrated tailored to the specific needs of each family / child? If yes, what were the strengths? If no, what were the challenges? 37. What has been the effect of the DOVCU-project’s interventions in the lives of the families that were targeted for prevention activities and who received support from the project? Examples of information the evaluation seeks to explore: - What kind of support packages did families receive in the end? - Who was targeted? - How did the project ensure to target the most vulnerable families? Was targeting successful or were there any cases, for example, were the wrong beneficiary was selected? - How successful were the various interventions in preventing separation of children at risk from their families? According to your own assessment, did the wellbeing of vulnerable families improve after DOVCU interventions? - What were the indicators of such success? - Were there any cases (households) when prevention methods of the project did not improve the situation for children in the household? Why was this? How did the project address this situation? - For which families were prevention efforts the most successful? What do you think were the circumstances that made these cases successful? 38. To what extent do you think the DOVCU project has responded to most common risk factors to separation (including for boys and girls) and was the support from the integrated package of interventions meeting the most urgent needs of children and families that the project targeted? Example of information we are interested to receive: - Which services/support offered by DOVCU have been the most useful and had the most effect for prevention? - What combinations of supports have been most useful to ensure child protection, and safe reintegration? - Were there, in your opinion, any missing elements in the support provided that should have been included in the support package to vulnerable families, in order to prevent the separation of children from these families? - To what extent did support and services offered to families need adaptation to the local context? Please give examples of how such adaptation was made. 39. Do you think the services and supports offered in DOVCU-districts would have the same effect if replicated / expanded in other parishes and/or districts in Uganda or are there any elements that would need better adaptation? Example of information we are interested to receive: - What kind of adaptation is suggested 58 Involvement of other stakeholders in project and effects of this involvement 40. We would not like to understand how other stakeholders, were involved in the project and what you assess as the effect of their involvement in the project. 41. In this district, have any Alternative Care Panel and/or Team Around the Child been established and how have these functioned? Examples of information the evaluation seeks to explore: - How were member selected - Who are the members - Regularity of meetings; - Level of participation of core members; - What were the core functions - Compliance with core functions; - The extent to which members knew and used foster care and adoption guidelines; - Coverage of various types of cases; - Regularity of reviews of placements and movements of children. - Any challenges and lessons learned from the work of ACPs and TACs. 42. How were community child protection mechanisms and para-social workers involved in the project and what was the experience of their involvement. Examples of information the evaluation seeks to explore: - Examples of activities they were involved in - Strength and weaknesses and challenges in their involvement 43. What changes in community child protection mechanisms and para-social workers do you think the DOVCU - project activities influenced: Examples of information the evaluation seeks to explore: - Improvements in understanding by community based child protection structures and para social workers of their key roles and contributions to prevent institutionalization and follow-up on child protection cases and cases of children who have been reintegrated. - Changes in awareness and skills to identify at risk children; - Changes in the frequency of referrals coming from these structures to PSWOs; - More proactive actions taken by these structures to solve the problems at their levels; - Proactive activities of these structures to make people aware of the existence of services that can help prevent separation pf children from their families. 44. What changes in professional practices of PSWOs, CDOs and para-social workers do you think the DOVCU - project activities influenced: Examples of information the evaluation seeks to explore: - If PSWOs and CDOs are more regularly involved in alternative care cases, cases of children to be reintegrated from the street or from remand homes - Changes in the quality and use of standardized processes and forms for alternative care case assessments and reintegration 59 - Regular follow-up to cases of risk for children (in CCIs, on the street, in remand homes, in families) - Improvements in the quality of case files for alternative care cases - Better regularity in follow-up in alternative care placements - Better regularity in PSWO and CDO monitoring of cases until reintegration - Better regularity and quality of inspections and reporting on CCIs and children placed in CCIs to central Ministry - Regular use of standardized referral protocols; - Better clarity among professionals on the trajectory of cases (e.g. referrals and admission practices). - Collaborative spirit between all professionals involved at district, community and service delivery level to first and foremost ensure prevention of separation, and as a last resort quality of placements of children into care. 45. What changes in the collaboration between different professionals has the DOVCU-project influenced: Examples of information the evaluation seeks to explore: - If the PSWO, as part of the DOVCU-activities, collaborated more or differently with CDOs and psychosocial workers with community-based partners to improve identification of children outside of family care - How they worked together with these professionals towards securing placements in family based care, and how they jointly monitored children's wellbeing? - How often did such collaboration take place and were you, through this collaboration, able to better identify and refer children living outside of family care? - What were the strengths in this collaborative work and in what way did DOVCU change how you do such work in the district - What are the challenges in collaboration with other professionals? - How did DOVCU project help you to address these challenges? Overall experience of being involved in these activities of the DOVCU project 46. How would you describe your overall experience of being involved in the project? Examples of information the evaluation seeks to explore: - Were the human resources (e.g. social work capacity, staff providing training and coaching activities), and other resources invested in the project such as coordination mechanisms (e.g. TAC and ACPs, social workers and project coordinators), time and other resources sufficient for there to be a lasting change in professional practices & were these resources complementary to what you can mobilize themselves. - Is there any activity that you think should have been included that was not included? Did you provide this feedback to the DOVCU-project staff? What did they do? 47. To what extent do you feel that the DOVCU-project has contributed to put in place structures and processes that are sustainable beyond the project’s timeframe? Examples of information the evaluation seeks to explore: - What structures have been established / strengthened? - What processes and procedures have been changed ((e.g. supervision/monitoring of CCIs and of reintegrated children, case management for children in need of alternative care or at risk) - What makes these changes sustainable? - What would be needed to make them sustainable? 60 48. The DOVCU project aimed at supporting 43,000 vulnerable households to prevent the separation of children from their families, and to reintegrate 2,000 children from child care institutions, from the street and from remand homes. What is your perception of the success of the project to reach those targets? Example of information we are interested to receive: - Did the DOVCU project support the implementation of the conditions for such ambitious targets to be reached? - What were the strengths and weaknesses of the changes introduced by the DOVCU project to reach those targets? Lessons learned: 49. What are the specific lessons learned from this project that would be useful to document and share: Examples of information the evaluation seeks to explore: - Reflections on the overall design of the project and if (in retrospect) selection of intervention areas and activities were sound. - Things that worked well, or were a challenge in the project’s work with child care institutions. - Suggestions for how to overcome such challenges in any future project with similar objectives. - Things that worked well, or were a challenge in the project’s work with different professionals involved in gatekeeping and outreach to vulnerable families (e.g. para professionals how are they working together with probation and social welfare officers and what is motivating them in their work) - Things that worked well, or were a challenge in the project’s integrated package of support? - Things that worked well, or were a challenge in the project’s cash-benefit and income generating approaches. Review of case files In the last part of the interview the interviewer should ask the respondent more about the work-flows, work volumes and work practices of PSWOs. The Interviewer should ask the PSWO to illustrate some of the practices and processes of work including On cases and case-files: 1. How many cases are referred to the PSWO within a 12 month period? How many follow-ups were done within this given period? How many cases were closed? 2. Ask the PSWO to show how cases are documented and where they are stored Observations (Look for whether the cases are documented in discrete case-files and if the case-files are stored in a way that guarantees the confidentiality of the clients, in a way that makes it easy to find case-files for follow-ups, etc.): 61 3. Ask the PSWO to show a few (between 3 and 5) social inquiry reports on families at risk; files for case assessments, etc. The interviewer should ask to see the report for one typical case that is representative of the type of cases dealt with in the district, as well as two cases which were standing out in different ways (e.g. because they were difficult to deal with, or because the social inquiry took a different than usual form: Observations (Look at whether a standard form is being used, quality of the information provided in the report, completeness of the report, whether follow up was provided, within how much time such follow up was provided): 4. A report from an inspection of a child care institution (please also show me where and how you enter this data into a system/report it to the Central Ministry): Observations (Look at whether a standard form is being used, quality of the information provided in the report, completeness of the report, whether follow up was provided, within how much time such follow up was provided): 5. (On trajectory of cases and integration and collaboration with other stakeholders and service providers – the respondent can be encouraged to make a drawing that illustrates flow of cases and collaboration lines) Can you list: a) Who are the actors who refers cases to you / from where are cases referred? b) What happens to the cases once they reach your office/you? c) To which instances do you refer cases? 6. After a case has been referred, do you usually get feedback on what happens to cases after you have referred them and how do you get this feedback? Please ask the PSWO to fill in the survey questionnaire (tool 1) and hand over the hard copy to you before you leave. Filling it in will take max 10 minutes. 62 Tool 10 – Guide for Focus Group Discussions with Community Development Officers (CDOs) (1 FGD / district) Introduction: Interviewer to start with an introduction, which organization he/she is representing and in which context the interview is taking place: The interview is being carried out in the context of an end-line evaluation of a project (DOVCU) that has been implemented in the area of child care and de-institutionalization by MoGLSD and a consortium led by ChildFund (RETRAK, TPO and Child’s I Foundation). The project started in 2014 and was implemented for 3 years across 12 districts of Uganda. It had for overall objective to prevent family separation and reintegrate children without appropriate family care back into nurturing family environments. The project worked with a large number of stakeholders, including district level officials, alternative care panels, probation officers, community development officers, para-social workers, community child protection structures, health facilities, child care institutions, vulnerable families and children and with children in institutions, remand homes and children found to be living in street situations. With this interview, we want to establish your involvement into the project, what and how you think it has contributed to changes in the lives of the beneficiaries. What you will tell us in the interview will remain confidential. The information you share with us will be compiled in a report that will have more than 100 respondents. We may quote you, but it will not be possible to link quotes or information discussed in the report with any specific respondent. Notes and recordings from the discussion will remain with the evaluator and will not be shared with anybody else. The interview is expected to take 1 hour. Ask for permission to start the discussion, and to record it. Before the discussion starts, the facilitator will establish how many of the total CDOs from the district are represented in the meeting as well as the demographics of group discussants: Male / Female Age Years in service Highest degree Male / Female Age Years in service Highest degree Male / Female Age Years in service Highest degree Male / Female Age Years in service Highest degree Male / Female Age Years in service Highest degree 63 Male / Female Age Years in service Highest degree Male / Female Age Years in service Highest degree Male / Female Age Years in service Highest degree Introductory engagement questions: 1. Please tell me about your role in working with children who have been separated from their families, and children who live in child care institutions. 2. Do you know about the Alternative Care Framework and DOVCU (the project Deinstitutionalization of Vulnerable Children in Uganda)? 3. What do you know about these and the priorities they introduce for services and supports to be provided to vulnerable families and children at risk (in CCIs or in street-situations)? Project implementation: 4. What was your involvement in the DOVCU-project implementation, the types of activities you were involved in, what went well and what were challenges in the project implementation and monitoring. Examples of information the evaluation seeks to explore: - What briefings, trainings and other activities that they were involved in, through the DOVCU project, what were the strengths and what were the weaknesses of these? - How many staff in this office were trained? On what topics were the trainings? - Particularly, what new did you learn from trainings? Have you changed any of your processes / practices as a result of these trainings? What changes did you introduce? - What were the strengths and what were the weaknesses of these trainings? - What activities and strategies used by the project have you found to be the most useful, to for example change the way you work (assessments, coaching, training, technical assistance activities etc.) 5. The DOVCU project aimed at supporting 43,000 vulnerable households to prevent the separation of children from their families, and to reintegrate 2,000 children from child care institutions, from the street and from remand homes. What is your perception of the success of the project to reach those targets? Example of information we are interested to receive: - Did the DOVCU project support the implementation of the conditions for such ambitious targets to be reached? 64 - What were the strengths and weaknesses of the changes introduced by the DOVCU project to reach those targets? Beneficiaries: vulnerable families targeted for prevention 6. What has been the effect of the DOVCU-project’s interventions in the lives of the families that were targeted for prevention activities and who received support from the project? Examples of information the evaluation seeks to explore: - What kind of support packages did families receive in the end? - How did the project ensure to target the most vulnerable families? Was targeting successful or were there any cases, for example, were the wrong beneficiary was selected? - How successful were the various interventions in preventing separation of children at risk from their families? According to your own assessment, did the wellbeing of vulnerable families improve after DOVCU interventions? - What were the indicators of such success? - Were there any cases (households) when prevention methods of the project did not improve the situation for children in the household? Why was this? How did the project address this situation? - For which families were prevention efforts the most successful? What do you think were the circumstances that made these cases successful? 7. How were families targeted for prevention support? Was this targeting appropriate? If not, why not? Examples of information the evaluation seeks to explore: - Who was targeted? - Did the project reach those families where children were the most at risk of separation? - If yes, what specifically made the project successful in reaching those most at risk? If no, what were the challenges? - Were activities to support families and children who were reintegrated tailored to the specific needs of each family / child? If yes, what were the strengths? If no, what were the challenges? Beneficiaries: reintegrated children 8. How were children to be reintegrated from child care institutions selected? Examples of information the evaluation seeks to explore: - What profile did the children have, who ultimately got reintegrated? - Who decided which children should be reintegrated? - How was the decision made? - Were these children the most vulnerable or those most in need of being reintegrated? 9. Overall, what has been the overall experience of these reintegration cases? Please talk both about the positive experience and the challenges that were faced. Examples of information the evaluation seeks to explore: - If in addition to the quoted number, how many cases of children leaving CCIs the project is thought to have influenced, even if not all these cases were followed by the project? - If the project reached its targets, what were the facilitating factors ensuring the target were reached? - If targets were not reached, what were the challenges in reaching targets? 65 10. Of the children that the DOVCU project reintegrated, do all children have a permanency plan and what has been the experience of implementing the permanency plan? Examples of information the evaluation seeks to explore: - What were successes and challenges? - In how many cases was it not possible to implement the permanency plan, what were the challenges? - Now that the project is ending, for how long will children continue to be followed up by ChildFund and the project partners, by PSWOs, CDOs or other government staff? - What will allow for such follow-up after the project has ended (e.g. availability of sustained mobility of PSWOs and CDOs, institutionalized government guidelines for case management, reintegration and permanency planning etc.) 11. What has been the impact of the reintegration on the children and the families where children were received? Examples of information the evaluation seeks to explore: - What kind of support did children receive (pre-, during and post reintegration activities)? - What kind of support did families receive (pre-, during and post reintegration activities)? - How successful were the reintegration activities overall? According to your assessment, did the wellbeing of reintegrated children improve after DOVCU interventions? - In what way did their wellbeing improve? Was there any area of wellbeing that showed greater improvements than other? - Did all reintegrated children remain in their placements? - Do you know of any case where the child did not remain in his/her placement? What happen? - What kind of actions did the project take in the cases where children did not show improved well￾being after they had been reintegrated - Do you know of any case where the reintegration of the child did not show improvements in the well-being status of the child? What happen? How did the project address this situation? - Do you know of any case where a reintegrated child returned to institutions or to the streets? What happen? How did the project address this situation? 12. For the children who have been successfully reintegrated with their families or placed in alternative family based care, what made the reintegration method successful and sustainable? Examples of information the evaluation seeks to explore: - What makes these reintegration methods sustainable? - For which children was impact most visible? - Why do you think reintegration efforts were more successful for some children? - What were the circumstances that influenced this? 13. For the children who were not successfully reintegrated, what were the challenges with these cases? Examples of information the evaluation seeks to explore: - For which children did reintegration not work? - Why do you think reintegration efforts did not work? 66 - What were the circumstances that influenced this? 14. To what extent was the project able to prevent child protection issues and human rights violations (e.g. child labour, violence against children, neglect, early marriage, etc.) and mitigate risks for creating vulnerability (e.g. child labour and any kind of harm)? Examples of information the evaluation seeks to explore: - How were such risks assessed and monitored - If risk was identified, how was it reduced Reintegration of children from remand homes (Question valid only in Mbale and Gulu districts) 15. Overall, how many children have been reintegrated or placed in alternative family based care from remand homes as a result of DOVCU activities and what has been the experience of these cases of reintegration? Please talk both about the positive experience and the challenges that were faced in the process. Question valid only in Mbale and Gulu districts Examples of information we are interested in: - Are there less children in remand homes now as compared to three years ago? - Do fewer children enter remand homes now as compared to three years ago? - Have the reasons for why children are placed in remand homes changed (e.g. now only for heavier crimes, or also petty crimes, behavioural or statutory offences) - Do more children leave a child care placement (outflow) now as compared to three years ago? - Do children more now than three years ago, leave remand homes quicker and easier or does it still happen that children miss their trials because of fuel shortages or inability to get to the court room for hearings? - For the children who are going home from remand homes, do they have a plan for how reintegration should be done to ensure safe and permanent reintegration? Reintegration of children from the street (Question valid only in Mbale district): 16. Overall, how many children have been reintegrated or placed in alternative family based care from the street as a result of DOVCU activities and what has been the experience of these cases of reintegration? Please talk both about the positive experience and the challenges that were faced in the process. Examples of information that we are interested to receive: - If there is an overall reduction in the phenomenon of children recorded to be on the street in the districts where DOVCU operated - What was the experience of implementing the permanency plan – successes and challenges? - In how many cases was it not possible to implement the permanency plan, what were the challenges? - Now that the project is ending, for how long will children continue to be followed up by ChildFund and the project partners, by PSWOs, CDOs or other government staff? - What will allow for such follow-up after the project has ended (e.g. availability of sustained mobility of PSWOs and CDOs, institutionalized government guidelines for case management, reintegration and permanency planning etc.) Professional practices CDOs 67 17. What changes in your professional practices did the DOVCU - project activities influence? Examples of information the evaluation seeks to explore: - If they are now more regularly involved in alternative care cases, cases of children to be reintegrated from the street or from remand homes - Changes in the quality and use of standardized processes and forms for alternative care case assessments and reintegration - Regular follow-up to cases of risk for children (in CCIs, on the street, in remand homes, in families) - Regularity in follow-up in alternative care placements - Regularity in CDO monitoring of cases until reintegration - Regularity and quality of inspections and reporting on CCIs and children placed in CCIs to central Ministry - Regular use of standardized referral protocols; - Better clarity among professionals on the trajectory of cases (e.g. referrals and admission practices). - Collaborative spirit between all professionals involved at district, community and service delivery level to first and foremost ensure prevention of separation, and as a last resort quality of placements of children into care. Community child protection mechanisms and para-social workers 18. What changes in community child protection mechanisms and para-social workers do you think the DOVCU - project activities influenced: Examples of information the evaluation seeks to explore: - Improvements in understanding by community based child protection structures and para social workers of their key roles and contributions to prevent institutionalization and follow-up on child protection cases and cases of children who have been reintegrated. - Changes in awareness and skills to identify at risk children; - Changes in the frequency of referrals coming from these structures to PSWOs; - More proactive actions taken by these structures to solve the problems at their levels; - Proactive activities of these structures to make people aware of the existence of services that can help prevent separation pf children from their families. Services and supports provided through DOVCU 19. Let’s talk about the services and supports that were provided: To what extent do you think the DOVCU project has responded to most common risk factors to separation (including for boys and girls) and was the support from the integrated package of interventions meeting the most urgent needs of children and families that the project targeted? Example of information we are interested to receive: - Which services/support offered by DOVCU have been the most useful and had the most effect for prevention? - What combinations of supports have been most useful to ensure child protection, and safe reintegration? - Were there, in your opinion, any missing elements in the support provided that should have been included in the support package to vulnerable families, in order to prevent the separation of children from these families? 68 - To what extent did support and services offered to families need adaptation to the local context? Please give examples of how such adaptation was made. Concluding comments on lessons learned: 20. Please share any reflections lessons learned from this project Examples of information the evaluation seeks to explore: - Reflections on the overall design of the project and if (in retrospect) selection of intervention areas and activities were sound. - Things that worked well, or were a challenge in the project’s work with child care institutions. - Suggestions for how to overcome such challenges in any future project with similar objectives. - Things that worked well, or were a challenge in the project’s work with different professionals involved in gatekeeping and outreach to vulnerable families (e.g. para professionals how are they working together with probation and social welfare officers and what is motivating them in their work) - Things that worked well, or were a challenge in the project’s integrated package of support? - Things that worked well, or were a challenge in the project’s cash-benefit and income generating approaches. 69 Tool 11 – Guide for Focus Group Discussions with members of Teams Around the Child / Alternative Care Panels (1 FGD / district) Introduction: Interviewer to start with an introduction, which organization he/she is representing and in which context the interview is taking place: The interview is being carried out in the context of an end-line evaluation of a project (DOVCU) that has been implemented in the area of child care and de-institutionalization by MoGLSD and a consortium led by ChildFund (RETRAK, TPO and Child’s I Foundation). The project started in 2014 and was implemented for 3 years across 12 districts of Uganda. It had for overall objective to prevent family separation and reintegrate children without appropriate family care back into nurturing family environments. The project worked with a large number of stakeholders, including district level officials, alternative care panels, probation officers, community development officers, para-social workers, community child protection structures, health facilities, child care institutions, vulnerable families and children and with children in institutions, remand homes and children found to be living in street situations. With this interview, we want to establish your involvement into the project, what and how you think it has contributed to changes in the lives of the beneficiaries. What you will tell us in the interview will remain confidential. The information you share with us will be compiled in a report that will have more than 100 respondents. We may quote you, but it will not be possible to link quotes or information discussed in the report with any specific respondent. Notes and recordings from the discussion will remain with the evaluator and will not be shared with anybody else. The interview is expected to take 1 hour. Ask for permission to start the discussion, and to record it. Before the discussion starts, the facilitator will establish how many of the total CDOs from the district are represented in the meeting as well as the demographics of group discussants: Male / Female Age Years in service Highest degree Male / Female Age Years in service Highest degree Male / Female Age Years in service Highest degree Male / Female Age Years in service Highest degree Male / Female Age Years in service Highest degree 70 Male / Female Age Years in service Highest degree Male / Female Age Years in service Highest degree Male / Female Age Years in service Highest degree Introductory engagement question: 1. What was your involvement in the DOVCU-project implementation, please describe to us the role and the functions of the Alternative Care Panels and Teams Around the Child: Examples of information the evaluation seeks to explore: - How were member selected - Who are the members - Regularity of meetings; - Level of participation of core members; - What were the core functions - How did they implement core functions; 2. What do you see as the mandate of the ACP / TAC? 3. Were you trained on any of the activities you were to be involved in through the DOVCU project? Examples of information the evaluation seeks to explore: - What briefings, trainings and other activities that they were involved in, through the DOVCU project, what were the strengths and what were the weaknesses of these? - On what topics were the trainings? - Particularly, what new did you learn from trainings? - What were the strengths and what were the weaknesses of these trainings? - The extent to which members knew and used foster care and adoption guidelines. 4. The DOVCU project aimed at supporting 43,000 vulnerable households to prevent the separation of children from their families, and to reintegrate 2,000 children from child care institutions, from the street and from remand homes. What is your perception of the success of the project to reach those targets? Example of information we are interested to receive: - Did the DOVCU project support the implementation of the conditions for such ambitious targets to be reached? - What were the strengths and weaknesses of the changes introduced by the DOVCU project to reach those targets? Mandate and functionality of TACs and ACPs 71 5. What kind of cases have you been looking at / discussing? Example of information we are interested to receive: - Coverage of various types of cases; - What have been the successes and challenges in working on these cases? - Were there any cases that were more difficult, less difficult than other? - What did the TAC / ACP do to ensure all cases of reintegration were successful? - Any challenges and lessons learned from the work of ACPs and TACs. Beneficiaries: vulnerable families targeted for prevention 6. What has been the effect of the DOVCU-project’s interventions on lives of the families that were targeted for prevention activities and who received support from the project? Examples of information the evaluation seeks to explore: - What kind of support packages did families receive in the end? - How did the project ensure to target the most vulnerable families? Was targeting successful or were there any cases, for example, were the wrong beneficiary was selected? - How successful were the various interventions in preventing separation of children at risk from their families? According to your own assessment, did the wellbeing of vulnerable families improve after DOVCU interventions? - What were the indicators of such success? - Were there any cases (households) when prevention methods of the project did not improve the situation for children in the household? Why was this? How did the project address this situation? - For which families were prevention efforts the most successful? What do you think were the circumstances that made these cases successful? 7. How were families targeted for prevention support? Was this targeting appropriate? Examples of information the evaluation seeks to explore: - Who was targeted? - Did the project reach those families where children were the most at risk of separation? - If yes, what specifically made the project successful in reaching those most at risk? If no, what were the challenges? - Were activities to support families and children who were reintegrated tailored to the specific needs of each family / child? If yes, what were the strengths? If no, what were the challenges? Beneficiaries: reintegrated children 8. How were children to be reintegrated from CCI selected? Examples of information the evaluation seeks to explore: - What profile did the children have, who ultimately got reintegrated? - Who decided which children should be reintegrated? - How was the decision made? - Were these children the most vulnerable or those most in need of being reintegrated? - Lessons learned from reintegration activities (what worked well, what were the challenges) 72 9. Overall, how many children have been reintegrated or placed in alternative family based care from CCIs and what has been your overall experience of these reintegration cases? Please talk both about the positive experience and the challenges that were faced. Examples of information the evaluation seeks to explore: - If in addition to the quoted number, how many cases of children leaving CCIs the project is thought to have influenced, even if not all these cases were followed by the project? - If the project reached its targets, what were the facilitating factors ensuring the target were reached? - If targets were not reached, what were the challenges in reaching targets? 10. Of the children that the DOVCU project reintegrated do all children have a permanency plan and what has been the experience of implementing the permanency plan? Examples of information the evaluation seeks to explore: - What were successes and challenges? - In how many cases was it not possible to implement the permanency plan, what were the challenges? - Now that the project is ending, for how long will children continue to be followed up by ChildFund and the project partners, by PSWOs, CDOs or other government staff? - What will allow for such follow-up after the project has ended (e.g. availability of sustained mobility of PSWOs and CDOs, institutionalized government guidelines for case management, reintegration and permanency planning etc.) 11. What has been the effect of the reintegration of children on the children and the families where children were received? Examples of information the evaluation seeks to explore: - What kind of support did children receive (pre-, during and post reintegration activities)? - What kind of support did families receive (pre-, during and post reintegration activities)? - How successful were the reintegration activities overall? According to your assessment, did the wellbeing of reintegrated children improve after DOVCU interventions? - In what way did their wellbeing improve? Was there any area of wellbeing that showed greater improvements than other? - Did all reintegrated children remain in their placements? - Do you know of any case where the child did not remain in his/her placement? What happen? - What kind of actions did the project take in the cases where children did not show improved well￾being after they had been reintegrated - Do you know of any case where the reintegration of the child did not show improvements in the well-being status of the child? What happen? How did the project address this situation? - Do you know of any case where a reintegrated child returned to institutions or to the streets? What happen? How did the project address this situation? 12. For the children who have been successfully reintegrated with their families or placed in alternative family based care, what made the reintegration method successful and sustainable? Examples of information the evaluation seeks to explore: - What makes these reintegration methods sustainable? 73 - For which children was impact most visible? - Why do you think reintegration efforts were more successful for some children? - What were the circumstances that influenced this? 13. To what extent was the project able to prevent child protection issues and human rights violations (e.g. child labour, violence against children, neglect, early marriage, etc.) and mitigate risks for creating greater vulnerability (e.g. child labour and any kind of harm)? Examples of information the evaluation seeks to explore: - How were such risks assessed and monitored - If risk was identified, how was it reduced Reintegration of children from remand homes (Question valid only in Mbale and Gulu districts) 14. Overall, how many children have been reintegrated or placed in alternative family based care from remand homes as a result of DOVCU activities and what has been the experience of these cases of reintegration? Please talk both about the positive experience and the challenges that were faced in the process. Question valid only in Mbale and Gulu districts Examples of information we are interested in: - Are there less children in remand homes now as compared to three years ago? - Do fewer children enter remand homes now as compared to three years ago? - Have the reasons for why children are placed in remand homes changed (e.g. now only for heavier crimes, or also petty crimes, behavioural or statutory offences) - Do more children leave a child care placement (outflow) now as compared to three years ago? - Do children more now than three years ago, leave remand homes quicker and easier or does it still happen that children miss their trials because of fuel shortages or inability to get to the court room for hearings? - For the children who are going home from remand homes, do they have a plan for how reintegration should be done to ensure safe and permanent reintegration? Reintegration of children from the street (Question valid only in Mbale district): 15. Overall, how many children have been reintegrated or placed in alternative family based care from the street as a result of DOVCU activities and what has been the experience of these cases of reintegration? Please talk both about the positive experience and the challenges that were faced in the process. Examples of information that we are interested to receive: - If there is an overall reduction in the phenomenon of children recorded to be on the street in the districts where DOVCU operated - What was the experience of implementing the permanency plan – successes and challenges? - In how many cases was it not possible to implement the permanency plan, what were the challenges? - Now that the project is ending, for how long will children continue to be followed up by ChildFund and the project partners, by PSWOs, CDOs or other government staff? - What will allow for such follow-up after the project has ended (e.g. availability of sustained mobility of PSWOs and CDOs, institutionalized government guidelines for case management, reintegration and permanency planning etc.) 74 16. According to you, to what extent, and which systemic changes did the project bring about at community and district levels that are enabling children to live in family care and preventing inappropriate placements in institutional care? Examples of information the evaluation seeks to explore: - Capacity strengthening of Probation and Social Welfare Officers (PSWOs) and CDOs - Changes in practices of Remand homes - Changes in practices of Child Care Institutions - Strengthened capacity of Para-social workers to address alternative care and issues of separation and risk of separation for children - Strengthened capacity of Community child protection structures to work on alternative care issues of separation and risk of separation for children 17. We are also interested to know what are the changes in child care institutions (CCIs) as a result of DOVCU-project activities: Examples of information the evaluation seeks to explore: - How many child care institutions in the DOVCU-districts district have been inspected? - For those CCIs that received recommendations for improvements to be made, how did the project follow-up with these CCIs? - How many developed an improvement plan? - What was the experience and lessons learned regarding the implementation of improvement plans? - Who followed up with CCIs on the improvements that were needed, and was there any re￾inspection of CCIs after some time? Who did that re-inspection? - Of the CCIs that were inspected by MoGLSD and their team, how many received a licence to operate? - What happened after CCIs received a license to operate, is there for example any obligation to regularly report on activities in the CCI to MoGLSD, as a requirement to maintain the licence? 18. Were there any CCIs that closed completely? Examples of information the evaluation seeks to explore: - In which districts did this happen and what were the names of these CCIs? - How were such closures planned and monitored? - What happened to children in the CCIs that closed? - Who was involved in the oversight and monitoring of such closures? - Were there any CCIs that closed spontaneously without oversight by the project or by MoGLSD? - In which districts did this happen and what were the names of these CCIs? - Is it known what happened to children in the CCIs that closed? - What did the project do, and what did MoGLSD do when they learned about these self-closures? - What attempts did the project make, or MoGLSD and district staff make to trace and check on the children from the CCIs that self-closed? 19. Regarding the practices of child care institutions, were there any observed and documented changes in how they operate as a result of the DOVCU-project? Examples of information the evaluation seeks to explore: 75 - Changes in governance and management culture (e.g. operating with formal registration, policies on having employed qualified social workers, overall human resource policies, and minimum qualifications for staff working in the CCI, policies for child protection, adequate descriptions of the types of care facility and services provided there etc.) - Changes in funding culture (e.g. fundraising for provision of residential care to individual children only or fundraising also to support children in their community) - Referral and admission practices (e.g. availability of care orders for all children staying at the CCI, criteria for admission, whether there are more streamlined referral and admission processes going through care planning and placement reviews with involvement of PSWOs and multi-disciplinary panels at district level) - Practices, policies and guidelines for in case review and reintegration - Policies for family visits and staying in touch with culture and community of children who are placed - Existence of standards of services provided to children in CCIs Changes in remand home practices (questions valid only for Mbale and Gulu districts) 20. We are also interested to know what are the changes in remand homes as a result of DOVCU￾project activities: Examples of information the evaluation seeks to explore: - The extent to which DOVCU has contributed to improvements in knowledge, skills and utilization by remand home staff of guidelines and tools (standard operating procedures) which project trained them on; - Improvements in the level of compliance with decisions on children (e.g. if children who have served their sentence are allowed to go home); - Availability of strategic partnerships and alliances for follow-up on cases of reintegration from remand homes. 21. What has been the effect of the DOVCU-project’s interventions in the lives of the families that were targeted for prevention activities and who received support from the project? Examples of information the evaluation seeks to explore: - What kind of support packages did families receive in the end? - Who was targeted? - How did the project ensure to target the most vulnerable families? Was targeting successful or were there any cases, for example, were the wrong beneficiary was selected? - How successful were the various interventions in preventing separation of children at risk from their families? According to your own assessment, did the wellbeing of vulnerable families improve after DOVCU interventions? - What were the indicators of such success? - Were there any cases (households) when prevention methods of the project did not improve the situation for children in the household? Why was this? How did the project address this situation? - For which families were prevention efforts the most successful? What do you think were the circumstances that made these cases successful? 22. To what extent do you feel that the work of the TACs and the ACPs can be sustainable beyond the project’s timeframe? Examples of information the evaluation seeks to explore: 76 - What makes these them sustainable? - What would be needed to make them sustainable? Lessons learned: 23. What are the specific lessons learned from this project that would be useful to document and share: Examples of information the evaluation seeks to explore: - Reflections on the overall design of the project and if (in retrospect) selection of intervention areas and activities were sound. - Things that worked well, or were a challenge in the project’s work with child care institutions. - Suggestions for how to overcome such challenges in any future project with similar objectives. - Things that worked well, or were a challenge in the project’s work with different professionals involved in gatekeeping and outreach to vulnerable families (e.g. para professionals how are they working together with probation and social welfare officers and what is motivating them in their work) - Things that worked well, or were a challenge in the project’s integrated package of support? - Things that worked well, or were a challenge in the project’s cash-benefit and income generating approaches. 77 V. Interviews to be conducted at sub-district level Tool 12 – Guide for Key Informant Interviews with staff in Child Care institutions (1 KII / institution x 2 institutions / district): Introduction: Interviewer to start with an introduction, which organization he/she is representing and in which context the interview is taking place: The interview is being carried out in the context of an end-line evaluation of a project (DOVCU) that has been implemented in the area of child care and de-institutionalization by MoGLSD and a consortium led by ChildFund (RETRAK, TPO and Child’s I Foundation). The project started in 2014 and was implemented for 3 years across 12 districts of Uganda. It had for overall objective to prevent family separation and reintegrate children without appropriate family care back into nurturing family environments. The project worked with a large number of stakeholders, including district level officials, alternative care panels, probation officers, community development officers, para-social workers, community child protection structures, health facilities, child care institutions, vulnerable families and children and with children in institutions, remand homes and children found to be living in street situations. With this interview, we want to establish your involvement into the project, what and how you think it has contributed to changes in the lives of the ultimate beneficiaries. What you will tell us in the interview will remain confidential. The information you share with us will be compiled in a report that will have more than 100 respondents. We may quote you, but it will not be possible to link quotes or information discussed in the report with any specific respondent. Notes and recordings from the discussion will remain with the evaluator and will not be shared with anybody else. Ask for permission to start the interview, and to record the interview. Project implementation: 1. Please give an introduction of your involvement in the DOVCU-project implementation, the types of activities you were involved in, what went well and what were challenges in the project implementation. Examples of information the evaluation seeks to explore: - What briefings, trainings and other activities that they were involved in, through the DOVCU project, what were the strengths and what were the weaknesses of these? - How many staff in this CCI were trained? - On what topics were the trainings? - Particularly, what new did you learn from trainings? - Have you changed any of your processes / practices as a result of these trainings? - What changes did you introduce? - What were the strengths and what were the weaknesses of these trainings? - What activities and strategies used by the project have you found to be the most useful, to for example change the way you work (assessments, coaching, training, technical assistance activities etc.) 2. From this project, what have you understood as being your role in supporting the implementing the Alternative Care Framework? Examples of information the evaluation seeks to explore: 78 - Their described role in identification and referral of cases of children in need of care and protection - Challenges or opportunities in their role to implement this policy 3. How would you describe your overall experience of being involved in the project? Examples of information the evaluation seeks to explore: - Were the human resources (e.g. social work capacity, staff providing training and coaching activities), and other resources invested in the project such as coordination mechanisms (e.g. TAC and ACPs, social workers and project coordinators), time and other resources sufficient for there to be a lasting change in professional practices & were these resources complementary to what you can mobilize themselves. - Is there any activity that you think should have been included that was not included? Did you provide this feedback to the DOVCU-project staff? What did they do? 4. We are interested to know what are the changes made in this child care institutions (CCIs) as a result of DOVCU-project activities: Examples of information the evaluation seeks to explore: - Was this child care institutions in the DOVCU inspected? - For those CCIs that received recommendations for improvements to be made, how did the CCI and the project follow-up with these CCIs? - Was there an improvement plan developed? - What things were identified for improvement? - What was the experience and lessons learned regarding the implementation of improvement plans? - Who followed up with CCIs on the improvements that were needed, and was there any re￾inspection of CCIs after some time? Who did that re-inspection? - Of the CCIs that were inspected by MoGLSD and their team, did the CCI receive a licence to operate? - What happened after CCIs received a license to operate, are there regular monitoring visits made to the CCI? - Does the CCI do any regular reporting to PSWOs, as a requirement to maintain the licence? 5. In this CCI what kind of dynamics has the DOVCU project influenced when it comes to number of children placed? Is there any changed dynamic? What change has taken place? Examples of information the evaluation seeks to explore: - Does the CCI receive fewer children? - Does the CCI receive a different profile of children as compared to before? - Do fewer children enter CCIs now as compared to three years ago? - Do more children leave a child care placement now as compared to three years ago? - Do children on average stay shorter periods in CCIs as compared to three years ago? 6. Regarding the practices of child care institutions, has the project influenced any changes in how they operate as a result of the DOVCU-project? Examples of information the evaluation seeks to explore: 79 - Changes in governance and management culture (e.g. operating with formal registration, policies on having employed qualified social workers, overall human resource policies, and minimum qualifications for staff working in the CCI, policies for child protection, adequate descriptions of the types of care facility and services provided there etc.) - Changes in funding culture (e.g. fundraising for provision of residential care to individual children only or fundraising also to support children in their community) - Referral and admission practices (e.g. availability of care orders for all children staying at the CCI, criteria for admission, whether there are more streamlined referral and admission processes going through care planning and placement reviews with involvement of PSWOs and multi-disciplinary panels at district level) - Practices, policies and guidelines for in case review and reintegration - Policies for family visits and staying in touch with culture and community of children who are placed - Existence of standards of services provided to children in CCIs Beneficiaries: reintegrated children 7. Did this child care institution reintegrate any children to their families Examples of information the evaluation seeks to explore: - Were these reintegration activities supported by the DOVCU project or did the CCI carry out these activities on their own? - How many children (boys and girls) have been reintegrated - What are the most recent cases that were reintegrated 8. For children to be reintegrated from child care institutions, how were children to be reintegrated selected? Examples of information the evaluation seeks to explore: - What profile did the children have, who ultimately got reintegrated? - Who decided which children should be reintegrated? - How was the decision made? - Were these children the most vulnerable or those most in need of being reintegrated? 9. Overall, what has been the overall experience of these reintegration cases? Please talk both about the positive experience and the challenges that were faced. Examples of information the evaluation seeks to explore: - Factors that facilitated reintegration efforts - Factors that made reintegration efforts more difficult 10. Of the children that the DOVCU project reintegrated do all children have a permanency plan and what has been the experience of implementing the permanency plan? Examples of information the evaluation seeks to explore: - What were successes and challenges? - In how many cases was it not possible to implement the permanency plan, what were the challenges? - Now that the project is ending, for how long will the CCI continue to follow-up on children 80 - What will allow for such follow-up after the project has ended (e.g. availability of sustained mobility of PSWOs and CDOs, institutionalized government guidelines for case management, reintegration and permanency planning etc.) 11. What has been the effect of the reintegration of children on the children and the families where children were received? Examples of information the evaluation seeks to explore: - What kind of support did children receive (pre-, during and post reintegration activities)? - What kind of support did families receive (pre-, during and post reintegration activities)? - How successful were the reintegration activities overall? According to your assessment, did the wellbeing of reintegrated children improve after DOVCU interventions? - In what way did their wellbeing improve? Was there any area of wellbeing that showed greater improvements than other? - Did all reintegrated children remain in their placements? - Do you know of any case where the child did not remain in his/her placement? What happen? - What kind of actions did the project take in the cases where children did not show improved well￾being after they had been reintegrated - Do you know of any case where the reintegration of the child did not show improvements in the well-being status of the child? What happen? How did the project address this situation? - Do you know of any case where a reintegrated child returned to institutions or to the streets? What happen? How did the project address this situation? 12. For the children who have been successfully reintegrated with their families or placed in alternative family based care, what made the reintegration method successful and sustainable? Examples of information the evaluation seeks to explore: - What makes these reintegration methods sustainable? - For which children was impact most visible? - Why do you think reintegration efforts were more successful for some children? - What were the circumstances that influenced this? 13. To what extent was the project able to prevent child protection issues and human rights violations (e.g. child labour, violence against children, neglect, early marriage, etc.) and mitigate risks for creating greater vulnerability (e.g. child labour and any kind of harm)? Examples of information the evaluation seeks to explore: - How were such risks assessed and monitored - If risk was identified, how was it reduced 14. At this point in time, does there exist, in this child care institution, any strategies to ensure that sustainability of the efforts that have been made with the support of the DOVCU-project? Examples of information the evaluation seeks to explore: - Is the CCI planning to reduce its capacity and to not fill up places with new children? - Are there any plans to ensure continuity of services provided in CCIs, (e.g. if children are enrolled for educational purposes) to children who have been reintegrated? - Will this continue after the end of the DOVCU project? For how long will you continue providing such services to reintegrated children? 81 - Under what circumstances will the CCI consider ending to provide such support to reintegrated children? - How do they estimate the chances that the cooperation will continue between CCIs and PSWOs to jointly monitor the situation of children who have been reintegrated? Is there expressed interest to continue this cooperation? - What mechanisms have been put in place to facilitate that such cooperation will continue beyond the lifetime of the DOVCU-project? Services and supports provided through DOVCU 15. Let’s talk about the services and supports that were provided to families who received children for reintegration: To what extent do you think the DOVCU project has responded to most common risk factors to separation (including for boys and girls) and was the support from the integrated package of interventions meeting the most urgent needs of children and families that the project targeted? Example of information we are interested to receive: - What combinations of supports have been most useful to ensure child protection, and safe reintegration? - Were there, in your opinion, any missing elements in the support provided that should have been included in the support package to vulnerable families, in order to prevent the separation of children from these families? - To what extent did support and services offered to families need adaptation to the local context? Please give examples of how such adaptation was made. Collaboration with district professionals and structures 16. In this project, what has been your overall experience in collaboration with professionals and different structures at district and community level? Examples of information the evaluation seeks to explore: - Examples of collaboration with PSWO and experience from this collaboration - Examples of collaboration with CDOs and experience from this collaboration - Examples of collaboration with Para-social workers and experience from this collaboration - Examples of collaboration with Community child protection structures and experience from this collaboration - Examples of collaboration with other CCIs and experience from this collaboration - Examples of collaboration with other Alternative Care Panels and Teams Around the Child and experience from this collaboration - Strengths and challenges in this collaboration Lessons learned: 17. What are the specific lessons learned from this project that would be useful to document and share: Examples of information the evaluation seeks to explore: - Reflections on the overall design of the project and if (in retrospect) selection of intervention areas and activities were sound. - Things that worked well, or were a challenge in the project’s work with child care institutions. - Suggestions for how to overcome such challenges in any future project with similar objectives. - Things that worked well, or were a challenge in the project’s work with different professionals involved in gatekeeping and outreach to vulnerable families (e.g. para professionals how are they 82 working together with probation and social welfare officers and what is motivating them in their work) - Things that worked well, or were a challenge in the project’s integrated package of support? In the last part of the interview the interviewer should ask the respondent to show and illustrate some of the practices and processes of work including to show: 1. A few (between 3 and 5) casefiles for children living in the CCI, including (if available) reintegration plans: Observations (Look at whether a standard form is being used, quality of the information provided in the report, completeness of the case file, such as health records, notes on family visits, contacts with family members etc. Check also if any other information is gathered on the child that can help the child understand where he/she is coming from, e.g. pictures, check what is included in the reintegration plan if such is available): 2. (On trajectory of cases and collaboration with other stakeholders and service providers – the respondent can be encouraged to make a drawing that illustrates flow of cases and collaboration lines) Can you list: 3. Who are the actors who refer children to this CCI / from where are cases referred? 4. What happens to the cases once they reach your CCI? 5. Which other services to do you refer children to from here (e.g. health institutions, school, etc.)? 6. When children leave from here, where do they go (adoption, reintegrated, foster care etc.)? 7. What kind of follow-up is provided from this CCI after a child has left? 83 Tool 13 – Guide for Key Informant Interviews with staff in Remand homes (1 KII / institution in 2 remand homes): Introduction: Interviewer to start with an introduction, which organization he/she is representing and in which context the interview is taking place: The interview is being carried out in the context of an end-line evaluation of a project (DOVCU) that has been implemented in the area of child care and de-institutionalization by MoGLSD and a consortium led by ChildFund (RETRAK, TPO and Child’s I Foundation). The project started in 2014 and was implemented for 3 years across 12 districts of Uganda. It had for overall objective to prevent family separation and reintegrate children without appropriate family care back into nurturing family environments. The project worked with a large number of stakeholders, including district level officials, alternative care panels, probation officers, community development officers, para-social workers, community child protection structures, health facilities, child care institutions, vulnerable families and children and with children in institutions, remand homes and children found to be living in street situations. With this interview, we want to establish your involvement into the project, what and how you think it has contributed to changes in the lives of the beneficiaries. What you will tell us in the interview will remain confidential. The information you share with us will be compiled in a report that will have more than 100 respondents. We may quote you, but it will not be possible to link quotes or information discussed in the report with any specific respondent. Notes and recordings from the discussion will remain with the evaluator and will not be shared with anybody else. The interview is expected to take 1 hour. Ask for permission to start the interview, and to record the interview. Project implementation: 1. Please give an introduction of your involvement in the DOVCU-project implementation, the types of activities you were involved in, what went well and what were challenges in the project implementation. Examples of information the evaluation seeks to explore: - What briefings, trainings and other activities that they were involved in, through the DOVCU project, what were the strengths and what were the weaknesses of these? - How many staff in this CCI were trained? - On what topics were the trainings? - Particularly, what new did you learn from trainings? - Have you changed any of your processes / practices as a result of these trainings? - What changes did you introduce? - What were the strengths and what were the weaknesses of these trainings? - What activities and strategies used by the project have you found to be the most useful, to for example change the way you work (assessments, coaching, training, technical assistance activities etc.) 2. From this project, what have you understood as being your role in supporting the implementing the Standard Operating Procedures for the Reintegration of Children from Remand Homes and the Street? Examples of information the evaluation seeks to explore: - Their described role in tracing and reintegration activities 84 - Challenges or opportunities in their role to implement this policy 3. How would you describe your overall experience of being involved in the project? Examples of information the evaluation seeks to explore: - Were the human resources (e.g. social work capacity, staff providing training and coaching activities), and other resources invested by the project sufficient for there to be a lasting change in professional practices & were these resources complementary to what you can mobilize themselves. - Is there any activity that you think should have been included that was not included? - Did you provide this feedback to the DOVCU-project staff? What did they do? Involvement and experience of work with children to be reintegrated from remand homes 4. Were any children reintegrated from this remand home with the support of the DOVCU project? Examples of information the evaluation seeks to explore: - What reintegration activities the stakeholder was involved in 5. Please describe your experience regarding the reintegration of children from this remand home: Examples of information the evaluation seeks to explore: - How they did family tracing activities - Preparations for reintegration that they were involvement in (preparations with the child and preparations with the family) - Strengths in the way these processes were implemented - Weaknesses and challenges in the way these processes were implemented - As compared to pre-2014, how their professional practices and involvement in reintegration activities has changed, been strengthened 6. Overall, how many children have been reintegrated or placed in alternative family based care from remand homes as a result of DOVCU activities and what has been the experience of these cases of reintegration? Please talk both about the positive experience and the challenges that were faced in the process. Examples of information we are interested in: - Are there less children in remand homes now as compared to three years ago? - Do fewer children enter remand homes now as compared to three years ago? - Have the reasons for why children are placed in remand homes changed (e.g. now only for heavier crimes, or also petty crimes, behavioural or statutory offences) - Do more children leave a child care placement (outflow) now as compared to three years ago? - Do children more now than three years ago, leave remand homes quicker and easier or does it still happen that children miss their trials because of fuel shortages or inability to get to the court room for hearings? 85 - For the children who are going home from remand homes, do they have a plan for how reintegration should be done to ensure safe and permanent reintegration? 7. We are also interested to know what are the changes in remand homes as a result of DOVCU￾project activities: Examples of information the evaluation seeks to explore: - The extent to which DOVCU has contributed to improvements in knowledge, skills and utilization by remand home staff of guidelines and tools (standard operating procedures) which project trained them on; - Improvements in the level of compliance with decisions on children (e.g. if children who have served their sentence are allowed to go home); - Availability of strategic partnerships and alliances for follow-up on cases of reintegration from remand homes. 8. To what extent was the project able to prevent child protection issues and human rights violations (e.g. child labour, violence against children, neglect, early marriage, etc.) and mitigate risks for creating greater vulnerability (e.g. child labour and any kind of harm)? Examples of information the evaluation seeks to explore: - How were such risks assessed and monitored - If risk was identified, how was it reduced 9. At this point in time, does there exist, in this remand home, any strategies to ensure that sustainability of the efforts that have been made with the support of the DOVCU-project? Examples of information the evaluation seeks to explore: - To what extent do the remand home perceives itself as having sufficient capacity to jointly follow-up on the situation of children who have been reintegrated in collaboration with the PSWO? - Challenges (if any) Services and supports provided through DOVCU 10. Let’s talk about the services and supports that were provided to families who received children for reintegration: To what extent do you think the DOVCU project has responded to most urgent needs of children reintegrated from remand homes? Example of information we are interested to receive: - What combinations of supports have been most useful to ensure child protection, and safe reintegration? - Were there, in your opinion, any missing elements in the support provided that should have been included in the support package to vulnerable families, in order to prevent the separation of children from these families? - To what extent did support and services offered to families need adaptation to the local context? Please give examples of how such adaptation was made. Collaboration with district professionals and structures 11. In this project, what has been your overall experience in collaboration with professionals and different structures at district and community level? Examples of information the evaluation seeks to explore: 86 - Examples of collaboration with Police and experience from this collaboration - Examples of collaboration with Judges and Courts involved in Juvenile cases and experience from this collaboration - Examples of collaboration with PSWO and experience from this collaboration - Examples of collaboration with CDOs and experience from this collaboration - Examples of collaboration with Para-social workers and experience from this collaboration - Examples of collaboration with Community child protection structures and experience from this collaboration - Strengths and challenges in this collaboration Lessons learned: 12. What are the specific lessons learned from this project that would be useful to document and share: Examples of information the evaluation seeks to explore: - Reflections on the overall design of the project and if (in retrospect) selection of intervention areas and activities were sound. - Things that worked well, or were a challenge in the project’s work with remand homes. - Suggestions for how to overcome such challenges in any future project with similar objectives. In the last part of the interview the interviewer should ask the respondent to show and illustrate some of the practices and processes of work including to show: 1. A few (between 3 and 5) casefiles for children living in the remand home, including (if available) reintegration plans: Observations (Look at whether a standard form is being used, quality of the information provided in the casefile, if the SOP seems to have been used to guide the process of reintegration): (On collaboration with other stakeholders and service providers – the respondent can be encouraged to make a drawing that illustrates flow of cases and collaboration lines) Can you list: 1. Who are the actors who collaborate with in the reintegration process for children from this remand home? 2. What happens to the cases once children are reintegrated? 3. Who provides the follow up with children and who ensures the children and their families get access to services and supports)? 87 Tool 14 – Guide for Key Informant Interviews with CBO working with children on the street (1 KII / institution in Mbale district) Introduction: Interviewer to start with an introduction, which organization he/she is representing and in which context the interview is taking place: The interview is being carried out in the context of an end-line evaluation of a project (DOVCU) that has been implemented in the area of child care and de-institutionalization by MoGLSD and a consortium led by ChildFund (RETRAK, TPO and Child’s I Foundation). The project started in 2014 and was implemented for 3 years across 12 districts of Uganda. It had for overall objective to prevent family separation and reintegrate children without appropriate family care back into nurturing family environments. The project worked with a large number of stakeholders, including district level officials, alternative care panels, probation officers, community development officers, para-social workers, community child protection structures, health facilities, child care institutions, vulnerable families and children and with children in institutions, remand homes and children found to be living in street situations. With this interview, we want to establish your involvement into the project, what and how you think it has contributed to changes in the lives of the beneficiaries. What you will tell us in the interview will remain confidential. The information you share with us will be compiled in a report that will have more than 100 respondents. We may quote you, but it will not be possible to link quotes or information discussed in the report with any specific respondent. Notes and recordings from the discussion will remain with the evaluator and will not be shared with anybody else. The interview is expected to take 1 hour. Ask for permission to start the interview, and to record the interview. Project implementation: 13. Please give an introduction of your involvement in the DOVCU-project implementation, the types of activities you were involved in, what went well and what were challenges in the project implementation. Examples of information the evaluation seeks to explore: - What briefings, trainings and other activities that they were involved in, through the DOVCU project, what were the strengths and what were the weaknesses of these? - How many staff in this CCI were trained? - On what topics were the trainings? - Particularly, what new did you learn from trainings? - Have you changed any of your processes / practices as a result of these trainings? - What changes did you introduce? - What were the strengths and what were the weaknesses of these trainings? - What activities and strategies used by the project have you found to be the most useful, to for example change the way you work (assessments, coaching, training, technical assistance activities etc.) 14. From this project, what have you understood as being your role in supporting the implementing the Standard Operating Procedures for the Reintegration of Children from Remand Homes and the Street? Examples of information the evaluation seeks to explore: - Their described role in tracing and reintegration activities 88 - Challenges or opportunities in their role to implement this policy 15. How would you describe your overall experience of being involved in the project? Examples of information the evaluation seeks to explore: - Were the human resources (e.g. social work capacity, staff providing training and coaching activities), and other resources invested by the project sufficient for there to be a lasting change in professional practices & were these resources complementary to what you can mobilize themselves. - Is there any activity that you think should have been included that was not included? - Did you provide this feedback to the DOVCU-project staff? What did they do? Involvement and experience of work with children to be reintegrated from the street 16. Were any children reintegrated from this CBO reintegrated from the street with the support of the DOVCU project? Examples of information the evaluation seeks to explore: - What reintegration activities the stakeholder was involved in 17. Please describe your experience regarding the reintegration of children from the street that was supported by this CBO with the support of DOVCU: Examples of information the evaluation seeks to explore: - How they did family tracing activities - Preparations for reintegration that they were involvement in (preparations with the child and preparations with the family) - Strengths in the way these processes were implemented - Weaknesses and challenges in the way these processes were implemented - As compared to pre-2014, how their professional practices and involvement in reintegration activities has changed, been strengthened 18. Overall, how many children have been reintegrated or placed in alternative family based care from the street as a result of DOVCU activities and what has been the experience of these cases of reintegration? Please talk both about the positive experience and the challenges that were faced in the process. This question is only for Mbale district Examples of information that we are interested to receive: - If there is an overall reduction in the phenomenon of children recorded to be on the street in the districts where DOVCU operated - What was the experience of implementing the permanency plan – successes and challenges? - In how many cases was it not possible to implement the permanency plan, what were the challenges? - Now that the project is ending, for how long will children continue to be followed up by ChildFund and the project partners, by PSWOs, CDOs or other government staff? 89 - What will allow for such follow-up after the project has ended (e.g. availability of sustained mobility of PSWOs and CDOs, institutionalized government guidelines for case management, reintegration and permanency planning etc.) 19. Of the children that the DOVCU project reintegrated do all children have a permanency plan and what has been the experience of implementing the permanency plan? Examples of information the evaluation seeks to explore: - What were successes and challenges? - In how many cases was it not possible to implement the permanency plan, what were the challenges? - Now that the project is ending, for how long will children continue to be followed up by ChildFund and the project partners, by PSWOs, CDOs or other government staff? - What will allow for such follow-up after the project has ended (e.g. availability of sustained mobility of PSWOs and CDOs, institutionalized government guidelines for case management, reintegration and permanency planning etc.) 20. What has been the effect of the reintegration of children on the children and the families where children were received? Examples of information the evaluation seeks to explore: - What kind of support did children receive (pre-, during and post reintegration activities)? - What kind of support did families receive (pre-, during and post reintegration activities)? - How successful were the reintegration activities overall? According to your assessment, did the wellbeing of reintegrated children improve after DOVCU interventions? - In what way did their wellbeing improve? Was there any area of wellbeing that showed greater improvements than other? - Did all reintegrated children remain in their placements? - Do you know of any case where the child did not remain in his/her placement? What happen? - What kind of actions did the project take in the cases where children did not show improved well￾being after they had been reintegrated - Do you know of any case where the reintegration of the child did not show improvements in the well-being status of the child? What happen? How did the project address this situation? - Do you know of any case where a reintegrated child returned to institutions or to the streets? What happen? How did the project address this situation? 21. For the children who have been successfully reintegrated with their families or placed in alternative family based care, what made the reintegration method successful and sustainable? Examples of information the evaluation seeks to explore: - What makes these reintegration methods sustainable? - For which children was impact most visible? - Why do you think reintegration efforts were more successful for some children? - What were the circumstances that influenced this? 22. To what extent was the project able to prevent child protection issues and human rights violations (e.g. child labour, violence against children, neglect, early marriage, etc.) and mitigate risks for creating greater vulnerability (e.g. child labour and any kind of harm)? 90 Examples of information the evaluation seeks to explore: - How were such risks assessed and monitored - If risk was identified, how was it reduced 23. We are also interested to know what are the changes in CBO practices and their work with children on the street as a result of DOVCU-project activities: Examples of information the evaluation seeks to explore: - The extent to which DOVCU has contributed to improvements in knowledge, skills and utilization by staff of guidelines and tools (standard operating procedures) which project trained them on; - Availability of strategic partnerships and alliances for follow-up on cases of reintegration from the street. 24. At this point in time, does there exist any strategies to ensure that sustainability of the efforts that have been made with the support of the DOVCU-project? Examples of information the evaluation seeks to explore: - To what extent does the CBO perceive itself as having sufficient capacity to jointly follow-up on the situation of children who have been reintegrated in collaboration with the PSWO? - Challenges (if any) Services and supports provided through DOVCU 25. Let’s talk about the services and supports that were provided to families who received children for reintegration: To what extent do you think the DOVCU project has responded to most urgent needs of children reintegrated from remand homes? Example of information we are interested to receive: - What combinations of supports have been most useful to ensure child protection, and safe reintegration? - Were there, in your opinion, any missing elements in the support provided that should have been included in the support package to vulnerable families, in order to prevent the separation of children from these families? - To what extent did support and services offered to families need adaptation to the local context? Please give examples of how such adaptation was made. Collaboration with district professionals and structures 26. In this project, what has been your overall experience in collaboration with professionals and different structures at district and community level? Examples of information the evaluation seeks to explore: - Examples of collaboration with Police and experience from this collaboration - Examples of collaboration with Judges and Courts involved experience from this collaboration - Examples of collaboration with PSWO and experience from this collaboration - Examples of collaboration with CDOs and experience from this collaboration - Examples of collaboration with Para-social workers and experience from this collaboration - Examples of collaboration with Community child protection structures and experience from this collaboration - Strengths and challenges in this collaboration 91 Lessons learned: 27. What are the specific lessons learned from this project that would be useful to document and share: Examples of information the evaluation seeks to explore: - Reflections on the overall design of the project and if (in retrospect) selection of intervention areas and activities were sound. - Things that worked well, or were a challenge in the project’s work with remand homes. - Suggestions for how to overcome such challenges in any future project with similar objectives. 92 VI. Interviews to be conducted at community level Tool 15 – Guide for Focus Group Discussions with Para-social workers (1 FGD / parish) Introduction Hello. My name is ______________________, and this is my colleague _______________ and we are working with Oxford Policy Management and Makarere University. We are conducting a study about child protection and caregiver wellbeing in Uganda to help the NGO ChildFund understand and improve the impact of their services and programmes. To gather this information we are speaking with men and women in different communities. We would very much appreciate your participation in this survey. Participation involves a discussion about parenting and child protection in this community. The discussion with you will take about one hour to complete. If you agree to participate, we will ask you questions from a printed questionnaire and we will note your answers. Other people will not know if you are in this study. We will put things we learn about you together with things we learn about para-social workers in other communities, so no one can tell what things came from you. When we tell other people about this research, we will never use your name, so no one will ever know what answers you gave me. Only a few researchers will have access to this information, and all information will be stored safely under the care of the lead researcher in Kampala. Do you have any questions? Do you agree to be part of the discussion today? Awareness of parenting skills (following training received), and ability to put it into practice through interactions with caregivers I would like to understand your role in supporting parents or caregivers who are at risk of separation or have been separated from their children. 1. Do you conduct home visits? If no, why not? If yes, how frequently? Which households do you visit, and how are they selected for visits? Do these include visits to children’s and youth groups? Has the frequency and content of your home visits changed over the last 18 months? If yes, what has caused this change? If no, why not? 2. How many children have you followed through home visits over the last 6 months? Of those, how many did you refer on to other services? In the case of referral, do you know if a child/family received services they were referred to? How many cases were you able to resolve without referring on to other services? 3. Can you describe how a typical home visit proceeds? 4. What are the challenges to being a good father/mother/caregiver in this community? How do people cope with these challenges? Has this changed in the last 18 months? 5. How do you assess whether someone has good parenting skills or whether the parenting skills need to be strengthened? What is the process? Can you give an example of when you identified good and poor parenting skills? 6. What are the main reasons a caregiver might find it difficult to have good parenting skills? Are there groups that are particularly likely to be in need of improved parenting skills? If so, who? What types of support do these caregivers receive and from whom? 7. During home visits, have you noticed any changes in caregivers’ attitude toward their children in the last 12-18 months? What kind of changes (positive or negative) have you noticed? What do you think has caused these changes? (Probe: Consider: Pay more/less attention to children’s needs; spend more/less time with their children; more aggressive towards their children; show more affection to their children; send children away from home; force children to stay inside the house; keep children from going to school; ensure children’s education despite difficulties; force/encourage children to marry at a young age; ensure children have access to recreational activities) 93 8. Have you received training on parenting skills? If yes, when/how frequently and from whom? And what did the training consist of? How do you rate the training? How could it have been improved? 9. Do you provide training to caregivers on parenting skills? If yes, what does this consist of? How do you assess whether parents have improved their skills? 10. What are the challenges to assessing parenting skills? How do you cope with these? 11. What are the challenges to providing training on parenting skills? How do you cope with these? Involvement in reintegration 12. Do you know any families that have had children who left home (because they were abandoned, neglected, abducted, mistreated, etc.) return to them in the last 12-18 months? Can you please tell us about this family/these families? 13. What support is available to such families? What is your role in providing this support? Who do you work with to provide this support? 14. What are some things that make it difficult to provide support to families with reintegrated children? 15. How could the support you provide to such families be improved? Referrals and access to services Now I would like to discuss the types of services available to at risk children and families who are vulnerable to separation in this community. 16. Do you sometimes refer families to services? If so, which services, and how frequently? Can you give an example and tell us what happened. If not, why not? 17. Have you had families referred to you? If so, when and by whom? Do these types of referrals occur often? 18. How many referrals have you made in the last 6 months? Who were you referring? To whom/where did you refer them? 19. What are the processes for referring a child or family to you, a CDO, or other services? Are these processes clear and efficient? 20. Do you work closely with the CDO? If so, how? Have there been any changes in the way you work with the CDO over the last 18 months? If so, what are these changes, and what has caused them? 21. Are families at risk of separation usually aware of the services available to them? Do you conduct activities to make these families aware of services? 22. Are there major barriers to accessing certain services? Are these mostly practical (e.g. long distances to medical facilities, inability to contact social workers, inability to pay school expenses, etc.)? Or social (e.g. embarrassed to be seen as needing help, etc.). Any changes to these barriers in the last 18 months? 23. Are there mental health services in this area? What do they consist of? (unpack: services for people who are so sad that their minds are having a difficult time recovering from bad things that happened to them) 24. Are there any other services needed? Are these available? Existing community-based child protection mechanisms- case management We would also like to understand all of the actors involved in child protection and support to vulnerable families, and how they work together. 25. Are there people/actors other than yourselves who are responsible for identifying vulnerable families or children living outside of family care? Who are they? Do you coordinate with them? If so, how? If not, why not? 26. Who in the community is responsible for referrals to support services for children and families at risk? (what is their role? Who do they target? How do they operate? 27. How long have these individuals/groups been providing referrals in this community? 94 28. What are the challenges to operating as a parasocial worker in this community? How do you cope with these challenges? 29. What could make these groups and the process more effective and efficient? 30. Are you aware of the District Inter-Religious Committee? How involved is this group in identifying and referring children and families who are at risk of separation or have been separated? Can you give an example of a time you have collaborated with them to identify or refer a child/family? How about other faith-based organisations? 31. Are you aware of community-based child protection structures? If so, who do these involve (e.g. Child Protection Committees, A/CDOs, LCs, religious leaders, teachers, etc.)? How do you collaborate with them? Are they effective? Any changes in your collaboration with them over the last 18 months? 32. What are the key constraints that will prevent community members from participating in a child protection mechanism now and in six months’ time? Can these be overcome? If these can’t be overcome, what can be done to engage the community informally? 33. Are men and women equally participating in these community-based child protection structures? 34. Who is responsible for case management of child protection cases? What is their role? Who do they target? How do they operate? (e.g. making and receiving referrals, permanent staff, volunteers, etc.) 35. What is the process that a case goes through? (e.g. identification, referral, follow-up, continued support, placement in family-based care, etc.)? What is you role at each stage? Which other individuals or groups do you work with at each stage? Do all cases follow this process? Why or why not? 36. Can you please tell me what are the places where people can get help if they have suffered physical, sexual or emotional violence? And if so, can girls and/or boys access such services on their own? Status of interactions (started or not started) with project coordinators and briefings received on the project Finally, we would like to discuss the a programme that is working in the area of child protection in this community. 37. Do you know about the NGO called [insert name of relevant implementing partner] If so, what do you know about this NGO’s work in this community? How do you know this? 38. Are you aware of [insert name of relevant implementing partner]’s project to provide help to families who are at risk of being separated from their children, and to help separated children return to family care? If so, what do you know? How did you hear about it? 39. Have you worked with staff of the programme regarding this project? 40. What do you think is the most significant impact the project has had on vulnerable families in this community? 95 Tool 16 – Guide for Focus Group Discussions with Community Child Protection Mechanisms (1 FGD / parish) Introduction Hello. My name is ______________________, and this is my colleague _______________ and we are working with Oxford Policy Management and Makarere University. We are conducting a study about child protection and caregiver wellbeing in Uganda to help the NGO ChildFund understand and improve the impact of their services and programmes. To gather this information we are speaking with men and women in different communities. We would very much appreciate your participation in this survey. Participation involves a discussion about parenting and child protection in this community. The discussion with you will take about one hour to complete. If you agree to participate, we will ask you questions from a printed questionnaire and we will note your answers. Other people will not know if you are in this study. We will put things we learn about your community together with things we learn about people from other communities, so no one can tell what things came from you. When we tell other people about this research, we will never use your name, so no one will ever know what answers you gave me. Only a few researchers will have access to this information, and all information will be stored safely under the care of the lead researcher in Kampala. Do you have any questions? Do you agree to be part of the discussion today? Background 1. What is your role in the community? How long have you been in this role? (note each participant) 2. Are you involved in any child protection activities in this community? What? 3. Have you ever received any training on child protection issues? If so when, and from whom? What did you learn in this training? Current understanding on risks that can lead to family separation, and capacity to assess and respond to these risks 4. Are there children between 0-17 in this community who have been separated from their usual caregivers in the last (12 months)? How many do you think there have been? 5. What do you think are the main causes of separation that occurred in the last 12 months? How do you know? (PROBE: intentional separation; sending children away; infant separation and what care arrangements were made for the infant; circumstances of separation; how common are these issues) 6. If families have to send children away, where are they most likely to send them? (to institutional care, extended family/friends, etc.?) 7. What are the biggest/main sources of stress for caregivers in the community? Consider: financial, ongoing conflict; lack of food; lack of shelter; loss of property; lost livelihood; children’s safety; violence within the community; being separated from family or community 8. What in your opinion constitutes a child protection risk to a child, which requires some kind of action? 9. How do you identify a family that is at risk of being separated from their children? What processes do you go through? How did you learn to do this? Has the way you identify at-risk families changed over the past 12-18 months? 10. What actions do you take once you have identified a family at risk of separation? Do you sometimes take no action? If so, why? 11. Which of these actions do you think are most effective at preventing separation? Why? 12. Do you conduct any type of monitoring of families that are at risk for separation or have been separated? What does this involve? Which individuals or organisations do you work with to do this? 96 13. Do you encourage at-risk families to consider family-based care rather than separation? How do you do this (in writing, verbally)? Are people usually responsive? Why/why not? Has this changed over the last 12-18 months? Perceived needs for capacity building and current understanding of how to do home visit and psychosocial support. Let us now discuss more about your role in child protection in this community. 14. Do you conduct home visits? If no, why not? If yes, how frequently? Which households do you visit, and how were they selected for visits? Do these include visits to children’s groups and youth groups? Has the frequency and content of your home visits changed over the last 18 months? If yes, what has caused this change? If no, why not? 15. Can you describe how a typical home visit proceeds? 16. Have you received formal instruction on how to conduct home visits? If yes, from whom and when? Was the training helpful? Why or why not? 17. How confident do you feel in conducting home visits? Why? Can you give an example of a time a home visit had a big impact on a family? 18. What would help you feel that you are doing a better job conducting home visits? During home visits, have you noticed any changes in caregivers’ attitude toward their children in the last 12-18 months? What kind of changes (positive or negative) have you noticed? What do you think has caused these changes? (Probe: Consider: Pay more/less attention to children’s needs; spend more/less time with their children; more aggressive towards their children; show more affection to their children; send children away from home; force children to stay inside the house; keep children from going to school; ensure children’s education despite difficulties; force/encourage children to marry at a young age; ensure children have access to recreational activities) 19. Do you provide emotional support to caretakers who are distressed and struggling to cope with the challenges of parenting or being separated from children in their care? What does this support look like? (e.g. listening, providing counselling, referring to other services, connecting the person with others in the community) Has this changed over the past 12-18 months? 20. Have you ever received formal training on how to provide emotional support to caregivers? If so, from whom and when? Was the training helpful? Why or why not? 21. How confident do you feel in providing this type of support? Why? Can you give an example of when you were able to help a family this way? 22. What would help you feel that you are doing a better job providing this type of support to these families? 23. How do you record and manage information you collect about children and families? Involvement in reintegration 24. Do you know any families that have had children who left home (because they were abandoned, neglected, abducted, mistreated, etc.) return to them in the last 12-18 months? Can you please tell us about this family/these families? 25. What support is available to such families? What is your role in providing this support? Who do you work with to provide this support? 26. What are some things that make it difficult to provide support to families with reintegrated children? 27. How could the support you provide to such families be improved? Access to services Now I would like to discuss the types of services available to at risk children and families who are vulnerable to separation in this community. 97 28. Do you sometimes refer families to services? How many referrals have you made in the last 6 months? If so, who were you referring and which services did you refer them to? Can you give an example and tell us what happened. If not, why not? 29. In particular, have you ever referred families to social workers, PSWO or CDO? If so, when? How often do you refer cases to them? What happens after you have referred a case? Do you ever get feedback on their referrals? 30. What are the processes for referring a child or family to a social worker, CDO, or other services? Are these processes clear and efficient? 31. How often do you meet, discuss with or receive any form of guidance from a PSWO or a CDO. Has this changed (e.g. increased or decreased) over the last 12-18 months? 32. Are families at risk of separation usually aware of the services available to them? Do you conduct activities to make these families aware of services? 33. What are the there major barriers to accessing certain services? (PROBE: mostly practical, e.g. long distances to medical facilities, inability to contact social services, inability to pay school expenses, etc.? Or social e.g. embarrassed to be seen as needing help, etc.) Existing community-based child protection mechanisms- case management and support services We would also like to understand all of the actors involved in child protection and support to vulnerable families, and how they work together. 34. Are there people other than yourselves who are responsible for identifying vulnerable families or children living outside of family care? Who are they? Do you coordinate with them? If so, how? If not, why not? Has this changed over the past 12-18 months? 35. Who in the community is responsible for case management of child protection cases? What is their role? Who do they target? How do they operate?(e.g. making and receiving referrals, permanent staff, volunteers, etc.) 36. Can you please describe the case management process to me? (e.g. identification, referral, follow￾up, continued support, placement in family-based care, etc.)? What is you role at each stage? Which other individuals or groups do you work with at each stage? Do all cases follow this process? Why or why not? 37. Are there places where people can get help if they have suffered physical, sexual or emotional violence? And if so, can girls and/or boys access such services on their own? 38. Who in the community is responsible for referrals to support services for children and families at risk? (what is their role? Who do they target? How do they operate? 39. How long have these individuals/groups been providing referrals in this community? 40. What are the challenges to operating in this community? How do they cope with these challenges? 41. What could make these groups more effective and efficient? 42. Are you aware of the District Inter-Religious Committee? How involved is this group in identifying and referring children and families who are at risk of separation or have been separated? Can you give an example of a time you have collaborated with them to identify or refer a child/family? How about other faith-based organisations? Status of interactions (started or not started) with project coordinators and briefings received on the project Finally we would like to discuss the programme we are doing this research for. 43. Do you know about the NGO called ChildFund? If so, what do you know about this NGO’s work in this community? How do you know this? 98 44. Are you aware of ChildFund’s project to provide help to families who are at risk of being separated from their children, and to help separated children return to family care? If so, what do you know? How did you hear about it? Have you benefitted yet from any activities of this project? Thank you for participating. Do you have any questions for me? 99 Tool 17 – Guide for conducting Case Studies with Households (caregivers): (8 household case studies / parish) Introduction & Consent Hello. My name is ______________________, and this is my colleague _______________ and we are working with Oxford Policy Management and Makarere University. We are conducting a study about child and caregiver well-being to help the NGO ChildFund understand and improve the impact of their services and programmes. To gather this information we are interviewing caregivers and children in some households. We would very much appreciate your participation in this survey. Participation involves an interview with you about your household. We will ask questions about all the children under 18 in this household, and would like any children over age 10 to join our discussion if possible. The interview with you will take about one hour to an hour and a half to complete. If you agree to participate, we will ask you questions from a printed questionnaire and we will note your answers. Some of the questions are personal and some people may find them difficult to answer. You do not need to answer any questions that you do not want to. The risks to you as a participant in this study are minimal. During the interview you may decide to share information that is personal in nature. But, again, you may skip any questions that you do not wish to answer or stop the interview at any time, without giving any reasons. Other people will not know if you are in this study. We will put things we learn about you together with things we learn about other people from your community and from communities in 6 districts altogether, so no one can tell what things came from you. When we tell other people about this research, we will never use your name, so no one will ever know what answers you gave me. Only a few researchers will have access to this information, and all information will be stored safely under the care of the lead researcher in Kampala. Your participation in this study will not benefit you directly, but it may benefit others in the future, as your responses will improve ChildFund’s and the government’s understanding about ways to provide better services to people in communities like yours. Your participation in this study is voluntary. If you don’t want to be in the study, it is OK. If you want to be in the study now and change your mind later, that’s OK too. You can stop at any time. If you agree for you and your children to participate, you can decide not to answer certain questions and can stop the interview at any time. Your decision about whether to participate in this study or to answer any specific questions will in no way affect any services that you receive. If you do choose to participate, please answer the questions honestly and openly, so that we can find out what people here in [community name] really think. Before you say yes or no to being in this study, we will answer any questions you have. If you join the study, you can ask me questions at any time during the interview. Do you have any questions now? Do you understand everything I have explained? Do you agree to participate in this interview? Do you agree for the children aged 10-18 years in your care to participate in this study? Discussion A: Household composition and overall wellbeing 1. Who lives in this household? How many people live in the household? How you are related or connected to each other? Has the number of people living in this household changed in the last year? Why is this? REMINDER: For each household member, probe age, gender, relationship to respondent 2. Do all the biological children between the ages of 0 and 17 born to the adults in this household live here? If not, where do they live and why don’t they live here? 3. What types of work and activities do the other people in your household do to earn a living? Does everyone do the same work, or do different people do different work and activities? Do the children contribute to the household income? How? 4. Who is the main provider for the household? 100 5. How would you describe the wellbeingg of your household? (provide examples if needed, e.g. struggling, poor, doing just ok, doing well, very poor, etc.) Why would you describe it this way? 6. How does this compare with others in your community? 7. Compared to last year, do you feel that your household is more or less financially secure, or about the same? Why do you think this? What, if anything has changed? 8. [IF cash transfer not mentioned in answer to previous question] Do you receive financial support from [implementing partner]? How long have you been a recipient? How much and how regularly? Does this support affect your household’s financial security? If yes, how? [PROBE: Things you are able to do that you couldn’t do before, consumption smoothing, etc.] If no, why not? How could the financial support be improved? 9. Do you receive any training on how to manage your household’s money, or how to improve your income? If yes, who provides this training and how regularly is it offered? How long have you participated? What kinds of things do you learn in these trainings? Can you give some examples of things you have done differently, or things that have changed in your household as a result of these trainings? 10. Does the household ever run out of money to buy food, rely on a limited number of foods, or cut size/skip meals? Do children eat less than they should? Has this changed in the last year? 11. What do you think your household’s situation will be in the future? Why is this? 12. What has been the most significant change in your household’s wellbeing in the last 12-18 months? 13. What are the main sources of stress experienced by caregivers/young children/adolescents (especially girls) in this household? Who do they turn to for support? Why do you turn to this person/group? What makes their support to you so effective or desirable? [PROBE: support from community support structures] How would you compare your stress as a parent now to the stress you experienced 12-18 months ago? What has caused these changes? Discussion B: Child Care & Parenting Support Now we would like to discuss childcare and parenting. 14. Who has primary child care responsibilities in this household? Are these responsibilities shared? By whom? 15. What do these responsibilities involve? (for example, feeding, educating, discipline, supervising, etc.) 16. What do you consider good parenting? Bad parenting? Why? Has the way you think about this changed in the last 18 months? If yes, what has caused these changes? 17. How does someone learn to be a good parent? 18. What factors influence your ability to be a good parent? How do you cope with these factors? Has the way you coped changed over the last 18 months? If yes, how, and what has caused these changes? 19. As a parent or caregiver of children, what helps you to feel supported? 20. Do you receive support or assistance from others within the community to help you when your household faces difficulties? If so, who from and what kind of support do they provide? Are certain 101 members of the family more likely to turn to people outside the family for support than others? Why? [PROBE: community-based support structures, parasocial workers]. Which of these has been the most helpful to you? 21. Do you receive education on parenting skills? If so, from whom? Do you feel that this parenting education is necessary and helpful? Why? What is the most helpful? What would make it more helpful? Who gives you the best advice on caring for children? 22. Can you share examples of things you have done differently because of what you learnt in the classes? 23. Are you part of a Parents Group? If no, would you find it helpful to be part of such a group? Why/why not (if not, probe reasons why not)? If yes, can you tell us about the group—who are the members, what is the purpose of the group? 24. If yes, what kind of support do you receive from this group? Is it helpful? How could it be more helpful? Which, if any, aspects of the group help you to be a better caregiver? Which aspects, if any, are not so useful or relevant to you? 25. Do you receive similar support elsewhere or from people not in this group? How would you compare the support from these other sources? 26. Sources of formal and informal support Tell me about the most significant sources of support to your household over the last 12 months. What are the characteristics of these people/institutions/organisations that makes them most helpful? 27. Are there ny kinds of support that are lacking? 28. Are you able to give feedback to those providing you with support? If no, why not? If yes, how do you give this feedback? Are they responsive? 102 Tool 18 – Tool for activity for reintegrated children Introduction & Consent Hello. My name is ______________________, and this is my colleague _______________ and we are working with Oxford Policy Management and Makarere University. We are conducting a study about child and caregiver well-being to help the NGO ChildFund understand and improve the impact of their services and programmes. To gather this information we are interviewing caregivers and children in some households. We would very much appreciate your participation in this survey. Participation involves an interview with you about your household. The exercise with you will do with you will take about half hour to complete. If you agree to participate, we will ask you questions from a printed questionnaire and we will note your answers. Some of the questions are personal and some people may find them difficult to answer. You do not need to answer any questions that you do not want to. The risks to you as a participant in this study are minimal. During the interview you may decide to share information that is personal in nature. But, again, you may skip any questions that you do not wish to answer or stop the interview at any time, without giving any reasons. Other people will not know if you are in this study. We will put things we learn about you together with things we learn about other people from your community and from communities in 6 districts altogether, so no one can tell what things came from you. When we tell other people about this research, we will never use your name, so no one will ever know what answers you gave me. Only a few researchers will have access to this information, and all information will be stored safely under the care of the lead researcher in Kampala. Your participation in this study will not benefit you directly, but it may benefit others in the future, as your responses will improve ChildFund’s and the government’s understanding about ways to provide better services to people in communities like yours. Your participation in this study is voluntary. If you don’t want to be in the study, it is OK. If you want to be in the study now and change your mind later, that’s OK too. You can stop at any time. If you agree for you and your children to participate, you can decide not to answer certain questions and can stop the interview at any time. Your decision about whether to participate in this study or to answer any specific questions will in no way affect any services that you receive. If you do choose to participate, please answer the questions honestly and openly, so that we can find out what people here in [community name] really think. Before you say yes or no to being in this study, we will answer any questions you have. If you join the study, you can ask me questions at any time during the interview. Do you have any questions now? Do you understand everything I have explained? Do you agree to participate in this interview? Exercise 3: ‘H-Assessment’—change in wellbeing & perceived quality of care This exercise explores the positive and negative changes children have experienced, by comparing their perceptions of their wellbeing before and after living in a child care institution, and gives children the opportunity to ask questions or raise suggestions about the care they are currently receiving. The facilitator draws a table on a large piece of paper and explains the activity to the children participating. 103 Before Now Future Now I would like you to think about some changes that might have happened since you came to stay here. These can be good changes, or sometimes they can be bad changes. We will also think about what you would like for the future when you are no longer living in this home. Also think about what you would change in each of these areas we will talk about. 1. Food & nutrition: Do you eat different things? Do you eat more now than before? More/less frequently? Going to bed hungry? Why do you think that is? Do you worry about having food at home in the future? 2. Health: Have you been more or less ill since you came to live here? Has the treatment you received changed? (more or less able to receive medical treatment) Would it be important for you to get medical treatment in the future? 3. Education: Has your schooling changed? Do you attend school more or less frequently? Has your ability to learn and focus in school changed? Why do you think that is? What would you change about schooling? Would you want to go to school in the future? Do you worry that you might not be able to? Why/why not? 4. Child labour: What kind of work were you doing before? Has the amount of work you do changed since you came here? Would you want to work in the future? Why or why not? 5. Adult supervision: Before coming here, was there an adult who always knew where you were or what you were doing? How about now? How do you feel about this? In the future, would you want an adult know where you are at all times? Why/why not? 6. Care: Who were the important adults in your life before? Who are they now? Do you spend more or less time with the important adults in your life now? Who would you want to stay with in the future? Why? 7. Violence against children: Has your experience of violence or abuse changed since you came here? How? Do you worry that you will experience violence or abuse in the future? Why/why not? 8. Stress: What things caused you stress before? Now? Why do you think these things have changed? Do you think these will be the same things that stress you in the future? 9. Support (formal and informal): Has the way you deal with stress and get support changed since you came here? How? Do you think this is how you will deal with stress in the future? 10. Harmful traditional practices: Are there things that are traditionally done to children (especially girls) that you experienced or were worried about before that you are not worried about living here? (for example, being married at a young age, not receiving enough food, being circumcised) Do you worry about these things in the future? 11. Which of these changes have been the most important or made the biggest difference in your day to day life? Why? 104 12. Which of these issues concerns you most about the future? Why? 105 Tool 19 – Guide for Focus Group Discussions with women and men in the community (1 FGD / parish per gender) Female caregivers of children 0-17 Introduction and Consent Hello. My name is ______________________, and this is my colleague _______________ and we are working with Oxford Policy Management and Makarere University. We are conducting a study about child and caregiver well-being to help the NGO ChildFund understand and improve the impact of their services and programmes. To gather this information we are speaking with men and women in different communities. We would very much appreciate your participation in this survey. Participation involves a discussion about parenting and child protection in this community. The discussion with you will take about one hour to complete. If you agree to participate, we will ask you questions from a printed questionnaire and we will note down your answers. Some of the questions are personal and some people may find them difficult to answer. You do not need to answer any questions that you do not want to. The risks to you as a participant in this study are minimal. During the interview you may decide to share information that is personal in nature. But, again, you may skip any questions that you do not wish to answer or stop the interview at any time, without giving any reasons. Other people will not know if you are in this study. We will put things we learn about your community together with things we learn about people from other communities, so no one can tell what things came from you. When we tell other people about this research, we will never use your name, so no one will ever know what answers you gave me. Only a few researchers will have access to this information, and all information will be stored safely under the care of the lead researcher in Kampala. Your participation in this study will not benefit you directly, but it may benefit others in the future, as your responses will improve Child Fund’s understanding about ways to provide better services to people in communities like yours. Your participation in this study is voluntary. If you don’t want to take part in the study, it is OK. If you want to participate in the study now and change your mind later, that’s OK too. You can stop at any time. Your decision about whether to participate in this study or to answer any specific questions will in no way affect any services that you receive. If you do choose to participate, please answer the questions honestly and openly, so that we can find out what people here in [community name] really think. Before you say yes or no to being in this study, we will answer any questions you have. If you join the study, you can ask me questions at any time. Do you have any questions now? Do you understand everything I have explained? Do you agree to participate in this discussion? Thank you. Now we will start. Discussion A: Community-wide perceptions of the needs of families that are vulnerable to separation & attitudes towards them 1. How would you describe a household that is struggling to provide for all the members of the family, or very poor? What are the characteristics of such a household? 2. What are the characteristics of parents/caregivers who are vulnerable to child separation in this community? Does this mean that the child is for some reason likely to become separated from their caregiver? 106 3. Do you know of households that have had children not grow up with their parents or guardians, when ordinarily they should have grown up under their care? Do you have an idea where those children are now? Did the community try to support them? If yes, how? If not, why not? 4. What are the reasons that could lead to such separation? If children are not under the care of legal guardians or parents, where do such children normally end up? NB: the use of the word separation in the local contexts may not communicate well what is acceptable and what is not in terms of child upbringing practices. Culturally, young children are taken up by their relatives for upbringing. This form of separation is not considered bad and it could be that the child has attained an acceptable age say 12 years that he is sent to live with his uncle. The separation we discuss here would be the involuntary or would include those situations where children do not grow up with their parents on account of maltreatment in the home/violence, abandonment etc. 5. Are you aware of any projects that are working to support these children and their families? IF YES 6. What kind of support is being provided to these families, and by whom? [PROBE: financial, psychosocial, parenting skills and legal] 7. Do you think this support is helpful? Can you give examples of how it has helped? 8. How could the support be improved? 9. What do vulnerable households do for help? Who do they turn to? Are there families that are less likely to seek or be able to access support? Who/why? 10. Do you know any families that have had children who left home (because they were abandoned, neglected, abducted, mistreated, etc.) return to them? Can you please tell us about this family/these families? 11. Has the community been able to support them in this reunification? Why/why not? If yes, how?In which ways is the community supportive in reuniting such children with their parents? What are some of the reasons/instances that the community sometimes fails to effectively intervene? 12. Do you/would you let your children play with re-unified children? Why/why not? ][PROBE: Does it depend on the reason the child was separated from his/her family?] 13. How should children be treated when they are reintegrated and re-enter the community? a. Is it easy or difficult to see the child as fitting back in to the community? Why? b. What are the reasons a community would/would not treat a re-integrated child well? c. Is it different for girls and boys? 14. Would the community’s attitude be different or the same towards reintegrated children who have committed a crime (compared to children who have been victims)? Why or why not? What about children with disabilities or those living with HIV/AIDS? 15. Do you think re-integrated children in this community face a lower, higher, or the same risk as other children as being targets of the community for: physical violence? Sexual violence? Emotional abuse? Does this depend on the reason the child was separated from his/her family? Is it different for girls and boys? 16. Is a child who has been re-integrated into his/her family likely to have a good life settling back into the community? What factors determine this? 107 17. What support have families whose children are being re-integrated into the community received? From whom? 18. Has this support been effective? Why/why not? How could it have been improved? Discussion B: Community-wide knowledge of and attitudes towards child protection Now we want to talk about something we call ‘child protection.’ Child protection is a term that describes measures that are taken to ensure that children have are not exposed to any kind of harm, and the measures taken to care for them if they do experience harm. Do you understand this term now? 19. What are the child protection concerns (anxieties, fears, apprehensions, alarms or distresses) in this area for girls (young and adolescent)? For boys (young and adolescent)? 20. Which traditional norms, values and practices in this community promote protection of children from exposure to harm (injury or maltreatment)? Which traditional norms, values and practices are harmful to children? 21. How are children protected from harmful practices (e.g. sexual abuse, physical violence, exploitation and emotional violence) in your community? (e.g. using traditional mechanisms such as elders, cultural leaders, aunts, uncles, religious mechanisms) Can you give examples of a child who received help using each/ any of these mechanisms and how it was done? What are the positive experiences? What are the negative experiences? Give an example of a child who was not helped. What were the effects – on the child, parents/guardians, community? 22. How do people here know when a child is likely to be abused or is beginning to experience violence and abuse or have negative impact to their wellbeing (i.e. not eating enough, not going to school, etc.)? What are the responses in these situations? How do they function?(they meaning who? The children or the people?) Has this changed in the last 12-18 months? 23. Are there persons unknown to the community who have offered to take children away from this community? Tell us what happened. Who came? What did they want? What did they offer? What happened? Were children taken away? How many girls? How many boys? What age group? Discussion C: Social norms and challenges related to caregiving 24. In most households here, who is responsible for taking care of children? What do these responsibilities involve (e.g. feeding, discipline, educating, playing, etc.)? 25. What are the characteristics of a good father/mother/caregiver? Are there ways one learns to be a good father/mother/caregiver? 26. What are the challenges to being a good father/mother/caregiver in this community? Discussion D: Need for support among children and caregivers across the community 27. What are the main sources of stress for children in the community? How are girls and boys affected— young and adolescent? (Consider: Attacks, kidnapping/abductions, trafficking, not being able to go back to school, not being able to return home, losing their belongings, being separated from their friends, being separated from their families, tension within the family, nightmares or bad memories, sexual violence, extra hard work, lack of shelter, going far from home for work, lack of food, bullying) How do 108 they deal with these stresses? Have these stresses and the ways that children cope changed over the last 12-18 months? 28. How might the most urgent needs of girls be addressed in this community? Of boys? 29. How do parents/caregivers cope with the economic and social constraints of daily life in [community name]? How do children cope? (For themselves? With others, “auto-protection”?) Are there specific coping mechanisms for girls? Have coping mechanisms changed in recent years? If so, why and how? Discussion E: levels of awareness of sources of psychosocial and practical community-based support structures; interaction between formal and informal child protection structures at community level 30. If you came across a child at risk of being harmed or in danger of being neglected or abused, or saw that a household was unable to provide the needed protection and support to the child and this could result into a child leaving the home (separation) what would you do? In case you chose to report the incident, where would you report it? 31. If you come across an abandoned child who does not have anyone who can care for him/her, what would you do? 32. Whose responsibility is it to identify vulnerable households/children? Who is responsible for acting to support or help them? Does the community provide interim solutions to separation? If so, what kind of support is provided? 33. Who do children turn to for support inside the family? Outside the family? Who do caretakers/parents turn to for support ( to support who vulnerable household incase of a separation?)? 34. Which structures, clubs, committees or similar groups exist in this community that also sometimes get involved to try to mediate, help solving child protection cases? 35. What are the responsibilities of such groups? Who are the members? Are they effective (do they handle things? properly?) and adequate (do they provide enough support to vulnerable children and households in the community)? Can you give an example of why/why not? How could they be made more effective? 36. Have you ever referred a family for support? Who did you call upon to help solve the problem with this family? Please explain what happened. 37. Are there facilities and institutions where people can get help if they have suffered physical, sexual or emotional violence? And if so, can girls and/or boys access such services on their own? 38. How are abuses and acts of violence against children reported? What else is done to help children who are abused/ experience acts of violence? (Probe for children: formerly associated with armed groups; in conflict with the law; abused at home; abused by authority, e.g. school teacher; affected by sexual violence; abandoned by parents/guardians; living on the streets) 39. Who are the most influential people within this community when it comes to helping families who are so vulnerable that their children might have to go and live somewhere else, or children at risk? What is the role of different actors within the community with regards to vulnerable families/separation of children? (Parents? Religious leaders? Teachers? Medical staff? LCs? Sub-county chiefs? CPCs? NGOs? Police/justice dept.? Others?) How capable are these actors in supporting parents? (Probe: Do they include women, men or both? Do they include people from different socio-economic groups? What are their aims? What kind of support do they provide—financial, psychosocial, training, etc.) 109 40. Who are the people outside this community who influence child protection here (e.g. the district government, national government, NGOs, etc.)? (Probe: what are their aims? Who do they target? How accessible are they? What kind of support do they provide—financial, psychosocial, training, etc.) 41. Which of these do you think are the most valuable? Why? 42. Which are the most accessible? What are the constraints to access? 43. Are there any gaps, any community needs that are not met? If so, what are they? 44. What could the actors you listed do to strengthen the coping capacity of vulnerable families and protect children at risk of being separated? What is the government’s responsibility to help people cope? 110 VII. Referral tool for field researchers This tool is to be used by field researchers if a child is identified to be at risk, suffer abuse or be in need of referral to services. An urgent action case is usually defined as “a situation in which lack of prompt response can put the life and/or wellbeing of a child in immediate danger.” Urgent action cases should always be immediately reported at daily debriefing sessions. The assessment team will:  Report the specific actions taken  Triangulate (i.e. compare information collected through different methods, by different people and from different sources)  Determine whether there are patterns emerging that require urgent follow-up or advocacy (e.g. risks to child protection are observed in several sites). It is essential that gender sensitivity be followed in the urgent action procedure: For example, a female member of the assessment team should handle urgent action cases involving a girl. Date: …………………………………………….. Location: ……………………………………… Name of person filing the report: …………………………………………………………………. Please fill in the first four sections, giving as many details as possible. In section 5, report any immediate action you yourself have taken and indicate any follow-up required. Hand this report to your team supervisor. If your supervisor is not available, contact national research leaders Eddy Walakira [insert mobile number] or Stella Neema [insert mobile number] as soon as possible. 1. What happened Circle all that apply and describe in the box to the right Child situation:  Concerns of abuse or neglect  Separated / unaccompanied child  Concerns of exploitation / child labor  Child without parental care  Without shelter  Not attending school  Child headed household  Medical issues  Other 111 2. Who? (by whom and to whom—please remember to note the gender of the people involved) 3. When? 112 4. Where? 5. Action taken and follow-up 113 6. Other relevant information (such as contact information and name of persons involved) Where possible, please also provide caregiver information: circle one: mother/father legal guardian grandparent other kin foster carer community member sibling none / self Other relationship:_____________________ Caregiver full name: Date of birth / age: Address or directions: County/District: Telephone: 114 Other children in the home: l ANNEX IX: SOURCES OF INFORMATION Primary data collection Gulu Kabarole Kamuli Kasese Luwero Mbale Total Community level qualitative instruments Caretaker at-risk HH 8 8 6 8 9 8 47 Child at-risk HH 8 6 6 5 6 6 37 Caretaker reunified HH 8 8 5 7 6 8 84 Child reunified HH 8 7 5 6 6 9 41 FGD Para social workers 2 2 2 2 2 2 12 FGD Community CP Mech. 2 2 2 2 2 2 12 FGD Men in community 2 1 2 2 2 2 11 FGD Women in community 2 2 2 2 2 2 12 Policy component instruments for district level Chairperson district council/commission 1 0 0 1 (secretary social services) 0 1 3 Chief Administrative Officer / Asst. CAO or DOVCC 1 1 1 1 1 1 6 DCDO 1 1 1 1 1 0 5 PSWO 1 1 (acting PSWO) 1 1 1 1 6 Representative remand home 1 1 N/A N/A N/A 1 3 FGD CDOs 1 1 1 1 1 1 6 FGD TAC 1 1 1 1 1 1 (interview) 6 Representative Child Restoration Outreach N/A N/A N/A N/A N/A 3 3 Representative CCI33 3 4 4 3 3 4 21 Policy component instruments for national level 33 CCIs visited Gulu: Children of Hope ; St. Jude’s ; Watoto Laminadera Children’s Village. Mbale: Lulwanda Children’s Home; Give Me Hope (closed interview w ex. director) ; My Father’s House ; St. Kizito Babies Home. Luwero: Another life; Children’s Ark; Home of hope kollektiv. Kamuli: His Grace Community Service; Kaako Orphanage; Kasozi (Former Uganda orphan fund); Child redeemed mission. Kabarole: Manna care center; SOS Children’s Villages; Toro babies home; Ibonde. Kasese: Nsirambi orphans talent Development Center; Chrystalis ; Nikibasika Development programme. li Reflective workshop with DOVCU Consortium N/A N/A N/A N/A N/A N/A 1 Retrak N/A N/A N/A N/A N/A N/A 1 TPO N/A N/A N/A N/A N/A N/A 1 ChildFund N/A N/A N/A N/A N/A N/A 1 Child's i Foundation N/A N/A N/A N/A N/A N/A 1 MoGLSD Officials & advisors N/A N/A N/A N/A N/A N/A 7 Total instruments 327 Document review Reference Documents ChildFund, (2013) Revised Technical Proposal – Deinstitutionalization of Orphans and Vulnerable Children in Uganda, ChildFund, Richmond, USA. Global Child Protection Working Group (2014). Interagency Guidelines for Case Management and Child Protection: The Role of Case management in the Protection of Children. Ministry of Gender, Labour and Social Development (2011) Uganda’s National Framework for Alternative Care, Kampala, Uganda Project Outputs Child’s i Foundation (draft – no date) Deinstitutionalization of Orphans and Vulnerable Children Project: CIF Training Handbook Child’s i Foundation. (no date) Working with Child Care Institutions: A strategy for the engagement with child care institutions to improve practice (document seems to be incomplete – ends abruptly) ChildFund (2015) DOVCU Case Management Tools (noted that document file name is listed as “DRAFT”) ChildFund International (2015) Village Savings and Loan Associations Field Officer Training Guide DOVCU (2015) Financial Education Manual for Project Officers and Social Workers DOVCU (2015) Key Messages Guide (draft) DOVCU (2015) Terms of Reference for the Alternative Care Panel DOVCU (2015) Visual Manual for Parenting Interactive Learning Sessions DOVCU (2015) Youths Vocational Training Guide DOVCU (2017, draft) Guidelines on Establishment of Alternative Care Panels in Uganda: Scope of Work for Consultancy DOVCU (2017) Household economic strengthening implementation guide DOVCU (no date) Guide to Social Workers. Alcohol Anonymous Meeting (AA) DOVCU (no date) Household Economic Strengthening: Micro-Enterprise Selection, Planning and Management – Trainers Manual for Savings, Credit and Investment Groups. DOVCU (no date) Terms of Reference for the District Team Around the Child (TAC) Ministry of Gender, Labour and Social Development (2015) Standard Operating Procedures for Deinstitutionalization and family reintegration from remand homes and the National Rehabilitation Centre in Uganda Ministry of Gender, Labour and Social Development (2015) Toolkit for Deinstitutionalization and family reintegration from remand homes and the National Rehabilitation Centre in Uganda lii Ministry of Labour, Gender and Social Development (2014) A holistic approach to psychosocial support: A national training manual for caregivers of OVC in Uganda (does not appear to be final draft, contains edit comments) Retrak (2015) Deinstitutionalization and family reintegration from remand homes and the national rehabilitation centre in Uganda: Facilitator guidelines for workshops with staff in remand homes and the National Rehabilitation Centre Project Reports ACF Training Reports & documentation Bugaari, A. (2016) Report for Micro-Enterprise Selection, Planning and Management. For ChildFund CCI Improvement Plans DOVCU (2015) Report on family reintegration standard operating procedures trainings DOVCU (2016) Inter-Remand Home Exchange Learning Visits Report DOVCU (2016) VSLA Staff and CDO Training Report DOVCU Quarterly progress reports to USAID/DCOF Effective Skills Development Consultants (2017) Final market assessment report (draft) Retrak (2015) Report on Family Reintegration Standard Operating Procedures Trainings conducted in remand homes, Kampiringisa National Rehabilitation Centre and Child Restoration Outreach Mbale Retrak (2016) Workshop Report on the Promotion of the International Reintegration Guidelines amongst Multi-agencies and Actors doing Reintegration Work in Uganda (Report on the workshop held at Grand Imperial Hotel, Kampala) Project reviews / evaluations Maestral International LLC (2016) Baseline report: Deinstitutionalization of Vulnerable Children in Uganda (DOVCU) Stuer, F. (2017) Reintegration of children from remand homes and the Kampiringisa National Rehabilitation Centre in Uganda: Review of Retrak’s capacity strengthening achievements under the DOVCU project liii U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523