EVALUATION Endline Report Moldova: Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project December 2017, Revised March 2018, Final May 2018 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by NORC at the University of Chicago. Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Endline Report Moldova: Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project (December 19, 2017; Revised March 2018, Final May 2018) Prepared under Contract No.: GS-10F-0033M / AID-OAA-M-13-00013. Tasking N003 Submitted to: USAID/DCOF Submitted by: Ritu Nayyar-Stone (Project Director) Mawadda Damon Gartner (Evaluator) Gary Gamer (Subject Matter Expert) Aaron Wilson (Senior Research Analyst) Carlos Fierros (Research Analyst) Samantha Downey (Research Assistant) Ilse Paniagua (Research Assistant) Contractor: NORC at the University of Chicago Attention: Renee Hendley Bethesda, MD 20814 Tel: 301-634-9489 Email: Hendley-Renee@norc.org DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. EVALUATION | i Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project TABLE OF CONTENTS EXECUTIVE SUMMARY................................................................................................. 1 1.0 EVALUATION PURPOSE & EVALUATION QUESTIONS...................................... 11 1.1 Evaluation Purpose.............................................................................................. 11 1.2 Evaluation Questions........................................................................................... 12 2.0 PROJECT BACKGROUND..................................................................................... 13 2.1 Moldova Program Context ................................................................................... 13 2.2 Moldova Program Objectives............................................................................... 14 2.3 Moldova Program Design and Management ....................................................... 15 3.0 EVALUATION METHODS & LIMITATIONS............................................................ 19 3.1 Evaluation Management...................................................................................... 19 3.2 Study Design ....................................................................................................... 19 3.3 Target Population ................................................................................................ 20 3.4 Sampling.............................................................................................................. 23 3.5 Limitations ........................................................................................................... 24 4.0 FINDINGS, CONCLUSIONS & RECOMMENDATIONS ......................................... 27 Question 1: By project end, to what extent have functioning structures been established that can continue to provide on an ongoing basis adequate case￾management services for children at risk? ................................................................ 27 Question 2: Did the program bring out systemic changes at the community, regional, and national levels that are enabling children to live in family care and preventing inappropriate placements in institutional care? ........................................ 62 Question 3: Have reintegration methods employed by the projects resulted in stable and sustained placements for children?.......................................................... 77 Question 4: Did the program measurably improve the safety, well-being, and development of highly vulnerable children, particularly those who are living without adequate family care? Did the program impact beyond direct services?................... 89 Question 5: Have prevention methods employed by the projects reduced risks of child/family separation? ........................................................................................... 119 Question 6: Did the project offer models and approaches for expansion, adaptation, and/or replication? ................................................................................ 140 EVALUATION | ii Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project TABLE OF TABLES Table 2.3.1: Level of Development of Childcare and Protection System by Project Raion ....................................................................................................... 15 Table 3.3.1: NORC Evaluation Survey Sample Sizes at Baseline and Endline ........... 21 Table 3.3.2: Number and Location of Focus Group Discussions at Baseline .............. 22 Table 3.3.3: Number and Location of Focus Group Discussions at Endline................ 23 Table 4.1.1: Support and Awareness of Child Protection Systems.............................. 35 Table 4.1.2: Structure/Situations Impacting Effectiveness of Child Protection Work ... 35 Table 4.1.3: Types of Training Received Between 2014 and 2017 ............................. 37 Table 4.1.4: Social Worker Opinions on Preparedness for Their Jobs ........................ 38 Table 4.1.5: Extent of Coordination and Collaboration between Stakeholders to Meet Child Care and Protection Needs of the Family .............................. 52 Table 4.2.1: Effectiveness of Methods in Increasing Parenting Skills and Knowledge............................................................................................... 72 Table 4.2.2: Acceptance and Support for Residential Care......................................... 73 Table 4.3.1: Number of Children with Disabilities in Residential Care ......................... 84 Table 4.3.2: Services Provided to Deinstitutionalized Children.................................... 87 Table 4.4.1: Numbers of Stakeholders Trained by Target Group ................................ 90 Table 4.4.2: Themes of main trainings provided.......................................................... 91 Table 4.4.3: Foster Care Service Development........................................................... 95 Table 4.4.4: Availability and Quality of Alternative Care in the Raion/Community ....... 96 Table 4.4.5: Challenges Faced by Deinstitutionalized Children................................. 104 Table 4.4.6: Types of Deinstitutionalized Children who face more Challenges ......... 105 Table 4.4.7: Deinstitutionalized Children’s Needs and Placements........................... 106 Table 4.4.8: Number of Children with Open Case Files by Gender across Project RCCs ..................................................................................................... 109 Table 4.4.9: Children Received Regular Medical Checkups and Necessary Treatment .............................................................................................. 109 Table 4.4.10: Feeling of Safety in the RCC and Community ....................................... 110 Table 4.4.11: Children’s Negative Emotions................................................................ 111 Table 4.4.12: Children’s Progress in School According to His/her Potential................ 112 Table 4.4.13: Case Files Reviewed in the Evaluation.................................................. 112 Table 4.5.1: Number of At-risk Cases Included for Child Well-being Analysis by Gender Across Project Raions............................................................... 120 EVALUATION | iii Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Table 4.5.2: Average Age of Children across Project Raions in Cases Included for Child Well-being Analysis ...................................................................... 121 Table 4.5.3: Parents in Household at Time of Case Opening.................................... 121 Table 4.5.4: Psycho-Emotional Status of Family at Time of Case Opening............... 122 Table 4.5.5: Parental Employment of Family at Time of Case Opening .................... 122 Table 4.5.6: Family Problems at Time of Case Opening ........................................... 122 Table 4.5.7: Child Support Service Provided at Time of Case Opening .................... 123 Table 4.5.8: Health Indicators at Baseline and Endline ............................................. 124 Table 4.5.9: Safety Indicators at Baseline and Endline.............................................. 124 Table 4.5.10:Education Indicators at Baseline and Endline........................................ 125 Table 4.5.11: Psychosocial Well-being Indicators at Baseline and Endline................. 126 Table 4.5.12: Material Indicators at Baseline and Endline........................................... 126 Table 4.5.13: Availability and Need of Services to Support Vulnerable Families – Raion Level............................................................................................ 131 Table 4.5.14: Availability and Need of Services to Support Vulnerable Families – Community Level ................................................................................... 131 Table 4.5.15:Effectiveness of Support Services to Families ....................................... 136 TABLE OF GRAPHS Graph 4.1.1: Frequency of Contact with Case Families (%)......................................... 40 Graph 4.1.2: Availability of Logistical/Financial Resources (%).................................... 56 Graph 4.1.3: Availability of Professional Resources (%)............................................... 57 Graph 4.2.1: When a Child Should Be Placed in a Residential Care Facility (% "Yes")....................................................................................................... 74 Graph 4.5.1: Adequacy of Existing Services for Special Populations of Families - Baseline................................................................................................. 133 Graph 4.5.2: Adequacy of Existing Services for Special Populations of Families - Endline................................................................................................... 133 Graph 4.5.3: Services Accessible to All Families........................................................ 135 EVALUATION | iv Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project ACKNOWLEDGMENTS We would like to begin by thanking all the social work specialists and community social assistants who took the time to respond to our survey. We would like to also thank all the children, parents, multidisciplinary team and gatekeeping commission members, and mayors who generously gave of their time to join our focus groups and share their opinions. It would not have been possible to collect all the survey and focus group data without our local data collection partner, IMAS, which we thank for working tirelessly to finalize the survey with us and track down reticent respondents. We would like to thank the P4EC team for their continual support of the evaluation, providing us with assistance, explanations, reports, and monitoring data. We would like to thank our Local Coordinator, Veronica Pelivan, for all her assistance in setting up the logistics for our team’s trip to Moldova and participation in the key informant interviews and qualitative analysis at baseline. We would like to thank Joseph Caldwell for his statistical assistance, Jeffrey Telgarsky for his guidance on report writing, and Clifford Zinnes for his guidance on the evaluation design. We would also like to thank the following individuals for their assistance in the data analysis: Letitia Onyango, Selena Zhong, and Gabriel Velez. EVALUATION | v Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project ACRONYMS ABC Advisory Boards of Children APP Asistenţi Parentali Profesionişti (Professional Parental Assistants – Foster Parents) CCTF Casa de Copii de Tip Familial (Family Type Children’s Home) CCF Christian Children’s Fund (formerly Hope and Homes for Children) CPCD Commission for the Protection of the Child in Difficulty CSAS Community Social Assistance Service CSA Community social assistants CSO Civil Society Organization DCOF Displaced Children and Orphans Fund DFID Department for International Development ED Education Directorate (District government) EC European Commission EvC EveryChild, UK charity FGD Focus Group Discussion GD 270 Government Decision no. 270 IGACC International Guidelines for the Alternative Care of Children IMAS Institutul de Marketing si Sondaje “IMAS-INC” SRL INGO International Non-governmental Organization IRC International Rescue Committee KAP Knowledge Attitude and Practice KII Key Informant Interview GoM Government of Moldova LA Local Authority LPA Local Public Authority M&E Monitoring and Evaluation MDT Multidisciplinary Team MHLSP Ministry of Health, Labor, and Social Protection (consolidation of ministries in 2017) MoE Ministry of Education MoLSPF Ministry of Labor, Social Protection & Family MoF Ministry of Finance MoH Ministry of Health MP Member of Parliament NGO Non-governmental Organization NPM National Practice Model OPM Oxford Policy Management PANDA Psycho-Social Assistance Program for Children P4EC Partnerships for Every Child PL 140 Public Law 140 PPAS Psycho-pedagogical Assistance Service RCC Residential Care Center SAFPD Social Assistance & Family Protection Directorate (District government) TACIS Technical Aid to the Commonwealth of Independent States ToR Terms of Reference USAID U S Agency for International Development UNCRC United Nations Convention on the Rights of the Child UNICEF United Nations Children’s Fund EVALUATION | vi Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project EXECUTIVE SUMMARY USAID’s Displaced Children and Orphans Fund (DCOF) provides financial and technical assistance to improve the well-being of children outside of family care or at risk of family separation, through direct interventions that affect children and strengthen human and institutional capacities at the family, community, and national levels. USAID/DCOF’s overall goal is to measurably improve the safety, well-being, and development of such children. However, DCOF also gives priority to projects that promise impact beyond direct services, which strengthen local capacity, and offer models and approaches for expansion, adaptation, and/or replication. Evaluation Purpose and Evaluation Questions The Moldova project Children in Moldova are Cared for in Safe and Secure Families was implemented by Partnership for Every Child (P4EC) from January 1, 2014 to December 30, 2017. The overall aim of the project was to advance child care reform in Moldova, increasing the shift from institutional care to family-based care. The project is operated at national and local levels, and worked most intensively with 10 raions (Romanian term for districts) in Moldova. The purpose of this performance evaluation report is to compare the situation from baseline to endline and report any improvements over time. However, since this is a performance evaluation and not an impact evaluation with a control or comparison group, the report does not attempt to attribute causality between project interventions and results in the country. The evaluation is based on six key questions proposed by USAID/DCOF. Project Background DCOF previously funded the project Protecting Children in Moldova from Family Separation, Violence, Abuse, Neglect & Exploitation from August 2010 – June 2014 (in Ungheni, Calarasi, Falesti, Telenesti and Singerei) which aimed to provide the authorities in Moldova with assistance to strengthen the child protection system, addressing the needs of vulnerable children and their families, and closing the gaps in their access to quality social services, both preventing family separation and protecting children without parental care. The current project, Children in Moldova are Cared for in Safe and Secure Families, built on the previous work both in the 5 “old” local authorities (LA/raion) and 5 “new” raions which include Soroca, Orhei, Causeni, Cahul and Nisporeni. The focus in the “old” districts was on developing prevention programs for vulnerable families (most of the residential care centers had largely been closed) while the focus in the “new” districts was on closing RCCs and developing alternative family-based care in addition to developing prevention programs. EVALUATION | 1 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project The project targeted three groups of children: (1) children living in birth families – supported to stay with their families; (2) children living in residential institutions – transitioning to family and community based care; and (3) children living in family-based alternatives – supported to reunite with their birth or extended families or placed in adoption. Project stakeholders included (1) those at the national level: Ministry of Labor, Social Protection & Family (MLSPF), Ministry of Education (MoE), the National Council on Residential Care and Inclusive Education Reform and the National Council for Participation (consists of leading NGOs that advise on legal acts); (2) those at the district level: Local Authorities and the Social Assistance and Family Protection Directorates (SAFPD); and (3) those at the local level: grassroots CSOs involved in service delivery, communities, children, parents, and caregivers. Evaluation Design and Methods The evaluation comprises of primary data collected by the NORC team in the form of KIIs, FGDs, case file reviews, and a survey of community social assistants and social work specialists of the SAFPD, who are the direct beneficiaries of the project (other direct beneficiaries include a large number of raion and national officials who were trained by the project). Baseline data was collected from February 18 - April 23, 2015 and endline data was collected from June 26 -- July 28, 2017. The evaluation also used secondary data collected by P4EC – child well-being indicators and monitoring data of capacity building training of stakeholders. NORC drafted and finalized the data collection tools (KII protocols, FGD protocols, and survey questionnaires). These were shared with DCOF and the implementing partner for their feedback prior to finalization. Findings And Conclusions Question 1: By project end, to what extent have functioning structures been established that can continue to provide on an ongoing basis adequate case-management services for children at risk? Findings: ■ Institutionalizing joint approaches as a norm in Moldova’s child protection system is a cornerstone for the project and there is now greater collaboration and coordination among the different sectors. ■ The project has improved the functioning of the gatekeeping commissions. Mayors and multidisciplinary teams are now more involved in and accountable for child protection. ■ Almost all social workers received training through the project. Training was seen as very relevant and they have largely applied what was learned. EVALUATION | 2 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project ■ At project end, social workers felt more supported in their work, caseloads had dropped somewhat, and there was more time to visit families. ■ However, social workers felt less confident in their own skills at project end, likely because of the significant changes in case management, new response tools, and new players with whom they must collaborate. ■ Government stakeholders to the project report Advisory Boards of Children are a welcome addition to better understanding the needs of at risk children and helping in responding to their needs. Conclusions: ■ P4EC has trained scores of mayors and members of multi-disciplinary team members which has contributed to the intersectoral collaboration and burden sharing required for more effective child protection. However, turnover in these positions requires ongoing training mechanisms to maintain the momentum generated by the project. ■ The involvement of the health sector and mayors in intersectoral approaches consistent with project objectives has strengthened over the project period, however, stakeholders report that greater involvement is needed, along with continued capacity-building and training in these sectors. Demands on the time of those in the health sector make their cooperation especially difficult. ■ Social service workers have a continuing need for their own psycho-social support given the challenges posed by their casework. This requires effective supervision, counseling mechanisms and greater funding to align caseloads with human resources needed to provide support. ■ The National Model of Practice shows promise in early identification of risk and vulnerability in families. The results in piloting of this model need to clearly and broadly shared among all stakeholders and NGOs and the model should be supported in its replication and expansion across Moldova. ■ Greater government funding of the social service sector is needed to maintain momentum in project activities. Adequate pay and staffing levels for the community social assistants employed by SAFPDs is the most pressing need, followed by financing foster care. ■ Though funding and service gaps still exist, prospects for project sustainability are high due to project-funded infrastructure investments and the high level of commitment by key players across the child welfare system. EVALUATION | 3 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Question 2: Did the program bring out systemic changes at the community, regional, and national levels that are enabling children to live in family care and preventing inappropriate placements in institutional care? Findings: ■ There is a perception that child protection laws, action plans, and national policies are now stronger and better implemented. Many different types of stakeholders perceived that there had been a significant change in the government’s emphasis on child protection, leading to more timely interventions on behalf of social workers, a focus on reintegrating the child back into the family, improved reporting, and better intersectoral communication. ■ While almost half of social workers at project end believed a child should be placed in an RCC when orphaned, this represented a significant drop of 14 percentage points from baseline. ■ There is no perceived change in the perception of social service workers on the attitudes of parents and the community towards residential care although most members of the community believe it is best for the child to stay with the family or relative ■ There has been a shift in perception of children in RCCs with some stakeholders now holding positive views and less negative stereotypes. Some described children from RCCs as hard-working, particularly in regards to their schoolwork, as having adequate behavior and being “brave.” ■ According to social workers, caregiver skills have not shown any improvement over the project period although parent support groups were seen as more effective at increasing parenting skills at the project end. ■ Child protection campaigns led to increased interest in foster care services. Conclusions: ■ There has been a moderate shift in attitudes regarding placement of children in RCCs and a greater sentiment that this is not appropriate. Public opinion is trending toward a greater number of people believing the family is the best place to raise a child. Social workers and specialists believe it is less necessary, for example, even when a child is disabled, there is a lack of community services, if the child is an orphan, or for juvenile delinquents. ■ In discussions with multidisciplinary teams, parents, ABC at national level, community social assistants, and the gatekeeping commission there was a perception that the government has a greater focus on child protection and that national policies and legislation are being better implemented. EVALUATION | 4 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project ■ Social workers and specialists believe the normative framework developed in recent years provides much greater opportunity for child protection and keeping children out of institutions. ■ National policies are beginning to shift the burden of child protection, more broadly from the SAFPD to other sectors, including health, education and municipalities. National policies and laws are beginning to be reflected in strategies and policies developed at the sub-national level. Question 3: Have reintegration methods employed by the projects resulted in stable and sustained placements for children? Findings: ■ Seven residential institutions were assessed and five were closed. The two that weren’t closed were special boarding schools for deaf and hard of hearing children. ■ The project deinstitutionalized all children in project-targeted RCCs in four of the six “new” districts: Soroca, Rezina, Orhei, and Nisporeni. ■ The total number of children de-institutionalized through project activities is 270, of which 166 were reintegrated in biological families, 59 were placed in extended families, and 45 were placed into family-based alternative care. ■ Follow-up social services to families of reintegrated children have risen in all categories of assistance, with more social workers reporting they put emphasis on material assistance and counseling Conclusions: ■ Children with disabilities face particular difficulties that may prevent smooth and successful transitions. Education inclusion, a key component of the deinstitutionalization process during this project, is essential as many institutionalized children have special education needs. There is also a need for more non-residential alternative care options for these children. ■ The project built capacity for the reintegration of children with disabilities through project stakeholders, but challenges remain particularly for children with more serious disabilities – including behavioral issues. Specialized foster care, vocational training, day support centers, and professional parental assistants are needed. ■ The project has contributed to making follow-up social services more available to children and families in need, including greater preparation of children and families prior to placement, support in the post-placement period, and multi-disciplinary assistance from health, education and other sectors. ■ The level of disruptions of deinstitutionalized children is low, with only five known cases. EVALUATION | 5 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Question 4: Did the program measurably improve the safety, well-being, and development of highly vulnerable children, particularly those who are living without adequate family care? Did the program impact beyond direct services? Findings: ■ P4EC does not provide direct services to children and families, but rather supports decision makers and direct service specialists in government sectors at raion and municipal levels. ■ The project strengthened the overall case management process and likely contributed to improvements in safety, health, and education indicators. ■ Community social assistants reported that when comparing the well-being of deinstitutionalized children from when they were in RCCs versus in family-case, their health and safety is adequate in both cases and their emotional wellbeing is much improved. The project successfully addressed issues of poverty and lack of effective parenting by strengthening government-provided social services. ■ There was an increased availability of alternative family care options by project end, especially in foster care and emergency/temporary placements in shelters. ■ The project strengthened guardianship arrangements and targeted services to families and children impacted by parents’ labor migration. ■ The project addressed adverse consequences of alcohol abuse. Alcohol abuse was identified by all stakeholders as a most serious issue in Moldova. The project addressed this service gap through its Psycho-Social Assistance Program for Children (PANDA) for children and adults with a household member who abuses alcohol. ■ Deinstitutionalized children did better on health indicators after placement, they mostly felt safe in their communities, and they showed a large improvement in emotional health. Conclusions: ■ Deinstitutionalized children struggled in school after reintegration, perhaps because those in RCCs had preexisting special education needs or had received schooling that was of lower quality and came with lower expectations. ■ For children who cannot be with their birth families, the project has contributed to increasing use of alternative care, in particular foster care and emergency/temporary placement shelters. APP foster care as an alternative care option has grown in the project raions and some stakeholders are investing in its expansion. Stakeholders in most project raions cite the need for a greater number of foster families, though this faces government funding shortfalls when compared to the need. EVALUATION | 6 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project ■ Alcoholism, negligent or abusive parents, poverty, and migration are the main reasons why children are separated from their parents. The project addressed these issues by strengthening the current government response capacity. The PANDA program shows promise as a mechanism to help fill an important support gap by addressing challenges faced by children in families affected by alcohol abuse. ■ Guardianship is taking on increased relevance as a means to protect children affected by migration. Question 5: Have prevention methods employed by the projects reduced risks of child/family separation? Findings: ■ Prevention is an overarching framework in the project ranging from early identification and response to risk through the NPM to building on the strengths of children and families to better withstand shocks and challenges in their lives. PANDA is designed to do this by strengthening psycho-social wellbeing of children impacted by alcohol abuse in the family. ■ All well-being indicators for children in vulnerable families showed at least some improvement from time of case opening to time of closing the case. The most progress was achieved in health and safety indicators, while the least progress was achieved in psycho-social indicators. ■ Availability, quality, and access to services improved over the project period. ■ Financial challenges and difficulties in securing employment in Moldova continue to be central issues for vulnerable families. While the availability of financial assistance has increased and it is viewed as one of the most effective support services, social workers also felt that the accessibility and effectiveness could be further improved. ■ Social workers viewed financial assistance and case management as the most effective support services, though social workers also felt they could be improved. ■ Social/psychological services for people with mental health challenges need improvement. ■ Parenting training was viewed as critical to help prevent separation of families. P4EC implemented a complementary Mellow Parenting program to the DCOF￾funded project that focused on improving parent-child relationships. Conclusions: ■ The well-being of children in vulnerable families who were monitored from 9 to 12 months between 2016 and 2017 showed some significant improvement in some areas. The most progress was achieved in health, safety, and education indicators for those with special needs. EVALUATION | 7 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project ■ Overall, social services have become more available and accessible; family support services have become more available but there was no change in their effectiveness; support to children has increased in availability and effectiveness. The project’s multi-service approach likely contributed to the increase in availability and relevance of social services. ■ There was improvement in the perceived adequacy of services to meet the needs of family at the raion level, in particular improvement was perceived for services to guardians. At the community level, gaps in services exist due to the distance between the families and the services. ■ Services in health care were also viewed as needing improvement in availability and quality by a little more than one third of respondents. There is a significant gap in alcohol and substance abuse treatment, though the PANDA program is being embraced as a mechanism that could be particularly helpful for the psycho-social health of children in families where abuse is occurring. ■ Parenting training is an effective mechanism for preventing the separation of families. Assessing the Mellow Parenting program will be important understanding its potential for replication. Question 6: Did the project offer models and approaches for expansion, adaptation, and/or replication? Findings: ■ The project effectively continued P4EC’s work on the inclusive education process to prevent separation and continue deinstitutionalization. ■ The intersectoral training and capacity building was an effective mechanism to put the key child protection sectors working toward the same goals and objectives, and conforms to the best practice of multidisciplinary approaches for child protection results. ■ The project approach gave child protection advocates and practitioners in project areas greater capital in helping to ‘responsibilize’ key government stakeholders – such as mayors – to fulfil their legally mandated duties in child protection and family welfare. ■ The project approach prioritized and successfully utilized lessons learned and their dissemination for impact, as models were piloted and then more broadly implemented based on these experiences. Trainers from “old” and “new” raions were utilized in cross-training processes bringing their experiences and lessons learned forward to new project areas. ■ The project implemented a case management process that leads to care plans based on individually identified child and family risks and strengths. EVALUATION | 8 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project ■ An important and noteworthy organizational practice of P4EC is facilitating a process of studying best practices and programs in other countries and then modeling and adapting them to the Moldovan context. The PANDA program innovation is a significant attempt to address alcohol abuse – one of the most challenging protection issues in Moldova and around the world. The roll out of PANDA has been particularly effective as the methodology has been met with considerable commitment from local government stakeholders who moderate and implement this mechanism. Conclusions: ■ The emerging comprehensive strategy for reducing institutional care for children with significant special needs is informed by this project. Children with disabilities have been deinstitutionalized through the project with adaptive inclusive education being instrumental to this process. The project and its stakeholders have helped to identify and demonstrate what is required in foster care to more effectively support children with disabilities. Community-based support linked to the project through training and other activities – such as day and respite care, personal assistant home care, and mobile teams supporting families with disabilities – are also strongly relevant. ■ Mechanisms for more efficient use of resources are broadly applicable for replication and adaptation across Moldova. Examples from the project include more efficient use of foster care resources to increase the number of caregivers and improve the quality of care, as well as demonstrable improvements in the well-being of children and families which spurred greater commitment of raion decision-makers to allocate funding. ■ Government stakeholders to the project reported that Advisory Boards of Children are a welcome addition to better understanding the needs of at risk children and helping in responding to their needs. ABCs are an innovative means for youth to become involved and advocate for children’s rights and needs in protection. ■ Government stakeholders in the project are so supportive of Advisory Boards of Children to the point that several raions are committed to financing support to the board after the end of the project. Other raions may have an interest in applying this child participation mechanism. ■ A community of expertise, committed to intersectoral collaboration, has been activated through the project. This comes through the project approach to vertical and horizontal capacity building and training. Stakeholders began to “speak the same language,” use the same points of reference and are assessing children and families in more uniform ways, leading to coordinated response. Many of the sectoral leaders and specialists are passionate about their work. They convey this through cross-training for replicating project priorities, such as from old to new raions, within raions, and strengthening intrasectoral mechanisms such as EVALUATION | 9 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project gatekeeping commissions and multidisciplinary teams. This experience and zeal can potentially be applied nationally. Overall Recommendations: ■ The project model for deinstitutionalization should be considered and adapted to raions still needing to close residential care centers, with appropriate incentives to create political support in these pockets of resistance. Guidelines, procedures, training modules, skilled and passionate trainers, case management tools, and systems developed and used in the project can all be applied to this process. ■ Through some project raions have found the means to invest in APP foster care expansion, the government needs to find a more stable means to grow foster care as an alternative form of care for children outside of parental care. ■ Efforts to strengthen guardianship regulations through the civil code associated with PL 140 will provide greater guidance in the procedures of guardianship protection – particularly as it applies to vulnerable children left by migrant parents. ■ The funding crunch at the local level brought about through decentralization needs to be resolved for project models to have significant expansion and replication, let alone improve prospects to grow more specialized support such as in psycho￾pedagogy. If enacted, a national funding mechanism for a basic package of services can potentially address this need. There should be a major push for this from the donor community and Moldovan civil society. ■ Local authorities in project raions are supportive of the National Practice Model as a means for early identification of risk and prevention. NPM piloting needs to be documented and refinements made to address concerns expressed by stakeholders such as time, human resources, and cost – particularly in the health and education sectors – for this support to be applied nationally. Other NGOs need to be collaboratively brought into this process. A normative act to “responsibilize" the NPM will be required for it to have national application. ■ All of the PANDA moderators interviewed (from Ungheni, Falesti, Nisporeni, Orhei, and Calarasi) believe the program should be replicated across Moldova. Government and the donor community should continue to support its development and research on its effectiveness as a model for psycho-social support outcomes for children and their families, and for replication across Moldova and other countries. EVALUATION | 10 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project 1.0 EVALUATION PURPOSE & EVALUATION QUESTIONS 1.1 Evaluation Purpose USAID’s Displaced Children and Orphans Fund (DCOF) provides financial and technical assistance to improve the well-being of children outside of family care or at risk of family separation, especially vulnerable children (defined as under 18 years of age) through direct interventions that affect children and strengthen human and institutional capacities at the family, community, and national levels. Children are defined as under 18 years of age. DCOF’s overall goal is to measurably improve the safety, well-being, and development of highly vulnerable children, particularly those who are living without adequate family care. However, DCOF also gives priority to projects that promise impact beyond direct services, which strengthen local capacity, and offer models and approaches for expansion, adaptation, and/or replication. USAID/DCOF attempts to ensure that all funded activities build upon and contribute to the knowledge base of evidence concerning the most appropriate practices for ensuring appropriate care, protection, and development of children. It has therefore funded the evaluation of two projects (in Moldova and Burundi) that seek to ensure that children are in protective and permanent family care by reducing unnecessary separation of children from their families and by placing children who are outside of family care in nurturing families. The project in Moldova (Children in Moldova are Cared for in Safe and Secure Families) was implemented by Partnership for Every Child (P4EC), and the project in Burundi (Family Care First: A Project to Ensure Children in Burundi are Place in Protective and Permanent Family Care) was implemented by the International Rescue Committee (IRC). Evaluations that collected data at baseline and endline were completed for both these projects. This report includes the performance evaluation results for Moldova; a similar report was also prepared for Burundi. The Moldova project implementation period was from January 1, 2014 to December 30, 2017. The overall aim of the project was to advance child care reform in Moldova, increasing the shift from institutional care to family based care. The project operated at national and local levels, and worked most intensively with the local authorities (LA) of 10 raions (Romanian term for districts) in Moldova. The purpose of this performance evaluation report is to compare the situation from baseline to endline and report any improvements over time. However, since this is a performance evaluation (an evaluation without a comparison group), the report does not attempt to attribute causality between project interventions and results observed in the country. EVALUATION | 11 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project 1.2 Evaluation Questions The evaluation is based on 6 key questions proposed by USAID/DCOF.1 Additional sub￾questions and indictors proposed by NORC for this evaluation are shown in Annex II: Evaluation Questions and Indicators. 1. By project end, to what extent have functioning structures been established that can continue to provide on an ongoing basis adequate case-management services for children at risk? 2. Did the program bring out systemic changes at the community, regional, and national levels that are enabling children to live in family care and preventing inappropriate placements in institutional care? 3. Have reintegration methods employed by the projects resulted in stable and sustained placements for children? 4. Did the program measurably improve the safety, well-being, and development of highly vulnerable children, particularly those who are living without adequate family care? Did the program impact beyond direct services? 5. Have prevention methods employed by the projects reduced risks of child/family separation? 6. Did the project offer models and approaches for expansion, adaptation, and/or replication? 1 The Concept Note/Scope of Work for this evaluation is provided in Annex I. EVALUATION | 12 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project 2.0 PROJECT BACKGROUND 2.1 Moldova Program Context The Republic of Moldova (hereafter Moldova) became independent from the Soviet Union in 1991, inheriting a child welfare system heavily reliant on institutional care for the protection for vulnerable children. A population of 3.56 million (2013) live in 32 raions and 2 autonomous republics. Moldova is the poorest country in Europe with 17% of the population living below the poverty line and many families struggling to care for their children. Approximately 58% of the population lives in rural areas where lack of employment and limited access to social protection services further exacerbate poverty￾related issues.2 In the period following independence from the Soviet Union, Moldova held one of the highest percentages of children living in residential care centers (RCCs). However, due to sustained reform efforts, the number of children in RCCs has reduced from approximately 14,000 in 2007, when the national strategy for deinstitutionalization began, to about 2,000 at present. Loss of parental care has been caused by several factors, including3: ■ Household poverty (Moldova is the poorest country in Europe); ■ Parental migration, with 1 in 4 children having at least one parent living or working abroad; ■ Violence, abuse, and neglect at home - alcohol dependency being one key causal factor; ■ Lack of access to good quality education and health care near home; and ■ The persistent belief among parents, practitioners and decision makers that the state can care for children better than families. USAID/DCOF previously funded the project Protecting Children in Moldova from Family Separation, Violence, Abuse, Neglect & Exploitation from August 2010 – June 2014 implemented by Every Child (EvC). EvC Moldova was established in 2003 and later supported the establishment of Partnerships for Every Child (P4EC), a Moldovan non￾governmental organization which started operations in April 2012. The previous project aimed to provide the authorities in Moldova (at the national, regional and local levels) with assistance to strengthen the child protection system, 2 From “Protecting children in Moldova from family separation, violence, abuse, neglect & exploitation: Final Evaluation Report.” By N. Beth Bradford. August 1, 2014. Chisinau, Moldova: Partnership for Every Child. 3 From “Children’s Reintegration. Longitudinal Study of Children’s Reintegration in Moldova.” By Dr. Helen Banõs Smith. February 2014. Chisinau, Moldova: Family for Every Child and Partnership for Every Child. EVALUATION | 13 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project addressing the needs of vulnerable children and their families, and closing the gaps in their access to quality social services, both preventing family separation and protecting children without parental care. Three regions were targeted by the USAID project – Ungheni, Calarasi, and Falesti. The project was expanded into Telenesti and Singerei with the support of Every Child UK and a private donor. The current project Children in Moldova are Cared for in Safe and Secure Families4 builds on the previous work both in the 5 “old” raions and 5 “new” raions which include Soroca, Orhei, Causeni, Cahul and Nisporeni. After the start of the current project, a sixth “new” raion, Rezina, was added at the request of the Ministry of Education and upon approval by USAID.5 Among the new components of the current project are those related to prevention such as the Strengthening Families Program, inter-agency work on the basis of child well-being indicators, and the Psycho-social Assistance Program for Children, as well as strengthening and diversification of alternative care for children. 2.2 Moldova Program Objectives The overall objectives of the project were: 1. Across Moldova, 100,000 children who are at risk of losing family care, are living with seriously inadequate family care, or are outside family care have increased chances to stay with their strengthened families or be placed in appropriate, protective, and permanent alternative family care; 2. 4,000 children have been prevented from being unnecessarily separated from their families in 10 regions of Moldova; 3. 3,000 children who are outside family care in 5 regions of Moldova live in safe and nurturing families. The project is targeted three groups of children: ■ Children living in birth families – supported to stay with their families; ■ Children living in residential institutions – transitioning to family and community￾based care; and ■ Children living in family-based alternatives – supported to reunite with their birth or extended families if possible or placed in adoption. The objectives and targets are result indicators for the project. However, USAID DCOF is also particularly interested in supporting a ‘national systems’ approach of informing 4 Shortened to ‘Strengthening Families for Children Project’ – a more concise project title commonly used by the project, including the Romanian translation. 5 P4EC is not implementing all project activities in this raion – the focus is on deinstitutionalization. As a result, it was agreed that only data on deinstitutionalization in Rezina, gathered by P4EC, would be included in this evaluation. EVALUATION | 14 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project and capacitating the national normative framework for government services and support systems for deinstitutionalization, including prevention of unnecessary family separation, family reunification, and family-based alternative care. 2.3 Moldova Program Design and Management The 10 project districts are at varying levels of development regarding their child care and protection systems.6 The residential care centers (RCCs) have largely been closed in the five “old” raions where P4EC previously worked. Table 2.3.1 below details their status, indicating whether it is an “old” or “new” raion and the level of development from low to high. Table 2.3.1: Level of Development of Childcare and Protection System by Project Raion Raion name Level of Development of Child Care and Protection System Low Medium High-Medium High Falesti Old Ungheni Old Calarasi Old Telenesti Old Singerei Old Orhei New Cahul New Soroca New Causeni New Nisporeni New Source: P4EC Project Director The focus in the “old” raions is on developing prevention programs for vulnerable families and testing new approaches and models of practice before replication to all raions. The initial focus in the “new” raions was on closing RCCs and developing alternative family-based care, although by project end, the “new” raions received the full package of project activities. The main activities of the project are as follows: 7 Objective 1 ■ Build capacity of the Ministry of Labor, Social Protection and Family (MoLSPF) and Ministry of Education (MoE) to implement the National Child and Family Protection Strategy, amend child care legislation, and develop the Child Participation Policy; 6 As noted above, the 11th raion, Rezina, is only included when reporting deinstitutionalization numbers from the P4EC monitoring data. 7 Adapted from P4EC quarterly project reports. EVALUATION | 15 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project improve communication (develop common vision on child care reform and communicate this campaign to the public); and improve monitoring and evaluation (M&E) framework to monitor children and vulnerable families; ■ Support MoLSPF to train the national workforce of community child protection specialists (to be recruited) and social workers to provide services in line with new legislation; gatekeeping commissions on revised regulations to prevent child separation; raion local authorities on inter-agency collaboration on prevention and child protection; and community multidisciplinary teams to work on prevention; ■ Facilitate the government’s alliances with civil society organizations (CSOs), including faith-based, academia, practitioners; and ■ Support 10 districts to develop communication plans and tools. Objective 2 ■ Support 10 raions to develop a holistic model for family strengthening and preservation; train family support teams, community social assistants, and child protection specialists to provide primary and secondary family support; train foster care teams, community social assistants, and child protection specialists to provide and support short-break foster care placements for children with disabilities living in families; and revise membership of and train gatekeeping commissions; and ■ Support five “old” raions to plan, deliver, and evaluate a Strengthening Families Program and Psychosocial Support Program for Children. Objective 3 ■ Support 7 raions to shift alternative care toward permanent family-based care and assess, plan, and reorganize residential institutions; ■ Support 5 raions to build capacities of community social assistants to plan, implement, and monitor individual plans for deinstitutionalization of children and to develop short-break, emergency, and short-term and long-term foster care; ■ Train and support 10 raions to develop inclusive education at district and school level; and ■ Build capacities of 10 raions to put in place sound child participation policies and practices. Project stakeholders include (1) those at the national level: MoLSPF, MoE, the National Council on Residential Care and Inclusive Education Reform and the National Council for Participation (consists of leading non-governmental organizations (NGOs) that advise on legal acts)8; (2) those at the district level: Local Authorities, the Social 8 There was a consolidation of ministries in 2017 and the new ministries are the Ministry of Health, Labor, and Social Protection and the Ministry of Education, Culture, and Research EVALUATION | 16 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Assistance and Family Protection Directorates (SAFPD), and gatekeeping commissions; (3) those at the local level: mayors, multidisciplinary teams, community social assistants, grassroots CSOs involved in service delivery, communities, children, parents, and caregivers; and (4) NGOs or multilateral organizations (UNICEF) working on child protection. P4EC has compiled a team of 23 key staff including a Director, Project Team Leader, Child Participation Specialist, Public Finance Reform Specialist, Training Manager (also in charge of academic collaboration), 5 Coordinators, 2 staff working on communications and PR, and 2 staff responsible for collaboration with NGOs and churches. Additionally, P4EC has subcontracted with Terre des Hommes to implement initial trainings of the multidisciplinary teams. Background Context on Old and New Raions The five “old” raions represent a continuation of DCOF/USAID and EveryChild UK support in the period immediately prior to the project (with many of the same project priorities such as deinstitutionalization and prevention). The five “new” raions did not have the prior involvement. However, these raions have had at least some earlier involvement with P4EC and other NGOs working on issues related to project objectives. In the current project, P4EC also provided support to an additional raion, Rezina, in deinstitutionalizing two residential care centers. There are general differences in project emphases in the old and new raions. Significant deinstitutionalization processes were already well underway in the old raions with residential care centers being closed and children reintegrating into communities. In these raions the project focuses strongly on strengthening prevention activities, such as taking intersectoral collaboration to the next level through piloting the National Practice Model (NPM) in three of these raions (Ungheni, Falesti and Calarasi). The NPM is an intersectoral early risk identification model. In Rezina and the new raions of Soroca and Nisporeni, the project was very involved in deinstitutionalization and the closure of residential care centers. Prevention is also an integral part of the project in old raions. Many of the same activities occurred in all raions based on needs and contexts.9 Examples include capacity building in case management, intersectoral approaches, strengthening family￾based alternative care options for children, Advisory Boards of Children (ABC), the Psycho-social Assistance Program for Children (PANDA) addressing alcohol abuse, communication strategies, finance assessment and restructuring, and parenting 9 P4EC has had some historic program involvement in new raions, such as Cahul, in supporting family strengthening and other activities. EVALUATION | 17 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project education/counseling. For the latter, a new model called Mellow Parenting was introduced by P4EC during the project period though this was not supported by DCOF/USAID funding. Deinstitutionalization and prevention of unnecessary separation of children from their families are intimately related. In a sense, they can be considered two sides to the same coin. As longer-term residential care centers are being closed, prevention slows down and eventually stops center intakes. Virtually the same child and family assessment and support activities, and multi-sectoral approaches are involved with prevention and deinstitutionalization. P4EC and other NGO prevention approaches in Moldova today have been greatly informed by the deinstitutionalization process that began more than 20 years ago. National and Local Contexts Performance in this evaluation is evaluated at both national and decentralized levels in the raions of the project. At the national level, the focus is on legal and policy frameworks; capacity building through training and development of models and systems applicable throughout Moldova; and through communication strategies on public and professional knowledge and attitudes. Communication is also a project focus at the raion level. At this level, the project also goes deeply into raion-specific social service systems built on family support, alternative care and child protection; capacity building/training; inclusive education; financial management, child participation and deinstitutionalizing children out of residential care. The project targeted its training and skill building at local and national levels. Training and skill building at decentralized levels in project raions brought together actors from across key child protection sectors to reinforce the multidisciplinary approaches and collaboration within local contexts. At the national level, the project held trainings with stakeholders from raions outside of the project target areas and with nationally-based actors as well, such as for gatekeeping commissions and the emerging NPM. Operationalizing Models and Best Practices and Informing the Normative Framework P4EC does not provide direct services to children and families. Its organizational approach is working closely with key stakeholders, usually government structures at decentralized levels, who do provide direct services. It begins with would be improved, in turn improving their well-being. EVALUATION | 18 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project 3.0 EVALUATION METHODS & LIMITATIONS To gather data required for this evaluation, NORC’s Evaluation Team used several techniques which entailed a mix of mutually reinforcing qualitative and quantitative methods that reflect the program design, research questions being addressed, and indicators. We combined the results of each technique to capture the diversity of opinions and perceptions of beneficiaries and stakeholders about key children/family care and protection issues at the start of the program. The qualitative analysis, which includes a document review, case file reviews, key informant interviews (KII) and focus group discussions (FGD), provides the local context and also represents concrete examples that illustrate in greater detail the quantitative findings. Our approach to selecting the appropriate methodology is based on the USAID Evaluation Policy as well as our experience conducting evaluations in the field. The NORC Evaluation Team conducted the evaluation in a participatory manner which involved engaging USAID/DCOF, implementing partner P4EC, program beneficiaries, and other stakeholders. A complete list of documents the Evaluation Team reviewed is included in Annex V, Sources of Information. 3.1 Evaluation Management The evaluation team for Moldova includes Ritu Nayyar-Stone (Project Director), Mawadda Damon Gartner (Evaluator), N. Beth Bradford (Subject Area Expert – Baseline), Gary Gamer (Subject Area Expert – Endline), Matthew Fisher-Post (Senior Research Analyst – Endline), Huyen Le (Research Analyst – Baseline), Aaron Wilson (Senior Research Analyst – Endline), Carlos Fierros (Research Analyst – Endline), Samantha Downey (Research Assistant – Endline), and Ilse Paniagua (Research Assistant – Endline). Graduate Research Assistants from the Harris School of Public Policy at Chicago, Selena Zhong and Gabriel Velez, provided research support during qualitative analysis. Veronica Pelivan (Local Coordinator) provided logistical support, took notes during KIIs, and led KIIs in two raions for the baseline. Local data collection was undertaken by IMAS INVEST SRL (IMAS) who conducted the FGDs and administered the evaluation survey. As a measure to ensure high data quality, NORC provided targeted training to IMAS for FGDs and survey administration/quality control; undertook a data quality review of the received data; and completed all the analysis. 3.2 Study Design The evaluation employs a pre-post design, comparing key indicators and findings for project beneficiaries at baseline to those at endline. This approach allows us to understand changes over time but does not allow us to attribute these changes to the project intervention since it was not possible to include a comparison group. EVALUATION | 19 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Concerns about confidentiality and sensitivity in contacting vulnerable children and their families directly, as well as the fact that children and families are indirectly beneficiaries of the P4EC project, led to a change in the target population for the evaluation survey. NORC had originally planned to survey caregivers of children in vulnerable families, but after consultation with P4EC and agreement by USAID/DCOF, a questionnaire was designed for community social assistants and social work specialists. This change in study design required NORC to depend solely on P4EC to obtain information on the well-being of children (i) currently in RCCs, and (ii) in vulnerable families, due to concerns about evaluators asking vulnerable families about often traumatic events and circumstances. NORC and P4EC worked closely together to develop indicators of child well-being (see Annex II), and P4EC built the capacity of and assisted social workers in obtaining child well-being information for both vulnerable children at home and vulnerable children currently in RCCs. The evaluation therefore comprises of primary data collected by the NORC team in the form of KIIs, FGDs, case file reviews, and a survey of community social assistants and social work specialists of the SAFPD, who are direct beneficiaries of the project (other direct beneficiaries include a large number of raion and national officials who were trained by the project); and secondary data collected by P4EC – child well-being indicators and monitoring data of capacity building training of stakeholders. NORC drafted and finalized the data collection tools (KII protocols, FGD protocols, and survey questionnaires) for Moldova. These were shared with USAID/DCOF and the implementing partner for their feedback prior to finalization. The tablet-based survey was programmed by IMAS and tested by both IMAS and NORC prior to beginning enumerator training. NORC developed all training materials and the NORC team traveled to Moldova prior to data collection at both baseline and endline to conduct the training for FGD and survey enumerators and conduct KIIs with P4EC staff and other stakeholders. Baseline data was collected from February 18 - April 23, 2015 and endline data was collected from June 26 -- July 28, 2017. 3.3 Target Population The quantitative survey (here on called the “evaluation survey”) targeted community social assistants and specialists of the SAFPD or “social work specialists” in the 10 project raions. Community social assistants are qualified as social workers in Moldova (holding a university degree) who provide social assistance services. Table 3.3.1 below shows the full sample framework for the baseline and endline survey as well as the attained response rates at each time. In this report, we use “social workers” to refer to these two populations (community social assistants and social work specialists”. EVALUATION | 20 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Table 3.3.1: NORC Evaluation Survey Sample Sizes at Baseline and Endline Raion Baseline Endline Community Social Assistants Social Work Specialists Community Social Assistants Social Work Specialists Cahul 30 4 30 5 Calarasi 31 4 28 4 Causeni 31 3 26 3 Falesti 30 7 30 4 Nisporeni 26 3 26 3 Orhei 30 5 30 5 Singerei 28 2 30 3 Soroca 30 4 30 3 Telenesti 30 2 30 2 Ungheni 30 4 30 4 TOTAL 296 38 290 36 Number of respondents 264 36 254 29 Number of eligible cases in sample 311 305 Response rate10 96.5% 92.80% EVALUATION | 21 Source: NORC Evaluation Survey Baseline qualitative data collection consisted of a series of 29 FGDs led by our subcontractor, IMAS, and 42 KIIs conducted primarily by Subject Area Expert Beth Bradford. Additional KIIs were conducted by Evaluator Mawadda Damon and Local Coordinator Veronica Pelivan. The KII team interviewed the P4EC project team and key national stakeholders such as representatives of the Department of Mother and Child at the Ministry of Education, Department of Pre-University Education at the Department of Education, and the Faculty of Social Work at Moldova State University. Additionally, Beth Bradford and Veronica Pelivan visited six project raions: Soroca, Telenesti, Singerei, Causeni, Ungheni, and Cahul where they interviewed the Vice Presidents on Social Issues, Heads of Social Assistance and Family Protection, Child and Family Protection Specialists, Heads of Education Departments, and Heads of Community Social Assistance Services in the SAFPDs and Heads of RCCs where they existed. Endline qualitative data collection consisted of a series of 22 FGDs led by IMAS and 53 KIIs conducted by Subject Area Expert Gary Gamer with a Moldovan team that included an interpreter and recordkeeper. The KII team interviewed P4EC project team and key national stakeholders, such as representatives from the Ministry of Labor, Social 10 NORC uses the AAPOR standard to calculate response rates where the numerator is the number of completed cases and the denominator is the number of eligible cases in the sample. Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Protection and Family (MoLSPF); the Ministry of Health (MoH); the Ministry of Education (MoE); the Lumos Foundation; and the Free International University of Moldova. Gary Gamer visited eight project raions: Soroca, Singerei, Causeni, Ungheni, Cahul, Nisporeni, Falesti, and Calarasi, where he spoke with the Heads of Education Departments, Raion Council members, Managers of Community Social Assistance Service, Child Protection Specialists, PANDA moderators, and Directors of RCCs where they existed. All KIIs and FGDs were recorded where consent was obtained. They were largely conducted in Romanian. All FGD transcripts were translated into English by IMAS and coded in NVivo by NORC for data analysis. FGDs were conducted with raion and community-level stakeholders. At the raion level, FGDs were conducted with: social work specialists (those working at the SAFPD such as the Child Rights Protection Specialist, Family Protection Specialist, Head of Community Social Assistance service, Foster Care Manager, Manager of the Community Social Assistance Service, and Social Worker Coordinators of services such as adoption, family-type children’s homes, guardianship (kinship care), family support, and community centers); advisory boards of children (ABC); and gatekeeping commissions (made up of a mix of professionals from schools, police, health providers, local authorities, and NGOs who review child cases). At the community level, FGDs were conducted with: community social assistants; multidisciplinary teams, (made up of a mix of professionals including mayors, teachers, police, social workers, and health providers who review child cases); mayors; girls; boys; and parents. Table 3.3.2: Number and Location of Focus Group Discussions at Baseline Target # FGDs #Participants #Male #Female Districts represented* Community social assistants 4 35 4 31 Causeni, Falesti, Singerei, Soroca, Ungheni Social worker specialists 2 15 0 15 Calarasi, Falesti, Nisporeni, Orhei, Ungheni Multidisciplinary Teams 2 16 4 12 Cahul, Singerei Gatekeeping Commissions 2 14 4 10 Calarasi, Falesti, Nisporeni, Orhei, Ungheni Mayors 3 18 16 2 Falesti, Nisporeni, Soroca, Ungheni ABC 2 15 2 13 Cahul, Calarasi Girls 4 34 0 34 Aged 12 -14 2 19 0 19 Falesti, Soroca Aged 15-17 2 15 0 15 Causeni, Orhei Boys 4 32 32 0 Aged 12 -14 2 19 19 0 Nisporeni, Telenesti EVALUATION | 22 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Target # FGDs #Participants #Male #Female Districts represented* Aged 15-17 2 13 13 0 Singerei, Ungheni Parents 6 46 1 45 At-risk children 2 16 0 16 Cahul, Singerei Deinstitutionalized children 2 13 1 12 Orhei, Ungheni Children in RCC 2 17 0 17 Nisporeni, Soroca Total 29 225 63 162 Source: NORC FGD data collection Table 3.3.3: Number and Location of Focus Group Discussions at Endline Target # FGDs # Participants #Male # Female Districts represented* Community social assistants 2 16 0 16 Causeni, Falesti, Singerei, Soroca, Ungheni Social worker specialists 1 11 0 11 Calarasi, Falesti, Nisporeni, Orhei, Ungheni Multidisciplinary Teams 2 17 1 16 Cahul, Singerei Gatekeeping Commissions 2 12 4 8 Calarasi, Falesti, Nisporeni, Orhei, Ungheni Mayors 2 12 8 4 Falesti, Nisporeni, Soroca, Ungheni ABC 1 6 4 2 National level Girls 4 26 0 26 Aged 12 -14 2 11 0 11 Falesti, Soroca Aged 15-17 2 15 0 15 Causeni, Orhei Boys 4 31 31 0 Aged 12 -14 2 15 15 0 Nisporeni, Telenesti Aged 15-17 2 16 16 0 Singerei, Ungheni Parents 3 25 3 22 At-risk children 2 17 2 15 Cahul, Singerei Reintegrated children 1 8 1 7 Nisporeni PANDA 1 7 6 1 Nisporeni Total 22 163 57 106 Source: NORC FGD data collection 3.4 Sampling At baseline, NORC included the full sample frame of all male community social assistants and all the social work specialists in the 10 project raions. Where the total number of community social assistants (male and female) was less than 32, NORC included all the community social assistants (Calarasi (31), Causeni (31), Nisporeni (26), and Singerei (28)). In the remaining 6 project raions, we selected a total of 30 community social assistants – first including all the males and then randomly selecting EVALUATION | 23 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project among the females. This resulted in a sample of 296 community social assistants and 38 social work specialists. At endline, NORC used an updated sample frame from P4EC of all community social assistants and social work specialists in the 10 project raions in July 2017. Names were matched to the baseline sample and NORC included all those in the 2017 sample frame who were also in the baseline sample frame (i.e., those who were employed throughout the project period). Additionally, NORC included all male community social assistants and all social work specialists from the 2017 list and excluded anyone who had refused the survey in 2015. Lastly, to complete a sample of 30 community social assistants per raion, NORC randomly sampled from among the “new” community social assistants (i.e., those who were not employed at baseline). This resulted in a sample of 290 community social assistants and 36 social work specialists. It is important to note that only 32 out of 283 endline survey respondents, just over 10%, were “new” community social assistants. Baseline FGD locations were chosen to represent the diversity of different raions so that each target group had representatives from the “old” and “new” raions; raions of low, medium/medium-high, and high level of development of child care and protection systems; and, to the extent possible given the other two criteria and limited total number of FGDs, northern, central, and southern raions. Endline FGD locations were chosen to match those at baseline as much as possible. Baseline and endline raions where the KIIs were conducted were chosen to include a range of low, medium/medium high, and high levels of child care and protection system development, and regional diversity. 3.5 Limitations The Evaluation Team encountered some limitations inherent to the design of this evaluation and during its fieldwork in Moldova. Some of the more relevant limitations are listed below: Baseline timeframe. The program started in January 2014, but NORC’s concept note for the evaluation was approved by mid-June 2014; the evaluation design was completed by December 2014; and the data collection was undertaken in March-April 2015. Thus some program implementation such as training and capacity building of stakeholders had already started prior to the baseline. This may have resulted in the Evaluation Team capturing less of the project impact through the survey than actually took place. Administrative and M&E data from the implementer. P4EC has been conscientious regarding data collection, making great efforts to work with the community social EVALUATION | 24 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project assistants to collect important data on child demographics, well-being, family status, etc. However, lack of sufficient community social assistant skills in following rigorous data quality assurance practices resulted in too few observations for certain indicators and inconsistency across districts, sometimes making the data unusable for analytic purposes. At baseline, NORC was only able to include information from administrative data on children in RCCs and was unable to use the data on vulnerable children and families. Additionally, baseline data on RCCs only included information from 4 raions where RCCs were located – Cahul, Ciniseuti/Rezina, Hirbovat/Calarasi, and Nisporeni. Information from RCCs in Visoca/Soroca was shared too late to be included in the baseline report. Part-way through the project, NORC conducted a capacity building exercise to improve the quality of the M&E data, and was therefore able to report data on vulnerable children and families for the endline, but issues remained with both the RCC data and vulnerable family data which is detailed below. Changes in collection method for deinstitutionalized children data. After the capacity building exercise, P4EC prepared consistent excel files which were shared with all community social assistants to collect follow up well-being information from deinstitutionalized children. While a dropdown filter of responses to questions made the data collection more consistent and less subjective across all project raions, it required community social assistants to recode into the excel files information from the baseline assessment of children while still in the RCCs. This caused a slight change in some response options and a difference in numbers reported in NORC’s baseline report for this project. Since consistency in data collection method is more important than a recall bias caused from reconciling data from 2 years ago we chose to use and present in this endline report the slightly different assessment data as baseline followed by endline information for deinstitutionalized children. It should be noted that the endline indicators for Ciniseuti are for one year after deinstitutionalization since most of the cases were then closed (either because the objectives of the plan were achieved, or children had become 18 years old); the endline indicators for Nisporeni and Visoca/Soroca are for two years after deinstitutionalization. The attrition seen for Nisporeni and Visoca/Soroca is due to case files closing or children becoming 18 years old. Using and reporting M&E data on vulnerable families and children. The vulnerable family database of P4EC includes case file information inputted by community social assistants on a rolling basis as they open and close case files on vulnerable children. It included information on files prior to the current DCOF project. Given our concerns regarding this database, we grouped and analyzed the follow case files in this endline report: (i) grouped together all case files opened during May 1 to July 31, 2016 (after the M&E capacity building trip by NORC) to be baseline data, (ii) Followed monitoring on these case files and grouped as endline those whose case files closed by December 31, 2016 (5-7 months of monitoring), (iii) followed monitoring for the baseline group of EVALUATION | 25 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project children and grouped together as endline those whose case file closed between March – May 2017 (10-12 months of monitoring). Survey respondents. The survey data respondents are social workers – both community social assistants and social work specialists – and are likely to have biases, since the evaluation is a reflection of their own performance. While this group is made up of project beneficiaries (in the training and capacity building they received from P4EC), they also gave feedback on community members and parents who were indirect beneficiaries of the project. NORC did not examine differences between baseline and endline results to see if these differences were statistically significant. EVALUATION | 26 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project 4.0 FINDINGS, CONCLUSIONS & RECOMMENDATIONS Question 1: By project end, to what extent have functioning structures been established that can continue to provide on an ongoing basis adequate case-management services for children at risk? Findings: a) Do government authorities and state and non-state service providers have adequate attitude, knowledge and skills to build family resilience, involve, support, and protect children at the local level? Social Service Provider Personnel Structure The project represents a shift in USAID support to initiatives focusing on national systems of care from projects providing services directly to beneficiaries. Two basic program approaches are to support a “responsibilizing” process of key institutions and stakeholder involvement in child protection, as mandated through PL 140, and strengthening the intersectoral collaboration underlying GD 270. The project provides support in strengthening the decision-making and engagement with gatekeeping commissions in preventing institutionalization of children with complex cases, and with mayors who are alerted that children and families may be at risk and appoint multidisciplinary teams to assess and respond to their needs. Multidisciplinary teams are composed of social service, health, education and public order sectors. Additionally, a National Practice Model (NPM) has been piloted in the project. It is strongly tied to both laws and is designed to provide early identification of risk to strengthen better care for children in families and prevent risky situations from escalating. These are further described below. Referral and Assessment System Community social assistants report children are referred to social services by their neighbors, doctors, the multidisciplinary team, their school, and police, though the latter is less frequent. It was thought by one focus group with gatekeeping commission members that at times children are not referred because local authorities think having a distressed child might reflect poorly on them. Service providers believe the general public is more active in contacting the village administration to refer vulnerable children than they have been in the past, though sometimes these referral calls are false alarms. The system of assessing a child’s well-being and status involves an initial assessment done by the community social assistant at the request of the mayor, who is the local guardianship authority. The mayor informs the social worker and the gatekeeping EVALUATION | 27 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project commission of the next steps. The multidisciplinary team conducts an individualized assessment of the child’s needs, checking for progress and contributing to a decision. According to the focus groups with multidisciplinary team members, their teams must follow a number of procedures to properly attend to a case. Every person on the multidisciplinary team has their own role and activity. To begin operations, the mayor issues an order, and based on this order the initial evaluation and the social inquiry are performed to see if the case is one that involves at risk children. If it is serious, another order is issued and the child’s case is opened. The team meets, and a plan is implemented. They are able to discuss problems with the beneficiary and go to their home. After visiting the home of the beneficiary, the team meets again to see if the child is assessed as being at risk. If it is an at risk case, the team gathers again and an order is issued for assembling a team with each member taking on a particular responsibility for the case. If a child is in kindergarten, a representative from the kindergarten is involved. The team looks at the case and determines if the police have been involved at any point and if the child has been harmed. If there is harm, the team also involves the police. If the child is in school, a representative from the school is involved. If the child is of pre-kindergarten age, a nurse would be involved. A community social assistant is the case manager in all cases. Depending on the care plan, medical professionals may take care of the child’s health. The gatekeeping commission - which is involved once the initial and complex assessment is complete and only in cases proposed for family separation and monetary family support - makes a recommendation to the SAFPD by majority vote, collaborating with other departmental teams as dictated by government legislation. As needed, attending the commission are mayors, SAFPD specialists and social workers and other professionals (including doctors). The SAFPD gather key documents for the commission to review. Family and extended family are always invited. The gatekeeping commission’s key role is to check whether the case was fully assessed and provide the most appropriate options for the best interest of the child. When considering child placement options, the commission reviews different viewpoints and assesses prospects for extended family and foster care, with placement centers considered a last option. Parents are given the opportunity to improve their situation and keep the child. The final decision is made by the SAFPD, which is the territorial guardianship authority. After a decision is made, the child’s file remains open, with regular review of the case in line with case management procedures, until the child’s wellbeing is improved in accordance with the care plan. In the event that the child needs immediate placement, emergency foster care, if available, is considered. EVALUATION | 28 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Some community social assistants, gatekeeping commission members, and mayors in focus groups spoke of the change in the assessment protocol through the P4EC project with the establishment of multidisciplinary teams: “Before I was going to the family, and I would mainly go all by myself. I would go with the social worker and we were evaluating in general. And now things changed, I already know, I have to go with the members of the multidisciplinary team, we have new files, we assess according to the well-being areas, according to the protecting factors for consolidating the parental skills. And everything is more detailed, when comparing between the old and the new management.” (Community social assistant, “new” raion) Gatekeeping Commissions The gatekeeping commissions are appointed by the raion councils, the elected governing body of the raion. According to gatekeeping commission members who participated in focus groups, the commission consists of two members of the district council, two area experts, and medical and educational representatives. They are independent and diverse to prevent conflict of interest and enable the council to review the child’s situation and possible placements with greater complexity. Involvement is on a voluntary basis. Calling the commission together comes through the raion officials and this is supported by social assistant managers and heads of other departments such as child protection. Members of the commission meet once a month. All members of the commission can vote. In some areas, the commission members can gather additional information from a child related to a case, which may include more information than a social worker would have been able to gather or process. The committee makes decisions on care issues concerning children, such as whether a child should remain with or be removed from a family and, if the latter, the appropriate type of placement, as well as emergency decisions to address child health and safety – all in accordance with Public Law 140 (PL 140). The key role of the commission is to check whether a case has been fully assessed and provides the most appropriate options for the best interest of the child. It also refers children to services they need and drafts plans of action in relation to children at risk and the roles of specific service providers. The gatekeeping commission also approves foster parents and reviews cases and extends care as needed. Gatekeeping commission members work closely with other services, making referrals to placement centers and collaborating with other departmental teams in keeping with Government Decision no. 270 (GD 270). Depending on the decision, commissions coordinate with the police and legal system, as well as with the social assistance department, for compliance purposes and for support with cases. Gatekeeping EVALUATION | 29 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project commission members described coordinating activities with Advisory Boards of Children (ABC), reaching out to doctors involved with the healthcare of a family or child, interacting with schools, and approving financial support on a monthly basis in collaboration with the government. Commissioners have been trained and supported by the project to guide the decision￾making for deinstitutionalizing children. This includes responsibilizing the mayors for their involvement and support: “Many cases are now resolved at the local level. For complex cases, the Commission does not discuss unless the relevant mayor and the social assistant are present. In fact, the Mayor must be knowledgeable and present the case.” (Manager of social assistants, “old” raion) The project has improved the functioning of the gatekeeping commissions. Some gatekeeping commission members from “old” raions noted organizational improvements thanks to the project’s skills and capacity building activities. These included community social assistants and mayors becoming more responsible in attending committee meetings and the commission now having diverse representation including healthcare, education, and policing; and many children receiving IDs, giving them greater access to services. In general, these gatekeeping commission members now feel they are adequately prepared for their responsibilities and that the trainings and seminars that were provided by the project have been helpful. Issues hindering the functioning of commissions mentioned in focus groups of gatekeeping commission members included: a sense of understaffing and a need for more child specialists; occasions of difference of opinion between commission members that slowed down the processes; and government institutions that played down the severity of issues a child faces to “save face” – this issue was also mentioned at baseline. Additionally, P4EC reported that the volunteer aspect of the commissions sometimes led to commissioners not having enough time to participate. Mayors Mayors are now more involved in and accountable for child protection. PL 140 requires mayors, as the local guardianship authority, to act decisively and be responsible when learning children are exposed to risk. They can be held accountable if they do not act. The mayor’s office is required to check on vulnerable children to make sure issues are resolved. The mayor’s office is supposed to employ community social assistants, who are subordinate to the SAFPD. As a child protection specialist in one of the “new” raions noted, the mayors were previously not aware of PL 140, but now they are aware that they are sanctioned if they don’t fulfill their duties. According to P4EC, a mayor has actually been jailed for lack of response. EVALUATION | 30 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project A child protection specialist from an “old” raion noted: “Now, mayors’ involvement is much more appreciable. These new duties are natural for mayors’ involvement since they know about the situation of a family and child in a community. It is the mayor who decides upon the separation of the child from family.” A child protection specialist in a “new” raion pointed out: “With the new laws and training there is much more responsibility from the side of the Local Public Authority on guardian identification and case monitoring. The old regulations did not cover the aspects related to monitoring, reporting the child’s status, and issues children were facing. The mayors are not delighted with the responsibility they have been assigned. Previously, political differences got in the way of decisions and support. The mayor may have been a member of one party and the social directorate the head of another. Now, there aren’t these barriers getting in the way of decisions and support. The right questions are being asked and collaboration is occurring.” By all accounts, this new accountability framework has stimulated the involvement of mayors in addressing the needs of at-risk children and vulnerable families in their communities. By project end, mayors’ offices were more often seen as supporting children. In focus groups with children from vulnerable families, participants were asked to create a flower using the petals to indicate which types of people helped children the most. Then, they were asked which types of people helped children the least. At the endline, most of the children felt that immediate family, grandparents, and social workers were supports. Many but not all children felt that teachers were a support. Fewer children felt adoptive or foster parents, friends, and mayor’s offices were supports. This was similar to the baseline, with the exception that the mayor’s office was only mentioned at endline. According to social service providers interviewed, although there are strides being made in responsibilizing mayors, and their involvement is increasing, there are still significant gaps where mayors are not involved. Multidisciplinary Teams Focus groups with those from the child care and protection system report that a multidisciplinary team consists of a doctor, the mayor (at least the initial assessment but sometimes for more), a social worker, and potentially one or more beneficiaries. The multidisciplinary team also cooperates and collaborates with educators, the police, health care facilities, psychologists, and NGOs. The team works to coordinate authority EVALUATION | 31 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project and demonstrate that rules and regulations have an impact and are enforced, as well as find or provide resources for vulnerable families. Focus groups with those from the child care and protection system also report that in cases of vulnerable children, the multidisciplinary team works with families, not just the child, and provides support to the family in the form of parenting advice and education. The multidisciplinary team is expected to ensure child well-being is up to the standards in the region. Cases may be started with calls to a hotline, after which the team comes up with a plan to address the situation, which can involve monitoring or material support. Depending on the need, the team may convene a local commission to secure financial resources to support vulnerable children and their families. The team assesses situations by going to visit homes. Later on, the team makes recommendations or referrals to the court or to a mayor, where decisions on child placement or separation are made. Turnover of mayors and other multidisciplinary team members means that frequent training is necessary. The mayor is chosen once every four years, and when the new mayor takes over, the multidisciplinary team from the previous mayor does not necessarily continue. This high turnover rate jeopardizes institutional knowledge being carried forward so regular training of team members is necessary, according to many multidisciplinary team members and social worker specialists. A multidisciplinary team member in a focus group said that even if they know how to do their jobs, training is helpful because they need refreshers when the law changes, and that there are relevant terms that they need to learn and understand. Some community social assistants felt that multidisciplinary teams would benefit from more trainings to understand the broader context of a child’s environment. Some multidisciplinary team members feel that it would be good to have trainings with all of the different members of a team together, including seminars and workshops when a multidisciplinary team is formed, because people only have deep knowledge in their own area of expertise, and they want to have everyone share their knowledge. Key informant interviews reported that joint multidisciplinary trainings occurred on a regular basis during the project. Indeed this is a significant aspect of the project approach to reinforce intersectoral collaboration. Multidisciplinary teams are now resulting in broader responsibility across sectors for child well-being and protection. Social work specialists and community social assistants mentioned that multidisciplinary teams are now more engaged. They are more familiar with the law and the responsibilities of different actors through interactions with social workers and trainings. Multidisciplinary teams have also noticed a greater receptivity and awareness of their work by the community at large. They believe this is a result of changes in legislation, media attention to the topic, and multidisciplinary team meetings EVALUATION | 32 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project in schools. The manager of social assistants in an “old” raion noted the change over the project period in multidisciplinary teams: “Previously, it was much more difficult to make specialists accountable and even to make the multidisciplinary team members work together. We cannot yet claim they are working now at their 100-percent capacity, but their level of accountability is now at 70 percent. The MDT [multidisciplinary team] convenes within 24 hours after the notification file is submitted to work on the case and proposes actions with deadlines. They meet again and check whether all the proposed actions have been met. The mayor acts as the chairperson of the MDT [multidisciplinary team] and his job is to make sure each specialist fulfils his/her actions. There have been cases when specialists had to intervene and take a child from the family at midnight and place him/her in the center until a solution was found. Cases are now solved much faster, since notifications are made quicker and the MDT [multidisciplinary team] started working better.” An example of training leading to improved collaboration was provided by a focus group of social worker specialists. They described initial obstacles facing involvement with the multidisciplinary team because team members said it wasn’t their duty to be involved. However, this has become rarer, and there is now greater understanding of the law – and the trainings have made multidisciplinary teams more aware of their responsibilities. Some multidisciplinary team members note their work has become easier because the team members now know each other better, and they are better recognized by the community. The process of strengthening multidisciplinary approaches represents a cultural change of practice and needs to be ongoing. It is not surprising there are gaps in multidisciplinary team development and varying levels of commitment and progress across the project area. Multidisciplinary members indicate they need more support. Some multidisciplinary team members expressed a need for better communication skills so beneficiaries can understand them better. In some extreme cases, they said that they need protection and psychological support because there are cases where team members are attacked by the beneficiary. Some multidisciplinary team members desired more guidance and written materials for engaging parents and children in kindergartens and other centers. Advisory Boards of Children Since 2010, P4EC has supported local authorities in establishing Advisory Boards of Children (ABC) whereby children monitor the situation of children in alternative care and influence social services planning and policy development. According to P4EC staff, ABC development was inspired by knowledge accumulated in child participation, EVALUATION | 33 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project particularly in the UK, though it is an innovation of P4EC. A widely-recognized international child participation specialist Roger Hart assisted P4EC in developing the mechanism, including reducing token or manipulative motivations for involving children. Significant capacity building by P4EC is occurring in each project raion in the formation, operationalization, and/or support of pre-existing ABCs in the pilot raions. A national ABC also exists that was established later in the project period and consists of children from all over the country. Moderators for the ABCs are selected and trained by P4EC. Children and youth are recruited to serve on the boards and receive extensive training in child and family well￾being and how to effectively talk to peers about child protection issues. Their activities are facilitated by the moderators. ABCs are involved in participatory research on case and service systems including focus groups with peers, providing input to District Councils and other decision-makers on policy and practice (including the UNCRC country report), and they raise awareness within their communities on issues facing children. ABC members visit certain categories of at-risk children, mostly children in fostering and guardianship care, to see how they are doing. They do not visit children in serious conflict with the law, children with significant disabilities, or victims of gender￾based violence. Guidelines and training for these visits exist, which include confidentiality and boundaries on what and how discussions occur. Raional Council The raional council (also referred to as “District Council”) is now perceived by community social assistants and social work specialists as more supportive of improvements in the child protection system, including through the provision of more financial resources. The project assigned a person to each raion to work with all actors at the raion level to support the institutional changes advocated by the project. The project impacted the council’s thinking and actions through technical staff and evidence of improvement in practice, evidenced by the following results. While there was no change in community social assistants’ and social work specialists’ perceptions of how aware the raional council was of national child protection policies from baseline to endline, there was an increase of 13 percentage points in the perception of whether the raional council was supporting improvements in the child protection system. When asked “Does the raional council make financial resources available for child protection?” 88% answered “yes” at baseline and 99% answered “yes” at endline. Similarly, the number of respondents who answered “yes” when asked “To your knowledge, are services for children and families included in the raional budget?” increased from 86% at baseline to 94% at endline. Almost all survey respondents at endline (99%), said that their raion had a child protection service development strategy – an increase of four percentage points from EVALUATION | 34 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project baseline. According to survey respondents at both baseline and endline, the top areas in which raional child protection service development strategy requires improvement are: (1) “strategy should include plans for increased budget allocation;” (2) “strategy needs to reflect local needs and realities;” and (3) “strategy should include plans for development of services.” Table 4.1.1: Support and Awareness of Child Protection Systems Support and Awareness of Child Protection Systems Baseline, % Endline, % Change Strongly Agree Do you agree or disagree that the raional council is aware of national child protection policies? 64 64 0 Do you agree or disagree that the raional council is supporting improvements in the child protection system? 51 64 13 Source: NORC evaluation survey Raional structures are viewed by social service workers as more effective and informal community structures more important by endline. Table 4.1.2 below shows evaluation survey respondents’ feedback on the effectiveness of structures and situations affecting child protection. Raional structures were viewed as more effective at endline, an increase of 8 percentage points; and informal community structures (such as parents’ associations, self-support groups, and community initiative groups) were viewed as more important, an increase of 14 percentage points. Gatekeeping commissions were viewed as equally effective and family situations were equally viewed as not too complex to affect social worker effectiveness. Table 4.1.2: Structure/Situations Impacting Effectiveness of Child Protection Work Training Type Baseline, % Endline, % Change Strongly Agree The raional structures in place for protecting children are effective 54 62 8 The gatekeeping commission is effective in making best interests determinations in the placement of children 71 74 3 The informal community structures in place for protecting children are important for helping children to be safe 59 73 14 Family situations are too complex for me to be effective in my work 16 13 -3 Source: NORC evaluation survey Community Social Assistants and Social Work Specialists Community social assistants work at the community level and are subordinated to the Social Assistance and Family Protection Directorate (SAFPD) at the raion level. EVALUATION | 35 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Community social assistants are qualified as social workers in Moldova (holding a university degree) who provide social assistance services. This Directorate employs social work specialists, who are trained social workers. In this report, we use “social worker” to refer to both these populations (community social assistants and social work specialists). The Directorate sits within the service and policy framework of the Ministry of Labor, Social Protection and Family. SAFPD specialists assist children, families, persons with disabilities, the elderly, and other populations identified as vulnerable through case services and specialized programs such as foster care, accessing social protection financial assistance, and home-based care. Mayors say community social assistants play an important role in alerting the mayor to the vulnerable children in the community. This is in part because people such as children and neighbors will tell the social assistant about children left alone. Almost all social workers surveyed received training between 2014 and 2017 from the project. At endline, survey respondents were asked whether they received any training related to their job from 2014 until the time of the survey administration. An overwhelming majority (96%) of survey respondents said that they received training since 2014. Over three quarters (76%) received training from P4EC or Terre des Hommes, an organization contracted through the project for training. Trainings that were part of the project were also led by Axa Management Consulting, various ministries, and individual consultants. Social workers found the trainings to be useful, particularly the guidelines, and they have largely started applying what was learned. Over 90% of the community social assistants and social work specialists surveyed were training regarding PL 140 and the interagency cooperation mechanism as seen below in Table 4.1.3. Each survey respondent that participated in one of the trainings presented in Table 4.1.3 were asked to rank the usefulness of the training with 4 being the most useful and 1 being not useful at all. Overall, all training types received high ratings of 3.62 or above, signifying that they find all the training offered highly useful. The highest ranked training was on child well-being and family protective factors approaches, with an overall ranking of 3.82. Many community social assistants and social work specialists confirmed in the focus groups that they found trainings to be enriching and beneficial. They specifically mentioned learning new terminology related to improved procedures and how to use certain tools. All stakeholders interviewed in the evaluation and some social work specialists who participated in focus groups indicated that the guidelines produced through the P4EC project were useful and influential. Guidelines exist on foster care, gatekeeping, supervision, case management and family support. Since they are officially approved, EVALUATION | 36 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project they carry weight. They offer concrete and practical examples, detailed descriptions and help specialists to deliver trainings and find answers to their questions. Each survey respondent that participated in these trainings was also asked whether or not they had begun implementing elements of the training. At least about 90% of respondents had started implementing all the trainings with the exception of communication with mass media and the PANDA program11. PL 140 was also the most highly implemented training (97%). Social work specialists in one focus group reported that they had spread the knowledge acquired in trainings with community social assistants and the mayor. Table 4.1.3: Types of Training Received Between 2014 and 2017 Training Type Participated, % Ranking Implementation, % Implementation of the Law nr. 140 regarding the special protection of the children in risk and children separated from their parents. 96 3.66 97 Implementation of the Inter-agency cooperation mechanism on Identification, prevention and assistance of cases of abuse, neglect, exploitation and trafficking. 92 3.62 88 Provision of Family support service: standards of care and new Regulations. Application of the case management and supervision tools. Application of the revised case management tool in Family support service. 88 3.68 89 Child Well-being and Family protective factors approaches. National Practice Model. 84 3.82 89 Communication with the Mass media entities. 77 3.7 79 Development of PANDA program, role of the CSA in the implementation. 64 3.68 75 Source: P4EC project training records Social workers at endline are less confident of their skills, but feel more supported in their work. As shown in Table 4.1.4, the percent of social work specialists and community social assistants who said they understood the roles and responsibilities of their jobs dropped from 88% at baseline to 82% at endline, despite 96% having reported that they participated in a training related to their profession since 2014. However, the percentage of respondents who “strongly agreed” that they felt supported at their jobs and got support from their supervisor to do better at their jobs increased from baseline to endline. Again in Table 4.1.4, the percentage of respondents who were “very confident” about their ability to do a good job in each area of their work” dropped for 11 All raions received PANDA training but due to financial constraints, only nine out of the 11 raions agreed to implement the program and integrate it within their local social protection system. EVALUATION | 37 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project each category between baseline and endline except for referrals to other services. This is in line with the idea that fewer respondents understood the roles and responsibilities of their jobs. While respondents at endline seem to have less confidence in their own skills, they have greater confidence in the social work workforce in general. Endline respondents who felt the workforce in their raion or community was “highly skilled” in adequately protecting children and fully supporting vulnerable families increased by 10 percentage points from baseline. Community social assistants who participated in focus groups were divided over whether they felt prepared to meet the needs of at-risk children. Some said that they did not have the authority and support to do so, nor the time needed to effectively do their work. Others said that they felt prepared and well￾trained to handle situations. Table 4.1.4: Social Worker Opinions on Preparedness for Their Jobs Do you agree or disagree with the following Strongly Agree Baseline, % Endline, % Change I understand the role and responsibilities of my job 88 82 -6 I feel supported in my job 65 75 10 My supervisor provides support that helps me to do my job better 69 76 7 How confident do you feel in your ability to do a good job in each of the following areas of your work? Very Confident Baseline, % Endline, % Change Identifying clients 60 47 -13 Completing assessments 74 63 -11 Interviewing / talking with clients 68 54 -14 Developing care / service plans 54 45 -9 Referrals to other services/organizations 57 61 4 Representing clients in court 48 41 -7 Representing cases in the gatekeeping commission 71 68 -3 Completing paperwork for government assistance 70 61 -9 Making home visits 83 82 -1 Working one-on-one with clients in my office 81 73 -8 Working with adult clients 72 63 -9 Working with child clients 74 58 -16 Running group sessions for clients 52 39 -13 Completing reports to my supervisors 69 68 -1 Supervising other workers 61 59 -2 To what extent do you believe that the social work workforce in your raion/community has the necessary skills and knowledge to: Highly skilled Baseline, % Endline, % Change Adequately protect children? 48 58 10 Fully support vulnerable families? 48 58 10 Source: NORC evaluation survey The findings in the above table indicate respondents in the survey felt they received greater support over the project period, though they expressed less confidence in EVALUATION | 38 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project aspects of their work. Key informant interviews with these same workers provided some perspective. The responsibilizing and intersectoral collaboration supported through the project has relieved some pressure these workers felt as often being the sole service providers for children and families in crisis. There are now more actors sharing this responsibility. This is significant. However, the system change that brought stronger case management assessment and response tools and greater intersectoral collaboration through the project is also significant and likely the reason that social service workers felt less confident at project end. Social service workers have learned more about their clients and the challenges they face, and are able to more definitively identify and define risks they have, revealing gaps in their own ability to address these risks. An example is behavioral issues of teenagers. Also, the system requires new procedures for paperwork and in casework, creating questions for these workers about how to efficiently work in the new environment. Greater collaboration with other stakeholders presents another set of responsibilities and work details that the social service workers are still coming to grips with. Many social workers at endline indicated that they struggle in assessing whether problems are likely to reoccur in vulnerable families and the sustainability of progress made in addressing risks. They would like greater training in building resilience through casework. At both baseline and endline, some community social assistants felt that they were burdened with too many diverse responsibilities and that there remained a great deal more progress to be made in sharing the burden with other sectors. A few community social assistants asked for greater protection and psycho-social support in dealing with beneficiaries they interact with. This is a common need globally with caseworkers supporting vulnerable and at-risk populations. These clients can be aggressive and demanding and present situations that are very disturbing to hear, affecting the psycho-social health of a service provider. Social worker caseloads have dropped somewhat and there is more time to visit families. The P4EC project provided technical assistance in modeling clinical case management supervision, through training and select case conferencing. Most of the endline social worker evaluation survey respondents made weekly contact with prevention and active cases, which was a similar pattern as baseline. There was a major shift in the frequency of contact for monitoring cases. At baseline, 58% of respondents reported following up with these cases weekly, while at endline only 6% of respondents reported making contact weekly and 48% reported meeting on a monthly basis (see Graph 4.1.1). Community social assistants reported having an average of 5.6 active cases at baseline, which dropped slightly to 5.0 active cases at endline. The workload was EVALUATION | 39 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project different among the social work specialists due to their supervisory role. At endline, the specialists reported having an average of 20.3 cases, which was a slight rise from the average of 19.5 cases reported at baseline. Social workers now have more time to visit families. The percent of respondents who reported “always” having time to visit the families they work with increased 8 percentage points from 19% at baseline to 27% at endline. Roughly 60% of respondents reported that they had enough time “most of the time” to visit with the families they work with at both baseline and endline. The main factor that prevented community social assistants and social work specialists from having enough time was “paperwork that has to be completed,” with over 90% of respondents reporting this reason. This is due to the new case file system, described below. Key informant interviews indicated that there is greater prioritization of SAFPD caseworker time on the active, at-risk cases. Also, the project is designed for greater burden sharing among other sector service providers such as health and education. Workers in SAFPDs reported a trend of spending less time on cases these other sectors should be managing more directly. Graph 4.1.1: Frequency of Contact with Case Families (%) Source: NORC endline evaluation survey Some community social assistants felt that the involvement of multidisciplinary teams increased the burden on them and created some new challenges. Although involving police, school, and medical sectoral participation was generally recognized as an improvement, some community social assistants say that they now have more responsibility and more communication demands made of them. Social assistants in this 14 13 0 55 58 6 18 15 11 10 11 48 0 10 20 30 40 50 60 70 Prevention case families with whom you work Active case families with whom you work Monitoring case families with whome you work Daily Weekly Bi-weekly Monthly EVALUATION | 40 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project evaluation reported they still face challenges in juggling their many responsibilities with vulnerable populations ranging from children to the elderly and individuals with disabilities. However, they feel the burden sharing through the capacitating of gatekeeping commissions, mayoralties, and multidisciplinary teams has shifted the burden more broadly across the service spectrum. Some community social assistants felt the many actors involved in decisions and actions were creating constant mistakes and obstacles. Also, community social assistants and social worker specialists cautioned that there needed to be safeguards against delays from the multidisciplinary team approach when decisive action was required for urgent situations like the removal of a child from a dangerous situation. One social worker specialist gave an example of a Roma boy that suffered epileptic seizures, as well as some deviant behavior, but was unable to be helped. Social Service Processes Placement Stakeholders at various levels (ABC, community social assistants, children, and multidisciplinary team members) confirmed that multidisciplinary teams assess placements jointly with community social assistants, psychologists, teachers, and doctors. The gatekeeping commission recommends the placement decision, though it may be reevaluated as new facts on the case emerge, and the SAFPD makes the final decision. ABC members are invited as observers and sometimes as participants, as are priests who sometimes organize activities for children and get to know them. In some families, the opinions of neighbors and close relatives are also solicited. The child’s homeroom teacher and the school principal also participate in the placement process, helping to talk with the child and their parents. Regular monitoring is necessary after placement. Contact with a psychologist is more frequent (daily) if a child is placed in formal care of a center versus being placed with relatives. In terms of procedural changes in the placement process, focus group participants had mixed feelings about whether or not there had been changes over the past few years. Some mayors, foster parents, and parents pointed to more intersectoral collaboration, an increase in guardianship assistance, and an increase in the benefit amount received by foster parents. Other community social assistants and foster parents said there had not been a change in the placement process. Child Participation Child participation is an important principle in protection and rights frameworks. The P4EC project trainings emphasized securing input from children on decisions impacting them. This occurred through skill development, talking with children in the assessment EVALUATION | 41 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project and case planning stages, and in follow-up activities. Advisory Boards of Children, as described below, are a mechanism to take into consideration the views of children. The child protection specialist in a “new” raion said, “When a child is deinstitutionalized, and before the best placement option is identified, the child’s opinion is very important.” Children’s opinions are considered, especially for those above 10 years old. Most children, parents, multidisciplinary team members, and community social assistants in “old” raions reported that the opinions of children are being considered, while some in child and foster parent focus groups believe that children’s opinions were not being taken into account. Some parents and foster parents felt that a child may not ever want to leave his/her parents, despite it being an inappropriate environment. Some parents and multidisciplinary team members felt that a child’s opinion should be taken “more or less” into consideration based on age – if a child is 10 years old or more, he/she should be asked his/her opinion: Mother: “I think that the child aged 2 years old will not understand if it is good or not. Of course the child wants to be next to his mother, because she is the mother, even if she is drunk. And we, the others, we look around, and we see that it should not be like this. And how to ask the small child aged two years old, if he wants to stay with the mother or not?” Mod: “So if the child is small, he cannot?” Mother: “He cannot say it. And if the child is older, the child may understand that this is better for them. That here it is better. And he has to live like this.” (Parent from family at risk of separation) Interestingly, only one focus group with children outright said that children’s opinions should be considered. They felt that the child might be more comfortable staying with his/her parents, despite being in an environment destructive to his/her well-being. One focus group with children said that though children’s opinions may be given, they may not be listened to. Children offered a variety of perspectives on how to make their opinions known, such as protesting, threatening to do something like running away, yelling, and explaining what they want to do to their parents. ABCs play an important role of participatory involvement of children in child protection issues and provide an important peer resource for children. An important premise of ABCs is that children at times are more likely to open up to their peers rather than adults about sensitive issues of concern to them. Many children in ABCs and those from the social service sector verified this is occurring through ABCs. EVALUATION | 42 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Children from the national-level ABC reported that children develop close relationships with ABCs and that there is a higher probability of students going to other children with issues than to a teacher or social assistant. A child protection specialist in an “old” raion explained that ABC members have been very involved in monitoring the situation of children placed in foster families and at placement centers. The directorate recently made an agreement with the ABC to also monitor children under non-paid guardianship: “Our directorate highly appreciates the support provided by the ABC, as the situation is different when a child speaks to another child. Following these visits/discussions, the ABC members produce an assessment report which they present to the social assistance heads of services and directorate. In their report, they describe how the children feel, what problems do they face and make recommendations. As specialists, we take these recommendations into consideration. We make sure they are not doing our work since these are just children. ABC members are never involved in very severe cases. Also, they are not invited to attend the gatekeeping commission sittings when it must discuss severe/complex cases. The work and expenses of ABCs are worth the effort, as there are many cases when ABC members provided valuable information specialists did not have access to. For example, a child’s needs are met, but he/she is still not feeling comfortable in the family and could not speak to the guardian about this. Subsequently, social specialists and ABC members provided assistance to this child.” (Child Protection Specialist, “old” raion) The family protection and support specialists in a “new” raion and the head of the Education Directorate in another “new” raion also spoke of the important input from the ABCs, including the meaningful input provided in decisions and support for children in fostering and guardianship care: “I have worked in child participation for a long time. I can tell you the ABC mechanism is useful since children feel more comfortable speaking to somebody of their age, rather than adults. ABC members have identified several cases when situations of children were not well known and specialists’ involvement was required.” (Deputy Director of Education Director, “new” raion) Some gatekeeping commission members and members of the national level ABC felt there is a need for additional ABC member training12. National-level ABC members mentioned needing more training in understanding children and using different 12 There are only a few children who are members of the National ABC who are from the pilot raions and received more training from P4EC. EVALUATION | 43 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project techniques such as interviews and games to achieve and measure their comprehension of the children’s well-being. An assessment on child participation models in Moldova, including ABCs, funded by the Oak Foundation occurred in 2016. 13 It found that because children living in alternative care settings did not know the ABC members, many were shy and reluctant to share their personal experiences. It was easier to meet with younger children to play games and to listen to their views, but harder to interview teenagers. In the assessment, some ABC members reported feeling powerless to help resolve other children’s problems, although a lot of efforts were made by ABCs to collaborate with the local authorities to improve children’s lives. The assessment also found that the competitive selection models for ABC membership (and other models) gave an unfair advantage to children who were good in their studies, resulting in a lack of diversity. It recommends involving more children who had experience in accessing social services or living in alternative care so that they could draw upon their own experiences and expertise. The report indicated that P4EC recognized this and was working toward greater diversity. P4EC clarified that the recruitment of ABC members was not competitive on the basis of school performance. Any child aged 12-17 who wants to engage in fostering the rights of children, including those placed in alternative care can become an ABC member. However, they must demonstrate skills that they can use to effectively advocate for children. The ABC is not a rehabilitation group for children, nor a tool to help them overcome their problems. P4EC reported that approximately 50% of ABC members were children who had been placed in kinship care, those whose parents were abroad, children with special needs, those in foster care, children of ethnic minorities, and children from families whose parents were divorced. The Oak-funded assessment says ABC contribution to child participation is recognized by professionals and authorities from the local to national level – particularly in informing the development of social services, promoting child rights, and ensuring children in adversity have a strong voice. The assessment encourages ABC participation to be “safe, sensitive to risk and address concerns relating to privacy and dignity” and to ensure confidentiality of children in care when they are visited by ABC members. The report stresses the need for children living in care to have sufficient information in advance of monitoring visits, with genuine opportunities to choose whether they would or would not like to be interviewed or be part of a group discussion with ABC members. There is at least some risk that children’s confidentiality and privacy may not be respected. ABC members do receive confidentiality training and sign confidentiality 13 Summary Report on child participation assessment Moldova Child Participation Assessment: Oak Partners, Moldova (summary on child participation, July 2016) EVALUATION | 44 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project agreements. P4EC reported that ABC members do not visit children they know and the children they interview are in different communities. Additionally, children who are placed in services are informed by the social workers that they have the opportunity to meet with ABC members if they so choose. They receive information about ABC and its purpose and sign an agreement to participate or not in the interviews. They can refuse participation at any stage. Case Management Over the project period, P4EC has led the development, piloting, and integration of a new case management tool for complex cases. The purpose is to establish a more uniform practice that assesses children on a set of eight well-being indicators, leading to more clearly articulated case management that identifies necessary multidisciplinary approaches. The second part of the tool assesses families for strength-based protection support to correlate with the well-being of children.14 The tool began to be used by all SAFPD social assistants in January 2017 for all new case files opened in project raions. However, the well-being assessment and family support approaches are intended for application to all previously existing and ongoing casework, either under the old or new case file formats. The tool was piloted in the 11 project raions and was approved by the Ministry as an official country-wide working tool. With project support, all community social assistants and district family protection specialists across the country were trained in the new case management tool. The project also developed a practice guide on case management implementation. Development of the tool began before the project. It blended two systems together into a unified tool adapted to the Moldovan context based on lessons learned, in part, through the deinstitutionalization process and experiences in strengthening family￾based care. Some community social assistants and social work specialists described it as a shift in focus to concentrate on the child and their interests. Reasons given were that more relevant information is now collected on individual circumstances of children – the old model was more of a general nature, whereas the new case management process is better suited to the needs of complex cases since it ensures the spectrum of well-being indicators are assessed and there are more clear-cut actions to pursue; the more comprehensive assessment of a child leads to clearer involvement of multidisciplinary approaches and teams in case work; and the new system is more 14 The well-being indicators center on activism (the ability to take action when needed, initiative) respect, responsibility, inclusion, safety, health, achievements and affection. The family support protective factors in the tool are parental resilience, practical support when needed, connections/social ties, knowing the parenting and child development principles, and knowing the socio-emotional abilities of children. EVALUATION | 45 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project dynamic in identifying the family support approach in case management to better meet the needs of children: “In old-type case management, we did the evaluation in general for the family. We were paying attention to the dwelling conditions, to how clean the house was. Now we do the evaluations for every child” (Community social assistant, “new” raion) “Social assistants were scared when the new [tool] was approved. Trainings occurred in groups, then smaller groups and then individualized with cases for social assistants. This approach enabled improved supervision on cases so the assistants are never left alone on a case. This is a very good tool for the child, as it helps specialists see what is good and what is bad in the situation of a child and where support is needed, and then the focus on the parents and the support they need to better care for the child. The tool involved all actors (across sectors) who see more clearly the situation of the family and a more coordinated approach.” (Social Assistant Manager, “new” raion) The case files reviewed were of high quality. The Child Protection Expert to the endline evaluation completed a review of eight different case files, including five organized under the new case file system developed through the project and three under the old system. These cases were purposively selected for various risk categories amongst a broader set of cases social assistants were asked to present for review. The files in all cases were thick with many documents and forms. The review randomly selected items to check in each file from a list of case file requirements, such as beneficiary identification documents, assessment forms, child protection/gatekeeping commission decisions, mayoral orders, referral forms, and monitoring/intake reports. In all of the case files reviewed in the evaluation, whether under the old or new system, disrupted or not disrupted, the review found all the necessary items on the list of items required by the SAFPD for the file process. The case files were thorough and the input and opinions of beneficiaries were well documented. All the social assistants using the old case file system are adopting child well-being indicators and assessment and family support methodologies developed through the new case management system. The intersectoral and accountability mechanisms being strengthened through the project were also evident in each case file. For example, the involvement of mayors and their decisions were well-documented, as were the activities of members of the multidisciplinary teams. The new case management tool is more useful and higher quality but, at least initially, requires more time to fill out and training of social assistants. Some community social assistants in focus groups said that the documentation has become more detailed and that procedures are clearer and cases are better recorded, allowing newcomers to the EVALUATION | 46 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project cases to understand the status because everything is present in the case documents. Now the case management files have sections for each specialist handling the case to fill out, giving everyone a role. Some social worker specialists feel the quality of the process has improved, as it is now clear, established, and structured with training and a set guide in which all the procedures are written down and described. After trying out the tool in both an “old” and “new” raion, the social assistants decided to transform all their case files to the new system since it was useful to them for their case work. “The new model gives easier and clearer milestones for social assistants to assess well-being and protection. Specialists can see accomplishments.” (Child protection specialist, “new” raion) Considerable training and time commitments will be required for social assistants to transition to the new process. This must happen within an already busy and sometimes stressful work environment. Many community social assistants and social work specialists complained of the large amount of paperwork. This is, in part, a result of the extensive written records that need to be kept with complex evaluations. A child protection specialist in a “new” raion explained how different templates are required for various categories of beneficiaries and a new file must be started individually for each child in the family. This specialist would rather have one process for an entire family. However, the new tool was seen as easier in coming to and drafting case objectives for complex cases and recording actions taken. The specialist said, “I am convinced that after a while, specialists will get used to it. Our input was not always taken into consideration but we assume since the tool is new, it will be evaluated and changed as needed.” The Vice Minister of MoLSPF believes the new “case management model is ‘friendlier’ for both the social assistant and the child. It eliminates social assistant’s bias since all fields need to be filled in.” The Vice Minister was aware of some of the difficulties in implementing the new case management process and attributed it, in part, to high social assistant turnover and lack of skills for some. The tool requires ongoing training and support. The Ministry explains the new case management information will be included in the Social Assistance Automated Information System that is under development. This will provide the capacity to assess the quality of case work and inform policy. Other child welfare NGOs in Moldova were invited to provide input and be involved in trainings on the new case management tool, though some wish they would have been involved at an earlier stage in its development. The Lumos Foundation is now using and promoting the tool. Lumos reports social assistants are slightly scared by the large volume and complicated format of the document. However, in the opinion of a Lumos leader, such changes are necessary and resistance is normal. Simple formats do not allow specialists to collect much useful information. The team at CCF (which represents EVALUATION | 47 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Hope and Homes for Children in Moldova) believes the tool is useful, provides needed structure, and gives quality to case work. Social assistants are finding it complicated and there is concern about whether and how it can be replicated in other districts. UNICEF identified the need to integrate the case management tool with the process of home visits and early identification of risk through the emerging National Practice Model and urged deliberate coordination among NGOs and stakeholders. b) Have government authorities and state/non-state service providers adopted a joint approach to build family resilience, involve, support, and protect children at the local level? Institutionalizing joint approaches as a norm in Moldova’s child protection system is a cornerstone for the project. While PL 140 delineates responsibilities of each sector, GD 270 identifies the intersectoral processes to reinforce the work of each authority. Important mechanisms requiring joint approaches by these two sets of laws include independent and diverse gatekeeping commissions appointed by raion councils and mayors as important child protection actors at the local level with jurisdiction over multidisciplinary teams for assessment, recommendations to decision-makers and response. The project approaches to collaboration emphasize intrasectoral training, guidelines applicable to all key child protection sectors, casework tools on child well￾being and family support requiring joint approaches, and enabling the intersectoral child protection community to “speak the same language.” The director of education in a “new” raion said, “After implementing the project with social assistants, we realized most of our failed cases are due to our lack of collaboration.” There is greater collaboration among different sectors. SAFPD leadership in the raions of the evaluation agreed that though the multidisciplinary team concept had been around for a long time, specialists were previously not really working meaningfully together and mayors lacked involvement. This has changed with the P4EC project though there remain pockets in raions where mayors are not very responsive: “Mayors are now on the spotlight, they are the ones who coordinate, and social assistants are their right hand. About 3-4 years ago, social assistants were working a case alone. Now they work on solutions together with the mayor. In the case of irresponsible mayors, the social assistant shows them the law and reminds them of the sanctions they might be subject to if they do not collaborate…In the past, our relationship with police was colder. Police never crossed with social assistance specialists while now, even they collaborate with social assistants including when police must issue a protection order or to make home visits.” (Social assistant manager, “new” raion) EVALUATION | 48 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project The same sentiment was expressed by SAFPD leadership in another “old” raion who said, “More open collaboration has replaced earlier difficulties in working with police and the health sector.” Participants in focus groups with community social assistants and gatekeeping commission members also noted that they were pleased with their collaborations with the multidisciplinary team. In all the raions, social assistance key informants reported that there was significantly greater collaboration as a result of the project. This is an indicator of changed attitudes related to operationalizing GD 270 and is evidence of greater involvement of multidisciplinary teams at the de-centralized level. This is particularly important to lifting the burden of social assistants who, at times, have been overwhelmed by their case loads. “Social assistants now only get involved when there is a more complex case and the other specialists do not know how to deal with it. Education is good example. For a child who is not attending school, the teacher submits a notification to the mayor, not just social assistance. Before, we used to ask the question, “Why only me?” Now we have a team to work on a case. There is much more active involvement of the MDT [multidisciplinary team] specialists and fewer requests are submitted to mayoralties requiring social assistants.” (SAFPD Head, “old” raion) There were a number of advantages to collaboration pointed out by actors in focus groups when discussing the referral and assessment system. Some community social assistants, gatekeeping commission members, and social worker specialists spoke of the support departments give to each other and the usefulness of the collaboration in decision making that, in turn, benefits the child: “This is a productive collaboration. It is efficient because each sector knows the case from their area of expertise. The issue is looked at from different perspectives. The police know one thing and the hospital knows something else, so we pool the information. Everyone expresses their opinions and gives their advice which makes it successful.” (Gatekeeping commission member, “old” raion) One mayor spoke of the positive collaboration between mayors and social workers: “In our community, the social workers and the head of the Social Assistance Directorate come to me and they point to the names of people in the list who should be accepted for social aid – unofficially. Yes, this is unofficial. They say “Look, there are 7 applications of which 3 should be approved and 4 should be rejected.” And when an issue arises, we tell the social worker about this, and the social worker reminds the mayor that it is he who had selected the families. The selection process is transparent and collegial. We have great relations with the social workers.” (Mayor, “new” raion) EVALUATION | 49 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Coordination across different stakeholders has improved. Coordination has improved, according to a variety of focus group participants. One focus group with community social assistants said they now have greater authority in their jurisdiction thanks to the involvement of police and the mayor. Participants from this group also said that communication between organizations has improved. Participants in a focus group with multidisciplinary team members said that they are more familiar with one another now, and that they have been able to provide more monitoring of at-risk cases. A focus group with social workers said that there is now more involvement of community members in helping decision makers become aware of at-risk cases thanks to such tools as the anonymous phone line. Two focus groups with gatekeeping commission members and multidisciplinary team members said that thanks to coordination, more children are protected. A focus groups with community social assistants reported that parents are now taking on greater responsibility with more active involvement of social service providers. Joint trainings are viewed as enabling greater responsibility and collaboration. Some community social assistants indicated that joint trainings amongst different social service providers such as mayors and medical assistants led to each of these sectors becoming more responsible and aware of the importance of their involvement. The increased collaboration across sectors has, in the opinion of some social worker specialists, helped to decrease the number of children separated from their families. For example, the health sector now works more directly with the mothers to enhance their child care skills. Civil society organizations were perceived as more engaged in protecting children and preventing separation by social workers by endline. 90% of respondents at endline “strongly or somewhat agreed” that civil society organizations were engaged in protecting children from violence, abuse, exploitation or neglect; 87% of endline respondents “strongly or somewhat agreed” that civil society organizations were sufficiently engaged in preventing separation of children from families. In both cases this was an increase of 9 percentage points from baseline. Some challenges with collaboration were reported. According to one focus group of community social assistants, it was difficult at times to bring multidisciplinary teams together due to scheduling issues. When the team could not be brought together, present members had to make a decision because of urgency. This was difficult because agreement should have come from a number of team members in order to proceed with placement procedures. This focus group also noted a lack of staff in rural localities, meaning that half of the members came from town and were unfamiliar with the children and their situations. Social assistants also still felt that they were taking on the largest burden of responsibility, and faced challenges of being overbooked. Some EVALUATION | 50 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project gatekeeping commission members mentioned a reluctance to be fully transparent in order to save face, such as a failure in the school to prevent dropouts: “A disadvantage may be that in some teams, team members may hide information, such as hiding the fact that the child is a dropout. The teachers don’t want to make the notifications. The school does not report to the Education Directorate or to the other authorities. Although the school reported that the child had finished the academic year, he actually dropped out of school and was wandering the streets of Chisinau.” (Gatekeeping commission member, “old” raion) Another issue expressed by mayors of newer raions was coordination among the various multidisciplinary actors. This made it difficult to track the status of a child’s case, leading to delays in processing and placement. Despite improvement, some challenges still exist with engaging the health sector. Though involvement of the health sector in child protection has strengthened throughout the project raions, most SAFPDs reported that there were still challenges in engaging medical specialists. This was due to the many demands of time on doctors and nurses, not necessarily due to their lack of commitment. Some community social assistants and mayors indicated that medical staff were particularly responsive members of multidisciplinary commission teams. Generally, respondents noted that health workers arrived at the most important meetings or when there was a situation that required their expertise. In other circumstances, social assistants explained that it was reasonable for health workers communicate or advise through email or over the phone if they were unable to physically attend meetings. A child protection specialist in a “new” raion said, “We need more involvement and time from medical specialists. They work at a different pace and we may need to find some new approaches with them. Some villages have one doctor only.” According to social service workers surveyed, the two hardest services to arrange at the endline period were health clinics (35%) and mental health services (30%). This finding was supported by key informants who said medical professionals were the least likely to collaborate on the intersectoral mechanism. Mental health services are lacking throughout Moldova, let alone for the at-risk populations targeted by this project. Additionally, mayors were viewed as the least engaged group. They were described as ill-informed and resistant, evidenced for example by not showing up for training. There was not much change in community social assistant perspectives on cooperation and collaboration between service providers. The survey data indicates a marginal change in perspective of social service workers as to the degree of coordination EVALUATION | 51 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project between actors (from low to high coordination) between the base and endline evaluation. However, focus group and key informant discussions contradict the survey data and indicate greater cooperation and collaboration between service providers at endline. Additionally, the points of reference may have changed the perspective on what constituted medium or high coordination/collaboration. Expectations could have been much higher at the endline. Table 4.1.5: Extent of Coordination and Collaboration between Stakeholders to Meet Child Care and Protection Needs of the Family Q40: What is the extent of coordination and collaboration between the following actors toward meeting the needs of families? High Coordination/ Collaboration Medium Coordination/ Collaboration Low Coordination/ Collaboration No Coordination Baseline, % Raional/local government and non-government actors 38 50 11 1 National government and raional/local government actors 27 50 16 4 Raional child protection actors and local actors 61 36 2 0 Endline, % Raional/local government and non-government actors 39 53 7 0 National government and raional/local government actors 31 55 9 4 Raional child protection actors and local actors 58 41 1 0 Source: NORC evaluation survey About half of the evaluation survey respondents (53%) “strongly agreed” that they were able to call on the support of other service providers in order to provide integrated care to families at baseline, which dropped to 47% at endline. Health clinics, the police, and schools/teachers were considered the most valuable service providers at baseline and endline by over 90% of survey respondents. By endline, domestic violence services and community organizations were also viewed as valuable by over 90% of respondents surveyed. National Practice Model When discussing GD 270, the Minister of MoLSPF explained the first step was to work with children already known to have risk. A shortcoming has been that prevention is not strongly enough reflected in the normative framework. The second step, therefore, is establishing mechanisms for early identification of risk. To do this, the project is EVALUATION | 52 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project establishing a National Protection Model (NPM) that has been formulated and piloted through the project and, according to the Ministry, will eventually be articulated in national regulations. The model builds from earlier joint work foundations of inclusive education and multidisciplinary team development. Lifted from an established practice in Scotland, the essence of NPM is equipping the education and health sectors with skills and tools to identify early risk in the settings where they can observe and interact with children – such as schools, clinics and home visits. “Known” or “appointed” persons are given responsibility to observe or track children or groups of children. As risk is identified, procedures in the NPM then prevent further escalation. Many of the existing services and supports at the local level can potentially swing into action as needed. P4EC’s director explains the NPM as, “The right thing at the right time for what we need. We aren’t smothered by big problems and we are seeing successes in its piloting. Scotland took ten years to develop the NPM and we are benefiting from this.” The piloting of the NPM is in final stages in Ungheni, Falesti and Calarasi. Other project raions are linked to the NPM since learnings from the pilot are being integrated into training and capacity building activities – recognizing the pilot will be expanded to these raions in the near future. The NPM is running parallel to other early risk identification processes such as medical home visiting being supported by UNICEF. Some NGO stakeholders see the NPM as being able to blend in well with other early identification initiatives, while others are confused or concerned about how the NPM will work within or even replace other child protection procedures. Two participants in NPM-related trainings are Cahul’s deputy head of education and the head of a family clinic. Around 765 specialists of various sectors attended the trainings delivered in 40 localities. Mayors and leaders of sectors such as education, health, and the police had special meetings reviewing the regulations and their responsibilities. The well-being indicators being introduced were new to everyone. Participants also learned observation skills for early risk identification and procedures for notification of risk. The top health and education officials in an “old” raion indicated the education and healthcare sectors have always collaborated to some degree, but the project brought the sectors closer together through trainings at all levels along with the SAFPDs. The collaboration is now more structured and focused, particularly on strengthening parenting skills. Educators expressed some initial reluctance on the greater amount of work involved with the NPM, but after going through training and application the value was better understood – particularly what risk factors to look for. The education director said, “With this understanding there comes a greater sense of responsibility. Our EVALUATION | 53 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project educators realized they were not aware of the child’s situation. To feel good, a child needs things apart from knowledge, they need understanding and respect.” Moving the NPM ahead will require continued training and refinement, according to the director. For example, the known person and observation process needs to be adjusted depending on the type of school. Kindergarten children are always within view, but older grades and students are not. There may need to be persons assigned during break periods, but this may conflict with preparing for lesson plans. The health director recommended more practical forms and processes that take less time for medical personnel to fill out. The director said, “The extra time it takes to do this poses some risk that other responsibilities, such a maternal health and improved infant mortality might be affected.” Both directors were supportive of the NPM and gave examples of results. For example, the model has led to identifying child homelessness and the local authority becoming involved to address this need. In another “old” raion which is one of the NPM piloted raions, the education director hopes and believes the NPM will become its own law. The changes from this, she expects, will go right down to the job descriptions of educators and other specialists. The piloting in this raion occurred in rural and urban areas, including two kindergartens, a primary school, a gymnasium, three high schools, and three medical units. An important lesson in the pilot is the importance of all specialists applying the same observation criteria for objective assessment. Peer colleagues not using the tool missed indicators of risk. Another lesson, cited by the director, is that inclusion is not just about having an individually adjusted curriculum. Much more knowledge about a child’s life is needed: this important for a child’s well-being and performance in school. The MoLSPF acknowledged there had been some resistance to the NPM from the health sector. The next steps after the piloting experience is concluded, well documented, and discussed is finalizing the model’s tools, inserting it into the normative framework of Moldova, and then developing the cooperation plan for all sectors. The Vice Minister said there was some risk the NPM would be not be uniformly applied at a national scale due to the big need for specialist availability to train stakeholders on applying the mechanism. Sustainability Greater government funding of the social service sector is needed. P4EC and stakeholders identify the funding gap for social services created by the national de￾centralization process as the biggest risk to maintaining momentum in project activities. Local authorities are struggling to fund legal responsibilities they now have in social assistance and family protection through their own resources. The Vice President of a EVALUATION | 54 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project “new” raion said, “It is most difficult to convince the district council to allocate funding for development of social services. Social assistance takes quite a lot of resources at the expense of other sectors.” Adequate pay and staffing levels for the community social assistants employed by SAFPDs is the most pressing need. Stakeholders report frequent turnover and burnout of social assistants with their caseloads. Strongly related to this is the stipulation in PL 140 for mayors to hire a social assistant to support child protection and social services, working with multidisciplinary teams and in collaboration with community social assistants, when needed. This would relieve the community social assistants’ pressure on their case work but very few have yet been hired. The above information from key informant interviews is verified by focus groups of social assistants and mayors. The manager of social assistants in one of the “old” raion said: “Lack of funding and overwork for social assistants creates turnover in staff and limitations in quality of work. This jeopardizes long-term stability and sustainability in services. Our social assistants are employed at 0.25 or 0.50 percent of salary but still carry a large workload. One could say these people are doing volunteer work. Their requests for an increase in salary could not be fulfilled, although the regulation provides for at least one unit of assistance in each mayoralty.” The head of the SAFPD in the same raion commented that, “The project mechanisms are very good. But wages paid in the system are also important, as one cannot have very high expectations from a specialist who is remunerated with 1,500 MDL (82 USD) per month.” A few mayors reported that more social services should be under the mayor’s office. One mayor felt the community social assistants should be under the mayor’s office instead of the SAFPD at the raion level. “There is a problem with the social workers. Social workers should be employed within the mayor’s office. Currently, social workers are not working in the mayors’ office, so the mayors have to cooperate with them. If the social workers work under the mayor, this would help solve national level problems through meetings and visits in the field” (Mayor, “new” raion) Another mayor suggested that social aid should not come from the Social Assistance Directorate. Instead, the mayor recommended that all funds should be transferred to the mayors’ office, and the mayor’s office with the councils should give contributions to vulnerable families, because aside from the fact that these families receive some social benefits for the children, they also receive the social aid to help them increase their welfare. EVALUATION | 55 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project This input from mayors needs to be balanced by additional information. Mayors are required by law to hire a child rights protection specialist at the community level to work with caseworkers and specialists from the SAFDP. They have not been able to come up with the financial resources to do so – it would require a national funding mechanism to enable this. However, the SAFPD specialists bring a level of expertise that it is unlikely to be sustained if mayors take control over these positions. Additionally, there is greater accountability in a division of labor in child protection. Mayors are now being held to a greater level of responsibility through other service providers – this could conceivably diminish if this division of labor was reduced. Finally, benefits and social aid are not administered by communities to avoid possible discrimination/nepotism by mayors and due to the lack of professional capacities on the ground to assess eligibility. Material and professional support for community social assistants has improved. As seen below in Graph 4.1.2, the percent of respondents who “strongly agreed” or “agreed” that logistical and financial resources were available to them rose from baseline to endline. For example respondents reporting that office space is available rose from 47% at baseline to 71% at endline. Key informant interviews indicated the P4EC project supported improving space for specialists to conduct their work and improved their connection with beneficiaries through more confidential spaces for meetings and greater dignity in the interactions. Similar improvements were seen in the availability of family contingency funds, financial resources, and workforce training. However respondents reporting that transportation was available dropped from 35% at baseline to only 9% at endline. Despite this improvement in material and professional support, some community social assistants and social work specialists spoke of continuing material needs such as Xerox machines, meeting spaces, and transportation. Graph 4.1.2: Availability of Logistical/Financial Resources (%) Source: NORC evaluation survey 21 53 20 7 22 61 15 2 0 10 20 30 40 50 60 70 Strongly agree Somewhat agree Somewhat disagree Strongly disagree Baseline Endline EVALUATION | 56 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Graph 4.1.3: Availability of Professional Resources (%) Source: NORC evaluation survey Financing foster care up to levels needed for alternative care within the raion basket of support options is another important government funding obligation and continuing need. When fostering turns more strongly to caring for children with disabilities, there will be greater financial and human resource support needed per foster family. Some foster families and social workers reported foster families felt like they were taking on more work and needed greater financial support. Raion officials also identified the need for more and higher skilled psycho-pedagogical specialists to address the needs of students with special needs. Fortunately, the education sector in a number of project raions has been able find mechanisms to continue to invest in inclusive education, according to education officials. Some gatekeeping commissions also reported their limited ability to finance new programs and some multidisciplinary teams reported a need for greater compensation for their time. Additionally, both mayors and multidisciplinary teams felt unable to finance programs that support children outside of their home environments. Stakeholders and consultants are working with the Ministry of Finance to reform laws on decentralization to establish a statutory “minimum package of services” funding mechanism for local public authorities to be supported by the national budget. Among the priorities identified for support are foster care, family support through social protection cash assistance, and personal assistants for children with disabilities. During the project period, P4EC provided important input on the need for this reform based on the experiences of working with local service providers, and was an advocate for this reform. 42 55 3 0 87 13 0 0 0 10 20 30 40 50 60 70 80 90 100 Strongly agree Somewhat agree Somewhat disagree Strongly disagree Baseline Endline EVALUATION | 57 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Sustainability of Project-Supported Components Prior to initiating a collaboration with a government stakeholder entity, P4EC enters into partnership agreements outlining commitments for sustaining project activities. Stakeholders in the evaluation were asked about prospects for continuation of services and new systems for at risk children in the post-project period. Sustainability findings are assessed below in financial resources generated locally to continue the investments made through the project; national government resources that can cover gaps identified in the evaluation; infrastructure and training investments by the project that carry over into the future and child welfare system changes that are here to stay. Though funding and service gaps still exist, the prospects for mechanisms supported by the project to move forward into the future are high. The project built on the normative framework of Moldova. Many of the necessary key laws and official strategies are in place. The role of the project was to capacitate them through models of best practice in the best interest of children and families through systems and procedures for stakeholders that will be sustained. There is strong evidence that this was occurring in the raions of the project at endline. Gatekeeping commissions were successful in deinstitutionalizing children and better ensuring highly vulnerable children are supported through family-based care. The diversity of sectors were working more effectively together through multidisciplinary teams involving health, education, legal and social assistance specialists. All SAFPDs in the evaluation reported they were experiencing much better collaboration with government stakeholders in other sectors and, as a result, they could spend more time on complex high risk cases. Mayors are now more “responsibilized” and are demonstrating greater involvement and accountability in child protection activities. This was echoed in a number of focus groups with social service providers. The new case management tool was deployed and is being used in all the project raions. Stakeholders reported that all of these project priorities and activities are having positive impacts on the well-being of children and families, which is supported by the evaluation survey findings. Though funding and service gaps still exist, the prospects for these mechanisms to move forward into the future are high. A Ministry official gave this forecast, “The momentum on deinstitutionalization is here to stay as in family-type social services, though funding is needed with political willpower. Absolutely essential is the NPM advancing early detection and intervention, moving this out of the piloting to national application.” The head of the education directorate of a “new” raion praised the clarity brought about from project procedures and training saying, “All that is needed to make this mechanism EVALUATION | 58 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project functional is for the specialist to do his/her job. This does not put any additional task on him/her. This mechanism eliminates indifference and superficiality.” As stakeholders are convinced of the utility of project-supported initiatives, greater effort is being made at the local level to more effectively utilize local financial resources. P4EC assisted Causeni (one of the “new” raions) in assessing social services in the raion. The need for more foster families was identified. To fund the expansion from existing resources, the assessment determined care for the elderly could be improved by strengthening home care versus the existing institutional care model. The growth in foster care, including allocating greater support to allowances and maintenance costs, was funded through these savings. In at least three other raions evaluated, a greater number of foster families were being supported by transitioning from CCTF to APP-style foster care. The VP of Causeni raion acknowledged that their budget was limited for social services. Because the training and case management systems developed through the project were having good results for children, the raion administrative leaders were confident the Council would continue to fund activities started through the project. In Ungheni (one of the “old” raion), the District Council has allocated new funding out of its budgets to continue investments in the PANDA, Family Support and Mellow Parenting programs. For PANDA, the first half of 2017 was financed by P4EC and the second half by the Council. In one of the “new” raions, a shortage in foster care also existed. According to the SAFPD head, the District Council came to understand the importance of this family￾based care for children and allocated the funding to it. The family support manager in this same raion reported that PANDA and Mellow Parenting financial components initially financed by P4EC would be taken over through council funding. Specialists from various sectors in the raion successfully advocated for this continuing support. The head of the Education Directorate of another “new” raion said, “During the project we learned how important the programs are for children. I am convinced we will find financial solutions to continue the programs, but I admit we will need to have some additional outside support.” Project-funded infrastructure investments are helping to sustain project priorities. The project invested in upgrading workspaces for support and protection specialists to do their work. For example, in one of the “new” raions, the workspace was not adequate for confidential intakes, meeting space for stakeholders, and training. The refurbishing built capacity and commitment to project priorities well into the future. EVALUATION | 59 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project There is a high level of commitment to continuing project initiatives and project-trained individuals will likely continue to help build capacity. Turnover of staff who have been trained, mentored, and are committed to project activities can be a potential threat after intensive external support. The comprehensive project training, vertically and horizontally, and across sectors, helps to mitigate this risk. The head of the Education Directorate in one of the “new” raion said that there was a consolidated network of specialists in the raion and the director was confident project initiatives would continue even when/if the director retires. In fact, there was an agreement the Education Directorate will continue the functioning of the system developed through P4EC support at the end of the project: “ABCs will continue activities even if the external support is over. Its functioning depends on the District Council and, given they are satisfied, I am confident ABCs will continue to be supported. I certainly will help to facilitate this great activity no matter what.” (Deputy Director of Education, “new” raion) NGOs have been instrumental in training and building capacity across government child protection sectors over the last 20 years and the government will continue to be reliant on NGO support into the foreseeable future. The Minister of MoLSPF says it is unrealistic to think this level of training investment can be sustained. To fill this potential gap in the social work field, the Ministry is facilitating the development of a National Agency for Social Assistance, with consulting help from Oxford Policy Management. One purpose of the Agency is to build training capacity within government structures for less reliance on external funding. This reservoir of training talent can be tapped from experts engaged through project with its strong trainer of trainer component. Many of the workshops, mentoring activities, and other training sessions are conducted by experts from the stakeholder pool within project raions. This peer-led training provides excellent context since the trainers have faced many of the same challenges and opportunities as those being trained. Stakeholders interviewed include specialists who were either trainers of trainers or had received training from Moldovans involved in project activities. The Vice Minister of MoLSPF says about sustainability, “Money doesn’t solve all the problems, there is a strong need for continuing to build capacity for such things as parenting education and other types of counseling, including working with young mothers.” Conclusions: The P4EC project has trained scores of mayors and members of multidisciplinary team members as documented in Evaluation Question 4. The training of these and others across sectors has contributed to levels of intersectoral collaboration and burden sharing required for more effective child protection – including preventing the separation of children from families and supporting vulnerable families with at-risk children. EVALUATION | 60 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Turnover in positions such as mayors, members of multidisciplinary teams, and social assistants require on-going training mechanisms to maintain the momentum generated by the project. All stakeholders interviewed in this evaluation report that gatekeeping commissions, mayors and multidisciplinary teams, raional officials and SAFPD specialists are far more active and becoming more effective in results for children and families because of intersectoral training, tools, and methodologies promoted through the P4EC project. The health sector and involvement of mayoralities in intersectoral approaches consistent with project objectives have strengthened over the project period, however, stakeholders report greater involvement is needed along with continued capacity￾building and training in these sectors. Government stakeholders to the project report Advisory Boards of Children are a welcome addition to better understanding the needs of at risk children and helping in responding to their needs. ABCs are an innovative means for youth to become involved and advocate for children’s rights and needs in protection. Social service workers have a continuing need for their own psycho-social support given the challenges posed by their casework. This requires effective supervision, counseling mechanisms, and greater funding to align caseloads with human resources needed to provide support. The NPM shows promise in early identification of risk and vulnerability in families. The results in piloting of this model need to be clearly and broadly discussed among all stakeholders and NGOs. The model should be supported in its replication and expansion across Moldova. This will strengthen a collaborative multi-sectoral environment for prevention and building resilience in families and communities of Moldova. There is evidence of sustainability of project results beyond the project period. The biggest risk to achievements of the capacities strengthened through the project is the lack of a funding mechanism to better support the human resources required by social assistants at the SAFPD and mayorality levels, continuing expansion of foster care as an alternative to institutionalization, and other resources targeted at children with disabilities still in institutions and struggling in local communities. A large reservoir of trainers exist from the project that can continue to be facilitated with national government support to provide on-going training, not just in project areas, but across Moldova in under-capacitized regions. EVALUATION | 61 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Recommendations: The psycho-social heath and protection of caseworkers is an important component to their being effective and achieving results with beneficiaries. The evaluation did not learn about how these issues are dealt with and the PSS available and provided to address these issues – though this does not mean such mechanisms are not part of the project. PSS systems need to be in place for caseworkers. This includes having reasonable caseloads and supervisory support for difficult cases, burden sharing with other stakeholders, and counseling support when needed to address issues of “secondary trauma” encountered by front line caseworkers. Question 2: Did the program bring out systemic changes at the community, regional, and national levels that are enabling children to live in family care and preventing inappropriate placements in institutional care? Findings: DOCF/USAID’s support of the P4EC project represents a shift to initiatives focusing on national systems of care from direct service projects. Two basic program approaches in the Strengthening Families for Children project for system change are “responsibilizing” key stakeholders as mandated through PL 140 which, in part, requires mayors to act decisively and be responsible when learning children are exposed to risk – and by strengthening the intersectoral collaboration underlying GD 270 through gatekeeping commissions and multidisciplinary teams. P4EC played a very active role in informing the development of these two frameworks and advocating for their passage. The project built capacity to effectively apply PL 140 and GD 270 through its various project approaches including intersectoral training, case management system development and guidance resources. “Responsibilizing” is a term used frequently by stakeholders in Moldova. The normative framework, particularly PL 140 and GD 270 are specific about the role of government sectors and institutions in project-related activities. “Responsibilizing” is the process of effectively using these laws and policies for greater involvement and accountability in government sectors. The project builds capacity with tools, mechanisms, processes, and systems across sectors to facilitate this. Key informant interviews with social service workers confirm health, education, mayoralities, and commissions are all being more responsible due to the laws and having the training and tools for involvement. The project provides support in strengthening the decision-making and engagement with Gatekeeping Commissions in preventing institutionalization of complex cases. Mayors are alerted that children and families may be at risk and appoint multidisciplinary teams to assess and respond to their needs. Multidisciplinary teams EVALUATION | 62 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project are composed of health, education and public order sectors. Additionally, a National Practice Model has been piloted in the project. It is strongly tied to both laws and is designed to provide early identification of risk to strengthen better care for children in families and prevent risky situations from escalating. All stakeholders interviewed in this evaluation report commissions, mayors, and multidisciplinary teams are far more engaged in prevention and child protection activities because of training, tools and methodologies promoted through the P4EC project. Gatekeeping Commissions The Gatekeeping Commissions are appointed by the raion councils. They are independent and diverse to prevent conflict of interest. Involvement is on a voluntary basis. P4EC reported that since commission members are volunteers, this sometimes leads to them not having enough time to participate. Calling the commission together comes through the raion officials and this is supported by social assistant managers and heads of other departments such as child protection. Commissioners are trained and supported by the project to guide the decision-making for deinstitutionalizing children. Their functions also include preventing separation and making decisions on support, such as financial assistance and placement into alternative care. The Director of Education in one of the “old” raion said, “Children from internats were experiencing deep problems because of their institutionalization. At that moment, we realized how important it was to apply the gatekeeping mechanism for support and decision-making for at risk children at an earlier age.” The manager of social assistants in an “old” raion said, “Many cases are now resolved at the local level. For complex cases, the commission does not discuss unless the relevant mayor and the social assistant are present. In fact, the mayor must be knowledgeable and present the case.” Mayors and Multi-Disciplinary Teams PL 140 went into effect in the first quarter of 2014. It requires mayors to act decisively and be responsible when learning children are exposed to risk. They can be held accountable if they do not act. The child protection specialist in one of the “new” raions said, “The local administration previously did not know about PL 140, now they know they are subject to sanctions if they fail to fulfill their duties.” According to P4EC, a mayor has actually been jailed for lack of response. The manager of social assistants in one of the “old” raions said: EVALUATION | 63 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project “Previously, it was much more difficult to make specialists accountable and even to make the multidisciplinary team members work together. We cannot yet claim they are working now at their 100-percent capacity, but their level of accountability is now at 70 percent. The MDT [multidisciplinary team] convenes within 24 hours after the notification file is submitted to work on the case and propose actions with deadlines. They meet again and check whether all the proposed actions have been met. The mayor acts as the chairperson of the MDT [multidisciplinary team] and his job is to make sure each specialist fulfils his/her actions There have been cases when specialists had to intervene and take a child from the family at midnight and place him/her in the center until a solution was found. Cases are now solved much faster, since notifications are made quicker and the MDT [multidisciplinary team] started working better.” In another “old” raion, the child protection specialist commented, “Now, mayors’ involvement is much more appreciable. These new duties are natural for mayors’ involvement since they know about the situation of a family and child in a community. It is the mayor who decides upon the separation of the child from family.” The child protection specialist in one of the “new” raions pointed out: “With the new laws and training there is much more responsibility from the side of the Local Public Authority on guardian identification and case monitoring. The old regulations did not cover the aspects related to monitoring, reporting the child’s status, issues children were facing. The mayors are not delighted with the responsibility they have been assigned with. Previously political differences got in the way of decisions and support. The mayor may have been a member of one party and the social directorate head of another. Now there aren’t these barriers getting in the way of decisions and support. The right questions are being asked and collaboration is occurring.” Specialists in key informant interviews said that this new accountability framework has simulated the involvement of mayors in addressing the needs of at risk children and vulnerable families in their communities. The P4EC project has trained scores of mayors, and as members of multidisciplinary team the mayors swing into action. However, the turnover of mayors and team members will require on-going training mechanisms to maintain the momentum generated by the project. Although PL 140 stipulates each mayoralty shall hire a child protection specialist, only several have done so due to lack of funding. A child protection specialist in a mayoralty would take over some of the responsibilities currently undertaken by social assistants and other specialists in the SAFPDs, and relieve pressure on their case work. Social assistants at endline reported that they still faced challenges in juggling their many EVALUATION | 64 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project responsibilities with vulnerable populations ranging from children to the elderly and individuals with disabilities. However, they feel the capacitating of gatekeeping commissions, mayoralties, and multidisciplinary teams has shifted the burden more broadly across the service spectrum. All stakeholders interviewed in the evaluation indicated guidelines produced in the P4EC project were useful and influential. Guidelines exist on foster care, gatekeeping, supervision, case management and family support. Since they are officially approved, they are influential. The guidelines offer concrete and practical examples and detailed descriptions, and help specialists to deliver trainings and find answers to their questions. a) Do professional and public attitudes show increased knowledge of and increased support of national policies that prevent unnecessary family-child separation: and promote appropriate family care for children without parental care? Communication Strategy of P4EC P4EC tracks public opinion through contracted surveys on issues related to program priorities and themes. At the time of the evaluation field work, data was available from the period 2014 – 2016. In the 2016 survey, about two thirds (65%) of the respondents totally agree with the statement, “the family is the most important for Moldovan citizens.” This represents an 11% increase over the 2014 measurement. The sentiment is especially strong from populations in larger urban areas of Moldova and those with secondary education. For the statement “families in the Republic of Moldova are a safe and protective environment for children,” there is similar growth – but these views are more evenly distributed between socio-demographic categories. The most significant change in the survey is the perception of parents having the knowledge level necessary to raise their children: 50% agreed with this statement in 2014, compared with 62% in 2015 and 69% in 2016 – a 38% increase over this period. The growth is primarily recorded from populations in small and medium towns. Lack of support from the government continues to be perceived by the public as the reason for children placed in residential care – 79% of respondents believe this. Poverty-related reasons are cited by 64% in the survey. An increasing number of respondents believe it is the government’ fault children are placed in residential institutions because there is not sufficient financial support to families. Agreement with this sentiment has moved from 56% in 2014 to 68% in 2016. These views are predominant in metropolitan areas, large cities, and among people of other ethnic EVALUATION | 65 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project backgrounds.15 The P4EC project has a communication strategy component carried out by two specialists. Activities in this work includes producing coordinating spots on radio and television; development and distribution of information in newsletters, through social media, and use of posters and brochures; orienting journalists to issues through exposure trips and recruiting/informing them on stories; and providing training of raion officials in communication – including developing district-wide communication strategies. Communication and Public Attitudes at the District Level SAFPD leadership in Ungheni describe how their communication strategy developed with P4EC assistance targets local journalists. They say this has improved journalists’ sensitivities to issues of vulnerable children and families, particularly how to report about children, minimize stigma, and better ensure protection. According to Ungheni social work specialists, the strategy is applied both externally and internally within their offices. Staff are functioning together more effectively and are more acutely aware of the importance of public perceptions about the problems of children. They see that all have a responsibility to raise public awareness of SAFPD activities and it gives staff greater pride and confidence in their work. In one of the “new” raions, the District Vice President stated, “People are no longer indifferent. Notifications about at-risk children are now also coming from doctors, police, teachers, and even ordinary people. Specialists became more careful with early warning signs“ According to the head of the Education Directorate in one of the “old” raions, the perception of teachers and parents four years ago was that deinstitutionalized children would be “lost” among other children and not get the attention they require. However, the director attributes the educational inclusion process as changing parental attitudes on children who previously may have been institutionalized and their ability to function in society and educated through regular schools. The director points out that even if not all teachers are 100% ready now to work with these children, people see the results. For example, four students with disabilities recently graduated from a school with an adapted curriculum used for two years. 15 The evaluation team did not conduct a review of the quality survey and statistics are reported here only as an example of the communications work completed by the project. After the data collection period, the evaluation team was informed that the last round of this survey, in 2017, found some backsliding in some of these trends although it is unclear whether these changes were statistically significant. There was a slight decrease in the percentage of respondents that believed “the family is the most important for Moldovan citizens” and in those who believed they had the knowledge level necessary to raise their children. EVALUATION | 66 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Professional Attitudes in Support of National Policies Many challenges exist to change attitudes in professional sectors. Much of this depends on experiential and more in-depth exposure to issues. A key informant interview with the director of social work in a major Moldovan university revealed the faculty only recently found out about challenges children face in institutions. This was by accident through a student internship. The Vice Minister of MoLFSP, in referring to the psycho-pedagogy teaching staff at another university, described how surprised she was of their mind-set. They had to be convinced of the importance of deinstitutionalization for children, let alone the basic methodologies needed to do this. An indicator of changed attitudes on and commitment to national policies is confirming and integrating these policies in local regulations and strategies. The evaluation found numerous instances of this in raions of the project. For example, in one of the “old” raions, a strategy on inclusive education exists for the period 2014-2018, developed in collaboration with P4EC. It is intersectoral in approach and thus in line with the national strategy and more sharply defined to the local district context. In one of the “new” raions, the district officials developed and began implementing a Child Protection Strategy in 2015. A major provision of it is APP foster care development. Another indicator of changed attitudes is the sense of urgency and commitment stakeholders have in responding to situations of vulnerability and risk, particularly in protecting children and supporting them in more nurturing and safe environments. This commitment is being expressed by SAFPD specialists in P4EC project raions. An example is the reference to cases of involving serious violence and abuse by the child protection head in one of the “old” raions: “Minimizing risk to vulnerable children often requires decisive procedures to protect children. The new regulatory provisions enable this. For example, the prosecutor’s office must be notified within 24 hours when a child has been removed from his/her family and, within 72 hours, the action must be put in front of the court.” An indicator of changed attitudes related to intersectoral collaboration (GD 270) is the evidence of greater involvement of multidisciplinary teams at the de-centralized level – particularly lifting the burden of social assistants who, at times, have been overwhelmed their their case loads. In all the raions, social assistant key informants report that there is now significantly greater collaboration as a result of the project. In one of the “old” raions, the SAFPD head remarked: “Social assistants now only get involved when there is a more complex case and the other specialists do not know how to deal with it. Education is good example. For a child who is not attending school, the teacher submits a notification to the mayor, not just social assistance. Before, we used to ask the question, “Why only me?” Now we EVALUATION | 67 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project have a team to work on a case. There is much more active involvement of the multidisciplinary team specialists and fewer requests are submitted to mayoralties requiring social assistants.” The emerging NPM, previously described and discussed in Evaluation Question 1, is further evidence of these changed attitudes. SAFPD leadership in all raions indicate through key informant interviews the weakest link in the multidisciplinary team approach is the health sector. This is due to the many demands of time on doctors and nurses, not necessarily their lack of commitment. Due to this SAFPD staff report sometimes health workers arrive only at the most important meetings and when there is a more urgent situation requiring their direct expertise. In other circumstances, social assistants explain it is reasonable health workers communicate or provide advice through email or over the phone, given the time and scheduling challenges they face. Living with the biological family still considered the best option for children. The project had attempted to shift the population’s attitude towards residential care through their communication campaigns. Similar to the baseline, a large majority of focus group participants (including children, community social assistants, foster parents, multidisciplinary teams, gatekeeping commission members, mayors, and parents at risk of separation) felt that while the best solution is for children to stay in their biological families, and when this is not a possibility, it is best to try to find a good placement among relatives. Focus group participants emphasized that guardianship allowed children to be in a familiar environment and reduce the likelihood that the child would engage in criminal or deviant behavior: “Guardianship is one of the most appropriate services, since the child is placed with the extended family. The foster parent is a stranger, but under guardianship, the child is placed within the family and maintains the connection with the family. The child is kept in the family.” (Community social assistant, “old” raion) “[Children] get a psychological trauma when you take them away from the family and place them somewhere else. They could clam up and stop communicating and they can also become more aggressive. It’s different for every child. Rather than separating them from the environment to which they have been used, it is better if they stay in the family, because this keeps them among relatives.” (Multidisciplinary team member, “old” raion) Foster care was also frequently mentioned as a positive alternative placement situation by various focus groups of children, social work specialists and community social EVALUATION | 68 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project assistants, foster parents, the gatekeeping commission, and multidisciplinary team members. Foster parents were viewed as well-trained professionals who could provide a loving environment for children, particularly in the case of younger children who could more easily adapt to a new situation: “They also feel very well under (foster) parental assistance. Our foster parents are good people, and trained. They don't accept everybody as foster parents as far as I know, and I worked a long time. And we treat them like our own children. We don’t act differently.” (Foster parent, “old” raion) “Do not agree [that being at home is best]. If the mother does not take care of the child. The mother just goes where she wants, and she does not take care of the child, and the child does not feel well with the mother. If the child would be integrated in a mixed family, meaning in foster care, he will feel better there. He will see many new things. He would feel another attitude.” (Girl, “old” raion) “They continue living in that family. Although they are supposed to live there just until the age of 16, they reach majority and they still live with that family. Neither the parents want to separate from them nor vice-versa.” (Gatekeeping commission member, “old” raion) There is a perception that child protection laws, action plans, and national policies are now stronger and better implemented. The project was designed to build capacity to help responsibilize stakeholders, consistent with the current laws, action plans, and policies. Many different types of stakeholders perceived that there had been a significant change in the government’s emphasis on child protection, leading to more timely interventions on behalf of social workers, a focus on reintegrating the child back into the family, improved reporting, and better intersectoral communication. Participants often attributed this improvement to changes in legislation over the past few years. “Generally speaking, in 2007, 2009-2010, not so big attention used to be paid to family and child protection. In the recent years, some new laws appeared, new methodologies for child protection, and actions taken at local level, where specialists get involved, we get involved. Before, it was not like this.” (Multidisciplinary team, “old” raions) “The strong points are that over the last years, the legislation changed a lot, adjusting it to what we have in the whole world and more services were created, which may be used and children have access to all of them. For instance, the children with disabilities, who never got out of the house, there is the ‘Respiro’ service, and parents may take these children during one year for 30 days.” (Gatekeeping commission, “new” raions) EVALUATION | 69 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Several focus groups emphasized that recent laws and initiatives were effective in their focus on integrating the child into a family rather than an institution. As a result, participants noted that the number of institutionalized children had decreased. Members of the national-level ABC, gatekeeping commission members, social work specialists, and community social assistant focus groups mentioned PL 140 as particularly beneficial, while a gatekeeping commission focus group also highlighted GD 270. The following quotes are from a male and two different females respectively, from gatekeeping commissions in newer raions at the endline: Male 1: “The focus is on the beneficiary, meaning on the child. They have a lot of rights. The focus is not on the one who is providing the service.” Female 1: “The children have a better condition.” Female 4: “And they are more integrated in the society, because they stay in a family, where there are fewer children, and they are provided more attention.” (Gatekeeping commission members, “new” raions) This finding is supported by survey results that show how social worker confidence in the child care and protection system has increased from baseline to endline. Community social assistants and social work specialists were asked in the survey if they agreed or disagreed that existing national policies and legislation were being adequately implemented to support the functioning of the child protection system. In the baseline evaluation survey, 35% of all respondents said that they “strongly agreed” and 57% said they “somewhat agreed.” These numbers rose slightly at endline to 38% and 60%, respectively. The majority of respondents at both baseline and endline agreed that the child care and protection system functioned as well as it was supposed to. One important difference is that the number of respondents who “strongly agreed” rose from 28% at baseline to 44% at endline, showing that social workers now have more confidence in the system. Social workers and specialists in key informant interviews said they believe PL 140 and GD 270 are essential pieces of law. However, there was frustration over the inability of the government to adequately fund the social service system, particularly having a sufficient level of caseworkers or foster families to meet the need. At baseline, the top three areas of child care and child protection that needed improvement were identified as service development and implementation (52%); resource allocation at the raion and local level and an adequate workforce for child protection (each at 49%); and resource allocation at the national level (46%). Additionally, key informants interviewed felt there was a need for further alignment of the laws, for example the Family Code with PL 140. At endline the top three areas of child care and child protection that need improvement were identified as national-level policy and/or strategy (22%); resource allocation at the raion level (28%); and an EVALUATION | 70 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project adequate workforce for child protection (26%). While there was some change in the most reported specific areas of child care and child protection that need improvement from baseline to endline, it is important to note that there was less agreement among respondents on priority areas for improvement, signaling that overall things are better. Child protection campaigns have led to increased interest in foster care services. P4EC provided technical assistance in the development of these campaigns through the communication strategy project component. Both focus groups conducted with community social assistants recalled video ads, brochures, and messages on foster care and guardianship. Community social assistants named the child protection campaign “Take care of me. I want to live” and the “When you leave abroad, inform the guardianship authority” campaign messages. Some social worker specialists and community social assistants in focus groups noted that these campaigns had led to an increased interest in foster care services, and some multidisciplinary team members and social worker specialists felt they led to a greater, clearer sense of how individuals are legally accountable to maintaining child well-being. For example, one participant mentioned that in the Orhei district, there were campaigns between 2004 and 2008 to increase the number of foster care parents. However, this had not necessarily led to an increase in the number of foster parents, and social workers highlighted that they would prefer to reintegrate the child as quickly as possible either with extended family or through adoption, rather than keeping children in foster care: “That’s why we have 40 foster care parents over the last 3 years, so the figure does not change, because children come and leave, and if they get integrated, this year we have 15 children integrated and other 11 have come. Meaning that children come and leave, we work with them.” (Social worker specialist) “Our main aim is to see how we can shorten the time of stay in the extended family. This is a period of transition. We have given the child to foster parents and forgot about it. No, on the contrary, if we start working, the second stage of the work starts so as to integrate the child in the biological family, and if this is not possible, to integrate the child in an extended family, and if this is not possible, the child gets the status and may be proposed for adoption.” (Social worker specialist) TV campaigns that were part of the project were viewed as both effective by some and ineffective by others. Some foster parents said that TV campaigns do not work because there is no time to watch it, but some community social assistants and multidisciplinary team workers say that TV works well to spread awareness. EVALUATION | 71 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project There is some need for greater awareness of other laws and national policies. Despite the perceived success of laws and policies by most stakeholders, there is a lack of awareness amongst beneficiaries on other issues. Some foster parents admitted that they are unsure about the reasoning and decisions behind the close of residential facilities and they are unsure if ABCs still exist. Some parents from vulnerable families say that parents like them are unclear and uninformed about the validity and enforcement of laws. b) Is there an improvement in caregivers’ parenting skills and practices? Social worker confidence in caregiver skills do not show any change. According to a little over half of baseline respondents to the evaluation survey, “most” of the vulnerable parents and parents of reintegrated children had the necessary skills to look after their child. This remained true at endline. The skills include: discipline, communication, knowledge about child development, knowledge about child health issues, and awareness of positive and negative patterns of behavior within a family. Parent support groups were seen as more effective at increasing parenting skills and knowledge by project end. While the confidence in caregiver skills remained consistent from baseline to endline, the views on what methods are most effective for increasing parenting skills and knowledge changed. At endline, parent support groups were cited as the most effective methods, with 50% of respondents saying they were “very effective,” an increase of 11 percentage points from baseline. Baseline respondents believed that individual/one-on-one parenting education with social workers was the most effective method. This number dropped by 5 percentage points at endline. Table 4.2.1: Effectiveness of Methods in Increasing Parenting Skills and Knowledge Q84: How effective are each of the following methods for increasing parenting skills and knowledge for the families with whom you work? Baseline, % Endline, % Change Very Effective Individual/one-on-one parenting education with a social worker 51 46 -5 Parenting classes/workshops 41 45 4 Parent support groups 39 50 11 Books, magazines, and other written resources 27 33 6 Source: NORC evaluation survey c) Is there any change in attitude towards residential care among parents, extended family and community members? There is no perceived change in the perception of social service workers on the attitudes of parents and the community towards residential care. The project attempted to shift the population’s attitude about residential care through its communication EVALUATION | 72 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project strategy. In Table 4.2.2 below, survey respondents share their perception on the extent to which residential care was considered acceptable by parents and by the community. Social workers’ perceptions of family and community acceptance and support for residential care remained largely the same from baseline to endline. At endline, there remain some parents and people in the community who believe residential care is acceptable for their children and a good service, at least for some children. As for the degree of support the community has for deinstitutionalization, respondents were more divided, with 13% saying “all”, 44% saying “most” and 38% saying “some” at endline. Table 4.2.2: Acceptance and Support for Residential Care Acceptance and support for residential care All Most Some None Baseline, % To what extent do parents believe that residential care is acceptable for their children? 3 25 56 13 To what extent does the community believe residential care is a good service to care for children? 4 28 61 5 To what degree is the community supportive of deinstitutionalizing children? 10 43 43 4 Endline, % To what extent do parents believe that residential care is acceptable for their children? 2 29 55 13 To what extent does the community believe residential care is a good service to care for children? 5 34 57 5 To what degree is the community supportive of deinstitutionalizing children? 13 44 38 5 Source: NORC Evaluation Survey There is some change in social workers’ perceptions regarding when a child should be placed in an RCC. While there is still widespread acceptance of residential care facilities at endline, especially when a child is an orphan, the overall trend was toward less acceptance of placement in residential care regardless of circumstances. The one exception was “when the child is failing school.” Also of note is that the percent of respondents who believe there is “never” any circumstance under which a child should be placed in an RCC dropped from 17% at baseline to 2% at endline. EVALUATION | 73 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Graph 4.2.1: When a Child Should Be Placed in a Residential Care Facility (% "Yes") Source: NORC evaluation survey Similar to baseline, placement in an RCC is viewed as acceptable at endline in certain situations. While community social assistants and social work specialists surveyed clearly view a child being orphaned as the most acceptable reason for placement in an RCC, multidisciplinary team members, gatekeeping commission members, mayors, children, and parents did not cite this as an acceptable reason in endline focus groups. Instead, when asked which children should be placed in RCCs, some from these populations mentioned children with severe disabilities, those with extremely violent or erratic behaviors, and children who must be removed from their families right away and have no other options available: “If we talk about the children with disabilities, who need medical assistance, for instance the child is fed through a special device, although there is personal assistance now and many persons, if it is recommended may take the child. In these cases, maybe in a residential institution to take is better.” (Social work specialist) Participants in FGDs mentioned that RCCs could provide specialized services that might not be available in other placements, including the opportunity to have regular access to a psychologist: “If placed in the center, you have to have a psychologist there, and also a pedagogue and persons to teach them ‘look, you have to take a shower, you have to take care of your clothes, you have to do this and that...’ they have to be taught, otherwise it is not possible.” (Multidisciplinary team member, “new” raions) 17 15 6 16 25 20 61 18 2 10 12 9 10 8 47 14 0 10 20 30 40 50 60 70 Never When the family is poor When the child is failing school In cases of juvenile delinquency When the community has no services to offer When the child has a disability When the child is an orphan When the child is left behind by migrant parents Baseline Endline EVALUATION | 74 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project On the other hand, a community social assistant spoke about one case where putting a disabled child into an auxiliary school was not best for the child’s happiness: “In my locality, there is a boy with the Down syndrome and his parents, because he is the first child, took him to centers with kindergarten, where they took him on Monday and brought him home on Friday. And then they tried the regular kindergarten and I understood that they were supposed to take him, and I don’t know why, but they did not. And when it came to school, the parents were forced to take him to auxiliary school, but it is not right. I went to the commission meeting and I saw the child holding his mother’s hand very tightly. I don’t know how to explain for you to understand. And now that he is back in the family, it’s very good. Even though he does not do a lot of learning, at least, he is in his family and everything is well. But there are cases when you take them back to the family, but they wander around… From case to case.” (Community social assistant, “old” raion) RCCs continue to be viewed as negative placement options for children who could be placed in family-based alternative care. Similar to the baseline, at endline RCCs were the most likely to be mentioned as a negative placement option, particularly by children and those from the child protection system. Some children and community social assistants felt that children in RCCs do not receive enough attention or warmth. Many children reported that children in RCCs might be physically or emotionally abused by other children or by center staff, such as being made to “stand in a corner,” “forced to kneel on corn seeds for 2 hours,” or “asked to do 100 push-ups.” Children expressed that these conditions could make children run away from RCCs. Some positive opinions of RCCs persist, mainly among children. While RCCs were viewed as negative placement options by some, others noted that RCCs treat children well and are an improvement to their home situation. Some children and gatekeeping commission members in focus group discussions noted that children who remain in RCCs are “integrated and treated with respect,” have “many friends,” and feel like the RCC is a “real home.” Some participants expressed that children may choose to stay in RCCs rather than face severe poverty or abusive parents at home: Mod: “Are there situations when children decide that it is better for them in the orphanage?” Male 9: “Yes.” Mod: “What situations?” Male 9: “Most [situations].” Mod: “Most? That is better for them in the orphanage?” Male 9: “Yes.” EVALUATION | 75 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Mod: “Why?” Male 4: “Because they have nothing to eat and nowhere to live.” Male 5: “Or their parents drink alcohol. And it is better to stay in the orphanage than to be beaten.” (Boys, “new” raions) By project end, there was a shift in perception of children in RCCs – some participants now hold positive views. When asked about children living in RCCs, focus group participants expressed both positive and negative stereotypes. Some children perceived that those in RCCs consume alcohol or drugs, while more believed them to be badly behaved, not obedient, and suffering from more financial or emotional issues than other children. Some foster parents and social work specialists said that RCC children in their care were not well socialized and did not have “life skills.” While similar negative stereotypes about children from RCCs were found both at baseline and endline, participants at endline also expressed positive views on children from RCCs that were not widely shared at baseline. Half of the children’s focus groups described those in RCCs as hard-working, particularly in regard to their schoolwork. Children from RCCs were also described by children and foster parents as having adequate behavior and being “brave.” One foster parent said that children from RCCs “knew how to behave, how to wash, how to arrange their things” because they had been taught good behaviors in RCCs. Further, at endline, some focus groups with children noted that they did not necessarily agree with the negative perceptions they had heard about children from RCCs. This is a significant change from the baseline study, where the views expressed about children from RCCs were almost entirely negative. Female 3: “I disagree when they speak negatively about [children from RCCs], because there are good children too.” Female 6: “Because sometimes they say a child is very bad, he is disobedient to his parents; however, some children are very obedient and they just don’t know about it and continue speaking negatively about him.” (Girls, “new” raion) Conclusions: ■ There has been a moderate shift in attitudes regarding placement of children in RCCs and a greater sentiment that this is not appropriate. Public opinion is trending toward a greater number of people believing the family is the best place to raise a child. Social workers and specialists believe it is less necessary, for example even when a child is disabled, there is a lack of community services, if the child is an orphan or for juvenile delinquents. EVALUATION | 76 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project ■ In discussions with multidisciplinary teams, parents, ABC at national level, community social assistants, and the gatekeeping commission at endline, there was a perception that the government now has a greater focus on child protection and that national policies and legislation are being better implemented. Social workers and specialist also believe the normative framework developed in recent years provides much greater opportunity for child protection and keeping children out of institutions. Stakeholders such as mayors are being forced to be more accountable. ■ National policies are beginning to shift the burden of child protection more broadly from the SAFPD to other sectors, including health, education, and municipalities. ■ National policies and laws are beginning to be reflected in strategies and policies developed at the sub-national level. ■ TV campaigns were viewed as effective by some and ineffective by others in generating increased interest in becoming a foster parent ■ There was a shift in the perception of children in RCCs at endline, with some holding positive views and less negative stereotypes. Some described children from RCCs as hard-working, particularly in regards to their schoolwork, as having adequate behavior and being “brave.” Question 3: Have reintegration methods employed by the projects resulted in stable and sustained placements for children? Findings: Educational inclusion is a key component in the deinstitutionalization process. P4EC has historically been at the forefront in the development of education inclusion in Moldova as part of the deinstitutionalization and reintegration program. This is explained (paraphrased) below in the text box by the head of the Education Directorate in one of the new raions of the project that underwent a deinstitutionalization process during the project period. EVALUATION | 77 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Box 1: Educational Inclusion and the Deinstitutionalization Process in a new Raion I had several internats under my subordination and I thought they were functioning well. I mostly relied on what I was being told but I went to them a couple times a year and the children were reciting poetry and singing. I was categorically resistant to change. I didn’t understand many aspects of the functioning of the centers. My shift in perception came when P4EC organized seminars and we learned more about the children. The deinstitutionalization process faced some barriers. One of the difficulties was it was an electoral year and not “convenient” for raion councilors to approve a decision to close residential care institutions. An independent commission was formed and the P4EC assisted the members to carry out an objective assessment of the children and institutions, with special attention to the psychological and social components of both the children and the integration of these children into their families. This assessment went deep and the councilors could really see and understand what the consequences were on children of institutional care. Before they finally approved the policy, there were discussions and planning on different topics including legal, financial, economic and social issues. I made sure the internat director was with me in all meetings with the raion council as she did not fully agree with deinstitutionalization. She had been in the position for a long time and was comfortable as head of the institution. As we (including the councilors) learned about the children and alternatives to institutional care, the director could not bring any effective arguments that the change shouldn’t take place. It was obvious. The assessment identified six children with mental retardation. The main challenge of the remaining 35 institutionalized children was “pedagogical neglect” – meaning the school failed in its mission. These children would not have ended up in this institution if the mainstream school system and teachers had done a proper job. The assessment showed that many children were from X and they were sometimes leaving the institution in the evening. The big question was, could these children successfully attend the mainstream school? The educational inclusion process was started in parallel with the deinstitutionalization process. P4EC got involved in the training of specialists and teachers. For each child assessed to have special educational needs, the deputy school director and teachers met to adapt curricula to the child’s capacities. The integration process addressed the individual characteristics of each child, rather than the more general approach of the other students. The process also stopped the risk and flow of children at risk of separation from going to residential institutions. In the past, children were easily placed in these institutions. But now, the mainstream schools are prepared to ensure the access to required services to these children. This occurred with the help of P4EC training in collaboration with the Psycho-Pedagogical Assistance Center. These two organizations were actively involved and complementary. Every six months or once a year, the center carries out an assessment of the situation of children enrolled in the mainstream educational institutions. Assisting in this process is the newly established Psycho-Pedagogical Assistance Service (SAP) at the district level. With the groundwork laid, the deinstitutionalization process took place relatively quickly in 2015, over approximately 9 months. The number of children referred to institutions also all but stopped as teachers realize that regardless of the situation, they are the ones who now must work with these children with their special needs. The head of the Education Directorate also went on to explain some of the existing challenges they are facing: “…Some teachers face difficulties, as they also must work with children with severe disabilities requiring much more attention and support. Educational institutions do not have yet the necessary conditions to do this effectively. What is left is the need to develop capacities to work with children with very severe disabilities, such as down syndrome, autism, or other very severe physical disabilities. Also, extremely difficult is working with children who have significant behavioral issues. The case load for SAP is huge and staffing is needed in this work.” EVALUATION | 78 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project In another “new” raion, the head of the Education Directorate said 12 out of 17 children who were bedridden in a residential institution were reintegrated in community schools where they get support and learning based on their specific needs. Fifteen reintegrated children with significant education needs are receiving special support to help facilitate their 9th-form graduation exams. In a third “new” raion, the SAFPD child protection specialist said that when the deinstitutionalization process started (before the start of the project) there were 250 children in residential care. Now there are no RCCs left in the district, aside from the center for the deaf and hard of hearing. According to the specialist, the most difficult placement was for children with mental disabilities, with the biggest issue being changing people’s mentality. Both society and the education sector were not ready to receive and accept these children. But all the joint training through the project changed this and showed deinstitutionalization could be successful. The head of the education directorate in one of the “old” raions in the project was used as an inclusive education national trainer by P4EC. After an inclusive education pilot led by P4EC, the process was launched across the raion about four years ago, which included establishing and training of resource centers in schools. The raion has a modified curriculum which is adapted to the needs of children, including individual plans. Two percent of the education budget is devoted to fund these centers and allocated based on the level of need of a particular school regarding children with disabilities. This also helps provide for the additional work involved with inclusive education. The director explained that an important input in developing inclusive education was the involvement of parents of children with disabilities who went to other countries to assess other systems and then shared their experiences and informed the development of the model. The deinstitutionalization process in this raion was completed in 2015-2016, with 70 children reintegrated into families from RCCs in the raion along with another 60 children from RCCs in other raions. One of the centers closed in the raion is now being transformed into a much-needed kindergarten. In another old raion, the head of education in a key informant interview said that all of the children deinstitutionalized from two RCCS are now attending mainstream schools. Children who are members of the National ABC also emphasized the importance of involving the school to make the transition to a new home and school easier for the child. A head of one of the education directorates in a “new” raion said that the P4EC project helped bridge the gap in the educational inclusion of deinstitutionalized children: “There had been many attempts to establish educational inclusion in the past, but without much success. A total of 17 resource centers in schools were already in place through state policies in the mainstream schools. P4EC got involved in the training of specialists and teachers. For each child assessed to have special EVALUATION | 79 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project educational needs, the deputy school director and teachers met to adapt curricula to the child’s capacities. The integration process addressed the individual characteristics of each child, rather than the more general approach of the other students…In the past, children were easily placed in these institutions. But now, the mainstream schools are prepared to ensure the access to required services to these children. This occurred with the help of P4EC training in collaboration with the Psycho-Pedagogical Assistance Center.” (Head of the Education Directorate in a “new” raion) Educational materials for parents and children was mentioned as beneficial in the reintegration process, including in-school education for the children about morals and values and TV shows to develop parental skills. Other support mechanisms mentioned by participants include employment for caregivers and material items, such as furniture and appliances. a) Is an increased number of children living in residential care facilities being placed in family care? P4EC has refined its basic deinstitutionalization model over time. In this project seven residential institutions were assessed and five were closed. The five closures were: 1. Sanatorium boarding-school for children with disorders of the cardiovascular system and of the joints from Ivancea village, Orhei district (closed); 2. Sanatorium boarding school for children with psycho-neurological diseases, from Ciniseuti village, Rezina district (closed); 3. Auxiliary boarding school for children with special educational needs, from Rezina district (closed). 4. Auxiliary boarding school for children with special educational needs, from Nisporeni district (closed); and 5. Auxiliary boarding school for children with special educational needs from Visoca village, Soroca district (closed). The centers that were not closed are:16 16 Likely a major factor was that USAID/DCOF raised significant questions about what kind of educational opportunities deaf and hard of hearing children need and whether these facilities should be closed. Institutions for deaf and hard of hearing children were included later in the project at the request of the Ministry of Education. The general aim of the intervention in these two institutions was to inform the design of a future education and care system for this group of children. Based on a complex assessment of children and their families, human and financial recourses, and mapping of services in the community of origins carried out by the project, the Ministry of Education devised - with project support - a strategic vision towards the development of the educational system for deaf and hard of hearing children and an interagency action plan. EVALUATION | 80 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project 1. Special boarding school for deaf and hard of hearing children from Cahul districts; and 2. Special boarding school for deaf and hard of hearing children from Hirbovat village, Calarasi district. The project deinstitutionalized all children in project-targeted RCCs in four of the six “new” districts: Soroca, Rezina, Orhei, and Nisporeni. The total number of children deinstitutionalized through project activities is 270, of which 166 were reintegrated in biological families, 59 placed in extended families, and 45 into family-based alternative care. b) During the tracing process, what percentage of children in residential care have been identified with a birth family or kinship care option that is safe and appropriate? All of the children in the RCCs targeted for deinstitutionalization by the project have been deinstitutionalized. As stated above, of the 270 children deinstitutionalized, 166 children were reintegrated in biological families, 59 were placed in extended families and 45 into family-based alternative care. Information about the well-being of deinstitutionalized children is provided in Evaluation Question 4. c) What factors prevented placement of children in residential care into permanent family care? There has been a substantial decrease in the number of children in RCCs. The number of children in RCCs has reduced from approximately 13,000 in 2007, when the national strategy for deinstitutionalization began, to around 2,000 today.17 About 25% of these 2,000 children only attend RCCs for educational and care services during the day and go home every night. Over this same time, family-based alternative care has grown from approximately 6,000 to 14,000. According to MoLSPF leaders, most of the remaining children in RCCs are in centers caring for children with significant disabilities and located mostly in the north of Moldova, where deinstitutionalization has not taken root due to lack of support from local government decision-makers. P4EC estimates there are about 700 children in this latter category in the North. The MoLSPF is finalizing a strategy for 2018-2024 to launch the next stage of deinstitutionalization for these remaining populations of children. Models that have proven effective in Moldova will be scaled to meet this need, such as the use of mobility 17 Sources: Interviews with MoLSPF and P4EC EVALUATION | 81 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project teams18 to reach communities and families caring for significantly disabled children, use of specialized day and respite care centers, and further development of educational inclusion. For children who are not able to be with their families, specialized therapeutic foster care is also an option for development. Funding mechanisms to drive this reform are important since specialized care for these children comes with significant costs that will be difficult for local governments to absorb. As centers caring for children with significant disabilities are phased out, at least some of this funding can be redirected from the high expenses of running the institutions. There are the additional challenges of changing attitudes and anxieties in the care of these children. For example, the Ministry of Health is concerned about deinstitutionalizing their TB center.19 An endline evaluation key informant interview with the Director of a special needs school summarized below, provides perspective on a special needs institution.20 Box 2: The School for Deaf and Hard of Hearing Children Two residential schools for deaf and hard of hearing children exist in Moldova, under the jurisdiction of the Ministry of Education (in Cahul and Calarasi). There is a third school in Chisinau under the jurisdiction of Chisinau Municipality that provides only day services. In the school in Cahul there were 42 residents in the most recent term and twelve graduated from the centers. There have been no new admissions into the first grade over the last three years and the resident population is steadily decreasing. There are also 42 staff members, including educators, cleaners, cooks and security personnel. Most graduates continue their education in a vocational school for the deaf in another raion. Others return home to live with their parents and with the household. All the children return to their biological families at the end of the school year and during extended holidays. If they live close by or can afford the transportation, they go home on weekends. The head of this raion’s Social Assistance and Family Protection Directorate (SAFPD) says there is considerable pressure from some parents of these students not to close the center. The center director believes children who can be assisted with implants can attend mainstream schools but, unlike other countries, schools are not ready with specialists to teach children who have completely lost their hearing. The director says: “In our special schools for the deaf the children are not neglected, dishonored or put in unpleasant situations, we are like a family. They can communicate with each other here at the school, while at home many parents do not know sign language.” There are no initiatives at the school to help parents learn sign language. The director does not think parents are interested in doing this nor in being involved in other initiatives to promote communication and contact with students. There are rooms at the center if parents want to come and stay, but this does not often occur. There is one psychologist at the school, otherwise the students receive psycho￾social support from teachers when they need it. At night a limited number of educators are supervising the dormitories. The director explains that if the school closes, the teachers and others may be able to find other employment. However, the children would be adversely impacted. 18 Keystone Moldova has been instrumental in developing this model, including in raions of P4EC’s Strengthening Families for Children project. 19 According to several sources, not directly discussed with MoH 20 At the time of the evaluation, DCOF/USAID was finalizing plans with P4EC on the best practices for children with hearing impairment to be pursued in the deinstitutionalization process. EVALUATION | 82 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Difficulties continue when placing children with disabilities.21 According to social service workers in focus groups, children with disabilities face particular difficulties that may prevent smooth and successful transitions from taking place. This is because such children are considered by some caregivers as too burdensome to raise given a lack of skills, money, or services to meet their needs. A mayor shared: “We had a sick girl. There were 4 sisters. One of them got married and adopted the two healthy ones. But X was not adopted because she had disabilities. We tried to place her with relatives from various villages. We tried to convince an aunt to take her. But then someone from the village said, “Do you really need this?” In the end, the aunt called and asked for X to be taken away. They did not have space in the house for her. Their biological children were jealous of X because the family took good care of her. I placed her in Glinjeni but was called and told to take her.” (Mayor, “old” raion) This is a particularly problematic issue for children with severe disabilities. A social work specialist went into detail about these challenges: “If we talk about the children with disabilities, who need medical assistance, for instance the child is fed through a special device, although there is personal assistance now and many persons, if it is recommended may take the child. In these cases, maybe in a residential institution to take is better. We are thinking about the children with hearing problems or children with eyesight problems, you have to find a good collaboration with the Educational Divisions for these children to have a possibility to go to school, because we have integrated such children and it is very good that we had nongovernmental organizations which helped them to go to school and the children can read and the class was prepared and the children can go to the center from Peresecina, but there are sometimes very complicated cases – one child per one thousand and we need highly specialized services for the child. Only in these cases.” (Social work specialist) Inclusive education is a significant factor in enabling deinstitutionalization of children with disabilities. About 1,400 institutionalized children in Moldova are those with disabilities. The table below shows the breakdown of children in residential care under control of the Ministry of Education and the Ministry of Health, Labor, and Social Protection (MHLSP). 21 Deinstitutionalizing children with severe disabilities was not a specific target group for the P4EC project. The Keystone Human services organization and Lumos were mostly involved in the deinstitutionalization of these particular group of children, especially from the Orhei residential boarding school for children with mental disabilities (boys) and Ialoveni boarding school for children with motor disabilities. EVALUATION | 83 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Table 4.3.1: Number of Children with Disabilities in Residential Care Children in RCC subordinated to the MoE, December 2017 Children in RCC subordinated to MHLSP, December 2017 Boarding schools (children without parental temporary care) Special boarding school (children with sensorial disabilities) Auxiliary boarding schools (children with special educational needs) Special boarding school for children with severe disabilities Baby homes (young children) Placed in RCC Without placement/ Receiving day care services Placed in RCC Without placement/ Receiving day care services Placed in RCC Without placement/ Receiving day care services 283 95 106 169 203 312 104 150 378 275 515 254 Source: P4EC project data However, a significant number of children formally diagnosed, or informally thought to have disabilities (rightly or wrongly), have been deinstitutionalized in Moldova and in raions of the P4EC project. The availability and application of inclusive education has been a significant factor enabling this. Children with disabilities are a significant group benefiting from specialized day centers of project raions. In one of the “newer” raions, the head of the SAFPD said: “Individual education plans have been developed for every disabled child. There are day-care centers for children with disabilities where they can benefit from speech therapy, kinetotherapy, psychological services and personal assistance. Each center employs psychologists although their number should be increased. It would be optimal if each school could employ a psychologist.” Mobile teams assisting rural families with disabilities developed through raion collaboration with Keystone Moldova is also an important support mechanism. At least three project raions – Calarasi, Orhei, and Causeni – have official strategies on inclusion of people with disabilities. In Causeni, this occurred over the project period. The strategy includes children who can benefit from home-based learning activities.22 The evaluation identified three project raions where children are learning through home￾based education: Singerei, Causeni, and Cahul. 23 22 For children with severe disabilities there is the option, regulated by law, for them not to go to school and instead have teachers come to their home and deliver the lessons under an adapted schedule and curriculum. This is valid until the age of 16. 23 According to P4EC, per national legislation, all children with severe disabilities are provided with home-based learning. EVALUATION | 84 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project There is a need for more non-residential alternative care options for children with disabilities. Children with disabilities benefit from foster care in project raions. In at least one raion, Ungheni, advocacy has led to a policy whereby financial support to biological families having children with disabilities can also be applied to foster families. However, key stakeholders interviewed in all the raions identified the need for greater alternative care options for children with disabilities. One member of a focus group with foster parents said that children who come from RCCs into foster care require a lot of attention, which can be difficult if they are taking care of other children in the same household. USAID/DCOF identifies therapeutic foster care as an important service needed for development in Moldova. This is particularly important as Moldova positions itself in the next phase of deinstitutionalization. This type of foster care is understandably more expensive than other foster care when factoring in the additional resources required in preparation and training of the families and greater material and financial assistance. Another resource available for families with disabled children is short-break foster care that consists of professional parental assistants, who help families manage and care for children with disabilities. However, one focus group with community social assistants in the “older” raions, noted that there are not enough of these qualified specialists available to support struggling families. A few parents in focus groups don’t support the closing of RCCs for children with disabilities and think that the lack of specialized care is worse for the children. One parent said they were not happy “because the children are out. They go and mix in schools with other children and they think otherwise and other have other thoughts, and there are problems with teachers.” (Parent at risk of separation, Cahul) d) What type of social service follow-up is provided to deinstitutionalized children, by whom and for how long? See evaluation Question 1 for social service follow-up procedures. The project stressed a multidisciplinary approach to placements. Institutional care for children presents significant issues for the well-being of children depending on the length of time they were institutionalized, the care experienced in centers, the circumstances leading to placement in the centers (such as care practices and trauma), and well-being deficits acquired by being institutionalized (such as attachment disorder). Best practice involves careful assessment of these factors to improve prospects for successful placement outside of the center. The risk of placement disruption can be high depending on a child’s individual circumstance and the level of support in the post￾institutional environment. The goal is to minimize risk of disruption while recognizing there are many unpredictable factors that can influence the stability of a placement. EVALUATION | 85 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Project training and the design of the deinstitutionalization process includes follow-up procedures to identify risks to the stability of the placement and adjust, if necessary, post-placement support provided by duty-bearers. The project has strengthened a comprehensive multidisciplinary approach, starting with the child and family assessment in the pre-integration period and the provision of supports needed to strengthen the placement. The basket of potential services and support is wide. This includes orientation and preparation of the foster carers, parenting education and counseling, in-kind and social protection and other financial support, links to the health and education sectors and specialized day centers or temporary shelters that children and caregivers can access. The cases remain open if needs are identified for follow-up. Examples of specialized centers exist in one of the newer raions, as explained by the family support manager in this raion. These specialized centers can be accessed by children if placements and alternative care are disrupted. One is for street children with multidisciplinary specialists including teachers. Another is for children and young people having challenges adjusting to society. The latter has a larger capacity and provides services such as social flats for young persons in difficulty, and persons over the age of 18 with housing needs and/or who want to add to their education. In one of the “new” raions, the Vice President of the district described another important set of challenging factors from the experience of deinstitutionalizing 77 children – all of whom continue to be monitored and supported. The raion has yet to come up with an effective vocational training service that was previously available in the centers that were closed. According to the legislation, mayoralties where the children are from are responsible for providing this support. However, they struggle to provide this. The VP says though many children were successfully reintegrated, some have attachment issues within families and may have preferred to stay in the institutions. A category of children the district finds difficult to work with are children with volatile behavioral issues. Of the 77 children deinstitutionalized, four were adopted and another four have accessed temporary centers. This raion had two disrupted cases of children from their families following deinstitutionalization. During the disruptions, the children were in and out of their parents’ homes and accessed foster care and temporary placement center support. Follow-up social services to families of reintegrated children have risen in all categories of assistance identified in the table below, with more social workers reporting they put emphasis on material assistance and counseling. The percent of community social assistants and social work specialists providing each type of follow-up service has increased from baseline to endline. Similar to baseline, the most common forms of EVALUATION | 86 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project social assistance are cash benefits and material assistance, with 99% of respondents reporting that they provide these services. The provision of each type of social service increased by between 9-11 percentage points, except employment/household income support and parent education, which increased by 20 and 19 percentage points, respectively. The survey data indicates the coverage of services for deinstitutionalized children is greater at endline than it was at baseline. Interviews with social workers also indicate the health and educational sectors are now more closely involved in service coordination with deinstitutionalized children and their families. Table 4.3.2: Services Provided to Deinstitutionalized Children Q77: In which of the following ways do you work with families in which a child has been placed? Baseline, % Endline, % Change Yes Yes Provide access to cash benefits 90 98 8 Provide other material assistance 90 99 9 Regular home visits, regular office visits 90 99 9 Visits to the child’s school or kindergarten 87 98 10 Represent the child at gatekeeping meetings 85 93 9 Arrange multidisciplinary meetings 84 95 11 Employment/household income support 67 87 20 Housing services 41 52 11 Referral to other services 82 90 9 Parent counseling 84 94 10 Parent education 68 87 19 Counseling for the child 87 96 10 Source: NORC Evaluation Survey Schools, parent and child education to prepare for deinstitutionalization, and activities for children placed in family care are helping to smooth the transition into home-based care. Preparing the family for the return of the child requires thoughtful planning. Some children and social service providers in focus groups stressed that counseling and discussion with the adoptive parents and foster care parents, in addition to that done with parents or relatives, is important prior to a child’s placement, including discussion between the family and the child about future expectations in order to avoid problems in their relationship. Employment, schooling, and involvement in activities were viewed as useful to help older children to reintegrate into society. Some social service providers and foster parents mentioned that children who are placed in families when they are older than 12 years old may not adjust as well. Some social service providers felt that older children needed to start working, and that social service providers could assist by helping them secure jobs and future schooling opportunities. EVALUATION | 87 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project “We help children who return from center that want to pursue additional education. One child wanted to be a nurse so I recommended her to go to Ungheni. We looked at all the certificates and letter of recommendations. We help if the child wants to study.” (Mayor, “new” raion) Participants named the following as stakeholders that could assist in the reintegration process: established social service institutions, local public administration, NGOs, schools, and neighbors. In particular, psychologists were named as valuable professionals who had expertise building rapport with children. Key informant interviews with local stakeholders indicated that all are involved in the reintegration process. Conclusions: The project played a strong role in building capacity in the following ways: ■ Inclusive educational systems and preparation and support to households receiving these children has proven to be critical to successful reintegration. ■ An intersectoral approach to reintegration of institutionalized children is occurring and essential to this process. ■ Reintegration of children with disabilities is occurring through project stakeholders. However, this remains a challenge nationally, particularly for children with more serious disabilities – including behavioral issues. Greater investments are needed for specialized multidisciplinary community-based strategies and services such as in specialized foster care, vocational training, day support centers, and professional parental assistants. ■ Follow-up social services to families in which a child has been placed have become more available from the baseline to endline period, including greater preparation of children and families, support in the post-placement period, and multidisciplinary assistance from health, education, and other sectors. The level of disruptions of deinstitutionalized children is low, with only five known cases. This low level of disruptions is most likely the result of the preparation and support in the placements provided by P4EC. See findings in Evaluation Question 4. EVALUATION | 88 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Question 4: Did the program measurably improve the safety, well-being, and development of highly vulnerable children, particularly those who are living without adequate family care? Did the program impact beyond direct services? Findings: a) Did the project provide a core package of services to help ensure that residential care is prevented when possible and that reunified and deinstitutionalized children and at-risk children remain in family care? The project provided substantial training and capacity building of project stakeholders to develop core packages of services. P4EC does not provide direct services to children and families, but rather provides support to government stakeholders who do. This is done through capacity building in system development and training. These services target decision-makers and direct service specialists in government sectors at raion and municipal levels who have responsibilities in preventative social services, deinstitutionalization, and alternative care. The stakeholders interviewed in the endline evaluation universally attribute significant and highly effective skill building and systems development to the technical assistance coordinated and provided directly by P4EC. Other key NGOs involved in child welfare acknowledged that they benefited as well, particularly through P4EC’s historic and continuous development and capacity building in foster care, multidisciplinary approaches, gatekeeping, and case management.24 The SAFPD director in an “old” raion, described P4EC’s support in this way: “A previous project left us with many unresolved issues, undefined structures, unexplained working tools, and lacking in service standards, organizational management, experience in intersectoral collaboration, and specialized skills to work with parents and children facing serious challenges. These changed with this project. As a result, our raion has become more confident in our work and relationships, and more open to collaboration with the needs we identify.” Social assistance and family support managers said they appreciated the fact that the P4EC team always gives them feedback on their activities. They said P4EC is not just coordinating activities, they always listen to other opinions and positions. In one of the 24 Through key informant interviews with Lumos Foundation and CCF (Hope and Homes for Children). UNICEF reiterated these same themes. EVALUATION | 89 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project “new” raions, the family support manager attributed her professional growth mainly due to collaboration with P4EC. The P4EC capacity building and training approach is comprehensively vertical and horizontal and often saturates a raion, such as involving all mayors in a raion in joint trainings and capacity building. Numerous trainings include participants from across Moldova, not just the raions in the project. With the intersectoral approaches emphasized in the project, different sectors attended many of the same trainings. Key informants discussing the training referred to the importance of “learning the same language” in prevention, deinstitutionalization, and alternative care. UNICEF stressed the importance of mentoring and monitoring in training and capacity building.25 Input from stakeholders involved in P4EC training indicates that the P4EC process excels at this. Numerous stakeholders interviewed in the endline evaluation attributed international exposure trips to strengthening their understanding, interest, and confidence in the systems being developed in the project. Noteworthy was the participation of education stakeholders in learning about the NPM component of observing and documenting behaviors of children in school for early identification of risk. Table 4.4.1 shows indicative (see footnote 27) summary data since the start of the project on the target group that has been trained and the number of people trained (disaggregated by gender). Table 4.4.2 shows the themes of the most frequently provided trainings. The project’s training reached a very large number of community members, school teachers, community social workers, and local-authority decision makers. Main themes addressed were the National Practice Model, implementation of the inter-agency cooperation mechanism, PANDA related activities and trainings, and development of inclusive education. 26 Table 4.4.1: Numbers of Stakeholders Trained by Target Group Group targeted by P4EC Female Male Total Policy makers 122 36 158 LA administration** 38 60 98 LA decision-makers** 1439 383 1822 Family support staff 228 20 248 25 From the interview with UNICEF Moldova’s Child Protection Specialist 26 The P4EC training monitoring database presented different numbers for the target groups trained and the total number of people at each event. The aggregation of the data did not allow us to understand the discrepancy between event totals and target group totals so we therefore suggest using these numbers indicatively. Additionally, P4EC has said these are the number of unique individuals trained, although the training database does not provide the ability to track unique individuals and we therefore believe these numbers may represent the same individuals in some cases who attended a training on more than one theme. EVALUATION | 90 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Group targeted by P4EC Female Male Total Foster care staff** 233 19 252 Community social workers** 2282 163 2445 Community members* 4549 1713 6262 School teachers* 2996 290 3286 Psycho-pedagogical assistance* 212 17 229 Journalists 53 10 63 District authorities* 223 69 292 Gate-keeping commissions Local Level 84 27 111 Gate-keeping commissions National Level 210 52 262 Child protection workers 0 0 0 Residential staff 56 14 70 Community workforce 0 0 0 Groups facilitators 166 13 179 International Organizations 22 15 37 Academics 38 4 42 Total 12,951 2,905 15,856 Source: P4EC administrative data *Multidiscplinary team members and coordinators came from the following groups: school teachers, psycho-pedagogical assistance, community members, and district authorities **SAFPD staff were categorized under community social workers, LA decision-makers, LA administration, and foster care staff Table 4.4.2: Themes of main trainings provided Theme of the most frequently provided trainings Number of people at the events National Practice Model related trainings 5951 Implementation of the inter-agency cooperation mechanism on identification, prevention and assistance of CANET 2845 PANDA related activities and trainings 2754 Development of inclusive education: working with children with individual educational needs 1331 Implementation of PL 140 regarding the special protection of the children in difficulty and children separated from their parents 806 Family support service methodology and application 748 Workshop for the consolidation of CSA and community social service managers in communication with the Mass Media entities 501 Training on the new revised Gate Keeping Regulation 395 Strategic planning workshops 321 Source: P4EC administrative data Alcoholism, negligent or abusive parents, poverty, and migration, continue to be the main reasons why children are separated from their biological families. At both the baseline and the endline, the primary reasons mentioned for children’s separation from their biological families were parental negligence or abuse and poverty at a level where parents were not able to care for their children. Migration to find better economic opportunities and a higher standard of living was also a large reason for separation. EVALUATION | 91 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project One reason that did not figure prominently in the baseline, but was present at endline, was illness of the parent or child, though it was mentioned less often than the other reasons. The project is successfully addressing issues of poverty and lack of effective parenting by strengthening government-provided social services. The foundation of the prevention system is support provided to families identified with vulnerable children. This comes in many forms: counseling such as in parenting education, links to schools for inclusive education, material and financial support, and services such as those offered at day care centers. The core package of services provided to families by the government includes a family support service (including financial support); means-tested cash benefit (as a social protection measure); alternative care services, mainly foster care; and psycho-social support in school in order to meet the child’s individual educational needs. Primary and secondary family support services include emotional, practical and monetary support to the family in order to strengthen parental skills, improve children's care, education, health and living conditions, and ensure access to means-tested poverty benefits and other relevant services. Primary support is provided to the family and children at the community level by the community social assistants in collaboration with other community actors such as the family doctor, school teacher, and community mayor. Some children and families who face complex and multiple problems are referred to the district level SAFPD by the community social assistants in order to receive coordinated family support services (secondary support) by professionals cooperating across different agencies at the district level, which is coordinated by the SAFPDs. One ministry official said: “The decreasing number of children in institutions is the proof of a functioning prevention system. However, there is more work needed to change the mind-set of local authorities. In the past, there was a perception that any problems faced by a family could be solved by providing financial support, but now we realize this is not enough.” Social assistants assist families in applying for the ajutor social economic benefit. This social protection mechanism amounts to the difference between the family income and amount identified for the family based on the national minimum consumption basket, adjusted to the size and characteristics of a family. In an “old” raion, a district official said: “What has changed (over the project period) is the modality of taking a child from the family; previously, a request from the mother was sufficient. Now a child is removed from the family in extreme situations only. Many families lack financial resources and EVALUATION | 92 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project this, in part, leads to separation. We have done much work on this in provision of the social benefits (Ajutor Social), family support and material support. Now all these problems are solvable, so that the lack of money should not be a reason for separation.” In a “new” raion, the service was established in 2014, following a decision and regulation approval by the District Council. The VP of the raion and SAFPD director attributed P4EC with establishing the financial support process for vulnerable families and said it was vital to the process of deinstitutionalizing children in the raion, including children with severe disabilities. In another “new” raion, the family support manager explained that P4EC helped the raion relaunch this support in parallel with an educational inclusion process. Family support helps to improve living conditions in the home and parenting skills of care givers. The database in this raion has detailed information about each family, including all stages of support and assistance the family went through. Family-based Alternative Care in the Core Basket of Services The availability of alternative family care options increased by project end. Overall, alternative family care options have become more available by project end. The forms of alternative care27 that experienced the biggest change in availability from baseline to endline are foster care/APP and emergency/temporary placement in shelters jumping by 14 and 13 percentage points respectively, as reported by social workers as seen in Table 4.4.3. There are two distinct types of foster care currently in Moldova. One is family-type children’s homes (“CCTF” - sometimes referred to as family-based group homes) caring for up to five or seven children. The second is a more traditional foster family (“APP” - Professional Parental Assistants) usually caring for one to three children or a small sibling group. The financial support to each is different. CCTF are provided resources for housing upgrades and upkeep. The service was mostly used/developed in the process of deinstitutionalization carried out by Hope and Homes, especially in early stages of their involvement in deinstitutionalization. Now they also develop/use foster care for deinstitutionalization and prevention of institutionalization. CCTFs can also be used to provide specialized care for children with disabilities and for larger groups of siblings. According to P4EC, the longer-term vision of the government is to transform CCTF into APP. 27 In Moldova, the term placement is used for non-permanent solutions only and adoption is considered a form of alternative care. Therefore, we use the Moldovan terminology in this report. EVALUATION | 93 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project P4EC is a pioneer in APP in Moldova. This model is promoted and supported in the project. The Minister of MoLSPF said, “The regulations and minimum standards for family-type alternative care did not have a unified and standardized structure. They were revised with the support of P4EC in 2014. P4EC is the organization that developed these services and they were ideally suited to do this.”28 Some community social assistants in focus groups noted that the availability of alternative care had increased to meet the need of residential care closure. They also noted the impact of campaigns around fostering children. In one focus group with social workers, participants noted that in two localities, people were initially skeptical about fostering children, but followed suit when they saw that the children and parents were doing well. In the quotes below, social work specialists note that the community became more sensitive to the plight of children in residential institutions. “[The number of foster parents’ services is increasing] because the residential institutions got closed, and it is better for them to be in the family, to raise a smaller number of children.” “The community became more sensitive. There are families whose children have already [left] and they would like for the house to be full of children’s laugh. These are people with big hearts, who would like to provide their love to somebody and they decide to be foster parents.” (Social worker specialists) APP foster care has grown considerably over the project period – both in the number of active foster families by 44% and the number of children benefiting from fostering by 83% (Table 4.4.3). Growth has occurred in each raion. Rezina and Singerei now have foster families when previously they had minimal or no foster families. Ungheni and Orhei have experienced significant growth in aggregate numbers of children served. Foster care has also become far more efficient: the greater number of foster families are caring for an even greater average number of children, growing from 1.66 to 2.11 children per family over the project period – a 27% growth in efficiency utilization. This confirms stakeholders and P4EC key informant input about the greater efficiency of APP fostering. Investment in fostering means more children can be cared for, the care is of higher quality since there are only one or two children (usually sibilings) being 28 The Minister was instrumental in this process as the director of P4EC prior to her appointment as Minister in January 2016. EVALUATION | 94 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project cared for by a family, and fostering fills a short term gap in care needed by children – an outstanding alternative to residential care. Table 4.4.3: Foster Care Service Development Districts 2014 Year 2017 Year Foster carers Children in placement at the end of the year Foster carers Children in placement at the end of the year Cahul 7 14 10 25 Calarasi 21 32 25 45 Causeni 6 18 12 30 Falesti 11 20 15 34 Nisporeni 7 13 10 19 Orhei 32 46 45 80 Rezina 0 0 4 7 Singerei 1 2 7 18 Soroca 17 21 19 28 Telenesti 8 6 11 36 Ungheni 29 59 42 100 totals 139 231 200 422 % change 2014 - 2017 44% 83% Average # children per foster family 1.66 2.11 (27% growth) Source: P4EC There was a shift away from CCTF to APP foster care. SAFPD stakeholders in one of the “old” raions said it was more cost effective for them to scale APP in their district when compared to CCTF, and the quality of care was superior. CCTFs were transitioned to APP due to high costs of the former with its costs in furniture and repairs. According to specialists in this raion, this helped to grow the number of foster families. Foster care is for a temporary time to meet a care gap. P4EC training included the challenging process of fostering children with disabilities with foster families. The raion instituted a 45-day respite period for foster parents in which these number of days can be used throughout the year. The training and new procedures were beneficial since no children with disabilities in recent times have needed to be referred to residential care except for a very young child placed in temporary care while permanency is worked out for this child. Fostering children with disabilities can be expensive. One of the “old” raions’ commitment to foster care included tapping a funding mechanism to better support this care option. The head of Child and Family Protection worked with another specialist to advocate for pensions available for parents with disabled children to also apply for foster carers. This now provides important additional support for fostering children with disabilities. Now, as per Law 66, the Civil Code of April 2017, a foster family can represent a foster child presenting identification and documents to access pensions. EVALUATION | 95 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project In a “new” raion, the raion specialist in charge of foster care indicated that the raion needed at least five more families and another specializing in respite care to adequately cover the raion’s family-based alternative care needs. The directorate had the necessary human resources to monitor this number of foster families (the regulations state one specialist shall monitor up to 15 foster families). However, the raion was not able to allocate the extra resources for this service. In another “new” raion, they are transitioning CCTFs to finance additional APP foster families. Specialists say they can transition two CCTFs into support for three new APPs where the quality of care is superior for children, especially those with disabilities. Even with this change, district officials indicate there is a funding gap to engage the number of foster families they require. All raion stakeholders interviewed in the endline evaluation identified a gap in alternative care services for older children in their teens with severe mental disabilities and “deviant” behavior. For the alternative care types that were available, survey respondents were asked to rate the quality of service for each type of alternative care as seen in Table 4.4.4. On average, 96% of endline respondents said that the alternative care was “excellent/good quality;” which is up from 90% at baseline. Boarding schools ranked lower, with 72% of endline respondents viewing them as “excellent/good” quality, a slip from 77% at baseline. Table 4.4.4: Availability and Quality of Alternative Care in the Raion/Community EVALUATION | 96 Q26: What types of alternative care are in place in your raion/community to meet the needs of children without adequate parental care? Yes Excellent Quality Good Quality Fair Quality Poor Quality Baseline, % Guardianship 97 31 58 10 1 APP 47 32 58 10 0 CCTF 28 24 67 6 1 Community group home 18 28 58 9 0 Adoption 67 42 51 5 1 Emergency/temporary placement center 26 29 57 10 1 Boarding schools 7 27 50 14 5 Other resident institution (not boarding school) 9 31 50 4 4 Endline, % Guardianship 99 30 62 7 0 APP 61 46 50 3 0 CCTF 27 39 56 4 0 Community group home 24 37 54 7 0 Adoption 69 49 47 2 0 Emergency/temporary placement center 39 43 53 4 0 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project EVALUATION | 97 Q26: What types of alternative care are in place in your raion/community to meet the needs of children without adequate parental care? Yes Excellent Quality Good Quality Fair Quality Poor Quality Boarding schools 6 11 61 22 0 Other resident institution (not boarding school) 14 36 56 5 3 Source: NORC evaluation survey Foster children are considered “difficult” to parent. At the endline, there were a variety of challenges mentioned with regard to raising foster children. Foster parents and multidisciplinary team members spoke of how it is difficult for foster parents to raise the children because they are poorly adjusted, uneducated at home, undisciplined, poorly behaved, and violent. One endline focus group with community social assistants said that it takes time for these children to bond with their families, and an endline group with foster parents said that the children have difficulty expressing themselves and understanding their new situations. Foster parents are not reaching out for assistance when it is needed. Foster parents face unique challenges in their care of children. One endline focus group with foster parents said that they refrain from seeking help even when they need it as they feel ashamed and worried about how others will perceive them. They also refrain from seeking help for paying for necessities like washing machines, electricity, and internet even when it is difficult to afford. One participant from this group said that they never told the social assistant that they do not have firewood even though the assistant was in a position to help. Besides not admitting that they need help, foster parents face challenges such as obtaining medicine for foster children. The process is delayed because certain paperwork is required to get the medicine that is not readily available. According to some gatekeeping commission members, there is also a need for training the foster parents, as well as a public campaign to raise awareness of how children’s lives are shaped by foster care. Other Care Centers: Day, Temporary and Multi-purpose Centers Day centers and temporary overnight shelters are vital to eliminating the need for residential care. They are also critical to preventing greater risk to children and families. Multi-service centers in some project areas combine these functions. They are supported by specialist staff from a variety of sectors, and volunteers from the community. In a center in an “old” raion, day care is offered to children whose parents need to work and have difficulty affording other care options. Youth engagement occurs through vocational training and arts and crafts activities. Often the beneficiaries receiving these services are children with disabilities. Parenting and health education classes are occurring. Separate accommodations exist for children and youth in transition to more permanent living situations and for children with disabilities whose caregivers need respite. Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project In one of the “new” raions where no residential institutions now exist, there are day care and multi-functional centers in three localities that socially vulnerable families and their children are accessing. Three centers also exist in another “old” raion – one specializes in children with disabilities in the city, while mobile teams initially supported by Keystone Moldova support families in rural areas having children with disabilities. Another center works with older children and, as the child grows up, he/she is transferred from one to the other. There are about 32 children in both centers and they provide day care thus enabling parents to work. The third center with about 14 children is a temporary shelter with overnight capacity providing emergency services. This facility can provide care for children when parents or foster carers need relief (respite) in caring for significantly disabled children. Temporary shelters can also be used in serious protection situations when children or adults need to separate from an abusive situation at home. In another “old” raion, a mother and child center was established last year based on a decision of the District Council. This center has a wide range of specialized services, including trainings on family budgets, a child helpline/hotline, emergency placement facilities including those for mother and baby placement, and temporary placements for young people who left the residential system. Mellow Parenting, a parenting education program of P4EC not funded through DCOF/USAID sources, is occurring there as well. Guardianship The project is strengthening guardianship arrangements and targeting services to families and children impacted by parents’ labor migration. The project has focused on guardianship through implementation of PL 140 at the local level – mayors (the local guardianship authority according to PL 140) have been trained in the implementation of this law. Additionally, the early identification and multidisciplinary approaches emphasized by the project are important in addressing the needs of children impacted by migration. Extremely high numbers of Moldovans migrate to work in other countries. In a country with a population of approximately 3.5 million, the number of Moldovans residing abroad at the end of 2015 was 753,800, according to Border Police data, with roughly 20,000 individuals who are absent for a period of at least three months.29 Project stakeholders report considerable risk for children whose parents migrate outside of Moldova for work. Children are sometimes left unattended and/or under the care of older youth. Other times children are left with relatives or other known individuals who may not have their best interests in mind. Children are also left with a single and vulnerable parent, such as a young mother, who is destitute or has mental or physical challenges. When asked why children remain without care, a mayor replied: 29 29 IOM Mission to Moldova on line fact sheet EVALUATION | 98 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project “Because parents leave the country. Some parents take the children with them, but most leave them here. In order to take the children with them, they need to find a job first. But I went there and I have seen the reality. In Vărzărești we also have families from which children have been take away forcedly, because parents did not take care and had a bad attitude towards the children. As much as 5 children have been taken.” (Mayor, “new” raion) Some parents and community social assistants in focus groups also spoke of migration and its effect on children who they feel lack love and appropriate care. A community social assistant commented: “They have to emigrate abroad looking for a salary, because there are no jobs here. And the children suffer a lot. The money does not replace parents’ love, we all know this. The children live with the old grandmother, who already needs silence and peace, while the child wants music. And all these impact the child. They are not taken care of, they are not checked at the doctor, they lack clothes, do not have adequate education.” (Community social assistant, “new” raion) Guardianship is identified by project stakeholders as a key mechanism to better protect children of migrant parents. The appointment of a guardian brings PL 140, the key Moldovan law on Special Protection of Children Separated from their Parents, into force so support and protection can be monitored and provided by the state. This requires identification of children whose parents are migrants and assigning guardianship. Guardianship exists in two basic forms in Moldova: supported financially by the state and that which is non-paid. The former is determined by raion authorities for children who are temporarily or permanently without parental care. For example, in one of the “old” raions, 125 children are in this guardianship category and there is a monthly allowance of 800 MDL per child. In another “old” raion, 100 children are under such guardianship arrangements. The second type of guardianship is applied for 90 days in the case of children whose parents leave abroad for work. Most of these children live in extended families or with other known persons without there being a legally appointed representative responsible for the well-being of the children. Stakeholders report progress is being made to better protect children in this form of alternative care. In one of the “old” raions, the procedure for guardianship appointment has been simplified and now does not involve costs. Therefore, number of these guardianships have grown from zero prior to 2015 to 45 children today. Official migration to countries such as Canada or Israel requires a certificate confirming children will not be adversely impacted by the migrants. However, unreported migration is a problem and difficult to control. The best solution is through communication EVALUATION | 99 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project strategies to disseminate information in the public media and through government offices. When a family asks for support for leaving to go abroad, specialists get involved to monitor a child’s well-being and the appointment of a responsible person. The mayor is informed and assigns the guardianship. Once identified, multidisciplinary approaches prioritized through the project can be applied to safeguard the well-being of children with migrant parents. One head of the Education Directorate interviewed explained: “We know there are children in our schools whose parents have left abroad, or children living with one parent only. It is good that specialists from education are aware of these children and know their identity. Schools have resource centers to assist them. These children, as well others who are at risk, are also being provided with meals – not just those in primary schools but in upper grades as well.” Stakeholders consulted in this evaluation were hopeful that current efforts to strengthen guardianship regulations through the civil code associated with PL 140 will provide greater guidance in the procedures of guardianship protection – particularly as it applies to vulnerable children left by migrant parents. Psycho-Social Assistance Program for Children The project addresses adverse consequences of alcohol abuse. In addition to poverty and the lack of effective parenting skills, alcohol is identified by all stakeholders as a most serious issue in Moldova putting family unity and well-being at risk. Alcohol abuse can lead to children being separated from their parents or parents not being able to take care of their children, and can also lead to children developing emotional problems down the line. One foster parent said: “They are psychologically affected for the rest of their lives. Maybe not a lot, if they manage to have a normal family of their own. But still, something lingers, they still remember. These are traumatized children.” The project addressed this service gap through its Psycho-Social Assistance Program for Children (PANDA). PANDA participants are children and adults living in households in which there is a member who abuses alcohol. The focus of the PANDA program is on helping family members cope and stay safe. It was developed by the project in all the “new” raions of the project and was scaled up to include Soroca, Causeni, Nisporeni, and Cahul over the project period. PANDA participants indicated alcohol abuse amongst parents often led to their children also abusing alcohol, physical abuse in families, and financial difficulties. One mayor gave an example of children being stigmatized or not being recognized for their achievements just because their parents are alcoholics: EVALUATION | 100 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project “I had a case about 20 years ago. Husband and wife worked in the collective farm, but they were drunkards and had 5 children…One girl from those children graduated school with a golden medal. And they still considered she was like her father, although she graduated with a golden medal.” (Mayor) Most PANDA moderators work out of Psycho-Pedagogical Assistance Service units and are psychologists. Those interviewed for this evaluation30 received extensive training and have been involved in PANDA for an average of about 1.5 years. They have moderated between two to four groups of children and/or parents. Some raions work exclusively with children in PANDA, others have include adult groups. Children are identified for PANDA participation through various means including from raion family support specialists and through referrals from schools. Parents are informed about the program and are asked for their written consent for their children to participate. All the moderators agreed that alcohol abuse is one of the greatest risks for children in family care and that there was a significant gap of services in their raions to address this crisis. Through PANDA they are learning a great deal about alcohol abuse and able to clarify participant misconceptions. Examples include: ■ Identification of alcohol addiction stages; ■ The error in thinking that the only solution is removal of a child from an alcohol￾addicted family; ■ Emotional consequences for children of living in these situations; and ■ Alcoholism should be treated not just as a protection issue, but as a disease. Moderators say the main purpose of PANDA is to help children understand alcohol addiction and, through this, strengthen their psycho-social well-being in self-confidence and respect and minimize the burden and guilt they carry with them. Two important PANDA goals are for children to realize alcohol addition of their parents is not their fault, and to minimize the risk that they will follow the pattern of their addicted parents. Boys tend to be at greater risk of this. Also, key is helping children to more effectively control and channel their emotions. Children are also learning there are persons whom they can ask for help when in need or when they need to talk about a problem. They develop empathy for the other parent who is not abusing alcohol. Often children start the PANDA sessions reluctant to talk about their issues, but by the end are freely talking. 30 This summary is based on key informant interviews with moderators from Ungheni, Falesti, Nisporeni, Orhei, Calarasi EVALUATION | 101 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project The program also has youth members that serve as confidants for their fellow students who may be in vulnerable families. Youth members learn how to tell other children about people they can go to for help, and where children can talk about their feelings with regards to alcohol abuse. Early outcomes of PANDA are very promising, though it is too early in the program development in the raions to draw firm conclusions on PANDA impact. However, PANDA moderators and youth members have no doubt the program contributes to preventing the separation of families. The PANDA moderators and youth members indicated that they see evidence that some children: ■ Are avoiding conflictual situations and have changed their behavior towards others for the better; ■ Feel more protected when they become aware of the fact that the adult is a support person and can provide help; ■ Feel better about themselves, including instances of lowering the threat of suicide; ■ Have a greater willingness to report abuse to the police; ■ Actually feel much better when parents are aware of their problems. For instance, one of the children said, “Did you see, mom, how much I am affected by the fact that dad consumes alcohol?” ■ Are pleased when their parents join groups and start learning about addiction; ■ See changes in their parents and parents becoming closer to them. Parents are encouraged to make complimentary statements to children in the program and this is occurring. One person observed, “Some of the children were moved to tears, as they have never heard their parents telling them kind words before. They were not aware of how important compliments are for children.” ■ Are becoming closer to their parents; ■ Are informing other children about what they are learning in PANDA, including during breaks at school, and it is helping these children; ■ Have a larger circle of friends; or ■ Are establishing boundaries with their parents, such as times for privacy. All of the moderators interviewed held a strong opinion that PANDA should continue and be replicated across the country. They say it is one of the few effective responses in Moldova to a most challenging problem affecting children and families. Moderators also recommended: ■ There need to be more male moderators/specialists to help explain men’s behavior. Many children believe it is normal for dads to consume a lot of alcohol; EVALUATION | 102 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project ■ PANDA sessions for parents are important – some believe the results of the program are greater when children attend sessions together with their parents; ■ Strengthen assistance to non-addicted parents, usually mothers, with resource person and protection support; and ■ Include video examples to demonstrate how to behave in good and bad situations; ■ Explore ways to make the program more convenient for parents given their work responsibilities. According to the PANDA youth members interviewed, the most useful activities were learning about family and the need for the family to change their behavior; learning about alcohol and drugs; and lessons preparing for careers and for the future. “We had two groups of children participate, those in grades 1-4 and in grades 5-7. As a result of the program, several changes were noticed, such as the fact that children are no longer hiding the fact that their parents consume alcohol. They admit that alcohol addiction is a disease and should be treated as such. They have learned to speak openly, have been informed about the persons they can ask for help if needed, to not be afraid and ashamed of their parents’ behavior. The program also helped renewing the child-parent relationship.” (Head of Education Directorate, “new” raion) Some PANDA youth members believed both parents and children should be participants. Others doubted behavior of alcoholic parents could change much. b) Are there fewer children living in residential care facilities? The project has resulted in RCC closure and reducing the need for RCCs through prevention of separation, family support and alternative care. The project has been very successful in this regard. The project was able to deinstitutionalize a total of 270 children, closing five RCCs in Rezina, Nisporeni, Soroca, and Orhei. c) Is the well-being of deinstitutionalized children assessed as being adequate? Deinstitutionalized children continue to face challenges in Moldova, although at somewhat lower rates, and the use of mitigation methods have risen. The evaluation survey shows that community social assistants and social work specialists believe deinstitutionalized children and their caregivers continue to face challenges in the country. This is not surprising given the global body of knowledge on the debilitating impact of institutions on the physical, emotional and cognitive well-being of children. While the percent of respondents who believe that there are no challenges facing deinstitutionalized children and their caregivers has dropped 7 percentage points by endline, the percentage of respondents who felt deinstitutionalized children face each of the specific challenges dropped for all categories. Endline respondents identified the EVALUATION | 103 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project same two primary challenges as baseline respondents, which are family risk factors and attachment between the family and child is lacking. Table 4.4.5 also presents the results for how the identified challenges are being addressed. The most common form of mitigation was social workers providing family care plans, with 99% of respondents saying that this form of mitigation was being used, a rise of 10 percentage points from baseline. Aside from family care plans, each of the mitigation methods was being used more at endline than at baseline. Based on stakeholders interviewed, this was due to the project’s work with SAFPDs and community social assistants. Table 4.4.5: Challenges Faced by Deinstitutionalized Children Challenges Faced by Deinstitutionalized Children Baseline, % Endline, % Change Yes Yes What, if any, are some particular challenges facing deinstitutionalized children and their caregivers? No challenges 21 14 -7 Stigma from the community 41 33 -8 Lack of access to school 12 11 -1 Family risk factors 59 50 -9 Attachment between the family and child is lacking 54 47 -7 Access to services 19 8 -11 How are these challenges being addressed? With the social worker through the family care plan 89 99 10 Through referral to other services 84 96 12 Through a special education plan with the school 88 96 8 Counseling services for the child 87 97 10 Counseling services for the parents/caregiver 87 97 10 Counseling for the family 84 94 10 Disability rehabilitation or other services 75 88 13 Work with the community 76 92 16 Source: NORC evaluation survey Similar to the baseline, many focus groups mentioned that children from RCCs had difficulties adjusting to their new living situations. The project could have helped community social assistants, and indirectly parents, respond more effectively to such behaviors. The reasons for these difficulties were also similar to those at baseline and include the perception that children had better living conditions in RCCs, including more freedom, better food, and a relationship with their peers. Some focus groups mentioned that children from RCCs may have had difficulties integrating into their communities because they were shyer and less likely to share their problems with their new caregivers. Other stakeholders said that they lacked the life skills necessary to communicate, manage their finances, and perform basic household chores. A few gatekeeping commission members felt there might be a mismatch between the child and the placement family and a lack of adaption period for the child into their new family. EVALUATION | 104 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project According to key stakeholders interviewed, raion specialists faced many challenges in their casework with formerly institutionalized children and youth who had behavioral issues. Focus groups with multidisciplinary teams, community social assistants, and children noted that children reunited or placed in family care can exhibit behavioral issues in their new placements, or that they can return to their homes and become aggressive and mean spirited. Multidisciplinary team members and community social assistants mentioned that these children can also engage in deviant behavior, such as drinking alcohol and theft. Examples included youth encountering the law, having emotional issues resulting in aggressive behavior, those with diagnosed personality disorders, and youth prone to running away or being unstable in other ways. While this is to be expected for children who have been institutionalized or have trouble in fragile family environments, many specialists interviewed in the evaluation did not anticipate the degree of challenges and amount of time it took to case manage these children and their caregivers. Conflict in families with reintegrated children was often mentioned in focus groups. Some parents and children felt that children from RCCs returned to their families with different habits and a broken relationship to their parents; some parents and stakeholders from the social service sector felt these children lacked the life skills necessary to communicate, manage their finances, and perform basic household chores. At endline, social workers seem to have a greater understanding of the needs of deinstitutionalized children. While the evaluation survey found little change in social worker perception of whether deinstitutionalized children are adapting well, the percent of social workers and social work specialists who believe these children face more challenges has increased for each of the categories in Table 4.4.5. This seems to indicate that social workers through the project likely have much greater understanding of the needs of deinstitutionalized children now versus at the baseline and are therefore more aware of children’s required needs and services. As seen in Table 4.4.6, the top three categories of deinstitutionalized children facing more challenges are children with learning disabilities, children with disabilities, and children from poor families. These are the same three top categories identified by baseline respondents. Table 4.4.6: Types of Deinstitutionalized Children who face more Challenges Q76: Are there any categories of deinstitutionalized children who face more challenges than others? Baseline, % Endline, % Change Yes Yes Children with learning disabilities 66 84 18 Children with disabilities 65 80 15 Children from poor families 62 82 20 Older children 51 72 21 Boys 47 72 25 EVALUATION | 105 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Q76: Are there any categories of deinstitutionalized children who face more challenges than others? Baseline, % Endline, % Change Yes Yes Girls 41 61 20 Children with chronic illnesses 41 63 22 Children who are of an ethnic minority 37 53 16 Younger children 30 51 21 Source: NORC evaluation survey At endline, almost all social workers feel that needs of deinstitutionalized children are being met - a slight increase from baseline. Over 90% of the endline social worker evaluation survey respondents “strongly/somewhat agreed” that the needs of deinstitutionalized children are being met (Table 4.4.7 below), which is an improvement from baseline by 2-8 percentage points across all areas. The largest improvements between baseline and endline were made in the categories of “needs are being adequately met in the community,” “[children] are in protective family care that ensure their safety and well-being,” and “[children] are placed in well-planned placements.” The endline evaluation revealed that there are more stakeholders involved in child protection coordinated activities at the endline compared to the baseline – not just social protection, but health and education sectors are also more involved. Based on stakeholders interviewed, this improvement is likely due to the project. Table 4.4.7: Deinstitutionalized Children’s Needs and Placements Q78: Do you agree or disagree with the following regarding deinstitutionalized children: Baseline, % Endline, % Strongly Change agree Somewhat agree Total Strongly agree Somewhat agree Total Needs are being adequately met in their family placement 17 73 90 24 69 92 2 Needs are being adequately met in their school placement 33 56 89 35 60 95 6 Needs are being adequately met in the community 17 69 86 24 70 94 8 Needs are being adequately met by the services available 31 59 90 44 53 97 7 Are in protective family care that ensures their safety and well-being 37 50 87 48 46 95 8 Are placed in well￾planned placements 39 45 84 55 37 92 8 Are placed with their permanency in mind 44 45 89 53 43 96 7 EVALUATION | 106 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Q78: Do you agree or disagree with the following regarding deinstitutionalized children: Baseline, % Endline, % Strongly Change agree Somewhat agree Total Strongly agree Somewhat agree Total Are placed with adequate consideration to their best interests 58 33 91 67 30 97 6 Source: NORC evaluation survey An indicator for reintegration is the number of children who “disrupted” from their placement into homelife. Only five children of all of those placed were identified in project raions as having disrupted. If this data is accurate, this is a very low number of disruptions.31 The evaluation reviewed the files and interviewed the case workers of three of these cases. A summary of the disruptions and follow-up support exists further below in this section. For these children, the casework remained intensive to protect and find appropriate care options for the children. Negative stereotypes of deinstitutionalized children continue at endline. Stigmatization is also a strong concern for reintegrated children. A number of focus groups with adults discussed how formerly institutionalized children are not accepted by other children. Social work specialists reported that reintegrated children are bullied and teased by their classmates and that they are not supported or assisted by community members in their reintegration. “A 6th grade boy returned from the boarding school and was taken to the doctor to get his eyes checked. The boy wasn’t able to get his eyes checked because he couldn’t read the alphabet, and the other children laughed at him.” (Mayor, “old” raions) Reasons for successful placements include household living conditions, training, and monitoring. When asked what they thought were the key determining factors for the success of alternative care, children and community social assistants said that a household’s living conditions, such as adequate housing, regular meals, and medical assistance, were important for a successful transition. Providing training to guardians and monitoring the child after placement were also mentioned as important factors for the success by community social assistants, social work specialists, and gatekeeping commission members. Assessment and ongoing support in reintegration by caseworkers to families and training were project priorities; this evidence shows that they were correctly prioritized and a reason for successful reintegration. There were a couple notable successful reintegration cases that participants were happy to share. In 31 This is the opinion of the evaluation technical expert Gary Gamer based on anecdotal evidence in other countries. He has been working in adoption and alternative care globally for over 20 years. EVALUATION | 107 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project one instance, the children have been reintegrated with the biological family after they were separated for one year. A mayor shared one case: “The mother had an immoral lifestyle. She was visited twice a week until the children were finally separated from her. After, she said she would reduce alcohol intake. She worked to renovate the house and even planted the flowers and worked the garden.” (Mayor, “new” raion). Another mayor shared a case of a family with three children: “They were considered socially vulnerable, but [the parents] quit drinking and made some repairs to the house. The oldest child was considered to be a challenge to work with as he leaves classes, does what he wants, and is aggressive in school. He reportedly has changed little but the other two children are better. The family has been doing well for 2-3 months and not drinking.” (Mayor, “old” raion) Deinstitutionalized Children Database The below tables present the results of the analysis of P4EC’s Monitoring and Evaluation database for deinstitutionalized children. The reported baseline data is from the date when the child’s case was opened and a full vulnerability assessment was conducted while in the RCC. The reported endline data is from the last follow-up visit conducted by a community social assistant for each child. It is important to note that the data reported is based on secondary perception, that of the social worker, and not directly from the deinstitutionalized children. It is also important to note that the children deinstitutionalized were from auxiliary boarding schools and so the majority of these children have learning disabilities/special education needs. The endline indicators for Ciniseuti are for one year after deinstitutionalization since most of cases were closed after this (either because the objectives of the plan were achieved or children had become 18 years old); the endline indicators for Nisporeni and Visoca/Soroca are for two years after deinstitutionalization (either because of case files closing or children had become 18 years old). Table 4.4.8 below shows the gender breakdown of the children included in the P4EC Deinstitutionalized Children Database who still had a case file open after two years. As can be seen, a large number of case files either closed or the children turned 18 years old in Ciniseuti between the baseline and the endline. In Nisporeni and Visoca/Soroca, there were fewer case files closed or fewer cases of children turning 18 years old. The database does not give the breakdown for the two reasons. EVALUATION | 108 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Table 4.4.8: Number of Children with Open Case Files by Gender across Project RCCs Children in RCCs Ciniseuti Nisporeni Visoca/Soroca Baseline Endline Baseline Endline Baseline Endline Male 43 7 24 22 32 20 Female 18 3 17 13 9 7 Total 61 10 41 35 41 27 Source: P4EC administrative data on children in RCCs Strengthening of case management likely contributed to the improvements in safety, health, and education indicators. The stakeholders interviewed generally said the project led to an improvement in the case management process and caseworker skills. Child safety was an important consideration in assessment for placement and on-going monitoring of the case. It was also an issue in schools, raising awareness for educators for inclusive education, which was part of the inclusion capacity building facilitated by the project. The overall case management process became more comprehensive in identification and assessment of risk and more responsive in prevention and taking action through multidisciplinary teams. As a result, the strengthening of the case management likely contributed to the improvements in safety, health, and education indicators documented below. Deinstitutionalized children were doing better on health indicators after placement and most received necessary medical checkups and treatment. Table 4.4.9 shows the percent of children in the database that received regular medical checkups and necessary treatments. Nisporeni saw the biggest increase (60 percentage points) in the percent of children who do receive their regular medical checkups and treatment with 29% reportedly receiving these services at baseline and 89% receiving them at endline. Ciniseuti and Visoca/Soroca both saw increases in the percent of children who “sometimes” or “most of the time” received regular medical checkups and necessary treatments from baseline to endline, making this the most common response for these two raions. The project, in strengthening the multidisciplinary approach to child care/protection, tied the health sector more closely to the target population of children. This is likely to have contributed to the improvement in health indicators. Table 4.4.9: Children Received Regular Medical Checkups and Necessary Treatment District Yes Sometimes/ Most of the time No Baseline, % Endline, % Change Baseline, % Endline, % Change Baseline, % Endline, % Change Ciniseuti 2 15 13 69 85 16 29 0 -29 Nisporeni 29 89 60 71 11 -60 0 0 0 Visoca/ Soroca 24 20 -4 32 60 28 45 20 -25 Source: P4EC administrative data on children in RCCs EVALUATION | 109 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Deinstitutionalized children mostly felt safe in their communities, although the change over time is mixed. Table 4.4.10 below presents the results of the children’s safety indicators. The first indicator explores whether the child feels safe and loved in the RCC or safe and loved at home; and the second explores whether the child feels safe in the community. At baseline, the children in Ciniseuti reported largely feeling safe “sometimes” in the RCC, which shifted drastically at endline with 54% reporting feeling safe “always” at home and 46% reporting feeling safe “most of the time” at home. Nisporeni and Visoca/Soroca showed different results with the majority of respondents “always” or “most of the time” feeling safe and loved at the RCC at baseline. Both raions saw a drop in the percent of children who reported feeling “always” safe and loved at home during endline. Regarding safety in the community, at baseline this referred to the community where the RCC was located and at endline it referred to the community where they were placed. Baseline data was not available for Visoca/Soroca. For this raion, we can only see that 92% of the children felt safe “most of the time” at endline. In Ciniseuti, the majority of children “never” felt safe in the community. This changed drastically at endline, with 89% of children in this raion feeling safe “most of the time.” In Nisporeni, the majority of children “always” felt safe in the community at baseline, which changed to most children feeling safe “most of the time” at endline. Table 4.4.10: Feeling of Safety in the RCC and Community Ciniseuti Nisporeni Visoca/Soroca Baseline, % Endline, % Change Baseline, % Endline, % Change Baseline, % Endline, % Change Children feel safe and loved in the RCC for baseline (is not exposed to violence, the staff applying non￾violent forms of managing challenging behavior of the child, etc.); feel safe and loved at home for endline Always 0 54 54 68 54 -14 44 9 -35 Most of the time 7 46 39 32 46 14 56 83 27 Sometimes 92 0 -92 0 0 0 0 9 9 Never 2 0 -2 0 0 0 0 0 0 Children feel safe in the community (able to move freely around the community) Always 0 2 2 53 14 -39 - 0 - Most of the time 23 89 66 37 69 32 - 92 - Sometimes 7 8 10 5 0 -5 - 8 - Never 66 0 -66 5 0 -5 - 0 - Under supervision (in case of young child ) 5 2 -3 0 0 0% - 0 - Source: P4EC administrative data on children in RCCs. There was no data for Soroca at baseline. EVALUATION | 110 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Children show large improvement in emotional health after deinstitutionalization, although children in Visoca/Soroca could further improve. Table 4.4.11 shows the proportion and frequency that children felt negative emotions. At baseline, Visoca/Soroca case workers reported that the majority of children in their raion, 62%, raion had negative emotions “often” while at the RCC. This proportion dropped at endline to 19% after deinstitutionalization. For Ciniseuti and Nisporeni, the majority of caseworkers reported that the children in their raion had negative emotions “seldom” (23% and 22% respectively) or “only sometimes” (38% and 24% respectively) at baseline. Both of these raions saw a shift at endline where the majority of case workers reported that the children didn’t display negative emotions at all after deinstitutionalization. This verifies an important premise of the project that family-based care contributes more positively to children’s psycho-social health. Table 4.4.11: Children’s Negative Emotions District Often Seldom Baseline, % Endline, % Change Baseline, % Endline, % Change Ciniseuti 30 3 -27 23 21 -2 Nisporeni 12 14 2 22 3 -19 Visoca/Soroca 62 19 -43 3 37 34 District Only Sometimes* Doesn't Display Baseline, % Baseline, % Endline, % Change Ciniseuti 38 10 75 65 Nisporeni 24 41 83 42 Visoca/Soroca 8 28 44 16 *The "only sometimes" option was only used during the baseline period Source: P4EC administrative data on children in RCCs Deinstitutionalized children struggled in school after reintegration. Finally, Table 4.4.12 below shows how the children in the deinstitutionalization database progressed in their school grades. The majority of children in all three districts were reported as having good grades at baseline, but each of the three raions experienced a drop in the percent of children who reportedly had good grades at home after deinstitutionalization. At endline the majority of children in each of the districts were reported as having “low grades but striving.” This finding is not surprising for a couple possible reasons. First, schooling received in RCCs was probably of lower quality with less expectations and therefore going to school outside of an RCC was likely more challenging. Second, children in auxiliary schools were placed there due to their real or perceived learning disabilities or special educational needs. It is understandable many would still be struggling to varying degrees in schools learning to be more inclusive. EVALUATION | 111 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Table 4.4.12: Children’s Progress in School According to His/her Potential District Has good grades Has low grades but striving No progress Baseline, % Endline, % Change Baseline, % Endline, % Change Baseline, % Endline, % Change Ciniseuti 44 11 -33 39 89 50 16 0 -16 Nisporeni 56 41 -15 39 45 6 5 14 9 Visoca/Soroca 51 23 -28 49 50 1 0 27 27 Source: P4EC administrative data on children in RCCs Review of Cases The endline evaluation assessed eight cases selected from the SAFPD caseloads in three project raions: Soroca, Causeni and Nisporeni. Three of the eight cases reviewed were disruptions32 of children placed with families out of residential care. The case selection in Soroca involved one of the two disruptions from deinstitutional placements that have been reported in the raion and selection of two additional cases from a pool of six cases presented by the social assistants for review. The four cases in Causeni were selected from a pool of approximately 20 presented by social assistants for review. In Nisporeni, the two cases assessed are disrupted. Five cases are reported as disrupted overall in the P4EC project raions, out of which three are reviewed here. All the cases assessed are male; however, female children live or are associated with the families in about half the cases and potentially benefit from support such as assistance in home repair, parental counseling, medical assistance provided to parents, and/or removal of an abusive parent from a household. For each case, the file was reviewed and discussed with the social assistant assigned to the case. Five of the cases reviewed are organized under the new case file system developed through the P4EC project and the three others are under the old system. Table 4.4.13: Case Files Reviewed in the Evaluation Raion Disrupted ? Case file status Case file system Primary risks/ vulnerabilities Approx. age Nisporeni (1) yes active new Poor single mom with moderate mental disability, child in and out of institutions (7-8 years) 12 Nisporeni (2) yes active new Poor single mom with mild disability, child with disability and difficulty with mom’s boyfriend, discrimination (Roma family) 12 32 Disruptions, sometimes referred to as ruptured cases, are when the placement or reunification of the child does not work out and he/she has to be placed again with another family, foster carer, or other type of alternative care. EVALUATION | 112 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Raion Disrupted ? Case file status Case file system Primary risks/ vulnerabilities Approx. age Soroca (1) yes active old After placement: poor single mothers without ID and her boyfriend’s alcohol abuse, 10 years of institutionalization 17 Soroca (2) no closed old Single mother and child with significant disability, significant poverty 10/11 Soroca (3) no active old Parental conflict, poverty, illness of mother resulting in death & adoption 5/6 Causeni (1) no active new Migrant father, young destitute mother, severe neglect 3/4 Causeni (2) no active new Violence and neglect by unemployed father abandonment by mother, care by grandma 6/7 Causeni (3) no active new alcohol abuse by mother, violence and removal from household of father 5/6 Causeni (4) no Closed (still monitoring) old Serious health issues of mother, domestic violence, funning away, temporary shelter, reintegration mother dies, adoption 8/9 Source: NORC evaluation team interviews and field observation Disrupted Case Review Two of the reviewed disrupted cases are from Nisporeni and one is from Soroca. The cases involve teenagers 12-17 years of age over the case management process. They had been in residential care before placement for four, eight, and ten years respectively. All have been identified as having moderate “mental” deficits (cognitive and emotional deficits could have resulted from institutionalization). One youth has epilepsy and another is of Roma descent facing issues of discrimination as identified in the case file. Alcohol abuse by a parent is identified in two of the cases and one of these involves abuse from an unmarried partner to his parent. Two of the cases identify a female parent as being young and fragile when the child was born and this contributed to the child entering residential care. Significant poverty is identified in all the cases, and lack of proper housing for children is identified in two cases. The disrupted case in Soroca involves a single destitute birth mother originally from Ukraine without proper ID who thus had difficulty in accessing steady employment. The mother wanted the child home, but the child was initially resistant to do this due, at least in part, to witnessing alcohol abuse by the mother’s partners in the household. Support services to the mother for the reunification included assistance in securing an ID, repairs to the home, and accessing support for individuals with disabilities. Although substance abuse was not identified in the initial assessment, the mother was later found to be an abuser, along with her partner (who was not identified in the assessment). The child fled the household (or was chased out) several times. The child was then placed in a foster family, is pleased with this arrangement, and plans are being made for when EVALUATION | 113 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project the youth becomes 18 for independent living support through vocational education. The placement is being monitored and the child is assessed to be doing well. Disrupted case #1 in Nisporeni involves a child institutionalized three years after his birth by a young and destitute single mother with moderate “mental” deficits. The child is also diagnosed with this deficit and having enuresis. The father is no longer alive. The case disrupted, in part, due to insufficient housing and neglect for the child, though counseling and parenting education was part of the pre-placement support. The child was placed in temporary foster care (CCTF) and has since returned to the mother’s household now fortified with improvements in infrastructure, financial support and parenting education for the mother and the child’s grandmother who is at home assisting. Disrupted case #2 in Nisporeni involves reunification of a youth diagnosed with emotional and behavior issues, and epilepsy. The mother is unstable with alcohol-abuse issues, compounded by being in a partner relationship with a man the child could not tolerate. This is a Roma family facing discrimination. The child has been in and out of school and runs away frequently. He accesses temporary shelters, stays with friends, returns home on occasion, and works various jobs to support himself. There is a trust relationship with SAFPD specialists and there is regular contact between the child and the SAFPD. Additional Disrupted Case Key Assessment Points: ■ All cases remain active and are closely monitored by the SAFPD. ■ In two out of the three cases, alcohol-abuse in the home to where the child re-unified contributed to the disruption, along with complications relating to the domestic partners of these mothers. To minimize risk to disruption, the family assessment and preparation phases for deinstitutionalization requires learning about and addressing these issues. ■ In all three cases, case work by specialists and the child protection system provided protection and stability for the children. In two cases, foster care and temporary shelters are utilized. There was one case of re-unification back into the biological home after the disruption. In the others, the social assistants have a close relationship with child and they monitor his situation carefully to provide support as opportunities present themselves. The first and third bullet indicate the disruptions were receiving comprehensive and effective case management services, and this is a capacity that was built by the project. EVALUATION | 114 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project There were some community social assistants and mayors who mentioned they had experienced ruptured placements to some degree or another during focus groups.33 Community social assistants and mayors noted that this could be due to poor treatment of children by their caregivers. To help avoid these cases, community social assistants said that parents needed to accept they needed help and have a desire to cooperate with the social system. Community social assistants and mayors said that the experience of having to take children away or of watching reintegration fail was stressful. Some community social assistants noted that they were seen as an unwelcome presence in the community, and were even threatened by parents who saw them as people who would take their children away from them. “Rumors abound and we are accused. I got a call from a woman asking me ‘why are people talking in the village that you will take my child away?’” (Community social assistant, “old” raion) Case Review of Non-Disrupted Cases Five of the cases reviewed are from Causeni and two from Soroca. Significant poverty and the need for economic support is identified in all these cases, except for one Causeni case. Depending on the case, this support includes linking to monthly social protection or disability support funds, repairs to homes and purchasing of assets to strengthen care and support to children within a family setting. Counseling, and particularly parent education, is indicated for all the cases. In two cases, issues of violence and alcohol abuse are being addressed. In several cases, parents or children are identified as having a disability, ranging from moderate “mental” disabilities to a more severe “mental” disability of a “bed-ridden” child. Table 4.4.13 above shows the range of risks and vulnerabilities. The following are impacts identified in the case review: ■ Two situations of protection removal: one of a child from an abusive home, and the other involving temporary removal of an abusive father. ■ One case of inclusive education of a deinstitutionalized child. ■ One case of transferring a child from a family environment to temporary shelter and foster care for protection and care, and later re-integration after family strengthening support. 33 However, it is unclear if FGD participants were referring to placements done by P4EC specifically, or commenting on placements in general. EVALUATION | 115 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project ■ One assignment of guardianship to a family relative (grandmother) to better protect and care for the child. ■ One child in process for adoption due to the death of his mother and inability of father to care for child. d) Have there been other unanticipated positive or negative results of the program? Increased adoption over the life of the project: In Causeni, the number of adopted children is increasing attributed in part to better assessment of children and the ability of workers to assess families. These capacities are a result of training received through the project, though the project did not directly focus on adoption. According to a raion child protection specialist, by mid-2017 a total of seven children have been adopted, compared with three children adopted in 2015. This is due to improved skills of matching authorities, monitoring of children and working in the best interest of the child. Degree of challenge and time to case manage children with behavioral issues: As discussed previously, raion specialists face many challenges in their case work with children and youth having behavioral issues. Examples include youth encountering the law, having emotional issues resulting in aggressive behavior, those with diagnosed personality disorders, and youth prone to running away or are unstable in other ways. This is to be expected for children who have been institutionalized or have trouble in fragile family environments. However, many specialists interviewed in the evaluation did not anticipate the degree of challenges and amount of time it takes to case manage these children and their caregivers. Mixed negative feedback on P4EC collaboration/coordination with other NGOs: Moldova is a small country where significant resources have been devoted to deinstitutionalization and related services over the last two decades. Relative to other countries in the region, the progress has been significant. NGOs have figured prominently in this process working in collaboration with government stakeholders, with considerable donor support such as through USAID/DOCF and from other governments, the EC, UNICEF, and private foundations. In the period of 2014-2016, approximately $19 million was allocated by donors for child protection, even though child protection was not a priority for global development partners. A recent study by the Global Alliance for Children concludes, “The implementation of the AP (Government Action Plan for Child Protection) remains fragmented with little or any coordinator across donor or implementers.”34 34 See draft of Global Alliance for Children Donor Mapping of Moldova Action Plan for Child Protection 2016-2020, funded through Oak Foundation EVALUATION | 116 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project P4EC is one of the prominent Moldovan NGOs involved in child protection, particularly deinstitutionalization, alternative care, and prevention activities to strengthen normative and government capabilities. The resources from donors to drive this work have been considerable. This comes with a responsibility to collaborate with other NGOs and stakeholders. Several NGOs interviewed in this evaluation request greater transparency and input-seeking from P4EC in activities impacting their work and with nation-wide influence. The emerging NPM and new case management tool are examples. There were complaints from some parties that results from piloting activities are not being made public. There is concern about how and whether the models will be reflected in regulations and how this will impact existing child protection mechanisms being implemented with government stakeholders in other raions.35 At the same time, all NGOs interviewed express significant appreciation for the work P4EC is doing, such as developing family-based alternative care, capacitating inclusive education and strengthening intersectoral collaboration. The guidelines produced and the trainings have been broadly collaborative and effective. PL 140 has broad-reaching consequences in child protection. The Vice Minister of MoLFSP explained there was considerable opposition to the law from some powerful stakeholders in the faith community. Though the church is strongly in favor of charitable support to families, there was opposition to aspects of the law that gave authorities the responsibility to act and intervene in family situations when the safety and well-being of children be at serious risk. There was risk that essential provisions in the law would not be enacted. P4EC was actively involved with other organizations to counter this risk so the law could be passed. This collaboration organized public debates and advocacy resulting in the law’s ultimate successful approval. Models being implemented through the project have informed the development of other social service sectors: The SAFPD head in an “old” raion said, “Motivation and confidence are the major drivers of our work. Our staff and other specialists have become more confident. They manage to develop specific plans for every family. Our district would like to apply aspects of these models in all public administrations.” The raion is planning for a “family club” in the future, based on a model from the U.S. In the club, families would share their experiences and mentoring would occur between families. Parenting education will be provided. Support is aimed to address marital conflict, supporting families having children with disabilities, educating men on violence 35 After the field work for this evaluation was completed, P4EC reported that the result of the piloting of the national model of practice were made public. In the contract extension period to the end of the year, capacity building was delivered to stakeholders across the country. This included community social assistants, child protection specialists, multidisciplinary teams, and staff of the National Social Assistance Agency and the National Accreditation Council and Social Protection. EVALUATION | 117 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project issues and provision of psychological support. This is being set up as a collaboration between P4EC, the District Council, some civil society associations, and the mayor’s office. Each of the involved parties are committed to a certain financial contribution. Improved care for adults was an unexpected positive outcome: An example of an unexpected development in Ungheni is how family-based alternative care development for children has impacted care for adults. The SAFPD was inspired by foster care for children to develop a similar program for the elderly. Specialists recognized residential care has negative impacts on both children and adults.36 According to Ungheni officials, foster-type care for the elderly has been assimilated from Ungheni into other raions. Conclusions: ■ The project provided widespread training and capacity building in skills and systems supporting a core package of services preventing the need for residential care and in support of family-based care. This applies to supporting reintegration; alternative family-based care – particularly APP foster care; strengthening guardianship as a protection mechanism; and reducing risk of separation for vulnerable children and families. The overall capacity building placed strong emphasis on intersectoral and multidisciplinary approaches to protecting children and strengthening families. ■ Social assistants reported that when comparing the well-being of deinstitutionalized children from when they were in RCCs versus in family-case, their health and safety is adequate in both cases and their emotional well-being is much improved. They indicated children faced challenges in the schools in the communities of their reintegration, more so than in the RCCs. However, the quality of education in the RCCs was likely far inferior and the demands and expectations for children in these schools were at a low level. Inclusive education is one of the essential components to successful deinstitutionalization in the project. ■ Social service workers reported that deinstitutionalized children continued to face challenges in Moldova, namely family risk factors and a lack of family and child attachment, although at somewhat lower rates than baseline. There was not much change in their perception of whether deinstitutionalized children’s needs are being met or the level of challenges they face in transitioning to a home environment. At the same time, at least half of social workers perceived deinstitutionalized children face greater challenges than other categories of at risk children. ■ For children who cannot be with their birth families, the project has contributed to increasing use of alternative care, in particular foster care and emergency/temporary placement shelters. APP foster care as an alternative care option has grown in the 36 The Director noted that this was made clearer after seeing the movie “One Flew Over the Cuckoo’s Nest.” EVALUATION | 118 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project project raions and some stakeholders are investing in its expansion. Stakeholders in most project raions cited the need for a greater number of foster families, though government funding shortfalls make this difficult. ■ Foster parents and multidisciplinary teams spoke of the challenges of raising foster children who were viewed as having violent or undisciplined behavior and are poorly adjusted to living in a home environment. Additionally, foster parents were somewhat reluctant to reach out for material or other assistance when they needed it. ■ Alcoholism, negligent or abusive parents, poverty, and migration are the main reasons why children became separated from their parents. The project addressed these issues by strengthening the current government response capacity. The PANDA program shows promise as a mechanism to help fill an important support gap by addressing challenges faced by children in families affected by alcohol abuse. Guardianship is taking on increased relevance as a means to protect children affected by migration. ■ PANDA is an important innovation by the project given the serious service gap to address alcohol abuse in Moldova. It should be further supported by the government and donors, if necessary, in order to assess longer term psycho-social impacts and its potential for replication nationally. ■ Through some project raions have found the means to invest in APP foster care expansion, the government needs to find a more stable means to grow foster care as an alternative form of care for children outside of parental care. ■ Current efforts to strengthen guardianship regulations through the civil code associated with PL 140 will provide greater guidance in the procedures of guardianship protection, particularly as it applies to vulnerable children left by migrant parents. Question 5: Have prevention methods employed by the projects reduced risks of child/family separation? The foundation of the prevention system is support provided to families identified with vulnerable children. This comes in many forms: counseling such as in parenting education, links to schools for inclusive education, material and financial support, and services such as those offered at day care centers. Prevention is an overarching framework in the project ranging from early identification and response to risk through the National Practice Model to building on the strengths of children and families to better withstand shocks and challenges in their lives. The PANDA is designed to do this by strengthening psycho-social well-being of children impacted by alcohol abuse in the family. Mellow Parenting is prevention oriented based on the premise that parenting education leads to a better understanding of children and EVALUATION | 119 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project the support in their development. These are further discussed below and in Evaluation Questions 1, 2, and 4. Findings: a) Are households with children at risk of family separation stabilized and strengthened? Child Well-being The below tables present the demographic information of the children included in the P4EC Monitoring and Evaluation database for vulnerable/at-risk families. In order to construct a database that could be analyzed as part of this evaluation NORC included only cases that were opened between May 1, 2016 and July 31, 2016 which was after the monitoring and evaluation capacity building training conducted by NORC staff in April 2016. All children in this database were living at home, but were at risk of separation when their case file was opened. Furthermore, NORC restricted the database to only those cases that had between 5 months and 12 months of case monitoring. In our analysis, we disaggregate results for those that received roughly 5-7 months of case monitoring and those that received roughly 10-12 months since receiving different amounts of case monitoring may have led to different results. It is important to note that P4EC did not work directly with these vulnerable families and children; rather their focus was on improving the skills and capacity of social workers by training to increase their knowledge of social protection mechanisms for vulnerable families in the community, improve their use of case files and develop individualized assistance for families based on their circumstances and need. Thus improvement of child well-being and closure of case file is indicative of social workers’ effectiveness, but we lack previous information to compare results to a period prior to receiving training by the project. Table 4.5.1 shows the number of at risk cases included in our analysis by raion. Table 4.5.1: Number of At-risk Cases Included for Child Well-being Analysis by Gender Across Project Raions Region 5-7 month monitoring 10-12 month monitoring Male Female Total Male Female Unspecified Total Cahul 4 1 5 13 13 2 28 Calarasi 0 0 0 7 2 0 9 Causeni 0 0 0 3 2 0 5 Falesti 1 3 4 6 8 0 14 Nisporeni 1 0 1 1 0 0 1 Orhei 7 7 14 18 9 0 27 Singerei 0 0 0 0 1 0 1 Soroca 4 0 4 1 0 0 1 Telenesti 13 9 22 2 1 0 3 EVALUATION | 120 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Region 5-7 month monitoring 10-12 month monitoring Male Female Total Male Female Unspecified Total Ungheni 14 12 26 6 1 0 7 Total 44 32 76 57 37 2 96 Source: P4EC Monitoring and Evaluation database Table 4.5.2 shows the average age of children across raions and by cohorts. Table 4.5.2: Average Age of Children across Project Raions in Cases Included for Child Well-being Analysis Region 5-7 month monitoring 10-12 month monitoring Cahul 7.6 7.7 Calarasi 0 6.4 Causeni 0 12.6 Falesti 5.5 10.1 Nisporeni 12 10 Orhei 6.6 8.8 Singerei 0 5 Soroca 11.3 12 Telenesti 8.7 7.3 Ungheni 8.1 6.9 Source: P4EC Monitoring and Evaluation database Characteristics of vulnerable families at time of case opening: The tables below present the family and child indicators at the time that the child’s case file was opened. While these help understand the child’s circumstances at what we use as the baseline, these indicators were not monitored during the follow-up visits and thus we do not report any changes over time. We can see that the characteristics are about the same for both those monitored for 5-7 months and those monitored for 10-12 months. These children are in poor (unemployed) households, with a high percentage of female headed households, and some social problems – generally neglect. The difference seems to be in the level of abuse and violence, and it makes sense that the caseworkers would stay with these cases longer due to the risk for the children. For most families (54%), the children lived with both parents or with a single mother (38%-40%) as seen in Table 4.5.3. Table 4.5.3: Parents in Household at Time of Case Opening Status, % 5-7 month monitoring 10-12 month monitoring Both parents 54 54 Mother only 38 40 Father only 3 1 Temporarily without parental care 1 3 Without parental care 4 2 Source: P4EC Monitoring and Evaluation database EVALUATION | 121 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Over half of the families exhibited “distant, cold relations” or “tensions, conflict relations” at time of case opening, as seen in Table 4.5.4. Table 4.5.4: Psycho-Emotional Status of Family at Time of Case Opening Status, % 5-7 month monitoring 10-12 month monitoring Harmonious relations 37 41 Distant, cold relations 37 35 Tension, conflict relations 26 24 Source: P4EC Monitoring and Evaluation database Less than one third of families had a parent who was employed or registered with an employment agency. This number was as low as 5-8% for those who received 10-12 months of monitoring, as seen in Table 4.5.5. Table 4.5.5: Parental Employment of Family at Time of Case Opening Status, % 5-7 month monitoring 10-12 month monitoring Mother is employed or registered with employment agency 15 5 Father is employed or registered with employment agency 34 8 Source: P4EC Monitoring and Evaluation database The most common family problem was neglect (49-53%), as seen in Table 4.5.6. Across the board, those receiving 10-12 months of monitoring had higher rates of each problem, which is linked to their receiving longer case monitoring given the higher risk for children. Table 4.5.6: Family Problems at Time of Case Opening Problem, % 5-7 month monitoring 10-12 month monitoring Child abuse 5 22 Alcohol/substance abuse 26 38 Domestic violence 16 28 Neglect 49 53 Source: P4EC Monitoring and Evaluation database The most prevalent child support service provided to the children at time of case opening was psychological counseling (49%-68%). Other common responses were “child needs additional support,” “integration supported by community centers,” and “child receives medical treatment,” as shown in Table 4.5.7. Again, those receiving 10- 12 months of monitoring were more likely to receive child support services. EVALUATION | 122 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Table 4.5.7: Child Support Service Provided at Time of Case Opening Indicator, % 5-7 month monitoring 10-12 month monitoring Child has psychological counseling 49 68 Integration supported by community centers 28 47 Child needs additional support 32 56 Child with special needs is evaluated and IEP is carried out 6 44 Child receives medical treatment 26 12 Child has special diet or restrictions in activity 4 12 Source: P4EC Monitoring and Evaluation database Changes in child well-being indicators: The following section presents the evaluation of the different child well-being indicators from the P4EC M&E data. The different broad categories of indicators include health, safety, education, psycho-social, and material. All well-being indicators showed at least some improvement from time of opening to time of closing the case across all indicators. The most progress was achieved in the health and safety where all the individual indicators had at least 10 percentage point increases. The education indicators were mixed with very high increases in provision of an Individual Education Plan for children with special needs – 33 points for those monitored for 5-7 months and 47 points for those monitored for 10-12 months. Additionally “child is included in the educational system” increased by 7 percentage points for those monitored for 10-12 months, “child systematically attends school” increased less than 10 percentage points for both groups of at-risk children – those monitored for 5-7 months and those monitored for 10-12 months. The psycho-social indicators exhibited the least progress, with increases of less than 10 percentage points across both groups. Health indicators: Table 4.5.8 below presents the results for the health indicators. The four health indicators evaluated reflected whether or not children were registered in health institutions, how frequently the child had regular checks, whether or not the child understood self-care, and how often the child had three meals a day. Focusing first on those monitored for 5-7 months, each of the indicators improved from baseline to endline. The biggest improvement was made in percent of children who reportedly “always” had regular health checks. The average increased from 21% of children at baseline to 47% at endline. Another notable improvement was a 10 percentage point increase in the percent of children who understand self-care, moving from 87% at baseline to 97% at endline. Similar improvements were made by those monitored for 10-12 months, with a 10 percentage point increase in the percent of children who know and understand self-care. The percent of children who always had regular health checks dropped from 39% at baseline to 33% at endline, which is of concern. However, under the same indicator the percent of children who “never” had regular checks remained at 0% and those who “sometimes” had regular checks dropped by 27 EVALUATION | 123 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project percentage points, leaving the majority of children getting regular checks “most of the time.” Also of note is that 100% of children in both cohorts were reportedly registered with a health institution at endline. Table 4.5.8: Health Indicators at Baseline and Endline Indicator 5-7 month monitoring 10-12 month monitoring Baseline, % Endline, % Change Baseline, % Endline, % Change Child is registered with health institution 95 100 5 98 100 2 Child with health issues has regular checks Always 21 47 26 39 33 -6 Most of the time 33 13 -20 17 50 33 Sometimes 42 40 -2 44 17 -27 Never 4 0 -4 0 0 0 Child knows and understands self-care 87 97 10 81 91 10 Child has three meals a day Always 37 48 11 44 75 31 Most of the time 45 49 4 36 22 -14 Sometimes 14 3 -11 20 3 -17 Never 3 0 -3 0 0 0 Source: P4EC Monitoring and Evaluation database Safety indicators: Table 4.5.9 shows the results for the safety indicators. The indicators included in this evaluation include to what degree the child feels loved and safe at home, to what degree the child feels safe at school, and to what degree the child feels safe in the community. Those monitored for 5-7 months showed improvement for children who “always” feel safe at home, school, and in the community. Those monitored for 10-12 months saw similar improvements for children reportedly feeling “always” safe at home, school, and in the community. Table 4.5.9: Safety Indicators at Baseline and Endline Indicator 5-7 month monitoring 10-12 month monitoring Baseline, % Endline, % Change Baseline, % Endline, % Change Child feels loved and safe at home Always 34 46 12 23 51 28 Most of the time 52 47 -5 42 38 -5 Sometimes 13 7 -7 34 12 -22 Never 0 0 0 1 0 -1 Child feels safe at school Always 51 79 28 54 70 16 Most of the time 47 17 -30 41 29 -12 Sometimes 2 2 0 6 2 -4 Never 0 2 2 0 0 0 EVALUATION | 124 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Indicator 5-7 month monitoring 10-12 month monitoring Baseline, % Endline, % Change Baseline, % Endline, % Change Child feels safe in the community Always 43 57 13 54 58 5 Most of the time 38 30 -8 22 24 2 Sometimes 3 0 -3 4 1 -3 Never 0 1 1 2 0 -2 Attended (small child) 16 12 -4 18 16 -2 Source: P4EC Monitoring and Evaluation database Education indicators: Table 4.5.10 presents the results for the education indicators in the P4EC M&E database. The indicators included in this analysis were percent of children who are included in the educational system, percent of children who systematically attend school, and percent of children with special needs who are supported by an Individual Education Plan (IEP). Both cases made progress on all three indicators between baseline and endline. The most progress made by the cases was in the percent of children with special needs who were supported by an IEP. Those monitored for 5-7 months reported a 33 percentage point increase between baseline and endline with 100% of children with special needs being supported by an IEP. Those monitored for 10-12 months made even more progress with an increase of 47 percentage points of special needs kids who are supported by an IEP. Table 4.5.10: Education Indicators at Baseline and Endline Indicator 5-7 month monitoring 10-12 month monitoring Baseline, % Endline, % Change Baseline, % Endline, % Change Child is included in the educational system 77 81 4 70 86 17 Child systematically attends school 85 91 6 84 93 9 Child with special needs is supported by IEP 67 100 33 33 80 47 Source: P4EC Monitoring and Evaluation database Psycho-social well-being indicators: Table 4.5.11 shows the results for the psycho￾social well-being indicators for children in vulnerable families. The two indicators evaluated were whether a child can discuss emotions with an adult (and who that adult is), as well as whether or not the child displays challenging behavior. The percent of children who cannot talk to any adult about their emotions dropped by 6 percentage points for those monitored for 5-7 months and 3 percentage points for those monitored for 10-12 months. Each cohort also saw an increase of 8-9 percentage points for children who said they could talk to a family member about emotions. The percent of children who “never” exhibit challenging behavior increased for both groups, with those monitored for 10-12 months seeing a 9 percentage point increase. The percent of children who “often” display challenging behavior also dropped for both groups. EVALUATION | 125 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Table 4.5.11: Psychosocial Well-being Indicators at Baseline and Endline Indicator 5-7 month monitoring 10-12 month monitoring Baseline, % Endline, % Change Baseline, % Endline, % Change Child can discuss emotions with adult Family Member 84 93 9 81 89 8 Family Friend 0 0 0 8 1 -7 Non-related Adult 9 6 -3 2 5 2 No 7 1 -6 8 6 -3 Child displays challenging behavior Never 82 83 2 74 84 9 Seldom 13 15 3 17 16 -1 Often 6 1 -4 9 0 -9 Source: P4EC Monitoring and Evaluation database Material indicators: Table 4.5.12 presents the results for the material indicators collected through the P4EC M&E database. The two indicators measured was whether or not the child had access to a safe place to sleep and whether or not the home was heated during the winter. Both indicators increased substantially for both cohorts. Those monitored for 5-7 months reported a 36 percentage points increase in children who always had a safe place to sleep and a 24 percentage points increase in children who always had a heated home in the winter. Those monitored for 10-12 months saw a 25 percentage points increase in the children who always had a safe place to sleep and a 15 percentage points increase in the percent of children who always had a heated home in the winter. Table 4.5.12: Material Indicators at Baseline and Endline Indicator 5-7 month monitoring 10-12 month monitoring Baseline, % Endline, % Change Baseline, % Endline, % Change Child has a safe place to sleep Yes 53 88 36 41 66 25 Partially 38 12 -26 42 31 -10 No 9 0 -9 18 3 -15 Home is heated during winter Yes 66 89 24 67 81 15 Partially 30 11 -20 24 16 -8 No 4 0 -4 9 3 -6 Source: P4EC Monitoring and Evaluation database Perceived decrease in the number of children needing alternative care: In contrast to the baseline study where stakeholders were more likely to feel that the number of children without parental care was increasing, the more dominant perception at endline was that the number of children without parental care was stable or decreasing. One focus group with social worker specialists noted that this decrease was a result of prevention at the family level and stronger legislation: EVALUATION | 126 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Female 8: “This separation [between children and parents] is decreasing. Analyzing 2016-2017 the number of children who got separated from their families has decreased considerably, and this fact is caused by prevention in the family. Many times the reasons for separation were alcohol, lack of jobs, and the need to leave abroad to get a job, but also the fact that the family did not know anything about child development stages. The young mothers did not know how to care about their children and how to supervise them, thus many cases would end up in health services, and we had to get the children out of the family. Now, there is collaboration with the health sector, we identify the persons with separated children, we collaborate with mothers to get them acquire child care skills, and we do our best in this direction.” Female 9: “Especially, taking into account that we have the PL 140 informing the parents that if they leave abroad they are entitled to establish the guardianship for the child and even though the child is separated from the family environment, nevertheless the child has a guardian and is protected, and I don’t know if we can call this separation in an institution or other services, because many of the relatives take over the child, and this is good. The law has its advantages too.” (Social work specialists) Caregiver Challenges and Needs Financial challenges continue to be a central issue for vulnerable families. As mentioned in answering Evaluation Question 4, poverty is one of the main reasons for family separation. The variety of financial challenges affecting vulnerable families was a common topic of discussion amongst focus group respondents. The most commonly mentioned issue was that parents lacked the necessities needed to support child well￾being, including housing, adequate plumbing, food, and clothing. There were a number of other specific issues that discussed managing the costs associated with raising children. The amount of money given to families to support their children is reportedly very small and not sufficient to cover the necessities of the child. Parents at risk of separation mentioned that this is an especially large challenge amongst families with disabled children, who feel a financial strain when making arrangements for their children. Many parents and those providing social services felt it was a priority to provide financial and material support for basic needs, such as providing housing or paying for school fees and clothes. Organization of events that provide material resources but also promote community involvement and support was considered to be an important action by community social assistants, mayors, and social work specialists. Some foster parents and mayors said that the lack of activities in the community encourages people to drink and at times use their child’s benefits to buy alcohol. EVALUATION | 127 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Vulnerable families continue to face challenges in securing jobs in Moldova. According to baseline and endline focus group respondents of all types, the lack of jobs in Moldova is a key issue for vulnerable families. One parent of a child at risk of separation went into detail about some of the obstacles that parents face in obtaining legal employment: “If someone wants to get employed legally, they have to be vaccinated. And the Ministry of Health must vaccinate that person. We don’t want to be vaccinated for our own reasons and we can’t get a certificate, and my husband cannot get a job legally and he works as a daily worker. Not having a job is an obstacle to supporting our children.” (Parent at risk of separation, “old” raion) Some community social assistants commented that parents at times do not have the skills to get jobs, as they were also raised in an RCC that did not focus on their development. Foster parents and parents at risk of separation mentioned that the search for employment and related lack of money means that both father and mothers cannot spend as much time with their children, and their relationship becomes more stressed. Criticism of a system that generates dependence on social services and parents who use funds for purposes other than child care persists while others feel that families have made steps to improve their resource management and accountability. Even more than at the baseline, there were several community social assistants, mayors, and multidisciplinary team members at the endline who felt that there was too much dependence on social assistance. They believed it can lead to greater dependency on the government to provide for their families instead of a growth in independence. At times, this overreliance was connected to parental mismanagement of money meant for children, spending it on their own needs and wants. A mayor gave an example of a woman who declined to look for work because she would lose her benefits that she received from caring for 4 children and her disabled sister if she were to return to work. Some gatekeeping commission members from Orhei, Calarasi, and Nisporeni felt that families should not receive aid without something in return – for example they should be required to provide community service work and those families that do receive social assistance should be closely monitored to make sure it is being used as intended. On the other hand, some multidisciplinary team members, gatekeeping commission members, and parents feel that vulnerable families have made lifestyle changes to better manage their families. They feel that vulnerable families now have greater feelings of accountability, use the resources they have more effectively and prudently, have fewer children, and work for their social benefits. P4EC is implementing a complementary Mellow Parenting program with other funding. Stakeholders in the project place great importance on parenting education and the need EVALUATION | 128 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project for stronger mechanism and skills to assist vulnerable families and children. Mellow Parenting (MP) is a relationship-based early intervention focused on improving parent￾child relationships. The methodology was piloted in Scotland in 1996 and has spread with many different applications for mothers, fathers, expectant parents and different age groups of children. The Scottish organization began working in partnership with P4EC in 2016 to prevent babies at risk of being abandoned and improving support for vulnerable parents to better meet the needs of children. These are parents and children dealing with mental health, substance abuse, child protection, and learning disability issues. Reflective and practical techniques are used for parents to address their personal challenges and those they face with their children. Fun activities between parents and children are facilitated. Videos of parent-child interaction are used for strengths-based feedback on positive parent child engagement. The model does not aim to directly solve participant problems, but enable collaboration between group members. Positive relationship models are demonstrated at various program levels. Evidence in Scotland shows increases in parenting confidence and positive behavior with children, decreases in the negative behavior of their children and a reduction in the intensity of interpersonal friction resulting from challenging behavior. Though not funded through DCOF/USAID, MP is strongly complementary to prevention and family support priorities of the Strengthening Families for Children project. Since 2016, MP has been piloted in five raions (Cahul, Falesti, Ungheni, Calarasi and Nisporeni) and Chisinau since 2016. So far, seven MP babies’ and toddlers’ groups have been established and evaluation data is currently being assessed. P4EC leadership and raion specialists indicate the early signs of this program are promising. Parents in the PANDA program learn about MP through targeted services offered to them. Community social assistants report that MP and PANDA are very useful because they help parents become aware of their problems and help them learn how to build relationships with and understand their children. The purpose of MP is to educate parents and to help them develop skills. Community social assistants have seen changes in behavior as a result of this program, but mention that the program can only be implemented in villages that are not too far away from Ungheni. b) Were the relevant families chosen for inclusion in the project? P4ECs project did not directly work with families. The evaluation team did not hear from any stakeholders that certain populations were being left out, nor was there any confusion about who can access financial support, which is usually a point of concern. EVALUATION | 129 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project c) Are children at risk of losing family care continuing to live in appropriate, permanent and protective family care due to improved national policies and local child welfare human resource capabilities and service delivery? See Evaluation Question 1 related to national policies and local child welfare human resource capabilities. Below we provide evidence showing that services have become more available and relevant to the needs of families and children. These were important project emphases through multi-service approaches and likely contributed to these improvements. Services have become more available. Improvement in the availability of services was documented across almost all services in the evaluation survey. This is highlighted by the drop in the percentage of community social assistants and social work specialist respondents who report that these services are needed but not available as shown in Tables 4.5.13 and 4.5.14 below. This was supported by focus groups. At baseline, all focus groups with parents spoke about how challenging it is to actually receive social assistance. At endline, there were some that spoke to this but it was much less often. The biggest improvements in service availability at the raion level were made in substance use/abuse services, job skills training programs for adults, continuing education for adults, and juvenile delinquency programs, each with an increase of at least 25 percentage points in availability from baseline to endline. At the community level, access to parenting support services rose by 25 percentage points. Other services that showed increases in availability at the community level are financial assistance – nongovernmental, parenting support services, mental health counseling, substance use/abuse services, employment/income generation services, job skills training programs for adults, continuing education for adults, and daycare services. It is important to note that social work specialists were asked about service availability at the raion level and community social assistants were asked about service availability at the community level. Social assistance, case management, housing assistance services, public schooling, and kindergartens are still amongst the most available services to support vulnerable families. Endline respondents still expressed high needs for juvenile delinquency prevention programs at the raion level and services for children with disabilities at the raion and community levels. EVALUATION | 130 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Table 4.5.13: Availability and Need of Services to Support Vulnerable Families – Raion Level Services to support vulnerable families Baseline, % Endline, % Available Strongly needed but not available Available Strongly needed but not available Financial assistance – government cash assistance / other social assistance program 100 19 97 0 Financial assistance – nongovernment 78 19 90 0 Case management for families at-risk 100 15 100 0 Parenting support services 89 30 100 6 Other family support 36 37 28 0 Mental health counseling 61 30 72 17 Substance use/abuse services 33 48 66 17 Housing assistance services 94 15 97 0 Employment / income generation services 78 19 76 0 Public schooling for children 89 15 97 0 Job skills training programs for adults 47 22 72 0 Continuing education for adults 39 30 69 0 Daycare services 94 26 100 6 Kindergartens 92 15 97 0 Crèche 64 26 79 11 Respite care for children with disabilities 81 30 90 28 Special services for children with disabilities 86 30 90 22 Juvenile delinquency prevention programs 44 48 76 39 N 36 27 29 18 Source: NORC evaluation survey Table 4.5.14: Availability and Need of Services to Support Vulnerable Families – Community Level Services to support vulnerable families Baseline, % Endline, % Available Strongly needed but not available Available Strongly needed but not available Financial assistance – government cash assistance / other social assistance program 99 15 99 2 Financial assistance – nongovernment 33 45 51 5 Case management for families at-risk 95 22 98 2 Parenting support services 43 38 68 14 Other family support 16 17 13 4 Mental health counseling 40 45 56 10 Substance use/abuse services 37 43 56 14 Housing assistance services 92 18 96 7 Employment / income generation services 33 38 48 6 Public schooling for children 91 16 93 2 Job skills training programs for adults 12 41 28 2 Continuing education for adults 13 41 29 6 EVALUATION | 131 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Services to support vulnerable families Baseline, % Endline, % Available Strongly needed but not available Available Strongly needed but not available Daycare services 25 60 44 19 Kindergartens 92 14 98 0 Crèche 32 24 46 6 Respite care for children with disabilities 34 45 45 22 Special services for children with disabilities 31 51 40 28 Juvenile delinquency prevention programs 30 45 41 6 N 264 191 254 125 Source: NORC evaluation survey There was improvement in the adequacy of services to meet the needs of family at the raion level, with particular improvement seen in adequacy of services to guardians. Respondents to the evaluation survey were asked “How adequate do you believe the services generally are to meet the needs of families to help ensure that children can stay in or be returned to family care?” At baseline about a third of the raion level social work specialists responded “fully adequate” while about two thirds responded “somewhat adequate.” This reversed at endline, with two thirds of the raion level social workers reporting that services were “fully adequate” to meet the needs of families and one third responding that services were “somewhat adequate.” Community level responses remained the same from baseline to endline with roughly one third of community level respondents reporting that childcare and protection services were “fully adequate” and two thirds reporting that they were “somewhat adequate.” The evaluation survey also asked respondents to rate the adequacy of existing services to meet the needs of special populations of families in their raion/community. Graphs 4.5.1 and 4.5.2 below show the respondent’s views on the adequacy of existing services to meet the needs of each special population. Respondents primarily rates services as “somewhat adequate” for each population type at both baseline and endline, except in the case of kinship caregivers at endline, where the majority of respondents rated services as “fully adequate.” EVALUATION | 132 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Graph 4.5.1: Adequacy of Existing Services for Special Populations of Families - Baseline Source: NORC evaluation survey Graph 4.5.2: Adequacy of Existing Services for Special Populations of Families - Endline 35 45 17 46 38 23 22 32 37 13 58 49 51 49 58 71 58 57 54 39 5 3 26 4 3 5 14 7 6 17 0 10 20 30 40 50 60 70 80 Fully Somewhat Not at all 33 58 20 46 40 36 27 37 41 26 61 40 65 51 58 61 57 57 53 39 5 1 12 2 2 3 5 2 4 6 0 10 20 30 40 50 60 70 80 Fully Somewhat Not at all EVALUATION | 133 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Source: NORC Evaluation Survey Family access to services rose significantly between baseline and endline. When asked “Are families in need generally able to access the services in your raion/community provided to help ensure that children can stay in or be returned to family care?” a large majority of community social assistants and social work specialists felt that services were accessible to most families. At baseline 80% of respondents reported that services were available to most or all families, which rose to 94% at endline. At baseline, respondents who said that only some families or no families had access to the services they needed reported that the least accessible services were mental health counseling and daycare services with 62% of respondents saying mental health counseling was not generally accessible and 56% of respondents saying daycare services were not generally accessible. At endline 25% of respondents reported that case management for families at-risk was not accessible to families, and 19% reported that government cash assistance and job skills training programs for adults were not available. When asked what the reasons were that services were not accessible, 81% of endline respondents reported that it was the distance between the families and the services. This was a shift from baseline findings which indicated that the top two reasons for services not being accessible were: (1) the service does not exist in the families’ community (69%) and (2) due to physical disability the family cannot get to the service (66%). Respondents were also asked to rate the level of access to services by special populations of families, as those being accessible to all families, most families, some families and not accessible to any families. Graphs 4.5.3 below show that according to respondents’ perception of those services that are “accessible to all families,” access to services has increased significantly for all groups – a range between a 9 percentage points increase for single mothers to a 23 percentage points increase for children from ethnic minority groups. That being said, children from ethnic minority groups, families with substance use/abuse issues, and teenage parents face more challenges compared to other family groups, a trend at both baseline and endline. EVALUATION | 134 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Graph 4.5.3: Services Accessible to All Families Source: NORC evaluation survey Support to children has increased in availability and effectiveness. At baseline, support services for disabilities and early childhood were somewhat lower with 72% stating they provide services to address cognitive/learning disabilities, 68% for referrals to early childhood support, 61% for mental health issues, and 52% to address visual/auditory disabilities. These numbers rose at endline, although these populations continue to be underserved. Similarly the percent of respondents reporting that the support was “effective” rose from 63% at baseline to 73% at endline. Availability of support to the whole family unit has improved although there has not been a change in effectiveness. The most impressive improvement in services provided to caregivers was in social/psycho-social services, referrals to alcohol or substance abuse treatment, and parenting education which improved by 14, 17, and 18 percentage points respectively. When asked “to what extent do you believe the support provided to caregivers by service providers improves their caregiving?” there was not much change between baseline and endline. Slightly less than half responded “greatly improves” and slightly more than half responded “somewhat improves” at both baseline and endline. Financial assistance and case management viewed as the most effective support services although social workers also felt they could be improved. Survey respondents identified the top three family support services that are most effective in improving a families’ ability to care for their children; those that need improvement in terms of 56 52 27 56 51 38 31 44 54 20 65 72 53 72 68 58 55 57 68 43 0 10 20 30 40 50 60 70 80 Single mothers Kinship care givers – guardians Families with substance use/abuse issues Families with children with disabilities Families living in poverty Families at risk of living in poverty Teenage parents Children left behind by migratin parents Families with children with disabilities or special education Children from ethnic minorities groups Baseline Endline EVALUATION | 135 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project availability, quality, effectiveness; and those that generally need to be improved. Table 4.5.15 shows only the highest three data points for each column. Support services that are considered the most effective in improving a families’ ability to care for their children include cash benefits or other financial assistance and family case management at both baseline and endline with almost all social workers in agreement by endline. Looking at these services, there were cases where they also were ranked among the highest three data points for additional improvement. For cash benefits or other financial assistance, nearly one half of respondents at endline feel it needs some form of improvement in quality and effectiveness and about a third of respondents felt that family case management services needs improvement in terms of effectiveness. Social/psychological services for people with mental health challenges need improvement. Survey results in Table 4.5.15 also show that social/psychological services for people with mental health challenges were viewed as needing improvement in availability, quality, and effectiveness by about a third of social workers. Some parents, children and social work specialists supported this view in focus groups by saying that having psychologists and counselors available to support parents was an important prevention service to provide. Specialist health care was also viewed as needing improvement in availability and quality by a little over a third of respondents. Alcohol and substance abuse treatment seems to need improvement in availability (although in a different question regarding availability, only 14% of respondents felt it was a strongly needed but unavailable service as seen in Table 4.5.14 above). Table 4.5.15: Effectiveness of Support Services to Families Support services to families In your opinion what are the top three support services to families that: (% responding yes) Are most effective in improving a families’ ability to care for their children (Q35) Need improvement in terms of availability (Q37) Need improvement in terms of quality (Q38) Need improvement in terms of effectiveness (Q39) Can be improved Baseline Cash benefits or other financial assistance 64 26 26 38 -- Family case management 43 -- 13 -- -- Referral to services 16 -- -- -- -- Social/psychological services for people with mental health -- -- -- -- 76 Specialist health care -- -- -- 13 76 Disability services for children -- 15 -- 11 82 Alcohol and substance abuse treatment -- 15 17 11 83 Parenting education -- 17 -- -- -- EVALUATION | 136 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Support services to families In your opinion what are the top three support services to families that: (% responding yes) Are most effective in improving a families’ ability to care for their children (Q35) Need improvement in terms of availability (Q37) Need improvement in terms of quality (Q38) Need improvement in terms of effectiveness (Q39) Can be improved Endline Cash benefits or other financial assistance 87 -- 49 53 -- Family case management 84 -- -- 35 -- Counseling 26 -- -- -- -- Social/psychological services for people with mental health -- 42 34 34 -- Specialist health care -- 45 42 -- 92 Disability services for children -- -- -- -- 92 Alcohol and substance abuse treatment -- 41 -- -- 93 Parenting education -- -- -- -- -- Source: NORC Evaluation Survey Parenting training was viewed as critical to help prevent separation of families. There was agreement across child care and protection providers that parental skills training to educate parents on how to care for their children was a very important prevention service. Many parents and those providing social services felt that parenting training through the MP program was key to separation prevention, as it gave parents important information, training, and awareness of their roles and responsibilities. They also said that it promotes resilience, builds capacity, and enhances parenting abilities. One gatekeeping commission member spoke about the importance of the school in her eyes in providing guidance to untested parents: “Yes, there should be parents’ school, parents’ “pedagogic training”. Sometimes parents come to school, they come for an advice; they don’t know how to act with their child. Very frequently, parents visit me asking for advice. Because when they used to be children they also didn’t know, there was nobody to advise them and they really don’t know how to act with their child in specific situations. So, there needs to be a parents’ school. I agree with what Mr. X (doctor) said: they shouldn’t be allowed to get married before they attend parents’ school. Sort of “pedagogic training” (Gatekeeping commission member, “old” raion) Some focus groups with parents and those from the child care and protection system felt that parents should also receive counseling services. This is also highlighted by survey results that show that social workers think counseling is one of the more effective support services for families. EVALUATION | 137 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project There was not much change in community social assistants’ and social work specialists’ self-assessment of effectiveness in family strengthening and stabilization. On average, the majority of respondents (at least 55% at baseline and 60% at endline), felt that all of the activities undertaken by social workers were “very effective” in strengthening and stabilizing families. Both baseline and endline respondents reported that among these activities, assessments and in-home monitoring visits were the most effective. 79% and 78% of baseline respondents, respectively, said that these services were “very effective” and 82% and 85% of endline respondents said these services were “very effective,” respectively. While the levels of effectiveness varied among the different services provided, it is important to note that the highest response rate for a service being “not useful at all” was only 1% at baseline and at endline. This shows that social workers believe that their work is effective in family strengthening and stabilization. Laws holding parents accountable for providing adequate care need to be better enforced. Several gatekeeping commission members, mayors, and social work specialists felt that parents should be held more accountable. This was understood by social service providers to be enforcing the laws that require parents to ensure certain standards of education, care, and supervision for their children. Failure to meet these standards can result in parents being taken to court and fined. However, one gatekeeping commission member reported that at times, judges that are presented with cases of parents failing to adequately care for their children refrain from punishing the parents, particularly in cases where the children ran away from home. This member said, “Judges should be trained to understand that children don’t run away because of good conditions, they run away because of bad conditions (Gatekeeping commission member, “old” raion). Other gatekeeping commission members felt that the child protection system was falling behind what it could be and that keeping families accountable was a great struggle. One gatekeeping commission member spoke about the need for institutional change to protect the children of vulnerable families: “I had an idea since there was the first Commission for Children in Difficulty, there are not enough mechanisms for making the family accountable. There are rather complicated cases, we try to come up with subsequent decisions at the implementation level and there are gaps. And hence, the family cannot be made accountable. I remember again the experience from Norway, if it is a case of psychological violence, if the parents shout at the child, the child is evaluated periodically and is taken out of the family, and subsequently the family undertakes correction measures, it is assessed and verified. We have rather serious cases in families, but we cannot find specific mechanisms for rendering them accountable.” (Gatekeeping commission member, “new” raion) EVALUATION | 138 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Some children, parents, mayors, and gatekeeping commission members also mentioned the need for more frequent monitoring and accountability of vulnerable families by local public officials. Conclusions: ■ Child wellbeing for children in vulnerable families who were monitored from 9 to 12 months between 2016 and 2017 showed some significant improvement in some areas. The most progress was achieved in health, safety, and education indicators for those with special needs. ■ Overall social services have become more available and accessible; family support services have become more available, but there was no change in their effectiveness; support to children has increased in availability and effectiveness. ■ There was improvement in the perceived adequacy of services to meet the needs of family at the raion level, in particular improvement was perceived for services to guardians. At the community level gaps in services exist due to the distance between the families and the services. ■ Financial challenges and difficulties in securing a job in Moldova continue to be central issues for vulnerable families. While availability of financial assistance has increased and it is viewed as one of the most effective support services, social workers also felt that the accessibility and effectiveness could be further improved. There is limited budget for this, as discussed in Evaluation Question 1. ■ Case management was also viewed as one of the most effective support services although social workers also felt it could be improved to be more effective. This is discussed in more detail in Evaluation Question 1. ■ Social/psychological services for people with mental health challenges need improvement. ■ Specialist in health care was also viewed as needing improvement in availability and quality by a little over a third of respondents. There is a significant gap in alcohol and substance abuse treatment though the PANDA program is being embraced as a mechanism that can be particularly helpful for the psycho-social health of children in families where abuse is occurring (see Evaluation Question 4) ■ Parenting training was viewed as critical to help prevent separation of families due to education about parents’ roles and responsibilities ■ Many service providers believe laws holding parents accountable for providing adequate care need to be better enforced since the current system does not always result in negative consequences for parents who do not adequately care for their children. EVALUATION | 139 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project Question 6: Did the project offer models and approaches for expansion, adaptation, and/or replication? Findings: a) Were lessons learned widely discussed and disseminated during project implementation? The project approach prioritized and successfully utilized lessons learned and their dissemination for impact, as models were piloted and then more broadly implemented based on these experiences. Trainers from “old” and “new” raions were utilized in cross￾training processes bringing their experiences and lessons learned forward to new project areas. P4EC has been a long time actor in Moldova in alternative care – particularly foster care development and deinstitutionalization. The project took the learnings from these processes and applied them in the broader set of prevention activities of the project. This includes intersectoral, multidisciplinary approaches; training modalities; casework and management practice; and relationship building with local authorities to build trust in the technical assistance and approaches provided by P4EC. Several NGOs in Moldova feel they have been left out of discussions of emerging initiatives, most notably the NPM. This, they say, has a detrimental impact on their programs and planning, leaves an important gap on perspective without their input, and hampers collaboration among NGOs. P4EC points out the NPM was a project initiative that took a lot of time to conceptualize and pilot, and only closer to the end of the project did they have the information to consult with stakeholders outside the project sphere. Nevertheless, in June 2016 representatives leading Moldovan NGOs participated in four days of a NPM participatory training. In consultation with participants in this training, it was decided who would constitute the “named person” for different age groups of children in universal services of health, education and protection. b) Were any best practices or successful techniques institutionalized? Though the inclusive education process began long before the project, P4EC took lessons learned from earlier experiences and applied these effectively in the prevention and continuing deinstitutionalization process of the project. Prior to the project, the case management system utilized by the SAFPD, though thorough and well documented, did not necessarily lead caseworkers to care plans based on individually identified child and family risks and strengths. The case EVALUATION | 140 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project management process implemented through the project does this. Though there is a learning curve and the process can be time-consuming, the new system is based on proven best practices in other countries, adapted to the Moldovan context. Intersectoral training and capacity building occurs at national and subnational levels with project components. This helps to put the key child protection sectors working toward the same goals and objectives, and conforms to the best practice of multidisciplinary approaches for child protection results. Additionally, a cadre of trainers exist from within government stakeholders to the project. They can potentially be tapped to continue capacity building in the project areas and in other raions of Moldova. By the end of the project, momentum was building in the development of an early identification of risk system, involving key government stakeholders in social protection, health, education, law enforcement, and others. The NPM utilizes best practices in training leaders and specialists in these sectors in an early identification and coordinated multidisciplinary response, within a context of preventing risks from escalating to children separating from families, dropping out of school, and other social crises. The coordination and fine-tuning of a national practice system will take time, resources, and fine-tuning – but it is based on global best practices. The P4EC project model is an interplay between practice and informing the national normative framework of Moldova. This occurs through research on possible best practice models (often global, see below), adaptation to the context of Moldova, piloting and scaling. The process gives P4EC significant credibility, including input and influence into more effective laws and regulations. Examples from prior to and into the project period include: ■ The Strategy and Plan of Action on the Reform of the Residential Care System (July 2007-2012) ■ The National Program for Inclusive Education 2011- 2020 (public funding began in 2012) ■ Government Decision 350 (May 29, 2012) – this regulation introduces the minimum package of social services provided to deinstitutionalized children and young people including family support, specialized family placements, foster care, family-type homes, temporary placement, personal assistance, mobile teams and centers of social assistance for the family and child. ■ PL 140 On the Special Protection of Children at Risk and Children Separated from Their Parents (June 2013 entering into effect January 1, 2014) – This is the influential law that specifically identifies and mandates responsibilities of government stakeholders EVALUATION | 141 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project ■ GD 270 Guidelines on the Intersectoral Cooperation Mechanism for Identification, Evaluation, Referral, Assistance and Monitoring of Child Victims and Potential Victims of Violence, Neglect, Exploitation and Trafficking (April 2014) The project approach gives child protection advocates and practitioners in project areas greater capital in helping to ‘responsibilize’ key government stakeholders – such as mayors – to fulfil their legally mandated duties in child protection and family welfare. An important and noteworthy organizational practice of P4EC is facilitating a process of studying best practices and programs in other countries and then modeling and adapting them to the Moldovan context. PANDA, the program dealing with alcohol abuse, is modeled after the Swedish program Ersta Vandpunkten. According to P4EC staff, it is based on scientific research and results found through implementation of prevention programs. Coming to fruition over the project period is a melding of well-being indicators for children with family support methodology into a new case management tool for more uniform and clear-cut assessment and action. This tool, integrated into the project’s NPM, is adapted from the child well-being framework Getting It Right for Every Child (GIRFEC) used by the Government of Scotland, and a U.S. family strengthening framework from the Center for the Study of Social Policy. This NGO has worked for over 12 years to develop the strengthening families approach now used broadly in child and family serving systems in the U.S. The PANDA program innovation is a significant attempt to address alcohol abuse - one of the most challenging protection issues in Moldova and around the world. The roll out of PANDA has been particularly effective as the methodology has been met with considerable commitment from local government stakeholders who moderate and implement this mechanism. Early signs are that there are positive outcomes, at least anecdotally in psycho-social health of children. Conclusions: ■ The project model for deinstitutionalization should be considered and adapted to raions still needing to close residential care centers, with appropriate incentives to create political support in these pockets of resistance. Guidelines, procedures, training modules, skilled and passionate trainers, case management tools and systems developed and used in the project can all be applied to this process. ■ The emerging comprehensive strategy for reducing institutional care for children with significant special needs is informed by this project. Children with disabilities have been deinstitutionalized through the project with adaptive inclusive education being instrumental to this process. The project and its stakeholders have helped to identify EVALUATION | 142 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project and demonstrate what is required in foster care to more effectively support children with disabilities. This includes mechanisms for greater financial support and preparation of foster families. Community-based support linked to the project through training and other activities – such as day and respite care, personal assistant home care and mobile teams supporting families with disabilities – are also strongly relevant. Stakeholders report intersectoral collaboration at the community level between health, education, social assistance specialists and mayoral authorities has grown significantly through the project. This momentum will be required in Moldova’s next big deinstitutionalization phase involving children with significant disabilities. ■ The funding crunch at the local level brought about through de-centralization needs to be resolved for project models to have significant expansion and replication, let alone improve prospects to grow more specialized support such as in psycho￾pedagogy. If enacted, a national funding mechanism for a basic package of services can potentially address this need. There should be a major push for this from the donor community and Moldovan civil society. ■ Mechanisms for more efficient use of resources are broadly applicable for replication and adaptation across Moldova. Examples from the project include more efficient use of foster care resources to increase the number of caregivers and improve quality care, and demonstrable improvements in the well-being of children and families creating greater commitment of raion decision-makers to allocate funding. ■ Local authorities in project raions are supportive of the NPM as a means for early identification of risk and prevention. NPM piloting needs to be documented and refinements made to address concerns expressed by stakeholders such as time, human resources and cost – particularly in the health and education sectors – for this support to be applied nationally. Other NGOs need to be collaboratively brought into this process. A normative act to “responsibilize" the NPM will be required for it to have national application. ■ All of the PANDA moderators interviewed (from Ungheni, Falesti, Nisporeni, Orhei, and Calarasi) believe the program should be replicated across Moldova. Government and the donor community should continue to support its development and research on its effectiveness as a model for psycho-social support outcomes for children and their families, and for replication across Moldova and other countries. ■ Government stakeholders to the project report ABCs are a welcome addition to better understanding the needs of at risk children and helping in responding to their needs. ABCs are an innovative means for youth to become involved and advocate for children’s rights and needs in protection. ■ Government stakeholders in the project are supportive of ABCs to the point that several raions are committed to financing support to the board after the end of the EVALUATION | 143 Performance Evaluation of USAID/DCOF’s Children in Moldova are Cared for in Safe and Secure Families Project project. Child protection specialists and raion officials cite learning things about children in foster and guardianship care from ABCs that they did not previously know and it informs their casework. This indicates that other raions may have an interest in applying this child participation mechanism. ■ A community of expertise, committed to intersectoral collaboration, has been activated through the project. This comes through the project approach to vertical and horizontal capacity building and training. Stakeholders began to “speak the same language,” use the same points of reference, and assess children and families in more uniform ways, all leading to coordinated response. Many of the sectoral leaders and specialists are passionate about their work. They convey this through cross-training for replicating project priorities, such as from old to new raions, within raions and strengthening intrasectoral mechanisms such as gatekeeping commissions and multidisciplinary teams coordinated by mayor’s offices. This experience and zeal can potentially be applied nationally. EVALUATION | 144 CONSOLIDATED ANNEX | 145 __ DRG-LER CONSOLIDATED ANNEX Endline Report Moldova: Performance Evaluation of USAID/DCOF’s Children in Moldova Are Cared for in Safe and Secure Families Project TABLE OF CONTENTS ANNEX I: CONCEPT NOTE ........................................................................................... 146 ANNEX II: EVALUATION QUESTIONS AND INDICATORS............................................157 ANNEX III: EVALUATION METHODS AND LIMITATIONS ............................................ 165 ANNEX IV: DATA COLLECTION INSTRUMENTS ......................................................... 176 ANNEX V: SOURCES OF INFORMATION . ....................................................................247 ANNEX VI: CONFLICT OF INTEREST . ..........................................................................253 ANNEX I: CONCEPT NOTE | 146 ANNEX I: CONCEPT NOTE DRG Learning, Evaluation, and Research (DRG-LER) Activity REVISED CONCEPT NOTE TASKING N003 PERFORMANCE EVALUATION DESIGN OF USAID/DCOF’S CHILD CARE REFORM (MOLDOVA) AND FAMILY CARE FIRST (BURUNDI) PROJECT Contract No. GS-10F-0033M/AID-OAA-M-13-00013 Revised January, 2016 This publication was produced for review by the United States Agency for International Development. It was prepared by NORC at the University of Chicago. The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. __ DRG-LER ANNEX I: CONCEPT NOTE | 148 DRG LEARNING, EVALUATION, AND RESEARCH (DRG-LER) ACTIVITY TASKING N003: CONCEPT PAPER PERFORMANCE EVALUATION DESIGN OF USAID/DCOF’S CHILD CARE REFORM (MOLDOVA) AND FAMILY CARE FIRST (BURUNDI) PROJECT CONTRACTED UNDER GS-10F-0033M/AID-OAA-M-13-00013 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. ANNEX I: CONCEPT NOTE | 149 LIST OF ACRONYMS DCOF Displaced Children and Orphans Fund DRG-LER Democracy, Human Rights, and Governance Learning, Evaluation, and Research Activity ESOMAR European Society for Opinion and Marketing Research FGD Focus Group Discussions IRB Institutional Review Board IRC International Rescue Committee JSI John Snow Research & Training Institute KII Key Informant Interviews LA Local authorities NORC National Opinion Research Center P4EC Partnership for Every Child PMP Performance Management Plan RFA Request for Application USAID United States Agency for International Development ANNEX I: CONCEPT NOTE | 150 REVISED CONCEPT NOTE TASKING N003: PERFORMANCE EVALUATION DESIGN OF USAID/DCOF’S MOLDOVA AND BURUNDI PROJECTS As part of the DRG Learning, Evaluation, and Research (DRG-LER) Activity, USAID has requested NORC to design and budget for a performance evaluation of USAID’s Displaced Children and Orphans Fund (DCOF) projects in Moldova (Children in Moldova are Cared for in Safe and Secure Families), and Burundi (Family Care First: A Project to Ensure Children in Burundi are Placed in Protective and Permanent Family Care). DCOF’s overall goal is to measurably improve the safety, well-being, and development of highly vulnerable children, particularly those who are living without adequate family care. DCOF gives priority to projects that promise impact beyond direct services, which strengthen local capacity, and offer models and approaches for expansion, adaptation, and/or replication. While DCOF support in some cases is directed toward parents or other adults, the primary beneficiaries must be children younger than 18 years of age. John Snow Research & Training Institute (JSI), acting on behalf of USAID and DCOF, issued a Request for Application (RFA) through the Advancing Partners and Communities Cooperative Agreement. Three applications were selected for awards of $4.4M each to be implemented over 42-month periods. NORC will be evaluating two of the three awarded projects: Burundi, implemented by the International Rescue Committee (IRC), and Moldova, implemented by Partnership for Every Child (P4EC). The overall goals of these projects is to ensure children are in protective and permanent family care by reducing unnecessary separation of children from their families and by placing children who are outside of family care in nurturing families. The IRC-Burundi project will work with government and community-based stakeholders to ensure children under 18 years of age in Burundi are in protective and permanent family care, mainstream family-based child protection approaches at all levels of the government, and contribute to shifts in fundamental skills, social attitudes, and norms regarding child protection and welfare in 10 provinces. The P4EC-Moldova project will advance child care reform in Moldova, increasing the shift from institutional care to family-based care. The project will operate at national and local levels and will work intensively with 10 local authorities (LAs) in Moldova. EVALUATION QUESTIONS NORC’s approach to performance evaluation entails a mix of mutually reinforcing qualitative and quantitative methods that reflect the program logic, research questions ANNEX I: CONCEPT NOTE | 151 being addressed, and indicators. The qualitative analysis will provide local context and represent concrete examples that illustrate in greater detail the quantitative findings. Our approach to selecting the appropriate methodology is based on the USAID Evaluation Policy as well as our experience conducting evaluations in the field. NORC’s evaluation will address the following key questions identified by DCOF on January 8 2014. 1. Have reintegration methods employed by the projects resulted in stable and sustained placements for children? 2. Did the program measurably improve the safety, well-being, and development of highly vulnerable children, particularly those who are living without adequate family care? Did the program impact beyond direct services? 3. Have prevention methods employed by the projects reduced risks of child/family separation? 4. Did the program bring out systemic changes at the community, regional, and national levels that are enabling children to live in family care and preventing inappropriate placements in institutional care? 5. By project end, to what extent have functioning structures been established that can continue to provide on an ongoing basis adequate case-management services for children at risk? 6. Did the project offer models and approaches for expansion, adaptation, and/or replication? In consultation with DCOF and the implementing partners P4EC (Moldova) and IRC (Burundi) NORC also developed several sub-questions and indicators to answer the above questions. The table with key questions, sub-questions, and indicators is in Annex A of this revised concept note. EVALUATION DESIGN Moldova The evaluation employs a pre-post design, comparing key indicators and findings for project beneficiaries at baseline to those at endline. This approach allows us to understand changes over time but does not allow us to attribute these changes to the project intervention since it was not possible to include a comparison group. Concerns about confidentiality and sensitivity in contacting vulnerable children and their families directly, as well as the fact that children and families are indirectly beneficiaries of the P4EC project, led to a change in the target population for the evaluation survey. ANNEX I: CONCEPT NOTE | 152 NORC had originally planned to survey caregivers of children in vulnerable families, but after consultation with P4EC and agreement by USAID/DCOF, a questionnaire was designed for social workers. This change in study design required NORC to depend solely on P4EC to obtain information on the wellbeing of children (i) currently in RCCs and (ii) in vulnerable families, due to concerns about evaluators being in direct contact with this vulnerable population. P4EC built the capacity of and assisted social workers in obtaining child wellbeing information for both vulnerable children at home and vulnerable children currently in RCCs. The baseline evaluation therefore comprised of primary data collected by the NORC team in the form of key informant interviews (KIIs), focus group discussions (FGDs), and a survey of community social workers and social work specialists of the Social Assistance & Family Protection Directorate (SAFPD), who are the direct beneficiaries of the project; and secondary data collected by P4EC – child wellbeing indicators, a Knowledge, Attitude, and Practice (KAP) survey, and monitoring data of capacity building training of stakeholders. The sources of information for the baseline evaluation therefore included: ■ A total of 29 FGDs with children, parents, Mayors, social workers, multidisciplinary teams, and Gatekeeping Commissions. All FGDs were translated into English and coded into NVivo by NORC for analysis ■ 42 KIIs. Subject expert N. Beth Bradford analyzed and synthesized all the findings from the KIIs. ■ A survey of 296 community social workers, and 38 social work specialists. The cleaned data was inputted and analyzed in STATA. ■ Case management information including wellbeing data on 236 children in RCCs. ■ Results of a KAP survey conducted by P4EC ■ Information on capacity building training by P4EC: P4EC also shared M&E data on vulnerable children, but problems with the format and consistency of information in these files prevented us from using this important secondary data in the baseline. More details on this, and its implications going forward are provided below under the section Capacity Building of Implementing Partners in M&E. Burundi The baseline evaluation comprised of primary data collection by NORC in the form of KIIs, FGDs, and a survey of caregivers in families who are participating in the Village Savings and Loans Association (VSLA) and Family Support Groups (FSG) program components and thus direct beneficiaries of the project. Concerns about NORC approaching deinstitutionalized children as an evaluator also resulted in us relying on ANNEX I: CONCEPT NOTE | 153 IRC for all child wellbeing indicators for children currently in RCCs but being deinstitutionalized by the project. Other secondary data to be provided by IRC included information on capacity building / training of stakeholders. Qualitative data collection consisted of a series of 31 focus groups led by our subcontractor, CARD, and 21 key informant interviews conducted solely by Siân Long (Subject Area Expert), who also conducted field visits to the four provinces of Bujumbura Mairie, Gitegta, Ngozi, and Muramvya. All interviews and FGD notes were coded into NVivo by NORC for data analysis. The quantitative survey targeted a census of 250 households participating in the FSG. Due to a much larger population in Cohort 1 participating in the VSLA -- 3,918 participant households in 158 VSLAs located in 98 villages, the Evaluation team, IRC, and USAID agreed upon a sample size of 179 cases following a two-stage sampling methods. The sources of information for the baseline evaluation in Burundi therefore included the following: ■ A total of 31 FGDs with children, FSG and VSLA parents, caregivers of disabled children, child protection committees at the colline and communal level, social workers, and the provincial committee for the coordination of child protection. ■ 21 KIIs ■ Survey of VSLA and FSG caregivers ■ M&E data from IRC which included capacity building of stakeholders and child wellbeing indicators for deinstitutionalized children. NORC received highly aggregated M&E information from IRC with no details on how the wellbeing indicators were defined, protocols for data collection, or any assumptions made. The raw data files were not shared with NORC. CAPACITY BUILDING OF IMPLEMENTING PARTNERS IN M&E In this revised concept note NORC would like to propose a focus on capacity building of the IPs in M&E data collection rather than primary data collection for a midline evaluation by NORC. The reasons for this are: (1) poor quality of M&E information shared with NORC by the IPs which do not allow us to answer key evaluation questions; and (2) the limited changes we expect to see from another round of data collection within 9 months of the baseline evaluation. Suggestions for capacity building in both countries are provided below. Moldova The main problem with the M&E data collected by P4EC is that the data are collected only for reporting purposes and not for further analysis. Additionally, the issues ANNEX I: CONCEPT NOTE | 154 highlighted below do not enable ease in analysis even if attempted, or confidence in the reported results. NORC had to spend substantial time cleaning and making the data on children in RCCs consistent (in some cases deleting indicators that were coded inconsistently or outliers). The format of the data collected on children in vulnerable families and the large data size did not allow for merging or any analysis of this data at all. Key problems seen are: ■ Inconsistency in the way data are entered for the same variable across different districts. This makes it difficult to merge the file and analyze the whole dataset. ■ Missing information for many variables. ■ Answers inputted that are not among the available response options. ■ Multiple records for the same child. ■ Data entry for children that are too young to answer or have inapplicable responses for specific questions. ■ Separate line for a child, his/her sibling and parents which makes it impossible to link the child with the family. Inability to use the M&E data has implications on the ability of NORC to answer key evaluation questions for DCOF both for the baseline and subsequently for the endline. NORC therefore proposes to forego the midline evaluation and instead build the capacity of IPs in regularly collecting and using project M&E data. NORC proposes sending Mathew Fisher-Post, Senior Research Analyst to Moldova to work with P4EC. Matt will conduct an overall training of P4EC staff on Best Practices in M&E Data Collection and Use, followed by detailed one-on-one discussion and technical assistance with staff in charge of P4EC’s M&E data collection. Topics addressed by Matt will include: ■ Structure of excel file to collect M&E data ■ Definition of indicators collected ■ Protocol to be followed in collecting data ■ Data analysis for project decision making ■ Salvaging baseline data where possible on vulnerable children and families Following this capacity building trip, NORC will ensure that data shells are shared with NORC in advance of future data collection as well as data collection protocols. Burundi NORC received very limited M&E information from IRC and no raw data sets to examine how the aggregate percentages were calculated. Requests for the raw data led to a sharing of files that does not include information on all the indicators reported by ANNEX I: CONCEPT NOTE | 155 IRC and lack of supporting documents that explains how all the indicators were defined, and how they relate to the indicator list in the PMR. In the past NORC has faced difficulties in extracting information from IRC and only face-to-face interaction has proven to be most effective in getting program details and moving the agenda forward. Key problems seen are: ■ Limited documentation on how the raw data links to the PMP indicators. ■ Limited documentation on indicator definitions, possibly leading to inconsistencies in data entry. ■ Indications that indicators have been added, or their definitions modified, over time, meaning that indicators collected at baseline may not be comparable to indicators collected later in the process. ■ Not all PMP indicators were available in the raw data. ■ Not clear how to match up children from baseline to endline NORC proposes sending Russell Owen, Senior Research Analyst to Burundi to work with IRC. Russell is fluent in French and will conduct a training of IRC staff on Best Practices in M&E Data Collection and Use, followed by detailed one-on-one discussion and technical assistance with staff in charge of IRC’s M&E data collection. Topics addressed by Russell will be the same as those covered by Matt in Moldova, but specific to the project and challenges in Burundi around quality data collection which is more of an issues compared to Moldova. NORC will also try and ensure that all data shells and data collection protocols are shared by IRC prior to future data collection. ENDLINE EVALUATION NORC proposes replicating all the data collection at the baseline for the endline evaluation. In addition we will work with both IPs to identify any ruptured cases (deinstitutionalized children who are placed with families by the project, but then sent back to the RCCs) for the case study analysis. NORC will work with the same local data collection firm in both countries and will seek to hire a local coordinator in each country who will assist the team by arranging appointments during their scoping trip, providing administrative and logistical support during their visits to the field, and following up on outstanding issues after the Evaluation Team leaves. The local coordinators will also provide some technical support in the sense of working closely with P4EC and IRC to keep us updated on implementation of the project as it may affect our evaluation design, collecting secondary data from in-country sources, and doing some primary data collection for the case studies as needed, with the case study data collection tools developed by the NORC team. ANNEX I: CONCEPT NOTE | 156 Four trips are envisioned for the endline with one NORC evaluator and subject expert traveling to each country. While in country, the subject experts will conduct KIIs with the implementers, and other stakeholders. They will also assist the evaluators (Ritu and Mawadda) in the refresher training of the local data collection firm enumerators, and focus group facilitators, focusing on issues related to data collection protocols with vulnerable children. While in country, the evaluators (Ritu and Mawadda) will focus mainly on re-training the local data collection firm and the local coordinator in best practices of survey administration and focus group moderation in addition to training on the purpose of the data collection and the data collection instruments themselves. Training will last a few days and include role playing by the field staff. Even though the same data collection instruments will be used as for the baseline, the evaluators will observe the survey pilot and pilot focus groups and work with the team (technical specialists and local data collection firm) to provide feedback for the main field data They will train the local coordinator to continue oversight of the local data collection firm. They will also join the technical experts on a portion of the KIIs, particularly those with the implementers in order to better understand the program, and will assist with other KIIs once the data collection trainings and pilots are completed. For the endline NORC will explore doing model based analysis, but this is contingent on getting M&E data from the IPs that is of high quality and analyzable. EVALUATION TIMELINE Capacity Building NORC will target undertaking capacity building of the IP between January 2016 – March 2016. Once this revised concept note is approved we request that the IPs are informed by DCOF about the change in plans regarding the midline and the focus on capacity building. We will reach out to P4EC and IRC to schedule the trips to Moldova and Burundi1 and initiate preparation for the trips. Endline Evaluation NORC will initiate work on the endline evaluation for both Moldova and Burundi in fall 2016. Both projects are expected to end in June 2017 and the endline evaluation will be undertaken in the last 6 months of the projects. 1 Travel to Burundi will be subject to a decrease in violence and more stable conditions. ANNEX II: EVALUATION QUESTIONS AND INDICATORS | 157 ANNEX II: EVALUATION QUESTIONS AND INDICATORS ANNEX II: EVALUATION QUESTIONS AND INDICATORS | 158 Exhibit II.1: Evaluations Questions and Indicators Indicator Indicator Computation Variables Evaluation Question 1: Have reintegration methods employed by the projects resulted in stable and sustained family-based placements for children? (a) Are increased number of children living in residential care facilities being placed in family care?  % of targeted children in residential care placed in permanent family care, disaggregated by type of care and disabled/orphan children  # of targeted children in residential care  # of targeted children in residential care placed with biological families  # of targeted children in residential care placed with safe and stable alternative family￾based care (disaggregated by category) (b) During the tracing process, what percentage of children in residential care have been identified a birth family or kinship care option that is safe and appropriate?  % of targeted children in residential care out of all targeted children in residential care identified a birth family or kinship care option, (disaggregated by type of care and CWD/special needs children)  % of targeted children in residential care out of the total # of children in targeted residential care facilities(disaggregated by type of care and CWD/special needs children)  # of targeted children in residential care  Total # of children in targeted residential care facilities  # of targeted children in residential care that have birth family identified for placement (disaggregated by CWD/special needs children and non￾CWD/special needs children)  # of targeted children in residential care identified a kinship care option (disaggregated by CWD/special needs children and non￾CWD/special needs children) (c) What factors prevented placement of children in residential care into permanent family care?  Availability of family care options  Obstacles to deinstitutionalization  Children’s level of desire for reunification or deinstitutionalization  Process used to identify placement (d) What type of social service follow-up is provided to deinstitutionalized children, by whom and for how long?  Opinions on the adequacy of social service follow-up  # of families in which deinstitutionalized children are placed receiving cash transfers  # of families in which deinstitutionalized children are placed participating in VSLA+ HFC each of the types of social service follow-up  Level of social service workforce follow-up of reunited and deinstitutionalized children Evaluation Question 2: (a) Did the program measurably improve the safety, well-being, and development of highly vulnerable children, particularly those who are living without adequate family care? (b) Did the program impact beyond direct beneficiaries? ANNEX II: EVALUATION QUESTIONS AND INDICATORS | 159 Indicator Indicator Computation Variables (a) Did the project provide a core package of services to help ensure that residential care is prevented when possible and that reunified and deinstitutionalized children and at-risk children remain in family care?  % of targeted families that received cash transfer, participated in FSG, participated in VSLA  % of targeted children in at-risk families that remain in family care  % of targeted children in residential care who experience a ruptured placement  Reasons for ruptured placement  Was the relevant core package of services provided to achieve the objectives?  # of targeted children that received core package of services, by type of service  Total # of targeted children in at￾risk families  # of children in targeted at-risk families that remain in family care  of children in targeted at-risk families that are ruptured placements  (# of residential care children placed in permanent family care, measured above)  # of ruptured placements of deinstitutionalized children  Selection criteria of at-risk families (b) Are there fewer children living in residential care facilities?  Total # of children in residential care facilities in program-targeted areas  # of children admitted to residential care facilities in program-targeted areas (c) Is the wellbeing of deinstitutionalized children assessed as being adequate?  See separate wellbeing indicator document  Perspective of social workers and teachers of deinstitutionalized children’s’ wellbeing  Opinions of children on their wellbeing  Opinions of caregivers on children’s wellbeing. (d) Have there been other unanticipated positive or negative results of the program?  Unanticipated outcomes that emerge from qualitative inquiry from among direct beneficiaries  Perceived impact among indirect beneficiaries Evaluation Question 3: Have prevention methods employed by the projects reduced risks of child/family separation? (a) Are households with children at risk of family separation stabilized and strengthened?  (% of targeted children in at-risk families that remain in family care, measured above)  Wellbeing of targeted children in at￾risk families (see wellbeing indicator)  Opinions on support services received by families (b) Were the relevant families chosen for inclusion in the project?  Criteria used for selection of families with children at-risk of separation ANNEX II: EVALUATION QUESTIONS AND INDICATORS | 160 Indicator Indicator Computation Variables (c) Are children at risk of losing family care continuing to live in appropriate, permanent and protective family care due to improved national policies and local child welfare human resource capacities and service delivery?  Quality of changes in national policies  # of child welfare staff at national and local levels  Average number of active case load per social worker in target locations  Retention rates of child welfare staff  Training inputs and training curriculum Evaluation Question 4: (a) Did the program bring out systemic changes at the community, regional, and national levels that are enabling children to live in family care and (b) preventing inappropriate placements in institutional care? (a)Do professional and public attitudes show increased knowledge of and increased support of national policies that prevent unnecessary family-child separation; and promote appropriate family care for children without parental care?  Knowledge of national policies by gov. authorities, state, and non-state service providers, community members, and parents  Support for national policies by gov. authorities, state, and non-state service providers, community members, and parents (b)Is there an improvement in caregivers’ parenting skills and practices?  Quality of caregiver parenting skills and practices  % of parents/kinship caregivers with improved parental skills  Total # of caregivers targeted by the project to improve their parenting skills  #of caregivers with improved parental skills  Change in parenting assessment (doing tests of families on parenting pre and post training? ) (c) Is there any change in attitude towards residential care among parents, extended family and community members? Attitudes of parents, extended family and community members towards institutional care ANNEX II: EVALUATION QUESTIONS AND INDICATORS | 161 Indicator Indicator Computation Variables  Government regulations are in line with UN Alternative Care Guidelines  Government Inspection system is in line with UN Alternative Care Guidelines  Quality of inspection reports  # of residential care facilities that have been inspected over the last 3.5 years,  # of designated staff in government offices/department with mandate to oversee residential care facilities # of trainings conducted for staff and number of staff who attended the trainings  Total # of trained gov. authorities and state and non-state service providers  # of trained gov. authorities and state and non-state service providers demonstrating adequate knowledge  Pre-post training evaluations  Case file audits with CDFC Evaluation Question 5: By project end to what extent have functioning structures, workforce and services been established that can continue to provide on an ongoing basis adequate case management services for children at risk? (a)Do government authorities and state and non-state service providers have adequate attitude, knowledge and skills to build family resilience, involve, support, and protect children at the local level?  Perceived attitudes of gov. authorities, state, and non-state service providers  Perception of knowledge and skills of gov. authorities, state, and non-state service providers  % of trained gov. authorities and state and non-state service providers that demonstrate adequate knowledge  Total # of trained gov. authorities and state and non-state service providers  # of trained gov. authorities and state and non-state service providers demonstrating adequate knowledge  Pre-post training evaluations  Case file audits with CDFC (b)Have government authorities and state/non-state service providers adopted a joint approach to build family resilience, involve, support, and protect children at the local level? Approach used to build family resilience and involve, support, and protect children at the local level Evaluation Question 6: Did the project offer models and approaches for expansion, adaptation, and/or replication? (a)Were lessons learned widely discussed and disseminated during project implementation?  Mechanism for sharing lessons learned  The extent to which lessons learned were disseminated ANNEX II: EVALUATION QUESTIONS AND INDICATORS | 162 Indicator Indicator Computation Variables (b)Were any best practices or successful techniques institutionalized?  Perceived knowledge of new best practices among project partners, stakeholders, and beneficiaries (as evidenced through analysis to answer above evaluation questions)  Opinions of adoption of new best practices or techniques among project partners, stakeholders, and beneficiaries Exhibit II.2: Child Wellbeing Indicators No Wellbeing element Indicators 1. Materials: -shelter -clothing Shelter The child has personal space facilitated by: ■ Necessary conditions for sleeping; ■ Conditions for doing homework; ■ Conditions for play and toys. Accommodation (house, apartment) should be: ■ Heated during the cold season of the year, ■ Illuminated, especially in the zone where the child does his/her homework, ■ To be connected to electricity, gas (if the place is gasified), ■ To be provided access to drinking water. The dwelling should be facilitated by: ■ Space for meals, ■ Conditions for leisure time, ■ Conditions for personal hygiene, ■ Conditions for washing. The house is clean; there should be washing powder in the household. Personal hygiene items: The child has at his/her disposal the following items of personal hygiene: toothbrush, comb, a small towel for hands, a big towel for the body. He/she has access to the toothpaste, soap, shampoo, and scissors. Clothing: The child has clothing for each season of the year according to his/her age, thus for one-year time the child roughly should be provided with: ■ Winter time – thick outer clothing, a cap, a scarf, mittens, warm boots, pullover, a shirt/blouse, pants/a warm dress, tights and warm socks, thick nightgown/ pajamas; ■ Autumn (fall) and spring - waterproof jacket and footwear, a cap, a pullover, a shirt/a blouse, pants/autumns dress, tights and socks; ■ Summer – pants/ shorts/cotton dress, t-shirts, sandals, a summer cap; ■ Irrespective of the season the child should have the following clothing for school – a set of clothes for school, sports suits and footwear, underclothing, hander chief. ANNEX II: EVALUATION QUESTIONS AND INDICATORS | 163 No Wellbeing element Indicators 2. Psychosocial: -secure relationships -consistent engagement in age appropriate recreational, cultural and educational activities Does the child display aggressive, distressed or depressed behavior The child develops emotionally and socially according to his/her age. ■ The child has a sense of identity and belonging; ■ The child is confident; ■ The child has self-care skills; ■ The child expresses his/her emotions by socially accepted beahviour; ■ The child copes with traumatic events of loss and separation; ■ The child respects other children, he/she is not involved in persecution and discrimination of other children; ■ The child is not worried because of the fear towards somebody or something; ■ The child doesn’t manifest behavior of self-mutilation or suicide tentative. The child is in good relationships with parents/carers, siblings, other members who live together: ■ There is a strong attachment and affectivity on behalf of the parent/carer and other family members; ■ Has an adult person/persons (social support network) among family members, carers or friends with whom they can talk, share joy or express their anxiety, uneasiness; The child is involved in diverse activities in the family, social network, school and community, including cases of children with chronic disease or disability. The child understands and respects social norms and traditions applied in his/her environment; The parent ensures a balance between learning activities, recreation for the recovery of physical and mental potential to avoid extenuation. 3. Educational: - Access to school - Grades - Regular attendance Specific issues for children with disabilities The child is integrated in the educational process – pre-school institution, school (including at home studying), vocational school etc. according to his/her age, with children of his/her age; The child attend pre-school or school systematically; The child has passing grades. In case of poor school progress the child has access to teaching support. The child with special needs has a worked out individual educational plan, which is put in application; The child with special educational needs benefits from the services of resource centers within the school; The child is supported by the parents/carers in educational process for doing his/her homework, involvement in school, extra-curricular and community activities; Educational environment is inclusive, stimulating and friendly. ANNEX II: EVALUATION QUESTIONS AND INDICATORS | 164 No Wellbeing element Indicators 4. Safety: This looks at whether the child is protected from: - Violence (physical and verbal) - Abuse (physical and verbal) - Neglect in the home, school and community environment The child is not threatened or exposed to physical abuse or violence at home, at school or in community; The child is not exposed to emotional abuse at home, at school and in community; The child feels safe at home, at school and in community; The child is not physically neglected by the parent/carer (the child is fed, has shelter and clothing, is provided with hygiene and has access to medical and dental services); The child who cannot take care of himself/herself is not left unattended, is not left in the care of the other child or other persons who can’t provide safety to the child; The child is not neglected emotionally , is in permanent contact with the reference adult, who provides support, who the child trusts, is protected and guided by the parent/carer; The child is not exposed to alcohol/drug abuse on behalf of the family members or other persons in community; The child is not exposed to persecution or discrimination by other schoolmates or adults at schools, in community; The child is not exposed to sexual abuse or exploitation; The child doesn’t feel being pushed by other children or adults to make dangerous actions; The child is not exposed to or involved in anti-social or criminal activities in the community; The child knows, takes protection measures and acts responsibly in potential risk situations; The child is supported and manifests resilience in different hard circumstances at home; The child is informed about the forms and signs of abuse, neglect and exploitation; Parents/carers are informed about the forms and signs of abuse, neglect, exploitation and about applying non-violent forms of managing challenging behavior of the child; The child is not involved in manufacturing, transporting, purchasing and selling alcohol, cigarettes, toxic and narcotic substances; The child is not involved in economic activities and hard work that could harm his/her physical, mental and moral development of the child. 5 Health and nutrition: - Food intake - Sickness - Access to health care The child is registered with the family doctor (has health record, is vaccinated); The child is medical and dentally examined; The child’s physical development parameters (weight and height) correspond to his/her age; The child with health issues, chronic or long-term diseases, including disability makes regular checks, takes necessary treatment, respects a regime upon need, accesses necessary health and social services; The parent/carer knows and understands the child’s health needs; The child knows and understands the needs that refer to his/her own health and is trained to provide self-care in the limits of his/her potential; The child doesn’t smoke or drink alcohol, misuse drugs or other toxic substances; The child has a sexual behaviour appropriate to his/her age and development stage; The child is fed three times per day, provided at least with a warm meal per day in sufficient quantities, according to his/her age; In child’s nutrition there are quality and nutritional products, drinking water, which are appropriate to the child’s age and his/her individual needs; The child is involved in choosing and cooking meals depending on his/her age; The parent knows and understands nutritional needs of the child, including the child’s individual health needs; The child knows and understands nutritional needs and is trained to respect necessary regime; ANNEX III: EVALUATION METHODS AND LIMITATIONS | 165 ANNEX III: EVALUATION METHODS AND LIMITATIONS ANNEX III: EVALUATION METHODS AND LIMITATIONS | 166 To gather data required for this evaluation, NORC’s Evaluation Team used several techniques which entailed a mix of mutually reinforcing qualitative and quantitative methods that reflect the program design, research questions being addressed, and indicators. We combined the results of each technique to capture the diversity of opinions and perceptions of beneficiaries and stakeholders about key children/family care and protection issues at the start of the program. The qualitative analysis, which includes a document review, case file reviews, key informant interviews (KII) and focus group discussions (FGD), provides the local context and also represents concrete examples that illustrate in greater detail the quantitative findings. Our approach to selecting the appropriate methodology is based on the USAID Evaluation Policy as well as our experience conducting evaluations in the field. The NORC Evaluation Team conducted the evaluation in a participatory manner which involved engaging USAID/DCOF, implementing partner P4EC, program beneficiaries, and other stakeholders. A complete list of documents the Evaluation Team reviewed is included in Annex V, Sources of Information. EVALUATION MANAGEMENT The evaluation team for Moldova includes Ritu Nayyar-Stone (Project Director), Mawadda Damon Gartner (Evaluator), N. Beth Bradford (Subject Area Expert – Baseline), Gary Gamer (Subject Area Expert – Endline), Matthew Fisher-Post (Senior Research Analyst – Endline), Huyen Le (Research Analyst – Baseline), Aaron Wilson (Senior Research Analyst – Endline), Carlos Fierros (Research Analyst – Endline), Samantha Downey (Research Assistant – Endline), and Ilse Paniagua (Research Assistant – Endline). Graduate Research Assistants from the Harris School of Public Policy at Chicago, Selena Zhong and Gabriel Velez, provided research support during qualitative analysis. Veronica Pelivan (Local Coordinator) provided logistical support, took notes during KIIs, and led KIIs in two raions for the baseline. Local data collection was undertaken by IMAS INVEST SRL (IMAS) who conducted the FGDs and administered the evaluation survey. As a measure to ensure high data quality, NORC provided targeted training to IMAS for FGDs and survey administration/quality control; undertook a data quality review of the received data; and completed all the analysis. STUDY DESIGN The evaluation employs a pre-post design, comparing key indicators and findings for project beneficiaries at baseline to those at endline. This approach allows us to understand changes over time but does not allow us to attribute these changes to the project intervention since it was not possible to include a comparison group. ANNEX III: EVALUATION METHODS AND LIMITATIONS | 167 Concerns about confidentiality and sensitivity in contacting vulnerable children and their families directly, as well as the fact that children and families are indirectly beneficiaries of the P4EC project, led to a change in the target population for the evaluation survey. NORC had originally planned to survey caregivers of children in vulnerable families, but after consultation with P4EC and agreement by USAID/DCOF, a questionnaire was designed for community social assistants and social work specialists. This change in study design required NORC to depend solely on P4EC to obtain information on the well-being of children (i) currently in RCCs, and (ii) in vulnerable families, due to concerns about evaluators asking vulnerable families about often traumatic events and circumstances. NORC and P4EC worked closely together to develop indicators of child well-being (see Annex II), and P4EC built the capacity of and assisted social workers in obtaining child well-being information for both vulnerable children at home and vulnerable children currently in RCCs. The evaluation therefore comprises of primary data collected by the NORC team in the form of KIIs, FGDs, case file reviews, and a survey of community social assistants and social work specialists of the SAFPD, who are direct beneficiaries of the project (other direct beneficiaries include a large number of raion and national officials who were trained by the project); and secondary data collected by P4EC – child well-being indicators and monitoring data of capacity building training of stakeholders. NORC drafted and finalized the data collection tools (KII protocols, FGD protocols, and survey questionnaires) for Moldova. These were shared with USAID/DCOF and the implementing partner for their feedback prior to finalization. The tablet-based survey was programmed by IMAS and tested by both IMAS and NORC prior to beginning enumerator training. NORC developed all training materials and the NORC team traveled to Moldova prior to data collection at both baseline and endline to conduct the training for FGD and survey enumerators and conduct KIIs with P4EC staff and other stakeholders. Baseline data was collected from February 18 - April 23, 2015 and endline data was collected from June 26 -- July 28, 2017. All evaluators and program staff were asked to sign a confidentiality agreement prior to working on the evaluation. Copies of the signed statements are available upon request. Further, as part of the data collection training, the Evaluator included a substantive Confidentiality Training component for IMAS’s survey and FGD team. Each individual on the data collection team was asked to read aloud together, understand, and sign NORC's Pledge of Confidentiality Compliance before being accepted to work on the program. Signed copies from each field staff are securely filed with NORC, as is required by NORC’s Institutional Review Board (IRB). ANNEX III: EVALUATION METHODS AND LIMITATIONS | 168 In addition to the Evaluation Team and Survey Administration Team’s pledge of confidentiality, each FGD and Survey Questionnaire began with an informative introductory statement that describes to respondents the subject of the survey and some basic details about the confidential and voluntary nature of their participation. For example, the introduction informed respondents about the P4EC project and the purpose of the survey; the client and evaluators; and a statement that their participation is voluntary, that their responses will remain confidential and used in aggregated summaries only, that they may skip questions they don’t feel comfortable answering; and time required to complete the FGD or survey. We provide this information to respondents so that they may give an informed consent to participate, which is consistent with NORC’s professional commitment as members of the American Association for Public Opinion Research (AAPOR). Evaluation instruments for KIIs, FGDs and the Survey are included in Annex IV. TARGET POPULATION The quantitative survey (here on called the “evaluation survey”) targeted community social assistants and specialists of the SAFPD or “social work specialists” in the 10 project raions. Exhibit III.1 below shows the full sample framework for the baseline and endline survey as well as the attained response rates at each time. Exhibit III.1: NORC Evaluation Survey Sample Sizes at Baseline and Endline Raion Baseline Endline Social Workers Specialists Social Workers Specialists Cahul 30 4 30 5 Calarasi 31 4 28 4 Causeni 31 3 26 3 Falesti 30 7 30 4 Nisporeni 26 3 26 3 Orhei 30 5 30 5 Singerei 28 2 30 3 Soroca 30 4 30 3 Telenesti 30 2 30 2 Ungheni 30 4 30 4 TOTAL 296 38 290 36 Number of respondents 264 36 254 29 Number of eligible cases in sample 311 305 Response rate2 96.5% 92.80% Source: NORC Evaluation Survey 2 NORC uses the AAPOR standard to calculate response rates where the numerator is the number of completed cases and the denominator is the number of eligible cases in the sample. ANNEX III: EVALUATION METHODS AND LIMITATIONS | 169 Baseline qualitative data collection consisted of a series of 29 FGDs led by our subcontractor, IMAS, and 42 KIIs conducted primarily by Subject Area Expert Beth Bradford. Additional KIIs were conducted by Evaluator Mawadda Damon and Local Coordinator Veronica Pelivan. The KII team interviewed the P4EC project team and key national stakeholders such as representatives of the Department of Mother and Child at the Ministry of Education, Department of Pre-University Education at the Department of Education, and the Faculty of Social Work at Moldova State University. Additionally, Beth Bradford and Veronica Pelivan visited six project raions: Soroca, Telenesti, Singerei, Causeni, Ungheni, and Cahul where they interviewed the Vice Presidents on Social Issues, Heads of Social Assistance and Family Protection, Child and Family Protection Specialists, Heads of Education Departments, and Heads of Community Social Assistance Services in the SAFPDs and Heads of RCCs where they existed. Endline qualitative data collection consisted of a series of 22 FGDs led by IMAS and 53 KIIs conducted by Subject Area Expert Gary Gamer with a Moldovan team that included an interpreter and recordkeeper. The KII team interviewed P4EC project team and key national stakeholders, such as representatives from the Ministry of Labor, Social Protection and Family (MoLSPF); the Ministry of Health (MoH); the Ministry of Education (MoE); the Lumos Foundation; and the Free International University of Moldova. Gary Gamer visited eight project raions: Soroca, Singerei, Causeni, Ungheni, Cahul, Nisporeni, Falesti, and Calarasi, where he spoke with the Heads of Education Departments, Raion Council members, Managers of Community Social Assistance Service, Child Protection Specialists, PANDA moderators, and Directors of RCCs where they existed. All KIIs and FGDs were recorded where consent was obtained. They were largely conducted in Romanian. All FGD transcripts were translated into English by IMAS and coded in NVivo by NORC for data analysis. FGDs were conducted with raion and community-level stakeholders. At the raion level, FGDs were conducted with: social work specialists (those working at the SAFPD such as the Child Rights Protection Specialist, Family Protection Specialist, Head of Community Social Assistance service, Foster Care Manager, Manager of the Community Social Assistance Service, and Social Worker Coordinators of services such as adoption, family-type children’s homes, guardianship (kinship care), family support, and community centers); advisory boards of children (ABC); and gatekeeping commissions (made up of a mix of professionals from schools, police, health providers, local authorities, and NGOs who review child cases). At the community level, FGDs were conducted with: community social assistants; multidisciplinary teams, (made up of a mix of professionals including mayors, teachers, police, social workers, and health providers who review child cases); mayors; girls; boys; and parents. ANNEX III: EVALUATION METHODS AND LIMITATIONS | 170 Exhibit III.2: Number and Location of Focus Group Discussions at Baseline Target # FGDs # Participants # Male # Female Districts represented* Community social assistants 4 35 4 31 Causeni, Falesti, Singerei, Soroca, Ungheni Social worker specialists 2 15 0 15 Calarasi, Falesti, Nisporeni, Orhei, Ungheni Multidisciplinary Teams 2 16 4 12 Cahul, Singerei Gatekeeping Commissions 2 14 4 10 Calarasi, Falesti, Nisporeni, Orhei, Ungheni Mayors 3 18 16 2 Falesti, Nisporeni, Soroca, Ungheni ABC 2 15 2 13 Cahul, Calarasi Girls 4 34 0 34 Aged 12 -14 2 19 0 19 Falesti, Soroca Aged 15-17 2 15 0 15 Causeni, Orhei Boys 4 32 32 0 Aged 12 -14 2 19 19 0 Nisporeni, Telenesti Aged 15-17 2 13 13 0 Singerei, Ungheni Parents 6 46 1 45 At-risk children 2 16 0 16 Cahul, Singerei Deinstitutionalized children 2 13 1 12 Orhei, Ungheni Children in RCC 2 17 0 17 Nisporeni, Soroca Total 29 225 63 162 Source: NORC FGD data collection Exhibit III.3: Number and Location of Focus Group Discussions at Endline Target # FGDs # Participants # Male # Female Districts represented* Community social assistants 2 16 0 16 Causeni, Falesti, Singerei, Soroca, Ungheni Social worker specialists 1 11 0 11 Calarasi, Falesti, Nisporeni, Orhei, Ungheni Multidisciplinary Teams 2 17 1 16 Cahul, Singerei Gatekeeping Commissions 2 12 4 8 Calarasi, Falesti, Nisporeni, Orhei, Ungheni Mayors 2 12 8 4 Falesti, Nisporeni, Soroca, Ungheni ABC 1 6 4 2 National level Girls 4 26 0 26 Aged 12 -14 2 11 0 11 Falesti, Soroca Aged 15-17 2 15 0 15 Causeni, Orhei Boys 4 31 31 0 Aged 12 -14 2 15 15 0 Nisporeni, Telenesti Aged 15-17 2 16 16 0 Singerei, Ungheni Parents 3 25 3 22 At-risk children 2 17 2 15 Cahul, Singerei Reintegrated children 1 8 1 7 Nisporeni PANDA 1 7 6 1 Nisporeni Total 22 163 57 106 Source: NORC FGD data collection ANNEX III: EVALUATION METHODS AND LIMITATIONS | 171 Exhibit III.4: List of Key Informant Interviews at Baseline No. Title / Organization-Agency / Raion 1. Department of Pre-University Education, Ministry of Education 2. Department of Mother and Child, Ministry of Health 3. Dean, Faculty of Social Work, Moldova State University 4. Director, P4EC 5. Project Team Leader, P4EC 6. Child Participation Specialist, P4EC 7. Public Finance Reform Specialist, P4EC 8. Communications Specialists, P4EC 9. Training Specialist, P4EC 10. Site Coordinator, P4EC 11. Site Coordinator, P4EC 12. Site Coordinator, P4EC 13. Site Coordinator, P4EC 14. Site Coordinator, P4EC 15. Project Coordinator, Terre des hommes Moldova 16. Vice President on Social Issues, Raion Council Soroca 17. Head of Social Assistance and Family Protection, Soroca 18. Child and Family Protection Specialist, Soroca 19. Head of Education Department, Soroca 20. RCC Director, Soroca 21. Vice President on Social Issues, Telenesti 22. Head of Social Assistance and Family Protection, Telenesti 23. Child and Family Protection Specialist, Telenesti 24. Child Protection Specialist, Telenesti 25. Manager of Community Social Assistance, Telenesti 26. Head of Education Department, Telenesti 27. Vice President on Social Issues, Singerei 28. Head of Social Assistance and Family Protection, Singerei Child and Family Protection Specialist 29. Manager of Community Social Assistance, Singerei 30. Child Protection Specialist, Singerei 31. Head of Education Department, Singerei 32. Head of Social Assistance and Family Protection, Causeni 33. Deputy Head of Education Department, Causeni 34. Vice President on Social Issues, Causeni 35. Child and Family Protection Specialist, Causeni 36. Manager of Community Social Assistance, Ungheni 37. Vice President on Social Issues, Ungheni 38. Head Social Assistance and Family Protection, Ungheni 39. Head of Social Assistance Department, Ungheni 40. Vice President on Social Issues, Cahul 41. Child Protection Specialist, Cahul 42. Director of Crihana Veche, Residential Care Center, Cahul ANNEX III: EVALUATION METHODS AND LIMITATIONS | 172 Exhibit III.5: List of Key Informant Interviews at Endline No. Title/Organization-Agency/Raion 1. Child Protection Officer, UNICEF, Chișinău 2. Director, P4EC, Chișinău 3. Site Coordinators, P4EC, Chișinău 4. Consultant--Public Finance Reform, P4EC, Chișinău 5. Consultant--Training, P4EC, Chișinău 6. Communication Specialists, P4EC, Chișinău 7. Vice-President on Social Issues, Raion Council, Soroca 8. Head of Education Department, Soroca 9. Manager of the Community Social Assistance Service, Soroca 10. Head of Social Assistance and Family Protection, Soroca 11. Director of the Lumos Foundation, Chișinău 12. Department of Mother and Child, Ministry of Health, Chișinău 13. Chairperson, CCH Moldova/Hopes for Homes, Chișinău 14. Chief of the Social Assistance and Sociology Department, ULIM, Chișinău 15. Project Coordinator, Terre des Hommes Moldova, Chișinău 16. Senior Consultant, Department of Pre-University Education, Ministry of Education, Chișinău 17. Child Protection Specialist, USIAD, Chișinău 18. Director, CNPAC, Chișinău 19. Consultant--Child Participation, P4EC, Chișinău 20. Vice Minister of Labour, Social Protection, and Family, Chișinău 21. Vice President for Social Affairs, Sîngerei 22. Head of Social Assistance and Family Protection, Sîngerei 23. Social Assistance Manager, Sîngerei 24. Child Protection Specialist, Sîngerei 25. Head of the Department of Education, Călărași 26. Head of Social Assistance and Family Protection, Căușeni 27. Deputy Chairperson on Social Issues, Căușeni 28. Manager of Community Social Assistance, Căușeni 29. Child and Family Protection Specialist, Căușeni 30. PANDA Moderators, Nisporeni 31. PANDA Moderators, Falesti 32. Vice-President for Social Affairs, Ungheni 33. Manager of Community Social Assistance, Ungheni 34. Family Protection Specialist and Manager of the Family Support Service, Ungheni 35. Head of the Social Assistance Department, Ungheni 36. Deputy Head of the Social Assistance Department, Ungheni 37. Head of the Department of Education, Ungheni 38. Head of Family Doctor's Center, Ungheni 39. Specialist in Child and Family Protection, Cahul 40. Specialist in Child Protection, Cahul 41. Head of Social Assistance and Family Protection, Cahul 42. Head of Education Department, Cahul ANNEX III: EVALUATION METHODS AND LIMITATIONS | 173 No. Title/Organization-Agency/Raion 43. Head of Family Doctor's Center, Cahul 44. RCC Director, Cahul 45. Head of Education Department, Nisporeni 46. Director, P4EC, Chișinău 47. Minister for Labor, Social Protection, and Family, Chișinău SAMPLING At baseline, NORC included the full sample frame of all male community social assistants and all the social work specialists in the 10 project raions. Where the total number of community social assistants (male and female) was less than 32, NORC included all the community social assistants (Calarasi (31), Causeni (31), Nisporeni (26), and Singerei (28)). In the remaining 6 project raions, we selected a total of 30 community social assistants – first including all the males and then randomly selecting among the females. This resulted in a sample of 296 community social assistants and 38 social work specialists. At endline, NORC used an updated sample frame from P4EC of all community social assistants and social work specialists in the 10 project raions in July 2017. Names were matched to the baseline sample and NORC included all those in the 2017 sample frame who were also in the baseline sample frame (i.e., those who were employed throughout the project period). Additionally, NORC included all male community social assistants and all social work specialists from the 2017 list and excluded anyone who had refused the survey in 2015. Lastly, to complete a sample of 30 community social assistants per raion, NORC randomly sampled from among the “new” community social assistants (i.e., those who were not employed at baseline). This resulted in a sample of 290 community social assistants and 36 social work specialists. It is important to note that only 32 out of 283 endline survey respondents, just over 10%, were “new” community social assistants. Baseline FGD locations were chosen to represent the diversity of different raions so that each target group had representatives from the “old” and “new” raions; raions of low, medium/medium-high, and high level of development of child care and protection systems; and, to the extent possible given the other two criteria and limited total number of FGDs, northern, central, and southern raions. Endline FGD locations were chosen to match those at baseline as much as possible. Baseline and endline raions where the KIIs were conducted were chosen to include a range of low, medium/medium high, and high levels of child care and protection system development, and regional ANNEX III: EVALUATION METHODS AND LIMITATIONS | 174 LIMITATIONS The Evaluation Team encountered some limitations inherent to the design of this evaluation and during its fieldwork in Moldova. Some of the more relevant limitations are listed below: Baseline timeframe. The program started in January 2014, but NORC’s concept note for the evaluation was approved by mid-June 2014; the evaluation design was completed by December 2014; and the data collection was undertaken in March-April 2015. Thus some program implementation such as training and capacity building of stakeholders had already started prior to the baseline. This may have resulted in the Evaluation Team capturing less of the project impact through the survey than actually took place. Administrative and M&E data from the implementer. P4EC has been conscientious regarding data collection, making great efforts to work with the community social assistants to collect important data on child demographics, well-being, family status, etc. However, lack of sufficient community social assistant skills in following rigorous data quality assurance practices resulted in too few observations for certain indicators and inconsistency across districts, sometimes making the data unusable for analytic purposes. At baseline, NORC was only able to include information from administrative data on children in RCCs and was unable to use the data on vulnerable children and families. Additionally, baseline data on RCCs only included information from 4 raions where RCCs were located – Cahul, Ciniseuti/Rezina, Hirbovat/Calarasi, and Nisporeni. Information from RCCs in Visoca/Soroca was shared too late to be included in the baseline report. Part-way through the project, NORC conducted a capacity building exercise to improve the quality of the M&E data, and was therefore able to report data on vulnerable children and families for the endline, but issues remained with both the RCC data and vulnerable family data which is detailed below. Changes in collection method for deinstitutionalized children data. After the capacity building exercise, P4EC prepared consistent excel files which were shared with all community social assistants to collect follow up well-being information from deinstitutionalized children. While a dropdown filter of responses to questions made the data collection more consistent and less subjective across all project raions, it required community social assistants to recode into the excel files information from the baseline assessment of children while still in the RCCs. This caused a slight change in some response options and a difference in numbers reported in NORC’s baseline report for this project. Since consistency in data collection method is more important than a recall bias caused from reconciling data from 2 years ago we chose to use and present in this endline report the slightly different assessment data as baseline followed by endline information for deinstitutionalized children. It should be noted that the endline indicators ANNEX III: EVALUATION METHODS AND LIMITATIONS | 175 for Ciniseuti are for one year after deinstitutionalization since most of the cases were then closed (either because the objectives of the plan were achieved, or children had become 18 years old); the endline indicators for Nisporeni and Visoca/Soroca are for two years after deinstitutionalization. The attrition seen for Nisporeni and Visoca/Soroca is due to case files closing or children becoming 18 years old. Using and reporting M&E data on vulnerable families and children. The vulnerable family database of P4EC includes case file information inputted by community social assistants on a rolling basis as they open and close case files on vulnerable children. It included information on files prior to the current DCOF project. Given our concerns regarding this database, we grouped and analyzed the follow case files in this endline report: (i) grouped together all case files opened during May 1 to July 31, 2016 (after the M&E capacity building trip by NORC) to be baseline data, (ii) Followed monitoring on these case files and grouped as endline those whose case files closed by December 31, 2016 (5-7 months of monitoring), (iii) followed monitoring for the baseline group of children and grouped together as endline those whose case file closed between March – May 2017 (10-12 months of monitoring). Survey respondents. The survey data respondents are social workers – both community social assistants and social work specialists – and are likely to have biases, since the evaluation is a reflection of their own performance. While this group is made up of project beneficiaries (in the training and capacity building they received from P4EC), they also gave feedback on community members and parents who were indirect beneficiaries of the project. NORC did not examine differences between baseline and endline results to see if these differences were statistically significant. ANNEX IV: DATA COLLECTION INSTRUMENTS | 176 ANNEX IV: DATA COLLECTION INSTRUMENTS ANNEX IV: DATA COLLECTION INSTRUMENTS | 177 SURVEY OF SOCIAL WORKERS (Community Social Workers and Raion Level Specialists) INTRO. PROG: PLEASE DISPLAY THE FOLLOWING MESSAGE ON ITS OWN PAGE: Hello, my name is ____________ from the Institute of Marketing and Polls IMAS-INC Chisinau (IMAS). We are working with NORC at the University of Chicago on an evaluation of Partnership for Every Child’s Program that is working to strengthen child care and protection in Moldova. This program started in January 2014 and will continue until June 2017. The program and this evaluation are funded by The United States Agency for International Development - USAID. We are asking you to participate in a 40-minute survey in order to obtain your feedback on the child care and protection system in your raion and community and information on your experience as a social worker. You have been selected for this survey because you have received training or guidance from Partnership for Every Child. We conducted this survey at the baseline and are requesting your opinion at the end of the program. Your feedback will be very important to us and to Partnership for Every Child. The information you provide will be used to improve Partnership for Every Child’s current and future programs and will inform future programming funded by USAID. Your participation in this survey is voluntary and you may choose to skip a question or discontinue your participation at any time without any penalty. Your identity will be kept confidential and will not be shared with Partnership for Every Child. We will keep your contact information in order to contact you to obtain your feedback until the end of the program. If you have any questions about the survey, you may contact Serviciul de Operatiuni Teren IMAS at tel: 022 26 00 96 May we start now? 1. Yes 2. No PROG: SKIP TO CLOSURE PROG: INSERT NEXT, PREVIOUS, STOP BUTTONS ON THIS AND SUBSEQUENT PAGES PROG: COLLECT TIME STAMP DATA UPON ENTRY OF THIS PAGE, AND AT THE END OF EACH SECTION PROG: END MESSAGE ANNEX IV: DATA COLLECTION INSTRUMENTS | 178 PROG: BEGIN FIELD CONTROL SECTION: FIELD CONTROL ID. Respondent ID: ______________ PROG: DROP BOX WITH ALL IDs RAION Raion ID: 10 raions listed, select one LOC. Locality ID: ______________ PROG: DROP BOX WITH ALL LOCALITIES LIMBA Language Choice of Romanian or Russian COD_OP. Interviewer ID: _____________________ PROG: NUMBER [RANGE] PROG: END FIELD CONTROL. ANNEX IV: DATA COLLECTION INSTRUMENTS | 179 ENUMERATOR: READ OUT RESPONSE OPTIONS UNLESS OTHERWISE STATED PROG: NOTE ALL QUESTIONS WILL ALSO HAVE A DON’T KNOW=-1, REFUSED=-2, and NOT APPLICABLE=-3 OPTION. NOTE: Those who answer 1 to Question 5 (meaning they are social workers) will answer questions about/at community level. Those who answer 2 to question 5 (meaning they are specialist) will answer questions about/at raion level. SECTION 1: DEMOGRAPHICS 1. GENDER. Sex. ENUMERATOR: DO NOT READ RESPONSES. CODE FROM RESPONSE OPTIONS. 1. Male 2. Female 2. AGE. How old are you? _________ PROG: NUMBER [1-110] 3. EDU. What is your highest level of education completed? 1. Less than 11 years of schooling PROG: SKIP TO 5: JOB 2. 11 years of schooling PROG: SKIP TO 5: JOB 3. High school PROG: SKIP TO 5: JOB 2 College 3 University 4 Master Degree 4. EDU_MASTER. PROG: ASK IF EDU=4 What is your degree area? 1. Social work 2. Education 3. Medicine 4. Law 5. Public Administration 6. Other related Social Field: _________ PROG: OPEN ENDED FIELD 7. Other unrelated field: _______________ 5. JOB. What is your job title? ENUMERATOR: DO NOT READ ANSWER OPTIONS. IF IS COMMUNITY SOCIAL WORKER, SELECT 1, IF ANY OF THE RAION LEVEL POSITIONS, SELECT 2 1. Community Social Worker 2. Specialists of the SAFPD or “Social work specialist” ANNEX IV: DATA COLLECTION INSTRUMENTS | 180 6. RAION_GEO. In what raion do you work? ______________ PROG: DROP BOX WITH ALL RAISONS 7. COM_GEO. In what community do you work? ______________ PROG: DROP BOX WITH ALL COMMUNITIES 8. YEAR_POS. How many years have you been in your position? ENUMERATOR: IF RESPONDENT DOES NOT REMEMBER THE EXACT NUMBER OF MONTHS BEYOND THE YEAR(S) WORKED, PLEASE ENTER ONLY THE NUMBER OF YEARS. IF THEY HAVE WORKED LESS THAN A YEAR, PLEASE ASK FOR THE NUMBER OF MONTHS. _____________ YEAR _____________ MONTH PROG: YEAR, RANGE 1-40. MONTH: 1-12 PROG: END SECTION 1 (PLEASE COLLECT TIME STAMP) SECTION 2: OPINIONS ON SUPPORTS TO CHILDREN ENUMERATOR: PLEASE READ OUT LOUD. I will now ask you a series of questions about the services that are available in your raion/community. 9. SERVICES_RAION. Which of the following services to support vulnerable families are available in your raion? PROG: ASK IF JOB = 2 1= Yes 2= No A. Financial assistance – government cash assistance / other social assistance programs B. Financial assistance – nongovernment C. Case management for families at-risk D. Parenting support services (parenting education, support groups, etc.) E. Other family support (please specify) F. Mental health counseling (for parents, for children, for both) G. Substance use/abuse services H. Housing assistance services I. Employment / income generation services J. Public schooling for children K. Job skills training programs for adults ANNEX IV: DATA COLLECTION INSTRUMENTS | 181 1= Yes 2= No L. Continuing education for adults M. Daycare services N. Kindergartens O. Creche P. Respite care for children with disabilities (APP respiro) Q. Special services for children with disabilities S. Juvenile delinquency prevention programs T. Other service (please specify) ENUMERATOR: Ask if there are any services that have not been mentioned 10. SERVICES_COM. Which of the following services to support vulnerable families are available in your community? PROG: ASK IF JOB = 1 1= Yes 2= No A. Financial assistance – government cash assistance / other social assistance programs B. Financial assistance – nongovernment C. Case management for families at-risk D. Parenting support services (parenting education, support groups, etc.) E. Other family support (please specify) F. Mental health counseling (for parents, for children, for both) G. Substance use/abuse services H. Housing assistance services I. Employment / income generation services J. Public schooling for children K. Job skills training programs for adults L. Continuing education for adults M. Daycare services N. Kindergartens O. Creche ANNEX IV: DATA COLLECTION INSTRUMENTS | 182 1= Yes 2= No P. Respite care for children with disabilities (APP respiro) Q. Special services for children with disabilities R. Juvenile delinquency prevention programs S. Other service (please specify) ENUMERATOR: Ask if there are any services that have not been mentioned 11. SER_RAION_ADE. How adequate do you believe the services in your raion generally are to meet the needs of families to help ensure that children can stay in or be returned to family care? PROG: ASK IF JOB = 2 1. Fully adequate 2. Somewhat adequate 3. Not at all adequate PROG: SKIP TO 13. SERVICE_RAION_ADE_B 12. SER_RAION_ADE_A. PROG: ASK IF 11. SERVICE_RAION_ADE = 1 OR 11. SERVICE_RAION_ADE = 2 What makes them adequate? Please choose all that apply. PROG: SELECT ALL THAT APPLY. PROG: ASK IF JOB = 2 1. A wide range of services exists 2. Services are community-based 3. Services are accessible to all families 4. Families are aware of the services 5. Services are designed based on the needs of family 6. Professionals are well prepared to work with families in these services 7. Other, please specify _____________ PROG: OPEN-ENDED FIELD PROG: UPON ENTRY, SKIP TO 14. SER_RAION_NEED IF SER_RAION_ADE=1. 13. SER_RAION_ADE_B. PROG: ASK IF 11. SER_RAION_ADE = 2|11.SER_RAION_ADE=3 What are the reasons that services do not fully meet the needs of families? Please choose all that apply. PROG: SELECT ALL THAT APPLY. PROG: ASK IF JOB = 2 1. Services are not in place at all 2. Too few services are in place to meet the needs 3. Families cannot access the services 4. Families do not know about the services 5. Services do not match the needs of families 6. Family needs are complex and cannot be met by existing services 7. Professionals running the services are not proficient at working with families 8. Other, please specify ______________________ PROG: OPEN-ENDED FIELD 14. SER_RAION_NEED. Are there any services that are strongly needed in your raion that are not available? PROG: ASK IF JOB = 2 ANNEX IV: DATA COLLECTION INSTRUMENTS | 183 1. Yes 2. No PROG: SKIP TO 16. SER_COM_ADE 15. SER_RAION_NEED_A. PROG: ASK IF 14. SER_RAION_NEED = 1. What are the services that are strongly needed in your raion but are not available? 1. Financial assistance – government cash assistance / other social assistance programs 2. Financial assistance – nongovernment 3. Case management for families at-risk 4. Parent support services (parenting education, support groups, etc.) 5. Other family support (please specify) 6. Mental health counseling (for parents, for children, for both) 7. Substance use/abuse services 8. Housing assistance services 9. Employment / income generation services 10. Public schooling for children 11. Job skills training programs for adults 12. Continuing education for adults 13. Daycare services 14. Kindergartens 15. Creche 16. Respite care for children with disabilities (APP respiro) 17. Special services for children with disabilities 18. Juvenile delinquency prevention programs 19. Other service, please specify ______________________ PROG: OPEN-ENDED FIELD 16. SER_COM_ADE. How adequate do you believe the services in your community generally are to meet the needs of families to help ensure that children can stay in or be returned to family care? PROG: ASK IF JOB = 1 1. Fully adequate 2. Somewhat adequate 3. Not at all adequate PROG: GO TO 18. SERVICE_COM_ADE_B 17. SER_COM_ADE_A. PROG: ASK IF 16. SER_COM_ADE = 1 OR 16. SER_COM_ADE = 2 What makes them adequate? Please choose all that apply. PROG: SELECT ALL THAT APPLY. PROG: ASK IF JOB = 1 1. A wide range of services exists 2. Services are community-based 3. Services are accessible to all families 4. Families are aware of the services 5. Services are designed based on the needs of families 6. Professionals are well prepared to work with families in these services 7. Other, please specify _____________ PROG: OPEN-ENDED FIELD PROG: UPON ENTRY SKIP TO 19. SER_COM_NEED IF 16. SER_COM_ADE=1. ANNEX IV: DATA COLLECTION INSTRUMENTS | 184 18. SER_COM_ADE_B. PROG: ASK IF 16. SER_COM_ADE = 2 | 16. SER_COM_ADE= 3 What are the reasons that services do not fully meet the needs of families? Please choose all that apply. PROG: SELECT ALL THAT APPLY. PROG: ASK IF JOB = 1 1. Services are not in place at all 2. Too few services are in place to meet the needs 3. Families cannot access the services 4. Families do not know about the services 5. Services do not match the needs of families 6. Family needs are complex and cannot be met by existing services 7. Professionals running the services are not proficient at working with families 8. Other, please specify ______________________ PROG: OPEN-ENDED FIELD 19. SER_COM_NEED. Are there any services that are strongly needed in your community that are not available? PROG: ASK IF JOB = 1 1. Yes PROG: SKIP TO 21. SER_SPECIAL_POP 2. No 20. SER_COM_NEED_A. PROG: ASK IF 19. SER_COM_NEED = 1 What are the services that are strongly needed in your community but are not available? Choose all that apply. PROG: SELECT ALL THAT APPLY. PROG: ASK IF JOB = 1 1. Financial assistance – government cash assistance / other social assistance programs 2. Financial assistance – nongovernment 3. Case management for families at-risk 4. Parent support services (parenting education, support groups, etc.) 5. Other family support (please specify)______________ 6. Mental health counseling (for parents, for children, for both) 7. Substance use/abuse services 8. Housing assistance services 9. Employment / income generation services 10. Public schooling for children 11. Job skills training programs for adults 12. Continuing education for adults 13. Daycare services 14. Kindergartens 15. Creche 16. Respite care for children with disabilities (APP respiro) 17. Special services for children with disabilities 18. Juvenile delinquency prevention programs 19. Other service, please specify ______________________ PROG: OPEN-ENDED FIELD ANNEX IV: DATA COLLECTION INSTRUMENTS | 185 21. SER_SPECIAL_POP. In general please rate how adequate existing services are to meet the needs of each of the following special populations of families in your raion/community: 1. Fully Adequate 2. Somewhat adequate 3. Not at all adequate A. Single mothers B. Kinship care givers – guardians (For example, grandmothers) C. Families with substance use/abuse issues D. Families with children with disabilities E. Families living in poverty F. Families at risk of living in poverty G. Teenage parents H. Children left behind by migrating parents I. Families with children with disabilities or special education J. Children from ethnic minorities groups K. Others, please specify specify ENUMERATOR: Ask if there is any other special population and then ask them to rate how adequate existing service is for this group ______________________ PROG: OPEN-ENDED FIELD 22. SER_ACCESS. Are families in need generally able to access the services in your raion/community provided to help ensure that children can stay in or be returned to family care? 1. Yes, all families PROG: SKIP TO 25. SER_ACCESS_RATE ANNEX IV: DATA COLLECTION INSTRUMENTS | 186 2. Yes, most families PROG: SKIP TO 25. SER_ACCESS_RATE 3. Yes, some families 4. No 23. SER_ACCESS_NO. PROG: ASK if 22. SER_ACCESS = 3 OR 22. SER_ACCESS = 4. Which services are generally not accessible to families in need in your raion/community? Choose all that apply. PROG: SELECT ALL THAT APPLY. 1. Financial assistance – government cash assistance / other social assistance programs 2. Financial assistance – nongovernment 3. Case management for families at-risk 4. Parent support services (parenting education, support groups, etc.) 5. Other family support (please specify) 6. Mental health counseling (for parents, for children, for both) 7. Substance use/abuse services 8. Housing assistance services 9. Employment / income generation services 10. Public schooling for children 11. Job skills training programs for adults 12. Continuing education for adults 13. Daycare services 14. Kindergartens 15. Creche 16. Respite care for children with disabilities (APP respiro) 17. Special services for children with disabilities 18. Juvenile delinquency prevention programs 19. Other service (please specify) _____________________ PROG: OPEN-ENDED FIELD 24. SER_ACCESS_REASON. PROG: ASK if 22. SER_ACCESS = 3 OR 22. SER_ACCESS = 4. What are the reasons that these services are not accessible? Choose all that apply. PROG: SELECT ALL THAT APPLY. 1. Service does not exist in family’s community 2. Distance – services are too far for families to reach 3. Physical accessibility – due to physical disability family cannot get to the service (example lack of access ramps to building) 4. Transportation issues 5. Cost prohibitive 6. Stigma attached to services 7. Working hours offered do not meet family’s schedule 8. Language barrier – service is not offered in family’s language 9. Other , please specify _____________________ PROG: OPEN-ENDED FIELD ANNEX IV: DATA COLLECTION INSTRUMENTS | 187 25. SER_ACCESS_RATE. For each of the following special populations of families, please rate their level of access to existing services. 1 = Services accessible to all families 2 = Services accessible to most families 3 = Services accessible to some families 4 = Services not accessible to any families A. Single mothers B. Kinship care givers – guardians (For example, grandmothers) C. Families with substance use/abuse issues D. Families with children with disabilities E. Families living in poverty F. Families at risk of living in poverty G. Teenage parents H. Children left behind by migrating parents I. Families with children with disabilities or special education J. Children from ethnic minorities groups K. Others, please specify specify ENUMERATOR: Ask if there is any other special population and then ask them to rate how adequate existing service is for this group ______________________ PROG: OPEN-ENDED FIELD 26. ALT_CARE. What types of alternative care are in place in your raion/community to meet the needs of children without adequate parental care? Choose all that apply. PROG: SELECT ALL THAT APPLY. 1. Guardianship 2. APP 3. CCTF 4. Community group home 5. Adoption 6. Emergency/temporary placement center 7. Boarding schools ANNEX IV: DATA COLLECTION INSTRUMENTS | 188 8. Other resident institution (not boarding school) 9. Others, please specify ______________________ PROG: OPEN-ENDED FIELD PROG: BEGIN ALTERNATIVE CARE RATING LOOP TO BE APPLIED ON EACH OPTION CHOSE IN 26. ALT_CARE 27. ALT_CARE_RATE. How would you rate the quality of the service? 1. Excellent 2. Good 3. Fair 4. Poor PROG: END ALTERNATIVE CARE RATING LOOP. 28. DEPT_CHILD_SUPPORT. Does your department provide support to children? 1. Yes 2. No PROG: SKIP TO 31 DEPT_CAREGIVER_SUPPORT 29. DEPT_CHILD_SUPPORT_A. PROG: ASK IF 28. DEPT_CHILD_SUPPORT = 1. What supports are provided to children? Choose all that apply. PROG: SELECT ALL THAT APPLY. 1. Services to address visual/auditory disabilities 2. Services to address cognitive/learning disabilities 3. Services for mental health issues 4. Family support 5. Counselling 6. Material support (for example, school supplies) 7. Referral to services to address physical/mobilities disabilities 8. Referral to health services 9. Referral to special education support 10. Referral to early childhood support 11. Referral to child care 12. Other, please specify ______________________ PROG: OPEN-ENDED FIELD 30. DEPT_CHILD_SUPPORT_B. PROG: ASK IF 28. DEPT_CHILD_SUPPORT = 1. To what extent do you believe the support provided to children is effective? 1. Effective 2. Somewhat effective 3. Not at all effective 31. DEPT_CAREGIVER_SUPPORT. Does your department provide support to caregivers (the person responsible for the care of the child including biological parents, guardians, foster parents, etc.)? 1. Yes 2. No PROG: SKIP TO 34. SER_ACCESS_RESULT ANNEX IV: DATA COLLECTION INSTRUMENTS | 189 32. DEPT_CAREGIVER_SUPPORT_A. PROG: ASK IF 31. DEPT_CAREGIVER_SUPPORT = 1. What supports are provided to caregivers? Choose all that apply. PROG: SELECT ALL THAT APPLY. 1. Access to cash benefits or other financial assistance 2. Family case management 3. Counselling 4. Social/psychological services for people with mental health issues 5. Disability services for children 6. Parenting education 7. Referral to services 8. Referral to specialist health care 9. Referral to child care 10. Referral to alcohol or substance abuse treatment 11. Other, please specify ______________________ PROG: OPEN-ENDED FIELD 33. CARE_GIVER_SUPPORT_B. PROG: ASK IF 31. DEPT_CAREGIVER_SUPPORT = 1. To what extent do you believe the support provided to caregivers by service providers improves their caregiving? 1. Greatly improves 2. Somewhat improves 3. Does not improve 34. SER_ACCESS_RESULT. In general, as a result of access to family support services, do you see the families’ ability to care of their children as: 1. Improving considerably 2. Improving some 3. Staying the same 4. Not improving much 5. No change 35. SER_TOP3. In your opinion, what are the top three support services to families that are the most effective to improving a families’ ability to care for their children? Choose only three. PROG: ALLOW UP TO THREE CHOICES. 1. Cash benefits or other financial assistance 2. Family case management 3. Referral to services 4. Counseling 5. Social/psychological services for people with mental health issues 6. Specialist health care 7. Child care 8. Disability services for children 9. Alcohol and substance abuse treatment 10. Parenting education 11. Other, please specify ______________________ PROG: OPEN-ENDED FIELD ANNEX IV: DATA COLLECTION INSTRUMENTS | 190 36. SER_IMPROVE. Which support services to families do you feel can be improved? Choose all that apply. PROG: SELECT ALL THAT APPLY. 1. Cash benefits or other financial assistance 2. Family case management 3. Referral to services 4. Counseling 5. Social/psychological services for people with mental health issues 6. Specialist health care 7. Child care 8. Disability services for children 9. Alcohol and substance abuse treatment 10. Parenting education 11. Other, please specify ______________________ PROG: OPEN-ENDED FIELD 37. SER_AVAIL_IMPROVE. What are the top three support services that need improvement in terms of availability? Choose only three. PROG: ALLOW UP TO THREE CHOICES. 1. Cash benefits or other financial assistance 2. Family case management 3. Referral to services 4. Counseling 5. Social/psychological services for people with mental health issues 6. Specialist health care 7. Child care 8. Disability services for children 9. Alcohol and substance abuse treatment 10. Parenting education 11. Other, please specify ______________________ PROG: OPEN-ENDED FIELD 38. SER_QUALITY_IMPROVE. What are the top three support services that need improvement in terms of quality (how well the service is administered)? Choose only three. PROG: ALLOW ONLY UP TO THREE CHOICES. 1. Cash benefits or other financial assistance 2. Family case management 3. Referral to services 4. Counseling 5. Social/psychological services for people with mental health issues 6. Specialist health care 7. Child care 8. Disability services for children 9. Alcohol and substance abuse treatment 10. Parenting education 11. Other, please specify ______________________ PROG: OPEN-ENDED FIELD ANNEX IV: DATA COLLECTION INSTRUMENTS | 191 39. SER_EFFECT_IMPROVE. What are the top three support services that need improvement in terms of effectiveness (use of the service results in an improvement)? Choose only three. PROG: ALLOW UP TO THREE CHOICES. 1. Cash benefits or other financial assistance 2. Family case management 3. Referral to services 4. Counseling 5. Social/psychological services for people with mental health issues 6. Specialist health care 7. Child care 8. Disability services for children 9. Alcohol and subtance abuse treatment 10. Parenting education 11. Other, please specify ______________________ PROG: OPEN-ENDED FIELD PROG: END SECTION 2 (PLEASE COLLECT TIME STAMP) SECTION 3: OPINIONS ON COORDINATION AND COLLABORATION AND OTHER SERVICE PROVIDERS ENUMERATOR: PLEASE READ OUT LOUD I will now ask you questions about your opinions of the different actors of the child care and protection system. 40. COOR_EXTENT. What is the extent of coordination and collaboration between the following actors toward meeting the needs of families? 1 = High Coordination/Col laboration 2 = Medium Coordination/ Collaboration 3 = Low Coordination/Col laboration 4 = No Coordination/ Collaboration A. Raional/local government and non￾government actors B. National government and raional/local government actors C. Raional child protection actors and local actors (For example, school, police, health officies ANNEX IV: DATA COLLECTION INSTRUMENTS | 192 41. COLLAB_WHO. With whom do you collaborate to meet the needs of families with whom you work? Choose all that apply. PROG: SELECT ALL THAT APPLY. 1. The families themselves 2. Extended family 3. Family’s neighbors 4. Community leaders (mayors, council members, other leaders) 5. Peer and colleagues in my department 6. Colleagues in other departments (such as education, health) 7. Educational institutions (schools, teachers, university, etc.) 8. Community social workers 9. Raion social work specialists 10. Services providers – child protection related 11. Services providers – non-child protection related (for example, employment or housing) 12. Law enforcement 13. Other, please specify ______________________ PROG: OPEN-ENDED FIELD 42. PROVIDER_RATING. To what extent do you agree or disagree that you are able to call on the support of other service providers in order to provide integrated care to families? 1. Strongly agree 2. Somewhat agree 3. Somewhat agree 4. Strongly disagree 43. SER_VALUABLE. Which other service providers do you find highly valuable when you are supporting families? Choose all that apply. PROG: SELECT ALL THAT APPLY. 1. Health clinics 2. Police 3. Domestic violence services 4. Schools and teachers 5. Mental health services 6. Community organizations 7. Churches 8. Other, please specify ______________________ PROG: OPEN-ENDED FIELD 44. SER_WEAK. Which other service providers do you find provide the weakest support? Choose all that apply. PROG: SELECT ALL THAT APPLY. 1. Health clinics 2. Police 3. Domestic violence services 4. Schools and teachers 5. Mental health services 6. Community organizations 7. Churches 8. Other, please specify ______________________ PROG: OPEN-ENDED FIELD ANNEX IV: DATA COLLECTION INSTRUMENTS | 193 45. SER_DIFF. Which other service providers do you find are the most difficult to arrange? Choose all that apply. PROG: SELECT ALL THAT APPLY. 1. Health clinics 2. Police 3. Domestic violence services 4. Schools and teachers 5. Mental health services 6. Community organizations 7. Churches 8. Other, please specify ______________________ PROG: OPEN-ENDED FIELD 46. CIL_VIOLENCE. Do you agree or disagree that civil society (for example non-governmental organizations) is sufficiently engaged in protecting children from violence, abuse, exploitation, or neglect? 1. Strongly agree 2. Somewhat agree 3. Somewhat disagree 4. Strongly disagree 47. CIL_SEPARATION. Do you agree or disagree that civil society (for example non-governmental organizations) is sufficiently engaged in preventing separation of children from families? 1. Strongly agree 2. Somewhat agree 3. Somewhat disagree 4. Strongly disagree PROG: END SECTION 3 (PLEASE COLLECT TIME STAMP) SECTION 4: SOCIAL WORKER CAPABILITIES AND PRACTICES 48. RISK_FAM_VUL. What risk factors make a family more vulnerable? Choose all that apply. PROG: SELECT ALL THAT APPLY. 1. Limited or lack of family income 2. Lack of adequate housing 3. Unemployment/under employment 4. Substance use/abuse 5. Conflict with the law 6. Domestic violence history 7. Mental health 8. Physical or other disabilities 9. Isolation from the community/stigmatization by other community members 10. Lack of access to services 11. Relationship within the family 12. Lack of attachments 13. Other, please specify ______________________ PROG: OPEN-ENDED FIELD ANNEX IV: DATA COLLECTION INSTRUMENTS | 194 49. RISK_FAM_PROTECT. What types of protective factors make a family more able to protect and care for their child? Choose all that apply. PROG: SELECT ALL THAT APPLY. 1. Ability to meet family needs with resources available 2. Good family relationships 3. Bonding and attachment between child/children and family 4. Emotional resiliency 5. Knowledge of child development 6. Knowledge of child health and education 7. Access to basic services 8. Connection to community – other parents, neighbors, teachers, other professionals 9. Connection to extended family 10. Other, please specify ______________________ PROG: OPEN-ENDED FIELD 50. TOOLS. Do you use a tool to assess families’ ability to protect and care for their children? 1. Yes 2. No PROG: GO TO 52. WORKLOAD 51. ASSESS_FACTOR. PROG: ASK IF 50. TOOLS = 1 What types of factors does that assessment look at? Choose all that apply. PROG: SELECT ALL THAT APPLY. 1. Family income 2. Housing conditions 3. Family size/ Number of children 4. Age of children 5. Health issues for family members 6. Family relationship 7. Substance use/abuse 8. Domestic violence/ child abuse 9. School attendance (of children) 10. Mental health (of adults in family) 11. Family ability to meet needs with resources available 12. Family’s involvement with various services 13. Family’s relationships with extended family, neighbors, community 14. Caregiver knowledge of child development 15. Parenting skills and knowledge 16. Other, please specify ________________ PROG: OPEN-ENDED FIELD 52. WORKLOAD. How often are you in the situation where you work overtime? 1. All of the time 2. Most of the time 3. Some of the time 4. Never ANNEX IV: DATA COLLECTION INSTRUMENTS | 195 53. CASE_NO. How many active cases do you currently have? _________ PROG: NUMBER [0-50] 54. CONTACT_PREV_FREQ. On average, how often do you make contact with the prevention case families with whom you work? 1. Daily 2. Weekly 3. Bi-weekly 4. Monthly 5. Every other month 6. Less than every other month 55. CONTACT_ACTIVE_FREQ. On average, how often do you make contact with the active case families with whom you work? 1. Daily 2. Weekly 3. Bi-weekly 4. Monthly 5. Every other month 6. Less than every other month 56. CONTACT_MONITOR_FREQ. On average, how often do you make contact with the monitoring case families with whom you work? 1. Daily 2. Weekly 3. Bi-weekly 4. Monthly 5. Every other month 6. Less than every other month 57. CONNECT_WAYS. How do you connect with families with whom you work? Choose all that apply. PROG: SELECT ALL THAT APPLY. 1. Face-to-face meetings in my office 2. Home visits 3. Telephone 4. Email 5. Meetings at the child’s school 6. Other, please specify ________________ PROG: OPEN-ENDED FIELD 58. TIME_ASSESS. To what extent do you feel you have enough time to visit with the families you work with? 1. Always PROG: GO TO 60. JOB_ASSESS 2. Most of the time ANNEX IV: DATA COLLECTION INSTRUMENTS | 196 3. Some of the time 4. Never 59. TIME_ASSESS_A. PROG: ASK IF 58. TIME_ASSESS! = 1 What prevents you from having enough time? Choose all that apply. PROG: SELECT ALL THAT APPLY. 1. High case load 2. Paperwork that has to be completed 3. Family willingness to meet 4. Lack of transportation (family or worker) 5. Location of the family 6. Lack of private meeting space 7. Other, please specify ________________ PROG: OPEN-ENDED FIELD 60. JOB_ASSESS. Do you agree or disagree with the following? 1 = Strongly agree 2 = Somewhat agree 3 = somewhat disagree 4 = strongly disagree A. I understand the role and responsibilities of my job B. I feel supported in my job C. My supervisor provides support that helps me to do my job better 61. JOB_RESPONSIBILITY. Which of the following are part of your job responsibilities? Choose all that apply. PROG: SELECT ALL THAT APPLY. 1. Identifying clients 2. Completing assessments 3. Interviewing / talking with clients 4. Developing care / service plans 5. Referrals to other services/organizations 6. Representing clients in court 7. Representing cases in the gatekeeping commission 8. Completing paperwork for government assistance 9. Making home visits 10. Working one-on-one with clients in my office 11. Working with adult clients 12. Working with child clients 13. Running group sessions for clients 14. Completing reports to my supervisors 15. Supervising other workers 16. Other (please specify) ANNEX IV: DATA COLLECTION INSTRUMENTS | 197 PROG: BEGIN JOB RESPONSIBILITY LOOP. ASK 62. JOB_RESPONSIBILITY_A FOR EACH OPTION SELECTED IN 61. JOB_RESPONSIBILITY. 62. JOB_RESPONSIBILITY_A. How confident do you feel in your ability to do a good job in each of the following areas of your work? 1. Very confident 2. Somewhat confident 3. Not at all confident PROG: END JOB RESPONSIBILITY LOOP. 63. RESOURCE_FINANCE. Do you agree or disagree that you have the resources (such as office space, office supplies, transportation, financial resources for services) necessary to do your work? 1. Strongly agree 2. Somewhat agree 3. Somewhat disagree 4. Strongly disagree 64. RESOURCE_PROF. Do you agree or disagree that you have the professional resources (such as supervision, training, and guides/forms) necessary to do your work? 1. Strongly agree 2. Somewhat agree 3. Somewhat disagree 4. Strongly disagree 65. SER_STRENGTH. Do you agree or disagree that the services provided by social workers strengthen and stabilize families? 1. Strongly agree 2. Somewhat agree 3. Somewhat disagree 4. Strongly disagree 66. SER_STRENGTH_ACT. How effective do you find are each of the following activities undertaken by social workers to strengthen and stabilize families? 1 = Very effective 2 = Somewhat effective 3= Not at all effective A. Case management in general from identification to case closure B. Assessment in particular C. Case or care planning D. Decision-making done with the family/child E. Psycho-social counselling F. Referral to other services ANNEX IV: DATA COLLECTION INSTRUMENTS | 198 1 = Very effective 2 = Somewhat effective 3= Not at all effective G. Monitoring visits – in office H. Monitoring visits – in home I. Direct support with financial or other materials assistance J. Accompany family to other services K. Other, specify 67. SKILL_PROTECT. To what extent do you believe the social work workforce in your raion/community has the necessary skills and knowledge to adequately protect children? 1. Highly skilled and knowledgeable 2. Somewhat skilled and knowledgeable 3. Not at all skilled and knowledgeable 68. SKILL_SUPPORT. To what extent do you believe the social work workforce in your raion/community has the necessary skills and knowledge to fully support vulnerable families? 1. Highly skilled and knowledgeable 2. Somewhat skilled and knowledgeable 3. Not at all skilled and knowledgeable PROG: END SECTION 4 (PLEASE COLLECT TIME STAMP) SECTION 5: ATTITUDES ON RESIDENTIAL CARE ENUMERATOR: PLEASE READ OUTLOUD. I will now ask you questions about residential care centers 69. TIME_RES_CARE. When might it be appropriate to place a child in a residential care facility? Choose all that apply. PROG: SELECT ALL THAT APPLY. 1. When the family is poor 2. When the child is failing school 3. In cases of juvenile delinquency 4. When the community has no services to offer 5. When the child has a disability 6. When the child is an orphan 7. When the child is left behind by migrant parents 8. Never 9. Other, please specify _ ________________ PROG: OPEN-ENDED FIELD 70. PARENTS_RES_CARE. To what extent do parents believe that residential care is acceptable for their children? 1. All parents believe it is acceptable 2. Most parents believe it is acceptable 3. Some parents believe it is acceptable ANNEX IV: DATA COLLECTION INSTRUMENTS | 199 4. No parents believe it is acceptable 71. COM_RES_CARE. To what extent does the community believe residential care is a good service to care for children? 1. All community members believe it is acceptable 2. Most community members believe it is acceptable 3. Some community members believe it is acceptable 4. No community members believe it is acceptable 72. COM_DEINST. To what degree is the community supportive of deinstitutionalizing children? 1. All community members are supportive 2. Most community members are supportive 3. Some community members are supportive 4. No community members are supportive PROG: END SECTION 5 (PLEASE COLLECT TIME STAMP) SECTION 6: OPINIONS ON THE CURRENT WELLBEING OF DEINSTITUTIONALIZED CHILDREN 73. DESINT_CHILD_ADAPT. In general, are the children who have been deinstitutionalized in your raion/community adapting well? 1. Mostly 2. Somewhat 3. Not at all 74. DEINST_CHILD_CHALLENGE. What, if any, are some particular challenges facing deinstitutionalized children and their caregivers? Please select all that apply. PROG: SELECT ALL THAT APPLY. 1. No challenges PROG: SKIP TO 77. WAYS_REINTEGRATE 2. Stigma from the community 3. Lack of access to school 4. Family risk factors (substance use, mental health issues, disability, income, etc.) 5. Attachment between the family and child is lacking 6. Access to services 7. Other, please specify: ________________ PROG: OPEN-ENDED 75. DESINST_CHILD_CHALLENGE_A. PROG: ASK IF 74. DEINST_CHILD_CHALLENGE!= 1. How are these challenges being addressed? Please select all that apply. PROG: SELECT ALL THAT APPLY. 1. With the social worker through the family care plan 2. Through referral to other services 3. Through a special education plan with the school 4. Counseling services for the child 5. Counseling services for the parents/caregiver ANNEX IV: DATA COLLECTION INSTRUMENTS | 200 6. Counseling for the family 7. Disability rehabilitation or other services 8. Work with the community 9. Other, please specify: ________________ PROG: OPEN-ENDED 76. DESINST_CHILD_CHALLENGE_B. PROG: ASK IF 74. DEINST_CHILD_CHALLENGE!=1. Are there any categories of deinstitutionalized children who face more of the above challenges than others? Please select all that apply. PROG: SELECT ALL THAT APPLY. 1. Girls 2. Boys 3. Children with disabilities 4. Children with learning disabilities 5. Children with chronic illnesses 6. Younger children 7. Older children 8. Children who are of an ethnic minority 9. Children from poor families 10. Other, please specify: ________________ PROG: OPEN-ENDED 77. WAYS_REINTEGRATE. In which of the following ways do you work with families of reintegrated children? Please select all that apply. PROG: SELECT ALL THAT APPLY. 1. Provide access to cash benefits 2. Provide other material assistance 3. Regular home visits, regular office visits 4. Visits to the child’s school or kindergarten 5. Represent the child at gatekeeping meetings 6. Arrange multidisciplinary meetings 7. Employment/household income support 8. Housing services 9. Referral to other services 10. Parent counseling 11. Parent education 12. Counseling for the child 13. Other, please specify: _______________ PROG: OPEN-ENDED ANNEX IV: DATA COLLECTION INSTRUMENTS | 201 78. DESINST_CHILD_RATE. Do you agree or disagree with the following: 1=Strongly agree 2=Somewhat agree 3=Somewha t disagree 4=Strongly disagree A. Deinstitutionalized children’s needs are being adequately met in their family placement B. Deinstitutionalized children’s needs are being adequately met in their school placement C. Deinstitutionalized children’s needs are being adequately met in the community D. Deinstitutionalized children’s needs are being adequately met by the services available E. Deinstitutionalized children are in protective family care that ensures their safety and wellbeing F. Deinstitutionalized children are placed in well-planned placements G. Deinstitutionalized children are placed with their permanency in mind H. Deinstitutionalized children are placed with adequate consideration to their best interests PROG: END SECTION 6 (PLEASE COLLECT TIME STAMP) SECTION 7: OPINIONS ON PARENTAL SKILLS 79. FAM_RISK_CAREGIVER. Of the families at-risk you work with, what proportion of caregivers… Of the families at-risk you work with, what proportion of caregivers… 1=All 2=Most 3=Half 4=Some 5=None A. Have positive discipline skills?(able to discipline their children without using physical punishment such as spanking or hitting) B. Have strong communication skills with their children C. Are knowledgeable about child development D. Are knowledgeable about child health issues E. Are aware of positive and negative patterns of behavior within the family? ANNEX IV: DATA COLLECTION INSTRUMENTS | 202 80. FAM_REINT_CAREGIVER. Of the families of reintegrated children you work with, what proportion of caregivers… Of the families of reintegrated children you work with, what proportion of caregivers… 1=All 2=Most 3=Half 4=Some 5=None A. Have positive discipline skills?(able to discipline their children without using physical punishment such as spanking or hitting) B. Have strong communication skills with their children C. Are knowledgeable about child development D. Are knowledgeable about child health issues E. Are aware of positive and negative patterns of behavior within the family? 81. FAM_RISK_CAP. Of the families at risk you work with, how would you rate the change in their capacity to care for their children as a result of involvement with services? 1. Highly improved 2. Somewhat improved 3. A little improved 4. No change 82. FAM_REINT_CAP. Of the families of reintegrated children you work with, how would you rate the change in their capacity to care for their children as a result of involvement with services? 1. Highly improved 2. Somewhat improved 3. A little improved 4. No change 83. CAREGIVER_LOWSKILL. In what areas do caregivers in your community generally have lower skills and knowledge? Please select all that apply. PROG: SELECT ALL THAT APPLY. 1. Positive discipline skills (able to discipline their children without using physical punishment such as spanking or hitting) 2. Strong communication skills with their children 3. Knowledge of child development 4. Knowledge of child health issues 5. Awareness of positive and negative patterns of behavior within the family 6. Other, please specify_______________ PROG: OPEN-ENDED ANNEX IV: DATA COLLECTION INSTRUMENTS | 203 84. CAREGIVER_SKILL_IMPROV. How effective are each of the following methods for increasing parenting skills and knowledge for the families with whom you work? 1= Very effective 2 = Somewhat effective 3 = Not at all effective A. Individual/one-on-one parenting education with a social worker B. Parenting classes/workshops C. Parent support groups D. Books, magazines, and other written resources 85. METHOD_PARENT_OTHER. Is there another effective method for increasing parenting skills and knowledge we have not mentioned? 1. Yes 2. No PROG: SKIP TO 87. POLICY_ADE 86. METHOD_PARENT_OTHER_OE. PROG: ASK IF 85. METHOD_PARENT_OTHER =1. What is this method? ______________________ PROG: OPEN-ENDED PROG: END SECTION 7 (PLEASE COLLECT TIME STAMP) SECTION 8. POLICIES AND LEGISLATION ENUMERATOR: PLEASE READ OUTLOUD I will now ask you question about policies and legislation. 87. POLICY_ADE. Do you agree or disagree that existing policies and legislation are being adequately implemented to support the functioning of the child protection system? 1. Strongly agree 2. Somewhat agree 3. Somewhat disagree 4. Strongly disagree 115. POLICY_ADE_DTL. PROG: ASK IF 87. POLICY_ADE =3 or 4. Which policies or legislation do you disagree with? Choose all that apply. 1. Family Code Of The Republic Of Moldova 2. Inter-sector cooperation mechanism for the identification, evaluation, referral, assistance and monitoring of child victims and potential child victims of violence, neglect, exploitation and trafficking 3. Law 140 on the Special Protection of Children at Risk and Children Separated from Parents 4. Law 338 on Children’s Rights 5. Law on Local Public Finance ANNEX IV: DATA COLLECTION INSTRUMENTS | 204 6. The Regulation Frameworks of the organization and functioning of the district and raional social assistance entities 7. Regulation for the Organization and Function of Raional and Municipal Psycho-pedagogical Services 8. Social worker guide to case management 9. Service regulations (example APP, adoption, shelters) 10. National Strategy for Child Protection 2014-2020 11. Law on adoption 12. Framework Regulations on the organization and functioning of the specialized local body in Social Assistance and family protection area and its structure (this is the official title of the newly approved Regulations for the functioning of the SAFPD - Gov. Decision 828) 13. Regulation for the Organization and Functioning of the Gate keeping Commissions (the official title is Commission for the protection of children in difficulty) 14. Regulations on Organization and Functioning of social services: Foster care service (APP); Social service for the families with children support (family support service); Family type children’s home; Temporary placement Centre for children without parental care 88. SYSTEM_RATE. Do you agree or disagree that the child care and protection system functions as well as it is supposed to? 1. Strongly agree PROG: SKIP TO 90. CHILD_ACT 2. Somewhat agree 3. Somewhat disagree 4. Strongly disagree 89. SYSTEM_RATE_A. PROG: ASK IF 88. SYSTEM_RATE !=1. What areas of the system need improvement? Choose all that apply. Do not read out answer options; code from response. PROG: SELECT ALL THAT APPLY. 1. National level policy and/or strategy 2. Raional level policy and/or strategy 3. Standards and guidelines for minimum package of services 4. Resource allocation – national level 5. Resource allocation – raional/local level 6. Workforce – adequate workforce to protect children 7. Workforce – adequate capacity of workforce to protect children 8. Coordination of actors in child protection 9. Collaboration between actors in child protection 10. Public awareness 11. Service development and implementation 12. Other, please specify: _____________ PROG: OPEN-ENDED 90. CHILD_ACT. Please list one national child protection policy/legal act that is important to child protection in your raion or community. Do not read out answer options; code from response. 1. Family Code Of The Republic Of Moldova ANNEX IV: DATA COLLECTION INSTRUMENTS | 205 2. Inter-sector cooperation mechanism for the identification, evaluation, referral, assistance and monitoring of child victims and potential child victims of violence, neglect, exploitation and trafficking 3. Law 140 on the Special Protection of Children at Risk and Children Separated from Parents 4. Law 338 on Children’s Rights 5. Law on Local Public Finance 6. The Regulation Frameworks of the organization and functioning of the district and raional social assistance entities 7. Regulation for the Organization and Function of Raional and Municipal Psycho-pedagogical Services 8. Social worker guide to case management 9. Service regulations (example APP, adoption, shelters) 10. National Strategy for Child Protection 2014-2020 11. Law on adoption 12. Framework Regulations on the organization and functioning of the specialized local body in Social Assistance and family protection area and its structure (this is the official title of the newly approved Regulations for the functioning of the SAFPD - Gov. Decision 828) 13. Regulation for the Organization and Functioning of the Gate keeping Commissions (the official title is Commission for the protection of children in difficulty) 14. Regulations on Organization and Functioning of social services: Foster care service (APP); Social service for the families with children support (family support service); Family type children’s home; Temporary placement Centre for children without parental care 91. FINANCE_RESOURCE_CHILDCARE. Are there resources in place to support the functioning of the child care and protection system? 1. Yes 2. No PROG: SKIP TO 95. RESOURCE_PROF 92. FINANCE_RESOURCE_CHILDCARE_A. PROG: ASK IF 91.FINANCE_RESOURCE_CHILDCARE ==1. Please list the resources available. Choose all that apply. Do not read out answer options; code from response. PROG: SELECT ALL THAT APPLY. 1. Financial resources for services 2. Financial resources for workforce hiring, training 3. Financial resources for workforce salaries 4. Office space 5. Technical resources (example computers, cell phones) 6. Transportation 7. Office supplies 8. Family contingency funds (for example to provide for specific urgent needs, such as school supplies or clothing) 9. Other, please specify _________________ PROG: OPEN-ENDED 93. FINANCE_RESOURCE_CHILDCARE_B. PROG: ASK IF 91. FINANCE_RESOURCE_CHILDCARE ==1. Are there additional resources needed that are not currently available? 1. Yes ANNEX IV: DATA COLLECTION INSTRUMENTS | 206 2. No PROG: SKIP TO 95. RESOURCE_PROF 94. FINANCE_RESOURCE_CHILDCARE_C. PROG: ASK IF 93. FINANCE_RESOURCE_CHILDCARE_B ==1. What are they? Choose all that apply. Do not read out answer options; code from response. PROG: SELECT ALL THAT APPLY. 1. Financial resources for services 2. Financial resources for workforce hiring, training 3. Financial resources for workforce salaries 4. Office space 5. Technical resources (example computers, cell phones) 6. Transportation 7. Office supplies 8. Family contingency funds (for example to provide for specific urgent needs, such as school supplies or clothing) 9. Other, please specify _________________ PROG: OPEN-ENDED 95. RESOURCE_PROF. Are there professional resources (such as social work supervision or training) in place to support the functioning of the child protection system? a. Yes b. No PROG: SKIP TO 99 CHILD_DEV_STRAT 96. RESOURCE_PROF_A. PROG: ASK IF 95. RESOURCE_PROF ==1. Please list the professional resources available. Choose all that apply. Do not read out answer options; code from response. PROG: SELECT ALL THAT APPLY. 1. Social work supervision 2. Initial training for new social workers 3. Ongoing / in-service training 4. Working methodologies (guides and forms) 5. Other, please specify _________________ PROG: OPEN-ENDED 97. RESOURCE_PROF_B. PROG: ASK IF 95. RESOURCE_PROF ==1. Are there additional resources needed that are not currently available? 1. Yes 2. No PROG: SKIP TO 99. CHILD_DEV_STRAT 98. RESOURCE_PROF_C. PROG: ASK IF 97. RESOURCE_PROF_B ==1. What are they? Choose all that apply. Do not read out answer options; code from response. PROG: SELECT ALL THAT APPLY. 1. Social work supervision 2. Initial training for new workers 3. Ongoing / in-service training 4. Working methodologies (guides and forms) 5. Other, please specify _________________ PROG: OPEN-ENDED 99. CHILD_DEV_STRAT. Does your raion have a child protection service development strategy? ANNEX IV: DATA COLLECTION INSTRUMENTS | 207 1. Yes 2. No 100. COUNCIL_FINANCE_CHILD. Does the raional council make financial resources available for child protection? 1. Yes 2. No 101. BUDGET_SER. To your knowledge are services for children and families included in the raional budget? 1. Yes 2. No 102. COUNCIL_SUPPORT_RATE. Do you agree or disagree that the raional council is supporting improvements in the child protection system? 1. Strongly agree 2. Somewhat agree 3. Somewhat disagree 4. Strongly disagree 103. CHILD_PROTECT_STRATE_IMPOV. What improvements in raional child protection service development strategy are needed? Please select all that apply. PROG: SELECT ALL THAT APPLY. 1. There is no strategy 2. Actors need to be more aware of the strategy 3. Strategy needs to be better aligned to national policy 4. Strategy needs to be reflect local needs and realities 5. Strategy should include plans for increased budget allocation 6. Strategy should include plans for workforce development 7. Strategy should include plans for development of services 8. Other, please specify _________________ PROG: OPEN-ENDED 104. RAION_AWARE_POLICY. Do you agree or disagree that the raional council is aware of national child protection policies? 1. Strongly agree 2. Somewhat agree 3. Somewhat disagree 4. Strongly disagree 116. MAYOR_AWARE_POLICY. Do you agree or disagree that the mayor(s) in the areas in which you work is/are aware of national child protection policies? 1. Strongly agree 2. Somewhat agree 3. Somewhat disagree ANNEX IV: DATA COLLECTION INSTRUMENTS | 208 4. Strongly disagree 105. MAYOR_FINANCE_CHILD_PROTECT. Does the local or community government (mayor) make financial resources available for child protection? 1. Yes 2. No 106. RAION_STR_RATE. Do you agree or disagree with the following: 1=Strongly agree 2=Somewhat agree 3=Somewhat disagree 4=Strongly disagree A. The raional structures in place for protecting children are effective B. The gatekeeping commission is effective in making best interests determinations in the placement of children C. The informal community structures in place for protecting children are important for helping children to be safe D. Family situations are too complex for me to be effective in my work SECTION 9. TRAINING RECEIVED ENUMERATOR: PLEASE READ OUTLOUD I will now ask you questions about training and capacity building you have received. 107. TRNG. Have you received any training related to your job from 2014 until now? 1. Yes 2. No PROG: SKIP TO Q114 ENUMERATOR’S COMMENTS 108. TRNG_TYPE. What was the subject of the trainings received? Please choose all that apply. PROG: SELECT ALL THAT APPLY AND ASK THE FOLLOWING QUESTIONS FOR EACH TRAINING SUBJECT SELECTED. 1. Implementation of the Law nr. 140 regarding the special protection of the children in risk and children separated from their parents. 2. Implementation of the Inter-agency cooperation mechanism on Identification, prevention and assistance of cases of abuse, neglect, exploitation and trafficking. 3. Provision of Family support service: standards of care and new Regulations. Application of the Case management and supervision tools. Application of the revised case management tool in Family support service. 4. Child Wellbeing and Family protective factors approaches. National Practice Model. ANNEX IV: DATA COLLECTION INSTRUMENTS | 209 5. Communication with the Mass media entities. 6. Development of PANDA program, role of the CSA in the implementation. 7. Other (specify)_____________ 109. TRNG_YR In what year did you receive the training? 1. 2014 2. 2015 3. 2016 4. 2017 110. TRNG_LGNT How many days of training did you receive? |_|_| number of days 111. TRNG_SKILLS On a scale of 1 to 4 where 1 is most useful and 4 is not at all useful, how useful was this training in improving your skills in the subject matter? |_| number 1-4 112. TRNG_PRVDR Who provided the training? 1. P4EC 2. Terre des hommes 3. Axa Management Consulting 4. Other non-government entity 5. Government entity 113. TRNG_IMPL Have you started implementing elements of this training PROG: CREATE BELOW TABLE WITH DROP DOWN MENUS SO THAT TRNG_YR, TRNG_LGNT, TRNG_SKILLS, TRNG_PRVDR, TRNG_IMPL ARE ASKED FOR EACH TRAINING SUBJECT SELECTED. 108. TRNG_TYPE (List of training subjects) 109. TRNG_YR Year of training (2014, 2015, 2016, 2017) 110. TRNG_LGNT # days 111. TRNG_SKILLS Scale of 1-4 112. TRNG_PRVDR Training Provider 113. TRNG_IMPL (Yes/No) 1. 2. 3. 4. 5. 6. 7. 114. ENUMERATOR’S COMMENTS ANNEX IV: DATA COLLECTION INSTRUMENTS | 210 FOCUS GROUP DISCUSSIONS WITH ADVISORY BOARDS OF CHILDREN Suggested number of participants per group: 6-8 members of ABCs, anticipated to come from the national board, and a board from a district (or combination of district members) Duration: This exercise should take around 45 minutes to an hour. Note: Ethical guidance applies here. Some possible ethical considerations are attached below this FGD outline. Required prior to the FGD: Signed parental consent forms for the child’s legal guardian and signed agreement to child protection policy from the moderator, including attending the child protection briefing. Required for the FGD: There must be two IMAS moderators for these FGDs, one facilitating discussions and the other taking notes. The facilitator needs to be able to engage fully with the participants, making it essential that all notes are taken by the other team member who does not actively engage in the FGD. Context of FGDs: The purpose of these focus group discussions is to explore the views of children/youth who are members of ABCs within the context of the endline evaluation of the P4EC program. The ABC mechanism is the specific topic of this FGD and its relationship to the evaluation questions. To collect before start of FGD from each participant: A. What is your age? B. How long have you been on the ABC? C. What city or part of Moldova are you from? 1. Could you please first briefly describe why ABCs exist? Probe on the main areas of individual and board responsibility 2. How is it that you came to be a member of the ABC? Probe on the inspiration and interest, who can be members, and the process for becoming a part of the board. 3. What does child participation mean and how does this apply to ABCs? Is this important for children. Why? 4. What preparation, information and training do ABC members (and the group as a whole) have in being able to function as the ABC? 5. In your communities, what are the reasons why children are separated from their parents? a. Is anything being done to prevent separation and, if so, is this working? ANNEX IV: DATA COLLECTION INSTRUMENTS | 211 b. Do you have suggestions for improving foster care for children? c. Are the numbers of children who cannot be cared for by their parents getting larger or smaller? Why? Probe on the monitoring and evaluation ABCs may be doing. 6. Please describe the situation of children reintegrating from residential care into family care. Probe on factors that prevent reintegration and what is most needed for this to occur successfully. a. Are children reintegrating with their families doing better? Why or why not? 7. Does the situation of at risk children get brought to your attention? If so, probe on how this occurs and probe on how ABC members get their information about at risk children, children outside of family care and those returning to families from RCCs, and if this information assists them in being part of the ABCs. 8. Who makes decisions about where children will be cared for? a. Are the opinions of children being sought? If so, how is this done and is this occurring adequately? b. Is this changing any? If so, how? c. Do you have suggestions on how children’s opinions can best be sought? Probe on child protection councils, raion meetings, gatekeeping. 9. Let’s talk about the organizations and individuals who assist children who are in RCCs, foster care and those in challenging situations in their families. a. Who are making the biggest contribution to the wellbeing of these children? b. Which services need to be strengthened? c. Are service providers working together very effectively? d. Has there been any recent changes in the ability of these organizations and individuals to help children? Describe. Probe on schools and teachers, health providers, social service entities, faith based organizations or leaders, NGOs, community leaders. 10. What policies and laws are important for the wellbeing of the children we are talking about? a. What are their strengths and weaknesses? b. Is your opinion ever sought in the policies or services that are being provided? Probe on how this occurs, i.e. child protection policies (local and national), child guide associated with the Guidelines on Alternative Care; how valued this is and if there are any changes that result. 11.Are there areas of child participation that need to be developed or more fully developed to assist at risk and vulnerable children? Please describe 12.Thus far, what have the ABCs accomplished? ANNEX IV: DATA COLLECTION INSTRUMENTS | 212 If not already come up probe on the role of ABCs in information dissemination, awareness building, child protection reporting. a. Have you benefited from being on the ABC? If so, how? b. What have you personally learned from being on the ABC? 13.What plans or ideas do you have for ABCs into the future? Probe on obstacles and how ABCs can best be supported to achieve these. a. Should ABCs be expanded or adapted in any way? b. What is essential for ABCs to exist and be productive into the future? 14.What would be three priority actions to help prevent children from being separated from their parents or help children who cannot live with their parents or families be better protected? a. Who should lead these actions? ANNEX IV: DATA COLLECTION INSTRUMENTS | 213 FOCUS GROUP DISCUSSIONS WITH CHILDREN MOLDOVA Children in at-risk families 12-14 age group, single gender (4 total – 2 new, 2 old) Children in at-risk families 15-17, single gender (4 total – 2 new, 2 old) Suggested number of participants per group: 6-8 children Duration: This exercise should take around 1.5 – 2 hours. It is important that children have refreshment and opportunities to get up and move around if needed. Note: Ethical guidance applies here. Some possible ethical considerations are attached below this FGD outline. Required prior to the FGD: Signed parental consent forms for the child’s legal guardian and signed agreement to child protection policy from the moderator, including attending the child protection briefing. Required for the FGD: There must be two IMAS moderators for these FGDs, one facilitating discussions and the other taking notes. The facilitator needs to be able to engage fully with the children and draw all the children out, making it essential that all notes are taken by the other team member who does not actively engage in the FGD. Context of FGDs: The purpose of these focus group discussions is to explore the views of children who are living within communities in vulnerable families about care arrangements. The objective is to elicit their assessment of the different forms of care arrangements that exist when parental care is either impossible (e.g. parental death) or not adequate (e.g. family abuse, poverty, parental migration), identifying the strengths and challenges of different forms of family-based care, the reasons why families might opt for a child to be in residential care and whether these reasons are valid in the children’s own view, and their views on what types of support from family and outside the family are mostly likely to promote child wellbeing. To create a safe space where children can feel free to share their ideas and insights, the tool will not require boys and girls to speak of their own individual experiences (although they may choose to do so). Instead its aim is to help them share with each other the experience of children more generally who live in a range of family-based care arrangements. ANNEX IV: DATA COLLECTION INSTRUMENTS | 214 TOOL 1: FIRE FIRE: IDENTIFYING DIFFERENT CARE ARRANGEMENTS Purpose: To break the ice, then identify which different living arrangements are known to participants Timing: 30 minutes Key steps 1. Introduce the facilitators and ask everyone to introduce themselves – use the exercise below or an alternative ice breaker as an introduction that the facilitator feels comfortable using: The two facilitators introduce themselves, giving their name and using an adjective to describe themselves that begins with the same letter e.g. I am Fantastic François; I am Turbo Tharcisse; then ask all the participants to introduce themselves describing the name. 2. Fire fire: Explain that the next game will get us all to move around and have some fun, and to start to hear from you about where children live in your community. a. Explain that the floor is on fire. People need to move around and try to keep their feet off the floor as much as possible, or they will get burnt. When you call out a number, they need to make an ‘island’ made up of that number of people (i.e. if the number called out is 3, then three people must join arms and they become an ‘island’ and can stand still because they are safe from the fire). b. Call out Fire Fire! Encourage them to run around, make sure that all the adults do so as well. c. Call out a number – e.g. Three! Depending on how many participants you have, try to choose a number that means they get into groups. When everyone is in an island of three people, ask them a question and say they must discuss the question together. First question must be something simple e.g. What was the most exciting thing that happened to me this week; What was the nicest thing that has happened to me this week. d. After 1-2 minutes, call out Fire fire! again. After they are moving around a bit, call out a different number. When all are in islands of that number, ask another ice￾breaking question e.g. If you could be an animal, what animal would you choose to be? What would you most like to be when you grow up? ANNEX IV: DATA COLLECTION INSTRUMENTS | 215 e. After 1-2 minutes, repeat the exercise. This time, get participants into groups of 2. When they have got into pairs (one group of 3 if there is an odd number), ask: ‘Who do you live with and how many people live in your home?’ 3. After they have had 1-2 minutes to talk about the question with their partner(s), go around the groups and ask everyone to say who they live with in their home. Write the answers down on some flip chart. 4. Once the answers are down, ask participants to sit back down and ask them if there are any other types of home in which children in their community are living. This could include other types of homes or living arrangements the children lived in previously. Add these to the list. If it has not already been mentioned, ask them about: living with one parent, both parents, extended family (who), someone in the community who is not a relative, children living as heads of household; residential care centers. 5. Discuss: - In general, which among these options is best for children? Which is worst? Why? - When do children usually live with their parents? Are there children who are more likely to live apart from their parents than others? (probe: children whose parents travel for work, children whose parents have abandoned them, children living with HIV, those with disabilities, girls, boys, younger, older, children who were having problems with their parents, children in families who are very poor, etc). Are there some groups of children who you think need a particular kind of family than others? (probe: children living with HIV, those with disabilities, girls, boys, younger, older, children who were having problems with their parents, children in families who are very poor, etc). What are these care preferences and on what basis are they chosen (probe: long-term, short-term)? Explain that we are now going to discuss a bit more what it is like for children who do not live with their own mother and/or father. ANNEX IV: DATA COLLECTION INSTRUMENTS | 216 TOOL 2: BODY MAP – IMPACT OF CARE ARRANGEMENTS ON CHILDREN’S WELLBEING Purpose: To explore children’s views about the positive and negative impact of different care arrangements on children. Timing: 30 minutes Options: Draw an outline of one of the participants on the ground (paper, draw in the dust) or have an outline of a body on a tablet. Key steps 1. Draw the image (one child is asked to volunteer to lie on the sheets to have the outline of his/her body drawn to create the outline of a body) or introduce the drawing of a body on a tablet(s). Key questions 2. Introduce the exercise – Ask the children to look at the different parts of the body. Explain that we want to discuss what children think, feel inside themselves, hear from others, see what is happening and do when they live in different situations. We are not asking what they feel about their own arrangement (although if they want to talk about this they can) but more in general about what children in their communities might feel. 3. Then follow the questions below starting with the community-based arrangement first mentioned by the child. Note the answers as they are discussed. Write down the key points on the paper / tablet a. (Pointing at the head) – The head is where we take decisions. Why do children live in the different settings that they have mentioned in Exercise 1 in the community? Who makes these decisions? - Why do children move to homes a different caregiver? Who usually makes the decision? Who moves the child / takes the child to the new home? Are certain ‘categories’ of children (i.e. age, gender, ability, etc) more likely to be placed in residential care? Which children are not brought to live in children’s homes? Why not? Where are the families of children who live in homes? b. (Pointing at the heart) The heart is about feeling. What are the good things about living in [the extended family – the family-based arrangement most commonly described by children]? What are the bad things? ANNEX IV: DATA COLLECTION INSTRUMENTS | 217 (Probe social, emotional, cognitive, health, nutritional and personal development impacts of the care arrangement, including for different ‘categories’ of children). c. (Pointing at the ear) What do you hear people say about the children who do not live with their parents? Do you agree with what is said? Why/why not? Also pointing at the ear are people in authority listening to the views of children about their situation and what they want to improve their lives? If not, why not? If so, who is doing this and how is this occurring? d. (Pointing at the hands) How are children who live in [other than extended family care] treated? (probe issues around stigma, discrimination, safety; probe whether there are different care arrangements that make it more likely for a child to be treated well or badly? e. (Pointing at the mouth) Do children have a chance to express their opinion in these decisions? If so, to whom and about what? 4. Having noted the key points about the family care arrangement, repeat with a new drawing / image on the tablet and ask about children who live in residential care or who have lived in residential care. Note, if this is an experience that none of the children find easy to imagine, skip this set of questions. a. (Pointing at the head) – Who decides about when a child must go to residential care? Who moves the child / takes the child to the new home? Are certain ‘categories’ of children (i.e. age, gender, ability, etc) more likely to be placed in residential care? Which children are not brought to live in children’s homes? Why not? Where are the families of children who live in homes? (Probe social, emotional, cognitive, health, nutritional and personal development impacts of the care arrangement, including for different ‘categories’ of children). b. (Pointing at the heart) What might be the good things about living in a residential care centre? What might be the bad things? (Probe social, emotional, cognitive, health, nutritional and personal development impacts of the care arrangement, including for different ‘categories’ of children). c. (Pointing at the ear) Do you hear anything said about the children who live in residential care centers? Do you agree with what is said? Why/why not? ANNEX IV: DATA COLLECTION INSTRUMENTS | 218 Also pointing at the ear are people in authority listening to the views of children about their situation and what they want to improve their lives? If not, why not? If so, who is doing this and how is this occurring? d. (Pointing at the hands) Do you know how children might be treated in the care center? Do you know how they may be treated when they come back to live with their family? (probe issues around stigma, discrimination, safety; probe whether there are different care arrangements that make it more likely for a child to be treated well or badly? e. (Pointing at the legs/feet) Do some children leave their children’s home? Why do they leave? (probe: family reunification, foster care, adoption, ageing out, run away, etc). If not all children leave the children’s home, then why not? (probe: why are some children not fostered or adopted? What are the challenges or barriers to being placed in family and community-based care, as opposed to residential care?). Are there any circumstances when a child has to leave the home? (probe: aging out of care and associated preparation for leaving care, expectations, maintaining contact). TOOL 3: SUPPORT FLOWERS (OPTIONAL, DEPENDING ON TIME) The purpose of this activity is to explore children’s views on who they and or their parents seek and/or gain support from, the kinds of support that they do and do not receive, and the kinds of support they wish to receive. Key steps 1. Ask children to get in pairs or groups of three. Give each group a big piece of paper and some pencils / or a table that they can draw on. Ask them to draw the centre of a flower in the middle of the paper. In the centre, they can draw pictures of themselves and/or their families. 2. Ask children to think about the different people (or organizations) who provide them and/or their caregivers and/or families with help and kindness. Give examples if necessary, i.e.: Who helps you when you feel sad? Who plays with you? Who helps you when you have a problem with school work? Who helps you when you have worries about the future? With whom do you share jokes? Who is kind to you? etc. Ask them to write the names/category (i.e. ‘parent’) of the person or draw a picture of each of these people in the area outside the flower. Ask them to think if there is anyone who supports their family including their caregivers or parents and to consider adding that to their pictures? Probe on what kind of support their caregivers/parents or others get, and if that helps them (the children)? Ask them to ANNEX IV: DATA COLLECTION INSTRUMENTS | 219 draw a petal around those who give children a lot of help, to show that they are part of the flower. For those who provide less support, smaller petal can be drawn to link them into the larger flower. 3. Support children to show their drawings with the larger group and explain to the rest of the group which people provide most support to them during difficult times and the kinds of support provided. Use this activity to enable a broader discussion on: The characteristics of those people that are most helpful or supportive as well as the characteristics that make it harder for some people to provide support to children. • The kind of support most sought by children • Any kind of support that is lacking – probe on some of the practical issues that may support their caregivers or themselves to stay together in a home. Suggestions or recommendations they may have for support, and why • The reasons why it can be difficult to get the desired/needed support (i.e. individual staff who are supportive but who cease working at the home, or someone who they would like to go to for advice in the local community but they are not allowed to leave the home to do so, etc). • The role that children can play in supporting one another when a child has a difficulty. Ethical procedure for conducting FGDs with children The following are the key steps required for undertaking these FGDs. These issues are to be discussed during training of moderators by NORC, with P4EC support and a revised, signed version will be agreed at the training: • All children who participate in the meetings will be informed about the purpose and content of the discussion before they arrive. All children who are invited have the right to choose not to participate. • If children choose to participate, their legal guardian will receive a consent form, explaining the purpose of the session. They have the right to not attend, even after agreeing and their legal guardian signs the consent; a right not to participate upon arriving at the FGD; and a right to stop participating at any time during the FGD itself. ANNEX IV: DATA COLLECTION INSTRUMENTS | 220 • All information in the discussion will be noted, but there will be full anonymity. Where quotations are used in any reports, identifying details will be removed, with only gender and age group information being documented in any published paper. • There will be a script to explain the purpose and use of the research. Each participant will have this explained before agreeing to participate and at the start of the session. Legal guardians will also be read the script and asked for signed consent before children take part in the focus group discussions. • Meetings will be held in a venue and at a time that is agreed to by caregivers and P4EC project staff as safe and suitable. Refreshments will be provided if deemed suitable. • All FGD facilitators will have received training on, and signed, P4EC’s code of conduct and NORC’s Pledge of Confidentiality. If moderators have any concerns about child protection breaches, they are to speak with the social workers linked to the family. • If the moderator has any concern regarding the emotional wellbeing of any child following the FGD, she can contact the social worker who can follow up. ANNEX IV: DATA COLLECTION INSTRUMENTS | 221 FOCUS GROUP DISCUSSIONS WITH PARENTS AND COMMUNITY￾BASED, FAMILY TYPE CAREGIVERS IN MOLDOVA This protocol will be adapted to at least three types of parents which may include those with children at risk of separation, parents of post-institutionalized children and foster parents (APP, CCTF) and, perhaps, parents of children in residential care. It may also be adapted to a key informant interview with care givers in family based group homes (CCTF). 1. What are some of the challenges that families in Moldova face in general that affects their ability to care for their children? 2. What kinds of skills, information or knowledge do you need to be good parents/caregivers of children? a. Why is this skill/information/knowledge important? b. Who can help parents obtain these skills/information/knowledge if they don’t already have it? c. Have skills, information and knowledge changed in recent years and, if so, as it been sufficient to meet family needs to care for their children? Verify where this is coming from. 3. From your own experience, and also your knowledge of others in the community, who is involved in making decisions about children who cannot, for whatever reason, be looked after by their parents? a. When and how does the child get involved in discussions? b. Who else is involved (direct family, extended family, mayor or community leaders, social workers). c. Are there any differences for different age/gender/disability, etc., of children? d. Has there been any change in this in recent years? Is it better, the same or worse…and why? 4. What kinds of support help you to be strong and effective in caring for children? a. Does this support exist? b. How easy is it to access this support? c. How effective is this support (provide examples)? Meaning, if there is impact on the family’s ability care for the children and if there are any improvements in the child’s wellbeing. d. What recommendations do you have for this support? e. Have you seen any changes in helping to support families in caring for children in recent years? (Explore) Probe on child-focused and family-focused interventions 5. Are your opinions being taken seriously by authorities or others about what is needed to improve your ability to care for children? Probe particularly on examples ANNEX IV: DATA COLLECTION INSTRUMENTS | 222 of when this has or has not occurred, and accountability of service providers to parents, caregivers and children. 6. In general, what do you feel would be the ideal form of care for children who cannot live with their parents? a. Can you give examples of where children are living with others and it is working really well? b. Are there examples of when it is not working well? (least favorite) c. What are the reasons for these different experiences? d. Have there been any changes in the good and bad forms of care in recent years? If so, why? 7. The national policy recommends that children do not live in residential care centers, as long as there are other family options available. a. What are your opinions of this policy? b. Is this policy having any impact on families? If so, how? c. Do you have recommendations on how policy or strategies can be more effective to help children, parents or other caregivers? 8. How easy is it for children returning from residential care centers to their family home or to another family member or foster care giver to reintegrate into the community? a. What challenges are faced by these children and their caregivers? b. How can you know whether a child is doing well or not? c. Are there some children that do better than others? (probe: boys/girls, disabled, ages) d. What kind of support is provided to families and e. In recent years, have prospects improved for children from residential centers to return to a family environment and integrate into the community? If so, how has this happened? 9. Let’s talk about children who are at risk of falling out of family care? If so, what kind of situations in families exist that lead to this risk? a. When should children stay in families and it might not be in their best interest to separate? b. What support can families and children receive for children to stay in their families if in their best interest? c. Is there such support in the community? If so, please describe it and if it is very effective? d. Do you know if anything has changed in recent years in the availability of support to families in these types of situations? Probe into reasons, and ask for examples. 10. We know that excess alcohol or drug use by family members can weaken families. Can you please describe how this may be affecting children in families? ANNEX IV: DATA COLLECTION INSTRUMENTS | 223 a. Is there any support to families who are affected by alcohol or drug abuse? If so, please describe the support and if has any results in the wellbeing of children in these families. 11.Are children/youth in this community participating in any formal or informal meetings or activities or efforts that enable the community to better understand the risks they face as children? Probe further here. If there is an ABC in this community, does anyone know about the group and what it does? More generally probe on the ability of children to express their views on issues that affect them, such as placement decisions, if they face challenges or crises in their families. Who can they go to? How do parents react to this? Is the ability of children to do this changing in recent years? Why or why not? 12.Have there been changes in attitudes, knowledge or support in your community about vulnerable children and families such as what we are talking about today? Probe on stigma, community leaders, FBOs, and the media. 13.If you could give one message to the people responsible for making sure that all children in Moldova are protected and about supporting the caregivers of vulnerable children, what would it be? ANNEX IV: DATA COLLECTION INSTRUMENTS | 224 FOCUS GROUP DISCUSSIONS WITH COMMUNITY SOCIAL ASSISTANTS IN MOLDOVA To collect before start of FGD from each participant: A. How many years have you been in your post as a community social assistant? B. How many localities do you cover? C. Are there other community social assistants in the same areas you are working in? 1. Could you please first briefly describe your work as a social assistant? Probe: Main areas of responsibility – child protection, family support, elderly, disabled, maintenance, probation work etc. 2. In your communities, are there many children who cannot be looked after by their parents? a. What are the reasons why children are separated from their parents? b. Are the numbers of children who cannot be cared for by their parents getting larger or smaller? Probe: Possible reasons they cannot be looked after by their parents might be because they have died or are sick, have had to move for work or, sometimes, because their own circumstances or behaviors make it hard for them to care for their children 3. In general, who cares for the child when biological parents cannot do so (NOTE: this is only an introduction question and you don’t need to probe too much)? Probe: Informal extended family care, foster care (APP), adoption, children are sent away. 4. Who makes the decision about where children will be cared for if they cannot remain in their families? a. Who is involved in the decision-making? b. Whose opinions are sought? c. Has this changed any since you’ve been in this type of work? How? Probe: child, him or herself, direct family, extended family, traditional or religious leaders, social workers 5. Do children do better in some types of alternative care arrangements than others? a. In which types of care do they fare the best? b. What are the core factors that make a care arrangement ‘successful’? c. In which types of care do they do less well? d. Are there any trends in this since you’ve been in this work? Probe: type of care arrangement, age of child, gender, disability, etc. ANNEX IV: DATA COLLECTION INSTRUMENTS | 225 6. How are cases brought to your attention? a. What are the main sources of referral (who brings these cases to your attention)? Trends? Probe: Examples may be informal referrals from families or schools; families seeking assistance for care with a child; neighbors who are concerned about a child’s wellbeing. 7. What happens to these children once their situation is brought to your attention? a. What are the steps taken once a situation is brought to your attention? b. Are there any formal procedures in place that are followed? c. What are the roles played by the different entities involved? What are their strengths and weaknesses in improving the wellbeing of the child? Probe: Explore their understanding of role of community multi-disciplinary teams, other local entities involved, raion specialists, gatekeeping commissions, extent of referral to social worker (raion-level experts) and other sources of support for family strengthening or alternative care. 8. What are the main services for children without appropriate care (ingrijirea neadegvata) in your raion? Probe: APP, CCTF, family support 9. Are there procedures for referral of children at risk of harm to social work case management? a. What do you do in these cases? b. What tools do you use? c. Are they effective? Probe on why or situations where they may not be. 10.What supports and services at the community level aimed at family support and strengthening exist? a. How effective are they at supporting and strengthening families? b. Which work the best? c. Which don’t work as well? Probe: community social assistants, the environment for cooperation, multi￾disciplinary team, gatekeeping, government offices, mayor, neighbors, Community Based Organizations, churches 11.To what extent do you feel that you have sufficient authority, training, and support to meet the needs of children at risk? a. To what extent do you feel that you have everything you need to do your job well? b. (In areas where don’t feel have everything) what would help you to do this work better? ANNEX IV: DATA COLLECTION INSTRUMENTS | 226 Probe: clarity of their role; clarity of responsibilities, level of workload, support, teamwork, supervision and training, resources. Probe also on training they have received, its impact and training priorities for the future. 12.The national policy recommends that children do not live in residential care centers, as long as there are other family options available. Do you agree that children should ideally stay in the community? a. If so, why? b. Is it ever appropriate for a child to be placed in a residential care center? (if yes) Please explain under what circumstances. 13.Do you think that children returning from residential care centers to their family home or to another family-based care arrangement might face challenges in the community? a. If so, what type of support to the child do you think might be most helpful? b. Who should provide this support? c. Are there any new or old procedures in place for this support? How effective are they? 14.Do you think that caregivers or families who receive a child who has been living in a residential care center may face challenges? a. What type of support to the family is most helpful? b. Who should provide this support? c. Are there any procedures in place for this support? Are they new or old procedures and how effective are they? 15.In the period you have worked as a CWS what are the most important lessons you have learned through training or experience working with families, children or others? 16.Based on the issues that you have discussed just now, what would you say are the most important and effective interventions that can help families care for children appropriately and reduce family separation? a. What do families need in order to care for their children? b. What do parents need to be better parents? Probe: community social worker and raion-level specialist’s role 17.If you could pick three priority actions that would make children who cannot live with their parents or families better protected, what would they be? a. Who should lead these actions? ANNEX IV: DATA COLLECTION INSTRUMENTS | 227 FOCUS GROUP DISCUSSIONS WITH RAION SOCIAL WORK SPECIALISTS IN MOLDOVA To collect before start of FGD from each participant: D. What is your position? E. How many years have you been in your post? F. What areas do you cover (all the raion or what is the number of localities)? 1. Could you please first briefly describe your work as a social worker? Probe: Main areas of responsibility – child protection, family support, elderly, disabled, maintenance, probation work etc. 2. In your raion, are there many children who cannot be looked after by their parents? a. What are the reasons why children are separated from their parents? b. Are the numbers of children who cannot be cared for by their parents getting larger or smaller? Probe: Possible reasons they cannot be looked after by their parents might be because they have died or are sick, have had to move for work or, sometimes, because their own circumstances or behaviors make it hard for them to care for their children 3. In general, who cares for the child when biological parents cannot do so (NOTE: this is only an introduction question and you don’t need to probe too much)? Probe: Informal extended family care, foster care (APP), adoption, children are sent away. 4. Who makes the decision about where children will be cared for if they cannot remain in their families? a. Who is involved in the decision-making? b. Whose opinions are sought? c. Has this changed any since you’ve been in this type of work? How? Probe: child, him or herself, direct family, extended family, traditional or religious leaders, social workers 5. Do children do better in some types of alternative care arrangements than others? a. In which types of care do they fare the best? b. What are the core factors that make a care arrangement ‘successful’? c. In which types of care do they do less well? d. Are there any trends in this since you’ve been in this work? e. Probe: type of care arrangement, age of child, gender, disability, etc. ANNEX IV: DATA COLLECTION INSTRUMENTS | 228 6. How are cases brought to your attention? a. What are the main sources of referral (who brings these cases to your attention)? Trends? Probe: Examples may be community social assistants, community multi-disciplinary teams, other local entities, informal referrals from other sources. 7. What happens to these children once their situation is brought to your attention? a. What are the steps taken once a situation is brought to your attention? b. Are there any formal procedures in place that are followed? c. What are the roles played by the different entities involved? What are their strengths and weaknesses in improving the wellbeing of the child? Probe: Explore their understanding of role of community multi-disciplinary teams, other local entities involved, gatekeeping commissions, extent of referral other sources of support for family strengthening or alternative care. 8. What are the main services for children without appropriate care (ingrijirea neadegvata) in your raion? Probe: APP, CCTF, family support 9. What supports and services at the community level aimed at family support and strengthening exist? a. How effective are they at supporting and strengthening families? b. Which work the best? c. Which don’t work as well? Probe: community social assistants, the environment for cooperation, multi￾disciplinary team, gatekeeping, government offices, mayor, neighbors, Community Based Organizations, churches 10.To what extent do you feel that you have sufficient authority, training, and support to meet the needs of children at risk? a. To what extent do you feel that you have everything you need to do your job well? b. (In areas where don’t feel have everything) what would help you to do this work better? Probe: clarity of their role; clarity of responsibilities, level of workload, support, teamwork, supervision and training, resources. Probe also on training they have received, its impact and training priorities for the future. ANNEX IV: DATA COLLECTION INSTRUMENTS | 229 11.The national policy recommends that children do not live in residential care centers, as long as there are other family options available. Do you agree that children should ideally stay in the community? a. If so, why? b. Is it ever appropriate for a child to be placed in a residential care center? (if yes) Please explain under what circumstances. 12.Do you think that children returning from residential care centers to their family home or to another family-based care arrangement might face challenges in the community? a. If so, what type of support to the child do you think might be most helpful? b. Who should provide this support? c. Are there any procedures in place for this support? How effective are they? 13.Do you think that caregivers or families who receive a child who has been living in a residential care center may face challenges? a. If so, what type of support to the family do you think might be most helpful b. Who should provide this support? c. Are there any procedures in place for this support? Probe on effectiveness of new or old procedures. 14.In the period you have worked as a SW what are the most important lessons you have learned through training or experience working with families, children or others? 15.Based on the issues that you have discussed just now, what would you say are the most important and effective interventions that can help families care for children appropriately and reduce family separation? a. What do families need in order to care for their children? b. What do parents need to be better parents? Probe: community social worker and raion-level specialists’ role 16.Have there been more recent and perhaps unexpected initiatives that have emerged that are innovative and have good prospects to improve the wellbeing of children, families and alternative care? Describe 17.If you could pick three priority actions that would make children who cannot live with their parents or families better protected, what would they be? a. Who should lead these actions? ANNEX IV: DATA COLLECTION INSTRUMENTS | 230 FOCUS GROUP DISCUSSIONS WITH GATEKEEPING COMMISSION MEMBERS MOLDOVA To collect before start of FGD from each participant: A. How many members are part of your gatekeeping commission? B. How long have you been a part of the gatekeeping commission? C. What is your profession? 1. Could you please first briefly describe your work as members of the Gatekeeping Commission? a. What is your role? b. What are your responsibilities? Probe: areas of activity e.g. protection decisions, material support, education access, awareness raising on children’s rights etc. 2. Who are the other actors you collaborate with in your work as a commission? a. How would you describe the collaboration? b. What are the positive aspects? c. What are the negative aspects? d. Has this changed much in the period of your involvement in the commission? 3. Who is involved in alternative placement decisions? Have there been any changes in approaches to decision-making and, if so, why? Probe: child him or herself, direct family, extended family, traditional or religious leaders, social workers 4. Under what circumstances is a child placed in a residential care center? Have there been any changes in circumstances and, if so, why? 5. In your experience, what determines whether an alternative care arrangement is successful? a. What is most important to ensure a successful arrangement? b. What challenges exist to a successful alternative care? Probe: type of care arrangement, age of child, gender, disability, etc. 6. To what extent do you feel that your commission has sufficient authority, training and recognition or support to meet the needs of children at risk? a. What would help you to do this work better? 7. In general, what do you feel would be the ideal form of care for children who cannot live with their parents? ANNEX IV: DATA COLLECTION INSTRUMENTS | 231 a. What are the advantages for children to living in the community as compared to within a residential care facility? If it is not apparent, probe on the type of community arrangements. b. What are the advantages to living in residential care facility? What areas of care and procedures in facilities require strengthening to improve the wellbeing of children? 8. In your opinion, what are the strengths of the current child care and protection system? Probe on evidence of these. 9. In your opinion, what are the weaknesses of the current child care and protection system? Probe on evidence of these. 10.In the period you have been on the Commission or involved in related work, what are the most important lessons you have learned of relevance to the Commission through training or experience working with other stakeholders, families, children or others? Probe on training they have received, its impact and training priorities for the future. 11. In the work of de-institutionalization, community-based alternatives for children and reducing risk of children separating from families, are there models and approaches for expansion, adaptation, and/or replication? 12.Have there been more recent and perhaps unexpected initiatives that have emerged that are innovative and have good prospects to improve the wellbeing of children, families and alternative care? 13.Are you concerned about structures that have more recently been established will be able to sustain themselves well into the future? Why or why not? Probe on ability for continued training, system development, and financial resource availability. 14.If you could pick three priority actions that would make families stronger in caring for their children, what would those be? 15.If you could pick three priority actions that would make children who cannot live with their parents or families better protected, what would they be? 16.Is there anything else you would like to discuss? ANNEX IV: DATA COLLECTION INSTRUMENTS | 232 FOCUS GROUP DISCUSSIONS WITH MAYORS IN MOLDOVA 1. How long have you been the Mayor? Prior to becoming Major, did you have much exposure or involvement in the situation of children at risk of being outside of family care and the challenges of residential care in the community? 2. In your communities, are there many children who cannot be looked after by their parents? a. What are the reasons why children are separated from their parents? b. Are the numbers of children who cannot be cared for by their parents getting larger or smaller? c. Has this changed in recent years? Probe on why or why not Probe: Possible reasons they cannot be looked after by their parents might be because they have died or are sick, have had to move for work or, sometimes, because their own circumstances or behaviors make it hard for them to care for their children 3. In general, who cares for the child when biological parents cannot do so? Probe: Informal extended family care, foster care (APP), adoption, children are sent away. 4. Do cases of children at risk get brought to your attention? a. If so, who brings them to your attention? Probe: Examples may be informal referrals from families or schools; families seeking assistance for care with a child; neighbors who are concerned about a child’s wellbeing, collaboration with other duty bearers. b. What happens to these children once their situation is brought to your attention? c. Have there been changes in recent years in the role and activities of the municipalities in both prevention and responding to the situations of at risk children. If so, how has this come about? Probe: Explore understanding of role of community social assistants, community multi-disciplinary teams, other local entities involved, raion-level social workers, gatekeeping commission, and other sources of support for family strengthening or alternative care. Identify any programs providing alternative care. Probe for understanding of mayor’s mandate. 5. Do children do better in some types of alternative care arrangements (when they cannot be cared for by their biological parents) than others? a. In which types of care do they do the best? b. In which types of care do they do less well? ANNEX IV: DATA COLLECTION INSTRUMENTS | 233 c. Are some care arrangements better than others for children of different ages, genders or facing particular issues? Probe: what are the core factors that make an alternative care arrangement ‘successful’? 6. The national policy recommends that children do not live in residential care centers, as long as there are other family options available. Do you agree that children should ideally stay in the community? a. If so, why? b. Is it ever appropriate for a child to be placed in a residential care center? (if yes) Please explain under which circumstances. 7. Do you think that children returning from residential care centers to their family home or to another family-based care arrangement might face challenges in the community? a. If so, what type of support to the child do you think might be most helpful? b. Who should provide this support? c. Are there any procedures in place for this support? 8. Do you think that caregivers or families who receive a child who has been living in a residential care center may face challenges? a. If so, what type of support to the family do you think might be most helpful b. Who should provide this support? c. Are there any procedures in place for this support? 9. To what extent do you feel that families have the skills and knowledge they need to care for their children? a. What actions do you believe would help parents and families to become better in their protection and care of children? 10.In recent years there have been a number of new national strategies, action plans and laws related to children living in and out of their biological families – and capacity building to support these frameworks. What impact have you seen as a result? Probe on strengths, weaknesses, gaps and momentum. 11.In the period you have been Mayor or involved in municipality leadership, what are the most important lessons you have learned through capacity building, working with other duty bearers, families, children or others? 12.In the work of de-institutionalization, community-based alternatives for children and reducing risk of children separating from families, are there models and approaches for expansion, adaptation, and/or replication? Probe to see if there are any unique initiatives in this municipality in novel and effective work of other NGOs/CSOs ANNEX IV: DATA COLLECTION INSTRUMENTS | 234 13.Are you concerned about structures that have more recently been established will be able to sustain themselves well into the future? Why or why not? Probe on ability for continued training, system development, and financial resource availability. 14.What would be priority actions to help prevent children from being separated from their parents? 15.What would be priority actions that would make children who cannot live with their parents or families better protected? ANNEX IV: DATA COLLECTION INSTRUMENTS | 235 FOCUS GROUP DISCUSSIONS WITH MULTI-DISCIPLINARY PROFESSIONALS *Make sure have a few teachers in each group. Need to identify who specifically should be in this group: TSAS, PPAS, CSAS, CPCD, community health workers or the equivalent, who else last time To collect before start of FGD from each participant: D. What is your position? E. How many years have you worked in this post? F. How long have you been part of the multi-disciplinary team for child care, protection, and safety? 1. Could you please first briefly describe your work in the community? a. What are your main areas of responsibility? 2. What is your work related to child care, protection and safety? a. How much of this work is part of the multi-disciplinary team? Probe into the functioning of the multi-disciplinary team; how it works, how frequently it meets what it accomplishes b. What do you do beyond your activities as a member of the multi-disciplinary team? 3. In your communities, are there many children who cannot be looked after by their parents? a. What are the reasons why children are separated from their parents? b. Are the numbers of children who cannot be cared for by their parents getting larger or smaller? Why? Probe: Possible reasons they cannot be looked after by their parents might be because they have died or are sick, have had to move for work or, sometimes, because their own circumstances or behaviors make it hard for them to care for their children 4. In general, who cares for the child when biological parents cannot do so (NOTE: this is only an introduction question and you don’t need to probe too much)? Probe: Informal extended family care, foster care (APP), adoption, children are sent away. 5. Who makes the decision about where children will be cared for if they cannot remain in their families? a. Who is involved in the decision-making? b. Whose opinions are sought? ANNEX IV: DATA COLLECTION INSTRUMENTS | 236 c. Has this process changed in recent years? Probe: child him or herself, direct family, extended family, traditional or religious leaders, social workers 6. Do children do better in some types of alternative care arrangements than others? a. In which types of care do they fare the best? b. What are the core factors that make a care arrangement ‘successful’? c. In which types of care do they do less well? d. Are there new trends since you have been doing this work? Probe: type of care arrangement, age of child, gender, disability, etc. 7. Do cases of children at risk get brought to your attention? a. If so, who brings them to your attention? b. Has this changed in recent years? If so how? c. What are the steps taken once a situation is brought to your attention? d. Are there any formal procedures in place that are followed? e. What are the roles played by the different entities involved? What are their strengths and weaknesses in improving the wellbeing of the child? Probe: Examples may be community social assistants; informal referrals from families or schools; families seeking assistance for care with a child; neighbors who are concerned about a child’s wellbeing. Probe: Explore their understanding of inter-agency cooperation, other local entities involved, community social workers, raion specialists, gatekeeping commissions, extent of referral to social worker (raion-level experts) and other sources of support for family strengthening or alternative care. 8. What supports and services at the community level aimed at family support and strengthening exist? a. How effective are they at supporting and strengthening families? b. Which work the best? c. Which don’t work as well? d. Has this changed in recent years? Probe: community social assistants, environment for cooperation, multi-disciplinary team, government offices, mayor, neighbors, Community Based Organizations, churches 9. To what extent do you feel that you have sufficient authority, training, and support to meet the needs of vulnerable children? a. To what extent do you feel that you have everything you need to do your job well? ANNEX IV: DATA COLLECTION INSTRUMENTS | 237 b. (In areas where they don’t feel they have everything) what would help you to do this work better? Probe: clarity of their role; clarity of responsibilities, level of workload, support, teamwork, supervision and training, resources. 10.The national policy recommends that children do not live in residential care centers, as long as there are other family options available. Do you agree that children should ideally stay in the community? a. If so, why? b. Is it ever appropriate for a child to be placed in a residential care center? (if yes) Please explain under what circumstances. 11.Do you think that children returning from residential care centers to their family home or to another family-based care arrangement might face challenges in the community? a. If so, what type of support to the child do you think might be most helpful? b. Who should provide this support? Probe on effectiveness of what’s been tried and any new types of support. 12.Do you think that caregivers or families who receive a child who has been living in a residential care center may face challenges? a. If so, what type of support to the family do you think might be most helpful b. Who should provide this support? Probe on effectiveness of what’s been tried and any new types of support. 13.In the work of de-institutionalization, community-based alternatives for children and reducing risk of children separating from families, are there noteworthy models and/or innovative approaches for expansion, adaptation, and/or replication? Please describe. Probe to see if there are any unique initiatives in this jurisdiction in novel and effective work of other NGOs/CSOs? 14.In the period you have worked on the multi-service team what are the most important lessons you have learned through training or experience working with families, children or others? 15.To what extent do you feel that families have the skills, knowledge and support they need to care for their children? a. What do you think families need to be able to care for their children? b. What do parents need to be better parents? c. how has this changed in recent years? Probe: community social worker and raion-level specialist’s role ANNEX IV: DATA COLLECTION INSTRUMENTS | 238 16.What would be three priority actions to help prevent children from being separated from their parents? a. Who should lead these actions? 17.What would be three priority actions that would make children who cannot live with their parents or families better protected? a. Who should lead these actions? ANNEX IV: DATA COLLECTION INSTRUMENTS | 239 FOCUS GROUP DISCUSSIONS WITH PARENTS MOLDOVA 6 FGD: 3 in “old” and 3 in “new” districts Likely types of groups: parents of children at risk of separation, parents of post institutionalized children, parents of children currently in institutions, foster parents 1. What are some of the challenges that families in Moldova face in general that affects their ability to care for their children? 2. What kinds of support help families to be strong in caring for children? a. How does this support help them? b. Who provides or should provide it? c. How easy is it to access this support? d. What is your opinion of the quality of this support? e. How effective is this support (provide examples)? Meaning, if there is impact on the family’s ability care for the children and if there are any improvements in the child’s wellbeing. f. Have you seen any changes in helping to support families in caring for children in recent years? (Explore) Probe: community support, extended family, service provision or facilitating access? Identify child-focused and family-focused interventions 3. What kinds of skills, information or knowledge do families need to be good caregivers to their children? a. Why is this skill/information/knowledge important? b. Who can help parents obtain these skills/information/knowledge if they don’t already have it? c. Have skills, information and knowledge changed in recent years and, if so, as it been sufficient to meet family needs to care for their children? Verify where this is coming from. 4. From your own experience, and also your knowledge of others in the community, who is involved in making decisions about children who cannot, for whatever reason, be looked after by their parents? a. When and how does the child get involved in discussions? b. Who else is involved (direct family, extended family, mayor or community leaders, social workers). c. Are there any differences for different age/gender/disability, etc., of children? d. Has there been any change in this in recent years? Is it better, the same or worse…and why? 5. In general, what do you feel would be the ideal form of care for children who cannot live with their parents? ANNEX IV: DATA COLLECTION INSTRUMENTS | 240 a. Can you give examples of where children are living with others and it is working really well? b. What has made it work well? c. Are there examples of least favorite options? d. What makes this so and when and why do you think this happens? e. Different perspectives on care – what are the reasons for different views e.g. type of child, type of family? f. Have there been any changes in the good and bad forms of care in recent years? If so, why? 6. The national policy recommends that children do not live in residential care centers, as long as there are other family options available. Have you heard of this policy? a. What are your opinions of this policy? b. Is this policy having any impact in families that you know and in the community? If so, how? 7. How easy is it for children returning from residential care centers to their family home or to another family member to reintegrate into the community? a. What challenges are faced by these children and their caregivers? b. How can you know whether a child is doing well or not? c. Are there some children that do better than others? (probe: boys/girls, disabled, ages) d. In recent years, have prospects improved for children from residential centers to return to a family environment and integrate into the community? If so, how has this happened? 8. Do you know what kind of support is provided to families when a child is returned home from residential care centers? Probe into these, ask for illustrations, and what impact if any this has had and any suggestions to help make the return successful 9. In your community are there children who are at risk of falling out of family care? If so, what kind of situations in families exist that lead to this risk? a. When should children stay in families and it might not be in their best interest to separate? b. What support can families and children receive for children to stay in their families if in their best interest? c. Is there such support in the community? If so, please describe it and if it is very effective? d. Do you know if anything has changed in recent years in the availability of support to families in these type of situations? Probe into reasons, and ask for examples. 10.We know that excess alcohol use by family members can weaken families. Can you please describe how this may be affecting children in their families? ANNEX IV: DATA COLLECTION INSTRUMENTS | 241 a. Is there any support to families who are affected by alcohol or drug abuse? If so, please describe the support and if has any results in the wellbeing of children in these families. 11.Are children/youth in this community participating in any formal or informal meetings or activities or efforts that enable the community to better understand the risks they face as children? Probe further here. If there is an ABC in this community, does anyone know about the group and what it does? More generally probe on the ability of children to express their views on issues that affect them, such as placement decisions, if they face challenges or crises in their families. Who can they go to? How do parents react to this? Is the ability of children to do this changing in recent years? Why or why not? 12.If you could give one message to the people responsible for making sure that all children in Moldova are protected, about supporting the caregivers of vulnerable children, what would it be? ANNEX IV: DATA COLLECTION INSTRUMENTS | 242 FOCUS GROUP DISCUSSION WITH CHILDREN IN MOLDOVA PANDA PROJECT PANDA is the Program for Alcohol Non-dependence Assistance for children and adults living in households in which there is a member who abuses alcohol. Moldova has the second highest rate of alcohol consumption in the world according to some sources. Alcohol abuse is a significant contributing factor to family conflicts and vulnerability, including the ability to parent. The focus of PANDA is not directly helping the alcoholic to recover, but on helping family members cope and stay safe. In PANDA there is a structured program of 15 sessions, each with a particular theme. PANDA has been initiated for children in nine districts and groups for parents have also started. Composition of Focus Group: 13-18 year age group, mixed gender, participants in PANDA Duration: This exercise should take around 1.5 hrs. Ethical guidance: Since the children are in PANDA it is obvious they have experienced abuse of alcohol in their families. This is a sensitive subject. The questions are worded indirectly and it is important to stick to this to make it less personal and in a safe participatory space so the children can be more comfortable to share their ideas and insights. Required prior to the FGD: Signed parental consent forms for the child’s legal guardian and signed agreement to child protection policy from the moderator, including attending the child protection briefing. Required for the FGD: There must be two IMAS moderators for these FGDs, one facilitating discussions and the other taking notes. The facilitator needs to be able to engage fully with the children and draw all the children out, making it essential that all notes are taken by the other team member. Context of FGD: The purpose of the discussion exercise is to explore the experiences and perceptions of children to elicit their assessment of PANDA for the endline evaluation. A Body Map Tool is used to explore children’s wellbeing in addressing alcohol abuse through the experience of participating in PANDA. ANNEX IV: DATA COLLECTION INSTRUMENTS | 243 The Process: The FGD begins with an explanation of the purpose of the discussion: to learn from children about their experiences with PANDA in addressing alcohol abuse in Moldovan families. The moderator will draw an example of a body map (maybe a PANDA?) on the newsprint and then describe we are going to be discussing what children think and learn (head), feel inside themselves (heart), hear and see from others, what they can do (hands and feet). Importantly, we are not necessarily asking what they feel about their own experience (although if they want to talk about this they can) but more in general about what children in their communities might feel. It is good to emphasize to the children that their input will provide information to help strengthen services for families impacted by alcohol. The moderator will talk about this being a participatory exercise, what will be used with the information, its confidentiality, etc. (the endline format for children FGDs) Step One: Introductions and Breaking the Ice Have the children break up into 2 -4 groups depending on the number of participants. In each group, each participant will be asked to introduce themselves to each other. First they will say something about what their favourite animal is and why (build off of the PANDA concept, actually a most cherished animal that lives in China but the whole world wants to protect). Then they will describe indicate how long they have been involved with PANDA, if possible how many sessions. Coming back to the group each will introduce each other or one person can serve as the spokesperson. Step Two: Drawing the body map and some initial questions The youth will again be divided up and given poster paper. First they will select one of the participants to serve as the model to draw an outline of their body on the poster paper on the ground. Once they have done this, they will be asked a number of questions listed below. They will have time to discuss among themselves and then report back to the larger group. The moderator will jot down some key words in response at the point of the body being discussed. The moderator will ask some follow up questions to the group for overall input from the group. ANNEX IV: DATA COLLECTION INSTRUMENTS | 244 Key questions 1. (Pointing at the head) – “The head is where we make decisions and how we learn about things.” How do children find out about PANDA? (for example, did someone ask them to, word of mouth, referral) And why do they decide to participate? After reporting back facilitate a group discussion on: a) What are obstacles to children participating in PANDA b) Why do some children drop out of PANDA and some decide to stay? 2. (Pointing at the eyes) “With the eyes you can see things happening, observe.” When there is alcohol use in a family and the community, what do you see in how this affects the lives of children? In the group discussion: Probe on what are the most difficult impacts on children – ask for examples particularly in safety, education, parenting support, financial situations, and others that children may raise 3. (Pointing at the heart) “The heart is about feeling, emotions” How does alcohol abuse in a family make children feel? (note this is a very general question, it can be prompted by asking them to think about emotions such as sadness, anger, confusion, guilt Group discussion: a) Probe as to what causes the specific emotions and feelings, have children give examples b) Have there been any changes in feelings of children and parents involved in PANDA about alcohol use? c) Has changes in feelings cause any changes in behaviours in dealing with alcohol issues? How so? 4. (Pointing at the ear) “Ears allow us to hear things and to listen and also to learn.” What have children learned about alcohol abuse in PANDA? a) Probe on what the children have heard and learned about alcohol issues that was not known before PANDA. b) Please describe the type of group activities that occurs with PANDA for learning. c) What were the most and least helpful for children in PANDA activities? d) Back to listening, are people in authority listening to the views of children about their situation and what they want to improve their lives? If not, why not? If so, who is doing this and how is this occurring? ANNEX IV: DATA COLLECTION INSTRUMENTS | 245 e) Are parents listening in any new and helpful ways, to children or others? 5. (Pointing at the hands and legs) “These allow us to make things, do things, to go places” Are there places or persons children can go to for help from people they can trust and be helpful? (who and what support is available and trustworthy) a) Is this new support? Please to identify where it came from, how it came about. Probe on if any support existed before this. Identify and ask about how the differences to the new support. b) What results were there from this these forms of support? Try to get some examples that they have heard about. c) Is there anything that needs to improve or strengthen in the support that now exists to improve the lives of children and parents? 6. (Pointing at the mouth) “Speaking helps people to explain things, express opinions and feelings, and also to communicate effectively.” Do children in difficult situations related to alcohol abuse, express their feelings about what they are going through? Who can they talk to? a) What hinders and helps children to communicate their feelings and challenges? b) Is there stigma attached for children if they are in families where there is alcohol abuse? Is this changing, why or why not? c) Has involvement with PANDA helped to raise awareness among friends or others in the community on alcohol abuse? After the participants have finished reporting on the questions above, and the give and take after each key question, the group will be assembled as whole and the following questions will be addressed with the entire group – referencing the different body maps that are on the walls or on frames: 7. Lets talk now about situations where there is alcohol abuse in families and family unity may be threatened and there may be separation of parents or children from families. Do you heard about these situations? If so, please describe. a) do you think programs like PANDA lowers the risk of this separation from occurring, if so, how? 8. When children are around parents who have had too much alcohol, what can they do lower the chance there will be physical or emotional hardship to children? Probe if there are any new behaviors they have learned 9. Is there much change in families who have had children and/or parents in PANDA? Please describe. ANNEX IV: DATA COLLECTION INSTRUMENTS | 246 10.What has been most important thing for children participating in PANDA? (Get each participant’s perspective) 11. What recommendations do you have for PANDA in its activities to improve the wellbeing of children and their families in dealing with alcohol abuse? ANNEX V: SOURCES OF INFORMATION | 247 ANNEX V: SOURCES OF INFORMATION ANNEX V: SOURCES OF INFORMATION | 248 1. Survey: A survey of social workers and community social assistants targeted by the project was conducted in Moldova. 300 respondents participated in this survey. The survey consists of 8 sections. • Section 1: Demographics. The goal of this section is to draw out basic demographic information, including gender, age, education, raion/community location, and years the social workers have been at the position. • Section 2: Opinion on supports of children. This section asks a series of questions about the services that are available in the raion/community; whether such services are adequate and needed; and whether they can be improved. • Section 3: Opinions on coordination and collaboration and other service providers. This section asks questions about different actors of the childcare and protection system. It also elicits responses on their rating and value assessment of the service providers. • Section 4: Social workers’ capabilities and practices. This section asks a series of questions about the social workers’ assessment of the vulnerabilities of the families, the tools social workers employed, their work load, frequency of contact with the families, job responsibilities, and their self-assessment. • Section 5: Attitudes on residential care. This section asks a series of questions about the social workers’ attitude towards residential care, and their perception of community’s support towards deinstitutionalization. • Section 6: Opinions on the current wellbeing of deinstitutionalized children. This section asks a series of question about the adaptability of deinstitutionalized children, the challenges these children are facing, and how the social workers are working with families to help the children. • Section 7: Opinions on parental skills. This sections ask a series of questions about the skills of caregivers and their assessments on the methods they are using to improve caregivers’ skills. • Section 8: Policy and Legislation. This section asks different questions about social’s workers’ opinion of existing policies and legislation related to childcare and protection in terms of adequacy, functionality, rating, etc. This section also assesses their knowledge and awareness of the resources available to them. • Section 9: Training received. This section asks about training and capacity building received. 2. Qualitative Data: We conducted a series of Key Informant Interviews as well as Focus Group Discussions 3. Data from P4EC: P4EC transferred the following datasets to NORC. (a) Vulnerable Children Database. This database included case file information for all 10 project raions: Cahul, Calarai, Causeni, Falesti, Nisporeni, Orhei, Singerei, Soroca, Telenesti, and Ungheri. The file contains the following information: ANNEX V: SOURCES OF INFORMATION | 249 1. General demographic data about the child 2. Evaluation of the child with respect to child care, psycho-emotional, education, safety, and health 3. Evaluation of the family examining issues such as housing, employment, income and any family problems 4. Child support plan including child care and psycho-social care, education, health care, and family support plan 5. Follow-up case review on point 4 above. (b) Deinstitutionalized Children Database. This database included information on children in 3 RCCs (Ciniseuti/Rezina, Nisporenti, and Visoca-Soroca), and contains the following information: 1. Basic demographic information: Gender, data of birth, data of entry to RCC, expected date of leaving RCC, district, addresses, reasons for living in RCC, siblings’ information 2. Education: how well the child is doing in school, educational progress, how socially accepted and supported the child feels within the educational setting (by teachers and peers) and within the community and home, special support for children with disabilities, 3. Safety: the child’s perception of safety at the institutions, schools, and communities, hardwork that can affect development of the child 4. Health: disability status, health status, registration with family doctors, frequency of checkup, receipt of treatment when necessary 5. Psychosocial component of well-being: child’s self-care, social skills, negative emotions/aggressive behaviors, contact with family/parents/caregiver, frequency of contact with family/parents/caregiver, opinions about living in RCC 6. Family status: family status, parents’ status, family issues, source of income 4. List of Documents Reviewed References Chapman, Jenifer (2013). "Measure Evaluation Core Ovc Program Impact Indicators." USAID, PEPFAR, MEASURE Evaluation. Save the Children (2012). "Alternative Care in Emergencies Toolkit." DFID & SIDA (2009). "Practival Guide - System for Prevention of Child Separation from the Family." edited by Family Ministry of Social Protection, and Child. Moldova. International Child Fund (2010). "Child- and Youth-Friendly Participatory Action Research Toolkit." Richmond, VA. Aaron Greenberg (2010). "Families, Not Orphanages." Save the Children. ANNEX V: SOURCES OF INFORMATION | 250 Langfod, Judy (2009). "Helping Families Get What They Need to Support Their Children's Development." Center for the Study of Social Policy. Ministry of Social Protection, Family, and Child of the Republic of Moldova (2013). "Social Assistant's Guide." P4EC (Dec, 2014). "P4EC Quarterly Report " Children in Moldova Are Cared for in Safe and Secure Families"." Moldova: P4EC. National Scientific Council on the Developing Child and National Forum on Early Childhood Policy and Programs (2010. "The Foundations of Lifelong Health Are Built in Early Childhood." Harvard University. "Protective Factors Surve." FRIENDS National Resource Center for Community-Based Child Abuse Prevention, University of Kansas Institute for Educational Research & Public Service Center. Resolution Adopted by the General Assembly. "Guidelines for the Alternative Care of Children." National Registry of Evidence-Based Programs and Practices (2007). "Strengthening Families Program." Zeanah, Charles H., Charles A. Nelson, Nathan A. Fox, Anna T. Smyke, Peter Marshall, Susan W. Parker, And Sebastian Koga (2003). "Designing Research to Study the Effects of Institutionalization on Brain and Behavioral Development: The Bucharest Early Intervention Project." Development and Psychopathology 15, no. 04: 885-907. NGO Working Group on Children without Parental Care. (2013). Identifying Basic Characteristics of Formal Alternative Care Settings for Children: A Discussion Paper. Bradford, N.B. (2014). Protecting Children in Moldova from Family Separation, Violence, Abuse, Neglect and Exploitation: Final Evaluation Report. Martin, F. Deinstitutionalizing the Alternative Care System for Children. 14th Webinar Social Service Workforce Strengthening Webinar Series. UNICEF and Oxford Policy Management. (2009). Assessment Of The Child Care System In Moldova And Technical Assistance To The Government Of Moldova To Host The Sub-Regional Consultation On Child Care System Reforms In ENP Countries. DCOF (2013). Flier for Presentation on Child Care Reform in Moldova. Evans, P. and Bradford, N.B (2013). Every Child Moldova’s Programme Experience: Improving Children’s Lives through Deinstitutionalisation. McCreery, K.B (2013). Foster Care Services for Children in Moldova: Report to UNICEF. Every Child (2009). Final Evaluation Report on the Implementation of the Foster Care in Chisinau Municipality Project. Larter, D. and Veveritsa, E. (2005). Services for Children in Difficulty: Residential Care / Community-Based Services. ANNEX V: SOURCES OF INFORMATION | 251 Larter, D. and Veveritsa, E. (2005). Expenditures on the Residential Care of Children in the Republic of Moldova: A Financial Analysis. Bilson, A. (2006). Evaluation of Raional Gatekeeping System: The Commission for Child Protection. Evans, P. (2009). Improving Gatekeeping Functions of Child Protection Systems. Better Care Network. (N.D.). Moldova Country Care Review Global Alliance for Children (2014). Moldova Trip Report. Every Child and IMAS. (N.D.) Protection of the Rights of Children Deprived of Parental Care or who are at risk of Being Separated from Their Families: Qualitative Study. IMAS (2013). Protecting Children of Moldova from Family Separation, Violence, Abuse, Neglect and Exploitation: Research Report. Family for Every Child (2014). Children’s Reintegration: Longitudinal Study of Children’s Reintegration in Moldova. USAID (2009). The Job that Remains: An Overview of USAID Child Welfare Reform Efforts in Europe and Eurasia. Government of Moldova Documents Ministry of Education, Order 255 from 25 April 2012 on revision of the list of residential institutions including the plan for transformation (Romanian) Minimum Standards of Quality for the Functioning of Foster Care Service, 2008 (English translation) Briefing Note for the Minister of Finance: Controlling the Costs of Residential Care (2006) Government Decision 784 from 9 June 2007 Annex document: Plan of Action on the Reform of the Residential Care System for Children. 2007-2012 (Romanian) Government Decision 924 from 31 December 2009 on Allocations for Children Placed in the Professional Parental Assistance Service (Romanian) Law 123 on Social Services, 2010 (Romanian) Government Decision 1479 of 25 December 2008 on Minimum Standards of Quality for the Professional Parental Assistance Service (Romanian) Ministry of Education, Order 444 of October 2009 on Approval of the Transformation Plan (Romanian) Action Plan for Implementation of the Guidelines for Alternative Care of Children (English) Government Decision 1361 of 7 December 2007 on Regulation of the Professional Parental Assistance Service (Romanian) Strategy for the Protection of Children and Families 2013-2020 ANNEX V: SOURCES OF INFORMATION | 252 Government Decision 432 of 20 April 2007 on the Minimum Quality Standards for the Care, Education and Socialization of Children in Residential Institutions (Romanian) Law 140 of 14 June 2013 on the Special Protection of Children at risk and of Children Separated from Their Parents (English) Ministry of Social Protection, Family and Child. (2008). Case Management Social Assistants Guide: Order 71 October 2008 (English) ANNEX VI: CONFLICT OF INTEREST | 253 ANNEX VI: CONFLICT OF INTEREST ANNEX VI: CONFLICT OF INTEREST | 254 Conflict of Interest Name Nancy Elizabeth “Beth” Bradford Title Subject Area Expert and Moldova Technical Lead Organization Maestral International LLC Consultant to NORC Evaluation Position? Team Leader  Team member Evaluation Award Number (contract or other instrument) GS-10F-0033M / AID-OAA-M-13-00013. Tasking N003 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) No evaluation of directly funded USAID project. Evaluation of World Learning SPANS-016 funded project in Moldova – Every Child Moldova / P4EC I have real or potential conflicts of interest to disclose. Yes  No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. Note: As discussed with USAID/DCOF when I was selected for the evaluation, I was the mid-term and final evaluator on the previous P4EC USAID funded (through World Learning SPANS-016) project “Protecting children in Moldova from family separation, violence, abuse, neglect & exploitation”. In addition I have completed other work for the implementing organization including “EveryChild Moldova’s Programme Experience: Improving Children’s Lives through Deinstitutionalisation” in 2013 and evaluation of the EU funded, “Developing Short Break Foster Care Service for Children with Disabilities in the Republic of Moldova” in 2012. As a result I do have existing relationships with Partnerships for Every Child and knowledge of their program experience, but it will not affect my ability to provide objective input to the evaluation. ANNEX VI: CONFLICT OF INTEREST | 255 I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 15 July 2015 ANNEX VI: CONFLICT OF INTEREST | 256 Conflict of Interest ANNEX VI: CONFLICT OF INTEREST | 257 Conflict of Interest Name Mawadda Damon Gartner Title Performance Evaluator Organization NORC Evaluation Position?  Team Leader  Team member Evaluation Award Number (contract or other instrument) GS-10F-0033M / AID-OAA-M-13-00013. Tasking N003 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Good Governance in Georgia (G3), implemented by Management Systems International, RFTOP No. SOL-114-I-1300001 Moldova Civil Society Strengthening Program (MCSSP), implemented by FHI 360, RFTOP No. SOL-121-14-000001 I have real or potential conflicts of interest to disclose.  Yes  No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. ANNEX VI: CONFLICT OF INTEREST | 258 I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 15 July 2015 ANNEX VI: CONFLICT OF INTEREST | 259 Conflict of Interest