November 2015 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Emerging Markets Consulting (EMC), Phnom Penh. Promoting Family-based Care in Cambodia: Evaluation of Childcare Reform Projects PROMOTING FAMILY-BASED CARE IN CAMBODIA: EVALUATION OF CHILDCARE REFORM PROJECTS EVALUATION OF THE STRENGTHENING CHILD PROTECTION SYSTEMS IN CAMBODIA TO PREVENT AND RESPOND TO VIOLENCE, ABUSE, EXPLOITATION AND UNNECESSARY SEPARATION OF CHILDREN (INCLUDING THE PARTNERSHIP PROGRAM FOR THE PROTECTION OF CHILDREN) AND FAMILY+ PROJECTS November 2015 DISCLAIMER This report is made possible by the support of the American People through the United States Agency for International Development (USAID). The contents of this report are the sole responsibility of Emerging Markets Consulting (EMC) and do not necessarily reflect the views of USAID or the United States Government. Cover photo: Kalyan Mith staff providing non-formal education to a group of children in Siem Reap province. ©UNICEF Cambodia/July 2015/Plong Chhaya i CONTENTS Acronyms............................................................................................................................ ii Executive Summary.......................................................................................................... iii Evaluation Purpose and Questions .................................................................................. 1 Project Background ........................................................................................................... 3 Goals and Objectives..................................................................................................................3 Strengthening Child Protection Systems.............................................................................................................3 3PC...............................................................................................................................................................................3 Family+.........................................................................................................................................................................4 Methods and Limitations................................................................................................... 5 Methodology ...............................................................................................................................5 Limitations ..................................................................................................................................6 Findings, Conclusions and Recommendations................................................................ 7 Quantitative Analysis .................................................................................................................7 Impacts on Key Stakeholders....................................................................................................8 Impacts on Government..........................................................................................................................................8 Impacts on RCIs ..................................................................................................................................................... 13 Impacts on Vulnerable Families........................................................................................................................... 22 Impacts on Donors to RCIs................................................................................................................................. 23 Impacts on Overarching Issues ...............................................................................................24 Research Commissioned...................................................................................................................................... 24 Specific Systems Changes that Contributed to National Care Reform.................................................... 25 Sustainability.............................................................................................................................26 Conclusion.................................................................................................................................27 Recommendations....................................................................................................................30 Annexes............................................................................................................................. 31 Annex I: Logframes ..................................................................................................................31 SCPS .......................................................................................................................................................................... 31 3PC............................................................................................................................................................................ 32 Family+...................................................................................................................................................................... 38 Annex II: Evaluation SOW ......................................................................................................45 Annex III: Project Venn Diagram ...........................................................................................51 Annex IV: Information Sources...............................................................................................52 List of Reviewed Documents.............................................................................................................................. 52 List of Interviews.........................................................................................Error! Bookmark not defined. Annex V: Data Collection Instruments ..................................................................................54 Annex VI: Evaluation Schedule ...............................................................................................79 Phase 1...................................................................................................................................................................... 79 Phase 2...................................................................................................................................................................... 79 Annex VII: List of COI..............................................................................................................80 ii ACRONYMS 3PC Partnership Program for the Protection of Children AC Alternative Care APLE Action Pour Les Enfants CCT Cambodian Children’s Trust CCWC Commune Committee for Women and Children CM Case Management CP Child Protection CPO Child Protection Officer CSO Civil Society Organization DCOF Displaced Children and Orphans Fund DoSVY Department of Social Affairs, Veterans and Youth Rehabilitation EMC Emerging Markets Consulting EAG Evaluation Advisory Group FCF Family Care First FI Friends International HBP Home Based Production HL Homeland HVC Honor Village Cambodia IGA Income Generating Activities KAP Knowledge, Attitudes and Practices KM Kalyan Mith KMR Komar Rikreay MCH M’lop Children’s Home ME Micro Enterprise MEF Ministry of Economy and Finance MoCR Ministry of Cults and Religion MoI Ministry of Interior MoJ Ministry of Justice MoSVY Ministry of Social Affairs, Veterans and Youth Rehabilitation MoWA Ministry of Women’s Affairs MS Mith Samlang MSACC Minimum Standards on Alternative Care for Children NGO Non-governmental organization PACC Policy on Alternative Care for Children RCI Residential Care Institution RGC Royal Government of Cambodia RUPP Royal University of Phnom Penh SCPS Strengthening Child Protection Systems UNICEF United Nations Children's Fund USAID United States Agency for International Development VC Village Chief VCAO Vulnerable Children Assistance Organization WL World Learning iii EXECUTIVE SUMMARY This evaluation assesses two alternative care reform projects in Cambodia: The first, “Strengthening child protection systems in Cambodia to prevent and respond to violence, abuse, exploitation and unnecessary separation of children”, hereafter referred to as the Strengthening Child Protections Systems (SCPS) project, implemented by UNICEF, was initiated in June 2009, extended in March 2013 and will end in September 2017. The second, the Family+ project, was implemented by Friends International (FI) and funded from April 2013 to June 2015. SCPS manages the sub-project Partnership Program for the Protection of Children (3PC) in collaboration with FI and the Ministry of Social Affairs, Veterans and Youth Rehabilitation (MoSVY). 3PC was initiated in 2011 and is ongoing, and is implemented by 9 NGOs led by FI. These projects and sub-project have their own timeframes and objectives but share a common aim of strengthening child protection systems and services from grassroots to national levels, including supporting reform of alternative care systems and practice in favor of family-based care. The projects have worked to influence and benefit the stakeholders in the alternative care system; government, residential care institutions (RCIs) and the children within them, donors to RCIs and lastly, vulnerable families and their children. The evaluation assesses the impact of these projects on each group of stakeholders, and then goes on to broadly discuss the crosscutting areas of systems change, sustainability and research. The evaluation found that, together, SCPS, Family+ and 3PC had an impact on alternative care system and services in Cambodia. UNICEF supported MoSVY to create a regulatory framework for Cambodia, which provides guidelines, roles and responsibilities to promote appropriate and permanent family care, and to prevent unnecessary family-child separation. MoSVY was further supported to conduct a mapping of RCIs in the five target provinces, which brings a large number of residential care institutions (RCIs) under MoSVY oversight. With support from UNICEF, MoSVY has conducted annual inspections of RCIs registered with MoSVY nationally, and has partnered in closures of substandard RCIs. However, given the lack of funding for MoSVY implementation of services, it is doubtful that these interventions are sustainable. The projects have also impacted RCIs and the children who live within them. Several Family+ RCI partners have transitioned from residential care to centers offering support services to families in the community, offering a combination of residential care service and services to reintegrated children, or have reintegrated all children into the community, and then closed. Reintegration of children from Family+ and 3PC partners is reported to have increased as a result of project efforts, and children who enter 3PC partner RCIs now spend less time in residential care than prior to reintegration. However, those interviewed cautioned that reintegration should always be thoughtfully planned, and supported with adequate follow-up and support, such as income generation activities, to avoid placing children at risk. At the community level, children and families have also been impacted through a number of project activities. 3PC supported delivery of direct services to significant numbers of children and families; 50,391 children and youth and 18,315 families received direct services through the work of 3PC partners. 3PC and Family+ both worked to develop and improve foster care programs, which have not been widely implemented in Cambodia, and 3PC reported a 36 percent increase in foster placements among 3PC partners. However, the lack of adequate policy guidelines and legal framework regarding foster care continue to warrant concern. iv One of the Family+ project’s aims was to change funding behaviors of international donors who fund RCIs in Cambodia, and voluntourists1 who visit them. The resulting advocacy campaign far exceeded its targets, reaching an estimated 3.9 million people internationally. Although it is difficult and too early to gauge impact on targets as dispersed as international tourists and funders, it was possible to find evidence of the impact in international media discussion, responses to the campaign by RCIs, and a program that ceased placement of volunteers in Cambodian RCIs. The SCPS, Family+ and 3PC projects achieved significant impact at all levels of the alternative care system in Cambodia. However, all three faced challenges regarding the monitoring and measuring of achieved impacts. Addressing these challenges would support the projects in planning for future interventions, and sharing their overall results. Extensive recommendations are made at the end of the report. Priority recommendations are listed below. UNICEF should support the government to:  Develop foster care minimum standards and guidelines.  Develop a policy to clarify laws and procedures regarding domestic adoption.  Strengthen the system for responding to reports of violence from RCIs, through stronger enforcement of the Minimum Standards on Alternative Care for Children (MSACC), enforce punitive actions for failing to meet these standards, and strengthen the existing mechanism managed by the government and 3PC partnership for responding to allegations of abuse from within RCIs nationally.  Expand the mapping of RCIs to all provinces so that all unregistered RCIs can be monitored and regulated by MoSVY.  Expand RCI inspections to all RCIs identified in the mapping, and conduct inspections every six months in accordance with MSACC.2  Redirect organizations requesting permits to open RCIs to opening family-based services instead.  Continue to close substandard RCIs, with a priority on closing those in which children are being physically or sexually abused, in a process which includes supported reintegration.  Scale up interventions to reduce the number of children in institutional care and to strengthen reintegration approaches.  Regulate the alternative care sector through the implementation of the Sub-Decree on Residential care Institutions. The evaluation team also recommends that the 3PC project strengthens its programming in the following ways:  Clarify among 3PC partners how indicators are reported, measured and verified. Particular attention should be paid indicators recording the number of children receiving services such as placements into families by 3PC partners.  Reintegration o Ensure that reintegrated children receive adequate follow up and support o Ensure that staff assisting in reintegration are adequately trained and skilled  With government collaboration, increase focus on children with disabilities within RCIs. 1 Voluntourist is a term used to describe both short-term and long-term volunteers who travel to another country to volunteer as part of a larger holiday. 2 MoSVY, 2008, Minimum standards on alternative care for children. Phnom Penh: MoSVY. 1 EVALUATION PURPOSE AND QUESTIONS This evaluation has a dual purpose. The evaluation aims to provide information to USAID regarding the impact of Family+, SCPS and 3PC, and to provide recommendations for SCPS and 3PC projects, which are ongoing. The evaluation also aims to identify lessons learned and best practices from these projects to inform the new Family Care First (FCF) intervention within Cambodia3. FCF is a global initiative to reduce the number of children growing up outside of families. FCF identified Cambodia as the first country to initiate a project, and in March 2015 FCF held a co-creation workshop to begin this process. Shortly after this workshop this evaluation commenced, and findings from this evaluation were shared in November 2015 with the USAID team overseeing FCF in Cambodia, and at the second FCF workshop, in the expectation that the findings would inform FCF Cambodia project design. The evaluation addresses seven overarching research questions as stated in the RFP. 1. To what extent are the projects on track to achieve their objectives and numerical targets specified in their project descriptions? 2. Identify best practices and lessons learned from these projects that help lead to the safety, well-being, and development of highly vulnerable children that they target. a. Is there evidence that the projects have measurably improved the conditions of these vulnerable children, particularly those who are living without adequate family care? b. Have the reintegration methods used resulted in stable and sustained family placements for children? To what extent does it appear that particular attention is being given to the reintegration of children under three years of age or who have disabilities? 3. Have the projects achieved or do they appear to be on track to achieve any specific system change(s) that will contribute to national care reform? Identify best practices and lessons learned. To what extent does it appear that functioning system elements (public and civil society) have been developed that can continue to provide adequate case-management services for children at risk? Identify best practices and lessons learned. 4. What does the potential appear to be of the capacities and systems that the projects are strengthening being sustained over time, after the end of the project? 5. What actions are being taken to respond to reports of violence or abuse toward children in residential institutions? 6. Identify lessons learned from the placement of 37 children into family care in 2012 in Siem Reap through a pilot initiative carried out by MoSVY with the support of UNICEF and Project Sky. 7. Make recommendations for a way forward that would address the issues identified in the evaluation. In order to answer these questions, the evaluation assesses whether the projects have met project objectives, as stated in their project proposals, and in the section titled, Goals and Objectives, below. Each of the seven research questions and many project goals require the evaluation of impact on several stakeholders. In order to prevent repetition, the evaluation is therefore structured around stakeholders. A chart of the sections in which specific research questions are answered, within this structure, is included below. 3 The US Government Action Plan on Children in Adversity was issued in December 2012. The second of its three primary objectives is: Put family care first U.S. Government assistance will support and enable families to care for their children; prevent unnecessary family-child separation; and promote appropriate, protective, and permanent family care. The FCF initiative aims to contribute to meeting this objective. 2 Research Questions Answered in Section Titled 1. To what extent are the projects on track to achieve their objectives and numerical targets specified in their project descriptions? Findings, Conclusions and Recommendations: Quantitative Analysis 2. Identify best practices and lessons learned from these projects that help lead to the safety, well-being, and development of highly vulnerable children that they target. a. Is there evidence that the projects have measurably improved the conditions of these vulnerable children, particularly those who are living without adequate family care? Methods and Limitations: Limitations Findings, Conclusions and Recommendations: Impact on RCIs Impact on Vulnerable Families 2. b. Have the reintegration methods used resulted in stable and sustained family placements for children? To what extent does it appear that particular attention is being given to the reintegration of children under three years of age or who have disabilities? Findings, Conclusions and Recommendations: Impact on RCIs Reintegration of Children from RCIs into Family-Based Care Have the Projects Resulted in Positive, Stable and Sustained Reintegration? Special Attention to Reintegration of Children under Age three or with Disabilities Lessons Learned Regarding Reintegration 3. Have the projects achieved or do they appear to be on track to achieve any specific system change(s) that will contribute to national care reform? Identify best practices and lessons learned. To what extent does it appear that functioning system elements (public and civil society) have been developed that can continue to provide adequate case-management services for children at risk? Identify best practices and lessons learned. Findings, Conclusions and Recommendations: Impact on Overarching Issues Specific Systems Changes that Contributed to National Care Reform System Elements Providing Case Management Services within RCIs 4. What does the potential appear to be of the capacities and systems that the projects are strengthening being sustained over time, after the end of the project? Findings, Conclusions and Recommendations: Impacts on Key Stakeholders Sustainability 5. What actions are being taken to respond to reports of violence or abuse toward children in residential institutions? Findings, Conclusions and Recommendations: Impacts on RCIs Child Safeguarding Policies Responding to Reports of Abuse from Non-partner RCIs 6. Identify lessons learned from the placement of 37 children into family care in 2012 in Siem Reap through a pilot initiative carried out by MoSVY with the support of UNICEF and Project Sky. Findings, Conclusions and Recommendations: Impacts on RCIs Lessons Learned Regarding Reintegration 7. Make recommendations for a way forward that would address the issues identified in the evaluation. Executive Summary (Prioritized recommendations) Conclusion (Comprehensive recommendations) 3 PROJECT BACKGROUND This evaluation assessed two childcare reform projects within Cambodia. Each project had its own timeframe and objectives. The first project, SCPS, was implemented by UNICEF, for the period June 2009 to September 2012 and extended for the period March 2013 to September 2017. As part of the SCPS, UNICEF managed a sub-project, 3PC, which was initiated in 2011 and was ongoing at the time of the evaluation. The second project, Family+, was implemented by FI, funded for the period April 2013 to June 2015 it has now ended. UNICEF partnered with FI and MoSVY to coordinate and manage 3PC, whilst NGOs partners, led by FI, offered services to beneficiaries. These three interventions addressed childcare reform, and each included multiple national partners, including governmental bodies and NGOs. All three have worked closely together, in five project provinces, and shared some activities, as illustrated in the Venn diagram, included in annex III. In a preliminary project review, it became clear that SCPS, Family+ and 3PC all evolved from their original work plans and substantial changes had been made to the projects’ original results frameworks. The ongoing revisions of the projects’ results frameworks contributed to creating significant overlap in project activities and result areas among the three projects (annex I). As a result, the Evaluation Advisory Group (the role of which will be elaborated in the next section) asked project implementers to identify key, relevant outcomes to create an evaluation logframe for each project. This resulted in the SCPS logframe (phase one and two), the 3PC logframe and the Family+ logframe. GOALS AND OBJECTIVES Strengthening Child Protection Systems The goal of the SCPS project is to strengthen systems for child protection in Cambodia to prevent and respond to violence, exploitation, abuse and unnecessary separation.4 It has four objectives: 1. Vulnerable children and their families in five target provinces have access to specialized, quality child protection and social welfare services. 2. Separated children in five target provinces are able to access mechanisms for family reunification, community reintegration and alternatives to institutionalization, and receive quality care. 3. Enhanced capacity of government officials at the national level, and government and NGO partners in five provinces, to effectively prevent child maltreatment and to move children into family-and community based care, in line with the ongoing Sub-National Democratic Development (SNDD) process requiring increased responsibility of sub-national actors. 4. Rigorous evidence-based policy and baseline data are used for advocacy, leveraging resources and measuring trends. 3PC The goal of the 3PC project is “to strengthen civil society’s involvement and coordination in child protection systems building”. Its objective is to strengthen child protection through CSOs’ enhanced capacity, coordination with and contribution to national and sub-national protection responses.5 4 Source: UNICEF Proposal to DCOF/USAID - Strengthening Child Protection Systems in Cambodia to prevent and respond to violence, abuse, exploitation and unnecessary separation of children, Nick Axelrod, 2010. 5 Source: List of indicators for the 3PC project provided by UNICEF and Friends International. 4 Family+ The goal of the Family+ project is for “children in Cambodia live in safe, healthy and sustainable family units rather than residential facilities institutions.”6 It has the following objectives: 1. To increase the number of children reintegrated from residential facilities and reduce intake through improved family support and prevention from abandonment. 2. To affect positive perception / behavior change amongst public, civil society and government target audiences on keeping families together (family based care). The shared aim of the projects was to strengthen child protection systems and services from grassroots to national levels, including supporting reform of alternative care systems and practice in favor of family￾based care. Together, the projects worked to influence the key stakeholders addressing alternative care in Cambodia. The analysis begins with a brief summary of the progress of the three projects towards achieving their objectives and numerical targets. In order to evaluate the success of these three complex and overlapping programs, this summary is followed by a discussion structured around the impact of projects on key stakeholders; government, residential care institutions (RCIs) and the children within them, donors to RCIs and lastly, vulnerable families and their children. The analysis concludes with a discussion of the cross cutting issues of system strengthening, the impact of research, and sustainability. 6 Source: List of outcome indicators for Family+ project provided by Friends International. 5 METHODS AND LIMITATIONS METHODOLOGY A document review of qualitative data was conducted reviewing SCPS, 3PC and Family+ reports, research studies produced by the projects, and relevant to the subject, and policy documents and guidelines. Quantitative data was gathered from reports and documents from SCPS, 3PC and Family+; and after analysis the results have been incorporated in three logframes, which are presented in annex I. The report focuses on a qualitative analysis of the projects, and it is predominantly drawn from semi￾structured interviews conducted with stakeholders, in four project locations, listed below. Further details regarding the methodology can be found in the project proposal in annex II. It was also informed by SCPS, 3PC and Family+ project documents, through a data triangulation and verification process. The table below summarizes the kinds, location, and number of stakeholders interviewed in the various locations. A full list can be found in annex IV. Respondent Location Phnom Penh Siem Reap Battambang Sihanoukville FI and UNICEF staff 9 2 - - MoSVY / DoSVY officials 1 2 1 1 CCWC* members - - 5 3 RCI management 1 2 3 1 Parents 4 8** 4 4 Children 7 (F: 4) 6 (F: 3) 5 (F: 1) 6 (F: 2) Notes: * commune council for women and children, **including four parents from the Project Sky reunification pilot in 2012. A defining methodological aspect of this evaluation was the piloting of a so-called participatory approach by USAID. This meant that, in addition to USAID/World Learning providing the overall oversight in the implementation of the evaluation, an Evaluation Advisory Group (EAG) was formed, to provide guidance to the evaluation team. The EAG was consulted at all key stages of the evaluation, and meeting minutes as well as all evaluation tools and deliverables were circulated for feedback by the EAG. The EAG was responsible for the following:  To ensure the evaluation team gets the information they need quickly, meets with the right people, and is able to get answers to its questions quickly and comprehensively.  To help ensure the recommendations the team proposes are relevant and supported by accurate information.  Make recommendations to USAID that it may consider, on steps during the course of the evaluation. As such, the EAG influenced major decisions that shaped the course of the evaluation. This included specifically the definition of project logframes and priority outcomes. In order to manage the large amount of different logframes and indicators – some logframes had reportedly changed over time – the EAG oversaw the compilation of the logframes that would be used for this evaluation of the three projects, based on existing logframes that the implementing partners had been using. In order to maintain transparency and neutrality USAID was a member of the EAG and the evaluation team reported on all the indicators in a transparent manner. Additionally, in order to cope with data gaps, it was decided to focus on qualitative evaluation methods for many outcomes. 6 The EAG consisted of UNICEF Child Protection and Monitoring and Evaluation Officers, Friends International personnel selected by the Executive Director, a technical level professional of MoSVY as well as USAID. Four meetings were held between the evaluation team, USAID and the EAG.  The first meeting the group was held on Tuesday, June 2nd 2015, 09:00am – 10:20am  The second meeting was held on Tuesday, June 9th 2015, 08:00am – 10:00am  The third meeting was held on Monday, July 27th 2015, 08:00am – 9:15am  The fourth meeting was held on Tuesday, October 29th 2015, 08:00am – 10:00am In addition there was follow-up email exchange among members of the EAG and the evaluation team to address issues identified during the meetings and to provide feedback on draft deliverables. As intended by USAID, the meetings helped frame the evaluation as a learning process to inform both ongoing and future programming related to children without adequate family care. There was general recognition that the point of identifying any shortcomings in projects would be to inform such programming, rather than to assess blame. Recognizing data limitations and the complex interrelationships between the three projects led to the adoption of a holistic approach to evaluate outcomes (especially systemic changes) achieved by concerted efforts across all projects. LIMITATIONS The following limitations were encountered:  Since the projects did not always collect baseline data, it was not possible to measure increases in relation to some targets. Consequently, the evaluation of some outcomes was based on qualitative primary data collected during field research. This constrained the evaluation team’s ability to assess measurable improvements in relation to the project goals and objectives, while it did contribute to identifying best practices and lessons learned.  Due to confidentiality issues, beneficiary respondents were pre-selected by the visited RCIs, from which they had received services. This may have biased the data collected, even though RCI staff assured the evaluation team that also beneficiaries from difficult or less successful reintegration cases were interviewed.  A large part of the qualitative data gathering relied on interviews, introducing inherent limitations thereof, particularly related to subjectivity and personal biases. The evaluation team mitigated this as much as possible by interviewing multiple respondents per organization and by triangulating with field observations and secondary sources. The EAG also explicitly dealt with this issue, by providing verification of finding at the highest level by all stakeholders.  Data to measure progress on some of the indicators was not collected, such as surveys or case file sampling. Some data that was collected by members of the 3PC partnership and then aggregated by FI, was also, reportedly, not verified.  Despite being part of the research questions, the projects generally did not collect data that could be disaggregated by age for children under age three or with disabilities. During field visits the evaluation team was able to get some information from RCI staff, but this was largely qualitative in nature (none of the visited RCI had any children under age three living there).  Many project logframe indicators focused on activities or outputs rather than outcomes. There was some duplication of activities across projects’ logframes.  As a result of the broad combined scope of SCPS, Family+ and 3PC, coupled with the large number of research questions, the evaluation focuses on the interventions that best answer the research questions, and offer lessons learned and best practices, in line with the evaluation’s objectives. This was based on input from, and agreed by, the Evaluation Advisory Group. 7 FINDINGS, CONCLUSIONS AND RECOMMENDATIONS QUANTITATIVE ANALYSIS Overall, all three projects have been broadly successful in meeting the majority of numerical targets set in the logframes included in this evaluation. The evolving nature of the projects, previously discussed in the Methods and Limitations section, did result in underperformance on a few indicators. The most important lessons learned from this relate to CSO service quality and reintegration of children to immediate and kinship family units. This reflects the ongoing realization that successful reintegration is a very resource-intensive and time consuming process. Another issue is that indicators were sometimes not defined clearly or not operationalized. Some targets were unrealistic or defined in such a way that their targets were unachievable from the start. Some indicators also lacked baselines or measurable targets altogether, and as such the evaluation team was not able to give a definitive assessment for these. The logframes considered for this evaluation were compiled through a consultative process involving the Advisory Committee, as described in the Methods and Limitations section, and approved in a meeting on 27 July 2015. It includes separate logframes for the 3PC project, the Family+ project, and the SCPS project. A summary of the number of indicators for each logframe that met their defined targets is included in the table below. The three logframes in their entirety, including analysis and explanatory comments, are included in annex I. Project logframe Number of indicators Met target Inconclusive Did not meet target SCPS 3 1 1 3PC 19 3 3 Family+ 17 5 7 Of the five indicators provided for the SCPS project, three met their target, one did not, and one had no target defined. The project was successful in supporting MoSVY by strengthening the regulatory framework and training officials on its implementation, which resulted in inspections of residential care centers. However, the number of children directly affected by this did not reach the target number for the specified years (year two and year three). More children may benefit from these services provided by MoSVY staff in the coming years, but no further targets were specified. The 3PC project met or exceeded 19 of the 25 numerical targets considered for this evaluation, and fell short on three. Additionally, eight of the 19 successfully met indicators were linked to multiple outcomes in the logframe. For three indicators no assessment could be made. For two of these, there were no defined targets. The third inconclusive indicator suffers from missing data; a survey to measure the impact of a campaign on child abuse awareness by 2014 was apparently never administered. In terms of absolute number of people reached through a number of campaigns however, the output targets were reached (outcome 2). The 3PC project was fully successful in meeting its targets for number of government policies influenced by 2014 (outcome 1), through a number of policy documents, best practices and thematic research efforts as well as meetings with government authorities. Other key outputs of the project were increased organizational capacity (outcome 3) and service quality (outcome 4) of participating CSOs. These outcomes were evaluated qualitatively by the evaluation team, at the request of the Advisory Committee. In terms of output indicators, all of the targets for organizational capacity increases were met, while service quality met 8 of its 10 targets but fell short on number of trainings provided and 8 number of beneficiaries reintegrated. In terms of beneficiaries’ service satisfaction (outcome 5), the target was greatly exceeded, but the evaluation team has some doubts as to whether “number of children [with] access to services” accurately reflects this outcome. Finally, a positive attitude towards networking and coordination (outcome 6) was reached by all participating CSOs. The Family+ project was ultimately successful in meeting or exceeding 17 of the 29 defined numerical targets. Seven targets were not met, and five indicators could not be assessed because no data was collected or no target was ever defined. Much of this could have been prevented if the project had taken a more systematic approach to monitoring and evaluation from its inception, including more clearly defined indicators, baselines and targets as well as implementation of data collection efforts. The Family+ project succeeded in developing and implementing best practice models for AC systems and prevention services (outcome 1), with all three of its numerical targets met. In terms of improved CSO services based on RGC minimum standards (outcome 2), none of the three specified targets were met. The evaluation team notes however that these indicators were poorly conceived both in terms of linkage to the outcome as well as feasibility of targets. This is discussed in more detail in annex 1. Improved and increased prevention and family support services (outcome 3) was broadly met with two out of three indicators, but failed to meet the target for number of microenterprise businesses still in business after 12 months, which does cast some doubt on sustainability. Improved and increased reintegration to family, kinship and foster family (outcome 4), had mixed outcomes. The target number of children was not reached, broadly reflecting the lessons regarding the difficulties and time requirements of successful reintegration. The other three targets were met however, including number of CSOs carrying out reintegration for the first time and reintegrated children still in place after 12 months. It was not possible to definitively assess the improvements of case management systems at participating CSO partners (outcome 5), as two out of three indicators did not have any baselines to evaluate against. CSOs recorded improved research, data collection and sharing of good practices (outcome 6) by meeting all of its three indicators, largely through increased codified knowledge in best practice and research documents. Increases in the skills and alternative care capacity of RGC (outcome 7) can be observed in two indicators which met their targets, while two more could not be assessed due to lack of data and baselines. Finally, changes in perceptions and behavior within the wider public concerning alternative and family based care (outcome 8) were largely achieved, with four out of six indicators meeting their targets. One target was not met (number of students reporting increased knowledge), but the evaluation team notes that this is based on reports from the training facilitator which could not be verified. The last indicator (number of local media reports) the evaluation team considered likely to have been met, but this cannot be definitively stated due to inadequate data. The three complete logframes are included in annex I. The overall findings discussed in the remainder of this section also refer to quantitative data from those logframes where contextually appropriate. IMPACTS ON KEY STAKEHOLDERS Impacts on Government SCPS, Family+ and 3PC all aimed to strengthen government child protection capacities and systems at the national, sub-national and local level. SCPS aimed to “enhance capacity of government officials at the national level, and government and NGO partners in five provinces, to effectively prevent child maltreatment and to move children into family and community based care” (SCPS Objective 3), 3PC aimed to enhance CSOs coordination with and contribution to national and sub-national protection responses, while Family+ aimed to “affect positive perception/behavior change amongst…government audiences on keeping families together,” (Family+ objective 2). The outcomes identified for the 3PC logframe are included in discussions of the impact of these three projects on different levels of government below. 9 Impacts on National Level Government Attitudes towards Residential Care within MoSVY Many participants in interviews said that in the last five years MoSVY has taken an increasingly strong position in supporting reform of alternative care systems and practice in favor of family-based care, and stated that UNICEF support had been the key force enabling this change. MoSVY’s position is evidenced in evaluation interviews, and is further reflected in the steps taken by MoSVY to strengthen the regulatory framework surrounding alternative care. At the same time MoSVY has noted in evaluation interviews that previously, in many cases, it lacked the authority to monitor RCIs due to the ability of RCIs to register with alternate Ministries. This is discussed in greater detail in the section on the Sub￾decree on management of RCIs, below. Regulatory Framework Since the inception of the first project in 2009, UNICEF has supported MoSVY to develop plans and policies which strengthen and support reform of alternative care systems and practice in favor of family￾based care. The 3PC logframe indicator regarding the number of meetings with government partners aimed to measure UNICEF impact on MoSVY. 3PC conducted 46 meetings, exceeded the target of 30 (3PC, outcome 1). 3PC also aimed to directly influence government policy, which was noted as a goal in the SCPS logframe, and 3PC logframe provided for this evaluation (3PC outcome 1). 3PC met its target of influencing two government policies, (the Prakas on Procedures to Implement the Policy on Alternative Care for Children, and the Sub-decree on Management of RCIs)7, and contributed to a third (The Explanatory Notes on Domestic Adoption), which is still being developed with the government, as described below in the section ‘Sub-Decree on Management of RCIs’. MoSVY Action Plan Through the SCPS Project, UNICEF worked with MoSVY to develop its 2013 two-year action plan detailing activities including capacity building, development of case management, reintegration, policy, awareness-raising and the promotion of foster and kinship care. Prakas on Procedures to Implement the Policy on Alternative Care for Children In 2008 MoSVY issued the MSACC8, which details standards for children in alternative care. However, a policy gap remained regarding the responsibilities for the implementation of alternative care services. As part of the SCPS Project, UNICEF supported MoSVY to issue the Prakas on Procedures to Implement the PACC9, issued in 2011, which defines roles and responsibilities of relevant agencies and establishes procedures, guidelines and forms to implement the PACC, recognizing the child’s right to grow up in a family. Sub-decree on Management of Residential Care Institutions MoSVY is the Ministry tasked with monitoring of alternative care. However, previously many RCIs signed Memorandums of Understandings with other Ministries, instead of MoSVY and as a result MoSVY was unable to conduct monitoring within these institutions (Jordanwood 2011)10. Mr. Oum Sophannara, Director of MoSVY Child Welfare Department, has spoken openly about the inability of MoSVY to force RCIs to comply with monitoring (Cambodia Daily, Sept 19, 2014) 11. The new Sub-decree on Management of RCIs, developed by MoSVY with the support of SCPS, and signed by the Prime Minister in September, 2015, has addressed this issue. The Sub-decree includes provisions for the mandatory registration of all RCIs with MoSVY, and tasks MoSVY with inspection of RCIs to ensure compliance with 7 MoSVY, 2011. Prakas on procedures for implementing the policy for the alternative care for children. Phnom Penh: MoSVY. MoSVY, Sub-decree on management of RCIs, in press. 8 MoSVY, 2008. Minimum standards on alternative care for children. Phnom Penh: MoSVY. 9 MoSVY, 2011. Prakas on Procedures for implementing the policy for the alternative care for children. Phnom Penh: MoSVY. 10 Jordanwood, 2011. With the best intentions. Phnom Penh: MoSVY and UNICEF. 11 https://www.cambodiadaily.com/news/ministry-powerless-to-reform-orphanage-industry-68218/, accessed November 28, 2015. 10 the MSACC. It includes a clause on sanctions and actions required in cases of reported sexual abuse or violence. MoSVY Mapping of RCIs MoSVY is tasked with monitoring RCIs, however, as noted above, until the signing of the Sub-decree on Management of Registration of RCIs, MoSVY lacked the authority to enforce registration that would provide access for monitoring. Studies suggested that large number of unregistered RCIs were operating in Cambodia, but there had been no attempt to systematically count unregistered RCIs12. In 2014 as part of the SCPS Project, UNICEF supported MoSVY to map all RCIs in the five operational provinces, meeting an outcome listed in the SCPS logframe. Initial unpublished results counted more than 400 centers in these five provinces alone, which significantly exceeds the previously reported number of registered RCIs nationwide. Prior to the mapping, MoSVY was unable to monitor these RCIs to assess whether they provided services to children in line with the MSACC. The results of the mapping, combined with the increased authority granted to MoSVY through the Sub-decree on Management of RCIs, will allow MoSVY to more comprehensively regulate RCIs. Standards for Foster Care There are currently no minimum standards guiding the implementation of foster care programs in Cambodia. 3PC supported both the Cambodia Foster Care Committee, chaired by MoSVY and the Foster Care Network, chaired by FI, to develop minimum standards for foster care. These are still in development. The Explanatory Notes on Domestic Adoption The law regarding domestic adoption within Cambodia does not provide comprehensive guidance regarding adoption procedures. The Civil Code of Cambodia (RGC 2003) 13 stipulates the eligibility, rights and obligations of prospective adoptive parents and children, as well as those of the children’s biological parents. However, it does not provide detailed procedures on the recruitment and assessment of prospective adoptive families, and care support for families in the process. The Prakas on the Procedures to Implement the PACC14 offers partial provisions but a gap still remains. These issues pose problems for Cambodian parents who wish to adopt children, and has been cited as an issue by long-term foster parents in this position. UNICEF is supporting the Ministry of Justice and MoSVY to finalize the Explanatory Notes on Domestic Adoption and provide related training to judges and prosecutors. Inspection of RCIs In addition to the development of policy, MoSVY has also taken steps to implement the regulatory framework. In 2008, the Department of Child Welfare began conducting monitoring visits to RCIs. According to participants in evaluation interviews, these visits, lasting only two hours, did not adequately assess conditions in RCIs. From 2010 to 2014, UNICEF supported the Child Welfare Inspection Team to conduct annual inspections of almost all RCIs in the country registered with MoSVY. The number of RCIs inspected in this process ranged between 216 and 269 each year, and is listed in the SCPS logframe. Targets were not set for this indicator. In 2011, a new rigorous, detailed inspection form and process was introduced, which required 1-2 days per assessment. The MSACC (MoSVY 2008)15 calls for bi￾annual inspections, but participants reported that in practice inspections occurred once a year due to under-staffing at MoSVY. RCIs were scored according to their adherence to the MSACC. While in a small number of cases, (estimated by participants to be less than ten) 16, RCIs were closed for failure to meet the MSACC, it was more common for RCIs to be advised on procedures for improvement. In 12 Holt, 2005. Cambodia orphanage survey. Eugene: Holt. Jordanwood, 2011. With the best intentions. Phnom Penh: UNICEF/MoSVY. 13 RGC 2003. Civil Code of Cambodia, Phnom Penh: RGC. 14 MoSVY, 2011 Prakas on Procedures for implementing the policy for the alternative care for children. Phnom Penh: MoSVY. 15 MoSVY, 2008, Minimum standards on alternative care for children. Phnom Penh: MoSVY. 16 11 RCIs have been closed, but reportedly not all were closed for failure to meet MSACC; some encountered issues with funding. 11 evaluation interviews, both MoSVY staff and UNICEF staff anticipated increased compliance in the near future due to the recently published Sub-decree on the Management of RCIs, discussed above. RCI Closures In 2012, MoSVY facilitated the first mandated closure of a RCI in Cambodia. Since then, 11 RCIs were reported by participants to have been closed by the government. Many participants said this represented a fundamental change and presented a model for the future. However, with over two hundred registered orphanages, and many more unregistered, there is more work to be done. There have been indications that some in MoSVY is ready to take further action. During a high level meeting to approve the UNICEF 2016-18 country program documents, MoSVY made a commitment to reduce the number of children in RCIs in Cambodia by 30 percent over the next five years. The commitment to reintegrate children indicates that high level staff at MoSVY perceive the promotion of reintegration as a key role of the Ministry. It is laudable that MoSVY has committed to reintegrating a significant percentage of children into communities, but it is important that this reintegration process does not become numbers driven, and focuses primarily on the welfare of children. The findings of this evaluation, discussed in the section ‘Impact on RCIs’, suggest that reintegration requires a high level of capacity among implementing staff, and the dedication of significant resources and time. The goal of reintegration 30 percent of children within RCIs may be overly ambitious. Reintegration from Government RCIs In 2014, with the support of UNICEF through the SCPS project, MoSVY hired 15 social workers to begin to conduct case file reviews of the children living in government RCIs in Phnom Penh and Sihanoukville. This step demonstrated a shift in MoSVY to promoting family-based care. In addition, many participants stressed the symbolic importance of this step, noting that for the government to begin reintegration from government orphanages indicated that government orphanages were not above improvement. In a culture in which “saving face” is valued, this further demonstrated the commitment of the government. The social workers of the Department of Social Affairs, Veterans and Youth Rehabilitation (DoSVY) and 3PC partners have already reviewed cases and conducted family tracing for all the children in those RCIs. UNICEF plans to expand the program to the provinces of Battambang and Siem Reap with 3PC partner support. Impact on Local Level Government Local authorities have been found to act as gatekeepers to alternative care and have previously favored placement in RCIs (Jordanwood 2011)17. Both 3PC and Family+ worked to strengthen the capacity of local authorities through training. Family+ aimed to increase the Royal Government of Cambodia (RGC)’s “skill and capacity in alternative care principles, services and monitoring”, measured by training, increased referrals from government and increased number of placements of children with families carried out with MoSVY or DoSVY. UNICEF supported the Child Welfare Department to train 245 members of local authorities in the Prakas Procedures to Implement the PACC (MoSVY 2011)18, exceeding their target of 200 members (SCPS, outcome 1). Family+ provided training on alternative care and “Keeping Families Together” to local authorities but did not specifically record the number of local authorities trained. Family+ conducted pre- and post-tests to assess training impact and found that 84 percent of participants showed improved knowledge after the training, exceeding their target of 75 percent (Family+, outcome 7). Impact on Attitudes of Local Government In evaluation interviews, most members of local authorities interviewed as part of the evaluation reported an increased knowledge of alternative care as a result of the training. All DoSVY and CCWC interviewed supported family-based care and were aware of the negative impacts of residential care. However, when asked to consider a specific scenario regarding the placement of a vulnerable child, many 17 Jordanwood, 2011. With the best intentions. Phnom Penh: MoSVY and UNICEF. 18 MoSVY, 2011 Prakas on Procedures for implementing the policy for the alternative care for children. Phnom Penh: MoSVY. 12 said that residential care was still a preferred option. Jordanwood (2011)19 found that most members of respondents from local authorities believed that residential care was the best option for a child without parents. The interviews with local authorities for this evaluation, suggested that core attitudes towards residential care among local authorities remained unchanged. DoSVY Evaluation interviews found that district level engagement by DoSVY in promoting alternative care practices and supporting reintegration efforts varied. Family+ reported 85 referrals from government or local authorities to NGOs on alternative care issues between April 2013-September 2014, and 169 between October 2014 and June 2015 (Family+, outcome 7), showing an increase in referrals. In evaluation interviews both local authorities and CSO partners confirmed these referrals were a common practice. One participant explained that once DoSVY staff members were aware of alternative care services offered by CSOs they were happy to use them. Beyond referrals DoSVY commitment varied. In many districts DoSVY fulfilled a ceremonial role, signing official documents and participating only on the day that the child was reunited with his or her parents. There were, however, a few districts in which DoSVY was reported to be conducting follow up visits to reintegrated children, and to promoting family￾based care. The reason for the different levels of engagement was not clear, although some participants believed this was a result of the long-term relationship between the CSO and the DOSVY office. Throughout the evaluation interviews, participants noted the chronic lack of sufficient funds and staffing within MoSVY and DoSVY, which has also been noted in a previous study (Harachi 2014) 20. A significant portion of the MoSVY budget is dedicated to the pensions of veterans, and the next largest allocation pays the salaries of central administration (Harachi 2014) 21. The central government does not allocate funds for the implementation of MoSVY and DoSVY’s child protection and welfare responses. Participants explained that these budgetary constraints limited DoSVY’s capacity to fulfill their proscribed role, “Our capacity is mismatched to what the partners want, we don’t even have money for transport. I can’t say that we fulfill the mission we have been given.” DoSVY staff member When reintegration programs had been designed to rely on DoSVY follow-up, this lack of engagement posed a child protection risk; as in the case of the 2012 MoSVY Sky Reintegration, discussed later in this report. Given these budget shortfalls, some participants advocated for a reconceptualization of DoSVY’ s role, arguing that DoSVY should focus on role of monitoring and quality assurance, and that CSOs should manage implementation, including follow-up monitoring and support until reintegration has been achieved. Role of Commune Councils for Women and Children (CCWC) The Prakas on Procedures to Implement the PACC (MoSVY 2011)22 assigns CCWC a range of tasks in relation to the alternative care of children. In evaluation interviews, many participants noted that CCWC members were more active partners in alternative care than DoSVY officials. Family+ and 3PC both trained and collaborated with CCWC members. Participants detailed examples of CCWC members monitoring the situation of vulnerable children in communities, linking children with 3PC services, and advocating for services for children. “CCWC provide the best information and referrals. They are invested and embedded in and their influence is increasing.” 3PC partner staff member 19 Jordanwood, 2011. With the best intentions. Phnom Penh: MoSVY and UNICEF. 20 Harachi, T., 2014. Review of Social Work Practice. Phnom Penh: UNICEF. 21 Ibid. 22 MoSVY, 2011 Prakas on Procedures for implementing the policy for the alternative care for children. Phnom Penh: MoSVY. 13 At the same time the concern was raised that a large number of NGOs rely on CCWC partnerships, and that the members of these committees were not trained social workers, and were overworked. CCWC members receive a small supplement as payment. Moreover, CCWC are elected officials and turnover has been noted as an issue. In conclusion, SCPS, Family+ and 3PC made strides in strengthening government child protection capacities at the national level. Through the support of SCPS, MoSVY was supported to develop and release two significant policies that addressed gaps in the alternative care framework. Moreover, MoSVY was supported to monitor all registered RCIs within the country, and to map unregistered RCIs, a step that will bring a large population of previously unmonitored children within MoSVY’s oversight. Evaluation interviews with MoSVY staff and CSOs showed that MoSVY is committed to promoting family-based care in the future. The impact of SCPS, Family+ and 3PC on local authorities, however, was less evident. Project trainings led to an increased level of knowledge but many local authorities had unchanged attitudes, continuing to believe that RCIs were the best placement options for a vulnerable children. While members of local authorities were happy to refer children to 3PC and Family+ partners, this was described as a result of the establishment of links to these NGO services, rather than the result of an attitudinal shift. The severe lack of government funding and staff for DoSVY limited the ability of these officials to implement services supporting family-based care. The CCWC were more likely to take on this role, but were also limited by lack of funds, time and capacity. The evaluation found that while the projects had resulted in local authorities overseeing placements and referring children to 3PC and Family+ partners, the implementation of services to vulnerable families and children reunited with families, was still largely dependent on NGO support. Impacts on RCIs Both Family+ and 3PC aimed to strengthen child protection systems through supporting improved practices and services amongst RCI partners. Family+ supported RCIs to develop best practice model systems and prevention services based on MSACC (Family+ outcomes 1 and 2), and improved and increased reintegration to family-based care (Family+ outcome 4). 3PC worked to improve organizational and service capacity of CSOs (3PC outcomes 3 and 4). Taken together the projects activities included training, documenting standards of practice, supporting improved child safeguarding policies, and supporting reintegration of children from RCIs into family-based care, as discussed below. Development of Tools and Training within RCIs Training Training was given with the aim of improving both organizational capacity, and service delivery. The Family+ logframe does not record the number of trainings given, although Family+ reports record that 16 trainings were given. 3PC recorded giving 23 organizational capacity development trainings, exceeding its target of nine, (3PC, outcome 4) and 100 service delivery trainings, failing to attain its target of 120 (3PC, outcome 4). However, it exceeded its target for the number of people trained (1200 people), by training 1539 people by 2015. It appears there were fewer trainings with more attendees. In the 3PC logframe, stakeholder groups are not disaggregated, so it is not possible to specify how many RCI staff were trained. Family+ and 3PC partners were assessed by FI to identify areas of weakness. In principle, these assessments were used to determine the training topics offered to partner organizations. However, both Family+ and 3PC partners reported that in practice they took whichever trainings were assigned, though these may not have been relevant. While some RCI participants of Family+ and 3PC trainings had found them useful, others said the trainings were not sufficiently informative, not targeted to their specific needs, and lacked sufficient detail regarding implementation. Trainings typically lasted from one to two 14 days. The majority of people acting as social workers in Cambodia have not received a social work degree (Harachi 2014) 23. This lack of prior education means that trainings need to be more thorough. A few participants suggested that trainings should address fewer topics, in greater detail, over a longer time period. Most participants were unable to offer examples of how information learned as a result of training had impacted service provision. “Usually the training just involved the trainer showing the goals and vision and the details how to do the work or not addressed.” 3PC partner Coaching and Exchange Visits In response to the limitations of the traditional training model, FI began to move towards coaching and exchange visits. In “coaching”, also called “support visits” a FI social worker visited an RCI and helped the staff address issues in the workplace. In “exchange visits” 3PC partner staff would visit other partner offices to learn about specific good practices. The staff who made these visits identified partner programs that they felt their own organization could learn from. This aspect of choice was identified as contributing towards the success of exchange visits. 3PC exceeded their target of 54 exchange visits, completing 64 (3PC logframe, outcome 4), and exceeded their target of 72 support visits, completing 76 (3PC logframe, outcome 4). Most participants noted that exchange visits and coaching had been very effective in building skills and promoting implementation change. There were several reasons for this. These methods allowed professionals to learn from other professionals, instead of trainers. Other professionals are better placed to offer relevant information, and they understand the complexity of the work. These models also allowed participants to view working programs. These were more relevant to practice than abstract concepts. Moreover, participants could see that the interventions could actually work in Cambodia. Coaching and exchange visits were also described as more appropriate given the cultural context. In the hierarchical social structure in Cambodia (Gourley, 2009) 24 participants are reticent to ask questions in a formal training session. They are more likely to ask questions in a personal interaction, such as those fostered by coaching and exchange visits. A final benefit of these models is that they allowed staff to build up personal relationships that they could then call in the referral process. The impact of these different skills-building interventions is not disaggregated in the logframes, so their combined impact is discussed. In an earlier review of 3PC, Hackett (2014) 25 found that the three 3PC partners that had the lowest level of organizational skill at the project inception, had increased their organizational capacity markedly. The Family+ logframe shows that 88 percent of trainees reported increased knowledge of alternative care systems and curricula, exceeding the target of 80 percent. Family+ RCIs showed improvement on assessments conducted during the course of the project. The percentages of improvement for each RCI are noted in the Family+ logframe, and fall short of the target of 15 percent. However, Family+ worked with some of the better resourced and managed RCIs in the country, and their initial scores on the pre-tests ranged from 66 to 77 percent, making a 15 percent increase per annum over two years unrealistic (and in a few cases mathematically impossible). Standards of Practice In order to improve service capacity of CSOs, both 3PC and Family+ developed Standards of Practice (SOPs). The 3PC logframe reports that seven SOPs, including guidelines, were completed, exceeding the target of 6 (3PC logframe, outcome 4). The Family+ logframe records that Family+ documented and shared best practices (Family+, outcome 6). The notes in the logframes explain that there may be some overlap in these indicators. The topics of the SOP guidebooks are listed in the relevant logframes. 23 Harachi, T., 2014. Review of Social Work Practice. Phnom Penh: UNICEF. 24 Gourley, S., 2010. The Middle Way. Phnom Penh: NGOCRC. 25 Hackett, L., 2014. Partnership Programme for the Protection of Children (3PC) evaluation. Phnom Penh: FI. Unpublished. 15 The Family+ project reports explain that progress on completing these guidebooks was slower than anticipated due to the need to adapt them to the specific requirements of the RCIs. While a few participants from RCI partners explained that they used these guidebooks, many said they did not know of them, or were not using them. It was not possible therefore to ascertain the impact of these SOPs. One NGO participant said that although SOPs were completed too late to be used by the end of the Family+ project, FI was continuing to use them. 3PC should ensure that these documents are used further in the future. Child Safeguarding Policies The child safeguarding protocols of all Family+ partners were assessed by FI. 3PC partners were required to report and verify the child safeguarding protocols of their organization. The Family+ and 3PC logframes did not record prevalence of child safeguarding protocols or training on these, so evaluation participants were asked to report these in interviews. There were some organizations that were initial members of the Family+ or 3PC projects but had since left. One of these had reintegrated all the children in its center and then closed, another left 3PC partnership in the first year, before 3PC had begun to implement planned activities with the RCI. The evaluation team were not able to reach these organizations to ask about their child safeguarding policies and procedures. As a result only current Family+ and 3PC partners are discussed below. Participants reported that all current Family+ partners were trained in child safeguarding protocols, and that all current Family+ and 3PC partners have child safeguarding policies in place, and staff have been trained in protocols for reporting abuse. Posters explaining how to report abuse five RCIs currently had child protection officers in place. A clear protocol exists for responding to allegations of abuse in Family+ and 3PC partner RCIs, and there have been less than five reports of abuse within these partners since the inception of the projects. One participant said that a RCI staff member had called to verify whether a situation violated the child protection policy (it did not), and the participant noted that this was a sign that the child protection code was taken seriously. All reports were responded to quickly, following organizational protocols. Responding to Reports of Abuse from Non-partner RCIs FI and the 3PC partnership receive reports of violence in non-partner RCIs on a regular basis. The ChildSafe Hotline, which is operated by 3PC as part of the ChildSafe Network discussed below, reported receiving 8,610 calls, exceeding its target of 5,300. Participants said that many of these reports were regarding physical and sexual abuse, suggesting that abuse in RCIs in Cambodia is a wide scale problem and is an area that warrants further research. These reports most commonly come from international volunteers or donors. Respondents offered a number of reasons these reports came from these sources. First, volunteers are “outsiders” and they have little to lose by reporting. Second, local RCI staff perceive the violence as normal. Third, RCI staff may fear reprisals if they report. FI and 3PC have developed a system to respond to these reports from non-partner RCIs. FI staff provide those reporting with a format for submitting concerns to the government, and link those reporting with a staff member at FI to act as the contact person throughout the reporting process. Once the report is completed, FI attempts to verify the allegations. The summary document is then translated and submitted to DoSVY. Any reports of sexual abuse are immediately passed by FI to Action Pour Les Enfants (APLE), a CSO focused on addressing this issue. If action has not been taken within a month, the report is passed to MoSVY by FI. In some cases FI calls on UNICEF to pressure the government to ensure that action is taken. One participant noted that while this, at first, had seemed futile, after some time the Child Welfare Department in MoSVY began to take action related to the reports, conducting investigations, contacting police in some cases, and even closing RCIs. Some respondents voiced the concern that NGOs were being required to take on a role that should be handled entirely by the government. However, in piloting a system for responding to reports of violence in RCIs, FI and 3PC have created a model for which the government could assume increased 16 responsibility as the project continues. The new Sub-Decree on the Management of RCIs (MoSVY 2015)26 includes a clause requiring action on abuse reports, which may prompt the government to take on more responsibilities. Another concern raised was that the government currently contacts a range of NGOs to act as partners in responding to the abuse, and it is unclear whether all these organizations have the adequate skills to respond appropriately. Interviewees advocated for a stronger government￾owned response mechanism, which was more standardized, and worked with partners whose capacities had been rigorously assessed. In order to protect children, caution needs to be taken, however, to ensure that the government is fully committed and funded to take on this role, before FI and 3PC step away. Reintegration of Children from RCIs into Family-Based Care Reintegration of children into families from RCIs can “support and enable families to care for their children; prevent unnecessary family-child separation; and promote appropriate, protective, and permanent family care”, which is the second goal of the US Government’s Action Plan on Adversity27. Family+ aimed to promote improved and increased reintegration from RCIs to family, kinship and foster family (Family+ Objective 1, and Outcome 4). UNICEF aimed to support separated children in five target provinces “to access mechanisms for family reunification, community reintegration and alternatives to institutionalization, and receive quality care” (SCPS logframe, objective 2). In addition UNICEF aimed to support MoSVY to provide case management, de-institutionalization for permanency planning and reintegration (SCPS logframe, outcome 5). Over the three year course of the project, Family+ partners placed 469 children into immediate and kinship family units, failing to meet their projected target of 770 children (Family+, outcome 4). No baseline data was collected regarding the number of placements into family-based care from NGO and orphanage partners prior to the project intervention. However, the Family+ logframe shows that four RCI partners began reintegration activities for the first time (Family+ logframe, outcome 4), and several participants confirmed this in evaluation interviews. Many participants also noted that the number of children placed in family care by Family+ partner RCIs overall had increased (although this increase could not be quantified). The Family+ logframe reported that 100 percent of the placements into family￾based care by 3PC partners were carried out with MoSVY or DoSVY (Family+ logframe, outcome 7). Moreover, UNICEF supported MoSVY to provide case-management for deinstitutionalization and permanency planning for 532 children (SCPS logframe, outcome 4). “Before the support from Friends International we lacked knowledge on the MSACC, and the importance of family reintegration, but thanks to our partnership with Friends International we have started the reintegration process.” Family+ partner RCI staff member The length of time children spend living in 3PC partner RCIs before reintegration has decreased. 3PC commissioned a report on the impact of reintegration on children from five 3PC partner RCIs. This report found that the average number of days that children spend in RCIs after admission has decreased from almost 400 in 2011, (the date of the first SCPS project), to under 200 days in 2015. This decrease is part of a larger trend, which began prior to the advent of 3PC (Jordanwood 2014)28, and it is difficult to isolate the impact of 3PC and SCPS on this decrease. However, many participants expressed the belief that UNICEF’s support of the MoSVY inspections of RCIs, had exerted pressure on RCIs to comply with the MSACC, which mandate that RCIs evaluate the possibility of reintegration annually. Some Family+ RCIs explained that they were aware of the negative effects of placement in residential care and therefore worked to reintegrate children as quickly as possible. 26 MoSVY, 2015. The Sub-decree on the Management of RCIs. Phnom Penh: MoSVY. 27 U.S. Government, 2012. Action plan on children in adversity; a framework for international assistance 2012-2017. Washington DC: 2012. 28 Jordanwood, M., 2014. Achieving positive reintegration. Phnom Penh: Friends International. 17 However, some Family+ and 3PC partners were concerned that a focus on the speed and number of placements in family-based care could result in hurried placements that did not sufficiently safeguard children. “Where the reintegration is numbers driven, that can be a real problem.” 3PC partner staff member Have the Projects Resulted in Positive, Stable and Sustained Reintegration? There has been much discussion on the criteria for positive, stable reintegration (Feeny 2005) 29. Reintegration is not defined in the policies related to residential care in Cambodia, (PACC 2006, the MSACC 2006, and the Prakas on Implementing the PACC, 2011)30. In assessing reintegration among 3PC partners, Jordanwood (2014)31 identified the criteria of education, medical care, shelter, safety and community. These criteria were also examined in the interviews with families and children for this evaluation. In both the study and these interviews, most reintegrated children were attending school, or were working, if they were over the legal age to work (15 years). Most children, both in Jordanwood (2014)32 and this evaluation also had access to medical care, many through the RGC Health Equity Fund system. 33 The majority of reintegrated children lived in adequate shelters, and 94 percent of children interviewed by Jordanwood in this study continued to live with family members. Except for the small number of families from the MoSVY and Project Sky 2012 family placement process, the sample of children and families for this evaluation was drawn from families that were actively involved with the Family+ or 3PC project. These projects do not continue to work with families who do not have children, and therefore the sample frame did not include families from which reintegrated children had left. The projects did not report the number of children who had left post-reintegration. As a result, the evaluation team was not able to quantify the number of children who had left families post-reintegration. After the placement of children into families, Jordanwood (2014)34 found that not all children were safe. The study found that 26 percent of children interviewed had been physically abused. The study found this number is lower than the national average but is still of concern since almost no children in the study reported being abused in the transitional shelters. It is important to note that the transitional shelters in which children had previously lived afforded a high level of support to children, resulting in part from the intervention of the Family+ and 3PC projects. This is unlike the situation in many other RCIs in Cambodia (Jordanwood 2011)35. Two children out of the twenty-four interviewed as part of this evaluation reported being physically abused. These two children had been placed into their families by a 3PC partner shelter, and were receiving follow-up services. Following the disclosure, the researchers, in accordance with the research guidelines in the consent forms, shared this information with the case workers responsible for the case, who committed to follow-up with the children concerned. At the same time, most children in Jordanwood (2014)36 and many children interviewed as part of this evaluation, noted the benefits of living in the community and living with their families. 29 Feeny, T., 2005. In best or vested interest: an exploration of the concept and practice of family reunification for street children. London: Consortium for Street Children. 30 MoSVY, 2006. Policy for the alternative care for children. Phnom Penh: MoSVY, MoSVY, 2008, Minimum standards on alternative care for children. Phnom Penh: MoSVY, MoSVY, 2011 Prakas on implementing the policy for the alternative care for children. Phnom Penh: MoSVY. 31 Jordanwood, M., 2014. Achieving positive reintegration: assessing the impact of family reintegration. Phnom Penh: Friends International. 32 Ibid 33 A program funded by the RGC and Health Partners through the MOH with technical support funded USAID. 34 Jordanwood, M., 2014. Achieving positive reintegration. Phnom Penh: Friends International. 35 Jordanwood, 2011. With the best intentions. Phnom Penh: MoSVY and UNICEF. 36 Jordanwood, M., 2014. Achieving positive reintegration: assessing the impact of family reintegration. Phnom Penh: Friends International. 18 Overall, children described reintegration as offering the benefit of family and community, with the cost of material provision. Some children were nostalgic for the time when they had lived in the center without having to worry about their family’s ability to provide for food and education. Both Jordanwood (2014) and this evaluation team found that many children reunited with, and placed in their families with support from SCPS, 3PC and Family+ enjoyed the benefits and warmth of living with families in communities, but also suffered from a lack of material provision and in a few cases abuse, and therefore cannot be considered fully positive reintegration cases. Special Attention to Reintegration of Children under Age three or with Disabilities The Family+ and 3PC logframes did not disaggregate data by age (other than between children and youth over 18) or disability. The evaluation team was not able to find data regarding special attention paid to children under age three or children with disabilities. However, participants who implemented programs did describe priority systems for reintegration. In interviews with RCI and transitional shelter directors, and MoSVY staff, all said that they made an effort to prioritize reintegration of children under three. Most directors reported that their RCI housed no children below this age, and that when babies and toddlers came to their centers they were placed in foster care if their families could not be traced. One center had piloted a successful program with the local hospital, so that babies abandoned at the hospital were transferred to the NGO, which then arranged foster placements. This program might be expanded to cover other provinces. Several participants noted particular challenges related to working with children with disabilities. In interviews, some participants noted that it is expensive to follow the MSACC when caring for children with disabilities, in part because the staff-child ratio must be lower in these cases. Some participants said there is a high incidence of children with mental disabilities in the alternative care system. The lack of a comprehensive social services network in Cambodia, coupled with a low level of mental health services, and a lack of cultural understanding of mental health, has led to RCIs becoming a placement option for children with disabilities (Jordanwood 2011)37. Some participants said they struggled to find RCI staff trained to work with children with disabilities, and that it was sometimes necessary to advocate for transitional shelters and RCIs to accept children with mental disabilities because they were often violent and disruptive. “Kids with mental disabilities are often depressed and when they’re placed in the center they get into fights, as a result it’s difficult to find organizations to accept them.” 3PC staff member Several participants interviewed who worked with and within RCIs expressed the belief that RCIs were not appropriate treatment options for children with mental health issues, and discussed ways they hoped to improve conditions for these children. One RCI had taken an exposure visit focused on learning how to take better care of children with disabilities. Another had worked with a long term volunteers to design better programing. However, the research team was not able to ascertain whether children with disabilities were given priority in reintegration processes. Lessons Learned Regarding Reintegration As mentioned earlier, reunification was also supported within these projects when MoSVY, UNICEF and Project Sky (not a 3PC affiliated partner) worked collaboratively to close an RCI in Siem Reap in 2012. This was the first government and CSO supported closure of a substandard RCI. In this process 30 children were placed, first in temporary care in a government RCI and then placed in their families following the closure of a severely substandard RCI. Two children were found to have been sexually abused by a staff member of the original facility, and four showed signs of physical abuse. This reunification process generated a large number of lessons regarding challenges to reunification and reintegration in the Cambodian context. These have been combined with lessons regarding reintegration in the Family+ and 3PC projects in the summary below. 37 Jordanwood, 2011. With the best intentions. Phnom Penh: MoSVY and UNICEF. 19 Jordanwood (2014)38 found that regular and long-term follow-up of reintegrated children was an important component of reintegration, and that it increased the chances of reintegration success. In the MoSVY-Project Sky reintegration pilot DoSVY was identified as the partner that would implement follow-up of children but was not allocated additional staff or resources by MoSVY or UNICEF. As noted earlier, DoSVY is significantly understaffed and under-resourced, and most participants interviewed who were involved with the MoSVY-Project Sky reintegration process reported that DoSVY had failed to make follow-up visits to these children. Project Sky as a result sought funding from an external donor to conduct follow-up for six months, but stressed this was not sufficient to meet the needs of these traumatized children. A lesson learned in this case is that if DoSVY is given a role implementing care provision to children, it must be allocated funds to do this job, or else the reintegrated children will not receive adequate services. A further lesson was the importance of making a plan that takes into account RCI staff prior to beginning reintegration. Staff in RCIs that are being closed have no incentive to support the process that will eventually result in them losing their positions. In the MoSVY-Project Sky reintegration process, staff actively undermined the process. Rosas (2012) 39 advises that caregiver staff be replaced during the process of closure. In a center in which staff have abused children this is essential. However, in a center in which staff have formed close relationships with children, as many caregivers in RCIs do, it might be wise not to withdraw these formative relationships during this period of transition for children. If relationships between staff and children are positive there might be other options. For example, Cambodian Children’s Trust (CCT) transformed its RCI into a family-based care center and the staff were given jobs in the new center. This allowed children to maintain valuable relationships as the center transitioned, and decreased the potential for staff sabotage of the process. Staff involved in reunification or reintegration must be highly trained. They must be able to identify and respond to abuse sensitively. This training must apply to the staff involved in all aspects of the reintegration process. It was reported that in the MoSVY-Project Sky pilot the police who brought children for medical checks regarding the suspected sexual abuse did not respect the children’s confidentiality, and the doctors who examined children were insensitive. Reintegration requires medical professionals trained in assessment for abuse, mental health professionals trained in support for addressing trauma in children, and legal professionals trained in issues related to child custody, prosecution of and child-friendly justice systems. Reintegration is an extremely complex, collaborative process. The MoSVY–Project Sky reunification pilot required the implementers to identify a wide range of partners, with expertise in mental health support, legal prosecution, income generation support to name a few. Family+ and 3PC partners drew on a similarly wide range of NGO and government partners. Another lesson learned is the importance of anticipating all these partners in advance, and recognizing that these service providers may require additional training. Effective NGO networking can link implementing agencies, with a range of necessary services, but it is important not to assume that local authorities will have knowledge of local services. A second aspect of this is the importance of designating very clear roles for each partner and to develop an enforceable monitoring mechanism to ensure services are delivered in a timely manner. Within the MoSVY-Project Sky pilot, the partner responsible for the legal prosecution of alleged offenders failed to act. Rosas (2012) 40 advocates for a strong legal response mechanism to be established prior to the beginning of any reintegration intervention. 38 Jordanwood, M., 2014. Achieving positive reintegration: assessing the impact of family reintegration. Phnom Penh: Friends International. 39 Rosas, S., 2012. Lessons learned: MoSVY-UNICEF-Project Sky reunification pilot. Phnom Penh: Project Sky. 40 Ibid. 20 A further issue is the lack of clarity regarding legal roles, particularly regarding guardianship. This was noted by participants in the interviews for the 3PC and Family+ evaluations, and by Rosas (2012) 41 describing the MoSVY-Project Sky pilot. In the latter, once the children had been placed in the care of the government RCI, the police were reportedly unsure as to who bore the responsibility for prosecution of offenders. In addition, laws addressing the custody of children separated from their families, as well as the laws required to place a child in protective state custody made it difficult for the Project Sky to take steps to protect abused children. In some cases it is necessary to ensure that duty bearers such as the police are aware of existing laws, in other cases the development of further legal frameworks is necessary. Both Jordanwood (2014)42 assessing the impact of reintegration from 3PC partner RCIs, and Rosas (2012)43, describing the MoSVY-Project Sky pilot recommended that reintegration programs recognize the importance of support programs to children once they are reunified with their parents. The Cambodian education systems relies so heavily on the payment of informal school fees, that it is often described as a hybrid public/private system (Brehm et al., 2012) 44. Some CSOs regard these informal school fees as a form of corruption, and therefore will not pay them. When they are not paid, children suffer significantly impaired access to education (Jordanwood 2013)45 Rosas (2012) 46 further noted the importance of facilitating the process of school transfer. While this was not found to be an issue with Family+ and 3PC reintegration processes discussed earlier, because these projects offered services to facilitate transfer, it was a problem in the Project Sky pilot reunification pilot. A desire to access education has been shown to act as an incentive to place children in RCIs, and participants believed that left unaddressed it would result into reentry into RCIs. However, the greatest incentive to place children in RCIs is poverty (Jordanwood 2011, Holt 2005)47. Studies of families who place children in care, and of families of reintegrated children have offered anecdotal examples of extremely poor and destitute families (Jordanwood 2011, Langler and Reimer 2007) 48. In these cases placement in RCI is a compassionate sacrifice by the family for the welfare of the child. Income generation programs are a key part of the Family + and 3PC reintegration models, and Jordanwood (2014) found they impacted reintegration success. Likewise, Rosas (2012) 49 identified them as a key factor in supporting reintegration in the 2012 MoSVY-Project SKY process. The initial plan to rely on DoSVY to provide follow-up services, without allocating resources, resulted in many children from this reunification initiative failing to receive adequate school support or income generation. The evaluation team was only able to reach five parents with children reunified in the pilot. In all these cases the children were enrolled in school, or were over the age of 18 and were employed. Seven out of the nine children continued to live with their families, while the other two live independently. However, some parents said their children had had better access to resources in the RCI. “My house has no roof or walls, and we are afraid we will be evicted.” Parent of child placed in family from the MoSVY-Project Sky pilot 41 Ibid. 42 Jordanwood, M., 2014. Achieving positive reintegration: assessing the impact of family reintegration.. Phnom Penh: Friends International. 43 Rosas, S., 2012. Lessons learned: MoSVY-UNICEF-Project Sky reunification pilot. Phnom Penh: Project Sky. 44 Brehm, W.C., Silova, I. and Mono, T., 2012. Hidden privatization of public education in Cambodia: the impact and implications of private tutoring. New York: Open Society Foundation Education Support Program. 45 Jordanwood, M., 2015. The impact of informal school fees on access to school of members of the care-leavers network. Phnom Penh. 46 Rosas, S., 2012. Lessons learned: MoSVY-UNICEF-Project Sky reunification pilot. Phnom Penh: Project Sky. 47 Holt, 2005. Cambodia orphanage survey. Eugene: Holt., Jordanwood, 2011. With the best intentions. Phnom Penh: MoSVY and UNICEF. 48 Langeler, E., and Reimer, K., 2007. The road home: Toward a model of ‘reintegration’ and considerations for alternative care of children trafficked for sexual exploitation in Cambodia. Phnom Penh: Hagar/World Vision. 49 Rosas, S., 2012. Lessons learned: MoSVY-UNICEF-Project Sky reunification pilot. Phnom Penh: Project Sky. 21 Reintegration was again portrayed as a compromise, bringing happiness when it reunified families, but also bringing hardship. “I felt happy (at reintegration) but I also felt hopeless, because at first my children couldn’t be in school and we didn’t have enough food to eat.” Parent of child placed in family from the MoSVY-Project Sky pilot System Elements Providing Case Management Services within RCIs A key objective of Family+ was “to increase the number of children reintegrated from residential facilities and to reduce intake through improved family support and prevention from abandonment” (Family+, objective 1). The Family+ logframe anticipated that the outcome of functional case management systems would contribute towards achieving this goal (Family+ logframe, outcome 5). The outcome is matched to three indicators intended to measure the percentage increase in family-based case management: the percentage increase in future planning, action planning and case notes, and the percentage increase in beneficiaries accessing at least three services through the NGO or referral. Unfortunately, as a result of the lack of a baseline assessment, it is not possible to measure the percentage increase in these indicators. However, Family+ did record that 400 new family-based cases were opened, as reported by all the partners, during the course of the project. In qualitative interviews conducted for this evaluation, Family+ and 3PC national staff and RCI partners were asked to describe case management systems. All Family+ partners received training in case management. All Family+ and 3PC partner RCIs were reported to have case management systems in place. Five out of the seven Family+ RCI partners, and all 3PC partners reported having monthly case management review meetings. All Family+ and 3PC partners visited as part of the evaluation were verified to be keeping case files on children in their care, although due to concerns regarding the confidentiality of clients, the details of case files were not shared with evaluators. Sample data collected during center assessments by FI (on 10 percent of the case files), indicated that 51 percent of beneficiaries had documented future planning, action plans, and comprehensive case notes. This was lower than the target of 75 percent, and suggests that Family+ could have done more to improve case file management. As part of the study (Jordanwood 2014)50 on reintegration amongst 3PC partners, case files of children who had been reintegrated within the last two years were assessed, and found to be largely incomplete. One participant noted that as a result of this study, the case file management of reintegrated children was being reformed, and that more accessible digital forms of case management were being developed. Several participants described lessons learned related to note taking in case files. Through the coaching activities, FI training staff identified a problem: social workers were not taking regular case file notes. Implementing social workers had not received formal university training. This is a common challenge faced in social work in Cambodia (Harachi 2014) 51. As the issue was explored, RCI social workers explained that they were people who “know how to do, not how to write”. They resisted taking case file notes, because it was perceived as a formal activity. When pushed to write case notes, some staff would write long formal compositions. Once the problem was identified FI staff were able to work with social workers to help them understand that case file notes need to be entered regularly, and usually will be only a few sentences long. FI training staff also said that RCI staff had previously perceived note taking to be a bureaucratic chore, and that key to behavior change was to help social workers to understand that good note taking could influence case management practice. The UNICEF SCPS project also worked to improve case management systems, through supporting case management in government RCIs. UNICEF supported MoSVY to hire 15 social workers who at the time 50 Jordanwood, M., 2014. Achieving positive reintegration. Phnom Penh: Friends International. 51 Harachi, T., 2014. Review of Social Work Practice. Phnom Penh: UNICEF. 22 of conducting research had completed case files and family tracing in Phnom Penh and Sihanoukville RCIs, as mentioned earlier, and were planning to expand this process to Battambang and Siem Reap. Impacts on Vulnerable Families SCPS, 3PC and Family+ were designed to work directly with families in the community, to address the second objective for the Action Plan for Children in Adversity, supporting and enabling “families to care for their children; prevent unnecessary family-child separation; and promote appropriate, protective, and permanent family care.” SCPS, 3PC and Family+ were designed to strengthen child protection systems and services from grassroots to national levels, in order to impact the lives of children. Essentially all activities were intended to have a direct or indirect impact on children. This section focuses on activities that provided direct services offered to children and families as well as outreach. Direct Services SCPS aims to support access for vulnerable children and their families in five target provinces to specialized, quality child protection and social welfare services (SCPS, objective 1). Families and children are being supported through a range of interventions. The 3PC logframe recorded significant numbers of children and families who had received direct benefits through CSOs, exceeding targets. The 50,391 children and youth who were reported to have received direct services through work with CSOs, far exceeds the target of 33,343, and 18,315 families received direct services, exceeding the target of 6,908 (the 3PC logframe, outcome 4). The SCPS logframe recorded that 532 children had benefited from case management, de-institutionalization for permanency planning and reintegration conducted directly by MoSVY staff. The target of 800 children (500 in year 2, 300 in year 3) was not reached. Family+ aimed to offer improved and increased prevention and family support services amongst NGOs. To achieve this outcome it provided reintegration services (discussed in the section on reintegration above), income generation activities (IGAs) both to families with reintegrated children, and to vulnerable children in the community as a prevention measure, and foster care support. The Family+ project measured household income before and after the IGAs, and its logframe reports that as result of IGAs family income increased by 52 percent, exceeding the target of 25 percent (Family+, outcome 3). Eighty￾nine percent of families said that IGAs helped facilitate children’s attendance at school, exceeding the target of 85 percent (Family+, outcome 3). Sixty-five percent of IGA businesses were still in operation after 12 months, which was lower than the target of 90 percent (Family+, outcome 3). The vulnerability of these very poor families, and the instability that was mentioned by families describing their lives post￾reintegration, may have contributed to the failure of some IGA businesses. Foster Care 3PC and Family+ both worked to develop and improve foster care programs, which have not been widely implemented in Cambodia. The Family+ logframe recorded a 36 percent increase in foster care placements amongst NGO partners, exceeding the target of 30 percent (Family+ logframe, outcome 4). 3PC developed and shared a foster care SOPs, and one RCI partner reported opening a new foster care program as a result of an exchange visit. In the last year one CSO that has the longest running foster care program in Cambodia joined the 3PC partnership, and this organization brings significant expertise to the partnership that should be shared as the project continues. Participants noted that foster care works well in a country if the definition of family is broader than the nuclear family. As a result families are accustomed to having large households with people from outside the nuclear family residing with them, and this provides a model for accepting children in foster care. In the few evaluation interviews with foster families, participants described loving, prosperous functional families. Participants from CSOs, however, did note that the legal provisions mean that foster families who would hope to adopt children find it difficult to do so. 23 Support Groups During evaluation interviews many participants described the new 3PC initiatives to bring support groups to families and children. Many parents in families who send their children to transitional homes have a history of alcohol, drug or domestic abuse (Jordanwood 2014)52. 3PC partners have set up support groups for these parents, and participants said these have been very successful in lessening harmful behaviors. Participants offered anecdotal evidence that these groups have a broad impact: when parents were free from substance abuse or ceased to beat their children, participants found that the family income improved, the children did better in school and the overall family dynamic improved. Advocacy in the Community Family+ and 3PC also conducted advocacy within Cambodia targeting Cambodian families to promote family-based care and raise awareness regarding child abuse among families. Family+ aimed “to affect positive perception or behavior change amongst public…on keeping families together” (Family+, objective 2) The campaign was disseminated through 15 local newspapers and other local media outlets (Family+, outcome 8). The 3PC logframe and the Family+ logframe report that this campaign reached 505,703 people (Family+, outcome 8). The impact of this campaign was not measured. Impacts on Donors to RCIs SCPS, through 3PC, and Family+ activities aimed to change funding behaviors of donors and key gatekeepers through advocacy. Family+ conducted an advocacy campaign, “Children are not tourist attractions” (Family+, outcome 8). This campaign targeted tourists who fund RCIs. Tourists may visit RCIs as a brief activity as part of a larger trip, or may arrange to volunteer at RCIs for both short and long periods. This latter group are often referred to as “voluntourists.” RCIs receive funding from tourists both during their visits and once they have returned home, although there have been no studies dedicated to assessing the funding streams of RCIs. The Family+ advocacy campaign recognized that RCIs draw on the emotions of tourists to fund their centers, and therefore countered this with an alternative emotional plea, not to create orphans through funding RCIs. Campaign messages were placed in key tourist destinations in Cambodia, as well as being placed in advertisements in magazines both within Cambodia and on in-flight magazines on airlines entering the country. Messages were also shared through websites in the expectation that this media form would reach younger potential voluntourists. FI also worked with an ethical business network that embedded campaign message in the guide for engagement with NGOs which they distribute to businesses. 3PC, in partnership with Family+, also managed the ChildSafe Network, which bought together groups in the larger community including vulnerable children, key gatekeepers in the business community (for example, tuk-tuk drivers, guest house personnel, vendors) and foreign travelers. Through the ChildSafe Network, 3PC advocated for child safe behaviors and prevented abuse. The network provided ChildSafe training and advocacy materials promoting ChildSafe behaviors. In addition, the Network operated a ChildSafe Hotline for reporting abuse. The 3PC logframe reports that over the funding period, ChildSafe Network enrolled 1,845 members, far exceeding the target goal of 885. While the parent group number (420) is lower than the target (595), the other groups have so significantly exceeded the target that the evaluation team has evaluated this as positive. The large and dispersed target groups of the ChildSafe Network made it difficult to assess the success of advocacy campaigns, however, the increase in membership suggests that the ChildSafe Network was positively viewed in the larger community, and the ChildSafe Hotline recorded 8610 calls, exceeding its target of 5300, demonstrating that a large number of people were aware of the ChildSafe Hotline number and had used this information to contact FI. 52 Jordanwood, M., 2014. Achieving positive reintegration: assessing the impact of family reintegration.. Phnom Penh: Friends International 24 Between 2013 and 2015, FI Family+ and ChildSafe campaign websites recorded over 10,000 visitors and messages in the ethical care network were shared with an estimated 60,000 partner businesses. Due to the extremely large readership of the inflight magazines in which advocacy advertisements had been placed, it is estimated that 3.9 million people in total were directly reached by the campaign message. Sixty international media channels and 15 local media outlets carried FI alternative care messages, suggesting that the issue of voluntourism and the RCI care model was being widely discussed. It is difficult to measure the impact of this campaign because of the wide international dispersion of the target group of tourists. However, there are some concrete signs that the campaign influenced RCI supporters. The website of Four Square Children of Promise, a prominent founder and funder of RCIs in Cambodia, addresses the FI campaign directly. The website embeds a picture of the campaign video, then labels the video as a “devilish deception”. 53 A few RCIs have contacted FI requesting support with reintegration of children citing reduced funding which they attributed to the changing climate towards residential care. FI also reports that it now receives inquiries from voluntourists who had planned trips to work in RCIs, and had subsequently encountered messages from the advocacy campaign. These volunteers contacted FI for advice on whether to proceed. In addition, as mentioned earlier, FI frequently receives reports of violence in RCIs from voluntourists who report that they learned of FI through the campaign. Moreover, after working with FI, Al Jazeera recorded a video story about voluntourism in Cambodia. 54 This piece identified a business called Projects Abroad, which charged individuals to arrange trips to volunteer in Cambodian RCIs. After the piece aired, Projects Abroad committed to stop sending volunteers to RCIs in Cambodia. IMPACTS ON OVERARCHING ISSUES Research Commissioned Research was conducted over the course of SCPS, Family+ and 3PC in order to inform decision-making, planning and advocacy. SCPS aims to use “rigorous evidence-based policy and baseline data …for advocacy, leveraging resources and measuring trends” (SCPS, objective 4). Project reports from 3PC phase 1 note that the project completed the following research: Mapping and Assessment of Child Protection System in Cambodia (2011), social service mapping in 117 villages, and the Primary Report on the Functional Mapping of MoSVY (MoSVY, 2011) and With the Best Intentions, a study on attitudes towards residential care (Jordanwood, 2011). The logframes document that 3PC and Family+ achieved their outcome targets regarding research during the course of these project. 3PC aims to influence government policy through the development of thematic research. 3PC project reports record that 3PC achieved its target number of research studies (3PC, outcome 1). 3PC also supported some research studies that were reported under Family+ outputs, and the UNICEF SCPS project achieved its aim of supporting MoSVY to conduct the Mapping of RCIs, discussed earlier. Family+ aimed to conduct research to improve practices of its partners, as well as the wider community. The Family+ logframe records that the project achieved its target of promoting greater understanding of beneficiaries through the publication of three Knowledge, Attitudes and Practice (KAP) surveys (Family+, outcome 6). These focused on special needs (disabilities and mental illness), foster care, and street children profile. SCPS and Family+ anticipated conducting baseline research in their project proposals. It was unclear why this was not done, and as a result it was more difficult to measure impact. Participants in interviews for this evaluation were able to point to many instances in which research had influenced policy and practice. MoSVY and NGO participants in interviews cited several ways in which the findings of With the Best Intentions55 had informed program planning. The Primary Report on the 53 Devilish Lie - FCOP International, accessed October 01, 2015. 54 http://www.aljazeera.com/programmes/peopleandpower/2012/05/201252243030438171.html, accessed August 24, 2015. 55 Jordanwood, M., 2011. With the best intentions. Phnom Penh: UNICEF. 25 Functional Mapping of MoSVY56 was credited with influencing the current push for decentralizing of some MoSVY roles. The significant impact of the Mapping of RCIs (MoSVY, in press) is discussed earlier in the section on government impacts. Several practitioners said that the findings of the KAP surveys had informed programmatic activities. Participants also described the ways in which the findings of Achieving Positive Reintegration57 had impacted practice, for example, promoting exchange visits to one successful project mentioned, and spurring an organizational review of reintegration case file management. Specific Systems Changes that Contributed to National Care Reform Regulatory Framework Through the SCPS Project UNICEF has supported MoSVY to develop policy and research that has contributed towards systems change. Prior to the beginning of these projects, MoSVY had issued the MSACC, which provided standards for alternative care. Over the course of SCPS Projects UNICEF has supported MoSVY to issue policy and research providing the tools to help to ensure RCIs meet these standards and to provide alternative care in the community. The Prakas to Implement the PACC58, clearly assigns roles and responsibilities within national and local government for providing alternative care. The Sub-decree for Management of RCIs59 also supported by UNICEF, states all RCIs must be registered with MoSVY and are subject to MoSVY inspections and oversight thus supporting the implementation of the MSACC. Finally, the mapping of RCIs provides information on the location of both registered and unregistered RCIs within target provinces, providing additional information that MoSVY will need to ensure that the Sub-decree is implemented. While gaps still exist in policy in the area of foster care standards, and clearer national adoption procedures are needed, significant progress has been made as a result of UNICEF support and advocacy to MoSVY. 3PC Partnership Over the course of these projects, UNICEF and FI worked together to build the 3PC partnership. 3PC aims “to strengthen child protection through CSOs enhanced capacity, coordination with and contribution to national and sub-national protection responses” (3PC objective). Many of the outcomes of 3PC, in terms of strengthening government policy and capacity, providing relevant research, supporting the improvement of services in RCIs, and advocating for family-based care have already been discussed. However, there is an additional outcome of 3PC, the system of partnership and collaboration between CSOs that warrants further discussion. CSOs are the main providers of alternative care services in Cambodia, due to the lack of budget allocation for implementing alternative care services within the government. Prior to 3PC, many of these CSOs offered similar services in different geographic locations, with limited or no coordination. 3PC brought nine CSOs together to work in a collaborative system. Through 3PC, partners could learn from each other’s experiences and best practices. The exchange programs, discussed earlier, allowed partners to learn about one another’s successful programs. In many cases these best practice programs were replicated. 3PC conducted networking meetings and an additional 46 meetings with government officials (3PC logframe, outcome 6). The networking meetings allowed partners to share difficult cases and learn from others experiences. A few participants noted that it was a sign that trust had been established, when partners began to share challenges as well as successes. 3PC also established a referral system. The 3PC logframe reports that the cases of 102 children were referred between 3PC partners by 2015 (3PC, outcome 6). 3PC meetings and trainings allowed 3PC members to establish personal relationships with other practitioners, which participants said were key in facilitating the referrals. The referral process allowed partners to link children and families with services 56 MoSVY, 2011. Primary report on the functional mapping of MoSVY. Phnom Penh: MoSVY. 57 Jordanwood, M., 2014. Achieving positive reintegration. Phnom Penh: Friends International 58 MoSVY, 2011. Prakas on Procedures for implementing the policy for the alternative care for children. Phnom Penh: MoSVY. 59 MoSVY, The sub-decree on management of RCIs, not yet published. 26 that the referring CSO did not offer. 3PC members gave examples of referring to mental health services and vocational training that would otherwise have been unavailable. Referrals also allowed partners to access services in another area of the country. One of the challenges of reintegration is that children from a single RCI may have families of origin in different provinces. Jordanwood (2014)60 found that children in the random sample had been relocated to every province in Cambodia. Follow-up visits, therefore, are often expensive and time consuming. Referrals within the 3PC network helped to address this issue. “If we have only one case in the province it’s a lot of money to spend to visit that child, that’s why 3PC is helpful.” 3PC partner staff member Participants did note areas for improvement within 3PC. Some felt 3PC trainings, discussed earlier, were not sufficiently rigorous or relevant. Others noted that 3PC could be more organized, and said they received notices of trainings in a different province less than 24 hours prior to the meeting. Some partners felt that 3PC management staff were not genuinely interested in learning from other partners, noting that 3PC often promoted the work of FI affiliated partners at the expense of other partners. In addition, several participants from CSOs found the 3PC logframe problematic. The logframe records, among other data, the number of children receiving different kinds of services, and a few participants explained that the criteria for these services were not sufficiently specific and were not verified. Overall, however, the CSOs interviewed supported and valued the network, “Nationally it is the strongest child protection mechanism in Cambodia.” 3PC partner staff member Piloting Models In evaluation interviews, several participants noted that many initiatives that had been piloted as part of the 3PC and Family+ projects had not been previously tried in Cambodia. These included the support to transition RCIs into community centers, the 3PC collaborative partnership of CSOs, the partnership with Angkor Hospital for Children to identify abandoned babies and place them in foster care, the development of case management within government RCIs, the advocacy campaign targeting tourists and the closure of RCIs with collaboration between MoSVY and UNICEF to reintegrate children. These interventions provided examples that acts as a rejoinder to the often voiced criticism that family-based care would not work in Cambodia. 3PC and Family+ demonstrated that family-based care can work, and this provides models for national care reform. SUSTAINABILITY Sustainability is achieved when host country partners take ownership of the development processes, maintaining projects and project impacts after the cessation of the project. SCPS, Family+ and 3PC have had a number of sustainable impacts. As noted earlier, with support from UNICEF, MoSVY has created a regulatory framework that outlines roles and responsibilities of government in alternative care, and mandates the registration and monitoring of the RCIs within Cambodia. The mapping of RCIs61 provides new information on the number of RCIs, registered and unregistered, within five provinces, This information brings a significant number of RCIs into the government monitoring data base, and the Sub￾decree on Management of RCIs62 affords MoSVY the power to more effectively regulate these RCIs. Participants in interviews predicted that these RCIs will continue to be monitored after the end of the USAID-funded project. In supporting this work UNICEF has helped MoSVY to build a sustainable framework at the highest level of government. 60 Jordanwood, M., 2014. Achieving positive reintegration. Phnom Penh: Friends International 61 MoSVY, not yet published. 62 MoSVY, 2015. The sub-decree on management of RCIs, Phnom Penh: MoSVY. 27 UNICEF and FI worked to create the 3PC network, which has established a system of sharing best practices and referrals among NGOs across the country. 3PC partners believed that the programs that were established as a result of the exchange visits would continue to operate independently of USAID funding. The 3PC referral system was also predicted to be a sustainable system, as NGOs would continue to require each other’s services and expertise, and the connections built as part of the 3PC system would allow partners to access this. In addition, FI senior staff described the 3PC partnership itself as a system that would continue to operate beyond the limit of the USAID project funding. Many participants noted that there had been a notable change in attitude regarding alternative care provision, and favoring family-based care over residential care. This attitude shift was evidenced in many stakeholder groups: MoSVY, DoSVY, Family+ and 3PC partners, RCIs and among some donors. This attitude shift was less established among local authorities, many of whom referred to 3PC partners and officiated at reintegration, but at the same time still offered views supporting RCIs. However, participants from government at the national level affirmed their commitment to continue supporting reform of alternative care systems and practice in favor of family-based care. MoSVY staff interviewed affirmed their intention to continue RCI inspections and to expand the development of case files in government RCIs. As noted earlier, several partner Family+ RCIs had begun reintegration or foster care programs for the first time as part of these projects, and these participants all predicted that these efforts would be ongoing. There was sense that Cambodia was on the path to creating an alternative care system that viewed family-based care as the preferred option for children, and that this would be sustained. However, at the same time MoSVY did not have sufficient funding to be able to implement the programs supporting regulation of RCIs and supporting family-based care independently. As noted earlier, the majority of MoSVY funding is allocated to veterans’ pensions and salaries at central level. UNICEF has worked with MoSVY to advocate with the Ministry of Economics and Finance to increase allocations to MoSVY. It is unclear whether this will occur, and in the meantime, MoSVY programs cannot be considered sustainable. RCI partners explained that it was more difficult to find funders for reintegration programs than for residential programs. The families that send children into RCIs are vulnerable, and this vulnerability remains post-reintegration. Families continue to require support after accepting their children, and to ignore this is to place children at risk. There is a danger that facing a lack of support, RCIs will either reintegrate all children, and then close their centers, leaving the children without follow up services, or will revert to placing children in residential care in order to attract funding. So while the attitude shift has occurred in many partner RCIs, it may not be sustainable unless a concurrent attitude shift occurs amongst more donors. CONCLUSION The SPCS, 3PC and Family+ projects worked to influence the key power-bearers at each level of the alternative care system in Cambodia; government at the national level (MoSVY), and sub-national level (DoSVY, local authorities), RCIs, donors to RCIs and lastly, vulnerable families. UNICEF supported MoSVY to make significant changes at the national level. As a result of USAID support, MoSVY was supported to issue two important policy documents, the Prakas to Implement the PACC63, which details roles and responsibilities within the government for providing alternative care and the Sub-decree on Management of RCIs64, which states that all RCIs must register with MoSVY, and includes a provision which mandates action if alleged abuse is reported within RCIs. 63 MoSVY, 2011. Prakas on Procedures for implementing the policy for the alternative care for children. Phnom Penh: MoSVY. 64 MoSVY, 2015. The sub-decree on management of RCIs. Phnom Penh: MoSVY. 28 In addition, MoSVY conducted the mapping of RCIs in five target provinces, identifying more RCIs in those five provinces than had previously been registered in all of Cambodia. As a combined result of the mapping and the sub-decree a significant number of previously unregistered RCIs will be bought under MoSVY oversight. Moreover, with UNICEF support, MoSVY has taken steps to improve alternative care services for children and families. These steps include the first rigorous, comprehensive annual inspections of RCIs, the first RCI closures and reintegration of children within them, and the first case management within government RCIs with the aim of reintegrating eligible children into permanent protective placements. Improvements have also been documented at the local level. With Family+ and 3PC support, DoSVY begun to refer significantly more children to Family+ partners, and DoSVY have been present to sign documents at all Family+ and 3PC placements. However, DoSVY engagement in implementation of service provision remains rare, due to a lack of staff and funding for service provision. CSOs have experienced greater success in collaborations with local government in the CCWC, due to their strong knowledge of the local context. The 3PC and Family+ projects improved the conditions of vulnerable children living without family care in RCIs. Both projects worked to increase staff capacity in RCIs. Initial efforts to achieve this through offering training sessions were less successful, however, coaching and exchange visits had a significant impact, and staff working in RCIs were able to offer concrete examples of programs in their RCIs that had been initiated or improved as a result. The projects also worked to strengthen the protective environment in Family+ partner RCIs, and to respond to reports of violence and abuse toward children in all RCIs. Family+ partner RCIs were supported to implement child safeguarding policies, and to create abuse reporting structures. The few cases of abuse that were reported within Family+ or 3PC partner RCIs were responded to quickly and appropriately. Moreover, UNICEF and Friends International received frequent reports of alleged violence or abuse from non-partner RCIs, suggesting that abuse is prevalent in RCIs in Cambodia. This is an issue that warrants further research. 3PC worked to ensure that the government responded to reports of abuse from non-partner NGOS, through preparing and verifying report summaries, which were then passed on to DoSVY and, if no action was taken, MoSVY. When necessary, UNICEF would then advocate with MoSVY to take action. Concerns were raised by participants that in this procedure CSOs were fulfilling a role that is the responsibility of government. However, the system did appear to be encouraging MoSVY to assume a greater responsibility for responding to reports of abuse. A few participants explained that although MoSVY had been inactive when the system was first introduced, MoSVY had become increasingly engaged in responding to reports of abuse from non-partner NGOs overtime. Given the importance of this reporting process, MoSVY must be both committed and funded to take on this entire role before 3PC passes this function to the Ministry. Family+ and 3PC reintegration methods contributed to stable and sustained placements for children. Children were reintegrated from Family+ and 3PC partner RCIs, and as part of the MoSVY-Project Sky 2012 reintegration pilot, supported by UNICEF. A report on the impact of reintegration on children from 3PC RCI partners found that most children lived in stable and sustained placements post￾reintegration, but that these families continued to be vulnerable to shocks, and required ongoing support. A number of lessons were learned from these reintegration processes. First, it was found that frequent and long term follow-up, and income generation support both positively impact reintegration success. It was noted that as a result DoSVY can only be relied on to fulfill these roles if additional funding is made available. Unless and until additional staffing and funding are allocated to DoSVY if children are to be placed in family care, CSOs will have to ensure that adequate monitoring and support is provided to these children and the families in which they are placed. Second, it is important to make provisions for longer term employment options for RCI personnel as part of a process to support an RCI to transition 29 to supporting family care instead of residential care in the reintegration plan, because otherwise they have an incentive to sabotage the process. Third, partnerships between supportive CSOs must be built to support the process, and fourth, all actors in the process must have received training on their roles. Finally, clear roles and responsibilities must be assigned in the reintegration plan, and these must be time￾bound and monitored. These may include the roles and responsibilities of MoSVY staff and local authorities, police, legal professionals, health care staff, and CSO staff, among others. Partners in the reintegration plan must be clear as to the actions they are required to implement, and the time in which these actions must be completed. It is not sufficient, for example, to assume that reporting to police abuse discovered as part of a reintegration process will automatically result in a timely investigation of this abuse. Interventions must be planned and monitored. SCPS, 3PC and Family+ worked to support vulnerable children and families. Through 3PC 50,391 children and youth, and 18,315 families received direct services through work with CSOs. Family+ provided income generation activities (IGA) both to families with reintegrated children and to vulnerable children in the community as a prevention measure. Families who had fostered children as part of 3PC program were also supported. 3PC pioneered successful support groups for people struggling with alcohol, drug and domestic abuse, which were believed to have a larger impact on their family welfare. 3PC advocacy campaigns worked to change the attitudes of international donors and voluntourists who constitute the funding base for RCIs in Cambodia. Between 2013 and 2015, FI campaign websites recorded over 10,000 visitors, messages in the ethical care network were shared with an estimated 60,000 partner businesses, and an estimated 3.9 million people in total were directly reached by the campaign message. Moreover, 60 international media channels and 15 local media outlets carried FI alternative care messages. Despite the efforts of the projects, the system for child protection and care remains dependent on the roles played by NGOs. This is found in both the role that RCIs assume in accepting children who could have been provided services at home if the government provided social services to children in vulnerable families living at home, and in the roles played by NGOs in providing services such as reintegration and family-based care support, or response to allegations of abuse that are considered by many to be the responsibility of the government. However, at this point, as a result of the lack of staffing and funding for MoSVY, particularly in the area of service delivery, it is unrealistic to plan for the government to take on these roles during the duration of the projects being evaluated. Within these larger constraints, the SCPS Project, 3PC and Family+ were able to contribute towards system changes contributing to national care reform. UNICEF’s support to MoSVY enabled the development of key policy and research that allowed for evidence-based decision making and strengthened the national regulatory framework. The 3PC project produced a partnership between CSOs that created a system of shared best practices and referrals across the country. Participants believed that both of these interventions would be sustained after the end of the three projects. 30 RECOMMENDATIONS Priority recommendations are listed in bold. The evaluation team after conducting this evaluation recommends to UNICEF that they support the government to:  Develop foster care minimum standards and guidelines,  Develop policy that clarifies laws and procedures regarding domestic adoption,  Advocate and promote attitude change in DoSVY and local authorities, to foster greater engagement in both prevention and reintegration follow up, and build greater engagement with CCWC,  Strengthen the system for responding to reports of violence from RCIs, through stronger enforcement of the MSACC, enforcing punitive actions for failing to meet these standards, and strengthen the existing mechanism managed by the government and 3PC partnership for responding to allegations of abuse from within RCIs nationally.  Expand the mapping of RCIs to all provinces so that all unregistered RCIs can be monitored and regulated by MoSVY,  Expand RCI inspections to all RCIs identified in the mapping, and conduct inspections every six months,  Redirect organizations requesting permits to open RCIs to opening family-based care centers instead,  Continue to close substandard RCIs, with a priority on closing those in which children are being physically or sexually abused, in a process which includes supported reintegration,  Regulate the alternative care sector through the implementation of the Sub-Decree on Residential care Institutions,  Develop links to existing foster care programs to provide alternatives to the current government RCIs, and  Commission research on the prevalence of violence in RCIs. The evaluation team recommends that 3PC strengthen its programming in the following ways:  Continue to build partner capacity focusing on coaching and exchange visits instead of training;  Promote the expansion of 3PC-partner existing foster care programs through sharing of best practices and successes in foster care among donors and 3PC partners;  Reintegration o Ensure that children who are reintegrated receive adequate follow-up and support; o Ensure that staff assisting in reintegration are adequately trained and skilled;  Expand the program linking babies abandoned at hospitals with foster care;  With government collaboration, increase focus on children with disabilities within RCIs;  Ensure that SOPs developed as part of Family+ continue to be available to 3PC partners and other RCIs and CSOs;  Continue to advocate for attitude shifts among donors and voluntourists to RCIs in Cambodia (Ongoing advocacy is important because of the high turnover of the 2 million plus tourists who visit Siem Reap each year.);  Expand advocacy messages so that examples of RCIs that have successfully transitioned into family-based care models are shared with RCI staff and donors,  Develop a baseline prior to implementing programming interventions; and  Clarify among 3PC partners how indicators are reported, measured and verified (Particular attention should be paid indicators recording the number of children receiving services such as placements into families by 3PC partners.). 31 ANNEXES ANNEX I: LOGFRAMES The final logframes used for the evaluation of the three projects. SCPS These indicators for the SCPS project were provided by UNICEF for inclusion in the evaluation on 21 July. This was agreed by the Advisory Committee. Outcome indicator Outcome baseline Outcome target Results/indicators achieved Comment 1. Number of local authorities trained by Child Welfare Department on the Prakas Procedures to Implement the Policy on Alternative Care for Children. (The wording in the Grant is: Introductory trainings on the AC framework to 200 stakeholders, as part of the dissemination of the AC Prakas) - 200 245 As reported by UNICEF. 2. Availability of Regulatory Frameworks related to Alternative Care for Children developed by MoSVY. (The wording in the Grant is: Development of the by-law regulating compulsory registration, sanctions and closures of institutions/national consultant) N/A Yes Yes Please see 3PC Outcome 3 and 4 3. Number of residential care institutions inspected by Child Welfare Inspection Team, (The wording in the Grant is: Inspection of Residential Care Institutions - None 2010: 269 2011: 216 2012: 219 2013: 225 2014: 228 There was no specific target defined for this, as the number of residential care institutions keeps changing. UNICEF supported MoSVY to do the inspection of all registered residential care institutions in the 5 priority provinces. 4. Availability of report on the Mapping of residential care institutions N/A Yes Yes Please see 3PC Outcome 3 and 4 32 ;5. Number of children that benefited from Case management, de￾institutionalization for permanency planning and reintegration conducted directly by MoSVY staff of 800 children (500 in Year 2, 300 in Year 3) - 800 532 As reported by UNICEF. 3PC Objective: Strengthen CP through CSOs enhanced capacity, coordination with and contribution to national and sub-national protection responses. Outcome Indicators Outcome Baseline Outcome Target Results / Indicator Achieved 01 Number of government policies influenced by 2014 0 2 2 Two policy documents, Foster Care Policy and Definition of Centers, were developed by FI. 02 Awareness of child abuse is increased by 2014 no data 60% of children, youth and adults in the communes have heard about the campaign and remember at least two messages. No data A pretest has reportedly been conducted in Siem Reap, but no survey data from after the campaign has been provided to the evaluation team. 03 Organizational capacity of CSOs is increased by 2014 59% average across partners 89% Qualitative; see report body FI could not provide an explanation for how the baseline and target figures were set for this indicator. It was suggested that a qualitative reviewed is done for this indicator. 04 Quality service capacity of CSOs is increased by 2014 59% 89% Qualitative; see report body FI could not provide an explanation for how the baseline and target figures were set for this indicator It was suggested that a qualitative reviewed is done for this indicator. 05 Service satisfaction beneficiaries no data 70% of respondents are satisfied with the quality of services 84% 33 Based on Hackett's 2014 survey: Regarding services, most children and young people rated a high (63%) or very high (21%) level of satisfaction. 06 Positive attitude towards networking and coordination no data 60% of respondents have positive attitudes 100% While it is not irrevocably clear what the target audience of this indicator is, all CSO directors and staff interviewed for this evaluation responded positively to networking and coordination and expressed a desire to continue or even expand these activities both within network and externally. Objective: Strengthen CP through CSOs enhanced capacity, coordination with and contribution to national and sub-national protection responses. Outcome Indicators Output Indicators Output Target Results / Indicator Achieved 01 Number of government policies influenced by 2014 Number of thematic researches documented 3 3 As listed in draft report, 5 research studies have been conducted and documented: - Child Protection and Child Service - Child Protection and Child Right - Alcoholic Abuse - Snapshot Survey and Street Children Profile 2014 - Family Reintegration Impact The evaluation team concludes that the last 2 have also been reported as part of the Family+ framework. The result is thus evaluated as 3. Number of meetings with government authorities 30 46 As reported in the baseline data report, regular meetings with DoSVY, other Provincial authorities or Commune Councils and other local authorities were held quarterly by some CSOs and monthly by other CSOs in 2011. In Hackett's survey in 2014, 7 of the 9 CSOs attend monthly meetings with commune or local authorities. Some meetings at provincial and national levels are also reported in Hackett's report in 2014. Number of good practices documented 6 8 34 Hackett's evaluation reports the following good practice documents: - FI Vocational Training guidebook - 3PC Database guidebook - 3PC Child Protection in Emergency Guidelines - Case Study Guidelines In draft report, 5 good practice documents are listed: 1. Guidebook for Outreach used by MT 2. Guidebook for Drop in Center used by MT 3. Guideline on how to develop 3PC Case Study (same as above, so not counted towards the total) 4. Instruction for 3PC monitoring sheet 5. Referral Database Service Guideline and directory for Hotline and Social Worker Some of these documents are also referred to as SoPs in another 3PC indicator (below). The evaluation team also notes the similarity of this indicator to one of the Family + indicator, which is associated with different best practice documents. Number of advocacy strategies 2 3 02 Awareness of child abuse is increased by 2014 3 campaigns launched: - Children are not tourists attractions - Don't create more orphans - Keeping family together Number of advocacy strategies 2 3 Duplicate indicator. Number of people reached 150,000 505,703 03 Organizational capacity of CSOs is increased by 2014. This indicator is also included in the Family + framework, with the same target. Number of good practices documented 6 8 Duplicate indicator. Number of child/youth beneficiaries (Directed benefit) 33,343 50,391 35 This number includes 40,861 (Female: 15,154) children and 9,530 (Female: 3,091) youth. Beneficiaries were also grouped under: - Outreach (34,461) - Referral children (1,808) - Children in prison (432) - Children from families (4,914) - Child/youth network members (607) The evaluation team also notes that in Hackett's report, the target for this indicator was listed as 29,900. Number of trainings provided to other 3PC partners 9 23 There were 23 trainings provided to other 3PC partners with a total of 238 participants. Topics conducted included IEC development, children with disability, alcohol, ChildSafe/ child protection, IPSS, the right of children, child conflict with law, storytelling, reintegration, case management, micro￾enterprise, child safe organization. Number of exchange visits to other 3PC partner 54 64 04 Quality service capacity of CSOs is increased by 2014 Total number of exchange visits stands at 64, up from 51 in Hackett's 2014 report. Number of trainings provided 120 100 In Hackett's 2014 report, FI was reported to have provided regular training to CSOs, for a total of 75 trainings. The figure 100 includes trainings to 3PC partners as well as non-3PC CSOs and also local authorities (CCWC). Number of training participants 1,200 1,539 In Hackett's 2014 report, the number of training participants was 1086. The currently reported number of participants compared to the number of trainings since then seems to indicate the average number of participants per training has increased. Number of support visits 72 76 There were reportedly 76 support visits to CSOs during 3 years of the program, conducted by FI. Number of SoPs developed 6 7 36 According to draft report, 7 SoPs were developed: 1. FI Home Based Production Guideline 2. FI Vocational Training Guidebook 3. 3PC Database Guidebook 4. Child Protection in Emergency (CPiE) Guideline 5. Foster care guidebook 6. Reintegration guidebook 7. Job placement guidebook The evaluation team notes that some of these overlap with "good practice" documents as reported in another indicator. Membership groups (including ChildSafe) ChildSafe: 885 PG: 595 C/YG: 320 ChildSafe: 1,845 PG: 420 C/YG: 356 PG = parent group, C/YG = children and youth group. While the parent group number is lower than the target, the other groups have so significantly exceeded the target that the evaluation team has evaluated this as positive. Number of beneficiaries reintegrated 2,067 1,604 draft report reported 3,509 for this indicator. However, this number included beneficiaries reintegrated to: - Public School (1,654) - Job (384) - Family/Relative (1,273) - Foster Care (72) - Others; independent living, group house (259) The evaluation team decided to only count reintegration into family, foster care, and other AC arrangements - excluding school and job. This is also in line with the Family + reported definition of reintegration. Number of calls to ChildSafe hotline 5,300 8,610 The number of calls significantly exceed the target. Number of family beneficiaries 6,908 18,315 The target was exceeded mostly through the provision of outreach services. Benefits received by families were: - Socio-Economic Support (593) - Material Support (2,205) - Emergency/Disaster Support (1,888) - Outreach to caretakers (13,803) Number of trainings provided to other 3PC partners 9 23 37 Duplicate indicator. Number of exchange visits to other 3PC partner 54 64 Duplicate indicator. 05 Service satisfaction beneficiaries Number of children access to services 9,012 18,841 Clarification on this indicator by UNICEF/FI, it refers to the number of children who actually received services, not how many had had access to them in principle. Types of services were: Drop in center, Transitional home, Pre￾school, Remedial class, Vocational training, and Diverted. How this indicator relates to service satisfaction is not clear to the evaluation team. 06 Positive attitude towards networking and coordination Membership groups (including ChildSafe) ChildSafe: 885 PG: 595 C/YG: 320 ChildSafe: 1,845 PG: 420 C/YG: 356 Duplicate indicator. Number of meetings conducted in communities by partners 904 807 It is unclear to the evaluation team to which degree this indicator could overlap with "network meetings" (another indicator, below) in so far that these were conducted with local authorities at the community level. If these were to be included, it is possible the target may be achieved. Number of exchange visits to other 3PC partner 54 64 Duplicate indicator. Number of trainings provided to other 3PC partners 9 23 Duplicate indicator. Number of children referred from (3PC) partners to partners (internal referral) No target 102 38 In Hackett's report in 2014, there're 38 whole cases referred between 3PC partners. This indicates that the practice of referrals has picked up significantly since then. There was however no target to evaluate against. Network meetings No target 232 These network meetings refer to meetings conducted with CSOs outside 3PC as well as government partners. The meetings were about collaboration, social work, child protection, the role of CCWC, identifying services for children, strengthening child protection systems, ChildSafe networking, health networking, victims of trafficking, children in conflict with the law, foster care networking, sexual abuse and violence towards women and children, school dropout issues, human trafficking, child rights. Number of meetings with government authorities 30 46 Duplicate indicator. Family+ Objectives Outcome Outcome indicator Outcome baseline Outcome target Results/indicators achieved To increase the number of children reintegrate d from residential facilities and reduce intake through improved family support and prevention from abandonme nt. 01 - Best practice model systems for Alternative Care and prevention services developed and implemented All project partners demonstrate improved AC approaches through reports, monitoring and evaluations 0 7 7 As reported by FI. The project partners are MS, KM, MCH, HVC, KMR, HL and VCAO. It should be noted that VCAO has since closed due to reasons not associated with the project (lack of donor funding). 80% of trainees report increased knowledge of AC systems and curricula 0 80% 88% This is the average of the pre- and post-test assessments of participants in all provided trainings. It was calculated and verified by the evaluation team from FI records. 39 Best practice models implemented by all project partners 0 7 7 All partners have received support on reintegration, case management, child protection policy, etc. and have implemented these best practice models. 02 - Improved TH/Center services amongst NGO and Orphanage partners, based on RGC Minimum Standards % increase in AC Center Standards Assessment carried out by FI 0 15% increase annually MS: 15.74%, KM: 5.2%, MCH: 6%, HVC: 19%, KMR: 2%, HL: 9.4% Unweighted average: 9.56% This indicator was explained by FI as referring to two AC standards assessments carried out for each partner, with the target being the difference between the two, as a measure of improvement. The results of the assessments were published in the final project report; the unweighted average was used to evaluate this indicator against the target. The evaluation team notes that most centers scored around 70% on the first test (the highest was 73%); this indicates that the target of 15% increase per annum over the life of the project may have been overly ambitious. # of children living in the centers that achieve increase of 15% per annum in AC Center assessments 0 1200 children / 2 years 742 (includes youth) Related to the previous indicator, as explained by FI, this refers to the number of children living in the centers meeting the 15% increase through assessments, during the 2 year period. The two centers meeting this criteria were MS and HV, and the reported number includes youth (18-24). The evaluation team would like to note that this indicator seems somewhat inappropriate compared to the projects' overall objectives. If it has already been determined which centers have improved, there does not seem to be a need for another indicator on how many children live in those centers. In fact, it could be argued that this indicator actually encourages taking in more children into those centers. 40 # partners with Child Protection Officers with clear role and internal reporting mechanisms 2 7 5 Five partners (MS, KM, HCV, MCH, and KMR) have a dedicated CP officer as described in this indicator. The other partners do not, but it should be noted that they do have clear internal reporting mechanisms for children - those centers are simply too small, in terms of staff numbers, to have dedicated CP officers. 03 - Improved and increased prevention and family support services amongst NGO partners. Average $ percentage increase in family incomes $112 25%/ 27 months 52% ($171) % HBP/ME families report IGA helped facilitate children’s attendance at school 70% 85% / 27 months 89% % ME businesses still in operation after 12 months 80% 90% 65% HBP = home based production, ME = micro enterprise. This is the average of estimates made by the partners, which was reported to FI. 04 - Improved and increased reintegration to family, kinship and foster family carried out by NGO and Orphanage partners. # children benefit through being reintegrated to immediate and kinship family units 0 770 / 27 months 426 children 69 youth This indicator does not cover additional forms of reintegration. On overall, 838 children and youth were reintegrated back to biological family, kinship care, foster care, group home, and independent living. FI notes regarding the wording of the indicator "children", that all indicator targets for Family + were designed also with youths in mind, because it does extensive work with 19 and 20 year olds. % increase in foster care placements amongst NGO partners 0 30% / 27 months 36% No baseline data was provided for this indicator, but the evaluation team was able to reconstruct this from FI records. These records show 16 children were placed in foster care during the course of the project, and that at the end of the project there were 60 placements, both old and new. There was thus an increase from 44 to 60. 41 % reintegrated children still in family / kinship units 12months after placement 0 65% 69% Data collected from the partners. # partners carrying out reintegration services for the first time. 0 4 (RCIs) 4 All orphanage partners started carrying out reintegration services after having received training from the Family + projects. Some had previously attempted reintegration, but were not successful. The evaluation team notes "carrying out reintegration services" should be interpreted as having been successful. 05 - Improved services to children through functional case management systems operational in all NGO partners % increase in family-based case management (rather than individual CM) 0 30% increase in Family CM per annum 400 This number refers to the number of new family cases opened, as reported by all the partners, during the course of the project. Due to lack of baseline data, it is not possible to determine whether the percentage increase target has been achieved. % of sample data demonstrates beneficiaries have documented future planning, action plans, comprehensive case notes. (demonstrates managed progression through services) 0 75% of random sample data 51% Based on a sample collected during center assessments by FI, of 10% of the case files. % of beneficiaries in sample data accessing at least three services provided by NGO or through referral (demonstrates holistic care) 0 70% of random sample data Not available 42 This sample data was not collected. It was however reported in the final Family+ report that around 1,645 families had received at least 3 services from Family+ partners (counseling/health/emergency support). To affect positive perception / behavior change amongst public, civil society and government target audiences on keeping families together (family based care) 06 - NGOs have improved research, data collection and good practices shared # NGOs utilize electronic data collection practices 0 3 4 This number refers to the ChildSafe data collection at Kaliyan Mith, Mith Samlanh, M’lop Tapang, and Friends-International (Hotline team). Good practices demonstrate benefits of family￾based approach 0 6 9 Although this indicator's wording is unclear, FI explained it refers to documentation and sharing of good practices. The number includes 4 guidebooks (first aid and vaccination, reintegration, foster care, and independent living and group homes), 1 research report (impact of family reintegration), 2 best practice reports developed along with RUPP interns (group home and prevention child abandonment) and 2 case studies developed and shared with partners. Greater understanding of beneficiaries through # of KAP surveys. 0 3 3 The three KAP surveys completed are on special needs (disabilities and mental illness), foster care, and street children profile. 07 - RGC increases skills and capacity in AC principles, services, and monitoring % Govt / Local Authorities report improved knowledge in curriculum of training 0 75 84% The average of pre- and post-tests conducted of participants for each training. % of reintegrations carried out with MoSVY / DoSVY involvement 60% 80% 100% All reintegration cases involve DoSVY to some extent, even if it is only signing the form. 43 % increase in referrals from Govt / Local Authorities to NGOs for AC issues 0 30% increase per annum Apr '13-Sep '14: 85 Oct '14 - Jun '15: 169 The number of actual referrals are given for each time period. Due to lack of baseline data it is impossible to assess whether the target percentage increase has occurred. increased # of AC Prakas assessments undertaken by RGC 10% 30% increase No data No data was collected. The meaning of the baseline is also unclear. 08 - Perceptions and behavior changes within wider public concerning Alternative Care and family based care developed / improved % RUPP students report increased AC knowledge through training 0 80% 70% A training workshop on “Social Work Child Protection Best Practice: Alternative Care” was provided to 58 social work students and approximately 70% of participants increased their knowledge as reported by the training facilitator. % RUPP report increased fieldwork confidence through participation in FI good practice assessments 0 80% 100% (see assumption) The evaluation assumes that this indicator was meant to read "% RUPP interns", based on information from FI. 8 interns have been working with FI during the project, of which 2 are currently still engaged. Based on oral interviews all of them reported some level of increased confidence and knowledge. # foreigners and tourists reached through ‘Children are not tourist attractions’ campaign. 500,000 (combine d total 2011 & 2012) 350 000 new people (expecte d reach over 27 months) 3.9 million people # people reached through Cambodian community campaign. 0 150 000 505,703 44 Estimated by FI. This is difficult to verify exactly, but based on news source website views, TV programme viewership, and the distribution of the in-flight magazine, it does not seem implausible. # local newspaper, mass media reports concerning community campaign 0 10 per year At least 15 media outlets; no data on number of reports According to the project report, around 15 national media outlets reported on the campaign, including radio, newspapers, and TV channels. There was no data collected on actual number of reports, so it is not possible to definitively validate this indicator. Based on experience in Cambodia, the evaluation team does find it likely that the target has been achieved given the nature of the outlets. Additionally, there was also a poster campaign, which the evaluation team believes should be included in this indicator as well. # international media / conference channels through which FI advocates for AC standards figures not currently recorded 9 per year 2013: 16 2014: 17 2015: 27 Based on list of media articles provided by FI for each year, verified via web links, and duplicate channels (which had more than one article) removed. 45 ANNEX II: EVALUATION SOW Attachment A: Statement of Work World Learning on behalf of USAID will contract with Emerging Markets Consulting (EMC) for a performance evaluation of two projects that it is funding in Cambodia. The timeframe for services is June 1, 2015 to August 3, 2015. The dual purpose of the evaluation is to inform USAID’s wider program for care reform in the country as well as the implementation of the projects, themselves. The first project is funded through UNICEF, “Strengthening Child Protection Systems in Cambodia to prevent and respond to violence, abuse, exploitation and unnecessary separation of children.” The second is “Family+” for which Friends International (FI) is responsible. Both projects are addressing care reform, and each includes multiple national partners, including governmental bodies and NGOs. While two separate projects are to be evaluated, each with its own objectives and time frame, USAID has decided that arranging for them to be evaluated at once will be helpful in providing timely information for its Family Care First initiative (http://familycarefirst.com/ ), which will address care reform in the country. It also anticipates that efficiencies can be achieved, since the two projects are operating in overlapping geographic areas. The two projects are expected to contribute to the achievement of Objective 2 of the United States Government’s Action Plan on Children in Adversity (APCA): “Put Family Care First” – U.S. Government assistance will support and enable families to care for their children, prevent unnecessary family-child separation, and promote appropriate, protective and permanent family care. Details on APCA are available at: http://www.childreninadversity.gov/about/how/action-plan . The evaluation is expected to consider the relevance of each of the projects to USAID’s care reform agenda in Cambodia (addressing APCA objective 2) as well as the projects’ efficiency and effectiveness and their potential for contributing to sustainable outcomes. Process and Deliverables USAID requires that the evaluation activities be planned and implemented in ways that actively demonstrate ethical information gathering from and about children and their families, are gender sensitive, and give attention to the rights and needs of children with disabilities. It is also expected that the process will be participatory and build upon information and perspectives provided by key governmental, civil society, and international stakeholders. It is also expected that findings will be based on triangulation of inputs from multiple sources. Deliverables for this contract are as follow: Deliverable #1: Following review and analysis of key documents prior to the initiation of key informant interviews and site visits, the evaluation team is to submit to World Learning, USAID/Cambodia, and USAID/DCOF for review a detailed evaluation plan, including:  Evaluation design: questions; methods; data analysis plan; 46  Schedule (evaluation timeline),  Key informant interview guides for the primary categories of key informants, and  Anticipated limitations to the evaluation.  List of the documents reviewed, and proposed data collection instruments (e.g. key informant interview guide). Deliverable #2: Submit to World Learning, USAID/Cambodia, and USAID/DCOF minutes from the in-country briefing with USAID/Cambodia and agreed upon site visits and proposed list of interviewees; Deliverable #3: Contractor will submit to World Learning, USAID/Cambodia, and USAID/DCOF a summary of key results that will be used in the in-country briefing with UNICEF, Friends International personnel, and perhaps Government stakeholders, as well as with USAID Washington and Cambodia personnel, on initial findings before writing a draft report. Meeting notes also to be submitted to World Learning by the contractor. Deliverable #4: Contractor will submit to World Learning, USAID/Cambodia, and USAID/DCOF a draft of the final evaluation report* prior to an out- with USAID/Cambodia. The draft report will be reviewed by USAID/Cambodia, USAID/DCOF, and the Evaluation Advisory Group. Meeting notes to be submitted by the contractor. USAID will provide comments on the draft report. The following describes the outline of the report to be prepared by the contractor*:  Table of Contents  List of Acronyms  Executive Summary  Introduction  Background  Methodology  Limitations  Analysis  Findings and Conclusions  Recommendations  References  Annexes o Evaluation SOW** o Data collection instruments i.e. interview guides o Evaluation schedule o List of persons contacted/interviewed. The names and positions need to be removed in the final version of the report that will be posted on USAID’s Development Clearinghouse. o Disclosure of any potential conflict of interest forms (for each evaluation member) o Statement of Difference (from evaluation team member, implementing partners, or other donors funding the projects) 47 *The contract agrees to include the following language in the final evaluation report: This study/report (specify) is made possible by the support of the American People through the United States Agency for International Development (USAID.) The contents of this (specify) are the sole responsibility of (name of organization) and do not necessarily reflect the views of USAID or the United States Government. **Attachment B of this contract Deliverable #5: Final evaluation report. The evaluation team will incorporate USAID and other comments and submit the final report after receiving these comments. USAID/Cambodia requests an electronic version of the final report in MS Word. The final report should be concise and specific. Annexes should be included with the final report. The evaluation report must not exceed 30 pages in length excluding annexes. Note: The contractor will provide an internal version of the final evaluation plan for USAID and an external version for public circulation. Attachment B – Program Description Approach and Methodology Background The past decade has seen a surge in the number of residential care centers in Cambodia (Jordanwood 2011). The Ministry of Social Affairs, Veterans and Youth (MoSVY) maintains a record of registered residential care centers in the Alternative Care Database. In 2005 there were 153 registered residential care centers; in 2015, this number now stands at 254,65 an increase of 66 percent. Moreover, the initial results of mapping project being conducted currently by MoSVY and UNICEF suggests that the number of unregistered residential care centers may be even higher. Cambodian government policy, the PACC and MSACC, (MoSVY 2006, MoSVY 2008) strongly states that placement in residential care should be a last resort. In line with this, MoSVY has begun working with members of civil society to close residential care centers that endanger children as they repeatedly fail to meet the MSACC, and to reintegrate children into their families. During this process MoSVY has worked closely with UNICEF and Family +, with the intent of strengthening systems that offer social services to children and promoting reintegration of children in safe, permanent placements (Jordanwood 2015). As USAID designs its Family Care First initiative, there is a need to evaluate the success of these projects, and to build on lessons learned for future programming. Project Objectives This evaluation will assess the degree to which the UNICEF’s project, Strengthening Child Protection Systems in Cambodia, and Friends International’s project, Family +, 65 This number has decreased due to MoSVY’s efforts since 2010, when it stood at 269, a 75 percent increase. 48 have met their stated objectives and performance targets. The 8 overarching questions listed in the RFP will provide a framework for the evaluation. In particular, this evaluation will assess the programs’ progress and best practices through in-depth assessment of quantitative and qualitative program data. Secondary data will be triangulated through in-depth interviews with project stakeholders, including NGOs, civil society organizations, relevant Cambodian government ministries, USAID, and target beneficiaries. Methodology The project will proceed in three phases, as outlined below. Document review A thorough desk review will be conducted at the onset of the evaluation process focused on the eight overarching research questions. The researchers will seek input from key informants through email and/or in-person interviews to access additional documents and to better understand listed documents. This desk review will be divided along qualitative and quantitative lines. The EMC team will develop an analytical framework and hypotheses regarding key factors for success in the project in order to guide research. Mia Jordanwood, who is the author of two studies in the list of documents (Jordanwood 2011a, Jordanwood 2015), will lead a qualitative review of program documents, working first towards a clearer definition of key terminology. Next she will analyze project descriptions to create a single document that summarizes goals and objectives of both projects. Finally, she will assess documents, identifying challenges, successes, and lessons learned to find evidence of meaningful, measurable impact of programs. In parallel with this research, the EMC team will undertake quantitative analysis of available program data in order to assess the extent to which projects are on track to reach initial targets, as well as to assess the pace of progress towards these targets during the project implementation period Research Planning Leveraging our extensive in-country networks and qualified expatriate and Khmer team members, EMC’s approach to this project will ensure participatory engagement of all relevant stakeholders. Based on desk review of project documentation, the EMC team will develop a plan for primary research, including questionnaires for key project stakeholders, and research tools and methodologies to collect data from program participants. Participatory research draws on the input of local stakeholders to inform the research process, incorporating local priorities in order to produce more relevant knowledge for action. This evaluation will strive to be participatory at each stage of the research process, soliciting input from local partners during document review, during research tool design, and regarding draft findings, and ensuring findings address the programming needs of UNICEF and Family +. Presentation of Methodology EMC will present the research plan and proposed discussion guidelines and other tools to the Evaluation Advisory Group and USAID Cambodia mission in order to incorporate comments and suggestions into the research plan. Stakeholder and Key Informant Interviews 49 Building on the initial results of the document review, the EMC team will design semi￾structured interview guides for key informants identified for the study. Sample Purposive sampling will be used throughout the evaluation, in order to identify participants with input that is relevant to the eight overarching questions. Purposive sampling will also allow researchers to ensure, in line with the tenants of participatory research, that the views of power-bearers within projects are included in the research. The researchers will ask participants in the initial list of stakeholders to recommend additional participants. Key informant interviews will be conducted in the designated research sites with members of USAID, UNICEF, Friends International, Mith Samlang, Kaliyan Mith, Komar Reakrey, Homeland and MoSVY and DoSVY staff, as well as additional stakeholders identified during the development of the research plan. Research with Project Beneficiaries The team will develop:  Semi-structured interview guide for parents of children who received UNICEF and Family + services  Semi-structured, child friendly interview guide for small group interviews with children. Guides for children will be visual conversation cards, which have already been used successfully in residential care in Cambodia. Research tools will be shared with both the Evaluation Advisory Group and UNICEF and FI project management staff, as well as key stakeholders, in order to refine tools. Adults and children who have received services provided by the projects will be contacted through project staff. In addition to these interviews, researchers will conduct in each province (Phnom Penh, Siem Reap, Battambang, and Sihanouk Ville)  4 interviews with adults whose children received services from the UNICEF and Family + projects.  2 group interviews with 2-3 children per group with children who were reintegrated or received services as a result of UNICEF or Family + projects. In Siem Reap alone researchers will also conduct:  4 interviews with adults whose children were reintegrated as part of the Project Sky MoSVY pilot 66 Key Elements of Research Methodology REGULAR UPDATES TO PROJECT STAKEHOLDERS Regular reporting and discussion with clients at key stages of research is incorporated into all EMC project work plans. For this project, we understand key stakeholders to be the USAID Cambodia mission and the Evaluation Advisory Group to be convened for the purposes of this evaluation. EMC will present research methodology to these stakeholders at the start of the project; will present summary overview of preliminary 66 Susan Rosas’s study (2012), Lessons Learned: MoSVY/UNICEF/Project Sky Reintegration Pilot Project will inform these interviews. 50 analysis upon completion of field work; and will present the draft project report for comments from key stakeholders. ETHICAL CONDUCT All members of the team have demonstrated expertise in conducting research in an ethical, culturally appropriate and child-centered manner. Throughout the research process researchers will: - Follow a child protection code of conduct - Follow guidelines to prevent causing participants (particularly children) distress - Ensure research conducted in a child-friendly manner - Ensure research is gender sensitive - Ensure data protection Appendix 2 outlines the specific guidelines and procedures that the team will follow in this regard. Incorporation of International Best Practices BEST PRACTICES IN EVALUATIONS Thomas Poulsen, the evaluation specialist proposed for this project, has over 5 years’ experience in evaluations in accordance with OECD best practices. His advisory role in this project, along with the extensive experience of key team members in undertaking evaluations for major clients (including USAID, the World Bank, the ADB, Oxfam, CARE, and others) will ensure this project is undertaken following international best practice. BEST PRACTICES IN PRIMARY RESEARCH All team members traveling to the field as part of this project – Olga Creamer, Matt van Roosmalen, Sao Setka – will undergo in-depth training by Mia Jordanwood on international best practices in research with children. These are detailed in Appendix 2 to this methodology. Analysis Data analysis will be conducted by EMC team members, and will be focused on finding meaningful answers to the eight overarching questions, assessing programs’ performance against targets and identifying evidence to make predictions and to inform realistic recommendations. Researchers will triangulate data from multiple sources to improve reliability of findings. Mid-Point Briefing Following initial analysis, the research team will assess and discuss initial findings, and present these to the USAID Cambodia mission and the Evaluation Advisory Group. The EMC team will incorporate USAID Cambodia and Evaluation Advisory Group comments into the analysis and these will be reflected in the draft report. Draft report The draft report will be written by the EMC team, and edited by the full research team. It will be shared with several key informants as well as the Evaluation Advisory Group, in order to verify data and solicit comments, which will inform the final draft. 51 ANNEX III: PROJECT VENN DIAGRAM Source: Evaluation plan, based on meeting with UNICEF and FI project teams on June 10, 2015, and approved by the evaluation advisory committee on July 15, 2015. 52 ANNEX IV: INFORMATION SOURCES List of Reviewed Documents 1. 3PC baseline report, 2011 2. 3PC review, 2015 3. A gap analysis towards strengthening the child protection system building in five provinces in Cambodia August 2011 Friends International UNICEF 4. AC Workplan Friends International April 1 2013-Sept 30 2013 5. AC Workplan Friends International Sept 01 2013- Aug 31 2014 6. AC Workplan Friends International Sept 01 2014-June 30 2015 7. Achieving positive reintegration: assessing the impact of family reintegration, M. Jordanwood, 2014 8. Cambodia orphanage survey, Holt International, 2005. 9. Child protection KAP survey 3PC partners 2012 10. Child protection KAP survey Phnom Penh MS 2012 11. Child protection KAP survey Siem Reap KM 2012 12. Children in informal alternative care, UNICEF, Jini Roby 13. Copy of DCOF summary budget 2013 14. Copy of DCOF summary budget Feb 2015 15. Copy of DCOF summary budget May 2015 16. DCOF Quarterly Progress Report Apr 1 2013-June 30 2015 17. DCOF Quarterly Progress Report April 1 2014-June 30 2014 18. DCOF Quarterly Progress Report January 1 2014-March 31 2014 19. DCOF Quarterly Progress Report January 1 2015-March 31 2015 20. DCOF Quarterly Progress Report July 1 2013-Sept 30 2013 21. DCOF Quarterly Progress Report July 1 2014- September 30 2014 22. DCOF Quarterly Progress Report October 1 2013-December 30 2013 23. DCOF Quarterly Progress Report October 1 2014-December 30 2014 24. Family Care First, Workshop Summary Report 11-13 March 2015 25. FI responses December 22 2014 26. FI responses May 26 2015 27. Final report with comments JW 28. Friends international Family+ budget 2013 29. Friends International Family+ DCOF application January 2013 30. Group home case study Battambang 31. Hidden privatization of public education in Cambodia: the impact and implications of private tutoring, Open Society Foundation Education Support Program, W.C Brehm, I. Silva, T.Mono, 2012. 32. In best or vested interest: an exploration of the concept and practice of family reunification for street children, Consortium for Street Children, T. Feeney, 2005. 33. Learned from the MoSVY/Project Sky, S. Rosas 34. Middle Way, NGOCRC, S. Gourley, 2010. 35. Minimum standards on alternative care for children May 2008 RGC 36. National action plan on early childhood care and development 2014-2018 37. National action plan on early childhood care and development 2014-2018 38. Partnership program with civil society organizations to strengthen child protection systems 2012 Friends International & UNICEF 39. Policy on alternative care for children April 2006 RGC 40. Potential short-term international volunteer’ perceptions of children’s residential care in Cambodia, Friends International, 2015. 41. Prakas on procedures to implement the policy on alternative care for children MoSVY 42. Program Description Family+ April 1 2013-June 30 2015 43. Review of social work practice: An emphasis on public and child welfare, Tracy Harachi, UNICEF 53 44. Revised DCOF-UNICEF expanded proposal January 2015 45. SCPS Final Report to USAID June 2009-Sept 2012 46. SCPS UNCEF First Progress Report May 2013-February 2014 47. SCPS UNCEF Second Progress Report January-June 2014 48. SCPS UNCEF Third Progress Report January-December 2014 49. Strengthening Child Protection Systems in Cambodia to prevent and respond to violence, abuse, exploitation and unnecessary separation of children, UNICEF Proposal to DCOF/USAID - January 2013 50. The prevention abandonment program, Soksophorn 2014 51. UNICEF answers comments January-June 2014 52. UNICEF proposal to the DCOF 2013 53. UNICEF responses April 8 2015 54. United States Government Action plan on children in adversity 55. USAID comments on Jan-June 2014 report 56. USAID comments on July-December 2014 report 57. USAID comments on June 2013 report 58. With the best intentions, M. Jordanwood, UNICEF 2011 54 ANNEX V: DATA COLLECTION INSTRUMENTS Semi-Structured Key Informant Interview Guides Interviewer: Location: Stakeholder: Residential Care Center Directors or Staff -Family+ Members Institution: The evaluator introduces her/himself and recalls the objectives of the interview, stressing that it is important to hear the interviewee’s perspectives. The evaluator also makes sure that no potentially interfering actors (such as UNICEF/FI project staff) are present. Before starting the interview, the interviewer will explain clearly that the interviewee’s privacy will be protected and that their answers will not be shared with others in a way that identifies them personally. Information about the Center 1. Can you describe in a few short sentences the main activities of your organization? Is your organization registered with a government ministry? If so, which ministry? (If participant begins to describe the entire role of the organization, remind the participant this is one or two sentences). 2. Does this center offer any other form of service to the community other than residential care? If the participant says the center offers social work services to families in the community, ask the participant to describe these services in a few sentences. 3. Can you briefly describe the residential care services offered by your organization? How long do children stay in the center? Where do children go after living in the center (e.g. reunited with their own family, placed in kinship care, move to independent living, returned to the street. Estimate percentages if you can.) 4. Can you describe challenges faced with your shelters? Can you describe successes of your shelters? How do residential care centers impact the lives of children? Does the FI impact your work in your residential care center? If yes, please describe how. Do you think this impact will be sustained after the end of Family+? 5. Can you describe the process of children entering this center? Do you recruit children from villages? If yes, please describe how. Does the center have criteria for admitting children? If yes, please describe these. Is anyone from DoSVY/MoSVY involved? 6. Does your center use the MoSVY Alternative Care forms in your center? Yes/No. If no, why not? 7. Do children in your residential care centers with families visit their families? How many times a year? 55 More than once a month Once a month On Khmer holidays only (eg. KNY/WF/PB) Once a year Less than once a year 8. Do international volunteers work in your center? What do they do? Are they ever alone with one or more children without a staff member being present? How long do they stay? 9. How many children live in this center? Can you tell us how many children there are in the following age groups: 10. 0-2 years 11. 2-6 years 12. 6-12 years 13. 12-18 years 14. How many caregiving staff are on duty at one time? Are there different child￾staff ratios for children of different ages? If yes, can you describe the child-staff ratio on duty for the different age groups above? If no, can you describe the overall child-staff ratio? 15. Can you describe any particular attention within this shelter focused on children with disabilities, or children under the age of three? Case Files 1. Do you keep case files for all children? If yes, how do you use these case files? How often do you review case files? Has Family+ helped you with your case management systems? If yes, please explain how. If no, please explain why. Do you share case files with MoSVY or DoSVY? Trainings 2. Which Family+ trainings did your center sign up for? Which Family+ trainings did your center receive? If there were any trainings you did not receive, what were the reasons? Did staff find the trainings helpful? If yes, please explain why. If no, please explain why not. 3. Please describe three things you learned about child protection. 4. Please describe three things you learned about family reintegration. 5. Please describe a situation in which you or a staff member applied something you learned in training, making a difference in the life of a child. Do you think you will continue to use things you learned from Family+ after the end of the project? 56 Reintegration Practice and TA from Family+ 6. Approximately how many children have been reintegrated from your center in the last 12 months? 7. Approximately how many children have been reintegrated from your center since March 2013? 8. Do you have individualized plan for each child to be reintegrated? 9. What kind of support do you provide to reintegrated children/families? What kind of support is most helpful? 10. Are reintegrated children visited to monitor how they are doing? How often? 11. Did FI support you in this reintegration process? For how many children did FI support you? How did they support you in this process? a. Approximately how many children have been told they should not enter your center, but have been redirected instead to family care options in the last 12 months? b. Approximately how many children have been told they should not enter your center, and have been redirected instead to family care options since March 2013? c. What kind of support do you offer to these children in families in the community? 12. Do you think that most children who were reintegrated are well cared for? Do you have concerns about the wellbeing of any of the reintegrated children or their families? If so, why? How do you address these concerns? Did any children that were returned to their family run away? If yes, please explain why. Can you describe the circumstances of those children now? 13. Can you describe one situation in which you received support from FI social workers and how it was helpful? If yes, please describe. If not, please explain why not. 14. Was DoSVY involved in helping children to return to families? If yes, please describe how DoSVY was involved. If DoSVY was not involved, do you know why? Local Authorities 1. Were local authorities other than DoSVY (e.g. CCWC or village chief) involved in helping children return to families? If yes, can you describe how? If the local authorities were not involved, do you know why? 57 2. Is there a specific social worker who is assigned to support your center? Is the social worker an FI or government or other institution social worker? If yes, please ask the following (possibly about different social workers if there are multiple and note differences):  How often does he/she visit you?  How does he/she provide support to you? 3. Can you describe some lessons learned from the reintegration process? 4. Can you describe best practices from the reintegration process? 5. Can you describe a situation in which reintegration or redirected a child to family-based care made a difference in the life of a child helped by your center? Child Protection 6. Does your center have a child protection policy? If yes, what does that policy say? What changes have been made to implement it? 7. How does this center respond to violence within the center? Has this center ever reported a suspected or known case of abuse? How was the report made? What action was taken as a result of that report? 8. Are there forms of violence that you do not report (for example, children beating other children?) Semi-Structured Key Informant Interview Guides Interviewer: Location: Stakeholder: Residential Care Center Directors -3PC Members Institution: The evaluator introduces her/himself and recalls the objectives of the interview, stressing that it is important to hear the interviewee’s perspectives. The evaluator also makes sure that no potentially interfering actors (such as UNICEF/FI project staff) are present. Before starting the interview, the interviewer will explain clearly that the interviewee’s privacy will be protected and that their answers will not be shared with others in a way that identifies them personally. Information about the residential care center 1. Can you describe in a few short sentences the main activities of your organization? Is your organization registered with a government ministry? If so, which ministry? (If participant begins to describe the entire role of the organization, remind the participant this is one or two sentences). 58 2. Does this center offer any other form of service to the community other than residential care? If the participant says the center offers social work services to families in the community, or supports children in family based care, ask the participant to describe these services in a few sentences. 3. Can you briefly describe the residential care services offered by your organization? How long do children stay in the center? Where do children go after living in the center (e.g. reunited with their own family, placed in kinship care, move to independent living, returned to the street). Estimate percentages if you can.) 4. What are challenges faced by your organization in reuniting children with their families? Can you describe a lesson learned and a success story in reintegration? 5. Can you describe challenges faced with your centers? Can you describe successes of your centers? How do residential care centers impact the lives of children? Does impact your work in your residential care center? If yes, please describe how. 6. Can you describe the process of children entering this center? Do you recruit children from villages? If yes, please describe how. Does the center have criteria for admitting children? If yes, please describe these. Is anyone from DoSVY/MoSVY involved? 7. Does your center use the MoSVY Alternative Care forms in your center? If yes, do you think these are helpful? If no, can you explain why not? 8. Do children in your residential care centers with families visit their families? How many times a year? 9. Do international volunteers work in your center? What do they do? Are they ever alone with one or more children without a staff member being present? How long do they stay? 10. How many children live in this center? Can you tell us how many children there are in the following age groups: 0-2 years 2-6 years 6-12 years 12-18 years 11. How many caregiving staff are on duty at one time? Are there different child-staff ratios for children of different ages? If yes, can you describe the child-staff ratio on duty for the different age groups above? If no, can you describe the overall child-staff ratio? 12. Can you describe any particular attention within this shelter focused on children with disabilities, or children under the age of three? Child Protection 1. Does your center have a child protection policy? If yes, what does that policy say? What changes have been made to implement it? 59 2. How does this center respond to violence within the center? Has this center ever reported a suspected or known case of abuse? How was the report made? What action was taken as a result of that report? 3. Are there forms of violence that you do not report (for example, children beating other children?) Case Files 4. Do you keep case files for all children? If yes, how do you use these case files? How often do you review case files? Has 3PC helped you with your case management systems? If yes, please explain how. If no, please explain why. Do you share case files with MoSVY or DoSVY? Training 5. How many training sessions were provided by FI or other CSOs in partnership each year? How did you select training topics? Have there been follow-up training sessions or support visits? Can you give an example of a time you organization used something you had learned in a 3PC training in practice? Will you continue to use these skills from the training after the end of the project? Did any member of staff take a learning exchange with another 3PC partner? Was this helpful? Can you describe how this visit impacted work at your center? 6. Did you receive training through 3PC to improve organizational capacity? If yes, has this training resulted in changes to your organizational practice? If yes, please describe how. If no, please describe why. If yes, ask whether the organization will continue to use these skills after the close of the project. 7. Did partnership develop any standards of practice guidelines, forms or manuals that it shared with your organization? If yes, does your organization regularly use these guideline, forms or manuals? Why/why not? Data System 8. How has data on children and other beneficiaries that your organization has supported been collected and stored? Does your organization contribute to a 3PC child protection database? If no, why not? If yes: a. can you describe the role of the database? I b. does this database contribute to a national system? c. Have you experiences areas of success or challenges related to your data collection system? If so, please describe. d. Can you give an example of a way in which data collection impacted the life of a child? 9. Does FI (through partnership) monitor or evaluate your organization? If no, why not? If yes, can you describe the type of monitoring and evaluation (report/telephone/meeting etc.)? Can you describe how frequently the monitoring and evaluation occurs? Does the monitoring and evaluation adequately identify challenges and successes? Do you feel adequately supported by when your organization encounters challenges or successes? 60 Best Practices 10. In addition to project reports, has your organization produced other documents such as good practices and lesson-learned reports? If yes, how have these documents been stored and disseminated to other partner CSOs/NGOs/government partners? Have the best practices CSOs/NGOs/government partners been shared with you? If yes, can you describe how your organization used these documents? 11. What were the outreach activities that you have implemented in the communities to reach children, family and other beneficiaries? Can you give an example of how your organizations outreach activities impacted the life of a child? Local Authorities 12. Can you briefly describe your work with local authorities? Do local authorities refer child to your organization or other 3PC partners? If yes, please describe how. Do you link children from your organization with local authorities? If yes, please describe how. Do you follow up with local authorities who do not make referrals to your organization? 13. Does your organization provide training to the local authorities? If yes, please describe this training. What is the impact of this training? Can you describe any successes or challenges your organization has encountered in working with local authorities? Can you give an example of a way in which working with local authorities impacted the life of a child? W 14. How does team work with other NGOs to support children in the communities? Have there been regular meetings held with other NGOs to discuss key issues in the communities? What have been the outcomes of these meetings? Referrals 15. Have there been children referred to your organization by other 3PC partners? Have you referred children to other 3PC partners? If yes, in either/both of these cases why were the children referred? Which services were offered to the children through the referral? What was the referral process? Did you follow a standardized process? If not, can you explain why? Do you think the referral process linked the child to adequate services? Were there any important successes or lessons learned? Can you identify any areas for improvement? Will these referrals continue after the end of the project? 16. What are the main benefits of partnership? What are the main challenges of partnership? Can you identify areas for improvement? Semi-Structured Key Informant Interview Guides Interviewer: Location: Stakeholder: Friends International Senior Management Staff￾Institution: The evaluator introduces her/himself and recalls the objectives of the interview, stressing that it is important to hear the interviewee’s perspectives. 61 Before starting the interview, the interviewer will explain clearly that the interviewee’s privacy will be protected and that their answers will not be shared with others in a way that identifies them personally. Family+ 1. Can you describe the relationship between the Family+ project and partner residential care centers? 2. One quarterly report noted it was difficult to establish partnerships with residential care centers? Why was this? Were there any lessons learned in this area? 3. Were Family+ materials (manuals, forms etc.) adopted for use by partners in these projects? Have they been adopted for use by any organizations outside the partners in the Family+ project? Can you describe the relationship between FI forms and the MoSVY alternative care forms? 4. Can you describe the process that lead to FI taking on the foster care working group? Can you describe any key successes in this group? Were there any lessons learned in this process? 5. Can you describe the role of MoSVY in leading Alternative Care interventions? Can you describe the role of PoSVY and DoSVY in supporting alternative care interventions such as reintegration of children, or supporting foster care? Can you share examples of best practices in this area? What are the barriers to MoSVY, PoSVY or DoSVY participation in these cases? 6. Can you describe the different training activities of Family+? Do FI staff members conduct the trainings themselves or do staff from MoSVY conduct the trainings? How successful were these trainings? Did you encounter any challenges in training activities? If yes, can you describe them? Can you describe how the different groups that received training will use the information they learned? Did you gather any data related to the outcomes of trainings? 7. What are the differences between the challenges faced by Transitional Shelters and the challenges faced by Residential Care Centers in terms of reintegration? 8. Have reintegration methods resulted in stable and sustained family placements for children? Are there best practices regarding Family+ or 3PC supported reintegration that you can share? Are there lessons learned in this area? 9. How many partner residential care centers have taken steps to transition from residential care centers to centers supporting family-based services? How many have completed that process? How many are on track to complete that process? 10. Has Family+ or 3PC supported MoSVY to improve its monitoring capacity? If yes, please describe how. If no, please describe barriers or lessons learned. 62 11. Has the Family+ or 3PC project has resulted in measurably improved conditions for children? If yes, can you describe what those changes have been? Can you describe how these changes were achieved? 12. Has the Family+ or 3PC project resulted in specific system change contributing to national care reforms? If yes, please describe. 13. What actions are being taken to respond to reports of violence or abuse towards children in residential care centers? (Differentiate between Family+ and 3PC). Is there a system-wide mechanism for reporting and responding to abuse? How does FI receive and respond to abuse allegations? 14. Can you describe advocacy activities that resulted from this project? What form did advocacy take? Can you explain the relationship between data/research and advocacy activities of this project? Whom did advocacy target? How many people were reached? Can you describe any changes in behavior that occurred as a result of this advocacy? How were these measured? Can you describe any best practices? Can you describe any lessons learned? 15. Is there evidence that Family+ or 3PC has measurably improved the conditions of vulnerable children in the target areas? Which documents best record these improvements? 16. Has Family or 3PC achieved any systemic changes that will contribute to national care reform? Can you describe these? 17. Do Family+ and 3PC partners maintain adequate case-management systems for children as risk? If yes, please describe the change and impact. If no, please describe barriers. 18. Which capacities or systems that Family+ or 3PC has strengthened will be sustained after the project, in your opinion? Can you explain why you believe this to be true? 19. What did you feel were the most successful activities of this project? Which activities were less successful than anticipated? 20. What were the challenges? 21. Can you describe examples of best practice from this project? 22. Can you describe lessons learned from this project? [Follow-up] Have any lessons been learned that could be applied more widely about how residential care centers can be influenced and supported to change their approach from long-term care to supporting family care? Semi-Structured Key Informant Interview Guides 63 Interviewer: Location: Stakeholder: UNICEF Staff Institution: The evaluator introduces her/himself and recalls the objectives of the interview, stressing that it is important to hear the interviewee’s perspectives. Before starting the interview, the interviewer will explain clearly that the interviewee’s privacy will be protected and that their answers will not be shared with others in a way that identifies them personally.  UNICEF and MoSVY have been leading a Mapping of unregistered residential care centers. Can you describe how this research might impact the work of government, CSOs, advocacy or the lives of children? Can you describe barriers encountered as part of this process?  Can you describe advocacy activities completed within this project? What form did advocacy take? Can you explain the relationship between data/research and advocacy activities of this project? Whom did advocacy target? How many people were reached? Can you describe any changes in behavior that occurred as a result of this advocacy? How were these measured? Can you describe any best practices? Can you describe any lessons learned?  Can you describe plans for the research on informal child protection mechanisms? Can you describe how this research might impact the work of government, CSOs, advocacy or the lives of children? Do you anticipate this research influence activities to strengthen informal child protection mechanisms? Can you describe bottlenecks/barriers experienced thus far?  Can you describe plans for the research on business model of orphanages being completed? Do you anticipate this research will impact the work of government, CSOs, advocacy or the lives of children? Can you describe bottlenecks/barriers experienced thus far?  How have lessons learned from this project been applied? How have they been documented?  Did UNICEF support the establishment of a regular reporting system on violence or abuse against children focusing on institutions and prisons? If yes, approximately how many incidents were reported? How were these recorded? Can you describe how these reports of violence were responded to?  Can you describe the process of reshaping the Buddhist Leadership Initiative? Can you describe the result of this reshaping? 64  We understand that UNICEF has been supporting MoSVY to develop case management systems in government orphanages. Can you describe these? Can you describe UNICEF’s contribution to these systems?  We understand that UNICEF has been supporting MoSVY to create a deinstitutionalization plan. Can you describe this? Have any children been deinstitutionalized yet? If yes, can you describe how the process has been carried out? How are children returned to family care monitored? Who is responsible for monitoring children who have been placed in families? How often does this occur? If no, can you anticipate when this process might begin? Can you describe UNICEFs contribution to the deinstitutionalization process? Can you share any lessons learned or best practices?  Can you describe the activities of the Foster Care Working Group? Can you describe why this group is important?  Can you describe the Foster Care Network in Siem Reap? What are its aims and what has it done so far?  MoSVY has worked to develop a “Prakas on Procedures to Implement the Alternative Care Policy”. Can you describe why this is important for children in Cambodia? Can you describe barriers to passing this legislation?  The MoSVY Alternative Care Data Base was developed to record information about registered residential care centers in Cambodia. Is the data base currently operating? If yes, can you describe how it is used? If no, can you describe barriers to use? Has UNICEF supported MoSVY to operationalize the Alternative Care Data Base?  Did UNICEF support MoSVY to train CCWC and other local authorities in their responsibilities as described in government policy? Do you believe that this training resulted in sustained changes in practice amongst local authorities? Please describe.  Can you describe the key components of the MoSVY deinstitutionalization plan for Siem Reap, Battambang, Phnom Penh, Preah Sihanouk and Kandal?  Can you describe any particular attention within this plan focused on children with disabilities, or children under the age of three?  Can you describe the 2012 MoSVY/Project Sky reintegration pilot in Siem Reap? What lessons have been learned from that pilot?  Likewise, can you describe lessons learned from the January 2015 process in Siem Reap that was intended to return children from a center to family care?  Can you describe how MoSVY supports residential care centers? 65  Do DoSVY or PoSVY staff carry out family visits, case management, family￾support services and referrals in the community? If yes, can you describe best practices related to these activities? If no, can you describe barriers? How many such DoSVY and PoSVY personnel are there who address child care and protection issues?  How has government capacity at the national and sub-national level been built as a result of the Strengthening Child Protection Systems Project? Please be as specific as possible in terms of the number of personnel concerned and the capacities that have been built. Semi-Structured Key Informant Interview Guides Interviewer: Location: Stakeholder: Phnom Penh Government Partners Institution: The evaluator introduces her/himself and recalls the objectives of the interview, stressing that it is important to hear the interviewee’s perspectives. The evaluator also makes sure that no potentially actors (such as UNICEF/FI project staff) are present who might influence responses. Before starting the interview, the interviewer will explain clearly that the interviewee’s privacy will be protected and that their answers will not be shared with others in a way that identifies them personally.  MoSVY worked with UNICEF to map all residential care facilities (registered and unregistered) and to enumerate children living in them in five provinces. Can you describe how this research might impact respectively: o The work of government o CSOs o Advocacy activities o The lives of children  Can you describe barriers encountered as part of this process?  MoSVY has taken a strong position in the MSACC and the PACC as well as developed the Prakas on Procedures to Implement the Alternative Care Policy. What changes have been made so far as a result of these? How have the lives of children been affected? Can you describe best practices? Can you describe lessons learned?  What future action is planned to fully implement the above framework?  I understand that MoSVY has been developing case management systems in government orphanages. Can you describe these? Can you describe UNICEF’s contribution to these systems? Can you describe the impact of these on children?  I understand that UNICEF has been supporting MoSVY to create a deinstitutionalization plan. Can you describe this? Have any children been deinstitutionalized yet? If yes, can you describe how they are monitored? Who is responsible for monitoring reintegrated children? How often does this occur? If no, 66 can you anticipate when this process might begin? Can you describe UNICEFs contribution to the deinstitutionalization process?  Can you describe the activities of the Foster Care Working Group(s)? Can you describe why this group is important?  MoSVY has worked to develop a “Prakas on Procedures to Implement the Alternative Care Policy”. Can you describe why this is important for children in Cambodia? Can you describe barriers to passing this legislation?  The MoSVY Alternative Care Data Base was developed to record information about registered residential care centers in Cambodia? Is the data base currently operating? If yes, can you describe how it is used? If no, can you describe barriers to use? Has UNICEF supported MoSVY to operationalize the Alternative Care Data Base?  Did UNICEF support MoSVY to train CCWC and other local authorities in their responsibilities as described in government policy? Do you believe that this training resulted in any changes in practice regarding children amongst local authorities? Please describe.  Can you describe the key components of the MOSVY deinstitutionalization plan for Siem Reap, Battambang, Phnom Penh, Preah Sihanouk and Kandal?  Can you describe any particular attention within this plan focused on children with disabilities, or children under the age of three?  Did this plan incorporate any lessons learned from the 2012 MoSVY/Project Sky reintegration pilot in Siem Reap? If yes, can you explain what they were?  Can you describe how MoSVY supports residential care centers?  How many PoSVY and DoSVY staff who give particular attention to child protection and care issues are there in the following provinces: Phnom Penh, Battambang, Siem Riep and Preah Sihanouk?  Do PoSVY or DoSVY staff carry out family visits, case management, family-support services and referrals in the community? If yes, can you describe best practices related to these activities? If no, can you describe barriers?  How was government capacity at the national and sub-national level affected as a result of the Strengthening Child Protection Systems Project? Semi-Structured Key Informant Interview Guides Interviewer: Location: Stakeholder: Local government-CCWC members, Commune Council members and Village Chief Institution: The evaluator introduces her/himself and recalls the objectives of the interview, stressing that it is important to hear the interviewee’s perspectives. The evaluator also 67 makes sure that no potentially interfering actors (such as UNICEF/FI project staff) are present. Before starting the interview, the interviewer ensures that the participants have completed consent forms, and explains clearly that the interviewee’s privacy will be protected and that their answers will not be shared with others in a way that identifies them personally. The interviewer then welcomes participants and thanks them for taking the time to meet with the interviewer. The interviewer then asks the following questions. 1. Describe your position in the village:  CCWC member  Commune Council Member  Village Chief  Other (specify)__________________________________ 2. How long have you held this position? 3. In a few sentences, can you describe the main problems facing children in your community? 4. How do you help children in your community? 5. Can you describe your work with 3PC /UNCEF/FI?67 6. Did you attend any trainings by 3PC /UNCEF/FI? If yes, what were the topics of these trainings? How many days did the training last? 7. Can you tell three things you learned during each training? 8. Can you give an example of a time when you used something you learned in a training to help a child? 9. Do you think that an orphanage is a good option for children who live in very poor families? 10. Which is better for a child with no living parents: an orphanage or support to live with an aunt in the community? 11. Have you ever referred a child to an NGO (in the Family+ or 3PC partnership)68? 12. If yes69, can you explain why you referred the child, and what happened to the child after the referral? 13. Do you know where the child lives now? In your opinion, how is the child now? 14. Have you ever referred a child under age 3 or a child with a disability to an NGO (in the 3PC partnership)? 15. If yes, can you explain why you referred the child, and what happened to the child after the referral? 16. Do you know where the child lives now? In your opinion, how is the child now? 67 The facilitator will know in advance whether this participant works directly with Family+, the 3PC partnership or the UNICEF Strengthening Project, and will use language specific to the participant. 68 The facilitator may need to explain what ‘refer’ means, and may need to allow the respondent to describe an NGO they received help from by name, since many respondents may not know the names of the 3PC partners. 69 If the respondent has referred many children, ask approximately how many, and then ask the respondent to tell you what happened in 3 cases. 68 17. Have you ever worked with 3PC /UNCEF/FI, or alone to reunite a child with his or her family? 70 18. If yes, how did you help? 19. Did you make any follow up visits to the child? 20. How is the child now? 21. Have you ever worked with 3PC /UNCEF/FI, or independently, to prevent a child from entering an orphanage? 22. If yes, can you describe how you did this? 23. How is the child now? 24. Is violence against children (including when parents beat children) common in your community? 25. How does the community respond when a family beats a child? 26. Have you ever sought help for a child who was beaten? Why/why not? 27. Can you describe your working relationship with the 3PC/UNICEF/Family+ project staff? 28. Were there any best practices with 3PC/UNICEF/Family+ project on your community? 29. Were there any lessons learned with 3PC/UNICEF/Family+ project on your community? 30. Is there anything you would do differently next time? 31. What was the most important thing 3PC/UNICEF/Family+ project did in your community? 32. Can you describe the impact of the 3PC/UNICEF/Family+ project on your community? 33. Do you think you will continue to use skills you learned through 3PC /UNCEF/FI in the future? 34. Do you think you will continue to refer to other NGOS that you worked with through 3PC /UNCEF/FI in the future? 35. Is there anything else you would like to tell me about this project? Discussion Guide for Children Location: Facilitator: Ages: Gender of children: Before the discussion, the facilitator ensures that written consent has been obtained from the parents or guardians of the participating children, using the appropriate consent forms. Introduction (10 minutes) The facilitator welcomes the children, and makes them feel comfortable, building a connection with children through chatting a little before beginning. The facilitator then explains: 70 By this I mean, have you, for example, been referred a child from 3PC who you assisted as she settled back into the community? 69  I am a researcher from Emerging Market Consulting (EMC) working for USAID to help Friends International (FI). This means that I ask people about their lives and then write reports about what they said.  Today I am going to ask about your experiences when you lived in an orphanage71 and when you came home from the orphanage.  What you say in this group is confidential. This means that I will write down what you said, but I will not write down your names. After this discussion, EMC will write a report about what children said, but it will not write the names of any children in this report. I also will not tell anyone outside this room, including your parents and teachers, what individual children said. This means you can feel free to tell me the truth.  You do not have to answer any questions. You can also choose not to answer certain questions. If you don’t feel like talking you can choose to stop or be quiet. You can also choose to leave the discussion at any point. (The facilitator should arrange for a safe place for children who choose to leave to stay while they await their parents. The facilitator should describe this to the children.)  No one will receive any money or benefits from EMC or FI for taking part in this research. Discussion with Children Ask children where they live, and who they live with. The facilitator explains to children that today we are going to be talking about children’s life in the center, and about their life when they returned home. The facilitator explains that she is going to show the children some pictures, and then ask children which pictures look most like their own experiences. School The facilitator takes out the first two pictures of children at school. The facilitator shows the children the images of school, and asks them to look them to say: Which picture looks more like children going to school in the center? Why? Which picture looks more like your house, and whether children go to school that you have moved home? Why? Can you describe any differences between the pictures and your own life? The facilitator asks the children: Did you go to school whilst in the orphanage? 71 A general note on the term “orphanage”; the Khmer word consistently used in this context translates roughly to “centers for orphans". In the experience of the evaluators’ subject matter expert, it is generally well understood what these centers are and they are not confused with boarding schools or other institutions. 70 If yes, were you studying at grade level? (If not, ask children to describe why not in both of these cases.) Do you go to school nowadays? Are you studying at grade level? (If not, ask children to describe why not in both of these cases.) Food Next the facilitator shows the children the images of food, and asks them to look at them say: Which picture looks more like the food in the center? Why? Which picture looks more like the food in your own house? Why? Can you describe any differences between the pictures and your own life? The facilitator then asks the children: Did you get enough food to eat in the orphanage? (If not, ask children to explain why.) Do you get enough food to eat nowadays? (If not, ask children to explain why.) Medical Care Next the facilitator shows the children the images of illness and medical care, and asks them to look them to say: Which picture looks more like the illness and medicine in the center? Why? Which picture looks more like the illness and medicine in your own house? Why? Can you describe any differences between the pictures and your own life? The facilitator then asks the children: Did you often get ill at the orphanage? If yes, please describe. Did you get medical help at the orphanage? Can you describe this care? Do you often get ill nowadays? If yes, please describe. If yes, do you get medical help at these times? Can you describe this care? Care Next the facilitator shows the children the images of care, and asks them to look at them say: Which picture looks more like the care in the center? Why? Which picture looks more like the care in your own house? Why? Can you describe any differences between the pictures and your own life? The facilitator then asks the children: How did the orphanage care for you? Can you describe the difference between the way the orphanage cared for you and the way your family care for you? Were you safe at the orphanage? If not, sensitively prompt the children to talk in more detail about threats to safety (but do not force). 71 Do you feel safe now? Did anyone hurt you at the orphanage? If yes, sensitively ask the children to describe how, (but do not force). If the children do not respond, ask whether there was fighting between children at the orphanage, or whether adults hurt children to discipline them. Be extremely sensitive to children’s feelings during this part of the discussion. Does anyone hurt you now? If a child was hurt, either at home or at the orphanage, ask the child how adults responded. Did adults report the violence? If yes, who to? What happened next? Stable Shelter Next the facilitator shows the children the images of shelter , and asks them to look at them say: Which picture looks more like where you slept the center? Why? Which picture looks more like where you sleep now in your own house? Why? The facilitator then asks the children: Where did you sleep at the orphanage? Where do you sleep nowadays? Have you moved house since you came back to your family? If yes, why? How many times? Do you think your family will be able to live in this house for a long time? If not, please describe why. Do you think this is a stable place to live? Please describe why or why not. Work There is no picture for this section. The facilitator asks the children: Did you work while you lived at the orphanage? How did this work affect you? Do you work nowadays? How does this work affect you? Do members of your family earn enough money to support you? Did anyone in your family receive training or money from Friends International to help you work or earn money? If yes, can you describe the impact of receiving this money on your family? Can you describe the impact of receiving this money on you? When you think back, were you happy or sad most of the time at the orphanage? (Please describe why.) Are you happy or sad most of the time nowadays? (Please describe why.) If the children are over 14 years of age, ask whether they have any plans for when they leave school. The facilitator explains that they are going to ask questions about the relationship between Friends International (or the implementing NGO partner) and the family. 72 (Friends International or the implementing NGO will henceforth be referred to as FI/NGO for the ease of the facilitator.)  Can you describe why you left the orphanage and returned to live with your family?  Can you describe how you first met a member of FI/NGO staff?  Did a FI/NGO staff member work to help you while you lived at the orphanage? If yes, can you describe how.  Did a FI/NGO staff member help you when you moved home? If yes, can you describe how.  How often did the FI/NGO staff member visit you when you moved home? Would you have preferred more or less visits?  Can you describe the most important thing that the FI/NGO l staff member did for you?  Is there anything you wish FI/NGO had done differently? If yes, can you describe this?  How did returning to live with your family change your life?  Is there anything else you want to tell me about the orphanage or returning to live with your family? The facilitator asks if the participants have any further questions, and remind them that no names will be recorded in the final report. Thank the participants for talking with you. Discussion Guide for Parents of Children Location: Facilitator: Gender of Participants: Mother/Father of Reintegrated Child: Introduction (10 minutes) The facilitator greets participants by saying:  I am a researcher from Emerging Markets Consulting (EMC) working for USAID to help Friends International (FI). This means that I ask people about their lives and then write reports about what they said.  Today I am going to ask about your experiences when your child lived in an orphanage and when your child came home from the orphanage. 73  What you say in this group is confidential. This means that I will write down what you said, but I will not write down your names. After this discussion, EMC will write a report about what you said, but it will not write the names of parents or children in this report. I also will not tell anyone outside this room, what individual parents said. This means you can feel free to tell me the truth.  No one will receive any money or benefits from EMC or FI for taking part in this research.  Participants do not have to answer any questions. They can also choose not to answer certain questions. The facilitator hands out the Consent Form and then reads it aloud to participants. Participants are asked if they have any questions. Participants are then asked to confirm that they have understood what the consent form says by checking the yes/no at the end of the form. For the ease of the facilitator Friends International or the implementing partner will henceforth in this interview be described as FI/NGO. Discussion During the following discussion, the facilitator must be sensitive if a participant is describing traumatic experiences, and should not force the participant to discuss traumatic experiences if she or he is reluctant to do so. The facilitator asks the following questions and prompts for more detail when brief answers are given. Questions:  Why did your child go to live in an orphanage? (Prompt parent to briefly speak about push factors.)  Can you describe the orphanage?  Can you describe the advantages and disadvantages of your child’s living in that orphanage? The facilitator explains that they are going to ask some questions, that ask the interviewee to compare the life of the child while in the orphanage with the life of the child today, living at home.  Did your child go to school whilst in the orphanage? Was your child studying at grade level? (If not, ask parent to explain why not in both of these cases.) 74  Does your child go to school nowadays? Is your child studying at grade level? (If not, ask parent to explain why not in both of these cases.)  Did your child get enough food to eat in the orphanage? (If not, ask parent to explain why.)  Does your child get enough food to eat nowadays? (If not, ask parent to explain why.)  Did your child get ill often at the orphanage? Please describe. If yes, did your child get medical help at the orphanage?  Does your child get ill often nowadays? If yes, does your child get medical help at these times?  How did the orphanage care for your child? Can you describe the difference between the way the orphanage cared for your child and the way your family cares for your child?  Was your child safe at the orphanage? (If not, sensitively prompt the parent to talk in more detail about threats to the child’s safety.) Is your child safe now?  Did anyone hurt your child at the orphanage? (If yes, sensitively ask the parent to describe how.) Does anyone hurt your child now?  Where did your child sleep at the orphanage? Where does your child sleep nowadays? Do you think your family will be able to live in this house for at least the next two years? If not, please describe why. Do you think this is a stable place for your child to live? Please describe why.  Did your child work while s/he lived at the orphanage? How did this work impact your child’s life?  Does your child work to earn money nowadays? How does this work affect your child’s life?  Do members of your family earn enough money to support your child? Have you received training or money from Friends International to help you work or earn money? If yes, can you describe the impact of receiving this money on the way your family lives?  Was your child happy or sad most of the time at the orphanage? (Please describe why.) Is your child happy or sad most of the time nowadays? (Please describe why.) 75  If the child is over 14 years of age, ask whether the family have any plans for when their child leaves school. What kind of work do you think the child will do when he or she leaves school? The facilitator explains that they are going to ask questions about the relationship between Friends International and the family.  Can you describe why your child left the orphanage and returned to live with you?  Can you describe how you first met a member of FI/NGO staff?  Did a FI/NGO staff member work with your child while s/he lived at the orphanage? If yes, can you describe how this staff member helped your child or your family at that time?  Did a FI/NGO staff member work with your child when s/he moved home? If yes, can you describe how this staff member helped your child or your family at that time?  Can you describe the most important thing that the FI/NGO staff member did for your family or your child?  Is there anything you wish FI/NGO had done differently? If yes, can you describe this?  Did help provided by FI/NGO affect the life of your child? If yes, please describe how. The facilitator asks if the participants have any further questions, and remind them that no names will be recorded in the final report. Thank the participants for talking with you. Discussion Guide for Parents of Children Reintegrated with MoSVY/Sky Pilot Location: Facilitator: Gender of Participants: Mother/Father of Reintegrated Child: Introduction (10 minutes) The facilitator greets participants by saying:  I am a researcher from Emerging Markets Consulting (EMC) working for USAID to help Friends International (FI). This means that I ask people about their lives and then write reports about what they said. 76  Today I am going to ask about your experiences when your child lived in an orphanage and when your child came home from the orphanage.  What you say in this group is confidential. This means that I will write down what you said, but I will not write down your names. After this discussion, EMC will write a report about what you said, but it will not write the names of parents or children in this report. I will also not tell anyone outside this room, what individual parents said. This means that you can feel free to tell me the truth.  No one will receive any money or benefits from EMC or FI for taking part in this research.  Participants do not have to answer any questions. They can also choose not to answer certain questions. The facilitator hands out the Consent Form and then reads it aloud to participants. Participants are asked if they have any questions. Participants are then asked to confirm that they have understood what the consent form says by checking the yes/no at the end of the form. Discussion During the following discussion, the facilitator must be sensitive if the participant is describing traumatic experiences, and should not force a participant to discuss experiences if she or he is reluctant to do so. The facilitator asks the following questions and prompts for more detail when brief answers are given. Questions:  Why did your child go to live in an orphanage? (Prompt parent to briefly speak in some detail about push factors).  Can you describe the orphanage?  Can you describe the advantages and disadvantages of your child’s living in that orphanage? The facilitator explains that they are going to ask questions about the reintegration process that occurred when children left the orphanage and returned home.  Can you describe why your child left the orphanage and returned to live with you?  Who was the first person to suggest that your child should return to your family? 77  Can you describe the reintegration process?  Do you think the staff at the orphanage supported the reintegration process? Why/why not?  Did you feel happy to have your child returned? Did you have concerns about having your child returned? Were you given a choice whether to have your child returned home? Was your child given a choice?  Can you describe the benefits to your child that were a result of reintegration?  Can you describe any challenges faced by your child after reintegration?  Where is your child living now? Who is your child living with? What is your child doing now (i.e. attending school/working)?  Can you remember how many days or weeks passed between the time when someone approached you about reintegrating your child, and the time your child came home?  Did anyone from the government or an NGO visit you at home after your child was reintegrated? If yes, who. How many times in total? How many times a month? Do you think that you were visited enough times?  Did anyone offer you support such as income generation training, or finding work during the reintegration process?  Was it easy for your child to enter school after s/he returned home? Why/why not?  What was the biggest challenge your family faced during the reintegration process? Is there anything you think should have been done differently?  Is there anything else you want to tell me about the orphanage or reintegration process? The facilitator explains that they are going to ask some questions that ask the interviewee to compare the life of the child while living at an orphanage and the life of a child at home.  Did your child get ill often at the orphanage? Please describe. If yes, did your child get medical help at the orphanage?  Does your child get ill often nowadays? If yes, does your child get medical help at these times? 78  Did your child go to school whilst in the orphanage? Was your child studying at grade level? (If not, ask parent to explain why not in both of these cases.)  Does your child go to school nowadays? Is your child studying at grade level? (If not, ask parent to explain why not in both of these cases.)  Did your child get enough food to eat in the orphanage? (If not, ask parent to explain why.)  Does your child get enough food to eat nowadays? (If not, ask parent to explain why.)  How did the orphanage care for your child? Can you describe the difference between the way the orphanage cared for your child and the way your family cares for your child?  Was your child safe at the orphanage? (If not, sensitively prompt the parent to talk in more detail about threats to the child’s safety.) Is your child safe now?  Did anyone hurt your child at the orphanage? (If yes, sensitively ask the parent to describe how.) Does anyone hurt your child now?  Where did your child sleep at the orphanage? Where does your child sleep nowadays? Do you think your family will be able to live in this house for at least the next two years? If not, please describe why. Do you think this is a stable place for your child to live? Please describe why.  Do members of your family earn enough money to support your child?  Was your child happy or sad most of the time at the orphanage? (Please describe why.) Is your child happy or sad most of the time nowadays? (Please describe why.)  What do you think the main difference is between living in an orphanage and living at home? Which is better for a child? The facilitator asks if the participants have any further questions, and remind them that no names will be recorded in the final report. Thank the participants for talking with you. 79 ANNEX VI: EVALUATION SCHEDULE Phase 1 Phase 2 Deliverable # Week 25-May-2015 1-Jun-2015 8-Jun-2015 15-Jun-2015 22-Jun-2015 29-Jun-2015 6-Jul-2015 13-Jul-2015 20-Jul-2015 27-Jul-2015 3-Aug-2015 Phase 1 Project Inception and Research Design Receive key documents for review from USAID and AC 1 Conference call with USAID/W; Meeting with USAID Cambodia mission X Meeting with evaluation AC X Hypothesis and evaluation framework development 1 Qualitative analysis of key documents 1 1 1 Develop questionnaires for key stakeholder interviews 1 1 Develop and localize data collection protocols for parents and children 1 1 In-Country Briefing with USAID/Cambodia and AC X 2 Submit minutes from in-country briefing X Quantitative analysis of reports and available logframe data 1 1 1 Develop detailed evaluation plan 1 1 1 1 1 Submit evaluation plan to USAID/Cambodia USAID DCOF and World Learning, as well as the AC x Meeting with USAID/Cambodia and the AC to present and discuss evaluation plan x Submit minutes from evaluation plan meeting x 1 Incorporate questions and comments on evaluation plan 1 X Deliverable # Week commencing 3-Aug-2015 10-Aug-2015 17-Aug-2015 24-Aug-2015 31-Aug-2015 7-Sep-2015 14-Sep-2015 21-Sep-2015 28-Sep-2015 5-Oct-2015 12-Oct-2015 19-Oct-2015 26-Oct-2015 2-Nov-2015 9-Nov-2015 16-Nov-2015 23-Nov-2015 30-Nov-2015 Phase 2 Field Research Pilot data collection protocols 1 Stakeholder meetings in Phnom Penh, Siem Reap, Battambang, Sihanoukville 1 1 1 1 \ Research with parents and children in Phnom Penh, Siem Reap, Battambang, Sihanoukville 1 1 1 1 Data entry 1 1 1 1 Analysis of interview findings 1 1 3 Develop and submit overview briefing of key findings 1 1 X Meeting with AC on key findings X Meeting with USAID/Cambodia on key findings X Conference call with USAID/DCOF X Submit minutes from initial findings brief X Phase 3 In-depth Analysis In-depth data analysis 1 1 Develop final evaluation report; circulate draft (by Oct. 23 ) 1 1 X 4 Present evaluation report to advisory committee X Submit minutes from draft evaluation report meeting X Briefing: USAID/Cambodia X Briefing: USAID/DCOF X * Additional stakeholder briefing (with NGOs and government) 1 Incorporate comments and questions into evaluation report 1 1 1 5 Submit final evaluation report (by Nov. 13) X * Time alotted for additional briefings and FCF workshop (up till Nov. 30) 1 1 Note: Grey shading denotes Pchum Ben and Water Festival holiday weeks 80 ANNEX VII: LIST OF COI 81 82 83 84 85 86 I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date July 27, 2015 Name Olga Creamer Title Research Consultant Organization EMC EvaluationPosition? Team Leader Team member Evaluation AwardNumber(contract or other instrument) Cooperative Agreement, No. DFD/A/00‐08‐00260‐00 Associate Award under LWA no. GPO‐A‐00‐04‐00021‐00 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) “Mid‐term evaluation of two projects being implemented in Cambodia by UNICEF and Friends International” I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or issignificant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct orsignificant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideastoward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could biasthe evaluation. 87 1 Cooperative Agreement, No. DFD/A/00-08-00260-00 Associate Award under LWA no. GPO-A-00-04-00021-00 “Mid-term evaluation of two projects being implemented in Cambodia by UNICEF and Friends International” U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523