FINAL REPORT FOR END OF PROJECT EXTERNAL EVALUATION FOR TRANSITIONAL LIVING PROGRAMME FOR CHILDREN IN STATE CARE PROJECT PROJECT CODE: AID-532-A-14-00001 Submitted by Dr. J. M. Moncrieffe MARCH 18, 2021 2 of 87 TABLE OF CONTENTS 1. BACKGROUND 9 2. METHODOLOGY 12 3. ASSESSMENT OF PROJECT RELEVANCE 15 4. ASSESSMENT OF PROJECT EFFECTIVENESS 16 5. CONSTRUCTION OF TLP COMPLEXES 49 6. EFFECTIVENESS OF PROJECT MANAGEMENT, RELATIONSHIPS AND MANDATES 51 ANNEX 1: TERMS OF REFERENCE 56 ANNEX 2 EVALUATION AREAS AND SUPPLEMENTARY LISTING OF QUESTIONS 63 ANNEX 3 SPECIFIC FOCUS GROUP AND INTERVIEW QUESTIONS - CPFSA TRAINERS AND CAREGIVERS 65 ANNEX 5 QUESTIONNARE FOR PROJECT IMPLEMENTATION TEAM 75 ANNEX 6 DOCUMENTS REVIEWED 78 ANNEX 7 LIST OF RESPONDENTS 80 1. 3 of 87 ACRONYMS Caribbean Child Development Centre CCDC Child Protection and Family Service Agency CPFSA Community Training Institution CTI Employment and Career Services ECS Government of Jamaica GoJ Human Employment and Resource Training /National Training Agency HEART/NTA Memorandum of Understanding MoU National Unattached Youth Programme NUYP Transitional Living Programme TLP University Project Management Office UMPO United States Agency for International Development USAID 4 of 87 TABLES Table 4.2.1 Effectiveness of the Training Received in Specified Areas 19 Table 4.2.2 Extent to Which Caregivers Have Been Equipped to Provide Training 21 Table 4.4.1: Progress of Skills Training Component Year 1 – Year 6 by Gender 29 Table 4.5.1 Focus Group Participants and Current Activities 40 Table 4.5.2 Growth Aspects 43 Table 4.5.3. WhatsApp Participants’ Current Activities 46 5 of 87 EXECUTIVE SUMMARY The USAID- supported Transitional Living Programme (TLP) for Children in State Care project, AID-532-A-14-00001 aims to achieve two key overarching goals: a) “improve the transition to independent living for children leaving State care at 18 years” and b) “reduce the risk factors associated with low skills development, inadequate life skills and poor self-image”. OVERARCHING AND SPECIFIC OBJECTIVES Beneath these overarching goals are the following specific objectives: 1. “To assess the level of preparedness of the children aged 14-16 who reside in State care facilities to exit the facility”; 2. “To develop an independent living protocol to prepare the children to exit State care”; 3. “To develop a Standard Operating Procedures guide for preparing children aged 14 to 18 for independent living”; 4. “To equip, through training, 900 children in State care facilities with the necessary skills and knowledge to reduce the effects of risk factors that affect safety and human rights”; 5. “To build the capacity of 40 trainers and 500 care staff with the techniques in life skills development training”; 6. “To establish safe places for at least 52 wards of the State, ages 18-21, for a maximum of two years after leaving State care”; 7. “To increase job/self-employment opportunities for the 52 housed youth”; 8. “To determine through a structured evaluation, the outcomes of the 52 young adults exposed to independent living facilities”; and 9. “To strengthen the leadership and management capability of the CPFSA to successfully implement and sustain project activities through the provision of an improved Child Case Management System and three vehicles to assist with the transportation needs of children in State care”. The attending project activities comprised: 1. Life Skills ‘Training of Trainer’ program for the staff of the CPFSA. The trainers then had responsibility for training 500 care staff and 900 children who were in care; 2. Skills training conducted by HEART NTA and/or other appropriate training institutions for 500 children in State care facilities. 3. Mental health Training; 4. Standard Operating Procedures (SOP); 5. Starter kits and support with buses, laptops etc. 6. Mentorship 7. The provision of 45 scholarships for youth transitioning out of care. The University of the West Indies Open Campus provided the scholarships. 8. Fifty-two youth (males and females) benefiting from the Housing Assistance Program 9. Development of a network of potential employers for the children leaving State care and who have employable skills. 10. Implementation of the second phase of the CPFSA’s Child Case Management System 6 of 87 11. Monitoring and evaluation. METHODOLOGY FOR THE EVALUATION The Institute for Applied Social Research (IASR) was contracted to conduct a process evaluation of the TLP, within a limited time frame (two months). In order to accomplish this, the IASR used diverse research tools, with the expectation and explicit request that all stakeholders make their full contributions, utilizing the mechanisms provided them. The research tools comprised: 1. documentary reviews of secondary data (including the Needs Assessment, annual and monthly reports, project proposal and inception documents, the monitoring and evaluation framework), 2. key stakeholder interviews, with senior officials/decision makers within the key implementing organizations (CCDC, the CPFSA, Vocational Skills Providers, Mentorship) and the Chair of the TLP Project Steering Committee; 3. open-ended questionnaires with all members of the TLP Project Implementation team (PIT); 4. surveys (with CPFSA trainers and caregivers); and 5. focus group discussions with the Project Implementation Team, a cohort of youth who occupy transitional housing and managers of the care facilities; and 6. electronic group discussions with children/youth who participated in the Life Skills, Entrepreneurship and Vocational Skills components. OBSERVATIONS AND CONCLUSIONS This evaluation concludes that the TLP Project succeeded in most components, although there were organization and management constraints that undermined some key aspects of project implementation. With respect to the Life Skills component, it is clear that the CPFSA’s capacity has been built; however, it is difficult to ascertain the actual reach and impact of the training since the monitoring and accountability mechanisms are inadequate. Further, the assumption that training would “trickle” from CPFSA trainers to caregivers does not necessarily obtain. The report has identified challenges with implementing the vocational skills and mentorship components; however, these have, in many respects, been used to devise circumventing tactics and strategies. While the TLP project managed to construct two of three anticipated housing facilities, that component was plagued with delays and, otherwise, negatively affected by the inability to procure a suitable site within the required time frame: 7 of 87 RECOMMENDATIONS Against the background of these findings, the evaluation team concurs with Project Management that the following actions are required to ensure that the TLP produces substantive outcomes for the children/youth who are exiting State care. Project Management 1. There is need for a dedicated team within CPFSA whose explicit mandate is to manage independent living planning for children and youth who are exiting care. 2. It is critical to enhance monitoring and accountability mechanisms and processes in order to measure the quality of training received, how training is being implemented and the real consequences for children and youth. 3. It is important to build skills in strategic project management, including the definition and monitoring of qualitative/outcomes targets. 4. The production of instructional guides is important for ensuring institutional knowledge. It may also be helpful to produce electronic copies of training materials and refresher courses, since these would remain even though staff may leave the institutions. Training for Sustainability 1. Provide ongoing training to strengthen and improve training undertaken as a part of the project. 2. Ensure that all staff within CPFSA are trained in Trauma Informed Care. This training should equip staff to deal with their own traumatic experiences as well as treat children and youth with greater compassion and respect. Training such as the Life Skills training of care staff requires ongoing reinforcement and updating if gains are to be sustained. The case management system needs to be embedded in the work of the agency, and sufficient training for the persons who will input the records. 3. Dealing with discrimination: This involves “continuing sensitization on the issue of stigmatization of the youth because of their status as wards or former wards of the state”. 4. Increased caregiver awareness of the stages of growth and the skills required for managing relationships with children and youth across age groups. Ensure Equitable Access to TLP 1. It is important to ensure that all children have access to the TLP: This means appending a transitioning/aftercare plan to the care plan of each youth in care. 2. It is critical to pay careful attention to the youth who are least likely to transition successfully; that is, those less prone to participate in programs; for these, those not involved in higher education or in vocational skills. This cohort of youth, likely the most at risk, must also transition. Without innovative and effective approaches and strategies, this cohort can be relegated to unemployment and drawn into 8 of 87 circles of crime and violence. Success with this category of youth will provide a good indicator of the effectiveness and impact of the TLP. Increase Resources: 1. The Government needs to ensure that the resources are available to CPFSA for them to be able to implement and improve the TLP. Further, resources should be allocated to the implementation of the SOP and the construction of additional facilities, including for males and children with disabilities. Finalize the Standard Operating Procedures 1. The Standard Operating Procedures needs to be in place and a policy document formulated to embed the operations. The SOP has been drafted and will be finalized with recommendations from this evaluation. 2. The key components in the SOP includes an educational plan for youth who want to pursue technical or tertiary education, life skills coaching as well as opportunities for internships and entrepreneurial training. 9 of 87 1. BACKGROUND Vision 2030, Jamaica’s National Development Plan (NDP), provides the backdrop for the USAID- supported Transitional Living Programme (TLP) for Children in State Care project, AID-532-A-14-00001. Specifically, Goals 1 and 2 of the NDP envisions the Jamaican Society as secure, cohesive and just, thereby empowering individuals to meet their full potential. The Government of Jamaica (GoJ) recognizes that in order to meet their full potential, children and youth need to be properly prepared to exit State care. Accordingly, the UWI Open Campus and the Child Protection and Family Services Agency (CPFSA) designed the TLP project in order to meet these critical needs. Prior to implementation of the TLP project, a ‘needs assessment’ was conducted in order to evaluate the status and experiences of wards of the state as well as their readiness to exit state care. In conducting the needs assessment, information was sought across the CPFSA divisions of South East (Kingston & St. Andrew, St. Catherine, St. Thomas); Southern (Manchester, Clarendon, St. Elizabeth); North East (St. Ann, St. Mary, Portland) and the Western Region (St. James, Westmoreland, Hanover, Trelawny) as well as from current and former beneficiaries of the state-care system, members of the management team, care-providers, and private sector stakeholders. The study examined the areas of: (a) life in care (b) perceptions and experiences of leaving care (c) youth preparedness to transition from care (d) challenges faced by youth transitioning from care, and (e) care planning. Having canvassed stakeholders, care providers and state-care beneficiaries, the process yielded several concerns, which resulted in recommendations for introduction of, or improvement in, the following areas: (i) care plans (ii) life-skills coaching (iii) education (iv) life-skills training (v) employment (vi) housing (vii) staff support (viii) proactive and robust care system, and (ix) children with special needs These recommendations were further translated into a streamlined programme to prepare those exiting state-care to transition with key skill-sets into independent living. The TLP’s overarching aims were as follows: a) “improve the transition to independent living for children leaving State care at 18 years” and 10 of 87 b) “reduce the risk factors associated with low skills development, inadequate life skills and poor self-image”. Accordingly, the Transitional Living Programme for Children in State Care Logic Model depicts the Project Inputs, Outputs and Outcomes. PROJECT OBJECTIVES AND ACTIVITIES The USAID/UWI Open Campus agreement has been modified on a number of occasions. Modification 7, which contains the final amendments to the agreement lists the following specific objectives: 1. “To assess the level of preparedness of the children aged 14-16 who reside in State care facilities to exit the facility”; 2. “To develop an independent living protocol to prepare the children to exit State care”; 3. “To develop a Standard Operating Procedures guide for preparing children aged 14 to 18 for independent living”; 4. “To equip, through training, 900 children in State care facilities with the necessary skills and knowledge to reduce the effects of risk factors that affect safety and human rights”; 11 of 87 5. “To build the capacity of 40 trainers and 500 care staff with the techniques in life skills development training”; 6. “To establish safe places for at least 52 wards of the State, ages 18-21, for a maximum of two years after leaving State care”; 7. “To increase job/self-employment opportunities for the 52 housed youth”; 8. “To determine through a structured evaluation, the outcomes of the 52 young adults exposed to independent living facilities”; and 9. “To strengthen the leadership and management capability of the CPFSA to successfully implement and sustain project activities through the provision of an improved Child Case Management System and three vehicles to assist with the transportation needs of children in State care”. Correspondingly, the project activities comprised: 1. Life Skills ‘Training of Trainer’ program for the staff of the CPFSA, initially led by the Social Welfare Training Centre (SWTC) personnel and then assumed by CCDC. The trainers then had responsibility for training 500 care staff and 900 children who were in care; 2. Skills training conducted by HEART NTA and/or other appropriate training institutions for 500 children in State care facilities. Children were then assisted to find employment. Approximately 50 youth, who after vocational skills training opted to pursue entrepreneurial ventures, were provided with start-up funding or other resources; 3. Mental health Training; 4. Standard Operating Procedures (SOP); 5. Starter kits and support with buses, laptops etc. 6. Mentorship 7. The provision of 45 scholarships for youth transitioning out of care. (The University of the West Indies Open Campus provided the scholarships.) 8. Fifty-two youth (males and females) who successfully achieved the program goals, were either working, in school and had no safe space to go back to on ageing out of care, were assisted with transitioning into adulthood and, upon exiting State care, provided with a shared apartment facility. Youth were supported by a CPFSA Officer based at each transitional housing facility in order to assist them with sustaining a safe and responsible lifestyle, on their own, by age 21. All youth were housed in a shared one-bedroom apartment (two adults/room) in a safe setting that is convenient to bus routes, community resources, employment opportunities and shopping. The facilities were fully furnished and equipped to allow the youth to fully concentrate on completing their education, succeed in vocational training, obtain employment, and develop a meaningful career path. 9. Development of a network of potential employers for the children leaving State care and who have employable skills. 12 of 87 10. Implementation of the second phase of the CPFSA’s Child Case Management System work-plan to enhance the operations of the Agency, improve information flow, and strengthen decision-making for the benefit of children who interface with the child protection system. Phase one was completed on or about March 2016. The execution of Phase two would take place at the CPFSA offices and facilities island-wide. 11. At the end of the implementation of the Child Case Management System, the system would be able to interface with the Family/Children Courts, the National Children’s Registry, the Centre for the Investigation of Sexual Offences and Child Abuse, and the Ministry of Education. 12. Monitoring and evaluation. 2. METHODOLOGY Consistent with USAID evaluation guidelines, the Terms of Reference (ToR) stipulates that the assessment should focus on the following: i. Project Relevance – the extent to which the goals and objectives were consistent with beneficiaries’ needs and priorities; ii. Project Effectiveness – the extent to which the targeted project objectives were achieved (or are expected to be achieved); iii.Project Efficiency – how efficiently the processes were carried out to achieve the results (time, other resources); iv. Project Sustainability – the extent to which the project activities are likely to continue after the project; and v. Project Outcomes – inferences on the likely long-term effects produced by the project (directly, indirectly, intended and unintended). Given the stage of implementation (some activities are still incomplete), the UWI/Open Campus requested a process evaluation rather than one that evaluates impact and the attending outcomes. Therefore, the scope of the evaluation reflects this. Specifically, the evaluation studies project relevance, effectiveness, efficiency and sustainability. The Methodology was structured to achieve evaluation objectives within a very limited time frame (two months). Given the challenge of completing a comprehensive evaluation of such a broad project within an extraordinarily tight time period, the IASR considered it prudent to use diverse research tools, with the expectation and explicit request that all stakeholders make their full contributions, utilizing the mechanisms provided them. The research tools comprised: 1. documentary reviews of secondary data (including the Needs Assessment, annual and monthly reports, project proposal and inception documents, the monitoring and evaluation framework), 13 of 87 2. key stakeholder interviews, with senior officials/decision makers within the key implementing organizations (CCDC, the CPFSA, Vocational Skills Providers, Mentorship) and the Chair of the TLP Project Steering Committee; 3. open-ended questionnaires with all members of the TLP Project Implementation team (PIT); 4. surveys (with CPFSA trainers and child carers); and 5. focus group discussions with the Project Implementation Team, a cohort of youth who occupy transitional housing and managers of the care facilities; and 6. electronic group discussions with children/youth who participated in the Life Skills, Entrepreneurship and Vocational Skills components. 2B. RESEARCH LIMITATIONS A. Locating Respondents One of the challenges to data collection was in locating children/youth who had benefited from the TLP project and were now out of care. The IASR team requested the children/youth’s contact information; however, the TLP Project team explained that some of these children/youth had changed their telephone numbers and could not be located. They proposed working with the subset of youth, whom they would have better prospects of locating. IASR also contacted the CPFSA for assistance with locating children and youth. While it was possible to speak with youth in care, CPFSA noted that except for youth in transitional homes, those who are resident in the care facilities would not be able to comment on the TLP (see reasons in Point B below). The CPFSA also acknowledged that they do not have contact with the majority of youth who were now out of care. The evaluation team recognized the need for a system that would keep better track of this cohort. There was an added complication: Given the protocols stipulated since the Covid-19 pandemic, it was considered prudent to conduct the meetings online. This further restricted the sample to respondents who had access to internet services. After much deliberation and full consideration of the potential advantages and limitations, IASR decided to consult with youth in the transitional homes and, with CPFSA support, youth from the vocational and entrepreneurship groups who were part of online discussion groups. B. Difficulties with Assessing TLP The TLP built on and streamlined initiatives, which the CPFSA had been implementing beforehand. Therefore, it was difficult for children/youth, CPFSA trainers and service providers to actually attribute the gains made to TLP, as opposed to the ongoing interventions. Many children had not heard of TLP. The only cohort of children/youth who was in a position to make conclusive statements on TLP was those who currently live in the transitional facilities. Therefore, as noted above, our team concentrated on conducting incisive focus group discussions with the youth who live in the housing 14 of 87 assistance facilities. Interviews were also conducted with youth who participated in the Entrepreneurship and Vocational Skills components. Given the limitations with locating respondents, IASR was not able to conduct the comparative analysis of youth who had benefited from employment opportunities and scholarships and those who had not. C. Levels of Responsiveness There was a low response rate to the questionnaires, despite repeated written and telephone requests for participation. Recognizing that caregivers had limited time to devote to completing the questionnaires (some also appeared daunted by the prospect of responding to the online survey), the IASR team opted to conduct telephone interviews and online focus groups with each care facility. This strategy proved fruitful. The team offered similar services to the CPFSA trainers; however, all the persons contacted preferred to complete the questionnaires themselves rather than being assisted with this. Despite their commitment, a low proportion (approximately 20%) of CPFSA trainers responded. Of the TLP project team, 9 officers responded to the invitation to complete an unstructured survey; eight of these were from CCDC and one from the CPFSA. These invitations were sent on different occasions; however, there was no increase in the number of respondents from the CPFSA. The IASR also facilitated other opportunities for input from the project management team. Two focus groups were held with the TLP project oversight team, two key stakeholder interviews with CPFSA staff and one with CCDC; one interview was held with the Chair of the TLP Steering Committee. Some key CPFSA officials did not respond to repeated requests for feedback to open ended questionnaires or to invitations to focus group discussions; therefore, views from the CPFSA are underrepresented in the evaluation. No response was received to requests for interviews with USAID representatives. Both IASR and CCDC submitted these. Further, attempts to organise group discussions with mentors proved unfruitful, although the IASR was able to meet with MultiCare Youth Foundation, which had responsibility for the administration of the mentorship component of the TLP. Finally, despite requests, the IASR was not provided with a draft copy of the Standard Operating Procedures for review. 15 of 87 3. ASSESSMENT OF PROJECT RELEVANCE This section of the evaluation addresses the relevance of the project. It focuses on two key questions: How pertinent is the project to national development priorities? How pertinent is the project to realizing international obligations? The Transitional Living Programme for Children in State Care project, AID-532-A-14- 00001 is pertinent to Vision 2030, Jamaica’s National Development Plan, which commits to building a secure, cohesive and just society, in which individuals are empowered to meet their full potential. The TLP also supports Jamaica’s bid to meet its international obligations to protect children’s rights to life and development, as specified in a range of human rights instruments to which the country is signatory: the Convention on the Rights of the Child (CRC); the Convention on the Eradication of all Forms of Discrimination against Women; ILO Optional Protocols on Child Labour (Conventions 138 and 182); the Beijing Rules (UN Standard Minimum Rules for the Administration of Justice); the Riyadh Guidelines (UN Guidelines for the Prevention of Juvenile Delinquency); UN Rules for the Protection of Juveniles Deprived of their Liberty); and the Declaration on Survival, Protection and Development of Children. In 2017, Jamaica also became a party to the Hague Convention on the Civil Aspects of International Child Abduction. Note, further, that the Government of Jamaica’s Child Care and Protection Act (2004) underpins the TLP. To what extent were the project goals and objectives consistent with beneficiaries’ needs and priorities? The Transitional Living Programme for Children in State Care project was designed to respond to identified youth needs and priorities. The needs analysis showed that children/youth (a) were fearful and apprehensive about leaving state-care (b) considered themselves inadequately prepared for independent living despite programmes to promote these skills being offered (c) were dependent and not ready psychologically to exit state-care; (d) required mental health support (h) had poor or no prospects of returning to a family. Correspondingly, the TLP’s Project Staff notes that in order to prepare youth for independent living, the project aimed to equip them with life skills coaching, skills training, mentorship and the provision of safe spaces in the form of dedicated housing for youth transitioning from state care to independent living. The TLP was also geared to meeting CPFSA’s needs and priorities. Thus, one key objective was to “streamline existing programmes” and upgrade the transitioning out of care protection system. 16 of 87 4. ASSESSMENT OF PROJECT EFFECTIVENESS What institutional changes/outputs/achievements have taken place? How have these changes/outputs/achievements been brought about? What stated objectives were achieved? What have been the unintended and unexpected outputs of the project activities? LIFE SKILLS 4.1 Background In order to assess the quality of the changes/outputs and achievements, the evaluation team conducted detailed reviews of the annual and quarterly reports as well as surveys, focus groups and interviews, where pertinent. In the first project year, the project management team focused on “the review of literature; development of measures for data collection; receiving ethical approval for the needs assessment phase; consultations with the CPFSA stakeholders; visits to facilities to inform the development of the interventions and the Independent Living Protocol and Standard Operating Procedures documents; development of the life-skills curriculum and manual; initiation of the skills training enrolments; identification of lands for the housing complex; drafting and review of the MOU and the Training of Trainer workshops, which started on September 14, 2015” (Annual Report 2014-2015). The first amendment to the cooperative agreement was made during this period, as additional funds were required to cover the costs of capacity building (this increased of the project budget by US$67,868.00); a portion of the funds was used for travel to the Unites States to visit transitional living programmes in order to learn the ways in which their operations could be used to develop the local initiatives. Project activities were, largely, consistent with the workplan, although the pace of implementation was slower than envisioned. Specifically, the project was on target with the literature review, development of measurements for the needs assessment and the Exit Readiness analysis, and for application for ethical approval. However, as the ethical approval was granted later than anticipated, the Exit Readiness assessment was pushed back to Year 2. Though the consultation sessions with stakeholders within CPFSA (including selected children’s homes) provided some of the material required for developing the Life Skills Curriculum, consultations with the private sector and the youth were not completed in the first year. This means that curriculum development, the design of training materials and the first Trainer of Trainers in Life Skills was started in September 2015, without the desired input from the youth. The Annual Report (2014-2015) notes that 16 CPFSA officers comprised the first September 2015 cohort and 24 officers comprised the second cohort. The feedback 17 of 87 garnered was positive, although there was a view that the child carers should have been included. The drafting of the Independent Living Protocol and the Memorandum of Understanding (MoU) between the University of the West Indies and the CPFSA begun during this period. The purpose of the MoU was to formalize the cooperation amongst the three entities, commitment to the project and sustainability of action In the second year of the project, the TLP focused on finalising consultations with the children, youth and private sector in order to complete the needs assessment phase. The pilot phase of the training of caregivers component was initiated during this period. In Year 3, following feedback and review of the initial training, the training material was revised, finalized and printed. These comprised 500 copies of the Life Skills for Children in State Care: Caregivers’ Coaching Manual, 150 copies of the Life Skills Development Programme for Youth Transitioning State Care: Facilitators’ Guide and 400 copies of the Independent Living Guide for Youth Transitioning from State Care. The rollout commenced but was then halted while the revisions to the material was being done. After the manuals were printed, a refresher training with trainers was convened in September 2017. Life Skills Training sessions were later (November 20-21, 2017) held across the regions. Participants included Social Workers, Monitoring Officers, and Foster Care Officers. These officers were then expected to train residential care staff and foster parents. Each region had two days of training. The Annual Report (2017-2018) noted that the feedback was positive. In Year 4 of the TLP, life skills training with carers was rolled out and life skills coaching with youth began. The project team’s effort at ensuring accountability is noteworthy: “To ensure quality as well as to address any issues that the trainers may face in training the caregivers, the CCDC team observed multiple training sessions in each region, particularly in the initial phases of the training roll-out. The project team members from the CCDC team conducted 5 observations and the life skills consultant conducted 2 observations. A CCDC member was also present during the youth coaching session.” (Annual Report 2017-2018) The report highlights the challenges encountered; specifically, the CPFSA team had difficulty with scheduling and adhering to training dates due to the dynamic nature of their work.” It is worthwhile to highlight salient issues with monitoring and evaluation: “The CCDC team experienced issues in tracking the staff who attended and completed the training sessions. The monitoring and evaluation instruments initially designed were not used as intended. Issues included many forms not being filled out completely, or had not been completed at all. Some respondents perhaps misinterpreted some questions and provided irrelevant information. The CCDC team therefore created a 18 of 87 new, simpler registration form for use at each training session to capture the information necessary to meet reporting requirements, and requested that trainers utilise this new form. However, the team continued to encounter difficulties as some trainers did use this form, while others continued to use their own registers, which has made it difficult to standardise the information received. The CCDC and CPFSA teams continue to coordinate to streamline the process.”(Annual Report, p. 13) In Year 5, trainers completed 11 sessions with caregivers across all four regions. One hundred and six 106 caregivers received training; 11 completed the training, meaning that they attended 80% of the training modules. The report notes that the challenges with accountability continued: “Receiving registers of workshop participants from CPFSA in a timely fashion has been an ongoing challenge.” Further, “the CPFSA team continued to experience challenges adhering to a training plan due to the dynamic nature of their work; staff, caregivers, and youth are often impacted by emergencies that arise” The Covid-19 pandemic caused delays in Life Skills Training; however, the Annual report notes that 235 training sessions were completed; 10 young ladies received Life Skills Training (bringing the total to 720 youth) and the CPFSA team assigned an officer to assist with meeting the training targets for caregivers and youth. This synopsis outlines the progress of the Life Skills Component; it includes the challenges encountered, particularly with implementing monitoring and accountability procedures and in meeting targets, given the demands on the time of the CPFSA staff. Sections 4.2 – 4.4 presents findings from surveys and discussions with CPFSA trainers and caregivers. These help to provide a more rounded picture of the effectiveness of this component of the TLP. 4.2. FINDINGS FROM SURVEYS COMPLETED BY CPFSA TRAINERS The IASR received a listing of trainers from the CPFSA. With support from the CPFSA, questionnaires were dispatched with requests for their completion. Twenty-four females completed the online questionnaires; 25% (6) had been employed by the CPFSA for 15- 19 years; 16% (4) for 10 -14 years and 33% (8) for 5 - 9 years. Respondents had diverse responsibilities, including social work, foster care managers, supervisors for children who are on Fit Person Orders and Supervision Orders; investigators; and case management supervision. All were aware of the TLP and were clear on the objectives: to prepare children and youth for independent living. Seventy percent (16) of the respondents were aware of the life skills component only while the remaining 30% claim to be aware of all components. Respondents were asked to state the number of training sessions they attended. The responses ranged from between two weeks to five years prior (2015). Fourteen percent 19 of 87 had attended 1 training session; another 14% attended 2 sessions; 9% attended 3 sessions and another 9 attended 4 sessions. One respondent reported attending 5 sessions. Table 4.2.1 below shows the trainers’ perspectives on the effectiveness of differing components of the training they received. The majority (82%:18) considered the training methods to be adequate; however, 9% suggested differing methodological approaches: “The topics are heavy. Each trainer should have been given at least 5 days training” “There should have been smaller group cohorts” “The sessions were long, and interest was lost in the later sessions”. CPFSA trainers were asked to state how many sessions they had conducted with caregivers since receiving training. Twenty three percent (5) had conducted 10 or more sessions; 27% (6) conducted between 2 and 4 sessions. Table 4.2.1 Effectiveness of the Training Received in Specified Areas Very effective Effective Somewhat effective Ineffective N/a I did not receive training in this area Total Weighted average Prepare children to apply for their birth certificates, passports, TRN 38.10% 8 23.81% 5 4.76% 1 0.00% 0 33.33% 7 21 2.67 Prepare children to manage their money through budgeting, opening a bank account, etc. 36.36% 8 31.82% 7 18.18% 4 0.00% 0 13.64% 3 22 2.23 Prepare children to manage their daily living (washing clothes, cooking a meal, etc.) 38.10% 8 38.10% 8 9.52% 2 4.76% 1 9.52% 2 21 2.10 Prepare children to solve daily problems they may encounter 33.33% 7 47.62% 10 14.29% 3 0.00% 0 4.76% 1 21 1.95 Prepare children to manage anger 30.00% 6 50.00% 10 10.00% 2 0.00% 0 10.00% 2 20 2.10 Prepare children to resolve conflict 35.00% 7 40.00% 8 15.00% 3 0.00% 0 10.00% 2 20 2.10 Prepare children to apply for jobs 28.57% 6 28.57% 6 28.57% 6 0.00% 0 14.29% 3 21 2.43 Prepare children to access healthcare 25.00% 5 50.00% 10 10.00% 2 0.00% 0 15.00% 3 20 2.30 Prepare children to make their own living arrangements such as searching for a place to live, renting, purchasing, etc. 15.00% 3 50.00% 10 20.00% 4 0.00% 0 15.00% 3 20 2.50% 20 of 87 Fifty six percent (10) of the trainers considered the caregivers they trained to be very receptive; 33% (6) thought they were receptive and 11% (2) somewhat receptive. Nevertheless, training would be more effective if the following amendments were made: “Information could be delivered in the form of video-taped skits” “More use of electronic technology” “CPFSA staff offered more frequent refresher sessions to caregivers and other facility staff”. The factors that facilitated achievement of the learning targets included the willingness of the participants; incorporating skits and activities and offering incentives; and the quality of the material: “the material provided was very detailed and helpful”. Conversely, the caregivers’ sporadic availability was the major factor that compromised training outcomes. “Not all staff members were able to attend all the sessions as some had to be rostered to provide supervision and care to the children at the facilities”. However, there were other factors: 1. “The modules are too long; not enough time”. 2. “Caregivers financial constraints, which hinders attendance” 3. “Caregivers inability to comprehend information received”. 21 of 87 Specified Areas Considering the children in State care within your region, what could future versions of the TLP add to its life skills training menu in order to meet the needs of those wards who are exiting the homes? Here, the CPFSA trainers recommend the following components: § On time monetary assistance and housing solution for each ward, up to 1 year § Cooking and home management skills § I would love to see more everyday topics be taught such as being responsible, having manners, public speaking etc. § Collaboration with organizations, such as Courts, Singer or ATL, in order to be able to provide our youth who are transitioning with basic furniture and appliances § A dedicated caseworker § Driving lessons and applying for a driver's license. § Job placement within agencies § Practical demonstrations of training (practicum aspect) where the children are taken on field trips to make the experience come alive for them § Relationship management § Navigating social media Table 4.2.2 Extent to Which Caregivers Have Been Equipped to Provide Training in Specified Areas 1-Not equipped 2- Partially equipped 3- Equipped 4- Not equipped 5- I am unable to assess Total Weighted average Prepare children to apply for their birth certificates, passports, TRN 0.00% 0 13.64% 3 31.82% 7 22.73% 5 31.82% 7 22 3.73 Prepare children to manage their money through budgeting, opening a bank account, etc. 0.00% 0 18.18% 4 50.00% 11 9.09% 2 22.73% 5 22 3.36 Prepare children to manage their daily living (washing clothes, cooking a meal, etc.) 0.00% 0 9.52% 2 38.1O% 8 28.57% 8 23.81% 5 22 3.45 Prepare children to solve daily problems they may encounter 0.00% 0 18.18% 4 40.91% 9 18.18% 4 22.73% 5 21 1.95 Prepare children to manage anger 0.00% 0 31.82% 7 40.91% 9 4.55% 1 22.73% 5 22 3.18 Prepare children to resolve conflict 0.00% 0 22.73% 5 50.00% 11 4.55% 1 22.73% 5 22 3.27 Prepare children to apply for jobs 4.76% 1 19.09% 4 38.10% 8 14.29% 3 23.81% 5 21 3.33 Prepare children to access healthcare 0.00% 0 9.09% 2 40.91% 9 22.73% 5 27.27% 6 22 3.68 Prepare children to make their own living arrangements such as searching for a place to live, renting, purchasing, etc. 0.00% 0 23.81% 5 33.33% 4 14.29% 3 28.57% 6 21 3.48% 22 of 87 With respect to modifications in training for the caregivers, CPFSA trainers recommended the following actions: • Dividing the module into two components where trainers would do one phase of the training and return for evaluation and training in Phase 2 for roll out - being guided by what obtained in phase 1 to improve on effectiveness; • Including a component on mental illness; • Providing one week’s training for caregivers; • Incorporating a home management component, which includes cooking; • Increasing the number of refresher courses and including periodic evaluations; • Including a 'practicum' component where the information is put in practice and trainees give a report on the outcomes as a result of their interventions; • More hands-on activities to prevent caregivers from writing; • Ensuring all caregivers are provided with the resources to assist children; and • Increasing refresher training for the caregivers who were trained, in which they would prepare mock training sessions for the youth. “This would allow us to better assess their level of readiness.” 4.3 SUMMARY OF FINDINGS FROM INTERVIEWS AND DISCUSSIONS WITH CAREGIVERS The evaluation team conducted interviews with respondents from all the children’s homes that were listed in documentation provided by the CPFSA. The interviews were necessary since many of these respondents were reluctant to complete the questionnaires, even with assistance. Thus, the methodology had to be changed. The interviews addressed the matters that were outlined in the questionnaires (See Annex). The findings are summarized below. Facility 1 Interviews were conducted with three (care) staff members. Respondents qualified that many of the residents are not cognitive; therefore, much of the training has not been pertinent. The training has been useful, primarily for one young man. Respondent: “You see, the thing is this, most of the residents that we care for are not cognitive [that is, not able to understand the material presented]. There is only one young man whom we have been working with for some time now. He's 24 years old. The staff has been dealing with him because of his behaviour. I see a little change there; it's not much of an improvement, but there is some improvement.” Respondents were asked to assess the value of the Trauma Informed Care component. Here, the staff concluded that this component has been useful primarily for one young girl, who is 10 years old. There was one boy who needed this trauma informed approach; however, he has been placed in another facility. 23 of 87 Given the needs of the home, staff explained that they would profit more from training that addressed development of their techniques, such as in feeding the clients (such as through the use of tubes and by hand] and providing adequate medical care. Nevertheless, they had been able to apply those aspects of Life Skills training that focused on matters such as hygiene and self-care. Across homes, staff requested increased training for themselves, such as to deal with their own experiences of trauma. Staff explained that there have been instances where their peers were unable to deal with the disrespect meted out to them by the children. Furthermore, while they receive some support, the type of care they are required to provide takes an emotional toll on the caregivers. Facility 2 Staff reported that six persons attended training in October 2020. Training was carried out for two days, lasting four hours each. The material taught included conflict resolution, budgeting, decision making and time management. Respondents considered the training to be effective; however, they rated its effectiveness for their institution at 4 out of 10, with 10 be the highest. This is because training was not entirely applicable, given their clientele. (With respect to the training they received and the general knowledge provided, they ranked this at 8/10.) Caregivers explained that they do not need refresher courses in Life Skills but, instead, training for rehabilitation or patient care for children who had little mobility. They are clear on how to implement the training received, although it is not applicable to them. Facility 3 Two persons received training either in 2017 or in 2018; respondents were not clear on the date. One respondent, who was listed by CPFSA as having received training, stated that she did not, in fact, attend the training sessions. Another who received training confirmed that this occurred a long time ago and so she does not remember much. However, she did recall that it was a good experience, which reinforced what she already knew. Nevertheless, all respondents believed that staff support and the psychosocial care components were insufficiently addressed. Interestingly, staff noted that, thus far, the children who had transitioned from the homes were still dependent on them for assistance with rent and other amenities. Thus, they questioned the effectiveness of the TLP. Facility 4 Staff from this institution stated that one person received training recently; that is, sometime in October or November 2020, via Zoom. The training was described as informative; it reminded them of critical problem-solving strategies. Staff recalled aspects of the Trauma Informed Care component, which included a session in which their peers shared their personal traumatic experiences. However, they indicated that much more 24 of 87 help is required in this area. They also believe that the training was too short; there was one Life Skills session that lasted 3-4 hours only. Despite these limitations, staff suggested that the children are benefiting from the training they received; they had seen changes in the children’s behaviours. Caregivers now do things differently; for example, they were less prone to “rush or get angry" with the children. Since training, they now tell themselves that the children have been affected by trauma in some way or the other; therefore, they now try to take things easier on them and talk things out more. Prior to the training, the caregivers would be quick to punish the children. Respondents believe that the training should be an ongoing process and “not just a onetime thing”. They suggest that training could be held “every three months, preferably face to face while enforcing the necessary protocols”. The caregivers agreed that it was difficult to concentrate on the material being presented during Zoom meetings; thus, learning is limited through that medium; many are unable to recall details from some of the topics. Facility 5 Interviews revealed that some persons whom the CPFSA office listed as trainees did not participate in the TLP sessions. Further, there are others, also listed, who care for persons with disabilities; therefore, the training was not applicable to them. Staff noted that in 2018, the CPFSA provided them with two manuals. Training received since then has been informative; however, they pointed to the limitations of receiving training via Zoom: these sessions did not allow for much interaction. Overall, while training helped, in some part, with the children; more focus is needed on behaviour management rather than just life skills. Staff pointed out that the training sessions “added a bit of trauma to them”. They noted that they were unable to focus while training was taking place, as their minds were occupied with whether or not things were operating smoothly with the children in their absence. Respondents felt that team building exercises to promote unity should have been included, as well as activities to build staff motivation. Respondents consider themselves to be parents, teachers, and, indeed, all the resources that the children need in one, while having “their own problems to deal with”. Staff made special note of the conditions at one location. There, staff live on site and are most times at work with no relaxation time or “me- time”, in a space that is very chaotic. Caregivers go beyond call of duty at all times; therefore, more talks to make them feel appreciated are needed along with more living space. 25 of 87 Facility 6 Eight persons completed the training in October 2020. Training lasted for two days, with two four-hour sessions each. Caregivers recall being taught how to assist children with various life skills such as: • applying for bank accounts and other necessary documents. • managing money • home maintenance • benefits of saving They rate the effectiveness of the training at 9 out of 10. Respondents have used the information from the Life Skills Training to help children in need of documents to get them. Nevertheless, refresher courses are needed. Staff are not clear on how to implement what was taught since the Zoom sessions went very quickly; however, they do have a manual that they can consult when needed. Facility 7 All staff who received training have since left this institution; no knowledge was transferred. One respondent explained that there is a high staff turnover rate at the institution, given the rampant disrespect they experience and underscore their reluctance to continue SUMMARY The findings indicate that while the trainings were useful for a significant proportion of children and youth, there are particular cohorts who needed more context relevant training. Furthermore, given the needs identified in the initial analysis (Needs Assessment), the Life Skills component could have included more psychosocial components and other practical subjects. While the psychosocial/ mental health component was addressed later in the project through the trauma informed care training, it would have been useful if this component were included earlier in the project. One assumption behind this Life Skills component is that it would provide a sound example of effective “trickle-down” training. However, without adequate monitoring mechanisms, it is difficult to gauge the extent to which the training actually profited the children. In this stream of activities, respondents indicate that the training provided to CPFSA trainers was, largely, effective. However, while the training transmitted to the caregivers confirmed material they knew and/or provided new knowledge, it was not, in some cases, applicable given the clients served. In addition, respondents note that they would have benefited from greater emphasis on the psychosocial component as well as the techniques required to serve in their own contexts. 26 of 87 There is some agreement that the training period was too brief and that additional sessions were needed in order to reinforce the material. The findings also raise questions about data accuracy. CPFSA staff maintain that the listing of persons who received training is correct; however, caregivers state that some persons who have been listed did not attend sessions. CPFSA trainers raise the need for greater monitoring and the conversations with caregivers confirmed that implementation has been patchy. In order to ensure that this training component has maximum impact, the CPFSA is encouraged to strengthen the corresponding management and accountability mechanisms. 27 of 87 4.4. EFFECTIVENESS OF THE VOCATIONAL SKILLS, ENTREPRENEURSHIP AND MENTORSHIP COMPONENTS Vocational and entrepreneurship activities, accompanied by job placement, addressed an important part of the second goal of the Transitional Living Programme (TLP) for Children in State Care, namely: “To reduce the risk factors associated with low skills development, inadequate life skills and poor self-image”. Correspondingly, two of the 13 anticipated results at the end of the project related, specifically, to this area: 1. Five hundred youth in State care trained in a vocational skills training programme. 2. Provision of employment for 92 youth, through a network of public and private companies. PROCEDURE AND METHODOLOGY In order to conduct a reliable assessment of the changes/outputs and achievements, the evaluation team conducted detailed reviews of the annual and quarterly reports as well as interviews and case studies with service providers: Cobbla Skills Training Centre, Spalding, Clarendon; Children First, Kingston/St Andrew and St Catherine; and 24/7 Productions. IASR project administrators made several attempts to contact all service providers. The team was unsuccessful in its efforts to secure interviews with HEART NTA and AVASANT. The findings from the documentary reviews and interviews are outlined below. 4.4.1 APPROACHES TO PROJECT IMPLEMENTATION A. SKILLS TRAINING Initially, the joint CCDC/CPFSA Project Management Team focused on getting wards accepted into HEART courses. Youth who had no subjects were given the standard diagnostic test in Maths and English; less than one third (30%) passed and were eligible to enter a HEART course. Thirty-three per cent were eligible for pre-Tech programme placement while 37% required remedial education. Those youth who had two or three subjects were eligible for direct acceptance. Applications were submitted later than desired; therefore, by the time the tests were taken and results given, all the various centres offering courses had completed enrolment. In Year 2, of the 314 tested, 41% passed. In this first year in which the youth took courses, 113 enrolled, 13 dropped out and six completed. In the second year of the programme, the Project Management Team embarked on a series of sensitization sessions with stakeholders, HEART Community Training Institution (CTI) managers and CPFSA staff and youth. The aim of these sessions was to address the issues that arose in the first project year and refine the vocational skills component. The sensitization sessions revealed the young people’s desire for wider course offerings. The following year 28 of 87 recorded more than double the enrolment, surpassing the target, and a much higher rate of completion, some likely rolling over from the previous year. Therefore, the project’s key partner in this component, whose training held the key to employment and to entrepreneurship, was found to be oversubscribed and under￾resourced. In principle, once a trainee was successful, he/she should have received a Statement of Competency from HEART to verify that they have completed their course. Within a reasonable time, they should then receive their Level 2 Certification. (Employers may accept a statement of competency when the trainee has recently graduated; however, after a while they may not acknowledge this if no certificate can be produced.) By the end of Year 2, only 5% of the youth had received the initial Statement of Competency. By Year 4 only 118 trainees out of the 275 who had been trained had received their Statement of Competency; this proved an impediment to job placement. The project managers had no recourse but to keep a good relationship in this essential partnership, while pressing for certification and seeking other ways to train their charges. A number of the training institutions, such as St. Joseph’s Teacher’s College, Avasant Foundation and Excelsior Community College, had their own recognized certification. Against this background, members of the project team met with HEART’s Executive Director to elicit his buy-in and voice their concerns about the challenges being experienced with placements. It emerged that HEART was experiencing an oversubscription to its courses. The Executive Director, nevertheless, proposed that the youth be accommodated under the National Unattached Youth Programme (NUYP), which was implemented through CTIs. He pointed out that course applications tend to come in late; therefore, it was agreed they should be sent in as soon as anyone wanted to apply. Other mechanisms were also developed for closer communication and improved sorting of applications. In light of the hurdles which resulted from oversubscription to HEART, coupled with the inability of many youths to pass the diagnostic test, the Project Management Team approached USAID for approval to contract additional accredited skills training institution and for customized programs. This request was approved. The options, outside of HEART, not only widened the scope of offerings; in some instances, they offered a new approach to teaching that incorporated a life skills approach, reaching students not just at the cognitive level but also at the level of their emotions through various tools, for example the Performing Arts. By the end of the project, wards had graduated in the following courses, with the first seven being the most popular: Cosmetology, Housekeeping, Allied Health and Practical Nursing, Commercial Food Preparation, Photography and Videography, Call Centre training, Food and Beverage, Plumbing, Administrative Assistance, Massage Therapy, Computer Servicing and Support, Furniture Making, Professional Driving, Front Office 29 of 87 Concierge, Welding, Data Operations, Early Childhood Education, Dining and Bartending, Electrical Installation, General Agriculture, Digital Animation and Art and Craft. Table 4.4.1 below demonstrate that the project was largely successful in achieving its enrolment targets in Skills Training and exceeded them in some cases. Table 4.4.1: Progress of Skills Training Component Year 1 – Year 6 by Gender Year Target Enrolment Actual Enrolment Target Completion Actual Completion Drop outs Yr 1 - 2015 0 0 0 0 0 Yr 2 – 2016 160 113 (69%F, 31%M) 16 6 (100%F) 13 Yr 3 – 2017 240 249 (89%F, 11%M) 136 108 (73%F, 27%M) 64 Yr 4 – 2018 180 181 (59%F, 41%M) 175 161 (60%F, 40%M) 40 Yr 5 – 2019 95 214 (66%F, 34%M) 150 179 (64%F, 36%M) 63 Yr 6 - 2020 0 13 (69%F, 31%M) 46 52 (69%F, 31%M) 0 Total: 679 770 (66%F, 34%M) 523 506 (64%F, 36%M) 180 N.B. Some participants who dropped out later returned to repeat a course but numbers or age or gender breakdown have not been provided. However, the evidence indicates that there was a dropout rate of one in four trainees. The Annual Reports noted that youth have dropped out of the skills training courses for reasons including: -Pregnancy -Lack of interest -Family commitments -Maladaptive behaviour -Indecision -Inadequate stipend (bus fare and lunch) -Other financial problems 30 of 87 -Ill health -Deferral of course (perhaps for some of the above reasons). Therefore, except for Years 5 and 6, actual completion rates were lower than targeted. Consistent with national trends, higher proportions of males than females completed training courses. The Importance of Comprehensive Strategic Interventions Some of the youth who participated in two major customized programmes, the BPO training course run by Avasant Foundation1 and the St. Joseph’s Teacher’s College Workforce Transition and Competency programme, presented serious behavioural issues. These behavioural issues included defacement of property, group conflicts and profanity. Partner meetings were convened to discuss problem-solving strategies, including unannounced visits by CPFSA and CCDC teams to meet with students. Eventually, some youth had to be taken off the courses and efforts made to find other options. The Project Management Team’s response was to institute Life Skills sessions. The Team also determined that, for the future, youth were to be exposed to life skills coaching before the commencement of their programmes. Furthermore, expectations regarding behaviour and attendance had to be stated from the beginning. The evaluation team conducted detailed interviews with two agencies who had responsibility for providing vocational skills training. The accounts (depicted below) demonstrate the types of comprehensive intervention strategies that are critical for behaviour change and successful outcomes. Trainers from three quite different settings were interviewed: 1.Cobbla Skills Training Centre, Spalding, Clarendon is a community training institution (CTI) that offers one-year Level 2 HEART courses. Since 2017, they have had 5 wards of the State (3 males and 2 females) take courses at different times in Commercial Food Preparation, Cake Baking and Decorating, and Beauty Therapy and Cosmetology. Two of the male wards dropped out, one because of mental illness and the other for unknown reasons. He said he had financial problems but did not disclose that CPFSA was supplying the funds to enable him to attend training. Of the three that completed their courses, the male works as a baker in a company in Mandeville, one female went on to complete another HEART course and is now a teaching assistant, and the other runs a cosmetology business from her home. 1 The Avasant Foundation trained 92 wards, found employment for 19 and one internship. The Foundation also facilitated interviews for 14 other graduates (outcome unknown). Ninteen youth successfully graduated from the St. Joseph’s Teachers College. 31 of 87 The wards attending Cobbla CTI did not exhibit any behavioural problems and got on well with the other students; however, two dropped out. In the photography and videography course, officially entitled Basic Digital Media Production and Information Technology, it took the wards two weeks to settle down. Remarks like: “Me nuh know how dis a go help!” “Waste a time business!” (I do not know how this will help. This is a waste of time) were heard. One student was particularly disruptive. By the third week, however, the teaching team of three reported that “There was a complete transformation.” They became fully engaged. The trainers’ analysis was that, at the beginning, the youth thought that this was another situation “where they would be judged”, including by their peers, as most of them did not know each other. Cobbla’s program managers have suggested that the youth were attracted to the hands￾on, very practical course, which is a learning approach that is particularly suited to most boys. This course gave immediate gratification as youth could instantly see their photograph or video clip. After two weeks, according to the Photography and Video teacher, they were taking selfies of each other at lunch time (he had told them to practice their skills on their phones) and correcting each other, repeating the trainer’s instructions where they thought it was relevant. He found it “a very exciting time” to watch them as their relationships with each other became transformed and they threw themselves into the work. Every Friday, he took them on a field trip which was (as for all students on any course) the highlight of the week. They explored Kingston with new eyes, getting exposure, in addition, to many places they had never been to before. This was not the only aspect of the course. There was a life skills component teaching personal development through, among other approaches, the Performing Arts. The Trainer in Personal Development and Performing Arts commented that within three months the youth had moved from “a total absence of self-confidence and self-worth to feeling very good about themselves”. He noted that Performing Arts reaches deep into people, touching their emotions. At the same time, he noted, they understood firmness; they recognized when this was required. What they needed was attention from people who cared about them. The teachers acknowledged that this applied to one young man who was the worst behaving participant at the beginning. He is now pursuing a degree in the field of Creative Arts at the University of Technology; his focus is on Videography. His teacher was moved by a recent phone call, in which this young man asked if he could do anything to help him, especially during holidays and when he had spare time. He was not just looking for paid work but to voluntarily help because he, himself, had benefited so much from his teacher’s expertise and support. 32 of 87 2. Children First, an NGO that offers HEART Level 1 certified courses, carried out two three-month, 3-day a week, customized courses, one in Cosmetology and the other in Art and Craft, for 26 wards resident in the Homestead Place of Safety for Girls, Stony Hill. The Children First Cosmetology Course is a Level 1 course which, by arrangement with HEART, gives the successful graduate matriculation into the Level 2 Cosmetology course. The Children First Art and Craft Level 1 course, again by arrangement with HEART, makes the successful graduate eligible to enter the HEART Special Arts Programme in jewellery making. It can be noted that the latter courses were customized courses, attended only by wards of the State. The Children First team reported that they experienced challenges on several levels: 1. Inadequate facilities & Supplies: The allocated training room was considered inappropriate and in dire need of renovation. Children First hired a handyman and, over four days, he repainted the room, fixed the door and windows and undertook other small renovations. Consequently, all the wards, not just those selected for this TLP project, wanted to join the courses. The freshly painted new-look training room, itself a magnet, combined with the course activities attracted all the wards. Children First adopted an inclusive approach; they reached out to all children regardless of the behaviour. This resulted in the courses being “flooded” within two weeks by most of the older Homestead wards, whether or not they were selected to be in the TLP. Children First ended up with course batches of 27 and 24 students, which required strategic management to ensure the training courses were effectively managed and delivered. 2. Youth with Mental Disorders: Children First assisted approximately eight students who displayed major abnormal behaviours and possibly mental health issues. The Executive Director engaged a psychologist who conducted: - Sessions with the Training Team on how to (a) deal with mental illness and (b) classroom management. It was decided that the Team Lead, who had to attend every session, should hold first degree qualification in Psychology. She would provide on the spot one-on-one counselling or mediation, should behavioural issues become disruptive. - One-on-one assessment of the most obvious ‘problem’ students. - Two one-day sessions with all the Homestead staff, some of whom were described as “just too angry”. These staff comprised caregivers, cooks, groundsmen, drivers and office personnel. The staff were divided into two groups, with one day allotted to each for psychological intervention. Children First would have liked that more time be allotted to this; however, this was not possible. For this reason, the Executive Director continued to hold sessions with the caregivers during the training period. Most had emotional issues, which they needed help with working through. One session, entitled ‘Young Again’, went back into childhood and 33 of 87 revealed that most staff had experienced abusive or challenging childhoods. The anger from these experiences was still there. Some were former wards of the state. Role play was used among students and staff in their different sessions. Some wards used drawings to depict staff as animals, which revealed the serious tensions within the home. Behaviour Progression through the Courses The youngsters’ behaviours gradually transformed as the training progressed. They were addressed as ‘Ladies’ and this, along with the provision of uniform T-shirts from Children First, led first to much better care with their appearance. Life skills training covering self￾esteem, having a sense of purpose, and conflict resolution were incorporated into the classes. Every morning started with a “Come mek we reason” (Come and let us talk) session to resolve the conflicts that occurred throughout the course of the sessions. On occasion, this conflict resolution session extended for the entire period allotted for training. For both customized courses, Children First acknowledged that participants displayed transformed behaviours and a tremendous sense of achievement. At the end of the course, which was convened at Excelsior, students had to produce a video. Trainers noted that the students were committed to the process. They regarded this activity as important for their own sense of self-worth; it was important to them that they prove to their teachers (and to themselves) that they could complete the task. Students displayed a great sense of pride when they produced the video. All passed the course. The Homestead girls demonstrated heightened levels of pride in their activities and in themselves. At the end of the course, Children First hosted an exhibition of products, which elicited genuine praise from surprised parents and staff, the latter however already having recognized positive changes. Case Example of the Effectiveness of Vocational Skills Training 24/7 Productions offers training courses in digital photography and videography at Excelsior Community College. The course also includes some entrepreneurship training. Trainers taught a customized three-month, 5-day a week course at the College for 15 wards of the State, 14 of whom were males. All successfully graduated, each having made a video as their course output. The company has information on four graduates: one is now at UTech; one graduate is very talented as a musician and in composing songs and the company is helping him to produce these; a third graduate is now in the army and keeps in touch with one team member, who gives him advice from time to time; and the single female graduate, who held her own very well with the group, is now pursuing a business course at Excelsior Community College. 34 of 87 A. Job placement The target for this project component, although relatively low at 18%, was the most difficult to achieve. The management team had conceptualized an Employment Network lobbying group and, from Year 2, they endeavored to accomplish this through: •Organizing early in the second year a breakfast meeting attended by several important corporate businesses •Attending another NGO organized network breakfast to which they were invited. •Meeting with the Jamaica Employers’ Federation •Holding individual meetings with employers like Boss Furniture and Manpower & Maintenance Services •Meeting the HEART Trust/NTA Employment and Career Services (ECS) Department •Other fact-finding missions taking them to other government ministries and agencies, and a World Bank project in digital animation. •Meeting the Executive Director of the Avasant Foundation to discuss the creation of a customized Business Process Outsourcing (BPO) training programme for wards of the State. •Taking the necessary steps to expand the UWIOC Scholarship, recognizing the need so assist the wards to obtain qualifications such as CSEC Math and English. •Following up on all networking opportunities that gradually arose from this work. In one early meeting, which was held with the Programme Coordinator for the USAID￾supported New Path programme at the South Camp and Metcalfe Juvenile Remand Centres, the management team was advised that there are several similar programmes currently existing “that compete in the same space for private sector support with job placements. Hence, although organizations may indicate initial enthusiasm, their experience has been that it has not sustained well enough to culminate into the desired numbers of job placements” (Annual Report Year 2, 2015-2016, p. 13). Added to this was the traditional reluctance to employ youth in care due to stigma and pre-conceived perceptions. Meeting the project target was therefore no mean feat. B. Transition to work The Transition to Work component constituted another thrust towards achieving employment for youth in State care by getting internships for those who possessed two or more CXC passes. It was hoped that they could be placed through the HEART Trust/NTA Employment and Career Services (ECS) Department. A meeting was scheduled, in which stakeholders would discuss the programme with interested youth. Several attempts to convene the session proved unsuccessful; correspondingly, the team decided to employ a new strategy whereby interested youth would be matched with participating companies and the HEART ECS department would implement the monitoring aspect of the programme. The reports indicate that the stability of the youth in this programme proved problematic. Some who were eligible for the programme were indecisive or indicated 35 of 87 they were not interested in working or pursuing training. By Year 5, of the 16 youth placed the year before, 13 had dropped out, one to take up a training course instead. Punctuality also proved problematic for some. One on one coaching was provided in these cases. Some companies were very understanding and recognized that a mentoring approach was essential. The mentoring programme, once established, would in the future also be a useful support for youth in internships. By the end of the project, 30 youth had passed through the Transition to Work programme; some have found permanent employment via this route. C. Entrepreneurship The Entrepreneurship component of the TLP was implemented in Year 4 of the project. This component was to be restricted to those who had successfully completed a vocational skills training course. Members of the TLP management team had encouraged various training groups to collaborate in providing this component of the TLP; however, they received responses from two only: Junior Achievement Jamaica (JAJ) and the Business Entrepreneurial Empowerment Programme (BEEP). JAJ’s approach is to train facilitators from within the interested organization; therefore, two CPFSA staff members were trained, along with two project staff members as back-stop. Under this component, USAID provided a number of grants for business starts. Youth were invited to submit an application, outlining a business plan. They were taken through a training session to improve the business plan, after which they participated in a pitch event. The judges either recommended the youth for business start-up support or for further development of business plans. The 19 youth who graduated with the Certificate in Workforce Transition and Competency Programme, a customized course held at the St. Josephs’ Teachers College in which they were exposed to an entrepreneurship module, were also invited to compete for these grants. In Year 5 of the TLP, entrepreneurship training started with the first cohort of youth. A second and third cohort were trained in Year 6. Every effort was made to give practical assistance to those who were successful in the first pitch for grants. The initial disbursement to the group came in February 2020 and was used for the registration of their businesses with the Companies Office of Jamaica as well for opening business accounts with a commercial bank of their choice. Business registration applications were completed, after which the group was chaperoned to the Companies Office of Jamaica to submit the documentation and to register their businesses. However, of the seven (7) in attendance, only five youth were able to successfully register. The two Cosmetology business applicants were informed that they needed to first obtain a license from the Municipal Council in order to practice as Cosmetologists. The next step was to obtain quotations for the equipment and supplies they needed for start￾up. The project team would then purchase the items on behalf of the trainees. The 36 of 87 process was conducted with rigour and they were expected to adhere to the budgets in their business proposals. With COVID-19, Cohort 2 was forced to pause their final pitch event. However, plans were put in place to continue the training of Cohort 3, virtually, and to take up the final steps for Cohorts 1 and 2, once restrictions were eased. Interested residents from the Mona and Lady Musgrave Apartment complexes were also included in the Entrepreneurship Training Sessions. D. Mentorship The MultiCare Youth Foundation (MYF) had responsibility for implementing the mentorship component of the TLP. MYF signed a contract in May 2018 and initiated what should have been an 18 month programme. However, although this component was slated for completion in October 2019, two extensions were requested and granted. A mentor programme committee, comprising representatives from the MYF, the UWI Open Campus, CCDC, CPFSA, the SWTC and the Project Management Office was appointed to steer this component. An interview was conducted with the Executive Director of MultiCare Youth Foundation. While lauding the achievements of this component, she noted that there were some key challenges that slowed the progress of implementation. First, there was slow recruitment of mentors and, especially, mentees. CPFSA was responsible for recruiting mentees. These mentees had to fit certain criteria for inclusion in the programme. For example, mentees had to be part of the TLP Housing Assistance. Programme. As delays were experienced in the Housing Component, this affected the pace at which mentees were recruited. Nevertheless, she reported that the team worked together to define a solution; it was decided that mentorship would be extended to youth who were participating in other programmes, such as the Vocational Skills and Entrepreneurship components. The Director described other challenges, such as the problems with securing appropriate matches and in sustaining relationships between mentors and mentees. MYF’s Final Report confirmed completion of the following deliverables, despite the identified constraints: (i) “Assessing existing mentorship programme materials from the Child Protection and Family Services Agency to ensure that there was no duplication of efforts in the creation of this programme”; (ii) “Drafting and submitting mentorship guidelines and a mentor training plan, along with a training handbook”;. (iii) “Revising mentorship guidelines and training handbook, based on feedback from the project team”; 37 of 87 (iv) “Reviewing mentee profile form and mentor application form”; (v) “Reviewing and revising mentor/mentee matching criteria and procedures”; (vi) “Conducting all training sessions with mentors and mentees, including refresher sessions”; and (vii)“Creating and executing a preliminary evaluation plan with accompanying instruments for the mentorship training to assess the following components: a. The success of the mentor/mentee match b. The success of the mentorship training c. Any other component deemed necessary based on research or in consultation with the mentor programme team d. Attending meetings with the mentor programme team to provide updates. e. Submitting a report based on outcomes of the mentor training, to include strengths, challenges, and recommendations for future training. f. Finalising mentorship guidelines and training handbook, including graphics and copyediting, for publication. SUMMARY The Vocational Skills, Entrepreneurship and Mentorship Components of the Transitional Living Programme for Children in State Care Project were, in large part, successful. Lessons for the future development of this very important programme can be learnt and gaps and insufficiencies strengthened. In terms of its effectiveness, the TLP: 1. Achieved many of its targeted outputs, despite the challenges experienced. 2. It built capacity in CPFSA and CCDC, at times through trial and error, to organize a skills training programme for wards in State care, to seek job placements, and to manage an entrepreneurship programme, which aimed at placing youth with the appropriate talents on the path to micro and small business. 3. It succeeded in overcoming a number of unexpected hurdles, such as HEART’s inability to provide enough course spaces, by finding alternative options, such as in customized courses. There are some noteworthy considerations, which the evaluation questioned but was unable to resolve. First, in selecting youth for mentorship, the TLP prioritized those who qualified for housing assistance and/or were enrolled in vocational skills or entrepreneurship programmes. It is critical to consider the implications for children who have not been selected. How consistent is the selection process with principles of equity? Second, in 2012, there were 5,336 children in State care, 50.5% being female and 49.5% being male (TOR. 1.3, p. 47). Yet, of those who completed skills training 64% were females. While this disparity is consistent trends in the Jamaican context, how does the TLP plan to address it? This question is especially pertinent in Jamaica’s socio-economic 38 of 87 context, where the major victims and perpetrators of violence are male youth who lack education and skills. It is noteworthy that the TLP has been making efforts to include groups who suffer compounded forms of disadvantage. Notably, in Year 5, the project reached out to a group of children with disabilities, those in the Mustard Seed Communities and a cohort of 30 participated in the Avasant BPO training; 21 of them graduated. These questions and observations are presented as challenges; they indicate the depth of programming that may be required to meet the needs of those least likely to transition successfully out of care. 39 of 87 4.4 EFFECTIVENESS OF THE TLP: VIEWS FROM YOUTH LIVING IN TRANSITIONAL HOUSING After several revisions of the target, the TLP aimed to reach 52 youth for inclusion in the transition housing intervention. These youth were to be placed in one of two transition housing complexes, both in the corporate area. The two-year initiative would allow youth to apply skills learned in the life-skills intervention while living independently. To be selected for the Transition Housing Programme, participants had to be enrolled in a training or tertiary education programme, be able to pay the nominal rental, if employed, and have no housing option on exiting state care. While the TLP While the TLP has provided a capacity for 52 youth; 38 are currently in housing due to COVID-19 Social Distancing requirements that restricts double occupancy. The evaluation team considered views from this group of youth to be especially valuable, given their participation in the varied aspects of the TLP and demonstrated capacity to assess the project components. Correspondingly, the IASR convened a focus group and subsequent Whats App survey with youth currently residing at the Lady Musgrave and Palmoral locations in Kingston and St. Andrew. As noted, the objective of the group discussion was to garner youth perspectives on the effectiveness of the TLP and its related activities. The CPFSA facilitated this engagement. The following questions formed the basis of the Focus Group Discussion. They were taken from the original list of interview questions and some were modified and aligned to specific themes of: (a) readiness for independent living, (b) knowledge and attitude improvement, (c)interpersonal relationships and social interaction; and (d) programme relevance and applicability. Readiness for independent living 1. How prepared were you to live on your own after leaving state care? Modified: Do you feel the programme prepared you for independent living? Knowledge or attitude improvement 1.What skills had you (a) learnt and (b) mastered? Modified: What skills have you learned or mastered as a result of the programme? Interpersonal relationships and Social Interaction 1. How did you relate to other children in care? Modified: How has the programme helped you to relate to peers, caregivers, or peers at school? 2. How do you handle conflict? 3. Did you have experiences or interactions in which you feel you were treated differently (for example, being pre-judged, discriminated against, marginalized) Programme Relevance and Applicability 1.Were you comfortable with the group activities in which you participated? 40 of 87 2.Did you have any issues with the material from the training session? In addition to these four (4) themes, participants were asked to reflect on their personal development from an emotional and psychological perspective, and evaluate their current competence of accountability, responsibility, and discipline. PROCEDURE AND METHODOLOGY With consent from the CPFSA, participants were informed of the invitation to the Focus Group. The invitation was facilitated by the Manager of one of the sites, and the Zoom Link was sent to her and shared with the participants. We were informed that there were approximately 38 possible participants. It was uncertain whether they would all be allowed to attend, as some had exams and would be on their way from school. The discussion was conducted using the Zoom platform which allowed participants to join via cellular phone, computers, either with or without an enabled video camera. The focus group methodology facilitated an easy conversational exchange between interviewer and interviewees. The session was recorded and has been used to summarize the findings presented in this report. The session had a total of 13 participants: 12 females from the Lady Musgrave facility and one male from the Palmoral facility. Participants were engaged in varying programmes in higher education programmes, as outlined below. Table 4.5.1 Focus Group Participants and Current Activities (N=13) Gender Current Programme of Study Point of Completion Institution F Economics 2nd year UTECH F Business Administration 1st year UTECH F BSc – Nursing 2nd year UTECH M Law 3rd year UWI F Practical Nursing 1st year Distinction College F Business Management/Computer Studies Final year UTECH F Dental Studies 2nd year UTECH F Sociology/Psychology 2nd year UWI F Public Policy and Management & Political Science 3rd year UWI F Science Education Final year Shortwood Teachers College 41 of 87 F Guidance and Counselling 2nd year Mico University F Environmental Biology Completed Internship F Literature/Language Final year Mico University College Participants were allowed to introduce themselves and inform the interviewer about their current status and activity and encouraged to speak freely. They were assured of confidentiality, and that if any specific comments were to be repeated, that would be done under absolute anonymity. The questions were posed generally, and to ensure each had a chance to contribute to the discussion, individuals were called upon to respond. In addition to the Zoom audio responses, at least three participants used the Chatroom to send their written questions or comments. The session was conducted with free and open-ended questioning, allowing for the conversational style of focus group interaction This section summarizes the discussions and comments resulting from the questions. A. Benefits of the Programme Several aspects of the programme were described positively. The comments are summarized as follows: Social interaction Here, respondents had grown in the following areas: • Patience and appreciation of persons • Appreciating diversity and different values, social experiences, responding to conflict and appreciating others’ perspectives • Learning to advocate for others Life skills Respondents noted that they benefited from improved skills in: • Budgeting and financing • Preparation to take care of yourself • Cooking, problem solving (dealing with difficult situations) • Applying for important documents (bank accounts, Drivers License, passport) • Transitioning to full independence • Taking the lead in one’s life • Full independence • Different experiences in communal living • Opportunities for entrepreneurship 42 of 87 Group activities Most activities were described favourably. Many of the activities that led to improved tolerance, social interactions, conflict resolution were described as helpful to their overall development. The sessions were described as being interactive and enough to promote a good feeling, allowing participants to appreciate different perspectives. Mentorship Participants were more interested in talking about the mentorship programme, which received mixed reviews. Note the following perspectives: Reasons for Positive Experiences with Mentorship • Relationships with mentors were maintained because of similar interests • Some mentorships have been good matches, leading to emerging friendships • Mentors maintain contact, are caring and are nice Reasons for Negative Experiences with Mentorship • Inconsistency with mentors who started out in the mentorship commitment • Interactions and relationships were not always consistent • Attempts to initiate contact were not mutual • Relationships discontinued because of frustration trying to find mentor (‘relationship went nowhere’) Two responses of note are stated here. One participant expressed absolutely no need for a mentor relationship based on how her mentor interacted with her. As she stated: ‘I have tried and I’ve concluded I can do without a mentor.’ In another instance, one mentor started out with the CPFSA’s mentorship program. The mentee reported that this mentor declined to be part of the formal mentorship programme because of a dislike of the position CPFSA took on certain matters. (The specifics were not shared, but it is important to mention it here because despite this observation, the person has continued to mentor this individual). Three other participants declined to clarify but stated they did not have a good experience with the mentorship programmes. Of the total 13 participants, 77% (10) viewed the mentorship component of the programme favourably. In order to allow an element of self-evaluation, participants were given five (5) developmental aspects with which to determine their growth. They were asked to think about (a) emotional growth (b) psychological growth (c) responsibility (d) accountability and (e) discipline. The most frequent responses are captured in the table below. 43 of 87 Table 4.5.2 Growth Aspects Growth Aspects Examples of growth Emotional Growth Learning to compartmentalize, conflict resolution Psychological Growth More accepting of leadership skills, approaches were realistic, learned to be more egalitarian Accountability Learning to deal with consequences of distraction, managing information, personal development Responsibility Developed more of a sense of community, more community-oriented, greater level of personal responsibility Discipline Time-management needs to be improved, working on self￾discipline over regimentation More specific elaboration on the responses indicated varying degrees of growth and impact of the programme elements on their development. For example: Emotional growth • While four (4) participants attributed their overall emotional and psychological growth to the general transition initiative, one stated emphatically that the mentorship programme had the most lasting impact on her development. • One participant credited the programme for her significant growth in the area of conflict resolution. • Another participant indicated that while there was some growth, it needed to be concretized, as more time was needed to develop the skills. • One participant stated that she had maintained her level of emotional growth, and that the programme had no significant impact on this. Of the 13 participants, 12 (93%) viewed the programme as having added to their emotional development. Psychological growth The perspectives on psychological growth revealed general acknowledgement of the positive impact of the programme. Participants noted the following: • The training from the programme allowed for more considerate behavior toward others, and a readiness to advocate for others. • One participant commented on the realistic approach and its impact on developing coping skills while being in a new environment • Another participant explained that she had learnt to function with restraint 44 of 87 • The sole male participant stated that while he would not necessarily attribute his psychological growth to the programme; his experience with the programme influenced him to be more accepting of his leadership abilities. Seventy-seven (77%) considered the programme to have influenced their growth positively. Accountability All participants acknowledged that accountability, defined as ‘taking ownership of one’s actions and the related consequences’ was an area in which they were growing. • One participant indicated that, because of personality training, she was engaged in more reflection and self-assessment. • Another elaborated on her growth experiences and stated that she had grown and was learning how to manage her personal development. All participants (100%) commented positively on their overall development in this aspect. Responsibility The aspect of responsibility was defined as ‘a sense of duty to self and others’. All participants commented on their growth in this area. • Two participants explained that they had maintained their sense of personal responsibility. One explained that she often took on tasks to help others if they needed help. Describing herself as the ‘mother of her group’ she stated that the programme had not added to her sense of responsibility. She credited the experiences of the programme to developing a more responsible awareness of community. • The other explained that the programme had not necessarily increased his sense of responsibility but had allowed him to maintain what was shown early in his development. • Another participant expressed her gratitude for the experiences through the programme which allowed her to develop a deep sense of service to others, and a desire to give back to community to others through acts of kindness. Of the 13 participants, 11 (85%) agreed that the programme had contributed to their improved sense of responsibility. Discipline In evaluating the aspect of discipline, participants were asked to distinguish between: acting on their own will (self-discipline), as opposed to doing what needed to be done because someone was watching, or they were told to do so (regimentation). The responses were varied, and while one indicated that she was always self-disciplined, others indicated that they were working on self-discipline. 45 of 87 • One participant commented that socialization and greater integration in different situations required more internal discipline. • Another expressed the need to do better with time-management and self￾discipline, particularly being in a higher education programme. Of the 13 participants, 12 (92%) indicated that the programme had influenced this aspect of their development positively. Concerns About Living in the Transitional Home Youth shared concerns about their experiences with living in the transitional housing. These included: Rigidity and inflexibility of leadership • Bureaucratic approach • Daunting and intimidating interaction • Leadership style causing undue stress • Invasion and breach of privacy (searching through personal belongings in the absence of the residents) • Leadership style is emotionally stressful and harmful to wellbeing • Leadership can be an issue primarily with communication styles • Communication between manager and residents can be difficult • Leadership sometimes immature and unwilling to listen, results in toxic and stressful environment • Unpleasant interactions because staff tended to project feelings on to residents, instead of dealing with the problem • Some members of staff in general, not knowing how to talk to persons with sensitivity • Method of interaction can be emotionally devastating and crushing • Caregivers need to know how to treat different people differently Unfairness and inconsistency • Contravening general values ... saying, insisting on one thing then doing the absolute opposite as exemplars • Unreasonable control over personal interactions or personal associations formed • Judging and dictating the interactions or relationships that should occur • Segregation among residents • Objection to exercise of rights to expedite situations – • Public embarrassment caused if residents go above someone in the chain of command to resolve an issue 46 of 87 • Personal biases imposed on objections or opinions of residents, • Threat of being blacklisted for being assertive or advocating for self and others • Conformity through state care experiences leads to conformity at this level Findings from the Electronic Survey Another phase of information-gathering was conducted with two cohorts of youth who had participated in the Entrepreneurship Programme. It was discovered that one of the participants in the Focus Group was also added to the WhatsApp groups. In this case, that person was excused from this activity. Table 4.5.3 provides details on the respondents current activities. Eight of the ten participants responded regarding their current level of activity Table 4.5.3. WhatsApp Participants’ Current Activities Gender Current Activity Institution M Unemployed/Unengaged N/A M Higher education UWI F Unemployed (Seeking) N/A F Out of school (programme interruption) VTC F Higher education UTECH F Unemployed (Seeking) N.A F Higher education UWI F Higher education UWI Ten respondents provided their answers via text or voice recordings. Of those referred for the survey, eight were actually living in the housing facilities in the corporate area. Respondents were asked: ‘What were the most helpful or memorable skills you learned in the entrepreneurial program?’ The skills reported included: communication, budgeting, time-management, finance, taking risks, patience building a business, starting a business, comparison shopping, writing a business plan, customer service, etiquette, talking to people more confidently, and registering a business. In relation to the question of skill mastery, respondents were asked: ‘What skills have you learned or mastered as a result of the entrepreneur program?’ They listed these skills: patience handling a business, risk-taking as a part of starting a business, money￾management, networking, comparison shopping, managing and running a business, writing a business plan, registering and managing a business, budgeting and planning, sales and finance, the do’s and don'ts of customer service, presentation of products, etiquette, and rapport building. One male respondent clarified that he had not 47 of 87 mastered a skill, but had learned how to improvise, specifically with repairing broken phones. In order to determine the respondents’ thoughts about areas of improvement to the program, they were asked: ‘What would you suggest could be different about any aspect of your training or preparation?’ Seventy (70%) of the respondents were satisfied with the programme as it was delivered. Those who had suggestions mentioned improving funds disbursement, extending the duration of the programme, and increasing face-to-face or individual sessions to support those who needed additional assistance. Summary In summary, a total of 23 youth who were formerly in State-care provided their opinions and perspectives about their experiences prior to and their preparation for transitioning into independent living. They were engaged through a Focus Group via Zoom, video conferencing and a WhatsApp survey. The feedback indicated highly favourable ratings of the effect of the life-skills and mentorship programmes. Skill areas such as budgeting and planning, starting and managing a business, money-management, partnering and networking were highlighted as being particularly beneficial. Personal growth areas, including conflict resolution, communication skills, appreciating differences and living as a community were also highlighted as important areas of development. However, there were some aspects of the TLP that were identified as requiring some attention, notably funds disbursement and the period of time allotted for the programme: youth believed that the TLP Project interventions should be implemented over a longer time frame. In addition, youth raised significant concerns about their interactions with the staff, inclusive of current facility managers and previous care-providers. The potential for youth who are exiting state-care may be overshadowed if these issues are not examined and addressed. For example, the youth felt that current staff tend to treat residents disrespectfully. Specific examples of these disagreeable staff actions include instances of the following: • Staff imposition of personal preferences for peer-to-peer associations; • Staff retaliation toward a resident if they act proactively to resolve an issue; • Staff invasion of the privacy of residents by conducting searches in the absence of residents; and • Staff communication style with residents, which contributes to a toxic environment. 48 of 87 It is worthwhile to note that residents from both housing facilities highlighted what they perceived to be challenges with staff interactions. Although there was some hesitation to discuss this matter, given their fears of repercussions, all respondents confirmed that their concerns are real. The youth were adamant that there is need for greater staff awareness that the approach used with children who are growing up should not be employed with adolescents and youth. The youth described those staff, whom they viewed as offending in these ways, as “immature and disrespectful”. Youth suggested that all staff, including current managers, needed to be trained in how to interact with children who are in and transitioning out of State-care. The following considerations are noteworthy: 1. One of the objectives of this evaluation activity was to have some comparison of youth in the housing complexes and those who have re-integrated with families. Given the relatively small sample of youth who were involved in discussions, there is definite need for a follow-up activity to ensure a greater range of perspectives and experiences is considered and used to inform subsequent steps of the evaluation of outcomes and impact. 2. The monitoring and evaluation plan noted the intent to match comparison groups (housed and not housed) for a two-year longitudinal study to assess impact and outcomes. In addition, a more substantive aspect of this activity would need to include a comparison of the exit-readiness profile with the current status of the youth who have gone through the programme. If possible, such an activity could be considered as a follow-up to this project. 3. The question which was asked to determine emotional and psychological development yielded positive responses from 80% of the respondents. In an expression of gratitude one participant commented that: ‘ ... it was good to know there are people out there who love them and see their potential. 4. Although we were unable to administer a formal evaluation of psycho-emotional status, it was important to determine their perspectives about their psychological and emotional growth as elements of their ability to transition and navigate the stresses and challenges related to the assumption of adult roles and responsibilities. 49 of 87 5. CONSTRUCTION OF TLP COMPLEXES 5.1 Background In its original conception, a transitional housing complex was to be developed for 40 female wards transitioning out of care. This was modified to three complexes (2 in Kingston and 1 in rural Jamaica); for 40 females, 40 males and 12 co-ed. In the end, two complexes were agreed for 40 females and 12 co-ed (both located in Kingston). According to the first (2014-2015) annual report, the first site that was earmarked for development became unavailable during the first quarter. The CPFSA then managed to secure another site in Mona, Kingston. However, the University Project Management Office (UMPO) indicated that this site was smaller than recommended for housing 40 occupants. A second site, located in Trelawny, was identified, and UMPO agreed that both locations could be used. A third location in St. Elizabeth or St. Ann was also identified to benefit 40 males. Ultimately the original site in Kingston (Lady Musgrave Road) was released to the project in addition to the site at Mona (Palmoral Avenue). The inability to secure the appropriate site at an early date contributed to a delay to the start of construction. On November 20, 2015, ground was broken for the first transitional living complex, located at 24 Lady Musgrave Road. Contracts with the design team were signed in March 2016. This team indicated that the budget was insufficient for covering the costs of the proposed building. USAID b. At this stage, boundary survey tests revealed covenant restrictions on the title that required amendment. CPFSA undertook to have the amendment rectified in the Supreme Court. In the third year, increases in construction costs and size of the building at Lady Musgrave resulted in a proposal to USAID that a number of activities be re-scaled in order to accommodate these. USAID approved “an increase the budget cap from US$500,000.00 to US$850,000.00 for two of the three sites, without increasing the overall project budget.” Project activities were re-organized to facilitate the additional amount required for construction: the scope of some activities was expanded while others were scaled down. Meanwhile, the project team issued advertisements of Requests for Proposals for Architectural and Engineering services at the second, Palmoral Avenue, site. The Annual Reports identified the challenges with changes to Scope of Works, securing timely permits and approvals, establishing a reliable sewer connection, escalating construction costs and slow response times from Government agencies. Other challenges which continued/manifested in Years 4 and 5 included missed timelines for completion of construction and delays in acquiring approval of a third site, which was designated as a facility for males. 50 of 87 In Year 6, the quarterly reports noted challenges with contractors completing repairs to defects involving plumbing and electrical works. Defects were eventually completed either by the contractor or sub-contractors as agreed by the main contractor so that project accounts could be closed. Furniture was procured for both sites separately from the construction contracts. Occupancy of both complexes commenced in November to December 2019. 5. 2 Feedback on the Challenges Presented Throughout the Construction Process One of the project team’s most frequently mentioned disappointments is that the living complex for boys was not realized, given problems with procuring a suitable site within the required time frame: “I'm saddened that the third facility couldn't be built for the males, as they are often a forgotten group in society. I'm also concerned that some youth were not able to benefit from the project due to some choices they have made, but you cannot force anyone to do something they are not ready to do." As noted, the analysis of project documentation points to substantial challenges with the construction process, which resulted in delays. The project team confirmed that the “consultants who were charged with leading construction activities did not perform as expected”. According to one respondent: “The failure to realize the male complex was a complicated issue. The original site was objected to, and there were tremendous delays in getting information for the second choice, such as titling, lot size and details, which pushed possible construction beyond the time frame of the project. There were problems with the agency—-perhaps there was insufficient buy-in from some of the management—-although kudos to the others who were outstanding.” 51 of 87 6. EFFECTIVENESS OF PROJECT MANAGEMENT, RELATIONSHIPS AND MANDATES How have relationships between partners helped or hindered the “delivery of outputs”? How can these relationships be improved? How have the mandates of each key stakeholder (CCDC, CPFSA, USAID) helped or hindered the accomplishment of the project objectives? How effective have the project’s monitoring, management and financial systems been? How could these be improved? The Mixed Outcomes of Financial Probity The TLP profited substantially from employing accounting procedures that ensured financial probity. However, the evidence suggests that the project would have benefited from greater flexibility in certain critical areas, such as approval for property for housing. The project outcomes demonstrate the importance of greater dialogue and cooperation between international development partners and local implementation agencies. Relationships Between the CPFSA and UWIOC Interviews with the teams from the University of the West Indies and those from CFPSA presented differing pictures of the process of project management and relationships between the key partners. One respondent from the UWIOC expressed what appears to be a fairly common frustration: “The research design could not be carried out as planned. The exist readiness profile should have been carried out in one wave, in the first year of the project. Instead, data collection was done over three years as it was difficult to identify participants. This also meant that all the interventions were not provided to the same group of youth beneficiaries. There is, therefore, no way to measure how the basket of interventions impacted a group of beneficiaries.” Another explained: “The responsiveness of CPFSA was not what I had hoped. While there was good camaraderie, and some excellent input by some people at times, there was sometimes a sense that the project was not of great importance to the CPFSA, that they were just anxious for the buildings to be finished so that they could get access and use them as they wished, and some of the staff were not up to the tasks assigned.” Others pertinent comments included: “CPFSA had too much to manage. The staff was handling multiple responsibilities. In the latter part, COVID-19 mandates impacted implementation of several activities.” “The project was seen as a partnership; however, as it took so long to receive information, it was necessary to question the partnership”. 52 of 87 “For construction, once project delays became protracted, consultants' attention shifted to other projects, further exacerbating the situation”. “Project activities were undermined because of miscommunication and politics”. “The movement of youth in and out of care and among different care programmes made it difficult to provide the interventions to collect data and track youth. There was no clear system to track youth once they aged out of care and, therefore, no longer on caseload.” “The delays in implementation led to shifts in project timelines. Delayed implementation is leading to slow rate of achieving output indicators Positive Unintended Effects of Managerial Challenges In many respects, these managerial challenges led to the development of innovative options: “We learnt how to work with these and develop another strategy”. However, they also qualify the impact of the capability building component. Respondents were asked: How would you qualify the level of impact that the project has had on CPFSA's capacity to manage projects such as these? The following is a selection of the responses: “The project helped streamline and standardize several activities, that will impact youth preparedness to leave care. Also, the case management system, the Standard Operating Procedure document and other capacity building support such as buses, laptops, tablets etc. will equip the CPFSA management better to prepare children for transition.” “There has been positive impact, as we have strengthened some internal systems. It is important to note that they have had a transitional living programme but it was more ad hoc and not structured. The TLPCSC project assisted with streamlining”. “There has been significant impact. The TLP was able to make a difference in the lives of young people who otherwise would not have a place to go, it helped to cement a programme of transition and create housing, which is a big factor for the poor in Jamaica”. Other respondents were more guarded: “It has had a medium level impact. Since the inception of the project, CPFSA recruited staff specific to the TLP CSC project as well as a project manager and has taken the effort to develop the Standard Operating Procedure to institutionalize the TLP programme.” 53 of 87 “The project was not managed by CPFSA, rather it was managed by the UWI. A statement on level of impact would be subjective. However, aspects of the project have been streamlined within CPFSA operations, and with the capacity building provided, the Agency should be able to implement some similar activities.” Now, in hindsight, the Project Team concludes that the following actions would have improved processes and outcomes: 1. A revision of some project targets and objectives, in order to ensure that the more feasible ones are addressed. 2. Proper scoping of the project. 3. Preparation of major stakeholders to improve understanding of the project concept. Narrow the target population a smaller number of youths in fewer homes in an effort to implement the intervention in a more controlled environment. 4. Better engagement and communication strategy with the care staff and other officers who engage directly with the youth. 5. Include, at an earlier stage, a Project Officer within CPFSA, with responsibility for managing the project at the CPFSA. The Transitional Office has now been established, and CPFSA has included budgetary provisions to sustain the interventions at different levels. For example, they will maintain the housing complexes and provide monthly allowance to residents. 6. Greater technical support for construction activities. 7. Less rigidity with certain reporting requirements (that is to USAID), which ended up confusing the project partners and significantly delaying reporting. 8. Greater flexibility in shifting when there was obvious difficulty in execution. The team agrees that the principal threats to project sustainability include: 1. Funding and human resources needed to carry out duties; 2. Organization and political will to keep the gains of the apartment complex, such that it does not become another children's home; 3. Budget limitations, especially for aftercare services for youth aged out of care. These could be mitigated through scholarships, adopt a child programmes and Government moving the age for ageing out from 18 to 21 years; 4. CPFSA not sufficiently tapping into existing social services, which could be mitigated by keeping an updated database of available services, for easy referral; 5. A decision to utilize the independent living facilities for other purposes; for example, for youth still in care rather than transitioning youth; 6. A change in senior management and new appointees not agreeing with the thrust of the independent living arrangements but deciding that 'care' will end once the youth exit. 7. Lack of Formal agreement for CPFSA to provide support for youth over 18 years. A Cabinet decision may be required to secure this. 54 of 87 7. CONCLUSIONS AND RECOMMENDATIONS The TLP succeeded in most components, although there were organization and management constraints that undermined some key aspects of project implementation. With respect to the Life Skills component, it is clear that the CPFSA’s capacity has been built; however, it is difficult to ascertain the actual reach and impact of the training since the monitoring and accountability mechanisms are inadequate. Further, the assumption that training would “trickle”from CPFSA trainers to caregivers does not necessarily obtain. The production of instructional guides is important for ensuring institutional knowledge. It may also be helpful to produce electronic copies of training materials and refresher courses, since these would remain even though staff may leave the institutions. The report has identified challenges with implementing the vocational skills and mentorship components; however, these have, in many respects, been used to devise constructive circumventing tactics and strategies. While the TLP project managed to construct two of three anticipated housing facilities, that component was plagued with delays and otherwise negatively affected by the inability to procure a suitable site within the required time frame: Against the background of these findings, the evaluation team concurs with Project Management that the following actions are required to ensure that the TLP produces substantive outcomes for the children/youth who are exiting State care. Project Management 1. There is need for a dedicated team within CPFSA whose explicit mandate is to manage independent living planning for children and youth who are exiting care. 2. It is critical to enhance monitoring and accountability mechanisms and processes in order to measure the quality of training received, how training is being implemented and the real consequences for children and youth. 3. It is important to build skills in strategic project management, including the definition and monitoring of qualitative/outcomes targets. Training for Sustainability 1. Provide ongoing training to strengthen and improve training undertaken as a part of the project. 2. Ensure that all staff within CPFSA are trained in Trauma Informed Care. This training should equip staff to deal with their own traumatic experiences as well as treat children and youth with greater compassion and respect. Training such as the Life Skills training of care staff requires ongoing reinforcement and updating if gains are to be sustained. The case management system needs to 55 of 87 be embedded in the work of the agency, and sufficient training for the persons who will input the records. 3. Dealing with discrimination: This involves “continuing sensitization on the issue of stigmatization of the youth because of their status as wards or former wards of the state”. 4. Increased caregiver awareness of the stages of growth and the skills required for managing relationships with children and youth across age groups. Ensure Equitable Access to TLP 1. It is important to ensure that all children have access to the TLP: This means appending a transitioning/aftercare plan to the care plan of each youth in care. 2. It is critical to pay careful attention to the youth who are least likely to transition successfully; that is, those less prone to participate in programs; for these, those not involved in higher education or in vocational skills. This cohort of youth, likely the most at risk, must also transition. Without innovative and effective approaches and strategies, this cohort can be relegated to unemployment and drawn into circles of crime and violence. Success with this category of youth will provide a good indicator of the effectiveness and impact of the TLP. Increase Resources: 1. The Government needs to ensure that the resources are available to CPFSA for them to be able to implement and improve the TLP. Further, resources should be allocated to the implementation of the SOP and the construction of additional facilities, including for males and children with disabilities. Finalize the Standard Operating Procedures 1. The Standard Operating Procedures needs to be in place and a policy document formulated to embed the operations. The SOP has been drafted and will be finalized with recommendations from this evaluation. 2. The key components in the SOP includes an educational plan for youth who want to pursue technical or tertiary education, life skills coaching as well as opportunities for internships and entrepreneurial training. 56 of 87 ANNEX 1: TERMS OF REFERENCE TRANSITIONAL LIVING PROGRAMME FOR CHILDREN IN STATE CARE PROJECT ________________________________________________________________________ The University of the West Indies, Open Campus Transitional Living Programme for Children in State Care Project CONSULTANCY TO CONDUCT AN EXTERNAL EVALUATION _____________________________________________ 1.1BACKGROUND Jamaica’s first long-term strategic plan for national development was tabled in the country’s Parliament in May 2009. The plan, ‘Vision 2030 Jamaica,’ its goals and actions are meant to lead to the attainment of development status of a first world nation by 2030. It envisions, ‘Jamaica, the place of choice to live, work, raise families and do business.’ The first of Vision 2030’s four goals speaks to a nation where ‘Jamaicans are empowered to achieve their fullest potential.’ This challenges the changing value of the children and youth, who between birth and 19 years comprised 34% of the island’s population in 20101 . Therefore the achievement of the goals must include a stronger approach to building the capacity of the country to prepare children in residential care for independent living, and in doing so, help improve their chances of achieving their full potential. 1.2 In 2012, the Planning Institute of Jamaica (PIOJ) in collaboration with the Child Protection and Family Services Agency (CPFSA) formerly the Child Development Agency (the government body with responsibility for children in need of care and protection) commissioned a study 2 to assess the adequacy of the child protection system of the Jamaican State, its ability to prepare its wards for independent living and make recommendations to assist their transition. The study revealed that CPFSA had made some efforts to facilitate successful transition for the children leaving care, however the agency’s services ended at age 18 and extended services were not consistent across the Government and private facilities. 1.3 As of 2012, there were 50 Children’s Homes and 9 Places of Safety in Jamaica. CPFSA stated that there were 5,336 children in State Care, comprising 2,640 males and 2,696 females. There were also other children who were in private institutions who were not wards of the state. At that time 62 percent (3,309) of the total children in care were living in a family-based environment (L.I.F.E.) while 43 percent (2,027) were living in the residential sector. Robinson-Hall (2012) conducted a review of the Child Protection and Family Services Agency’s (CPFSA) capacity to prepare children living in residential facilities for Independent Living3 . In the review, the children 1 Economic and Social Survey of Jamaica 2010 2 Robinson-Hall, R. (2012). Review of State capacity to prepare wards of the State for independent living. Unpublished Planning Institute of Jamaica 3 Ibid 48 who participated were at the threshold of leaving state care and had been exposed to the CPFSA’s current independent living programme. The data collected 57 of 87 suggested that the children were not confident about their readiness to exit care and did not identify with the current independent living programme, with respect to preparedness to enter the adult world. Over 51.5% (both males and females) of those interviewed were desirous of continuing with their education and training after leaving care as one of their priorities. They were also concerned with how they were going to manage in the outside world as they had gotten used to a highly structured way of living and now had to be solely responsible for themselves. Among their priorities listed in the study were (i) having somewhere to live, (ii) employment, (iii) start-up funds for living expenses, and (iv) opportunities for continuing education and training. 1.4 While some youth will have successful independent living outcomes, others will not due to risk factors such as disability, poverty, inadequate schooling, and a history of trauma, loss, and maltreatment. The ones who do not have a successful transition typically have individual circumstances such as: they do not have families with whom reintegration can be planned, or they had to leave their community of origin due to a dangerous situation. This is supported by relevant international literature that showed that young adults – both males and females – in transitioning from child protection systems to adulthood showed notably poorer outcomes in all domains of functioning in the adult world when compared with their peers in their respective societies. 1.5 The risk domains identified in the literature were: unstable living arrangements, unemployment, sustained deficits in education and training, poor mental and physical health, susceptibility to using drugs and alcohol, conflicts with the law, early parenthood and fragile social relationships. Given the risks for poorer outcomes, it is likely that a number of youth transitioning out of care in Jamaica will need extended services that can enable them to participate in attaining the desired national outcomes that are outlined in Vision 2030. 1.6 Against this background, the University of the West Indies Open Campus (The UWIOC), in partnership with Child Protection and Family Services Agency (CPFSA) formerly Child Development Agency (CDA) and United States Agency for International Development (USAID) sought to implement the Transitional Living Programme for Children in State Care project, over six years. 2.0 PROJECT DESCRIPTION On August 27, 2014, U.S. Agency for International Development (USAID), mission to Jamaica (Mission) approved the Transitional Living Programme for Children in State Care project, AID-532- A-14-00001 for the creation of safe neighbourhood spaces and services for protecting and addressing the social needs and concerns of marginalized populations. The cooperative agreement AID-532-A14- 00001 was initially approved for $1,388,521. United States Dollars in grant funds to The University of the West Indies Open Campus. The scope was increased in February 2016 together with additional grant funds. 58 of 87 2.1 Project Goal: The goals of the project are to i) improve the transition to independent living for children leaving State care at 18 years and ii) reduce the risk factors associated with low skills development, inadequate life skills and poor self-image. This is to be accomplished through the development of an exit-readiness programme involving life skills and job skills training, to address employability, lifestyle choices, health and the family and the 49 environment, as well as, the creation of safe and appropriate transitional living facilities to accommodate up to 92 children who have nowhere to go upon leaving State care, and create a public-private partnership to improve employment opportunities. 2.2 The specific objectives of the project are: i) To assess the level of preparedness of youth 17 years and older in State care facilities to transition out of care. ii) To develop a standard operating procedure guide for preparing children aged 14 to 18 for independent living. iii) To equip 900 children in State care with the necessary life skills and knowledge to reduce the effects of risk factors that impact safety and human rights. iv) To equip 40 Trainers and 500 care staff with the techniques in life skills development training. v) To establish safe transitional housing complexes for 46 females and 46 males ages 18-21 for a maximum of 2 years. vi) Increase the opportunity for jobs/self-employment for 500 young adults through workforce develop and job placement. vii) To determine the outcomes of 92 young adults provided with independent living facilities. 2.3 Outcomes/Results The anticipated outcomes/results at the end of the project are: i) Needs assessment to inform the development of the exit-readiness programme. ii) A Standard Operating Procedure for preparing youth in State care for independent living. iii) 40 CPFSA Officers trained as Life Skills Trainers. Iv) 500 caregivers equipped by Life Skills Trainers with techniques in life skills coaching. v) 900 youth in State care exposed to life skills coaching. vi) 500 youth in State care trained in a vocational skills training programme. vii) 46 females and 46 males, ages 18-21 years accommodated in safe and appropriate transitional housing complexes for a maximum of two years and provided with mentoring. viii) Provision of employment for 92 youth, through a network of public and private companies. ix) 400 youth provided with “starter-kits” to facilitate exit preparedness. x) A longitudinal study of the outcomes of 92 young adults who are housed in the independent living facilities compared with 92 matched young adults not housed in the facilities. 59 of 87 xi) 45 youth in receipt of scholarships for professional development courses from The University of the West Indies Open Campus. xii) Strengthening the leadership and management capability of the CPFSA to successfully implement and sustain project activities after project implementation through the provision of an improved Child Case Management System, trauma￾informed mental health training and procurement of buses for transportation needs and a 27-seater bus, retrofitted to support the Mobile Mental Health Programme. xiii) A Memorandum of Understanding between The University of the West Indies and the Child Protection and Family Services Agency (CPFSA) for the sustainability of the programme. 50 3.0 THE CONSULTANCY 3.1 The Caribbean Child Development Centre (CCDC), The UWI Open Campus wishes to utilize part of the funding provided by the USAID to contract an External Evaluator under a Contract for Services to undertake a process evaluation of activities associated with the execution of the project. 3.2 The results of this evaluation are primarily intended for internal use by CPFSA to inform future improvements to the operationalization of the Transitional Living Programme. The results and findings will also be used by CCDC to improve the design of intervention projects, and will be shared with USAID to inform country programme development. Relevant results and lessons learned will be shared with other stakeholders in academia and civil society. 3.3 Objectives of the Evaluation: A. Undertake an evaluation to assess the project’s: i. Relevance – the extent to which the goals and objectives were consistent with beneficiaries’ needs and priorities; ii. Effectiveness – the extent to which the targeted project objectives were achieved (or are expected to be achieved); iii. Efficiency – how efficiently were the processes carried out to achieve the results (time, other resources); iv. Sustainability – the extent to which the project activities are likely to continue after the project; v. Outcome – inferences on the likely long-term effects produced by the project (directly, indirectly, intended and unintended). The evaluation should assess the above in relation to: o What institutional changes /outputs/achievements have taken place? o How have these changes /outputs/achievements been brought about? In relation to this, the evaluation should consider the following sub-questions: What stated objectives were achieved? What have been the unintended and unexpected outputs of the project activities? Who has benefited (boys, girls, adolescents, men, women), in what ways? Did any intended beneficiaries not benefit? Have any changes been achieved in relation to 60 of 87 practice/knowledge of decision makers, line staff, etc.? Did any of the outputs achieved relate to broader national and international policies, conventions, targets etc.? How was the project implemented? How well did the logic framework for the project support the implementation of the project? Was it well defined, where were the gaps, how could it be adapted and improved for the future? What were the most effective approaches used by the project to achieve the goals and objectives? What worked, what didn’t, and why? What overall lessons have been learned by the project team? 5.1How have relationships between partners helped or hindered the ‘delivery of outputs”? How can these relationships be improved? How have the mandates of each key stakeholder (CCDC, CPFSA, USAID) helped or hindered the accomplishment of the project objectives. How effective have the project’s monitoring, management and financial systems been? How could these be improved? How readily is project learning transferable to local/regional/international knowledge development? B. Provide recommendations for improvement, sustainability and replicability. The evaluation should be strongly focused on recommendations for improvement relating to the central questions outlined above, and in particular – how activities could be adapted to better meet the needs of the target beneficiaries in a sustainable way. Learning from this evaluation should: Inform plans for the operationalization and expansion of the Transitional Living Programme within the CPFSA. This learning will also aid in the development of policy for the transitioning of youth out of care, with necessary budgetary provisions. Inform the design, implementation and monitoring of other similar projects by The UWI and USAID. 3.4 Methodology The evaluation should consist of: A. A review of project documentation: Review of material related to the project. This could include, but is not restricted to: various project reports, workplans, communications amongst CCDC, CPFSA and USAID, the original project proposal document and relevant modifications, relevant project data stored in databases at The UWI and CPFSA, data collection tool(s), training materials etc. This information will be provided to the evaluator by CCDC and CPFSA. Any other relevant statistics and secondary sources should also be reviewed by the evaluator. B. The development of an evaluation approach and data collection tools/methods to include: Detailed time line and work plan; Outline of any proposed changes to the scope of the evaluation; Key interview questions; Proposed sampling framework; List of stakeholders to be consulted; and Development of associated data collection and evaluation tools 61 of 87 C. The Evaluator is welcome to use mixed methods, and particularly participatory evaluation methods that are fun and engaging for the youth and stakeholders participating. 3.5 Sample size: Based on a target of 500 beneficiaries over 6 years, a sampling framework is to be proposed by the consultant. The proposed approach should be submitted to CCDC for approval prior to the fieldwork. 52 3.6 Stakeholders to be consulted: The consultant should plan to include the following groups of stakeholders in the evaluation: Direct project beneficiaries in CPFSA (potentially disaggregated by those who participated in different parts of the project (as indicated in the project summary above) CCDC implementing staff Relevant CPFSA staff Relevant USAID staff Volunteer Mentors Wider stakeholders – for example members of the Project Steering Committee, Liaisons at HEART Trust/NTA and other training institutions, Junior Achievement Jamaica, companies involved in Transition to Work programme, etc. 3 3.7 Participate in an inception meeting with CCDC, CPFSA and USAID. This should take place immediately after contract signing. 3.8 Field visits: Field visits should be made to the Transitional Housing Complexes, where some beneficiaries are housed in order to conduct interviews or focus groups. 3.9 Provide a debriefing to CCDC, CPFSA and USAID: using a draft set of results and PowerPoint presentation to summarise preliminary findings and recommendations. This should be shortly after the end of the field work. 3.10 Submit a draft evaluation report: that corresponds to the requirements outlined below in the ‘Deliverables’ section. 3.11 Submit a final evaluation report: incorporating any relevant feedback from CCDC, CPFSA and USAID on the draft report. 3.12 Deliverables Deliverables should include the following: An evaluation work plan, including: planned timeline, methodology /approach, planned stakeholders to be consulted and sampling framework, data collection tools if relevant, qualitative and quantitative protocols for data collection and analysis; any suggested improvements to existing evaluation scope, as outlined in this document Presentation of preliminary findings Draft evaluation report that meets the requirements outlined below Final evaluation report comprising i) One (1) printed bound copy of the clean report, with appendices, ii) One (1) electronic file in .pdf format of the clean report, One (1) electronic file of the final report with track changes as updated from the Draft report. One (1) electronic file of the clean (final) qualitative and quantitative data collected. Requirements for Final Report – 62 of 87 The final evaluation report should be written in English, jargon free, clear, concise and simply written - The report should include an executive summary (max 2 pages) - Technical information should be included in appendices only. - Analysis of project achievements should always be backed up with relevant data, with reference to the data source. 5.3 - Recommendations should be specific and include relevant details for how they might be implemented. - The report should be so organized that information relevant to the key stakeholders (CCDC, USAID and CPFSA) can be easily extracted. The structure of the report should cover the following: - Executive summary - Project background - Main findings relating to the evaluation questions and including detail of any unintended outcomes that are resulting from project activities - Recommendations for future action – In addition, the final report should contain at least the following annexes: o Terms of Reference for final evaluation o Details of evaluation methodology, o Milestone schedule o List of meetings/consultations held o List of persons interviewed o Summary of field visits o List of documents reviewed o Any other relevant material, including data collection tools 63 of 87 ANNEX 2 EVALUATION AREAS AND SUPPLEMENTARY LISTING OF QUESTIONS RFP Listing of Questions Supplementary Listing - IASR Respondents Relevance To what extent were the project goals and objectives were consistent with beneficiaries’ needs and priorities How pertinent is the project to national development priorities? How pertinent is the project to realising international obligations? CCDC, CPFSA What institutional changes /outputs/achievements have taken place? How have these changes/outputs/achievements been brought about? What stated objectives were achieved? What have been the unintended and unexpected outputs of the project activities? Who has benefited (boys, girls, adolescents, men, women), in what ways? Did any of the outputs achieved relate to broader national and international policies, conventions, targets etc.? How was the project implemented? How well did the logic framework for the project support the implementation of the project? Was it well defined, where were the gaps, how could it be adapted and improved for the future? What were the most effective approaches used by the project to achieve the goals and objectives? What worked, what didn’t, and why? How have relationships between partners helped or hindered the “delivery of outputs”? How can these relationships be improved? How have the mandates of each key stakeholder (CCDC, CPFSA, USAID) helped or hindered the accomplishment of the project objectives. Did the project objectives meet the needs of different sub-categories of youth, both at the level of outputs and of substantive outcomes? To what extent have the risk factors to successful transition been addressed for differing categories of youth? How was progress measured? Were the indicators used comprehensive? What categories of youth have benefited (a) the most and (b) the least? Why, and with what consequences? How were youth selected for employment opportunities? How has employment provision affected the lives, broadly, and transition experience, specifically, of recipients? Have there been unintended outcomes for non-recipients? How has this influenced their transition? How has the project mitigated unintended negative outcomes? Are there differential outcomes for youth housed versus not housed in independent living facilities? What are the long term programming implications? Did the project objectives meet the assessed needs of the key ‘implementing’ stakeholders, both at the level of outputs and of outcomes? How was capacity development defined and measured throughout the project? To what extent has capacity been built? What is the evidence? Where have there been gains? Where are there gaps? How was the trauma-informed mental health training developed and delivered? To what extent did it add to the CPFSA’s knowledge and skill base? CCDC, CPFSA, Regional directors, monitoring officers, facility managers, care givers, sample of children 64 of 87 How effective have the project’s monitoring, management and financial systems been? How could these be improved? Was project implementation staged in ways that produced optimal benefits for all stakeholders? Efficiency Did the project make the best use of human and financial resources? Did the project make the best use of time? CCDC, CPFSA, Regional directors, monitoring officers, facility managers, care givers Sustainability To what extent does the quality and scope of training facilitate sustainable? What measures have been put in place to ensure that training is lodged within the institution’s memory such that the knowledge gained is sustained even where persons depart their respective organisations? Are there sufficient funds to facilitate sustainability? If so, over what time period?? Has the relevant ministry ‘bought in’ to this project such that it intends to include the necessary funds in its annual budget? Has this ministry indicated interest in the evaluation for this reason? How is the relationship between partners likely influence sustainability? What mechanisms are in place to protect against adverse outcomes? CCDC, CPFSA, Regional directors, monitoring officers, facility managers, care givers Outcomes What overall lessons have been learnt by the project team? What are the likely long term effects for each group of stakeholders? How readily is project learning transferable to local/regional/international knowledge development? Did outcomes for youth differ, depending on variables, such as location, staff complement, sex, disability, learning disorders, youth at varying levels of risk etc.? How did the project provide for expected and unexpected challenges from different sub-groups? CCDC, CPFSA, Regional directors, monitoring officers, facility managers, care givers 65 of 87 ANNEX 3 SPECIFIC FOCUS GROUP AND INTERVIEW QUESTIONS - CPFSA TRAINERS AND CAREGIVERS RELEVANCE Prior to the TLP: - How would you describe the children’s life skills status? - How would you describe the children’s vocational skills status? - What proportion and category of the children experienced trauma? How was this displayed? - How did the TLP affect the children’s/youth development? EFFECTIVENESS - How were/are you involved in the TLP? - Have you received training on how to enhance the children’s life skills capacities? - How effective has this training been? What have you learnt? - FOR TRAINERS: - How effectively was training transmitted? - How were learning outcomes measured? - To what extent did the caregivers meet the learning objectives? What evidence exists to support your response? - What were the strengths and weakness of the training content and delivery methods? - What additional project inputs would you recommend to strengthen training programme and process? - FOR CAREGIVERS: Has your practice, as caregiver, changed since the introduction of the TLP? If so, in what ways (positive and negative)? IMPACT - What changes (in vocational skills, life skills, psychosocial development) have you noticed since the children participated in TLP? Have outcomes differed across different cohorts of children? If so, how? - Have you noticed differences between the TLP participants and children who have not had access to any of the project components (vocational skills, life skills, psychosocial development? - Of the children who are still participating in the TLP, have life skills and vocational skills consistently improved or have there been periods of decline in performance and growth? If there have been periods of decline, what factors account for this? - Are there additional ways in which you think that TLP could assist children? What improvements would you recommend? - Are there additional ways in which you think that you could assist the children’s skills development and emotional preparedness for exit? - What other interventions—in addition to PLED— are required to prepare the children for exit from state care? 66 of 87 ANNEX 4 QUESTIONNARE FOR CPFSA TRAINERS OF TRAINERS 67 of 87 68 of 87 69 of 87 70 of 87 71 of 87 72 of 87 73 of 87 74 of 87 75 of 87 ANNEX 5 QUESTIONNARE FOR PROJECT IMPLEMENTATION TEAM 76 of 87 77 of 87 78 of 87 ANNEX 6 DOCUMENTS REVIEWED Avasant Foundation, Summary, Youth Digital Employment Initiative MultiCare Youth Foundation, Transitional Living Programme for Children in State Care Project, 2021 University of the West Indies Open Campus, Transitional Living Programme for Children in State Care, Project Agreement and Modifications University of the West Indies Open Campus, Transitional Living Programme for Children in State Care Workplan. Plan, January 10, 2019 University of the West Indies Open Campus, Transitional Living Programme for Children in State Care Workplan. Plan, January 21, 2020 University of the West Indies Open Campus, Transitional Living Programme for Children in State Care Workplan. October 5, 2020 University of the West Indies Open Campus, Transitional Living Programme for Children in State Care Workplan, November 2017 University of the West Indies Open Campus, Transitional Living Programme for Children in State Care Feedback forms used for Life-skills training and Mental Health training University of the West Indies Open Campus, Transitional Living Programme for Children in State Care Project ,Performance report , October, 2017 – September, 2018 University of the West Indies Open Campus, Transitional Living Programme for Children in State Care Project ,Performance report , August 27, 2014 to September 30, 2015 University of the West Indies Open Campus, Transitional Living Programme for Children in State Care Project, Performance report, October 1, 2015 to September 30, 2016 University of the West Indies Open Campus, Transitional Living Programme for Children in State Care Project, Performance report, October, 2016 – September, 2017 University of the West Indies Open Campus, Transitional Living Programme for Children in State Care Project, Performance report, October 1, 2018 – September 30, 2019 University of the West Indies Open Campus, Transitional Living Programme for Children in State Care Project, Performance report, April 1, 2020 –June 30, 2020 79 of 87 University of the West Indies Open Campus, Transitional Living Programme for Children in State Care Project, Performance report, April 1, 2020 –June 30, 2020 University of the West Indies Open Campus, Transitional Living Programme for Children in State Care Project, Performance report, January 1, 2020 –March 31, 2020 University of the West Indies Open Campus, Transitional Living Programme for Children in State Care Project, Performance report, October 1, 2019 – December 31, 2019 80 of 87 ANNEX 7 LIST OF RESPONDENTS PROJECT IMPLEMENTATION TEAM CONTACTED FOR EMAIL SUBMISSIONS Contacts made on December 8, 2020, December 18, 2020 NAME ORGANIZATION Anaokar, Priya CCDC Anglin, Claudette USAID Kenneth Williams USAID Buchanan, Cerita SWTC Budhi, Audrey CDA Campbell, Marva CCDC Dillion, Nicole CPFSA Dixon, Rochelle CPFSA Douglas, Newton CPFSA Gage Grey, Rosalee CPFSA Gooden, Marlene CPFSA Meeks Gardner, Julie CCDC Minott, Ceceile CCDC Morrison, Annmarie UWI Finance Office Perkins, Andrien CCDC Thomas, Kathi-Ann CCDC Thomas, Joan CCDC Thompson, Jessica CCDC Reid Gleasia CPFSA Essue Patrick CPFSA Gooden Kaylyn CPFSA Brown Boyd Janeil SWTRC Thomas, Christopher O Projects Office Morrison, Annmarie Finance Office 81 of 87 Key Stakeholder Interviewees Ms. Mary Gibson-Clarke, Chair, Steering Committee Professor Julie Meeks Gardner, CCDC Managers of all Care Facilities Directors of Skill Training Institutions: Children First, 24/7, Cobbla Ms. Alicia Glasgow, MultiCare Foundation Nicole Dillion, Training Officer, CPFSA Mrs Audrey Budhi, Children and Family Programmes, CPFSA Focus Groups with Project Implementation Team -September 14, 2020 -November 25, 2020 -March 18, 2021 Interviews with Managers of Care Facilities Interviews were conducted throughout the month of December, 2020 Focus Groups with Youth Focus groups were conducted in December 2020 Interviews with Representatives From Skills Training Institutions Attempts to make contact with CTIs commenced on December 18, 2020. Professional Development Institution and Girls Town, Kettering, Knockalva and Village Academy were the first to be contacted. Mrs. Salmon-Russell from Excelsior Community College (24/7 Jamaica) was also contacted and an interview scheduled. Mr Russell facilitated an interview for Knockalava. Contact was made with Children's First and an interview date secured. An interview was secured with Ms. Angela Bailey of Cobbla Skills Training Centre for December 29,2020. On December 22, 23 and 28, 2020, attempts were again made to contact all CTIs listed. These attempts were unsuccessful as the phones either rang without answer (due to Christmas break), liaison persons were no longer at the institutes and/or promises were made to return telephone calls but this did not occur. With respect to HEART Trust, the CPFSA suggested a contact person; however, that individual was no longer employed to HEART. HEART provided another contact, Mrs Viva Hines-Gray, out on vacation leave. IASR was then advised that no one else could facilitate an interview because none of the available officers knew about the Transitional Living Programme. Numerous attempts were made to contact Mrs Gray on the mobile number provided; however, the phone consistently rang without an answer. 82 of 87 Trainer of Trainers Electronic Survey CPFSA List of Trainers of Trainers. Questionnaires were sent to all email addresses. Reminders were sent, electronically, on three occasions. Persons were also contacted by telephone. Name of Participant - Phase 1 1. Sanchia Ellis 2. Candice Williams-James 3. Sharon Bradford 4. Sheryl McKenzie 5. Wayne Grant 6. Nicola Dacres 7. Leasia Bromfield 8. Carlette Wallace 9. Carlyn Stewart 10.Kediene Shea Hood 11.Dotlyn Bailey 12.Racquel Remy 13.Dania Walker 14.Stacy Brady 15.Ardine Bartley 16.Georgette Muschette 17.Taneka Cassanova-Durrant Name of Participant - Phase 2 1. Shauna Samuels-Clarke 2. Lisa Hill 3. Camille Crooks Wright 4. Michael Clarke 5. Sellean Barracks 6. Morvetia Hunter 7. Carmen Mullings 8. Leonard Leslie 9. Michael Murray 10.Simone Smith Case 11.Julia Wesley 12.Donna Fuller 13.Francine Rhoomes 14.Karen Rowe 15.Nadine Anderson 83 of 87 South East Region 1. Kayeon Bernard 2. D’Andra Bingham 3. Jesanya Scale Menzies 4. Bobbieanne Johnson 5. Shanice Clarke 6. Lavern Belnavis 7. Chane Campbell 8. Stacy Ann Lindsay 9. Paulette Bannerbie 10. Judine Thompson 11. Shanice Risden 12. Sassah-Gaye McPherson 13. Joan Myrie 14. Fay Bryan 15. Angella Gibbons Suragh 16. Georgia Edwards 17. Sandra Brown 18. Kimesha Mundle 19. Raecine Bailey 20. Ericka Gilbert Hope 21. Remona DAcosta 22. Clive Mohalland 23. Joy Morrison 24. Judine Webb Brown 25. Camille Lewin LAmbie 26. Sheryl Hamilton 27. Angilla Gayle 28. Johanna Brown 29. Colette Gooden 30. Alexia Royal South East Region 1. Carol Cooper Rodriques – Western Region 1. Oral Hylton 2. Elvis Farquharson 3. Andrea Patterson 84 of 87 4. Kerry-Ann King 5. Adria Brown 6. Emery Baker 7. Shiona McDonald 8. Nicolette Prout 9. Olivia Kelly-Daye 10.Sasha McIntosh Dawkins 11.Marjon McPherson 12.Shauna Samuels Clarke 13.Avril Warren-Tracey 14.Camille Crooks Wright 15.Sanna Shae Simms (SER) – 16.Kerry Ann Swaby Dennis 17.Sherieka Burton 18.Shavel Desouza 19.Nordia fletcher 20.Candice Williams James (Trainer) 21.Sanchia Ellis (Trainer) 22.Lisa hill (Trainer) 23.Nordalee Stokes (Trainer) 24.Gailann Wilson Dixon (Trainer) 25.Sharon Bradford (Trainer) Western Region 1. Shereika Burton 2. Emery Baker 3. Shavel Desouza 4. Sasha McIntosh Dawkins 5. Oral Hylton 6. Adria Brown 7. Michael Wedderburn 8. Kerry-Ann King 9. Cnthia o’Connor 10.Nordia Fletcher 11.Elvis Farquharson 12.Olivia Kelly-Daye 13.Shanikee Pinnock 14.Kerry Ann Swaby Dennis 15.Andrea Patterson 16.Stephanie Taylor 17.Marjon McPherson 18.Shelly-Ann Johnson 19.Peter Brown 85 of 87 20.Sharon Bradford (Trainer) 21.Nordalee Stokes (Trainer) 22.Camille Crooks Wright (Trainer) 23.Candice Williams James (Trainer) Northeast Region 1. Nediesha Grey Russell 2. Kareen Kelly 3. Shannae Walsh 4. Tanesha Dilworth 5. Patricia Christie 6. Renee lindo 7. Sasha-Gay Brown 8. Valdean Harris 9. Kevin McKenzie 10.Kedar Sharpe 11.Rayboyd Johnson 12.Kediene hood 13.Raquel Mclean remy – 14.Shakira genus Corrodus 15.Patrice McGann 16.Dania Walker (Trainer) 17.Simone Smith Case (Trainer) 18.Carlette Wallace (Trainer) Northeast Region 1. Andrew Blake 2. Patricia Christie 3. Renee Lindo 4. Tanesha Dilworth 5. Dania Walker 6. Carlette Wallace 7. Nediesha Grey Russell 8. Hylgay Roache Campbell 9. Kevin McKenzie 10.Patrice McGann 11.Kedar Sharpe 12.Valdean Harris (SER) 13.Rayboyd Johnson 14.Vinette Jackson Beckford 15.Kareen Kelly 86 of 87 16.Kediene Hood (Trainer) 17.Raquel McLean Remy 18.Shakira Genus-Corrodus 19.Simone Smith Case Southern Region 1. Shereene Palmer Reid 2. Antonette Brooks Blake 3. Althea Knight 4. Kalena Pusey 5. Lorraine Beaton 6. Kimeahia Brown Reid 7. Kay-Ann Williams 8. Kemolyn Frost 9. Dacia Lawrence 10. Shelly Bent Parchment 11. Kimberly Ann Thomas 12. Michael Coward 13. Lacian Colquhoun 14. Genel Taylor 15. Shaneek Ramsaroup 16. Melesia Smile 17. Princton Herron 18. Andrea James Howe 19. Rohan Burrell Southern Region 1. Althea Knight 2. Antonette Brooks Blake 3. Rohan Burrell 4. Kemolyn Frost 5. Kimberly-Ann Thomas 6. Lorraine Beaton 7. Kimeahia Brown Reid 8. Kay-Ann Wilson 9. Princton Herron 10.Shaneek Ramsaroup 11.Melesia Smile 12.Andrea James Howe 13.Genel Taylor 14.Lacian Colquhoun 15.Michael Coward 87 of 87 16.Donna Fuller Henry 17.Dacia Lawrence 18.Shelly Bent Parchment Caregivers Electronic Survey Questionnaires were sent to the respective care facilities. Reminders were sent, electronically, on three occasions. Persons were also contacted by telephone. Eventually, online interviews were conducted, using the questions outlined in the survey.