External End-of-Program Evaluation Report 7/8/2014 Prepared by: Phung Thi Yen Submitted to: Viet Nam Assistance for the Handicapped (VNAH) and Hanoi Employment Introduction Center (HEIC) Interview with a person with hearing impairment Person with disability participated in the inclusive job fair in Tu Liem district Person with disability at HEIC Low vision group has succeeded in getting jobs External End-of-Program Evaluation Report Vocational Rehabilitation Program 2014 2 This evaluation was commissioned by the Vietnam Assistance for the Handicapped (VNAH). The opinions contained in this report, however, are of the evaluators only and do not necessarily reflect the views of United States Agency for International Development (USAID) or VNAH. External End-of-Program Evaluation Report Vocational Rehabilitation Program 2014 3 Acknowledgement This report has resulted from joint efforts and contributions by a number of participants. We sincerely thank United States Agency for International Development (USAID) and Vietnam Assistance for the Handicapped (VNAH), Hanoi Employment Introduction Center (HEIC), and Vocational Rehabilitation Demonstration Unit (VR Unit) for persons with disabilities at HEIC. We would like to thank Vocational Rehabilitation Unit for their support and favorable conditions created to the Evaluation Team. We would also like to thank the project staff and student volunteers participating in the collection of information. Their support and cooperation have significantly contributed to the outcome of this final evaluation report. The Evaluation Team would like to express our deep gratitude to the project Deputy Chief of Party (DCOP) - Mr. Bob Horvath, VNAH’s program officers - Ms. Pham Thi Lien and Mr. Nguyen Quoc Dung, and HEIC’s project officers for their valuable time providing information and comments to finalize this report, and of course all stakeholders and persons with disabilities who contributed to this report. We are looking forward to getting feedback from all interested readers! The Evaluation Team Team Leader: Phung Thi Yen External End-of-Program Evaluation Report Vocational Rehabilitation Program 2014 4 LIST OF ACRONYMS DOLISA Department of Labor, Invalids and Social Affairs NGOs Non-Governmental Organizations MOLISA Ministry of Labors, Invalids and Social Affairs M&E Monitoring and Evaluation HEIC Hanoi Employment Introduction Center VR Vocational Rehabilitation VNAH Viet Nam Assistance for the Handicapped External End-of-Program Evaluation Report Vocational Rehabilitation Program 2014 5 Contents Acknowledgement ........................................................................................................................... 3 EXECUTIVE SUMMARY ..................................................................................................................... 6 PART I - EXECUTIVE INTRODUCTION .............................................................................................. 8 1.1. Background...................................................................................................................... 8 1.2. Evaluation Objectives ..................................................................................................... 8 1.3. Approach and Evaluation methodology ......................................................................... 8 Evaluation Approach ................................................................................................................... 8 Information collection methodologies ....................................................................................... 8 1.4. Sampling size and survey areas ...................................................................................... 9 1.5. Limitations ..................................................................................................................... 10 PART II - SUMMARY OF OUTCOMES ............................................................................................ 10 PART III - PROGRAM EVALUATION ............................................................................................... 12 3.1. Relevance ........................................................................................................................... 12 3.2. Efficiency, effectiveness and unexpected impacts ........................................................... 14 Efficiency ............................................................................................................................... 14 Effectiveness ......................................................................................................................... 15 Unexpected impacts ............................................................................................................. 18 3.3. Participation of stakeholders and their satisfaction ........................................................ 19 Participation .......................................................................................................................... 19 Satisfaction ............................................................................................................................ 20 3.4. Strengths and points for improvement ............................................................................. 21 Strengths ............................................................................................................................... 21 Some points for improvement ............................................................................................. 23 PART IV - LESSONS LEARNED AND RECOMMENDATIONS ........................................................... 24 4.1. Lessons learned .................................................................................................................. 24 4.2. Recommendations for similar initiatives in the future .................................................... 25 PART V – CONCLUSION ................................................................................................................. 26 REFERENCES .................................................................................................................................. 28 Annex 1 - FRAMEWORK FOR EVALUATION .................................................................................. 29 Annex 2 – QUANTITATIVE SURVEY QUESTIONNAIRE .................................................................. 32 External End-of-Program Evaluation Report Vocational Rehabilitation Program 2014 6 EXECUTIVE SUMMARY The External End-of-Program Evaluation on the Vocational Rehabilitation Program for Persons with Disabilities (VR) took place from June to July, 2014 in Hanoi, Vietnam. There were a total of 134 key informants and targeted beneficiaries who actively participated in this evaluation, in which women counted for 51% (68/134). The main objectives of the external evaluation were to assess the extent to which the project achieved its stated objectives. The evaluation' scopes included the relevance, effectiveness, efficiency, unexpected impacts, participation and levels of satisfaction as well as lessons learned of the achieved results regarding knowledge, planning and management. Relevance: The demonstration model was fully consistent with the local context, strategies and policies. The VR model was relevant to the needs of employees with disabilities and their families. According to statistics by Hanoi Disabled People Association (Hanoi DP), about 75% of working-age persons with disabilities have needs of employment but are unemployed. The vocational rehabilitation demonstration model was established to help solve the employment issues for persons with disabilities and demonstrate the effectiveness of using a vocational rehabilitation system/process. The program design conformed to the implementing partner’s capacity. Efficiency, effectiveness and unexpected impacts: The program’s achievements during the last two and a half years were reliable evidence about the social and economic values of this model. The evaluation results showed that, if we pay approximately 1,500,000 VND ($72) to help finding jobs for a person with disabilities, he/she would generate an average annual income of 28 times over the cost of VR’s services1 payment to seek jobs, and about 1.8 times over the social allowance and family costs to support the person. This figure does not include other greater values that workers with disabilities can contribute to society such as personal tax, or social insurance. The final evaluation had sufficient evidence to confirm that the vocational rehabilitation demonstration model was highly efficient as it managed to mobilize large resources from its service providers’ network. 102 persons with disabilities were referred to and received free vocational trainings; and at least 140 persons with disabilities were involved in soft-skills and peer counseling skills training workshops. The model has also developed a network of stakeholders including employers, NGOs, vocational training and counseling centers, physical rehabilitation centers, and peer counseling groups involved in the provision of vocational rehabilitation services. The improvement of stakeholders’ capacity, changes in the awareness about persons with disabilities by employers, communities and persons with disabilities themselves were also good indicators reflecting the demonstration model’s effectiveness. Two unexpected positive impacts that the program achieved included: (1) the vocational rehabilitation model has helped improve the status and enhance the social responsibilities of the HEIC; and (2) the vocational rehabilitation model has created positive effects in the 1 Source: VR unit of HEIC; VR cost/case = total program budget (include management cost)/total service users. VR cost = USD 31.000/432 persons with disability registered to the VR program = 71,75$ each. External End-of-Program Evaluation Report Vocational Rehabilitation Program 2014 7 disability community. Nearly 100% (94/95) of persons with disabilities responding to the quantitative survey confirmed that they would recommend their peers to come to the program. Participation and levels of satisfaction: The program managed to mobilize the stakeholders’ participation. VNAH created a structure where the implementing partner was empowered during the program implementation, which enabled the HEIC’s proactiveness and creativity, making the model in conformity with reality during operation. More than two thirds of employers and persons with disabilities were very satisfied with the vocational rehabilitation services. Both employers and persons with disabilities had good impressions about the attitudes of the VR Unit staff. The counselors’ energetic attitudes and what they did to help persons with disabilities getting jobs were highly appreciated by their families. Strengths and weaknesses: The model had three outstanding strengths. The first strength was the economic value; the second was the humanitarian value; and the third, related to its sustainability, was the possibility to be easily replicated in other provinces as the result of the program documentation efforts. Thanks to these strengths, the Hanoi government has allocated funding for the HEIC to implement employment supports for persons with disabilities. Thus, the vocational rehabilitation model would be continued in Hanoi even when the project is ended. There were still some points for improvement, including communication to the right target groups, updating of database, and more effective approaches to improve the accessibility to vocational rehabilitation services to persons with hearing and visual impairments. The main learned lessons included mobilization of resources, strengthening empowerment to the implementing partners, and involving the participation of all stakeholders, including the direct beneficiaries. Great attention was paid to the development of the service providers’ network and work samples during the implementation of the VR model. The assignment of sufficient staff also helped improve the efficiency of the demonstration program. The appropriate documentation and filing of database were also helpful for the model evaluation and sharing of information. It is recommended to provide follow up supports to the established VR service for persons with disabilities. If the VR model is replicated and developed in other provinces and cities, the following activities should be focused on: - Conduct a survey on the labor market before the execution of the model; - Pay attention to and allocate budget for communication activities based on the assessment of communication needs of the target groups; - Focus on the development of the service providers network (employers, social organizations/associations, and peer networks); - Further strengthen the capacity for implementing partners and service users; - Develop a software for database management and regularly update that database; - Provide appropriate equipment for consulting unit (such as supportive devices) to ensure that different disability groups can access vocational rehabilitation services effectively. External End-of-Program Evaluation Report Vocational Rehabilitation Program 2014 8 PART I - EXECUTIVE INTRODUCTION 1.1. Background The Project’s Vocational Rehabilitation Program is aimed at providing vocational rehabilitation services for persons with disabilities, which is being implemented at the Hanoi Employment Introduction Centre (HEIC) under a cooperative agreement between VNAH and HEIC. The vocational rehabilitation services are based on an assessment of the unique needs of the person with a disability, and are provided by specialized service providers who can respond to those unique needs in assisting a person to maintain or retain employment and to achieve his or her highest level of functioning and independence. The vocational rehabilitation services are provided by the HEIC vocational rehabilitation staff or counselors. The steps to successful employment begins with an application for services from the person with a disability, leading to an initial intake interview to explore the person’s interests and background, which can be considered for the next steps in the vocational rehabilitation process including assessment, eligibility determination and planning. All these steps are conducted with specific forms, allowing the individuals’ data and information to be collected. 1.2. Evaluation Objectives The final evaluation of the vocational rehabilitation demonstration model included the four following specific objectives: 1. Assess the relevance and appropriateness of the program design and approach; 2. Assess the efficiencies, effectiveness and any other unintended effects of the program; 3. Assess the beneficiaries' and partners' participation, ownership and satisfaction of the program; 4. Identify areas of strengths and weaknesses of the program and provide recommendations for succeeding phases or similar interventions. 1.3. Approach and Evaluation methodology Evaluation Approach This evaluation was conducted in a participatory manner. Representatives of the direct implementing partner were involved, supporting the preparation of the information collection, arrangement for interviews with stakeholders and direct beneficiaries, and providing recommendations and comments for the completion of the evaluation report. The sensitive issues, such as gender and participation, were taken into consideration in the evaluation process, including the evaluation design, sample selection, field visits for discussions, conclusion and recommendations. Information collection methodologies The methods of information collection and analysis applied by the consultant are illustrated as follows: External End-of-Program Evaluation Report Vocational Rehabilitation Program 2014 9 Literature review: The consultant undertook literature review of the program’s available documents including the Project Proposal, -annual reports, program work plans, IEC materials, handbooks and other evaluation reports conducted during the program evaluation. Especially, the consultant worked closely with the HEIC to analyze the model’s database recorded from the beginning of program to the time of evaluation. Quantitative evaluation: A structured questionnaire was used to interview about 60% of direct beneficiaries, equally N=95. They were workers with disabilities receiving employment introductions. Qualitative methods: In-depth interviews, focus group discussions, and case studies were applied with two target groups including service providers (the direct implementing partner and stakeholders) and service users (employers employing persons with disabilities and employees with disabilities). 26 in-depth interviews, two focus group discussion and four case studies were conducted with the two target groups. Evaluation feedback meeting: After the field data collection, a meeting was organized with the participation of VNAH and HEIC, where main findings were reported by the consultant. Comments and suggestions were contributed by the participants. 1.4. Sampling size and survey areas Sampling size and characteristics of evaluation samples Both target groups, service providers and service users, were interviewed for the evaluation. Total number of sample size was 134 (N=134), 51% of whom were female. Among the sample size, 75% were persons with mobility disabilities, 16.8% were those with hearing impairment, and 6.3 % were those with visual impairments and others. Table 1 - Sampling size of the final evaluation Interview methods Number of participants Total Male Female In-depth interviews 14 12 26 Focus group discussions 3 6 9 Case studies 2 2 4 Questionnaire 47 48 95 Total 66 68 134 Percentage 49 51 100% External End-of-Program Evaluation Report Vocational Rehabilitation Program 2014 10 1.5. Limitations One of the problems encountered during the evaluation was the rate of refusal (15%) to participate in research, or inability to contact respondents. After participating in the program and getting jobs thanks to employment introductions by the VR Unit, some employees with disabilities changed their contact details (emails, phone numbers and addresses) or changed jobs. Especially, the Evaluation Team had to contact families or direct supervisors of respondents with hearing impairment when they changed their addresses, resulting in the extension of the data collection time, and affecting the evaluation progress. That is why the number of focus group discussions was lower than planned. To meet the evaluation targets, the research team leader applied flexible ways to access the respondents by making individual in￾depth interviews instead of focus group discussions, or distant interviews (via phone calls, emails or Skype) with respondents living too far away (abroad, or out of Hanoi city). At least 10 respondents joined distant interviews. The quantitative survey was conducted by social work student volunteers who were trained on interviewing methods by using the questionnaire. However, as the student volunteer team was inexperienced in checking survey questionnaires, the sample size did not match the estimated samples (expected sample size was N=100, but actual sample achieved was N=93). To meet the sample size, the research team leader had to conduct further interviews directly with two respondents to increase the sample size up to 95. Therefore, the achieved quantitative samples (N = 95) covered 60% (95/159) of persons with disabilities obtaining jobs for more 3 months. PART II - SUMMARY OF OUTCOMES Most of the planned activities stated in the cooperation agreement between the HEIC and VNAH have been done with high appreciation by the stakeholders and service users. The Table 2 below summarizes the outcomes and brief information on the status of activities. Table 2 - Summary of program activities and outcomes Planned activities Outcomes Status 1.Develop plans for the establishment and implementation of the VR model at HEIC - Completed - Timely implemented 2.Develop and complete necessary procedures for HEIC to implement and integrate program activities into their services - The vocational rehabilitation process was documented into the handbook on the procedures of vocational rehabilitation services for persons with disabilities; - HEIC was committed to integrate employment introductions for persons with disabilities into their existing services; - Forms of vocational rehabilitation were completed (5 forms) - The handbook on the procedures of vocational rehabilitation services for persons with disabilities was highly valued by the stakeholders; - In reality, these forms should be flexibly used; External End-of-Program Evaluation Report Vocational Rehabilitation Program 2014 11 Planned activities Outcomes Status 3.Provide trainings for HEIC and VTCs staff, teachers, parents and other stakeholders - The counselors and stakeholders involved in the implementation of the VR model were trained on some basic skills (project management, how to work with persons with disabilities, etc.) - No trainings for families with disabilities were organized (because of limited budget) 4.Develop the referral network and cooperation skills trainings - The SP network was established and developed (including 5 physical rehabilitation centers, 10 vocational training centers, four soft skills training centers, and 18 employers who were willing to employ workers with disabilities) - The linkage between the VR model and physical rehabilitation centers was still weak and the connections with persons getting disabilities by work accidents has not been set up. 5.Provide vocational rehabilitation services to persons with disabilities (vocational counseling, development of vocational work plan, vocational training, on-the-job coaching, seeking jobs, and follow-up supports when employed) - 432 persons with disabilities registered to the VR program (about 50% was female); - 346 persons with disabilities received employment introductions - 159 got jobs and retained those jobs for over 3 months - 3 persons with disabilities received on-the-job trainings - 102 received referrals to free vocational trainings - The number of beneficiaries getting successful employment was over the target of 150 - >70% of persons with disabilities were very satisfied with the VR model’s services 6.Cooperate with BREC to organize job fairs and forums in Hanoi - At least 5 inclusive job fairs at HEIC location and a community-based one were organized - >50% of respondents with disabilities preferred inclusive job fairs than those exclusively for persons with disabilities only as they liked being inclusive 7.Establish a work assessment center - a work practice room (or assessment center) was established - 3 work samples were installed and visual pictures of certain jobs performed by persons with or without disabilities were produced - The utilization of the work assessment center by persons with disabilities was low; - The work samples provided by the project were not regularly External End-of-Program Evaluation Report Vocational Rehabilitation Program 2014 12 Planned activities Outcomes Status for counseling purposes. utilized. Pictures and videos of common jobs have been developed by the HEIC for more practical utilization. 8.Monitoring and Evaluation - Regularly conducted Source: Summary from the project’s annual report and final evaluation results Both service providers and service users highly appreciate the social significance of Activity 2 and Activity 5. The documentation of the VR model under Activity 2 was of great significance for its replication possibility in other provinces and cities in the future. Under Activity 5, up to 159 persons with disabilities obtained jobs. Over 70% of users were satisfied with the VR services. Persons with mobility disabilities were the majority of direct beneficiaries from the program. Based on random surveys on 95 persons with disabilities receiving employment introductions, persons with mobility disabilities accounted for 75%; 16.8% were those with hearing impairment; 6.3% were with visual impairment; and 2% were others (Source: Quantitative survey - final evaluation). Some activities were planned but have not been conducted due to the program’s limited budget, including: - Trainings to raise awareness of parents/families of persons with disabilities(under Activity 3); - Connections to persons with disabilities, which were caused by job accidents, at physical rehabilitation centers. PART III - PROGRAM EVALUATION 3.1. Relevance The VR model implemented in Hanoi was fully relevant to the local context, strategies and policies. Vietnam Government has had clear policies on disabilities. Specifically, the 1992 Constitution (amended in 2001) and the National Law on Disability enacted in 2010 ensured that all civil rights including state support are provided to persons with disabilities. Vietnam has also been strongly involved in international and regional commitments on the rights of persons with disabilities. Regarding laws on labor and employment for employees with disabilities, the Labor Code (Section 4, Chapter XI) has provisions on employment policies for specific persons including laborers with disabilities. It can be said that policies and legal documents on employment for persons with disabilities have been systematically issued by the governmental authorities. However, in reality, there has been a lack of specific models of effective employment resolution for workers with disabilities. The VR model implemented in Hanoi by the HEIC since 2012 has provided a specific model showing that employment issues for this specific target group have been on the right track. External End-of-Program Evaluation Report Vocational Rehabilitation Program 2014 13 The VR model was relevant to the needs of employment by persons with disabilities and their families. The demonstration model was established when persons with disabilities had many needs of employment. According to Hanoi Disabled People Association (Hanoi DP), among 5.3 million persons with disabilities in the country, 60% were at working age, only about 25% of which were involved in income generation activities, or having paid jobs. This also means that 75% of working-aged persons with disabilities had employment needs but were unemployed (source: http://dphanoi.org.vn/index). Thus, the VR model is helping to resolve the difficult employment issues for persons with disabilities and the local government. It was also found that the program scale was compatible with the capacity of the implementing partner (i.e. the HEIC). It was reported by the HEIC representative that they had never implemented a similar program before. There had been few persons with disabilities coming to the center to find jobs. There had been no differences in services provided to persons with or without disabilities. However, since the demonstration of the VR model, the program activities have been more professional thanks to the technical support by VNAH. HEIC’s Counselors were trained on knowledge and skills working with persons with disabilities. ”HEIC used to provide job introductions to persons with disabilities when requested but the way we did was unprofessional. We have been more professional since involving in the demonstration of the VR model.” Source: Mr. Nguyen Toan Phong, HEIC’s Director However, the VR Unit’s counselors thought that the vocational rehabilitation forms (forms MS 01, MS 02, MS 03, MS 04, and MS 05 in the VR handbook) should be flexibly used or continue to be improved for better appropriateness with reality. Table 3 - Comments on VR forms VR Forms Comments MS 01 & MS 02 Was generally good; special pictures of persons with disabilities should be provided. MS 03 – Assessment of the persons with disabilities Was not regularly used. Information about physical functions and abilities was of low reliability. The persons with disabilities themselves sometimes did not know how much they could push/pull. MS 04 – Individual Work Plan This form was regularly used. However, the part on physical rehabilitation was not utilized as no physical rehabilitation￾related activities were provided by the VR Unit. Part III – Work practice: Rarely or hardly used for persons with disabilities receiving referrals to vocational trainings. Service Cost to persons with disabilities part was not used: Real expenditure of VR per each case can estimate when closing program2 . 2 VR cost for each case = total program budget (include management cost)/total service users External End-of-Program Evaluation Report Vocational Rehabilitation Program 2014 14 About 25.3% (24/95) of respondents with disabilities said that it was difficult for them to complete the program forms. 75% of them complained that “the forms were long and difficult to understand”; the other felt difficult as they were illiterate. The VR Unit’s counselor recommended that the model’s instruments should be flexibly and creatively utilized. Specifically, some vocational assessment instruments such as the Legos and Perdue Pegboard were not applied to all persons with disabilities coming to find jobs. The sensitivity of the counselor when interviewing the job seekers was very important. During the counseling process between job seekers and counselors, the counselors could provide appropriate advice for job seekers thanks to their feelings and assessment of the job seekers’ voice, gestures, or dress/appearance, etc. The HEIC’s Counseling Department staff created pictures and videos of jobs to show persons with disabilities when providing employment counseling. 3.2. Efficiency, effectiveness and unexpected impacts Efficiency When comparing the inputs and outputs, the evaluation team believes that the project has achieved the highest efficiency. The budget of the project was also accurately and effectively utilized thanks to regular monitoring on each activity by both parties (HEIC and VNAH). According to the Decree 28/2012/ND-CP and Hanoi People’s Committee’s policies, social allowance provided to a person with disabilities in Hanoi was 350.000 VND per month3 . Apart from social allowance, each laborer with disabilities received an average support of 1,628,000 VND (81$) by their families4 . Total average cost for an employee with disabilities was 1,978,000 VND per month or 23,736,000 VND per year. It should be noted that this was the minimum cost for basic living needs and excluded healthcare costs for them. The evaluation result showed that, over 70% of persons with disability had stable jobs for over three months with an average income of 3,500,000 VND/person/month. If a person with disabilities has a stable job for a year, his/her average income would be 42,000,000 VND/person/year (see chart 1). Currently, the average cost of VR services (including management cost, staff allowance, equipment, etc.) for a worker with disability till obtaining jobs was 1,500,000 VND/case (about $72US/case)5 . It means that, if we are willing to pay 1,500,000 VND (or $72) to find jobs for a person with disabilities with the monthly average salary of 3,500,000 VND, he/she would generate an average annual income of 28 times over the cost that VR services pay to seek the job (1,500,000 VND compared with 42,000,000vnd), and nearly 1.8 times of social allowance plus family costs to support the person (23,736,000 VND compared with 42,000,000 VND). This figure does not include other greater values that laborers with disabilities can contribute 3 The Decree 28/2012/ND-CP on the implemantation of some articles of the Law on disabilities. Source: http://www.moj.gov.vn/vbpq/Lists 4 Source: Program database, average statistic of 461 job seekers, support from family for a person is 1,628,000 VND /month (81 US$) 5 VR cost/case = total program budget (include management cost)/total service users; VR cost = USD 31,000/432 persons with disabilities registered to the VR program = 71,75 $/case; Source: VR unit of HEIC. External End-of-Program Evaluation Report Vocational Rehabilitation Program 2014 15 to society such as personal tax, or social insurance. Importantly, when persons with disabilities have jobs, the burden on families, communities and society will be largely reduced. Obviously, the VR model is a reliable evidence of economic efficiency when the investment of one dong (or dollar) on employment resolution for a specific target group will bring back much bigger profits. Unfortunately, within the scope of the final evaluation, it was difficult for the evaluation team to find statistics on social insurances and taxes contributed by persons with disabilities. Therefore, the evaluation team could not give an exact figure of the actual costs and contributions to the society by persons with disabilities. However, the following parts will provide valuable qualitative evidences on the social significance of the VR model for persons with disabilities. Although there were many programs supporting persons with disabilities in Hanoi, there was no duplication of assistance for beneficiaries in this program. All job seekers would be participated in vocational trainings or soft skills trainings by organizations under the VR network such as NGOs, or private and/or public organizations if they needed. Training costs were not part of program budget thanks to effective connections between HEIC and network service providers. Effectiveness The VR models achieve high efficiency for being able to mobilize huge resources from stakeholders and from members of the VR network. The evaluation team did not have specific data that was mobilized from stakeholders. Their contribution sometimes was not money but human resources or practice opportunities for persons with disabilities. As reported in the 1st quarterly report (2014) by the VR Unit, by the end of the first quarter of 2014, 102 persons with disabilities had received free vocational trainings. Expenses for these courses were mobilized from the local government funding and directly paid by HEIC, employers or NGOs. Details are as follows: - 2 cooking classes (30 participants each) conducted by HEIC trainers External End-of-Program Evaluation Report Vocational Rehabilitation Program 2014 16 - 1 office computing class (30 participants) by Gia Lam district’s Vocational Orientation and Training Center - 2 persons learned to do the laundry at Sofitel Plaza Hotel - 2 persons learned to do art nails at Kim Nhung Aesthetic Center - 6 persons got training on internet job seeking skills by Viet Net – ICT & Microsoft - 2 persons attended the bartenders class by REACH The development of the network of stakeholders involved in the provision of vocational rehabilitation services helped enhance the VR model’s effectiveness. VNAH and HEIC have formed a network of service providers (vocational training institutions, soft skills training centers, etc.) to support persons with disabilities. The network involved the participation of stakeholders (local government, employers, NGOs, and DPOs). Notably, in less than 3 years, a network of employers willing to employ persons with disabilities was established. More importantly, the service providers’ network would be maintained even when the program ends. The peer counselors group can be considered as an important link in the network of employment service providers for persons with disabilities. At least 140 persons were involved in peer counseling and soft skills trainings. The peer counselors would share their employment seeking knowledge and experiences, job retention and communication skills to unemployed persons with disabilities. Communication between peers would result in positive effects which sometimes are not easily measured by numbers or figures. The smooth coordination of stakeholders during the program implementation also helped improve the model’s effectiveness. It was confirmed by VNAH and the VR Unit Team that the project was on progress. Information was always timely exchanged and communicated among the stakeholders. A VNAH’s project officer was based at HEIC location, which was necessary to make regular and direct communications between VNAH and the implementing partner. The program’s achievements were regularly updated and reported to DOLISA /MOLISA, Hanoi DPO by the HEIC’s Employment Counseling Department. The VR model helped improve the implementing partner’s capacity. The capacity improvement of program stakeholders was a good indicator reflecting the program’s effectiveness. It was confirmed by the HEIC representative that, previously (before 2012), employment introductions to persons with disabilities had been provided at HEIC but it was unprofessional and the proportion of persons with disabilities coming to the HEIC to find jobs was small. Especially, seeking employment for persons with disabilities was not considered as an important task. Since the demonstration of the VR model, it has been well aware by the HEIC that it was an important task with positive social significance. The HEIC staff involved in the model received trainings on knowledge and skills on how to work with persons with disabilities. After two and a half years of demonstration, the process of working with persons with disabilities has been thoroughly understood by the HEIC and they can be confident to develop and maintain the model in the future even if no financial support from the program is provided. Regarding capacity building, the VR model has helped improve the awareness about persons with disabilities for both community and employers receiving persons with disabilities. Though the program has not communicated with families of persons with disabilities during the experimental phase, families’ perceptions of those benefiting from the program have been changed. External End-of-Program Evaluation Report Vocational Rehabilitation Program 2014 17 “…In a working environment with both persons with and without disabilities, those without disabilities feel happier themselves...” - Source: N. V. H. – Community Ltd. Company “T. was a good boy. He just stayed at home after school. We did not want him to travel a lot because we were afraid that he might fall with such weak legs. However, he has been very sturdy since he got a job. It seems that having a job opened his awareness and social interaction. For this reason, he has been able to live independently…” (Source: N.X.D., father of N.N.T.) However, the VR model still had some shortcomings affecting overall effectiveness of the program. The staffing structure was not equivalent with the workload and changes of staff during the implementation affected the model’s implementation quality. Some of those involved in the program from the beginning and receiving trainings on basic knowledge and skills to implement the program were transferred after a year. Unfortunately, additional trainings were not provided, making the new in-charge persons encounter many challenges. As the staffing structure was not equivalent with the workload, the program’s components were not properly focused. For example, when the counselors focused on the development of the employer network, the peer counselor network was not. Due to the lack of staff, coaching to the beneficiaries after getting jobs (during the first 3 months) was not done well; this directly affected job retention rate. According to the database provided by the HEIC’s Employment Counseling Department, 9% of employees with disabilities lost jobs after 3 months. The percentage of those retaining a job for 3 months and above was 76.7% (see chart 2). It is believed that the program would be entirely capable of enhancing the job retention rate for the workers with disabilities if the support to them in the first three months was improved. “…It is easy for workers with disabilities to quit a job in the first 3 months, especially those who never worked before. To increase the job retention rate, from our experience, it is important to provide support (coaching) in the first 3 months. Generally, persons with disabilities are often of inferiority complex and sensitivity. If the working environment is difficult with any discriminations or conflicts with their co-workers, or is distant from home, they easily become discouraged and quit. Thus, they need to be prepared before and during the beginning months External End-of-Program Evaluation Report Vocational Rehabilitation Program 2014 18 after employment. Those with no work experience should be given opportunities to work as interns in companies/offices to learn the minimum skills at work environments. Then come to plans to support after getting jobs..." Source: N.L.A., female, Action Center for the Development of Community (ACDC). The work assessment center was a component of the model - which we believe that both HEIC and VNAH spent a lot of effort with high expectations for its performance. However, during the past time, the work assessment center was not fully utilized. The underlying cause was because the work samples equipped for the work assessment center were simple and inappropriate, resulting in low frequency of utilization. The work samples included an industrial sewing machine, an internet-connected computer, helmet assembling samples, and some vocational assessment instruments (a Lego set, the Perdue Pegboard, vocational assessment and the pre-screening set). The actual work samples such as sewing machines and helmets will help job seekers with practical experience. However, not all job seekers but only applicants to the garment industry, or helmet companies should practice with a sewing machine or a helmet. Thankfully, the HEIC’s Employment Counseling Department actively and creatively sought new work samples. For example, pictures and videos of working places and jobs would replace the machines. The counselors used pictures of certain jobs performed by persons with or without disabilities to show job seekers during counseling. The outstanding advantage of the visual samples was that it was easy for both counselors and job seekers to imagine or visualize the job and work environment; the second advantage was its low cost as they could be used repeatedly. This initiative, which HEIC is planning to develop and apply in the future, is highly valued. Unexpected impacts It was found through the final evaluation that the program had two unexpected positive impacts. Firstly, the VR model has helped improve the HEIC’s status and social responsibilities. It was admitted by the HEIC representative that, since engaged in the provision of employment services to persons with disabilities, its reputation had been improved among the employers network, and under the observation by the local authorities (DOLISA) and the community. Based on the demonstration achievements, the HEIC was planning for the maintenance and development of the model in the future. Importantly, it has been considered as a part of their responsibilities. Thus, the HEIC would have the opportunity to assert its prestige and its role in the future, which was admitted as an added value that "we did not anticipate before getting involved in the demonstration of the model" (source: N.T.P., HEIC’s representative). Secondly, the VR model has created positive effects in the disability community. Workers benefiting from the program have also referred the services to their friends/peers, which were confirmed by nearly 100% (94/95) of respondents in the final evaluation. We think this was very important for persons with hearing impairment, who faced the most challenges to find jobs because of their communication difficulties. However, they were well￾known for their unity; when one got a job, the information would be actively shared with others. Thus, if the model is maintained in the future, it would be the opportunity for persons with hearing impairment to get jobs and overcome their challenges. External End-of-Program Evaluation Report Vocational Rehabilitation Program 2014 19 Positively influenced by the VR model, after benefiting from the program, some persons with disabilities having good qualifications and opportunities aspired to make contributions back to the community. This is also a good indicator showing the positive impact of the VR model. The following story of N.T.D is an example. Wishes of a person with disabilities N.T.D, a person with mobility disabilities, participated in the vocational rehabilitation model in 2013. D. accidentally accessed to the model after unsuccessful efforts to find a job via the Internet. After applying to the program, D. was introduced to work as an accountant for Chula company, which is now a member of BREC, with a monthly salary of 8,000,000 VND. D. was very happy with his job at Chula and its friendly working environment. After working there for 3 months, D. gained a leadership award scholarship in Japan. D. thinks that the VR model is of great significance for unfortunate persons like him. There are a lot of unemployed persons with disabilities, who live dependently on their families. D. expects that the model would be extended so that more opportunities would be created for students with disabilities. While studying in Japan, D. actively approached the Nippon Foundation of Japan to submit a project similar to the VR model demonstrated at the HEIC. D.’s ideas were highly evaluated by the donor. He hopes that, when returning home, he would be able to find reliable partners to implement his new project "vocational rehabilitation for student with disabilities". D said, “maybe I'll come back to the HEIC or the Vietnam Students’ Association to discuss on the cooperation plans.” 3.3. Participation of stakeholders and their satisfaction Participation The program managed to mobilize the intensive participation of stakeholders. The stakeholders were consulted on the satisfaction of their participation in the program. They were informed (with information about the program), and got involved in discussions and collaboration to organize inclusive job fairs. The HEIC cooperated with Hanoi DP and its sub￾associations at district levels to invite its members to participate in inclusive job fairs or attend vocational training courses. VNAH created a mechanism where the implementing partner was empowered, making them (i.e. the HEIC) active and creative in implementing the VR model in conformance with reality. As discussed above, visual work samples (pictures and video clips) were evidence of the program’s implementing partner’s creativity. All monthly, quarterly and annual work plans and cost estimates were actively developed by the HEIC and approved by VNAH. The model has also enhanced the participation of its direct beneficiaries. The peer counselors’ network was a way to mobilize the participation of persons with disabilities when they already got jobs. Through consultation, persons with disabilities having jobs expressed their External End-of-Program Evaluation Report Vocational Rehabilitation Program 2014 20 desires to get involved in the peer counseling network without any financial support from the program. Satisfaction More than 70 percent of persons with disabilities and employers were very satisfied with general services provided by the VR model. The employers were satisfied with the quality of workers referred by the HEIC’s VR Unit, while the persons with disabilities were satisfied with the jobs introduced. 75% of the respondents said that the jobs introduced were suitable for their health conditions; 64% said the jobs were suitable for their commuting. Levels of satisfaction on the suitability of the jobs with their expertise and work experience were lower, 40% and 32.6% respectively. Over 50% of the job seekers were satisfied with the jobs introduced by the VR Unit. In particular, both employers and persons with disabilities had good impressions with the attitude of VR Unit staff. Both employed and unemployed persons with disabilities talked favorably about the HEIC’s counselors. “…We are very pleased to work with Mr. D. As we mentioned our hotel’s recruitment criteria, they sent the right persons to us as expected. We did not have to spend a lot of time shortlisting the candidates.” Source: A. D., HR Manager – Fortuna Hotel “…They did not only introduce the candidates but also made phone calls to ask about the persons’ work performance. It proves that they are very concerned about us and the employees, and we don’t feel annoyed at all…” Source: T. T. N. Q. – Director - My Xinh Aesthetics Enthusiastic attitudes of the counselors and what they did to help persons with disabilities get jobs were highly appreciated by their families. "... My son talked about her ( Ms. O. – the HEIC counselor) so much. He said that she was a good person. I was really surprised by her enthusiasm. She did not only help my family but also many other disabilities. If I know other person with disability who is looking for jobs, I will recommend them meeting her... " Source: N. X. D., Thai Binh province "... At first, I was wondering by myself the question why? Why could he/she do so well with persons with disabilities? My younger brother has hearing impairment, so it was difficult for him to find a good job. We felt very happy when people at HEIC introduced the job at a hotel. At first, External End-of-Program Evaluation Report Vocational Rehabilitation Program 2014 21 we thought that we would have to pay a lot of money, but they said it was free. We do not know what to say but thanks….” Source: N. T.H. – N.V.H.’s sister, hearing impairment 3.4. Strengths and points for improvement Strengths Firstly, the VR model has brought a certain economic value not only for families of persons with disabilities but also for the society. Being undeniable of the positive economic and social values of the VR model, Hanoi city government has decided to allocate budget for the HEIC to maintain and develop the VR model in the future. Specifically, in fiscal year 2014, Hanoi has funded at least three vocational training courses and four inclusive job fairs this year with a funding of about 120 million VND (including communication costs). In particular, if the persons with disabilities from suburban areas come to inclusive job fairs in groups of district DPOs, transportation support (by bus) would be supported by the HEIC. This has shown that the VR model was a reliable evidence base to help change the local government’s funding mechanism. It should also be noticed that the program implementing partners were very active in advocacy. Secondly, the VR model was of humanitarian significance. The humanitarian significance of the VR model was also highly valued by the stakeholders. Service users reported that the model has contributed to the improvement of their life and their families. Eventually, the model has changed their lives. T.’s story below is typical of a person with disability getting rid of employment abuse thanks to the access to the vocational rehabilitation services. External End-of-Program Evaluation Report Vocational Rehabilitation Program 2014 22 Getting rid of employment abuse T. was born in a poor family in Thanh Hoa province. Though having graduated from the University of Technology and Business in Hanoi for 2 years, he could not find a suitable job. Being unemployed for two years, he received a lot of complaints from his family, felt him as a burden. In 2013, T. applied for the vocational rehabilitation model but could not find a suitable job. While unemployed, he was asked to move to Thai Nguyen province to work with a monthly salary of 2 million. He worked as vendor from 7am to 11pm. Every morning, the employer (the ruler) took him and some other disabilities to a location, from which they had to walk to every corner throughout the city, even wandering to district markets. Each day, they had to wander for about 50 - 60 km. It was such a challenge for a person with mobility disabilities, but there was no way for him to escape. The employer controlled all the money, and apparently cut off all contacts with the employee's families and relatives. Each night, T. came back to the rented room with weary limbs and could not do anything but fell asleep in fatigue. Six persons were put into a 7-m2 room hired by the employer and changed every two months. All the rented rooms were simply places to sleep with no minimum living facilities. He found that he was abused and exploited, but did not dare to say for being afraid of being hit. He repeatedly witnessed the other persons with disabilities being beaten by the ruler. One time, lucky, he got in touch with the VR model’s counselors. After numerous attempts by the counselors, he came back to Hanoi. Supported by the VR services, he was admitted to work for a foreign company, where could learn English and computing. He was very satisfied with his current job. We met T. at Donkey Bakery Company. He said: "now, I cannot imagine that I am working with foreigners. Every time thinking the days working as a street vendor, I just cry. I cry out of fear, I cry because now I have a good job. When going to bed every night, I am very happy when thinking about tomorrow. I sleep better, gain weight and love this life ... " Over half of responding persons with disabilities thought that the VR model was the bridge between employers and laborers with disabilities or vice versa. The model helped them get jobs and thereby better integrate into the community. Only very few (<4.2%) said that the model did not help persons with disabilities (see table 4). Table 4 - Significance of the VR model towards persons with disabilities and employers Description Yes (%) No (%) Help person with disability get a job easily (n=95) 86.2 13.8 Help employer access to laborers with disability (n=95) 56.8 43.2 Help person with disability integrate into society (n=95) 53.7 46.3 Did not helped (n=95) 4.2 95.8 Other (n=95) 6.3 93.7 Source: Result of final evaluation External End-of-Program Evaluation Report Vocational Rehabilitation Program 2014 23 Thirdly, the VR model for persons with disabilities could be easily replicated in other provinces/cities thanks to the program documentation efforts. During two and a half years of demonstration, the “handbook on the procedures of vocational rehabilitation and employment services for persons with disabilities” was completed by HEIC and VNAH. This was a striking result of the demonstration model with a standard procedure, which could be recommended to other provincial employment service centers. The Evaluation Team believes that this is a good indicator showing the sustainability of the model. Some points for improvement Communication activities should be improved. Four out of six employers involved in the final assessment were embarrassed when asked about the name of the model or the program implemented at the HEIC. Two employers said that they were trying to search on the Internet via Google for several times when having needs of workers with disabilities but no names or information on the VR services appeared. Two local employers (Huong Hoa Ha Co. and Community Co., Ltd.) participating in the inclusive job fair in April 2014 had no ideas of the model of vocational rehabilitation services. “We come here to look for employment candidates as we were invited and have needs of employing workers with disabilities.” Source: N.V.H. - Community Co., Ltd. “We had no ideas or information about the program. We just know Mr. Dung who referred workers with disabilities to us”. Source: A. D., Fortuna Hotel Similarly, only 18% of persons with disabilities got to know about the model via the Internet, 6% via the radio and television, nearly half (46%) via relatives, friends or peers, and one third (30%) via local DPOs. During the past time, the VR model has been promoted through many different channels including internet and television. However, the frequency of communication through these two channels mentioned is lower. As revealed from the evaluation, the internet is an important channel of information for qualified persons with mobility disabilities or those having received vocational trainings. If the VR model continues to be developed in the future, two following notes should be paid attention to when designing communication activities: Firstly, appropriate communication channels should be selected for the right target groups (employers and persons with disabilities); secondly, after selection of the right targets, a quick survey should be conducted to investigate the communication channel of their favorite or frequent access. Based on the results of the quick needs assessment, appropriate communication strategies should be designed to increase access to the program and its communication efficiency. Monitoring and evaluation (M & E): The program reporting mechanism was regularly well maintained and also shared with stakeholders. However, the program encountered difficulties in updating the database of beneficiaries. At the time of the evaluation, the database was not fully updated with all information in the VR forms (assessment of the employers’ satisfaction, and statistics on the laborers’ income before applying for the service, etc.). The lack of data made it difficult to analyze the program effectiveness. We understand that it was challenging to External End-of-Program Evaluation Report Vocational Rehabilitation Program 2014 24 keep in touch with workers with disabilities as they often changed their phone numbers and addresses, or rarely checked emails; especially, persons with hearing impairment who required communication through their relatives. However, the counselors should consider this as an important work during the provision of services and get agreement with them on a communication channel right from the beginning. Similarly, sufficient feedbacks from employers after employing a person with disabilities should also be collected. Feedbacks from both employers and employees should be monitored from the beginning, mid-term and end of the program. In addition, to make the database more user￾friendly and time-saving, appropriate software should be developed for the model. Because, as said, the database is not only an important instrument for program monitoring and evaluation but also makes the vocational rehabilitation services more professional. There was a lack of supportive devices to communicate with persons with hearing impairment. We believe that the counselor has utilized 100% of their energy, enthusiasm and sincere attitudes to work with workers with disabilities. However, it was still challenging for them to communicate with persons with hearing disabilities. Persons with hearing impairment had difficulty to access the model services. To date, only 16.8% persons with hearing impairment successfully accessed the program services. According to the final evaluation survey on persons with disabilities’ job retention rate, many employers, especially employers working in garment and fashion industry, had the needs of hiring employees with hearing impairment6 . This fact suggests that VNAH or its implementing partners should use supportive communication devices to communicate with persons with hearing impairment and improve their access to the program. At the same time, it should be ensured that different disability groups have equal opportunities to access to the VR services. PART IV - LESSONS LEARNED AND RECOMMENDATIONS 4.1. Lessons learned Resources mobilization: The Evaluation Team believes that the model would achieve high performance and efficiency if resources from stakeholders (employers, community, professional associations and NGOs) are well mobilized in the implementation process. Currently, there are many programs and organizations working in disability. In fact, the VR services for persons with disabilities in Hanoi did this well. Good resources mobilization will not only empower the project but also help avoid duplication of support for persons with disabilities. Empowering the implementing partner and enhancing the participation of beneficiaries and stakeholders ensured the program to be actively and creatively implemented. When the implementing partner was empowered, their initiative and creativity would be upheld in the implementation of activities. HEIC is a typical case study for other provincial employment service centers in advocating for changes in funding mechanisms by the local government in employment resolution for persons with disabilities. Creativity in the development of work samples in conformity with reality or the development of the service providers network are significant lessons that HEIC has done during the past time. 6 The job retention survey in Hanoi, Dong Nai and Da Nang was conducted by VNAH at the same time with this final program evaluation. External End-of-Program Evaluation Report Vocational Rehabilitation Program 2014 25 Appropriate staffing: Staff involved in the program implementation was an important part determining the success of the program. Additional trainings in various ways should have been provided to those directly involved in the program implementation in case of a personnel change, which would avoid the situation that the program's effectiveness is affected by a person. In particular, the staffing structure was relevant with the workload. Documentation: Documentation of success stories, case studies and program guidance was necessary because it would be utilized by other programs or provide the basic information for the design of new programs and projects. It was recommended by the MOLISA that lessons learned from the two demonstration models of vocational rehabilitation implemented should be documented and shared with the program implementing partners. Regarding the program activities, it should be noted to develop the network and work samples and flexibly apply them when implementing the VR services in other provinces/cities. The VR services would not be successful if a service providers’ network is not developed. Network development is also a way to mobilize resources and increase access to service users. In addition, work samples should be regularly updated to be in conformity with the requirements of professions in demand. 4.2. Recommendations for similar initiatives in the future Humanitarian and economic values of the VR services for persons with disabilities seem to be undeniable. The following points should be noted when replicating the services: Survey on the labor market for persons with disabilities: All the stakeholders stressed that the vocational rehabilitation model for person with disabilities should only be applied in a province/city if three following criteria are met: (i) there is a labor market; (ii) the implementing partners are committed to implement the program; (iii) it is supported by the local government. Of these, the criterion of “there is a labor market” is the first priority. To determine if there is labor market in a province/city, it requires statistics on workers with disabilities, types of disabilities, and employers’ needs of laborers with disabilities. Currently, MOLISA/ DOLISA and DPOs have difficulties in statistical work. Thus, it is necessary to conduct a survey on the labor market before deciding to implement the model. Increase of budget for communication activities: Communication does not only aim at improving access to persons with disabilities, but also improving the quality of the model’s services. Popular communication channels should be used for both service providers and service users. It is necessary to identify the target groups and communication needs of each so that the right communication channel would be selected in the locations. Development of employers and peer counselors networks: The employer network can be developed by different ways. Workshops (with the participation of policy makers and governmental agencies, etc.) to disseminate/update preferential policies for enterprises or share success stories of employers receiving employees with disabilities are the solutions proposed by employers involved in this final assessment. The VR services can be introduced to employers through workshops, social events and or awareness raising workshops for unemployed persons with disabilities. External End-of-Program Evaluation Report Vocational Rehabilitation Program 2014 26 Capacity building for program implementing partners and service users: Types of capacity building should be diversified such as training, workshops, site visits and on-the-job coaching. Capacity building topics should be built based on the training needs assessment. Communication should be provided to both persons with disabilities and their families. Development of a management software and regularly updated database: Regularly updated data of the program would support the program monitoring and evaluation as well as timely adjustment of activities. Ideally, a data base management software should be developed. The current database is updated on Excel, which is very difficult to follow, unfriendly and difficult to users. Equal access opportunities for all disability groups: It should be ensured that all persons with different types of disabilities have equal rights and opportunities to access to the vocational rehabilitation model. Since the beginning of the program, necessary skills and supportive instruments should be provided to the counseling departments to communicate effectively with different disability groups (with hearing or visual impairments, etc.). In addition to supportive devices, the counselors should be also trained on minimum skills to communicate with persons with hearing impairment. PART V – CONCLUSION At the final evaluation time, the VR program already achieved expected results as described in the agreement between VNAH and HEIC. At least 432 persons with disabilities registered to the VR program (about 50% was female), of which, 346 persons with disabilities (80%) received employment introductions; 159 got jobs and retained those jobs for over 3 months (37%) and 102 received referrals to free vocational trainings. However, two activities were planned but have not been conducted due to the program’s limited budget, including: trainings to raise awareness of parents/families of persons with disabilities (under Activity 3); and connections to persons with disabilities caused by job accidents through physical rehabilitation centers. More importantly, the SP network and work assessment center (or work practice room as called by HEIC staff) were established and developed. The steps of the VR process had been documented and would be easily replicated in other provinces. With strong commitment of HEIC in continuing maintenance and development of project’s results, Hanoi government has allocated funding for the HEIC to implement employment supports for persons with disabilities. Thus, project evaluation team strongly believed that the results mentioned will be sustainable. Regarding the relevance elements, it was observed by the evaluators that the design of the project was relevant to the prioritized issues and concerns of all stakeholders, particularly the needs of persons with disabilities. The VR model was also fully consistent with the local context, strategies and policies. In relation to the effectiveness and efficiency elements, the implementation arrangements were highly effective. The program set up a close financial management system to ensure proper use of the resources. The VR model has mobilized large resources from its service providers’ network. Interestingly, economic value of the model was also demonstrated through External End-of-Program Evaluation Report Vocational Rehabilitation Program 2014 27 comparison between costs and benefits. The improvement of stakeholders’ capacity, changes in the awareness about persons with disabilities by employers, communities and persons with disabilities themselves were also good indicators reflecting the effectiveness of the demonstration model. Participation and levels of satisfaction: Participation of both local authorities and service providers is also an important aspect. VNAH and HEIC created a participation mechanism to all stakeholders. Over 70 percent of employers and persons with disabilities were very satisfied with the vocational rehabilitation services. Both employers and persons with disabilities had good impressions about the attitudes of the VR Unit staff. Strengths and weaknesses: The model had three outstanding strengths. The first strength was the economic value; the second was the humanitarian value; the third was the sustainability and feasibility of the model in replicating the VR model to other localities. There were still some points for improvement, including communication to the right target groups, updating of database, and more effective approaches to improve the accessibility to vocational rehabilitation services to persons with hearing and visual impairments. External End-of-Program Evaluation Report Vocational Rehabilitation Program 2014 28 REFERENCES 1. Vocational Rehabilitation Community Base Guidline, World Health Organisation (WHO), 2010. 2. VNAH, Technical Application on Inclusion of Vietnameses Disabilities (IVWD) Program 3. Program materials including: Database, Workplan, Memorandum of Association between VNAH and HEIC, annual report (monthly, quarterly, yearly). 4. The Decree 28/2012/ND-CP on the implementation of some articles of the Law on disabilities. Source: http://www.moj.gov.vn/vbpq/Lists. 5. Retention rate survey of workers with disabilities at the work place in Hanoi, Danang and Dongnai, 2014, VNAH. 6. Website Hanoi Disabled People Association: http://www.dphanoi.org.vn/ 7. Website: The Association for the support Vietnam Handicap and orphans: http://asvho.org.vn/modules. 8. Website Information support People with disabilities– MoLISA http://nccd.molisa.gov.vn/index.php/uu-tien/day-nghe-tao-viec-lam Annex 1 - FRAMEWORK FOR EVALUATION Criteria/dimensions Key questions Methods Sources of information Efficiency • Please compare the inputs & outcomes or achievements. • To which extent have the human and financial resources been wisely allocated and coordinated to achieve project outcomes? • How are the expenses and time allocated for activities? Are program activities on progress? Are the activities in accordance with the tentative plans? • How is the quality of activities performance/achievements? • Is the budget allocation appropriate with each components or overall objectives of the program? • How are the financial regulations? • Document review • Focus groups • Key informant interviews • Review of primary and secondary data • Workshop • Project documents and government’s regulations and policies; • Implementing partners and stakeholders (HEIC, VNAH, DOLISA, etc.); • beneficiaries Effectiveness • What is the base for planning of activities? • How efficient are the project planning, M&E, coordination and communication systems? • Are any activities wasteful or irrelevant to the objective? • What is the participation mechanism during the program management and implementation? • Desk review • Focus groups • Key informant interviews • Review of primary and secondary data • Workshop • Project documents and government’s regulations and policies; • Implementing partners and stakeholders (HEIC, VNAH, DOLISA, etc.); • Persons with disabilities Impacts • How has the program contributed to the implementation and mainstreaming of cross cutting • Desk review • Focus groups • Project documents and government’s regulations External End-of-Program Evaluation Report Vocational Rehabilitation Program 2014 30 Criteria/dimensions Key questions Methods Sources of information issues (participation, gender, and human rights) • What are positive, negative, expected and unexpected impacts (socially, economically and institutionally) of the program? • Has the program influence any of the local government’s social welfare policies or of the implementing partners’ and stakeholders’ social responsibilities? • How has the program influence the funding allocation by the local government? Please specify. • Key informant interviews • Review of primary and secondary data • Workshop • Case studies and policies; • Implementing partners and stakeholders (HEIC, VNAH, DOLISA, employers, etc.); • Beneficiaries (persons with disabilities and their families) Relevance • Is the program approach appropriate with the implementing partners’ capacity and the situation in Hanoi city? Please specify. • Is the project design relevant to the need of disability community? • Are project objectives logically linked? • Desk review • Focus groups • Key informant interviews • Project documents and government’s regulations and policies; • Implementing partners and stakeholders (HEIC, VNAH, DOLISA, employers, etc.); • Persons with disabilities Sustainability • How has capacity of persons with disabilities and project partners been increased to maintain and improve the program achievement and impacts? • Is there a mechanism or institutionalization to ensure the project practices, approaches or ways of working to be continued? • Are there any indicators or opportunities for sustainability and expandability of project activities • Desk review • Focus groups • Key informant interviews • Review of primary and secondary data • Workshop • Case studies • Project documents and government’s regulations and policies; • Implementing partners and stakeholders (HEIC, VNAH, DOLISA, employers, etc.); • Persons with disabilities External End-of-Program Evaluation Report Vocational Rehabilitation Program 2014 31 Criteria/dimensions Key questions Methods Sources of information and the intervention model? Satisfaction • How is the satisfaction of the program stakeholders? What activities or services are the service users satisfied about? What are dissatisfied? • How are the employers as employers of workers with disabilities satisfied about? • Focus groups • Quantitative questionnaire • Case studies • Persons with disabilities and their families • Employers employing workers with disabilities Best practices • What are good approaches or best practices of the program regarding: - intervention approaches? - program design? - implementation and M&E? • Desk review • Focus groups • Key informant interviews • Review of primary and secondary data • Workshop • Case studies • Project documents and government’s regulations and policies; • Implementing partners and stakeholders (HEIC, VNAH, DOLISA, employers, etc.); • Persons with disabilities and their families Recommendations and lessons learned • What should be done the same in a familiar program in the future? • What should be done differently in the future familiar project? And how? • If the program is extended to other provinces, what can HEIC contribute? • Document review • Focus groups • Key informant interviews • Workshop • Project documents and government’s regulations and policies; • Implementing partners and stakeholders (HEIC, VNAH, DOLISA, employers, etc.); • Persons with disabilities and their families Annex 2 – QUANTITATIVE SURVEY QUESTIONNAIRE PROJECT NAME JOB NO. MAIN QUESTIONNAIRE BCH # SP # SURVEYOR DATED STAR TING TIME ENDI NG TIME SUPERVISOR SUP CODE INTERVIEW STATUS BY Yes No SIGNATURE Dated: ACCOMPANIED (FS) 1 2 LOGIC-CHECKED (FS) 1 2 TEL BACK-CHECKED (QC) 1 2 F2F BACK-CHECKED (QC) 1 2 INTRODUCTION AND ASKING FOR PERMISSION Hello! My name is ………. (surveyor’s name). We are conducting the final evaluation on the Vocational Rehabilitation demonstration model for persons with disabilities. This evaluation is conducted by Vietnam Assistance for the Handicapped (VNAH) and Hanoi Employment Introduction Center (HEIC). Results of this survey will be utilized to help the HEIC and its donor to look back at what has or has not been achieved. The interview will last for about 30 minutes. Are you willing to answer our questions? All your responses will be kept anonymous and absolutely appreciated. Your personal information will not be mentioned in any documents. We would like to thank your very much for your cooperation. If the respondent agrees to answer the interview  1.1 If the respondent refuse to answer the interview  FINISH External End-of-Program Evaluation Report Vocational Rehabilitation Program 2014 33 END-OF-PROGRAM EVALUATION VOCATIONAL REHABILITATION DEMONSTRATION MODEL FOR PERSONS WITH DISABLITIES PART 1: PERSONAL INFORMATION Respondent’s address Name of respondent: In which district are you living? ................................................................................... Are you working in urban or suburban areas in Hanoi? Urban area................................................................. 1 Suburban area ........................................................... 2 No. Question Code Shift 1.1 Sex Male……..……………………………………………..………. 1 Female………………………………….……………….……… 2 1.2 How old are you? Or When were you born? (calendar year) ...................................................................... 1.3 What disabilities do you have? (Select any appropriate answers) Visual impairment …………………………………….. 1 Hearing impairment …………………….………….… 2 Mobility disabilities …………………………………… 3 Mental disabilities ………………………………….….. 4 Other disabilities (please specify) ……………. ….. 5 ................................................. ……………………… External End-of-Program Evaluation Report Vocational Rehabilitation Program 2014 34 No. Question Code Shift 1.4 What is your highest education level that you have accomplished? (Select one answer only) Have never been to school……………..……………. 1 Primary school………………………………..……………. 2 Junior high school...…………….……….…….……….. . 3 Senior high school …….………………….……….……. 4 University/College ........................................... 5 Vocational training school...…………………………… 6 Others (please specify) ………………………..……….. 7 1.5 Have you ever been to the Hanoi Employment Introduction Center to get employment seeking support in the past three years? Yes………………………………………………………….……… 0 No………………….…………………………………….....……. 1 1.5.1 1.6 1.5.1 If Yes, what services have you received from the HEIC? (select any appropriate answers) Counselling to participate into the program …… 1 Intake interview……...…………………..…………………. 2 Development of individual work plan ….…………. 3 Referral to vocational training programs.…..…… 4 Employment introduction...….………..……………….. 5 Soft skills trainings ……..……….…..……….……………. 6 Follow-up support after getting jobs ...……..….… 7 Others …………………………………….…..…………………. 8 Don’t know...…………….…..………..………..………… 99 1.6 Have you participated in any inclusive or exclusive job fairs for persons with disabilities at HEIC? Yes…………………………………………….……….………… 0 No………………….……………………………………………. 1 1.6.1 2.1 1.6.1 How many job fairs have you participated? One……………………………………………………………… 1 Two……………….……………….…………….……..……… 2 Three ……………….…….……………………….….………. 3 Over three……..…….……………………….…..………… 4 Don’t know/don’t’ remember.…………………….. 99 External End-of-Program Evaluation Report Vocational Rehabilitation Program 2014 35 PART 2: EMPLOYMENT AND INCOME/PROJECT EFFECTIVENESS No. Question Code Shift Introduction (the interviewers read out loud): In this part, you will be questioned about your current work. Please feel comfortable to share your answers. 2.1 Are you currently employed? Yes……………………………………………………… 0 No……………….……………………………………. 1 2.1.3 3.1 2.1.1 If yes, what’s your job? (open question) …................................................................ …................................................................ 2.1.1. 3 2.1.2 If no, how long have you been unemployed? ……............................................................. ……............................................................. 2.1.2.1 Why did you leave your previous job? ……............................................................ ……............................................................. 2.1.3 Do you have a labour contract with your employer? Yes……………………………………………………… 0 No…………….……………………………………….. 1 2.1.4 How did you get that job? (choose any appropriate answers) Introduced by friends/relatives….......... 1 Introduced by the HEIC ……...……........... 2 Introduced by the DPOs…………...……..... 3 It’s my family’s secondary job ……….…... 4 Find it myself…...................................... 5 Other (please specify)............................ 6 2.1.5 Is the current job suitable for you? Yes……………………………………….……………… No…………….…………………………………..……. 1 2.1.5. 1 2.1.6 2.1.5.1 If Yes, how suitable is it? Health…............................................... 1 External End-of-Program Evaluation Report Vocational Rehabilitation Program 2014 36 No. Question Code Shift (select any appropriate answers) Profession….......................................... 2 Experiences…………............................... 3 Transportation……………........................ 4 Other (please specify)........................... 5 Don’t know………………………………….……. 99 2.2 How do you feel about that job? Very satisfied……………………………………. 1 Satisfied...…………………………………..……… 2 Neutral…………………………………..…………. 3 Dissatisfied…..……….…………………………. 4 Very dissatisfied…................................ 5 Don’t know……………………………………….. 99 2.3 What is your monthly average income? (Select one answer) Under 1,000,000 VND…………………..... 1 From1,000,000VND ->3,000,000VND... 2 From 3,000,000 VND ->5,000,000VND. .. 3 From 5,000,000VND -> 7,000,000VND... 4 From 7,000,000 VND ->10,000,000VND... 5 Over 10,000,000VND…….…….............. 6 2.4 How is your most current income compared to 3 years ago? Higher…................................................. 1 Lower…..….............................................. 2 The same…..…........................................ 3 Don’t know........................................... 99 2.5 How long are you committed to this job? under 1 year........................................... 1 In one to two years…….......................... 2 In two to five years................................ 3 Over 5 years.......................................... 4 Haven’t planned yet………...................... 5 External End-of-Program Evaluation Report Vocational Rehabilitation Program 2014 37 No. Question Code Shift Don’t know……………………..................... 99 PART 3: ACCESSIBILITY No. Question Code Shift 3.1 Did you encounter any challenges when seeking the job? Yes.......................................................... 0 No.......................................................... 1 3.1.1  3.2 3.1.1 If Yes, what were the challenges? ……………………………………………………………… ……………………………………………………………… ……………………………………………………………… ……………………………………………………………… 3.2 How did you get to know about the vocational rehabilitation model for persons with disabilities at HEIC? (Select any appropriate answer) Through friends/relatives ………………..... 1 Through TV/newspapers….…………………. 2 Through DPOs…………………………………….. 3 Accidentally see it……………...……………... 4 On the internet……................................. 5 Other ……………..……................................ 6 3.3 Did you have any challenges when completing the VR forms (application, individual work plan…) under VR program? Yes.......................................................... 0 No........................................................ 1 3.3.1  3.4 3.3.1 If Yes, what were they? ……………………………………………………………… ………………………………………………………………. ……………………………………………………………… ………………………………………………………………. 3.4 How does the VR model at HEIC help persons with disabilities? Make it easier for persons with disabilities to find jobs …..……………........ 1 Enable faster connections between employers and workers with disabilities.............................................. 2 Enable persons with disabilities become inclusive with workers without disabilities.............................................. 3 External End-of-Program Evaluation Report Vocational Rehabilitation Program 2014 38 No. Question Code Shift Does not help with anything…............... 4 Other (please specify)............................ 5 Don’t know…...………………………………….. 99 3.5 How do you value the VR services provided by counsellors at HEIC? Very satisfied…………………………….………. 1 Satisfied...…………………………………………… 2 Neutral………………………………………………. 3 Dissatisfied…..……….…………………..………. 4 Very dissatisfied….................................. 5 Don’t know……………………………………….. 99 3.6 Has the VR program had any impacts on your life in the last three years? ................................................................... ................................................................... 3.7 Are you intending to refer your peers or relatives with the same situation to participate in the VR program at HEIC? Yes.......................................................... 0 No........................................................ 1 External End-of-Program Evaluation Report Vocational Rehabilitation Program 2014 39 Thank you very much for your cooperation! LAST PAGE RESPONDENT’S DECLARATION: This interview was well conducted according to my satisfaction. DATE OF INTERVIEW ________________ SIGNATURE OF RESPONDENT____________________ INTERVIEWER’S DECLARATION I certify that this interview has been personally carried out by me with the respondent. I further declare that the respondent, whose name and address appear above, was unknown to me until the interview. I confirm that, before returning this questionnaire, I have checked that it meets and was carried out in accordance with the principles and instructions supplied to me for this study. I understand that the information given to me during the interview must be kept confidential. DATE OF INTERVIEW________________SIGNATURE OF INTERVIEWER_____________________