EVALUATION OF Vietnam Disabilities Programming November 2015 This report is made possible by the support of the American People through the United States Agency for International Development (USAID). The contents of this study are the sole responsibility of Mary F. Hayden, Hoang V. Tran, Tra H. Nguyen, Long T. Tran and Chang N.Q. Le and Management Systems International and do not necessarily reflect the views of USAID or the United States Government. Evaluation of Vietnam Disabilities Programming Contracted under AID-440-C-14-00003 Vietnam Evaluation, Monitoring and Survey Services DISCLAIMER This report is made possible by the support of the American People through the United States Agency for International Development (USAID). The contents of this study are the sole responsibility of Mary F. Hayden, Hoang V. Tran, Tra H. Nguyen, Long T. Tran and Chang N.Q. Le and Management Systems International and do not necessarily reflect the views of USAID or the United States Government. Evaluation of USAID Vietnam Disabilities Programming iii TABLE OF CONTENTS Acronyms ...........................................................................................................................................................ii Glossary ..............................................................................................................................................................ii Acknowledgments ...........................................................................................................................................iv Executive Summary .........................................................................................................................................iv Key Findings ...................................................................................................................................................................................... i Conclusions ..................................................................................................................................................................................... iv Recommendations ......................................................................................................................................................................... vi I. Overview of the Evaluation .....................................................................................................................1 II. USAID Disability Programming in Vietnam ..........................................................................................3 Historical Overview .......................................................................................................................................................................3 Current Portfolio ............................................................................................................................................................................4 III. Methodology ...............................................................................................................................................7 IV. Findings ...................................................................................................................................................... 12 Relevance of the USAID disabilities programming ...............................................................................................................12 Improving Lives and Opportunities ..........................................................................................................................................12 Development of the Legal and Policy Environment .............................................................................................................20 Strengthening Critical Institutional Capacities ......................................................................................................................29 Sustainability of Program Interventions ..................................................................................................................................56 V. Conclusions and Recommendations ................................................................................................... 60 Conclusions ....................................................................................................................................................................................60 Recommendations ........................................................................................................................................................................64 Annex 1: References ..................................................................................................................................... 67 Annex 2: Evaluation Scope of Work ......................................................................................................... 70 Annex 3: Differences Between Qualitative and Quantitative Research ............................................ 75 Annex 4: Summary of USAID Supported Sites ....................................................................................... 76 Annex 5: Key Informants ............................................................................................................................. 82 Annex 6: Evaluation Schedule ..................................................................................................................... 88 Annex 7: Interview Guides .......................................................................................................................... 97 Annex 8: Desk Review and Independent Data Source Review Form .............................................155 Annex 9: Sources of data for numbers of direct beneficiaries ..........................................................156 Annex 10: Performance Indicators: Targeted and Achieved .............................................................158 Evaluation of USAID Vietnam Disabilities Programming i List of Tables Table 1: Type of USAID Activities by Sector and by Implementing Partner……………………………… 5 Table 2: Selected Provinces and Districts…………………………………………………………………. 7 Table 3: Number of Beneficiaries of Medical Assistance Provided by DAI from 2013 to 2015………… 14 Table 4: Vietnam Disability Legal and Policy Provisions Assisted by USAID……………………………. 21 Table 5: Capacity Building on Physical Rehabilitation by PDSP from 2013 to 2015…………………….. 31 List of Figures Figure 1: Geographic Locations of USAID Disability Activities…………………………………………… 8 Figure 2: Impact of Assistance on the Lives of Persons with Disabilities………………………………… 12 Figure 3: Beneficiaries Reporting Improvement in Selected Life Domains………………………………. 12 Figure 4: Beneficiaries’ Awareness of the National Law on People with Disabilities …………………….13 Figure 5: Impact on the Quality of Life for Beneficiaries of Medical Assistance…………………………. 14 Figure 6: Impact on the Quality of Life for Beneficiaries of Education Assistance………………………. 17 Figure 7: Impact on the Quality of Life for Beneficiaries of Employment/Livelihood Assistance…………18 Figure 8: Level of Satisfaction among Beneficiaries of Medical Assistance………………………………. 32 Figure 9: Level of Satisfaction among Beneficiaries of Education Assistance……………………………. 36 Figure 10: Level of Satisfaction among Beneficiaries of Employment/Livelihood Assistance……………. 41 Evaluation of USAID Vietnam Disabilities Programming ii ACRONYMS AMSA Agency of Medical Services Administration (MOH) BDS Birth Defects Surveillance BREC Blue Ribbon Employer Council BOET Bureau of Education and Training BOE Bureau of Employment BOLISA Bureau of Labor, Invalids and Social Affairs CBR Community-Based Rehabilitation CDC Children with Disabilities Center CRPD Convention on the Rights of Persons with Disabilities CRS Catholic Relief Services CSA Committee on Social Affairs (a National Assembly committee) DAI Development Alternatives Inc. DHWC Danang Hospital for Women and Children DIS Disability Information System DLA Department of Legal Affairs, located in MOLISA DOET Department of Education and Training DOH Department of Health DOLISA Department of Labor, Invalids and Social Affairs DPO Disabled People’s Organization DPC District People’s Committee ESC Employment Service Center GVN Government of Vietnam HEIC Hanoi Employment Introduction Center HR Human Resources ICF International Classification of Functioning ICT Information Communications Technology IE Inclusive Education IEP Individual Education Plan ILO International Labour Organization INGO International Non-Governmental Organization IP Implementing Partner IT Information Technology ITTP Information Technology Training Program IVD Inclusion of Vietnamese with Disabilities Program KII Key Informant Interview MDRI Mekong Development Research Institute MOET Ministry of Education and Training MOF Ministry of Finance MOH Ministry of Health MOLISA Ministry of Labor, Invalids and Social Affairs MSA Medical Services Administration NA National Assembly NAP National Action Plan NBS Newborn Screening NCCD National Coordinating Council for Disabilities NGO Non-Governmental Organization PAP Provincial Action Plan PDSP Persons with Disabilities Support Program PPC Provincial People’s Committee PT Physical Therapy SDC School for Disabled Children SPD Social Protection Department SOB Survey of Beneficiaries VBPO Vietnam Business Process Outsourcing VCCI Vietnam Chamber of Commerce and Industry VEMSS Vietnam Evaluation, Monitoring and Survey Services Program VNAH Vietnam Assistance for the Handicapped VNEN Vietnam Escuela Nueva VWD Vietnamese with Disabilities UNICEF United Nations Children’s Fund USG United States Government USAID United States Agency for International Development VT Vocational Training Evaluation of USAID Vietnam Disabilities Programming ii GLOSSARY Cerebral palsy: The general term for a number of neurological conditions that affect movement and coordination. It is caused by a problem in the parts of the brain responsible for controlling muscles. Circulars and decisions: The ministry responsible for drafting the relevant law and decree also issues circulars and decisions. The purpose of these documents is to provide policy guidance for how the law and decree will be implemented. Cleft lip and cleft palate: Birth defects that occur when a baby’s lip or mouth do not form properly during pregnancy. A baby can have a cleft lip, a cleft palate or both a cleft lip and a cleft palate. Decree: A statement of significant legal importance issued by the government, without reference to the National Assembly, establishing detailed rules beneath a law. Developmental disability: A severe, chronic disability of an individual that is attributable to a mental or physical impairment or combination of mental and physical impairments; is manifested before the individual reaches age 22; is likely to continue indefinitely; results in substantial functional limitations in three or more of the following areas of major life activity: a) self-care, b) receptive and expressive language, c) learning, d) mobility, e) self-direction, f) capacity for independent living and g) economic self￾sufficiency; and reflects the individual’s need for a combination and sequence of special, interdisciplinary or generic services, individualized supports or other forms of assistance that are of lifelong or extended duration and are individually planned and coordinated.1 Disabled People’s Organizations (DPOs): Organizations in which persons with disabilities hold a majority (51 percent) of control at the board and membership levels. “The role of these organizations includes providing a voice of their own, identifying needs, expressing views on priorities, evaluating services and advocating change and public awareness. As a vehicle of self-development, these organizations provide the opportunity to develop skills in the negotiation process, organizational abilities, mutual support, information sharing and vocational skills and opportunities. In view of their vital importance in the process of participation, it is imperative that their development be encouraged.” (World Programme Action Concerning Disabled Persons, p. 8) Down syndrome: A congenital disorder arising from a chromosome defect, causing intellectual impairment and physical abnormalities including short stature and a broad facial profile. It arises from a defect that involves chromosome 21, usually an extra copy (trisomy-21). Epilepsy: A neurological disorder marked by sudden recurrent episodes of sensory disturbance, loss of consciousness or convulsions, associated with abnormal electrical activity in the brain. Vietnam Escuela Nueva (VNEN): VNEN is a Global Partnership for Education project. The managing entity is the World Bank. Its purpose is to introduce and use new teaching and learning practices in the classroom to target the most disadvantaged groups of primary students. The project’s immediate beneficiaries are children in primary school who belong to disadvantaged groups in 20 priority provinces. The project is a set of sequential activities that constitute a comprehensive package of pedagogical and administrative reform. The project includes four components: (1) materials development for pedagogical transformation, which includes the development of learning guides and other materials and capacity building for material development, (2) training and provision of materials, (3) school-level support for VNEN implementation and (4) project and knowledge management. 1 U.S. Developmental Disabilities Assistance and Bill of Rights Act Evaluation of USAID Vietnam Disabilities Programming iii Habilitation: The process of supplying a person with the means to develop maximum independence in daily living activities through training or treatment.2 Habilitation can include cognitive, social, fine motor, gross motor or other skills that contribute to mobility, communication and performance of activities of daily living and enhance quality of life. Individualized supports: Supports that enable an individual with a developmental disability to exercise self-determination, be independent, be productive and be integrated and included in all facets of community life. They are designed to a) enable an individual to control his or her environment, permitting the most independent life possible, b) prevent placement into more restrictive living arrangements than necessary and c) enable such individuals to live, learn, work and enjoy life in the community. Individualized supports include provision of rehabilitation technology, assistive technology and assistive technology services, as well as: 1) early intervention services, 2) respite care, 3) personal assistance services, 4) family support services, 5) supported employment services and f) support services for families headed by aging caregivers of individuals with developmental disabilities.3 Laws: Laws are drafted by the relevant ministry and approved by the National Assembly at one of its twice-yearly sittings, after being approved by the government. Rehabilitation: Services to restore an individual to her or his original or improved state or condition. Waardenburg syndrome: A group of conditions passed down through families that involve deafness and pale skin, hair and eye color. There are four types of Waardenburg syndrome; the most common are types 1 and II. Symptoms may include the following: cleft lip (rare); constipation; deafness (more common in type II disease); extremely pale blue eyes or eye colors that do not match (heterochromia); pale-color skin, hair and eyes (partial albinism); difficulty completely straightening joints; possible slight decrease in intellectual function; wide-set eyes (in type I); white patch of hair or early graying of the hair; and problems with the arms or intestines (less common). 2 Mosby’s Medical Dictionary, 8th edition. © 2009, Elsevier 3 U.S. Developmental Disabilities Assistance and Bill of Rights Act Evaluation of USAID Vietnam Disabilities Programming iv ACKNOWLEDGMENTS The assessment team wishes to thank the many Vietnam central, provincial, district and commune officials who agreed to meet. The team also extends thanks to the adults with disabilities, parents of children with disabilities, members of disabled people’s organizations, teachers, school managers, employers, doctors and technicians and faculty from higher education institutions who agreed to be interviewed. Additionally, we thank the management and field staff from Vietnam Assistance for the Handicapped, Catholic Relief Services and Development Alternatives Incorporated for generously providing documentation and information about their disabilities programs and activities. We also appreciate USAID’s assistance in meeting and assembling information, particularly Randy Flay and Ha Thi Nguyen of the Program Development Office, and Van Ha Le and Ezra Simon of the Environmental and Social Development Office. Cover Photo: Ms. Le Thi Het received a USAID-funded wheelchair and regular physical therapy services as part of a three-year project in Danang. Credit: Richard Nyberg @USAIDVietnam Evaluation of USAID Vietnam Disabilities Programming i EXECUTIVE SUMMARY Since 1989, the United States Government (USG) has been assisting persons with disabilities in Vietnam. To date, the USG has contributed $60 million in assistance to persons with disabilities, regardless of cause. USAID provides ongoing assistance through the Inclusion of Vietnamese with Disabilities (IVD) Program, which began in 2005, and the Persons with Disabilities Support Program (PDSP), which the Agency awarded in 2012. Assistance has helped address medical, education, employment and other needs and supports the broader inclusion of persons with disabilities in Vietnamese society. USAID/Vietnam requested that the Vietnam Evaluation, Monitoring and Survey Services (VEMSS) Program document the general impact and key contributions of the programs in building awareness, the enabling environment, institutional capacity and operation, and identify current opportunities and gaps in disability programming that can serve as a base of understanding for the ongoing work of the Government of Vietnam (GVN), program implementers, disabled people organizations (DPOs) and other stakeholders. The purpose of the evaluation was to examine project activities from 2005 to present. Specifically, the evaluation answers four key questions: 1. To what extent and in what ways have the programs improved the lives and opportunities for persons with disabilities in Vietnam? Have the planned benefits been delivered and received as perceived by beneficiaries? 2. To what extent has the programming assistance helped develop the GVN’s legal and policy environment, public awareness and support for disability services? 3. In terms of improving service delivery, did the USAID Disability Support Program provide an enabling environment and a supporting systems approach in technical assistance to strengthen critical institutional capacities? 4. Are the programs’ outcomes sustainable? What challenges in terms of sustainability remain? USAID requested that the evaluation focus on six sectors: legal framework, civil society development, health/medical, education, employment/livelihood and service delivery and management. Four provinces were selected for field research (Hanoi, Danang, Dong Nai and Quang Nam) based on poverty rate, disability prevalence, the type of activities being implemented in the province and consultations with USAID. The evaluation was based on an analysis of primarily qualitative data obtained from key informant interviews (KIIs), supported by findings from document reviews (desk review and reviews of independent data sources) and a field survey of 850 direct beneficiaries. Key Findings Improving Lives and Opportunities  USAID-provided assistance improved the lives of persons with disabilities: A majority of beneficiaries surveyed said their life had improved since receiving medical (55 percent), education (65 percent) and livelihood support (69 percent) services.  The enacted National Law on People with Disabilities and subsequent public awareness activities supported with USAID assistance increased the acceptance of persons with disabilities within the community and improved provision of services to them. However, knowledge of the law among persons with disabilities and government officials is not as far-reaching as it should be. Evaluation of USAID Vietnam Disabilities Programming ii  On the whole, people were generally satisfied with medical services provided; less than 15 percent said they were dissatisfied because they felt the services received were insufficient for their needs. Results were due, in part, to significant delays (six to 12 months) in receiving certain medical services (e.g., surgeries) through the case management program.  USAID assistance for education inclusive of children with disabilities in regular classrooms resulted in the children becoming more confident and developing a more positive attitude and behavior toward studying after their educational experiences. Parents benefited as they became more involved in their respective parent associations and kept in better contact with teachers to follow the learning progress of their children.  Beneficiaries, family members and the community viewed the USAID employment/livelihoods program in a primarily positive manner. Many beneficiaries found jobs or started a business that led to earning money for themselves and, in some cases, for their families. However, others reported difficulties securing and maintaining employment.  The school-to-work project enabled graduates to obtain skills that they could use to find work and earn a living. Consequently, graduates and parents were more confident about securing employment in the future.  USAID assistance for DPOs to become service providers resulted in increased livelihood opportunities, improved income and better social inclusion for their members. Development of the Legal and Policy Environment  USAID assistance helped develop the Law on People with Disabilities and related laws and policies that protect the rights of and expand opportunities for health care, education and employment for Vietnamese with disabilities.  Members of DPOs have a broader voice to advocate on their own behalf as a result of USAID assistance.  Gaps in the current framework inadvertently perpetuate old beliefs about persons with disabilities, particularly those with severe disabilities. These gaps are due in part to inconsistencies between national disability laws and the Convention on the Rights of Persons with Disabilities (CRPD). Strengthening Critical Institutional Capacities  The technical assistance provided by USAID contributed to improvement of disability law and policymaking and implementation capacities. It helped lay the foundation for the key ministries — including the Ministry of Health (MOH), the Ministry of Education and Training (MOET) and the Ministry of Labor, Invalids and Social Affairs (MOLISA) — to strengthen their capacities to serve and support persons with disabilities.  USAID programs supported DPOs to ensure that the organizations had a voice in policy formulation and implementation. Additionally, DPO training resulted in a service provider system for and by persons with disabilities. The DPOs appeared to be successful as service providers.  USAID assistance played a key role in strengthening capacity of medical and health service providers, specifically to meet the needs of persons with disabilities. Technician training and equipment were provided to improve the delivery capacity of services for persons with disabilities (especially physical rehabilitation and treatment for autism), as well as public health Evaluation of USAID Vietnam Disabilities Programming iii services related to disability screening and surveillance, especially in Da Nang. Long-term human resources (HR) development was facilitated by support for vocational rehabilitation curriculum development and testing in four medical universities in the country.  USAID assistance provided multiple services focused on inclusive education to expand access for children with disabilities. The government accepted the principle of inclusive education and passed decisions to adopt inclusive education into schools. Capacity building for teachers and administrators has been highly valued and effective, but HR capacity is too limited to fully implement inclusive education commitments. The educational system continues to foster a two￾tiered structure that will maintain both inclusive schools and special (segregated) schools.  Different training alternatives were developed, taking both formal and informal approaches, and supported by vocational rehabilitation and employment support models based at Employment Service Centers (ESCs) and DPOs. While largely successful in the projects, beneficiaries noted that training and employment services could better align to job market opportunities, and more could be done to better prepare trainees to enter the workforce.  Stakeholders highly value the role of the case management system and the Disability Information System (DIS) in assisting persons with disabilities and facilitating the community’s awareness and acceptance in their favor, as well as to serve planning and management purposes. However, the DIS is difficult for some communes to use due to insufficient equipment, time or human resource.  The case management system was used as a means to obtain services supported by USAID, not viewed as a strategy to improve service quality and outcomes while decreasing costs. More time than is available would be needed to fully integrate case management into GVN systems across all provinces and districts4 in which USAID assistance is being provided. Sustainability of Program Interventions  Full implementation of the laws and policies on persons with disabilities will require sustained human and financial resources from the GVN to ensure that persons with disabilities can fully participate in society. Setting up DPOs as independent non-governmental organizations (NGOs) that are for and by persons with disabilities may provide an alternative and more sustainable approach to advocating for the rights of persons with disabilities.  It remains unclear whether medical and health services and training and professional development for inclusive education that USAID assistance helped expand and strengthen will be fully sustained with adequate funding by the GVN when the projects end.  Officials noted that the vocational rehabilitation model continues beyond the life of the program, since it has been institutionalized and has become a routine activity. A need exists to closely examine other vocational training and employment services to identify more effective and efficient strategies to make the services more affordable and thus sustainable. Implementation of the National Action Plan (NAP) of Vietnam is an opportunity to examine further.  A number of factors firmly suggest that without stronger local government support (funding for staff, training, systems development and expansion, etc.), case management as now provided through USAID assistance cannot be sustained. 4 To date, the case management system has been implemented in all seven mainland districts of Danang City, two of 11 districts/towns of Dong Nai Province and two of 11 districts of Binh Dinh Province. Evaluation of USAID Vietnam Disabilities Programming iv  Several issues persist with the way performance indicators were developed, tracked and used. Less than 50 percent of listed performance indicators had targets, and few had verified baseline numbers; thus, the use of these indicators to track progress is very limited. On the whole, for indicators with targets, performance was fair; the programs met or exceeded two-thirds of targets. However, no narrative explains how the indicators may have been used for performance management or to help develop strategies to sustain program interventions. Conclusions Improving Lives and Opportunities USAID-provided direct support assistance improved the lives of persons with disabilities. Across all types of USAID assistance, recipients experienced improvements in mental and physical health, the ability to live independently and improved financial status. In the legal/policy sector, the participation of DPOs in disability policy formulation and implementation resulted in more persons with disabilities being accepted in the community and within the government. Many key informants indicated that they observed less discrimination at the commune level by other community members and commune officers. In the health/medical sector, people were generally satisfied with medical services provided, although some problems emerged in accessibility to and delivery of services. In the education sector, children with disabilities improved their learning capacities, confidence and school attendance, while parents benefited from more participation in the parent associations and better contact with teachers. However, depending on the local resources, the enforcement and implementation of the policies to support children with disabilities varied, and the rate of children accessing services differed. Moreover, findings suggested that conditions contributing to the effective implementation of inclusive education included resources and capacity, multi-sector coordination of services, financial resources and timely planning. In the employment/livelihood sector, persons with disabilities received support to obtain employable skills, find jobs and better perform work. While maintaining mixed opinions about the different models of training, they enjoyed the social benefits that stemmed directly and indirectly from the work they found. Development of the Legal and Policy Environment With assistance from USAID and other developmental partners, the GVN established an extensive and comprehensive framework of disability laws and policies to protect the rights of and expand opportunities for persons with disabilities. Members of DPOs were empowered to participate in policy formulation and implementation. Although great strides have been made in supporting equal protection for persons with disabilities, full implementation of the law is hindered by the lack of sufficient funding and human resources. Moreover, the presence of gaps in the Law on Persons with Disabilities contradicts basic principles established in the CRPD. These contradictions impede the implementation of the law and, as a result, impede the ability of persons with disabilities to participate fully in society. Strengthening Critical Institutional Capacities USAID assistance has helped improve critical institutional capacities across sectors that are important for comprehensive inclusion of persons with disabilities. Examples include the following: Evaluation of USAID Vietnam Disabilities Programming v Legal/policy sector. USAID assistance improved lawmaking and policymaking capacities by providing experts, workshops, study tours and other types of technical assistance to key ministries and drafting committees and by giving a voice to DPOs in the policy formulation process. Health/medical sector. USAID-supported technician training, curriculum development and equipment provision improved the capacities of direct service delivery in the project areas and long-term HR development through medical universities. Education sector. Capacity building for teachers and administrators has been highly valued and effective, and adoption of inclusive education into schools was formalized by government’s decisions, although the issues of too-limited HR resources and the two-tiered education structure persist. Employment/livelihood sector. A variety of training and employment service alternatives were developed and the ESC-based vocational rehabilitation model institutionalized. Lack of resources for ensuring accessibility and a lack of network with employers remain the key issues. In the civil society sector, through technical assistance, training and learning-by-doing, USAID expanded and strengthened the capacity of DPOs as service providers and as advocates and disseminators of laws and policy, although their advocacy role may be limited, since they are quasi-governmental organizations. Service delivery and management The case management project was a pilot with the purpose of examining the feasibility of implementing case management in Vietnam. If successful, case management services could be implemented at a larger scale (nationally). Consequently, case management was implemented in specific districts within a small number of provinces (an exception was Da Nang, where the program was implemented in all districts and communes). USAID supported the case management program with both technical and financial assistance for direct service delivery. However, given limited resources and time constraints, the program was unable to reach all persons with disabilities in the areas it served. As a result, a large amount of persons with disabilities, including those with the greatest needs, might not have been included in the USAID programs or received services. Therefore, USAID did not improve the service delivery system, per se. However, the recipients and case managers at the district level reported numerous positive outcomes from the case management program. Sustainability of Program Interventions Stakeholders confirmed that some services were sustained after the project, including the public health services in Da Nang, the vocational rehabilitation model in ESCs and livelihood support models in selected communes. Improved HR capacities are likely to be sustained in most areas, except for the case management program. However, local government officials are seeking to incorporate a case management model into their service delivery system in Danang. The availability of adequate human and financial resources from GVN for implementation is the key issue for sustainability across all sectors. Single issues for each sector include GVN’s inability to implement, coordinate and monitor the new laws and policies at the local level (legal framework sector); uncertainty about whether provincial Departments of Health (DOHs) will continue to provide many direct health and medical services funded through PDSPs to persons with disabilities, although the services are likely to continue (health/medical sector); dependence on development projects to ensure quality, coverage and replication of inclusive education services (education sector); the limited advocacy capacity due to DPOs being quasi-governmental organization rather than NGOs for and of persons with disabilities (civil society sector). In addition, a number of factors suggest that without stronger local government support, case management as now provided through USAID assistance cannot be sustained. Evaluation of USAID Vietnam Disabilities Programming vi Recommendations Legal/Policy Sector 1. Support and assist policymakers with reviewing and revising current disability laws and policies to ensure that (a) the current process to identify persons with disabilities, as well as the type and severity of the disabilities, is valid and accurate; (b) the commune officials who make disability determinations have the expertise, knowledge and tools to carry out their responsibilities; (c) persons with mental disabilities are adequately included in all disability laws and policies; (d) all persons with disabilities, regardless of the type and severity of disability, have equal access to services and supports; and (e) the laws and policies of Vietnam align with the CRPD, as appropriate. 2. Continue to support capacity development on evidence-based laws and policy formation. 3. Expand activities to enhance awareness of the rights of persons with disabilities for officials, service providers, persons with disabilities and the general public. 4. Work with the GVN to better enforce critical provision of the laws and policies that support persons with disabilities. 5. Support GVN in implementing the CRPD by supporting and assisting the establishment of a committee that will review and monitor implementation. 6. Assist GVN in developing a comprehensive and reliable national database to collect, synthesize and disseminate information on adults and children with disabilities to improve the government’s ability to formulate, implement and coordinate laws and policies. 7. Encourage those in charge to ensure that an effective mechanism exists to coordinate action plans, facilitate equality for persons with disabilities in the society and supervise the effective and comprehensive implementation of provisions in the laws for persons with disabilities. Health/Medical Sector 1. Work with the MOH to strengthen and implement the NAP on Rehabilitation by engaging all departments and agencies. 2. If direct service provision continues, incorporate the services into the government’s existing service system and include people with and without medical insurance. 3. Assess the feasibility and options of providing capacity-building activities and equipment to potential providers who are neither licensed nor eligible for health insurance reimbursement and may not continue services after the project ends, as the services would not generate revenue. 4. Strengthen rehabilitation capacities by expanding training courses to include longer-term certification programs, incorporating a monitoring system for quality and effectiveness. 5. Provide equipment to upgrade speech therapy services. 6. Provide more advanced training for service providers who work with children with autism. 7. Work with the GVN to improve accessibility of persons with disabilities to the existing government health services. Evaluation of USAID Vietnam Disabilities Programming vii Education Sector 1. Support GVN in establishing inclusive education for all children with disabilities, as provided for in Article 24 of the CRPD, which states that governments “shall ensure an inclusive educational system at all levels.” 2. Establish a transitional plan to transform special or segregated schools into inclusive educational programs to ensure that all children with disabilities receive education within the mainstream general educational system. 3. Work with GVN to dedicate more resources to providing community-based services and support for children with disabilities. 4. Work with GVN to implement standards for making schools physically accessible and with modifications for students with different disabilities. 5. Make new investments into training current teachers to provide appropriate curricula and support for children with intellectual and other disabilities. 6. Use future funding to promote best practices, rather than on programs and services that discriminate against persons with severe disabilities. 7. Work with the GVN to improve the physical accessibility of children with disabilities in all schools. Employment/Livelihood Sector 1. Work with stakeholders to improve collaboration among programs by the MOLISA, the provinces and the private sector to provide vocational training for persons with disabilities. 2. Continue to work with employers and the MOLISA to assess the labor market demand and identify and target additional vocational training that better match local labor market opportunities. If support for information technology (IT) training continues, strengthen it by providing more practical experience that matches the needs of potential employers. Proper training needs analysis and counseling for both persons with disabilities and employers before and after training would also be critical for the success of IT training and the inclusion of persons with disabilities. 3. Build career/employment counseling capacities at schools to facilitate a smooth transition to work for persons with disabilities. 4. Ensure that livelihood programs provide not only basic training and a minimum start-up seed grant, but also training in how to sustain the business, including long-term access to capital. 5. Work with the MOLISA, ESCs and DPOs to develop and implement additional job placement services for persons with disabilities, including a job board website, individual job referral support and job fairs that are better targeted to persons with disabilities. 6. Work with the MOLISA General Directorate of Vocational Training to strengthen and implement the NAP on Vocational Training for Persons with Disabilities. Evaluation of USAID Vietnam Disabilities Programming viii Civil Society Sector 1. Build capacity for DPOs in disability advocacy. This type of capacity building refers to teaching strategies, shaping leadership and advocacy skills and providing the tools to influence decision-making processes in public and social arenas to eliminate inequalities between persons with and without disabilities. The purpose of this type of capacity building is to achieve respect for diversity and difference and to contribute to cultural, social and political change for individuals’ full and active citizenship. 2. Support the expansion of disability civil society organizations. Identify organizations that are currently forming as an association. Build members’ capacity around disability advocacy. 3. Ensure that the contractor providing leadership and advocacy training is a civil society organization operated by and for persons with disabilities, preferably an NGO or international NGO (INGO) with an independent living or self-advocacy background. Service Delivery and Management 1. Work with the MOLISA, in coordination with the Department of Labor, Invalids and Social Affairs (DOLISA) and Provincial People’s Committees (PPCs), to strengthen, better disseminate and — where needed — train local officials on the Circular on Case Management. 2. Work with the GVN to modify case management practices to make it easier to incorporate case management into the government’s service delivery system. 3. Promote international best practices in case management to better advocate for persons with disabilities and to better coordinate efforts across the full range of needed services. 4. Employ a social worker with experience in establishing and implementing case management within a national service delivery system. This individual can review the case management curriculum to ensure that the curriculum can be a part of implementing the Circular on Case Management. 5. Continue to provide direct services, but include more planning prior to implementation. Work closely with USAID to apply procedures and policies that are more consistent with the local situation. Sustainability of Program Interventions 1. Support activities with practices that are inclusive and consistent with the CRPD to provide continuing guidance to the GVN on international best practices. 2. If pilot projects continue, strengthen implementing partners’ ability to monitor and evaluate progress and incorporate results into their work to ensure that the project is moving forward. 3. Future disabilities programming should have clear, realistic, reportable performance indicators, targets and results. 4. For future pilot projects, USAID should develop a strategy to use the monitoring and evaluation data to engage the central government in discussions on how USAID could play a role in supporting the government. 5. Programs should better document lessons learned from the implementation of the activities to guide implementation going forward. Evaluation of USAID Vietnam Disabilities Programming 1 I. OVERVIEW OF THE EVALUATION Since 1989, the United States Government (USG) has been assisting persons with disabilities in Vietnam. To date, the USG has contributed $60 million5 in assistance to persons with disabilities. Assistance has helped address the medical, educational and employment needs of persons with disabilities and supported their broader inclusion in Vietnamese society. The USAID disabilities programming assistance portfolio consisted of the following three projects: • Inclusion of Vietnamese with Disabilities (IVD), implemented by Catholic Relief Services (CRS). The initial purpose of the project was to improve the quality of and access to education for children with disabilities. It has grown, currently piloting assistive technologies within an educational setting and providing information technology (IT) training and employment assistance for young persons with disabilities. The project will end in 2015. • Inclusion of Vietnamese with Disabilities (IVD), implemented by Vietnam Assistance to the Handicapped (VNAH). The purpose of the project was to focus on the development and enforcement of Vietnam disability policies. The project will end in 2015. • Persons with Disabilities Support Program (PDSP), implemented by Development Alternatives Inc. (DAI). The purpose of the project was to meet the needs of Vietnamese with Disabilities (VWD) through comprehensive, integrated system. The project seeks to: (i) implement a sustainable and replicable case management and referral system; (ii) improve access to and quality of specialized services; (iii) strengthen the public health system to prevent and reduce the severity of disabilities; and (iv) expand the project’s strategy and models to other provinces. The performance period for the project is from 2012 to 2015. Purpose USAID/Vietnam asked the Vietnam Evaluation, Monitoring and Survey Services (VEMSS) Program to document the general “impact” and key contributions of the programs in building awareness, political will, the enabling environment, institutional capacity and operations and to identify current opportunities and gaps in disability programming that can serve as a base of understanding for the ongoing work of the Government of Vietnam (GVN), program implementers, disabled people’s organizations (DPOs) and other stakeholders. The evaluation was to examine project activities from 2005 to present. (See Annex 1: Evaluation Scope of Work.) Objectives and Key Evaluation Questions The objectives of the evaluation were to: a. Understand the extent to which the interventions have been consistent with beneficiaries’ needs, GVN government expectations and interests and USAID policies (Relevance). b. Identify/explain the causality of interventions and the impacts they have had on improving the policy environment and on the delivery of disabilities services (Effectiveness). c. Assess the benefits on the ground in the daily lives of persons with disabilities (Impact). d. Evaluate progress in sustaining positive impacts of the interventions and determine whether opportunity exists to further enhance the quality and quantity of the benefits of the programming (Sustainability). 5 Source: USAID Vietnam’s Disability Portfolio PowerPoint presentation, Hanoi, March 2015. Evaluation of USAID Vietnam Disabilities Programming 2 The key evaluation questions included the following: 1. To what extent has the programming assistance helped develop the GVN’s legal and policy environment, public awareness and support for disability services? 2. To what extent and in what ways have the programs improved the lives of and opportunities for persons with disabilities in Vietnam? Have the planned benefits been delivered and received as perceived by beneficiaries? 3. In terms of improving service delivery, did the USAID Disability Support Program provide an enabling environment and a supporting systems approach in technical assistance to strengthen critical institutional capacities? 4. Are the programs’ outcomes sustainable? What challenges in terms of sustainability remain? The remaining report is structured to provide the reader with a brief summary of the program (in Section II) and a detailed description of the methodology (in Section III). Section IV is divided into four major parts, which provide information to answer four key evaluation questions. Section V provides conclusions and proposes recommendations. Evaluation of USAID Vietnam Disabilities Programming 3 II. USAID DISABILITY PROGRAMMING IN VIETNAM Historical Overview Assisting persons with disabilities has been one of the top priorities for the U.S. Government (USG) in Vietnam.6 Since 1989, the USG has been improving the quality of life of individuals with disabilities and their families and supporting their inclusion into all aspects of society. From 1989 to 1998, USAID’s Disability Support Program focused on war legacy activities. The program was based on the traditional “medical model” with a humanitarian assistance approach. Activities focused on delivering direct assistance to persons with disabilities, such as health treatment, prosthetic and orthopedic services, assisted community-based rehabilitation, community vocational training and job placement and capacity building for health practitioners. With the addition of social activities in 1998, USAID based the program on the “needs-based model” with a social approach. Activities included disability policies development and enforcement, inclusive education and inclusive employment. In 2005, USAID based the implementation of activities on the “disability rights model” with an empowerment approach. Activities supported the protection of rights and full inclusion of persons with disabilities. Despite progress taking place in 2005, significant gaps still existed between the aim of the GVN’s disability policy, best practices and the actual assistance available to most persons with disabilities in Vietnam. In a 2005 assessment,7 authors stated: Little policy guidance has been developed and disseminated and actual implementation and enforcement of the Ordinance on Disabled Persons is weak. There is a lack of consensus and coordination amongst Vietnamese ministries regarding methods to implement the new laws and regulations, including how to meet the inclusive education commitment. Widespread lack of awareness of disability legislation still persists, as do negative attitudes about disability. Despite the great efforts made by GVN and donors from 2005 to 2013, many challenges remain:8,9  Longstanding relationships between donors and government officials at all levels appeared to be a necessary prerequisite to effective advocacy. However, maintaining those relationships at the district and commune levels was an outstanding issue.  A gap between policy and implementation resulted from underdeveloped policies that lacked evidence to give justification.  Implementation of laws and policies was not systematically monitored or evaluated.  Beneficiaries and implementing staff were not sufficiently involved in policy development.  Enforcement of laws and policies did not exist, thus failing to ensure the measure’s intended integration of persons with disabilities into Vietnamese society.  Limited multi-sectorial coordination took place.  Awareness and dissemination of the disability policies were lacking. 6 USAID’s disability programs address development objective 2 in the Country Development Cooperation Strategy, “Capacity Strengthened to Protect and Improve Health and Well-Being,” and intermediate result 2.3, “Expanded Opportunities for Vulnerable Populations.” 7 USAID. “Vietnam Disability Situation Assessment and Program Review.” Report prepared by Management Systems International (May 2005). 8 Ibid. 9 USAID/Vietnam. “Disability Projects Review Assessment and Analysis Report” (2013). Evaluation of USAID Vietnam Disabilities Programming 4 From 2005 through 2015, USAID adjusted its Disability Support Program to overcome these challenges. USAID’s assistance to Vietnam took the form of grants to three non-governmental organizations: Vietnam Assistance for the Handicapped (VNAH), Development Alternatives Inc. (DAI) and Catholic Relief Services (CRS). VNAH received USAID grants to implement the program “Inclusion of Vietnamese with Disability” from 2006 to 2014. The key focus of the VNAH-implemented program was to develop a strong legal and policy framework for disability inclusion and socioeconomic support. The program had three major components:  Law and policy development and implementation;  Strengthening national coordination for promoting and supporting the development and implementation of disability inclusion; and  Inclusive vocational training and employment through the piloting of a holistic, integrated service delivery approach on business development. CRS managed and implemented a separate award under the similarly named program “Inclusion of Vietnamese with Disabilities” from 2006 to 2014. The program focused on four activities:  Enhanced educational support for children with disabilities through improved and expanded inclusive education;  Ensuring that Vietnamese persons with disabilities have equal access to meaningful employment opportunities;  Developing and implementing policies and regulations related to inclusive education and vocational training/employment for children and adults with disabilities; and  Increasing participation of and support to disability self-help groups. In 2012, DAI started a three-year program. The program focused on four activities:  Assisting GVN with developing and implementing a social work case management program;  Improving the quality of and access to specialized services, including assistance to develop disability policies and direct services delivery to persons with disabilities;  Strengthening the public health system with the development and implementation of a birth￾defect surveillance system, newborn screening and pre-conception counseling; and  Replicating the project’s strategy and models in other provinces. Current Portfolio From 2008 to the present, USAID’s portfolio has included a comprehensive set of activities that address the disability system with a holistic approach to strengthening the legal framework, supporting law enforcement at subnational levels, developing human resources, fostering interagency cooperation and supporting DPOs. Additionally, USAID introduced a comprehensive and integrated disability service model addressing three crosscutting issues: capacity building, gender and strategic information. Table 1 presents a summary of current USAID activities by sector and implementing partner. Evaluation of USAID Vietnam Disabilities Programming 5 Table 1: Type of USAID Activities by Sector and by Implementing Partner Sectors/Types of Activities IPs CRS DAI VNAH LEGAL AND POLICY FRAMEWORK Policy Development (at central, provincial or district level) X X X Implementation (at central, provincial or district level) X X X Establishment of the National Coordinating Council on Disabilities X Capacity Building/Institutional Strengthening Study tours, technical assistance for drafting team X X HEALTH/MEDICAL Direct Services Newborn screening / birth defects surveillance X Pre-conception counseling X Physical rehabilitation (i.e., physical therapy [PT]) X (1) X X Speech therapy X Corrective surgery X (1) X Medical equipment and devices (individual beneficiaries) X (1) X Capacity Building/Institutional Strengthening Rehabilitation training for medical doctors and technicians X X Equipment for newborn screening (Danang) X Handbook, training and software to manage newborn screening data X Established rehabilitation unit in the Mothers and Children Hospital X Medical equipment for rehabilitation unit and in district hospitals X EDUCATION Direct Services Inclusive education X X Special education (stipends) X Home-based education X Online / distance learning X Capacity Building/Institutional Strengthening Training to regular and special education teachers: inclusive education X X Training to regular education managers: inclusive education X X Training to parents (e.g., PT, home-based education, homework) X X Resource rooms (equipment) X X Equipment for special education classrooms X X Curriculum – teachers’ / education managers’ training, online and distance learning courses X X Evaluation of USAID Vietnam Disabilities Programming 6 Sectors/Types of Activities IPs CRS DAI VNAH EMPLOYMENT/LIVELIHOOD Direct Services Vocational rehabilitation X Job placement X X X Vocational and employment counseling at employment service centers X X Vocational training courses X X X Integrated job fairs X X Start-up business kits X X Small, interest-free loans to start businesses X Animal husbandry X Capacity Building/Institutional Strengthening Established Blue Ribbon Employer Council (BREC) X Equipment for Hanoi Employment Service Center X SOCIAL INCLUSION Direct Services Case management X Housing renovation X Disabled People’s Organizations and Parent Associations Establishment of organizations / associations X (1) X X Capacity building for organizations / associations X (1) X X Public awareness campaigns X X (1) On a cost-share basis. Evaluation of USAID Vietnam Disabilities Programming 7 III. METHODOLOGY Qualitative Research Methods The evaluation used qualitative research methods. These methods are appropriate for the evaluation because the purpose of the evaluation was to seek a wide understanding of the entire disabilities program. These methods addressed people’s experiences of receiving support and assistance from the program. Moreover, the methods produced in-depth, comprehensive information and used subjective information and participant observation to describe the context of the variables under consideration, as well as the interactions of the different variables within the context. Quantitative Research Methods Although the key research methods used in this evaluation were qualitative, quantitative research methods supplemented the approach. A survey of beneficiaries enabled collection of the underlying data on the VWD situation in the selected provinces. Quantitative research is inappropriate to use as the key method for this evaluation because it relies on measureable data to formulate facts and uncover patterns in research. Since the purpose of the evaluation is not to quantify attitudes, opinions or behaviors and generalize the results from a large sample population, the evaluation used qualitative research methods. Annex 2 provides additional information about the differences between qualitative and quantitative research to further support the use of qualitative research methods for this evaluation. Timeframe and Topics USAID requested that the evaluation cover work from 2005 to 2015. Additionally, USAID requested that the evaluation focus on the legal framework, civil society, health/medical, education, employment/livelihood and service delivery and management sectors. Geographic Location USAID activities occurred within the central government and 13 provinces (see Figure 1). This evaluation focused on four provinces based on poverty rate, disability prevalence, the type of key activities being implemented and consultations with USAID. Annex 3 provides a summary of supported program sites by province, poverty rate, disability prevalence, key activities, implementing partner, sub￾grantees and other partners and stakeholders. The four provinces selected are Hanoi, Danang, Quang Nam and Dong Nai. Two districts from each province were selected. Selection of districts was based on population density (urban vs. rural) and the type of key activities being implemented in the province. Table 2: Selected Provinces and Districts Locations Poverty Rate Disability Prevalence Children (5–15 yrs.) Adults (6–59 yrs.) Hanoi Province (Ba Dinh-Urban, Hoan Kiem-Urban Districts) 6.44% 1.18 3.23 Danang Province (Cam Le-Rural, Hoa Vang-Rural Districts) 2.39% 2.39 6 Quang Nam Province (Hoi An-Urban, Hiep Duc-Rural Districts) 23.47% 1.97 5.53 Dong Nai Province (Bien Hoa-Urban, Vinh Cuu-Rural Districts) 11.73% 1.5 5.86 Evaluation of USAID Vietnam Disabilities Programming 8 Figure 1: Geographic Locations of USAID Disability Activities Key Informants Seven types of key informants were identified for each province and district: 1) implementing partners, 2) policymakers, 3) policy implementers, 4) service providers, 5) private sector (e.g., employers), 6) beneficiaries and 7) civil society organizations. Annex 4 contains a list of key informants. Evaluation Team The evaluation team includes four individuals. The team leader is from the United States and has expertise in intellectual and other developmental disabilities, disability policy, community integration, empowerment and the self-advocacy movement. Additionally, she has a strong background in quantitative and qualitative research. The other three members are from Vietnam and have expertise in the Vietnam disability service system, capacity building, disability policy, the legal framework of Vietnam, vocational rehabilitation and employment and research and statistics. One research assistant provided administrative and logistical support and verbal and written translation services. Procedures The evaluation was based on an analysis of primarily qualitative data obtained from the key informant interviews (KIIs), supported by document reviews (desk review, reviews of independent data sources, findings and secondary data from an independent quantitative evaluation conducted by another source). Data synthesis and analysis involved a seven-step procedure. These steps were the following: Step 1: Transcription: Team members transcribed KIIs into English. CURRENT USAID DISABILITY PROGRAMS IN VIETNAM Evaluation of USAID Vietnam Disabilities Programming 9 Step 2: Initial Findings: Team members met to discuss and summarize initial impressions/findings into the six sectors (legal and policy, civil society development, inclusive education, employment/livelihood, health and medical, and service delivery and management). Step 3: Confirmation of Initial Findings: The team leader reviewed the data from the KIIs to determine if the data were consistent with the initial findings. Step 4: Follow-up of Findings: Where data did not support the initial findings, the team met again to determine if any data from the KIIs supported the finding in question. Step 5: Incorporation of Documentation Reviews: Data from the desk review and the review of independent data sources was incorporated into the findings. Step 6: Confirmation of Findings: Where inconsistency emerged between the document review data and the findings, team members contacted the relevant stakeholders to confirm if data were correct. Step 7: Finalize Conclusions. Once the findings were verified, the team drew conclusions and incorporated them into the final report. Key Informant Interviews The purpose of key informant interviews was to collect information from a wide range of people — including government officials, community leaders, professionals and beneficiaries — who had first-hand knowledge about the program. These program experts, with their particular knowledge and understanding, provided insight on the nature of problems and gave recommendations for solutions. Additional reasons for conducting KIIs were to:  Obtain information about any pressing issues or problems in the community from a limited number of well-connected and informed community experts;  Understand the motivation and beliefs of participants on any particular issue;  Get information from people with diverse backgrounds and opinions and be able to ask in-depth and probing questions;  Discuss sensitive topics, get respondents’ candid discussion of the topic or get the depth of information required, as individual discussions create a comfortable environment where individuals can have a frank and open discussion; and  Get more candid or in-depth answers than other methods provided. The team conducted face-to-face interviews. The team leader and three team members divided into two interview units. Each unit had one male and one female interviewer. Each unit had a leader. Prior to each interview, one person was assigned to conduct the interview and the other person took notes. Document Reviews Document reviews determined how the program operated without interrupting the program. This included a desk review of program documents and a review of independent data sources. The document reviews enabled the evaluation team to obtain a full range and depth of information. Additionally, multiple sources of data enabled the team members to triangulate the KII findings. Desk Review. Each team member was assigned documents from USAID and the three implementing partners (IPs) to review and summarize findings. Only documents related to activities conducted from 2005 to 2015 were included (e.g., quarterly and annual reports, performance evaluations). The lead national team member incorporated all findings into the final desk review. Evaluation of USAID Vietnam Disabilities Programming 10 Review of Independent Data Sources. Each team member was assigned an independent data source to review. Data from government reports — e.g., 2013 General Statistics Office, National Survey on Disability, Ministry of Labor, Invalids and Social Affairs (MOLISA) data (on website), Ministry of Health (on website), National Coordinating Council for Disabilities (NCDD) and Ministry of Education and Training (MOET) — implementing partners (e.g., Disability Information System from VNAH) and international development partners (e.g., World Bank, UNICEF and the International Labour Organization [ILO]) were reviewed. Each person reviewed and summarized the findings and the team leader incorporated the findings into the final report. Survey of Beneficiaries. The Mekong Delta Research Institute, on contract with VEMSS, independently conducted a cross-sectional survey of 850 direct beneficiary persons with disabilities who received USAID assistance to ascertain their satisfaction with services provided and the effects of the assistance in the lives. The results are summarized throughout this report in Figures 2–10. The survey included beneficiaries who received direct assistance from USAID from 2010–2015. Although the targeted recipients of a number of USAID activities (e.g., training to take care of persons with disabilities, capacity building in inclusive education) are the caretakers and service providers instead of persons with disabilities themselves, the latter are the ultimate targeted beneficiaries. Therefore, the respondents of the survey were all persons with disabilities. In situations where the persons with disabilities were not able to fully articulate their responses, their caretakers answered the questionnaire on their behalf. USAID direct assistance covers an array of activities across time, and of different scales both in terms of the frequency of service provision and the level of services received. From the list of beneficiaries and the respective services that beneficiaries received, the survey categorized services into three main groups: medical support services, educational services and employment/livelihood support services. Any direct assistance that the persons with disabilities received could be classified under one of these groups. The time reference for the survey allows the respondents to recall the received support and the changes it may have made on their lives. It would have been very difficult to track recipients from ten years prior. This time frame however excludes some of the previous interventions that may have had significant impacts. Evaluation Schedule Planning started the week of March 9 and continued through the week of March 23. During the week of March 23, initial KIIs were conducted in Hanoi. The week of March 30, 2015, the teams conducted KIIs in Danang Province. The week of April 6, one team traveled to Quang Nam Province and the second team went to Nong Nai Province to conduct interviews. During the weeks of April 13 and April 20, team members completed the Hanoi KIIs and conducted follow-up interviews with the three IPs. During those weeks, team members finished transcribing the KIIs, worked on the document reviews and met for two days to discuss initial findings. Annex 5 summarizes a detailed evaluation schedule. Data Collection Instruments Key Informant Interview Guides. The evaluation team designed interview guides (see Annex 6) for each key informant. The guides included semi-structured questions to determine the relevance, effectiveness, impact and sustainability of the program. Additionally, open-ended questions elicited key stakeholders’ views on USAID programming and how it evolved, and on the programming’s progress toward achieving its goals and objectives. Evaluation of USAID Vietnam Disabilities Programming 11 Although each guide included questions directly related to the key informants’ areas of expertise, it also included the following key questions: 1. What are the KIIs’ roles and responsibilities within the project? 2. How were policies developed, implemented and monitored from the central level to the commune level? 3. To what extent did the project improve the lives of and opportunities for persons with disabilities? 4. What were the planned benefits of the project? Were these benefits delivered? 5. Have the planned benefits been delivered and received as perceived by beneficiaries? 6. To what extent has the programming assistance helped to develop the GVN government legal and policy environment, public awareness and support for disability services? 7. In terms of improving service delivery, did the program, across projects, provide an enabling environment to strengthen critical institutional capacities? 8. In terms of improving service delivery, did the program, across projects, provide supporting systems approach in technical assistance to strengthen critical institutional capacities? 9. Are the programs’ outcomes sustainable? What challenges in terms of sustainability remain? 10. What gaps in service delivery are still present? The interview guides assisted the interview teams to direct conversations toward the topics and issues they wanted to learn about. They also helped the interviewer know what to ask about, in what sequence, how to pose the questions and how to pose follow-ups. Moreover, they provided guidance about what to do or say next, after the key informant had answered the previous question. The interview guides offered techniques to accommodate four important facts of human social interactions that influence what people are likely to say to an interviewer. These facts are: (1) research questions are not the same as interview questions, (2) people’s espoused theories differ from their theories in use, (3) interviews are social occasions and (4) testimony by itself is a relatively weak form of evidence. Desk Review and Independent Data Source Form. For each document or data source reviewed, team members completed a form to summarize information related to the four key evaluation questions (see Annex 7). Evaluation of USAID Vietnam Disabilities Programming 12 IV. FINDINGS Findings in this section draw upon evidence from key informant interviews, a desk review of key project documents and the survey of beneficiaries. This section will first discuss and analyze the relevance of the USAID disabilities programming. Next, the findings are organized by the four key questions of the evaluation design relating to improved lives and opportunities, the development of the legal and policy environment, strengthening critical institutional capacities and sustainability of program interventions. Relevance of the USAID Disabilities Programming Vietnam has a large number of persons with disabilities. In 2014, 6.7 million persons with disabilities lived in Vietnam, and 1.2 million of them were children. Vietnam has a long history of supporting persons with disabilities, starting with the establishment of the National Association of the Blind in 1969. In recent years, the government of Vietnam has worked to improve the legal and policy environment and the public service delivery system to facilitate the inclusion of persons with disabilities as full contributing citizens. Examples include the enactment of the Law on Persons with Disabilities, the Convention on the Rights of Persons with Disabilities (CRPD) ratification, inclusion of persons with disabilities in the social protection schemes and formulation of various national action plans for their support. In this context, the USAID-supported disability programs that bring technical assistance and funds to support inclusion of Vietnamese with disabilities align with the interests and policies of the Government of Vietnam (GVN). As stated in USAID’s policies, the Agency is committed to disability-inclusive development and working across several fields, including inclusion in political processes, improving design of health and emergency services, education, job training and economic empowerment. In the area of disability, USAID applies the twin-tracked approach in which it 1) supports disability-specific programs to address targeted needs and 2) seeks to integrate disability into all of its programs. In short, the interventions have been consistent with beneficiaries’ needs and the expectations and interests of the Government of Vietnam, as well as USAID policies. Improving Lives and Opportunities USAID-provided assistance has improved the lives of persons with disabilities: A majority of beneficiaries surveyed said that their life had improved since receiving medical (55 percent), educational (65 percent) and livelihood support (69 percent) services. Key results include:  The enacted Law on People with Disabilities and subsequent public awareness activities supported with USAID assistance increased the acceptance of persons with disabilities within the community.  Beneficiaries of medical and health care assistance reported improved ability to move around and travel and are better able to care for themselves, acquire new knowledge and join activities.  Of all beneficiaries who received educational assistance, almost 50 percent of caregivers of young students reported improved learning capacity. More than 40 percent said the students were better able to attend school and had more confidence in their educational pursuits. Key Question 1: To what extent and in what ways have the programs improved the lives and opportunities for persons with disabilities in Vietnam? Have the planned benefits been delivered and received as perceived by beneficiaries? Evaluation of USAID Vietnam Disabilities Programming 13 Figure 2: Impact of Assistance on the Lives of Persons with Disabilities 0 10 20 30 40 50 60 70 80 Medical Assistance Education Assistance Livelihood Assistance Total Better Worse Same  Job training and support services have given persons with disabilities new and useful skills to help find employment. Nearly 40 percent of persons with disabilities trained in information technology (IT) found jobs within six months after completing the training. Acceptance and Service Delivery Improve with Better Legal and Policy Environment Framework: USAID, primarily through Vietnam Assistance for the Handicapped (VNAH), supported the ratification of CRPD and key national policies through a multifaceted approach (see findings for Key Question 2). As a result, GVN drafted and enacted 24 laws, decrees and circulars in support of persons with disabilities. Findings indicate that the policies created a strong foundation to improve the lives of persons with disabilities. However, contradictions between the National Law on People with Disabilities, the educational policies and the CRPD impede the implementation of the law. As a result, this impedes the ability of all persons with disabilities to participate fully in society, improve their lives and benefit from opportunities that their non-disabled counterparts experience. (See findings for Key Question 2.) Feedback from key informants within government institutions suggests that the policies were well￾designed and relevant. Former Committee on Social Affairs (CSA) members said the support from development partners (e.g., USAID grantee VNAH and UNICEF) ensured that the quality of the drafted law was practical and reflected the needs of persons with disabilities. USAID’s contribution via experts, workshops, study tours and other types of technical assistance and support were described as beneficial and helped to achieve consensus from National Assembly members. CSA members indicated that the policies also translated into tangible benefits for persons with disabilities. They noted that the disability laws and policies resulted in more persons with disabilities being accepted into higher education institutions and working within the government. For example, as a result of exemptions on the requirement to take an entrance examination for children with disabilities, more universities are now accepting students with disabilities. Figure 3: Beneficiaries Reporting Improvement in Selected Life Domains 0% 10% 20% 30% 40% Skills to Raise Income Stable Job Ability to Live Independently Family Financial Status Mental Health Physical Health Evaluation of USAID Vietnam Disabilities Programming 14 Figure 4: Beneficiaries’ Awareness of the National Law on People with Disabilities 0% 10% 20% 30% 40% 50% 60% 70% Aware of It's Existance Aware of It's Contents Aware Not Aware of its existence Aware of its contents Not aware Public awareness activities helped increase the acceptance of persons with disabilities within the community. The staff from the Social Protection Department (SPD) confirmed that the social behavior of community members toward persons with disabilities positively changed. For example: the term “handicapped” (nguoi tan tat in Vietnamese), which fuels stigmatization, is now rarely used in public communication. Persons with disabilities noted in interviews at a district in Danang that they and their families have become more confident as community attitudes have changed for the better. SPD staff noted that the living environments of disabled persons have improved, accompanied by an improvement in employment offers. A key informant in Danang noted that current law and policies give person with disabilities more assistance for their daily lives, including social welfare from the state budget, health insurance, livelihood support and free legal counseling. Key informants noted a gap in awareness of the law and its contents for persons with disabilities who are not members of disabled people’s organizations (DPOs). This aligned with survey findings: Only 20 percent of respondents were aware of a law or policy that could help empower and bring them more opportunities. Of those, only 11 percent could describe specific content of the law. Assistance for health and medical services: Health and medical assistance came primarily through direct medical services to beneficiaries and through material and technical support to health facilities. USAID programming provided some beneficiaries with access to medical services that they may not otherwise have had. Support services include clinical examinations, physical therapy, prostheses/orthopedic devices, speech therapy and corrective surgical procedures, as well as stipends to cover living expenses. The project also provided assistive devices such as wheelchairs and hearing aids. Table 3: Number of Beneficiaries of Medical Assistance Provided by DAI, 2013–2015 Types of Medical Assistance Danang Dong Nai Binh Dinh Quang Nam Total Clinical exam 690 533 2,010 0 3,233 Physical therapy and clinical exam 661 0 120 0 781 Speech therapy 154 0 0 92 246 Hearing aids and clinical exam 87 125 48 0 260 Prostheses/orthoses and clinical exam 130 32 0 0 162 Corrective surgery and clinical exam 8 7 0 0 15 Total 1,730 697 2,178 92 4,697 Evaluation of USAID Vietnam Disabilities Programming 15 Figure 5: Impact on the Quality of Life for Beneficiaries of Medical Assistance 0% 20% 40% 60% 80% Move & Travel Acquire New Knowledge Self-Care & Do Housework Join More Activities Better No Change Worse Evidence from the survey of persons with disabilities receiving medical assistance (Figure 5) shows that assistance via health and medical services has been beneficial. Almost 40 percent of recipients of medical assistance surveyed reported that their ability to move around and travel had improved. More than 20 percent were better able to care for themselves, acquire new knowledge or join more activities. However, field research found many implementation challenges, including programming gaps, insufficient investment in rehabilitation services and facilities that directly benefit persons with disabilities, long delays for certain medical services and problems with determining eligibility for needed services. Only one-third of survey respondents reported actual improvements in their physical and mental health. Nearly two-thirds (63 percent) of beneficiaries receiving medical assistance said they needed additional health or medical support, suggesting a gap in meeting needs with the available services. In the field, numerous reports cited delays in receiving corrective surgery and other types of medical support. According to case managers, delays ranged from six to 12 months. In one district, 145 persons with disabilities required medical support. After one year of implementation, the Persons with Disabilities Support Program (PDSP) provided services — mostly basic services such as medical examinations — for about half of them. The leader of that district expressed his concern about the long approval process of both PDSP and government agencies such as DOLISA. Key informants indicated many challenges in implementation, including insufficient investment in rehabilitation services and facilities that directly benefit persons with disabilities, problems with determining eligibility for needed services and, once eligibility was determined, long delays in obtaining approval for medical procedures. Key informants reported insufficient investment in rehabilitation services and facilities that directly benefit persons with disabilities. One senior official noted that health service providers did not want to invest in equipment for rehabilitation because it “does not make money.” Health insurance covered only 45 operations and techniques,10 and hospital administrators did not want to invest in rehabilitation equipment or services because they are not always reimbursed by insurance. The recent budget reprioritization by provincial and district hospitals indicate that rehabilitation is not a priority, resuling in an insufficient investment in rehabilitation services and facilities that directly benefit persons with disabilities. Determining eligibility appeared to be a major challenge to the implementation of medical services. Key informants also identified issues in accurately determining who is eligible to receive corrective surgery. The Department of Health (DOH) uses a tool to identify persons with disabilities and assess the severity of their disability. Only persons with “severe” or “very severe” disabilities are eligible for health 10 Per circular No. 50/2014 issued by the MOH, the ministry accepts 248 operations and techniques for rehabilitation. Of these, health insurance pays for only 45, and typically covers only a portion of the fee. Evaluation of USAID Vietnam Disabilities Programming 16 insurance, and the agreement between the DOH and PDSP states that PDSP will pay the portion of surgery costs that medical insurance does not cover. Some officials at the national level are concerned that the tool is not valid or reliable. As a result, some persons with disabilities may not be considered disabled, or may have their level of disability incorrectly classified. Other government officials at the national and commune levels indicated that commune officials lack the expertise, knowledge and tools to accurately identify persons with disabilities or the type and severity of their disabilities. Key informants at the commune level confirmed that they have difficulties with the tool and are uncomfortable using it. Given the concerns expressed about the tool used to classify persons with disabilities, it’s unclear if it results in potential beneficiaries being inadvertently excluded. Even for those who are eligible for health insurance, the process to obtain approval to receive services is complex and time-consuming. One administrator cited challenges in working with family members of the person with a disability, saying they often did not complete all of the administrative steps needed for surgery. Key informants gave a variety of reasons for delays in the implementation process, such as 1) delays in Development Alternatives Inc. (DAI) obtaining approval to work in provinces, 2) insufficient amount of time to implement activities, 3) PDSP management waiting for direction from its headquarters on how to fund corrective surgery (i.e., grant vs. direct payment), 4) PDSP waiting to receive a list of people who were to receive corrective surgery from the Department of Health and 5) lack of mapping and planning prior to implementation. As indicated in the example that follows (“Interviews with Parent and Teacher of a Child with Disabilities in an Inclusive School”), incidents of poor judgment also emerged. In addition to delays in medical assistance, another issue with medical assistance is the lack of support to transport patients to a service facility. One woman had two disabled children; she was able to take only one child to therapy at a time by pushing him in his wheelchair 20 minutes each way. Though students with disabilities receive stipends in part to help pay for transportation costs to get to school, transportation to medical facilities is not generally covered. Interviews with Parent and Teacher of a Child with Disabilities in an Inclusive School DA was a fifth-grader with development difficulties in a primary school receiving Catholic Relief Services (CRS) support for inclusive education. Until third grade, DA had a speech disorder and struggled to speak. Parents and teachers agree that DA also had difficulty studying. He had no friends and talked of wanting to drop out of school. The school and family worked together on an individual learning plan. Every day, teachers helped him with studying after class so he could keep up with his classmates. Teachers also worked with his family to provide extra help with his homework. Teachers and parents kept in regular communication on DA’s issues and progress. Where once he had no friends at school, now DA has become much more socialized. He has friends in class who help him with studying; some even come to his house to help. His parents are proud of him for being kind to people around him. He does not hesitate to share whatever he has with friends. DA has made great progress and has been awarded a certificate of achievement every school year. DA’s parents appreciate the devoted support from his school and teachers. In a meeting with the evaluation team, the school’s teachers demonstrated that the skills and knowledge they obtained from the CRS-supported training were very helpful and played a critical role in their approach to work with DA and other students with disabilities. Evaluation of USAID Vietnam Disabilities Programming 17 Education assistance: USAID education assistance focused on expanding the opportunities and capacities of providing inclusive education to children with disabilities, and home schooling or specialized education as needed. Assistance has primarily included stipends for children with disabilities to access education by helping to pay for fees, transportation and other costs. As well, caregivers have received training to help them better prepare children for schooling. Beneficiaries surveyed as part of the field research said they were satisfied with the education programming (61 percent); 40 percent said that they needed additional educational support. In terms of quality-of-life improvements for beneficiaries of educational assistance, almost 50 percent of those surveyed (caregivers for young students) reported improvement in learning capacity, and more than 40 percent said their ability to attend school and confidence in their educational pursuits improved. (See Figure 6.) Teachers, school managers and other key informants said children with disabilities were more confident and had a more positive attitude and behavior toward studying after their educational experiences. When asked about USAID assistance for their children, parents named and valued multiple supports they received: inclusive education in regular school, individual education plans, stipends, scholarships and especially the cooperation between school teachers and family members that USAID assistance fostered. Caregivers’ training was also noted as beneficial in helping them deal with daily challenges. Personnel observed that after parents learned about it via training activities on home-based education, homework and the like for disabled learners, more parents believed in and supported inclusive education. Some children received home-based education from the Catholic Relief Services (CRS) program and showed improvement in their life skills. Parents benefited as they became more involved in their respective parent associations and kept in better contact with teachers to follow their child’s learning progress. Currently no home-based schooling exists in project areas. The teachers may coordinate with the parents, but the parents do not receive curricula to use at home. Parents are expected to call the school if students have any problems. Without that call, the school assumes that everything is all right. When asked if the school does anything for these children, the Department of Education and Training (DOET) said the school maintains contact with the parents who report any problems. Employment/Livelihoods: USAID employment and livelihood assistance consisted primarily of vocational training and job orientation and placement support to help persons with disabilities prepare for and find jobs. In addition to direct support for students with disabilities, training of teachers and caregivers was provided. Significant positive impacts were identified in the survey of beneficiaries receiving employment/livelihood assistance, shown in Figure 7. More than 50 percent reported an improved capacity to perform work 0% 20% 40% 60% 80% School Attendance Learning Capacity Confidence Ability to Attend Events Ability to Make Friends Better No Change Worse Figure 6: Impact on the Quality of Life for Beneficiaries of Education Assistance Evaluation of USAID Vietnam Disabilities Programming 18 and more than 40 percent improved their ability to find a job, perceived better inclusion in the work environment and improved their income and time worked. A number of key informants described the school-to￾work model, which helped prepare children with disabilities for their working life, as an effective approach. Dong Nai Province implemented this model. In 2013, 25 teachers, students and parents received training on school-to-work transition services, including needs assessment, counseling and networking with community service providers; 39 youth with disabilities received support to pursue vocational programs as part of this model. Through 2014, 21 children at the School for Disabled Children (SDC) received vocational training in sewing and repairing sewing machines, and 11 of them found jobs based on the services provided. The school also followed up on 52 former students/graduates. The data shows that employment and livelihood programming is among the most appreciated disabilities programming initiatives. SDC personnel described the school-to-work project as very successful, commenting that the graduates obtained skills they could use to find work and earn a living. They observed that the graduates and parents became more confident about work and employment. SDC personnel noted that as a result, former students have opportunities to be more fully included in the community. VNAH/DAI implemented a school-to-work project by providing vocational training to youth at special education schools. The special education school at Dong Nai participated in the school-to-work project. Since 2008, 82 students have attended the vocational training and graduated from the school. Of those 82, 11 continued to higher levels of education or continued training. The majority (n=71) are working in jobs such as seamstress, shoemaker, baker, masseur, musician and goldsmith. Dong A University provided vocational training on IT for persons with disabilities and linked them to employers. Potential students learned of the opportunity through DPOs, CRS and public announcements. Students were selected through individual interviews and were accepted if they had graduated from a secondary school. 0% 20% 40% 60% Ability to Find a Suitable Job Capacity to Perform Work Improvement in Time Worked Income Improvement Inclusion in Work Environment Better No Change Worse Figure 7: Impact on the Quality of Life for Beneficiaries of Employment/Livelihood Assistance Cam Le Woman Sets Up Her Own Shop Ms. D set up a food stall and grocery store at her home. She is married and has two children and reports being very satisfied with the support that she received through the case management project. Her case manager assessed what she needed and helped her apply for the funding. She received the start-up money with no problems. Ms. D finds the livelihood allows her to manage to earn extra income for the family and take care of her children at the same time. Her only suggestion to improve the project is to increase the start-up fund. She received VND 5 million, and it was not enough to buy all of the products she needed. She borrowed more money. Evaluation of USAID Vietnam Disabilities Programming 19 The training lasted six months, but students with disabilities often took more time to complete it. In addition to IT training, students with disabilities received soft skills training (e.g., how to prepare job applications). During the course, students were sent to companies that employ persons with disabilities for either short visits or to participate in an internship program in which they worked on simple tasks. After the training, students were placed in jobs through job fairs or the provincial Employment Service Center (ESC). Of 131 people enrolled for the training, 111 finished the course, including 68 who received certificates from the university. A total of 46 people secured stable jobs, equaling 35.1 percent of the individuals who enrolled in the training and 41.4 percent of the individuals who finished the courses. Support to the Blue Ribbon Employer Council (BREC) promotes the contribution and efforts of employers hiring persons with disabilities and workers with disabilities. Activities to raise the awareness of businesses are integrated in job fairs, and events such as the BREC’s award and recognition ceremony honor the achievements of socially responsible employers. A number of key informants commented on the importance of USAID assistance in raising awareness, which ultimately impacts the lives of beneficiaries by expanding job opportunities for persons with disabilities. Self-employment projects, which included a combination of training and seed money, provided livelihood opportunities. Persons with disabilities, especially youth, enrolled in short-term vocational training courses with local enterprises or vocational training institutions. Once they completed the business training, they typically received equipment or toolkits to open a small business. This type of support often took a community-based approach, although in some interventions, the person attended training at the provincial level and returned to his or her home community to set up a business. However, beneficiaries cited issues with the post-training package; one did not receive the toolkit he was promised following his training and another said she did not receive sufficient funds to purchase inventory and had to borrow additional funds to start up her small retail business. Efforts to train recipients to sustain their new businesses were considered weak. Interview with Small Businessman in Hiep Duc District Mr. Q worked with an employer for one year to learn how to repair motorbikes. World Concern/CRS provided financial support to him and his employer for one year. He received a machine to wash the bikes, but did not receive the toolkit he was promised during the training. The team verified that another person who went through the same training received a toolkit and the machine to wash bikes. Mr. Q decided he wanted more training and arranged one more year of training with another employer. Mr. Q set up his shop in 2009 and has been successful. He earns enough income to meet the family’s daily expenses, to pay for the remaining surgeries to correct his cleft palate, to save money and to buy tools. Although he has a lot of clients, sometimes he loses clients because he does not have the tools to fix the bikes. Some clients do not want to wait while he goes to his former employers’ shops to borrow the tools. When the team talked to Mr. Q’s mother, she said that he used to have low self-esteem, was very shy and refused to go out. Now he is more confident and feels he can be a helpful member of society. He said he is very satisfied with the training. His only suggestion to improve the training is to for USAID to create a mechanism to ensure that participants receive toolkits. Evaluation of USAID Vietnam Disabilities Programming 20 Several key informants said they benefited from the project by obtaining skills needed to start businesses or to gain employment. As a result, beneficiaries earned money for themselves and, in some cases, for their families. The mother of one adult with a disability, who opened a grocery store, stated that family members received indirect benefits. As the income from the store became stable, everyone in the household felt more financially secure and as a result felt less stress. Furthermore, as her son became more independent, she found more free time to seek work outside the home. Another parent of a disabled adult said that livelihood projects showed the community that persons with disabilities are capable and can contribute to the community. Consequently, community members and commune officers have become more accepting of persons with disabilities. Development of the Legal and Policy Environment USAID, primarily through VNAH, has supported the ratification of CRPD and key national policy formulation, implementation and coordination in the past 10 years. USAID worked closely with GVN officials to provide a wide range of activities that included legal analysis and review, consultative workshops, technical focus groups, in-country training, study tours to the United States, consultations by international and national experts, public awareness activities and technical assistance to finalize laws and policies. Key results include: • USAID assistance has significantly helped develop the Law on People with Disabilities and related laws and policies that now secure the legal rights of persons with disabilities in Vietnam and expand opportunities for their health care, education and employment. Interview with Employee of Vietnam Business Process Outsourcing (VBPO) Company Ms. L attended the CRS-supported IT course from September 2014 to March 2015. During those months, she worked part-time as a designer for VBPO and attended training during the second half of the day. CRS paid for a part of the training fee (VND 12 million to the company; she paid the remaining VND 3 million through monthly salary deductions). VNAH paid for a travel allowance (VND 900,000/month for six months). At the end of the training, participants were required to pass a recruitment test. She noted that if persons with a disability did not pass the recruitment test, they had a probationary period during which they could study and re-take the test. Before coming to work for VBPO, she worked as a commercial designer in a design company. Ms. L has been employed by VBPO since March 2015. Her job tasks include data entry and design. She acquired the design skills prior to the CRS-supported courses. Ms. L said she was “very satisfied” with the training course and believes it was practical. Ms. L said a three-month course was suitable to achieve the speed and error requirements. Ms. L made two suggestions to improve the course. First, she noted a need for stricter management for the entire course to enable students to learn more quickly and take less time to reach required standards. Second, the trainers should speak in a more “gentle way.” “Persons with disabilities do not have exposure to society and can be shocked and lose courage when being shouted at.” Key Question 2: To what extent has the programming assistance helped develop the GVN’s legal and policy environment, public awareness and support for disability services? Evaluation of USAID Vietnam Disabilities Programming 21 • Person with disabilities have a broader voice to advocate on their own behalf as a result of USAID assistance to DPOs. • Continued focused assistance to build GVN capacities to implement the law is required, particularly at the local level of government. Assistance to develop laws and policies that support persons with disabilities. USAID programs provided crucial financial and technical assistance to support the GVN in drafting and enacting the legal framework on disabilities in Vietnam. This effort resulted in the passage of 24 legal documents, including the Law on People with Disabilities, and various implementing decrees and circulars in health care, education, vocational/employment and social inclusion sectors in support of persons with disabilities. (See Table 4.) Together, these initiatives established a platform to help secure and improve the rights of and opportunities for persons with disabilities in Vietnamese society. Highlights include: • The Law on People with Disabilities has shifted Vietnam’s policies from a charity and medical model of disability to a social- and rights-based model. The new law provides a strong foundation to protect and promote the rights and opportunities for persons with disabilities in Vietnam. • Decree 28 provides the critical guidelines needed to properly implement the law across various ministries and departments of the GVN. • The Vocational Training Law grants tax provisions to organizations that train persons with disabilities. • The MOET decrees mandate training for teachers and education managers to acquire the skills necessary to provide inclusive education. • The Labor Code Law regulates the employment of people with disabilities, ensuring equal opportunity for, and removal of discrimination against, persons with disabilities seeking employment. • National Disabilities Action Plans provide a detailed set of required actions and budgets outlining financial resources to implement the law. Table 4: Vietnam Disability Legal and Policy Provisions Assisted by USAID Year Provision Funder/ Implementer 2005 Education Law No. 38/2005/QH11 (June 14, 2005) GVN 2005 MOET National Action Plan on Education of Children with Disabilities for 2005–2010 CRS 2006 Decision No. 23/2006/QD-BGDDT Inclusive Education for Persons with Disabilities (May 22, 2006) CRS 2006 Decision No. 239 Issuance of First National Action Plan (NAP) on Disabilities for 2006–2010 VNAH 2006 Vocational Training Law No. 76/2006/QH1 (Nov. 29, 2006)11 VNAH 2006 “Disability” incorporated into Vietnam Household Living Standard Survey12 GVN 2007 Establishment of Blue Ribbon Employers Council (BREC) VNAH 11 Includes vocational training for “disabled and handicapped persons,” tax provisions for organizations providing training for persons with disabilities (i.e., professional skills and knowledge for persons with disabilities, policies for vocational training centers, persons with disabilities and trainers) and a chapter with six articles related to persons with disabilities. 12 As with results of more accurate statistical data collection, the percentage of persons with disabilities increased to 15.3 percent. Evaluation of USAID Vietnam Disabilities Programming 22 Year Provision Funder/ Implementer 2007 Decision No. 9/2007/QD-BGDDT (Aug. 29, 2007)13 GVN 2007 Revision of Vocational Law to include persons with disabilities VNAH 2010 Enactment of Law on People with Disabilities and Decree 2814 VNAH 2011 Issuance of Second NAP for 2011–2020 VNAH 2012 Circular No. 26/2012/TT-BLDTBXH (Nov. 12, 2012)15 VNAH 2012 Labor Code Law No. 10/2012/QH13 (June 18, 2012)16 VNAH 2012 NAP to Support Persons with Disabilities for 2012–2020 VNAH 2012 Inter-Ministerial Circular 34/2012/TTLT-BYT-BLDTBXH by MOH & MOLISA on Determination of Disability Level by Medical Council VNAH 2012 Inter-Ministerial Circular 37 on Determination of Disability Level by the Disability Council17 VNAH 2013 Decree on Administrative Fines for Violations of the Law on People with Disabilities VNAH 2013 Inter-Ministerial Circular No. 42/2013/TTLT-BGDDT-BLDTBXH-BTC18 VNAH, CRS 2014 Inter-Circular No. 01/2014/TT-BLDT-BXH on Case Management for Persons with Disabilities VNAH 2014 Circular No. 11/2014/TT-BGDDT (April 18, 2014)19 VNAH 2014 National Action Plan on Vocational Training VNAH 2014 National Action Plan on Rehabilitation by MOH20 VNAH 2014 Vocational employment and guidelines developed by Employment Bureau/MOLISA VNAH 2014 Ratification of the UN Convention on the Rights of Persons with Disabilities VNAH, CRS Building GVN capacities to develop laws and policies. MOLISA’s Social Protection Department (SPD) is a member of a drafting committee on laws and policies for persons with disabilities. VNAH supported its work by funding technical meetings. For example, VNAH provided funds and local experts to help SPD organize national workshops with all provinces for drafting laws and policies. Additionally, VNAH supported ministries in providing comments on the drafting circular on case management and promulgating the policy development process. SPD was very satisfied with the assistance it received from VNAH. The two staff people from SPD also indicated that they were very satisfied with the materials developed by the legal experts and with other experts who were hired to support them in law/policy formulation. 13 Mandates training for teachers and education managers to acquire the skills necessary to provide inclusive education. 14 Decree provides guidelines for the implementation of law. 15 Circular provides the determination of the rate of employees with disabilities; dossiers, procedures and order for recognition of production and business establishments employing persons with disabilities accounting for at least 30 percent of their total employees for enjoyment of incentives; the determination of levels of monthly social allowance and caretaking allowance; levels of allowance for persons with disabilities living in social protection centers; commune-level social allowance approval councils; and conditions and competence to send persons with disabilities living in social protection centers to families. 16 Code contains regulations on the employment of persons with disabilities and ensuring equal opportunity for and removal of discrimination against persons with disabilities. 17 The goal is to increase the number of persons with disabilities who receive disability social allowance. 18 Preferential education policy and supports for persons with disabilities. 19 Persons with disabilities are exempt from entrance exams when enrolling in schools. 20 Decision #4039/QD-BYT. Evaluation of USAID Vietnam Disabilities Programming 23 Other people from SPD who participated in the drafting of disability laws also said VNAH provided good international experts to support the drafting committee. As a result of the technical assistance, SPD staff and members of the committee built the capacity to draft laws. Staff from MOLISA’s Department of Legal Affairs (DLA) stated that they received support from international partners, with the majority of their funding coming from UNICEF. Although these staff stated that VNAH assigned one person to work with them to draft laws, they did not receive direct funding from USAID. DLA staff also stated that they need to build capacity in their roles and have yet to receive any USAID support for this. Evaluation team members interviewed members from the MOET’s Committee on Education of Children with Disabilities and Disadvantaged Children and a program officer from the committee who was involved in the development of the major education laws and policies related to children with disabilities. Team members indicated that the support from USAID through CRS and VNAH was invaluable in law and policy formulation. Giving a voice to persons with disabilities. The evaluation team members interviewed two former National Assembly (NA) members who were members of the Committee of Social Affairs (CSA). VNAH assisted CSA in conducting an assessment of the implementation of the Vietnam Disability Ordinance. Results from the assessment indicated the need to upgrade the ordinance to a law. The former CSA members were very satisfied with USAID assistance and support. As a result, CSA invited persons with disabilities to be involved in the formulation process for the Law on People with Disabilities. CSA now has a standing procedure to involve persons with disabilities in its work, and continues to consult with DPOs on the content of laws. One member said, through USAID, “The Committee on Social Affairs showed their respect to the voice of persons with disabilities through disabled people’s organizations.” The team interviewed members and officers from DPOs in all four provinces. DPOs function more as associations of persons with disabilities than as full-fledged civil society organizations. Through USAID support, DPOs participated in the policy formulation and implementation process for the first time. Although USAID provided training to DPO members to enable them engage in the conversation, it appears that the training did not provide them with the tools to alter the process to accommodate a new set of actors so they could truly participate. It appears that they may not have been effective participants in the policy discussions. DPO members were involved in policy and law formulation and coordination activities; however, the descriptions of how they participated in these activities suggested that their participation was consultative. Statements from other participants indicated that their involvement was passive, perhaps due to their “limited communication skills.” Statements from the KIIs suggest that the participation was not interactive or self-mobilized. For example, members of one provincial DPO stated that they participated in the development of the provincial action plan (PAP). They distributed the PAP to members for feedback. The officers reviewed the feedback and submitted only what they believed was important. When pressed for information on what criteria they used to determine what was important, key informants could not provide the criteria. When asked why they did not summarize all of the feedback and send everything to the local government, they indicated that the local officials wanted only certain information. Their description of the process indicated that their participation was due to a request of an external agency to meet predetermined objectives. There may have been some shared decision-making, but it Evaluation of USAID Vietnam Disabilities Programming 24 appeared to happen after the local government had already made the major decisions about the PAP and after officers filtered the feedback according to their perception of what was important. All of the DPOs interviewed found the USAID support and activities related to policy formulation, implementation and coordination to be relevant and said they were very satisfied with the activities. The primary benefit has been that DPOs have a legal foundation to advocate for rights and benefits of persons with disabilities. Other DPO members said it helps persons with disabilities to advocate for their benefits. However, these two statements refer to DPO members. When pushed to describe how the DPO advocated for persons with disabilities outside of their organizations, three DPOs were unable to provide examples. No leadership or advocacy skills training was provided to the DPOs that participated in USAID activities. When asked about support activities related to advocacy, one DPO said they were denied funding by the implementing partner because the USAID project did not fund that type of activity. As a result, they obtained funding from an international non-governmental organization (INGO). The lack of leadership and advocacy skill training and support for strengthening DPOs’ advocacy skills may be partly because the implementing partners are not civil society organizations for and by persons with disabilities. VNAH assisted the CSA with organizing three regional workshops to consult with the disability community via the DPOs and elect representatives to CSA on disability policy. Additionally, VNAH helped CSA explain and defend proposed laws to the NA and assisted CSA with a number of internal meetings to discuss the draft law. CRS also provided support for CSA to hold consultative workshops. However, statements from other key informants suggest that the participation in the workshops was not interactive. Comments from central government officials indicated concern about the quality of participation of DPOs. One key informant said, “Some of the representatives from the DPOs do not know about [the] law formulation process, so they could not provide good comments.” A government official stated that participation of members of the DPOs is a weak step in the law formulation process because members’ lack of communication skills hindered their ability to actively participate in discussions. Key informants from DPOs expressed that their contributions and suggestions were not adequately taken into consideration in the process of formulating the law and policies. For example, one DPO member provided his recommendation on the right of persons with disabilities who are able to drive to get a driver’s license. However, it was not accepted and persons with disabilities are not licensed to drive, even though many of them have to drive on daily basis. Although this is a legitimate concern, it should be noted that the issue of a driver’s license was not relevant during consultation of the Disability Law. It should be addressed in the non-code regulations of drivers’ health requirements. An Inter-Circular No. 24/2015/TTLT-BYT-BGTVT dated Aug. 21, 2015, on health standards of drivers has addressed the issue by allowing persons with disabilities to obtain driver’s licenses. Another DPO representative noted that persons with disabilities were involved in developing the law and policies through consultation workshops and meetings, but were not selected to be members of the drafting committee and were unable to engage in monitoring law and policies implementation. Having recognized that many DPOs and persons with disabilities were not well informed on the lawmaking process, the disability policies and their rights/benefits, VNAH started activities in 2015 to improve awareness among DPO leaders and members on disability rights and the procedures to claim those rights. DPOs also received support to improve their knowledge of disability policies and their advocacy skills. Implementing the law and policies on disabilities. The MOH collaborated with VNAH during the last three years. The activities included 1) the development of the National Action Plan (NAP) on Rehabilitation 2014–2020, and 2) the development of Circular 46/2013/BYT on the structure, role and Evaluation of USAID Vietnam Disabilities Programming 25 responsibilities of the rehabilitation units. Together, these activities provide for the underlying legal authority to budget for and deliver rehabilitation services for persons with disabilities. Provincial key informants demonstrated that they were aware of disability laws, due to VNAH and CRS providing consultations on the development and dissemination of new laws. With the support of VNAH, all four provinces developed PAPs to guide implementation of the Law on People with Disabilities. But not all of the districts have action plans. Many of the commune officers stated their belief that too many policies were developed in a short time and, as a result, it is difficult to know all of the benefits that persons with disabilities are allowed under the law. Many persons with disabilities were also unaware of all of the benefits available to them. USAID supported the provinces in developing provincial action plans. Key informants indicated that the ability to implement a PAP is dependent on a province’s ability to acquire resources and the willingness of local officials to make a commitment. For example, Quang Nam’s government officials are committed to implementing policies, but do not have the revenue to provide services. Danang City has a social fund that employers and others pay into each year and, as a result, it generates revenue that it can allocate to services for persons with disabilities and other vulnerable groups. Constraints on full implementation of the law. Implementation of the law was constrained by many factors, including insufficient resources. As shared by a National Coordinating Council for Disabilities (NCCD) official in an interview, not much progress had taken place in implementing the law since its issuance in 2012. Most efforts were about development of the legal framework, including non￾code decrees and circulars. Team members interviewed the head of the Ministry of Health’s Agency of Medical Services Administration (AMSA). VNAH worked with MOH for three years to develop Circular 46 and the NAP for Rehabilitation 2014–2020. Although the NCCD convinced the Ministry of Finance (MOF) to allocate funding for the MOH’s NAP, the increase in the budget allows them to pay only for surgeries for persons with disabilities. The ministry does not have the budget to pay for examinations, screening to identify persons with disabilities or implementation of other aspects of the NAP. Several key informants stated that the implementation of laws is not sufficiently monitored or evaluated. Monitoring and evaluation data and information regarding NAP and PAP implementation is lacking. Currently no data or information exists on how laws and policies are implemented. NCCD’s limitations hinder its ability to monitor the implementation of the disability law. Although the team was informed that NCCD is receiving support from USAID to develop and implement a monitoring system, it was unclear at the time of evaluation where NCCD was in the process. The team was told that NCCD was informing and consulting with local government officials, but senior provincial government officials who were interviewed said they were unaware of NCCD’s work in this area. In one province, officials said they never talked to anyone from NCCD. Limited budget and human resources hinder the full implementation of laws and policies. For example, the MOH developed its plan for its NAP through the assistance and support of USAID’s implementing partner. However, one official from MOH informed the team that the ministry does not have sufficient funds or human resources in its current budget or payroll scheme to implement all of the recommendations in the plan. It would take stronger and more sustained advocacy over time to build the support needed to develop additional resources. Former CSA members stated that local staff in the MOLISA system have low knowledge and lack the skills to access and work with persons with disabilities. More capacity building is needed among people who work with persons with disabilities. Members of CSA typically do not have knowledge about persons with disabilities, informants said. Evaluation of USAID Vietnam Disabilities Programming 26 The promulgation of the Disability Law in 2010 marked an improvement in the disability legal framework in Vietnam. Pursuant to the Disability Law, decrees and circulars are developed and enacted to provide guidance on the implementation of the law. However, a gap remains between formulation of policies and their implementation. In general, policies are not implemented as expected. One of the main causes for this is that many policies are developed with neither sufficient evidence nor a baseline survey to provide justifications for policy development. One MOET official indicated that no comprehensive and consistent mechanism exists to collect, synthesize and disseminate information related to children and adults with disabilities. Consequently, the MOET, MOLISA and MOH do not have a robust national database to assist in the implementation of policies. The individual suggested that support in developing a national database would strengthen the government’s ability to develop and implement relevant laws and policies. The MOH is concerned that the current tool to classify disabilities under the law is not valid or reliable and, as a result, the tool may cause some persons with disabilities to not be considered “disabled,” or the level of disability may be incorrectly classified. Additionally, the head of AMSA expressed concerns that current disability laws do not adequately address the needs of persons with mental disabilities, as no such classification exists. Staff from the Department of Labor, Invalids and Social Affairs (DOLISA) and commune officers stated the same concerns. These staff and officers also stated that commune officers do not have the expertise, knowledge and tools to carry out their responsibilities to identify persons with disabilities or the type and severity of the disability. A MOH/AMSA key informant stated that he is unhappy with Circulars 34 and 37 because he believes identifying the degree of disability at the commune level is inappropriate. Commune officers stated that they have difficulties with the tool and are uncomfortable making decisions. As a result, commune officers said they frequently bring in a medical doctor to assist them. Policy coordination. USAID provided support to the NCCD, which is responsible for coordinating the implementation of the National Disability Law. NCCD has low capacity to coordinate and its contribution toward law and policy formulation and implementation was limited at best. One senior ministry official indicated that the previous leader of NCCD had been in the position to coordinate across sectors. Since NCCD is a unit in the Social Protection Department of MOLISA, it has no authority to coordinate with other departments within MOLISA and it has no authority to coordinate with the MOH, the MOET or other ministries. Consequently, NCCD cannot take on its role of coordinator. It can only summarize and report on its work. VNAH supported the MOH in policy development. PDSP supported the provincial health departments. No coordination or consultation took place between the MOH and the health departments. As a result, the policies developed at the ministerial level and the services and support provided at the local level did not directly connect. Consistency with CRPD and the need to strengthen the law. Vietnam ratified the CRPD in December 2014. Of the CRPD’s 30 articles, 25 were referred and applied in the Law on People with Disabilities. Several gaps remain, as well as a few areas where the Law on People with Disabilities contradicts basic principles established in the CRPD. These contradictions impede the implementation of the law and, as a result, impede the ability of persons with disabilities to participate fully in society. Therefore, continued USAID assistance may be needed to help policymakers review current disability laws and policies and, where necessary, assist and support the drafting of new articles that will align the laws and policies with the CRPD. The next four subsections highlight significant inconsistences in the laws that should be addressed. Evaluation of USAID Vietnam Disabilities Programming 27 Definition of Persons with Severe Disabilities Persons with severe disabilities are those whose impairments render them unable to perform by themselves their personal daily life activities. (Law on Persons with Disabilities [2010], Article 3.2.a) This legal definition reinforces the old idea that individuals with severe disabilities are incapable. Although some may not be able to perform their personal daily life activities, they are entitled to equal rights under the law and to participate in society to the best of their ability. The CRPD requires that all persons with disabilities have access to the individualized supports and services they need to participate as fully as they can in society. The current education policies state that when a student with disabilities is in a regular class, the number of students in that class should be lower than the average class size at the school. This was considered a way to support the teacher. Schools hardly applied this regulation since most classes were too big to absorb more students from those classes having children with disabilities. Therefore, teachers did not have sufficient support from the education system to take care of students with disabilities. The evaluation team members heard many provincial and district officers, headmasters and teachers say things such as, “We are responsible only for children who can learn,” “Those children are too severe.” “They have severe disabilities. They lie in bed all day and cannot move.” Parents also stated that schools would not accept their children with severe disabilities. Common statements by parents included, “My child was not allowed to go to school because he is in a wheelchair.” “The special education school will not accept him because he cannot take care of himself.” Such statements indicate that some government officials, education personnel and parents espouse old ideas that hinder children’s ability access to public education. The GVN, with the support of USAID, would benefit from aligning the definition of “persons with severe disabilities” with the CRPD. Discrimination on the Basis of Disability Discrimination against persons with disabilities means the act of shunning, refusing, maltreating, disparaging, showing prejudice against or restricting the rights of persons with disabilities because of their impairments. (Law on People with Disabilities, 2010, Article 2.3) Although this definition of discrimination is consistent with the CRPD, it is not specific enough. CRPD requires that states’ parties “prohibit all discrimination on the basis of disability and guarantee to persons with disabilities equal and effective legal protection against discrimination on all grounds” (Article 5.2). In theory, the law applies to those who have disabilities that are physical, sensory and visual, mental and psychiatric and intellectual. In reality, the language of the law appears primarily tailored to persons with physical disabilities, particularly those with mobility disabilities. For example, persons with disabilities have the right to “functional” rehabilitation and orthopedic operations, but not the right to habilitation services that would be required for children with intellectual disabilities. Evaluation team members documented incidents of discrimination on the basis of disability in access to services and supports and in education. One reason for the incidents was due in part to the false belief of implementing partners and local government officials that it was important for the pilot programs to succeed. Therefore, beneficiaries were selected because they had “minimal” or “mild” disabilities and would be the most likely to succeed or “rehabilitate.” Evaluation of USAID Vietnam Disabilities Programming 28 The purpose of a pilot study is not to ensure success; rather, it is to provide a platform for the organization to test logistics, prove value and reveal deficiencies before spending a significant amount of time, energy or money on a large-scale project. Therefore, it is imperative to include everyone during the pilot stage to ensure that an accurate assessment is conducted. In the case of the education system, the evaluation team members found schools excluding children with disabilities because the children were not able to adapt to the environment and the school did not have the resources to meet the children’s needs (e.g., “She cannot sit for long periods of time.” “We don’t have the resources for those kinds of children.”). Given the limited resources for services and supports, it is necessary for any government to function within its means. If the government must limit benefits due to financial constraints, it must do so equally for all persons with disabilities. Regardless of a country’s resources, it is essential that all adults and children receive full protection under the law. The Law on Persons with Disabilities should continue to prohibit discrimination, but it should revise the definition of discrimination to guarantee equal and effective legal protection against all discrimination. Reasonable Accommodation “Reasonable accommodation” means necessary and appropriate modification and adjustments not imposing a disproportionate or undue burden, where needed in a particular case, to ensure to persons with disabilities the enjoyment or exercise on an equal basis with others of all human rights and fundamental freedoms. (CRPD, 2007, Article 2) According to CRPD, denying persons with disabilities reasonable accommodation constitutes discrimination on the basis of disability. Without reasonable accommodation, individuals with disabilities will not be able to realize their right to dignity, equality and freedom from discrimination because they will not be able to participate fully in society. The current Law on People with Disabilities provides a general definition of discrimination, but does not state that the failure to provide reasonable accommodation to persons with disabilities constitutes discrimination. An additional clause in the law to ensure reasonable accommodation would align the law with Article 5.3 of the CRPD, which states: “In order to promote equality and eliminate discrimination, States’ Parties shall take all appropriate steps to ensure that reasonable accommodation is provided.” Gaps in the Education Policies21 The MOET, with technical assistance and funding from VNAH/USAID, conducted a National Baseline Survey on the Disability Support Service System focused on education for persons with disabilities. Results were summarized in the Vietnam National University Journal of Education Research. 22 Survey findings indicated gaps between education policies and practices for children with disabilities. These gaps were the same as those identified by key informants of this evaluation, and they include the following:  Children with disabilities are entitled to school fee and other education exemptions and reductions. However, the families of many children with disabilities experience difficult economic conditions and cannot afford costs including textbooks, special learning equipment and transportation. As a result, they cannot access educational services. 21 The section is based on the team leader’s observations, statements from key informants and the following report: Nguyen Thi Hoang Yen & Nguyen Thi Thu Huong (2013). The Policy on Education of the Disabled in Vietnam – the Gaps and Its Impact on the Persons with Disabilities. VNU Journal of Education Research, 29(2), 24–33. 22 Ibid. Evaluation of USAID Vietnam Disabilities Programming 29  It is difficult to implement the policy on school fee exemption and the reduction of other contributions for children with disabilities. Additionally, the process to apply for support is too complicated and time-consuming for many parents.  Although policies support education for children with disabilities, these children face transport accessibility problems and difficulty accessing schools and offices. These are also obstacles for enforcement and implementation of disability support policies.  The conditions for enforcement and implementation of education policies and laws are limited. The localities depend on the development projects and programs to carry out the policies properly. This leads to the unsustainability of the programs. Additional findings indicated that educational policies and laws to support persons with disabilities promulgated at the national level were applied and implemented nationwide. However, depending on the local resources, the enforcement and implementation of the policies varied, and the rate of children accessing services differed. Moreover, findings suggested that conditions contributing to the effective implementation of inclusive education included resources and capacity, multi-sector coordination of services, financial resources and timely planning. Authors of the report concluded: The promulgation of the Disability Law in 2010 marked an improvement in the disability legal framework in Vietnam. Pursuant to the Disability Law, decrees and circulars are developed and enacted to provide guidance on the implementation of the Law. However, the gap between policies and the implementation of policies remains. In general, policies are not implemented as expected. One of the main causes for the poor implementation of disability policies is that many policies are developed with neither sufficient evidence nor baseline survey to provide justifications for policy development. The implementation and efficiency of policies are usually not monitored and evaluated on frequent basis. Awareness raising and dissemination of policies activities did not have proper attention; both beneficiaries and implementing staff at the local level have little say in policy development process. Other challenges included the capacity of the disability support system and limited multi-sector coordination. Strengthening Critical Institutional Capacities USAID-supported programs provided a broad range of assistance for direct support to persons with disabilities, including health and medical services, inclusive education and employment and livelihood. Assistance also helped strengthen the means of delivering the services through capacity building of service providers including DPOs, and the development and enhancement of supporting systems including case management and data information systems. Key results include: • USAID assistance played a key role in strengthening medical and health services specific to the needs of persons with disabilities. Assistance strengthened the capacities of rehabilitation service providers, improved care facilities to better accommodate persons with disabilities, developed new curricula at medical universities, introduced new and innovative birth defects surveillance and newborn screening services, and worked to incorporate community-based support activities. Beneficiaries interviewed were largely satisfied with these services; 75 percent reported that they were very satisfied or satisfied. Key Question 3: In terms of improving service delivery, did the USAID Disability Support Program provide an enabling environment and a supporting systems approach in technical assistance to strengthen critical institutional capacities? Evaluation of USAID Vietnam Disabilities Programming 30 • DPOs supported by USAID programs helped give a voice to and provide useful services for persons with disabilities. However, DPOs are quasi-governmental organizations rather than civil society organizations. Their status may limit their advocacy role. • USAID assistance has enabled multiple services focused on inclusive education to expand access for children with disabilities. Capacity building for teachers and administrators has been highly valued and effective, but HR resources were too limited to fully implement inclusive education commitments. • Vocational training and employment/livelihood supporting services developed and implemented through USAID assistance activities have expanded job and business opportunities for persons with disabilities. Training alternatives that take both formal and informal approaches have been developed. While largely successful, beneficiaries noted room for improvement. • DOLISA staff highly valued the Disability Information System, especially in data analysis for program planning and management purposes. However, it would have been more effective if DOLISA had played a more proactive role in developing and maintaining the system. While DOLISA staff said the DIS was very helpful and easy to use, commune officials may not have had computers to enter or track data. The findings are broken out into two subsections: Service Delivery and Supporting Systems for Service Delivery. The former section contains findings from the four main subsectors of service delivery (health/medical, community-based support, education and employment/livelihood) and is organized by key activities within each. The latter section assesses key findings for the case management and the disability information system (DIS) initiatives. Service Delivery Health and Medical Services Context: The health care service delivery system in Vietnam comprises a large number of institutions. Human resources are unevenly distributed in the service system, with shortages in some regions, facilities and specializations. The most qualified health workers are concentrated in urban areas, but only 30 percent of the population lives in urban areas.23 The government-run sector has been open to developing innovative strategies to improve, manage and integrate health and medical services for persons with disabilities. USAID, through VNAH and DAI, has played an integral role in strengthening rehabilitation and public health services for persons with disabilities by building the capacity of doctors and technicians and by providing direct medical and health support services. USAID-funded IPs worked with a wide range of partners, including medical universities, health departments, hospitals, clinics and commune health workers. Following is a summary of key achievements and challenges. Physical rehabilitation: A major focus of health and medical care assistance was strengthening the capacities of rehabilitation service providers. Table 5 highlights the specific services provided by PDSP. 23 See World Health Organization (2012). Evaluation of USAID Vietnam Disabilities Programming 31 Table 5: Capacity Building on Physical Rehabilitation by PDSP, 2013–2015 The director of the Danang Rehabilitation Hospital sent staff to training for short-term capacity building courses in rehabilitation and said that staff improved their capacity to provide speech therapy and rehabilitation, as well as to provide home-based rehabilitation for persons with disabilities. He suggested that the courses last six months or longer and that the project provide equipment to upgrade their speech therapy unit. The hospital received support to organize screening and examinations of persons with disabilities at their homes. The USAID/DAI disability program also organized medical consultations at project-funded hospitals and district health centers and supported travel for doctors and patients. The PDSP project also supported corrective surgery by paying a portion of the surgery fee and other fees for the hospital, and provided allowances to persons with disabilities to conduct rehabilitation at home. As a result of the USAID/DAI support, Danang Rehabilitation Hospital provides home-based rehabilitation for children with disabilities. USAID assistance also funded a children’s physical therapy room in the hospital after officials identified a need for it. The director noted that prior to the support, the hospital received three to four children each day for rehabilitation. At the time of evaluation, the hospital was receiving approximately 20 children each day. When asked what types of challenges the staff faced in implementing project activities, he said the procedure to receive surgery was complicated for persons with disabilities. Family members sometimes do not collaborate well and will not take all of the administrative steps needed for surgery. Some persons with disabilities have no health insurance. Since PDSP pays for only a portion of the surgery, economic status may prevent some surgical patients from being able to pay. For the same reason, some people who need multiple surgeries may decide not to undergo or complete the process. DOH officials said other training on rehabilitation in Danang was very beneficial, including a training needs assessment of 13 major hospitals and district health centers and one consultative workshop with 15 local health specialists and officers. These officials also commented positively on another follow-up to the needs assessment, a training of trainers rehabilitation workshop for 62 doctors and physical therapists that focused on “rehabilitation for children with autism spectrum disorders and children with developmental disabilities.” • Short-term courses: 30 medical doctors (MDs) & 41 physical therapists (PTs) trained on occupational therapy, speech therapy, autism; 19 MDs & senior speech therapists (ST) attended training of trainer (TOT) courses; 96 health care workers (HCWs) trained on home-based rehabilitation services; 575 caregivers/parents trained on basic rehabilitation techniques; 167 HCWs trained in early detection. • Six-month course on rehabilitation orientation for MDs in Hanoi: 12 participants (Danang: 4, Dong Nai: 3, Binh Dinh: 5) • Six-month course on rehabilitation transition for nurses in Danang: 24 participants (Danang: 5, Dong Nai: 8, Binh Dinh: 11) • Rehab Facilities Upgraded: Danang: 6 units; Dong Nai: 1 unit; Binh Dinh: 5 units. • New pediatric rehabilitation department opened at Danang Hospital for Women and Children. • New speech therapy unit established at Danang University of Medical and Pharmacy Technologies. Evaluation of USAID Vietnam Disabilities Programming 32 Support for rehabilitation training progressed in 2014; 36 general doctors and nurses received support to participate in two specialized training courses (lasting six months each) at Danang and Hanoi medical universities. This effort mostly aimed to support rehabilitation programs at Danang DOH and the Danang Hospital for Women and Children (DHWC). PDSP also conducted two training courses for 53 health case managers to provide home-based rehabilitation services in Danang, resulting in 170 persons with disabilities — including children — receiving home-based physical therapy. In Binh Dinh Province, 100 adults and children with disabilities received home-based and center-based physical therapy services. DOH officials considered home-based training and services a critical success of USAID assistance. USAID assisted in upgrading 12 rehabilitation units in provincial and district hospitals in Danang, Dong Nai and Binh Dinh. The Agency’s assistance helped establish a pediatric rehabilitation unit at Danang Hospital for Women and Children and a speech therapy unit in Danang University of Medical and Pharmacy Technologies. Several challenges affect the long-term delivery of rehabilitation services for persons with disabilities. Lack of sustained investment in rehabilitation services and facilities/equipment is the most critical. Medical services provision depends on local human and financial resources. The need for rehabilitation services may be higher than the willingness of local authorities to invest in rehabilitation. This is evidenced by the recent budgeting process at provincial and district hospitals that did not set rehabilitation as a priority. One reason for the low interest in providing rehabilitation service is that health insurance pays only a portion of fees and does not pay for all types of rehabilitation procedures and operations. A central government official noted that hospital administrators did not want to invest in rehabilitation equipment or services because they would not be reimbursed by medical insurance, and individuals could not afford to pay out of pocket to cover the cost. Another cost-related concern was that hospitals could not obtain health insurance reimbursement for rehabilitation services unless the doctor providing the service was certified to provide the services. Additionally, MOH issued Circular No. 50/2014 to list rehabilitation techniques that the MOH accepts. Of the 248 operations and techniques listed, only 45 were paid by health insurance at the time of the evaluation. Health insurance typically paid only a portion of the fee. Consequently, due to limited budgets, hospitals were placing less priority on providing rehabilitation services for which they would not be reimbursed. One senior official stated that rehabilitation faced many other challenges. Human resources were limited, and some medical universities could not afford rehabilitation faculty. Some specialties of rehabilitation did not even have appropriate teachers, such as speech therapists. The number, accessibility, condition and functions of facilities themselves was also a concern. Helping children with autism. PDSP conducted several trainings, including two advanced courses on autism spectrum disorders for doctors and physical therapists from Danang, Dong Nai and Binh Dinh. Additionally, three trainings on “Recognition, Assessment and Social Inclusion for Children with Autism” and a four-day communication training for 60 parents of children with autism conducted in Dong Nai by PDSP provided parents with a greater understanding and improved skills to successfully advocate for their children and access needed services. Parents said that they received services, they were very happy to see improvement in the children’s behavior: they were more interactive and their verbal communication skills had improved. The team interviewed the heads of the spinal cord injury and pediatric departments of Danang Rehabilitation Hospital. Two rehabilitation technicians were also interviewed. They participated in the rehabilitation training and learned “movement” therapy, speech therapy and non-pharmaceutical therapy Evaluation of USAID Vietnam Disabilities Programming 33 0% 20% 40% 60% 80% 100% Very dissatisfied Dissatisfied Neutral Satisfied Very satisfied Device/service Service provider Figure 8: Level of Satisfaction among People Who Received Medical Assistance for children with autism. They learned about screening and non-examination diagnostic tools. People from the hospital took turns attending; no one took every course due to their busy schedules. Hospital staff noted that overall, the training plan was not as comprehensive as they wanted and some courses and topics in the training were too similar. The training on physical rehabilitation for spiral cord injury patients, for example, closely followed the standard guideline document supported by Handicap International, which had been in use for a long time and medical staff were familiar with it. The guide on patient nutrition was the example of an irrelevant training topic in the Vietnamese context. The trainings were participatory and included national and international experts who lectured on specific topics; they were holistic, covering topics including therapy specific to the needs of children with autism, recognition and assessment of children with autism and the ways and means to improve social inclusion for children with autism. The trainees said the courses provided important knowledge for new health care professionals. The directors and technicians agreed that the training on how to work with children with autism was very useful, especially helping district staff in detect the condition in children. However, all four trainees agreed that some topics were too short and too basic. Since they had worked in the area for a long time, they were interested in more advanced topics. Birth defects surveillance and newborn screening services. PDSP implemented a pilot birth defects surveillance (BDS) system in Danang, with more than 110 doctors, midwives and nurses trained on birth defects identification. Also, newborn screening (NBS) services at the DHWC that focused on hearing loss and congenital heart diseases used an information system funded by USAID assistance. All of the staff personnel from the hospital said they were “satisfied” to “very satisfied” with the assistance and support that led to the provision of these services. DHWC officials told the evaluation team that early identification of problems led to earlier treatment and/or delivery of critical supporting services. The evaluation team was informed that the hospital had made arrangements to continue the services after the project ends. Support to improve health services at district hospitals. PSDP provided a variety of assistance to improve health and medical care services for persons with disabilities at district hospitals, including support for direct services such as clinical examinations PDSP worked with DOHs to provide health services including clinical examinations for diagnosis for 2,559 persons with disabilities, physical therapy for 649 persons with disabilities and surgeries for three persons with disabilities. Local health services facilitated provision of these services and the distribution of assistive devices (wheelchairs, walking sticks and hearing aids) to 1,344 persons with disabilities. As noted in Figure 8, 65 percent of medical services beneficiaries were “very satisfied” or “satisfied” with the service, and 58 percent were “very satisfied” or “satisfied” with the devices. Direct medical and health support services to persons with disabilities has been provided in the City of Danang and the provinces of Dong Nai (in Bien Hoa City and Vinh Cuu District), Binh Dinh (Phu Cat and Hoai An districts) and Quang Nam (12 of 18 districts). The services are managed through a referral mechanism in PDSP’s case management initiative that supports commune health Evaluation of USAID Vietnam Disabilities Programming 34 workers to conduct outreach to families of children or adults with disabilities. When asked about the PDSP case management system, a majority of KIs noted that it was complicated and time-consuming. For example, after identifying assistance needs, case managers had to submit a list of persons with disabilities and their needs to the district level and then to DOLISA offices. MOLISA and PDSP staff jointly make the final decision on service provision after a long verification process on both sides. But the case management program mainly provides services that were funded by PDSP. The process creates extra steps. Relating the case management system to the actual services that the government can provide would enhance service delivery. PDSP provided equipment to upgrade six rehabilitation units in Danang, Dong Nai and Binh Dinh districts. A senior Danang staff member was very satisfied and impressed with the equipment. He said: VNAH has sent a senior technical staff to our department. He is a well￾known doctor specialized in rehabilitation with extensive knowledge and experience. He worked with us to find out what additional and essential equipment we needed to provide good service to persons with disabilities. Equipment came after a few weeks, and the equipment was exactly what we ordered [showed rehabilitation equipment]. We have not had enough time to see how these actually help improve patient status, but I believe that many people will benefit from this investment. Despite the achievements in improving health services, and a high level of satisfaction among beneficiaries who received medical assistance (Figure 8), key informants highlighted a number of concerns. In summary, these concerns are as follows:  Delays in receiving services: Several key informants said a lot of money was invested in Danang Province, but persons with disabilities had been waiting a long time for services.24 KIIs at the district level within Danang indicated that people were still waiting for corrective surgery. Two key informants stated that more than 20 people were on a waitlist, while PDSP management stated that they were waiting for the Department of Health to provide a list of people who would receive corrective surgery.  Ability to access corrective surgery: Furthermore, not everyone with a disability had access to corrective surgery. Persons with “severe” and “very severe” disabilities qualified for health insurance and, as a result, had access to the corrective surgery provided through the USAID￾funded project. Those without health insurance and who needed corrective surgery could not access it. As indicated as part of Key Question 2, the tool to identify persons with disabilities may not be valid and, as a result, may have been denying some individuals access to government￾supported health insurance. Even for people with health insurance, the process to obtain approval to receive services was complex and time consuming. A hospital administrator in Dong Nai stated that persons with disabilities who did not have health insurance were unable to access corrective surgery. He said USAID should provide services to all persons with disabilities, regardless of their insurance coverage status. This practice is due to two factors: 1) The local government decides who will receive the surgery and 2) The agreement between the Department of Health and PDSP is that PDSP will pay the portion of the surgery that medical insurance will not pay. Consequently, the procedure excludes uninsured persons with disabilities. 24 Other KIIs confirmed that people have been waiting for six to 12 months to receive services and, in some cases, longer than 12 months. Evaluation of USAID Vietnam Disabilities Programming 35 Support to Hanoi Medical University’s Rehabilitation Department. Two initiatives provided support for the Rehabilitation Department of Hanoi Medical University Support to begin to institutionalize capacity building for disabilities medical services. The first was support to develop learning materials on vocational rehabilitation, and the second was designed to use these materials for training and other capacity-building opportunities for medical professionals. Development of learning materials: VNAH supported professionals from the Rehabilitation Department to collaborate with a team of medical teachers, health professionals and international experts from four universities to develop a textbook on vocational rehabilitation. The textbook will be a reference for medical students taking physical rehabilitation courses and a key textbook at nursing and medical technology schools, and is tailored for an audience of practicing medical professionals. VNAH anticipated that textbook printing would take place before the end of the calendar year. VNAH indicated that all four medical schools in Vietnam had agreed to use the book, but could not articulate an exact plan for how the textbooks would be incorporated. At the time of the evaluation, the plan for the universities to institutionalize the textbook was to include some of the textbook’s contents in a 45-minute section of a rehabilitation course for medical students. Postgraduate students and those in nursing and medical technology schools would then participate in more extensive sessions for specialized rehabilitation courses. The reason for the limited use was that the training curriculum for medical students was already full, and the rehabilitation course lasts only five days. As a result, the potential of the textbook to strengthen the institutional capacity of medical doctors remained unclear. Capacity building for medical professionals: The director of the Rehabilitation Department reported a discussion between Hanoi Medical University and the VNAH team on the needs of introducing the concept of occupational rehabilitation to health workers. Multiple pilots with post-course evaluations were conducted at the graduate (all medical universities) and post-graduate (Hanoi Medical University only) levels. In 2014, 47 medical doctors took part in one TOT on vocational rehabilitation principles in medical studies. After the workshop, participants completed a post-training survey and gave a positive evaluation. Trial teaching of the vocational rehabilitation curriculum took place with 236 university students and 13 postgraduate students at the provincial medical universities in Hanoi, Hai Phong, Thai Binh and Thai Nguyen. Also, a post-training survey collected feedback and recommendations from the participants. Most thought it was necessary to have lessons on vocational rehabilitation in the training program for general doctors at medical universities, although some believed that the training needed to be revised to better conform to economic, health and social contexts in Vietnam. In a follow-up with students and faculty, a drafting team organized in late 2014 was to revise five of the curriculum’s 11 units. Capacity-building activities included supporting physicians to enroll in a master’s course on rehabilitation at Hanoi Medical University. However, no one had started the program because of a lack of interest. Instead, the PDSP supported: 12 physicians from Danang, Binh Dinh and Dong Nai to attend a six-month training course on physical rehabilitation in Hanoi; 24 nurses to take a six-month transition course to become rehabilitation specialists in Danang; and a five-day rehabilitation course for doctors and technicians. Interviewed MOH officials were not sure VNAH used the same rehabilitation model as the community￾based rehabilitation model that MOH used. The head of AMSA asserted that all agencies and organizations involved in rehabilitation in Vietnam should use the model that is supported by the MOH. Evaluation of USAID Vietnam Disabilities Programming 36 Community-Based Support Context: To encourage and support the initiative of persons with disabilities to advocate on their own behalf and to provide alternative service delivery approaches to better meet their needs, USAID provides direct assistance to disabled people’s organizations. USAID also supports parent associations to work with local officials to mobilize resources to support their children with disabilities. Building capacities of DPOs and other community-based associations. The Law on People with Disabilities indicates that a DPO is a social organization that represents “the legitimate rights and interests of persons with disabilities. DPOs shall be responsible to participate in planning, monitoring the implementation of disability laws, policies, programs and plans.” In the past decade, DPOs across Vietnam have grown rapidly with assistance from multiple donors, including USAID. USAID assistance, through VNAH, CRS and DAI, helped establish and empower 17 DPOs and parent associations nationwide. As noted, DPOs have helped persons with disabilities participate in law and policy development. They also serve as a mechanism to deliver modest but essential services to improve the lives of persons with disabilities, including the provision of medical devices, referrals for specialized education and job fairs. USAID assistance has focused on building their service delivery capacities. The team interviewed members and officers from DPOs in all four provinces. The majority of board members were persons with disabilities, typically physical or mobility disabilities. DPOs function primarily as quasi-governmental service providers. DPOs support themselves through a government assistance annual allowance, membership fees, fundraising and funding from IPs. However, the majority of them expanded their funding base through financial support of INGOs. The organizational development support provided by USAID was viewed as relevant, and the key informants said they were “satisfied” to “very satisfied” with the assistance and support. All of the support related to operating service delivery programs. DPOs that participated in USAID activities did not receive any leadership or advocacy skills training. When asked about support activities related to advocacy, a representative from one DPO said it was denied funding by the implementing partner because the USAID project did not fund that type of activity. As a result, they obtained funding from an INGO. The lack of leadership and advocacy skills training and support for strengthening DPOs’ advocacy skills may be due in part to the implementing partners being quasi-governmental service providers, not civil society organizations for and by persons with disabilities. CRS promoted the activities of parent associations in Ninh Binh and Quang Nam provinces, the project sites where CRS implemented inclusive education activities. Thirty-seven associations were set up and their officers trained in Ninh Binh and Quang Nam provinces with small grants from the Inclusion of Vietnamese with Disabilities Program (IVD), and CRS established an additional 10 in Quang Nam using CRS- own funds but with training on establishing and sustaining an association provided through IVD. CRS expects that parent associations’ capacities will be developed to directly support their children and to collaborate with local government and the community in mobilizing resources to support children with disabilities. Members interviewed said they were very satisfied with the assistance. Evaluation of USAID Vietnam Disabilities Programming 37 Inclusive Education Context: From 2006 to 2010, CRS promoted the inclusion of children with disabilities in regular classrooms. The project focused on capacity building, early detection and intervention and the transition among different levels of education from nursery school through secondary school. Activities included training teachers , who serve as resource persons, and administrators in inclusive delivery and management; an inclusive education pilot project in six districts in Ninh Binh and Quang Nam provinces for secondary schools; equipment for schools; equipment and devices for children; and a file referral system between primary and junior secondary schools to ensure smooth transition. CRS also worked closely with MOET to develop guidelines and materials on inclusive education related to nursery, primary and junior secondary schools and construction and academic guidelines for national standardized schools. CRS built ramps in a number of schools and was piloting an information communications technology (ICT) project to improve the quality of education for children with disabilities. In a separate component, VNAH/DAI implemented a school-to-work project by providing vocational training to youth at special education schools. As shown in Figure 9, 71 percent of caregivers responded that they were “satisfied” or “very satisfied” with the service provider, and 67 percent were “satisfied” or “very satisfied” with the education assistance they received. School enrollment for children with disabilities. Team members interviewed the director of the Primary Education Department, the vice director of the Secondary Education Department and the officer of the Primary Education Department of the Ministry of Education and Training (MOET). Key informants stated that approximately 100,000 children with disabilities were attending school, with 56,000–57,000 of them in primary school. Key informants said few students with disabilities had obtained disability certificates because many families refused to go through the administrative process. As a result, their school enrollment was delayed. Once in school, they were stigmatized because they were much older than their counterparts without disabilities. Consequently, they would often drop out of school. Key informants said another reason for the high dropout rate among children with disabilities was that secondary schools frequently were farther away and the students with disabilities were unable to secure reliable transportation. Training for teachers. MOET identified topics for in-service training of inclusive education resource teachers in provinces (e.g., hearing disabilities, visual disabilities, autism, etc.) and the USAID-supported training would then be mainstreamed into the in-service training that teachers attend each year. Each summer, 120 45-minute sessions were typically offered. Central-level personnel trained a few resource people at the provincial level, and the provincial resource people trained their counterparts at the district level, who then trained a few people from each school. School resource people then trained 0% 20% 40% 60% 80% 100% Support/Service Service provider Figure 9: Level of Satisfaction Among Children (Care Givers Responding) Who Received Education Assistance Evaluation of USAID Vietnam Disabilities Programming 38 other teachers. Key informants stated that it might not be the most effective way to provide inclusive education training because it followed a cascading system and no quality control was available at lower levels, so MOET could not control who participated in trainings at the province or district levels. In addition, education for children with disabilities accounted for only a small portion of the 120 training sessions. School teachers could also receive training on inclusive education from provincial inclusive education centers, but only 13 exist in the country. Teachers told the evaluation team that they benefited from better teaching plans provided by USAID-assisted capacity-building activities. In the project provinces of Ninh Binh and Quang Nam, in line with the CRS’s technical approach, in￾service training for teachers was supplemented with follow-up through a sample teaching class, lesson observation and technical support for teachers in regular classes, thematic workshops, etc. In Danang and Dong Nai, the PDSP project supported nearly 80 teachers in a long-term training. Participants received a second bachelor of arts certificate in inclusive and special education at the Ho Chi Minh City University of Education. Accessibility. CRS built ramps for schools from 2006 to 2010. In Ninh Binh, all five schools participating in CRS’s activities on inclusive education constructed wheelchair ramps to increase accessibility for students with disabilities. CRS also mobilized funds from the government and communities in Ninh Binh and Quang Nam provinces to construct new school infrastructure, including ramps, or make repairs. MOET then took over this initiative to replicate in a later MOET- Asia Development Bank (ADB) secondary education development project. Key informants stated that the activity provided a model on accessibility for regular schools. Unfortunately, schools are unable to install ramps without support from CRS because they have no budget for it. Moreover, the provincial governments are unable to approve construction of new, accessible schools because of budget constraints. VNAH worked the Ministry of Construction to develop accessibility standards for buildings that were incorporated into the standards for national-level schools. Specifically, each school has to follow the accessibility standards published by the Ministry of Construction and have two inclusive education resource rooms. Monitoring. According to Decree 20 on General Education (2013), regular schools have to admit all children, with or without disabilities. MOET is responsible for monitoring the progress of implementation of general education by comparing the percentage of children with disabilities in school to the percentage of children with disabilities in the area. The specific decision to admit children with disabilities does not take place at the ministry level. An official noted that MOET can only recommend that inclusive education is best for children and recommend that children go to special schools if inclusive education cannot meet their educational needs. Online learning. The purpose of the three-year project is to improve the quality of education for children with disabilities who use ICT to attend primary and secondary school. CRS developed software to teach children with visual impairments and developed a set of videos to demonstrate to teachers how to implement the lessons by using sign language. It included 20 video lectures on Vietnamese language instruction and 20 video lectures for Grade 5 math. MOET reviewed the first version and developed an improved set of videos. MOET proposed a pilot project to provide distance learning to support school-based lessons for the junior secondary level, with 60 sessions in Vietnamese language and 50 sessions for Grade 6 IT. MOET decided to develop the curriculum for Grade 6. However, the online curriculum has been developed for Evaluation of USAID Vietnam Disabilities Programming 39 only two of 13 subjects required for the junior secondary education program. Since the project did not develop curricula for all courses, participants of the pilot cannot receive a certificate at the end of junior secondary school. Key informants from MOET explained that the purpose of this pilot was to test the feasibility of online learning, not to provide formal courses with certification. MOET is planning to develop a more comprehensive program with certification for learners. As part of the pilot project, children with disabilities, teachers and parents received ICT training. Students and schools received 55 computers and laptops, as well as 10 televisions and 11 DVD drives to support the use of video lectures. Both the project budget and MOET-mobilized resources covered the costs, demonstrating the interest and efforts of MOET in this intervention. However, key informants stated that MOET officials were concerned that the online curriculum pilot was too small to make any relevant policy recommendations. Home-based education. The pilot of home-based education was designed to examine alternative educational services for those who were unable to go to schools due to distance or mobility challenges (for example, they could not sit or move); the aim was to provide both academic and social/life skills to the children. Parent associations and nearby schools used individual education plans (IEPs) to conduct this activity in Quang Nam and Ninh Binh. Later, the home-based education initiative supported distance learning using ICT, implemented by MOET with technical and financial assistance from CRS. This took place in nine provinces and offered two subjects, literature and Grade 6 math. The latest students started in February 2015; as of March 2015, 277 were students enrolled in the pilot program at 21 schools in five provinces. Both teachers and caregivers could receive training in home-based education was offered. In nine provinces, children with disabilities use the site www.giaoduchoanhap.edu.vn to study with support from experts from the provincial department of education and training (DOET) or the Bureau of Education and Training (BOET). Students will not receive certificates. The DOET in Quang Nam Province said 156 of 198 children with disabilities in Hiep Duc District are in school. When asked about the other 42 children, the official said they had severe disabilities and could not attend school; they lie in bed and could not move. Nineteen children received home-based schooling from 2010 to 2013. Teachers developed independent education plans for each child, trained parents to implement the plans and conducted home visits to follow each child’s progress. CRS did not provide direct assistance for home-based schools during this period; however, knowledge and skills from CRS-assisted capacity-building efforts before 2010 enabled this home-based schooling. At the time of the evaluation, no home-based schooling was underway in project areas, as indicated in the MOET report on the pilot project on home-based education, as well as from the observation of the evaluation team. Teachers may coordinate with parents, but the parents do not receive curricula or learning materials to use at home. Parents are expected to call the school if students have problems. Without that call, the school assumes that everything is all right. When asked if the school does anything for these children, DOET said the school maintains contact with parents who report problems. Education reform. The national education reform program was approved a week prior to the evaluation. The agenda, assessment and test regulations were not finalized. Therefore, it was unclear how to continue the regulation on inclusive education. Given the MOET leaders’ strong commitment to inclusive education, it will be incorporated into the reform process. Education and Training Unit in Hoi An City. Evaluation team members interviewed three people from the unit, which received An Effective Study Tour Key informants said former Vice Minister Dang Huynh Mai participated in a 2012 study tour to the United States and was impressed by the inclusive education models. The vice minister instructed MOET officials to prepare and issue Decision 23. From these trips, MOET officials changed their attitudes toward inclusive education (IE) and became the first generation of IE resource people in Vietnam. Evaluation of USAID Vietnam Disabilities Programming 40 CRS support from 2006 to 2009. In the area of education, support included training for school managers and teachers, dissemination of information related to inclusive education, and school management. Key informants identified the same challenges that the DOETs identified. In addition to those challenges, they identified the following: • A lack of teaching aids and materials for children with disabilities; • A high turnover rate among inclusive education school managers and teachers; • A lack of training for new managers and teachers and, as a result, schools unable to maintain professional inclusive education personnel; and • No resource room in the unit to support inclusive education. Inclusive education schools. Evaluation team members interviewed school managers and teachers from An Phuoc Primary School (Commune Hoa Phong, Hoa Vang District, Danang Province); Cam Thanh Primary School (Hoi An City, Quang Nam Province); and Binh Minh Kindergarten (Que Binh Commune, Hiep Duc District). All key informants confirmed that their schools participated in the USAID inclusive education initiatives and received assistance, including inclusive education training for teachers and equipment for the resource room for children with disabilities. Teachers also received compensation (VND 300,000 or USD $15) for providing additional sessions to children with disabilities in the resource rooms. Key informants, including school managers, teachers and parents, were satisfied with the assistance provided. They noted that resource rooms, where children with disabilities received training on soft skills, were very useful and helped to facilitate integration of children with disabilities to the formal education environment. The biggest challenge they mentioned to the evaluation team related to the sustainability of the resource room after the project ends. Other challenges included student load (too many children in a class) and that some equipment and learning materials in the resource room were inappropriate for children with disabilities. For example, some education materials were too difficult for children with intellectual disabilities. Children with disabilities and family members. Evaluation team members interviewed JM, a young girl from Tan An Township (Hiep Duc District, Quang Nai Province). Her mother attempted to take her to school, but the school said they could not accommodate her. JM received home-based schooling from 2008 to 2013. Her mother said the schooling was very good. The teacher taught her handwriting, math and literature. He came to the home three times a week for two hours each time. When possible, he would come on weekends. JM was enrolled in home-based schooling for five years. In the middle of fifth grade, the project ended and she was unable to finish primary school. She was unable to earn her certificate. At the time of the evaluation, she was taking the online course. JM’s mother confirmed with MOET that teachers no longer came to the home. JM quickly mastered the computer (typing and how to use certain software). She said she played games, used Facebook to make friends online and watched movies for entertainment. She was taking the literature course and wanted a certificate of accomplishment for the program, as well as to finish high school. A grandfather from Hiep Duc District was interviewed. His grandson, attending inclusive kindergarten, had undiagnosed Waardenburg syndrome and is deaf. Although the child was in an inclusive classroom, the teacher did not know sign language and did not have materials to teach the child sign language. The grandfather wanted to send the child to a special education school in Danang, but could not afford the tuition. Although a school in Ho Chi Minh City serves children who are deaf, the grandfather did not want to send his grandson to that school because it was too far from home. Evaluation of USAID Vietnam Disabilities Programming 41 A MOET official informed the evaluation team members of an inclusive education school that she recently visited. The teacher said that inclusive education appeared to be successful for a student who was blind. The teacher said the young girl participated in the classes and seemed to be taking notes. But upon inspection of the girl’s work, the official realized that the girl had been scribbling, not writing. Human resources for inclusive education. The current education sector is a two-tiered structure with a network of special education and regular education schools. The quality of the inclusive education is limited, in the judgment of the staff and parents’ level of satisfaction. Also, “human resource capacity is limited; services provided are not comprehensive and do not meet the diverse needs of persons with disabilities; facilities and infrastructure are not well-equipped; interagency coordination is still poor and there is big gap in quality of services provided by different levels. Services are better developed at central and provincial levels than those provided at local and community levels.”25 Given the high turnover rate of teachers and school managers, no plan to train new teachers and school managers appeared to be in place. MOET incorporated inclusive education into the standard summer teacher training courses, but several teachers and local education officials said that not enough time was available to train educators fully in inclusive education. DOET, the Education and Training Unit and inclusive education school key informants identified several human resources issues. First, teachers experienced work pressure to accomplish the standard teaching content as regulated by MOET while also meeting their responsibilities for students with disabilities. Second, districts could not afford to pay the allowance to inclusive education teachers, as required by Decree 28 (April 10, 2012). Third, local commune officers lacked the capacity and expertise to sit on the assessment board that identified the type and level of disability, especially for children with autism or intellectual disabilities. Special education schools. Evaluation team members interviewed senior staff from DOET in Quang Nam, Dong Nai and Hiep Duc District. Staff from both DOETs confirmed that MOET instructed them to incorporate inclusive education in schools, but Dong Nai maintains a special education school “since there are children with too severe disabilities to blend and develop in inclusive education.” Evaluation team members interviewed senior staff from two special education schools: Children with Disabilities Center (CDC) in Dong Nai and Nguyen Dinh Chieu Special Education School in Danang. One school’s admission criteria stated that the child must be at least 6 years old, have a medical certificate showing that he/she cannot go to a standard school but can dress, toilet and feed himself or herself. Also, the child is required to have “ability to learn.” Key challenges in implementing USAID education assistance. Key informants identified several challenges in implementing USAID-supported education interventions for persons with disabilities. First, they noted a high turnover rate of CRS staff, which resulted in delayed activities as new staff are hired and trained. Second, key informants found the process and procedures to implement the educational assistance in the case management system to be complicated and time-consuming, sometimes needlessly, and resulting in delayed services. Third, the financial mechanism in CRS projects does not encourage MOET to participate and have ownership over activities. For example, CRS makes direct payments to service providers and bears full 25 Nguyen Thi Hoang Yen & Nguyen Thi Thu Huong (2013). The Policy on Education of the Disabled in Vietnam – the Gaps and Its Impact on the Persons with Disabilities. VNU Journal of Education Research, 29(2), p. 31. Evaluation of USAID Vietnam Disabilities Programming 42 responsibility to arrange meetings, consultants, training and building resource rooms. MOET is responsible for professional content but has no control over the activities, resources and conditions to do it. Key informants noted that other donors, such as UNICEF, provide direct funding assistance to MOET, which can then organize activities. This gives them greater incentive to take full ownership of the activity. Fourth, several KIs noted that individual program components in specific sites or provinces may be too limited in resources to achieve the desired impact. For instance, the online/home school project included only two subjects and was doing little to influence the entire school program. Employment and Livelihood Support Context. Vietnam has a long history of promoting a work life for persons with disabilities. On the heels of a slogan from Ho Chi Minh that “Being disabled is not being unable,” persons with disabilities in Vietnam try to work and earn a living. Employment provides opportunities for income, capacity building and social inclusion and hence for becoming a full contributing citizen. Historically, employment of persons with disabilities primarily occurred in an exclusive environment. There were “enterprises of persons with disabilities” and “factories of invalids with severe disabilities.” Foreign-invested companies, most of which were manufacturing contractors of international retail brands, later introduced the inclusive working environment. The approach in delivering employment and livelihood assistance for persons with disabilities has been to develop and support partnering relationships with various local institutions and the private sector to build capacities to provide vocational training, job preparation and placement services, IT career development, livelihood apprentices and a small microfinance program to seed livelihood ventures. Overall, beneficiaries receiving employment or livelihood assistance had high satisfaction, more than 70 percent rated the services “very satisfactory” or “satisfactory” (Figure 10). Less than 10 percent expressed dissatisfaction. Vocational training. Vocational training has played an important role in USAID employment support interventions. Different training alternatives have been developed and implemented, taking both formal and informal approaches. The formal training has been provided at the public or private training institutions, and the students received certificates upon graduation. Among the formal training, the Information Technology Training Program (ITTP), for which CRS developed the materials and trained the trainers, is a model providing nine-month to one-year formal training with certification. ITTP has been implemented since 2007 in colleges in three geographical areas of Vietnam: Ho Chi Minh City, Hanoi and Danang. Reported data as of March 2015 documented 787 persons with disabilities graduating from the training, with 54 percent finding jobs or paid internships within 12 months after graduation. 0% 20% 40% 60% 80% 100% Very dissatisfied Dissatisfied Neutral Satisfied Very satisfied Tool/Service Service Provider Figure 10: Level of Satisfaction Among People Who Received Employment/Livelihood Assistance Evaluation of USAID Vietnam Disabilities Programming 43 Views on IT training’s effectiveness on inclusion of persons with disabilities in employment. Stakeholders in IT interventions have differing views about the effectiveness of the training on employment for persons with disabilities. Officials with the Da Nang-based Dong A University, for example, stated the formal training institution’s view that a six-month course was too short, as persons with disabilities needed more time to acquire the skills. They also noted that the course contents should be more advanced and provide skills for a wider labor market, instead of specific skills for some employers, which made trainees dependent on those employers and unable to change jobs. They said the course should also be appropriate for the local provinces. For example, the persons with disabilities coming from outside Da Nang learned skills that made them employable in Da Nang, but it was difficult for them to find employed when they returned to their original provinces. The provincial DPO in Da Nang shared another observation: They introduced young persons with disabilities to IT training at universities, and the individuals returned to the DPO to seek assistance for a job because the university could not help them. The DPO questioned the quality and appropriateness of the training, as well as the ability of formal training institutions to link to employment opportunities after the course. A third view came from employers. The KIs with IT companies in Ha Noi and Da Nang showed that a low rate of persons with disabilities passed the entry skill test to get employed after graduating from the formal IT training courses supported by USAID’s programs. The employers explained that the formal training institutions may have problems with the practice, equipment and software used for training, or did not provide a format for candidates to share their experiences after passing or failing, to help them improve their job search skills. They also noted that the formal institutions did not provide proper employment counseling before referring the graduates to companies, resulting in a high drop-out rate among those who were offered employment. The companies also suggested that the training should not be too long or too advanced. Students should learn basic and specific skills and then get employed first; after they accumulate some workplace experience, they could pursue more appropriate and advanced training to foster further career development tailored to the more advanced, better-paying path they wanted to follow. Some key points:  Information technology encompasses a wide range of skills and levels, requiring different types of training. Choosing what skill or level a trainee should pursue would depend on the needs and priority of the person with a disability and the job he or she wants after training, as well as the requirements of employers and the status of the local labor market.  The contribution of IT to employment and inclusion of persons with disabilities would depend greatly on the linkage that the training contents and training institutions have with the demand of the labor market and the most common employers in the market. If the training institution closely follows the market and works with employers to help them understand what to expect from persons with disabilities and how to accommodate them, they could better adapt and update the curriculum and practice, vastly improving graduates’ employment opportunities.  Giving sufficient information to persons with disabilities through proper vocational counseling before the course and employment counseling after the course is vital to both the persons with disabilities and the employers. In this context, training needs analysis and counseling for both persons with disabilities and employers would be critical for the success of IT training and employment inclusion for persons with disabilities. Evaluation of USAID Vietnam Disabilities Programming 44 The informal training approach contracted companies to train persons with disabilities in the skills needed to work in the company, with the activity offering capacity-building assistance to develop materials and train providers. At the end of training, students took a test to determine if they learned the skills. Students who passed the test were offered positions in the company. Students could also look for a job through the ESC or file an application directly with a company introduced by the training school, family, friends or organizations for persons with disabilities. Development of vocational training materials. The team interviewed a senior official in the Bureau of Employment (BOE) at MOLISA. VNAH supported the BOE in developing vocational rehabilitation training materials and organizing three training courses for 230 social workers at the provincial level who work at ESCs. The informant stated that the training materials were the most important activity to improve services for job seekers with disabilities, and said he was “very satisfied” with the activity. He noted the bureau had not previously had standard training materials in vocational rehabilitation. The informant also expressed satisfaction with how the project mainstreamed vocational training for persons with disabilities in the employment events at the Hanoi Employment Introduction Center (HEIC). He said he was “satisfied” with the training courses, but noted room for improvement. He added that pre- and post-tests should be used to assess the knowledge transfer of participants and suggested that fewer lectures and more practical or participatory activities would be beneficial. Moreover, he said field visits would enable participants to see how the vocational rehabilitation model worked in the field. The bureau planned to expand its model on employment and vocation rehabilitation to employment centers in Hai Duong, Danang, Dong Nai and Can Tho. The bureau had funding and training materials to replicate the model, but did not have experts to demonstrate how to consult with persons with disabilities in job fairs. The bureau wanted training in a practicum with an employer in Hanoi’s ESC. Support to the HEIC vocational rehabilitation model. The team interviewed a senior staff member and two other key staff members. HEIC worked with VNAH assistance to build its capacities to establish a persons with disabilities vocational rehabilitation model and integrated job fairs26 and to support regular employment counseling/job placement for workers with disabilities. Starting in 2013, HEIC worked with CRS to provide employment counseling to students with disabilities who graduated from CRS-supported ICT courses at Hanoi ICT College. HEIC implemented the vocational rehabilitation model in 2013. From 2013 through the spring of 2015, HEIC served 189 persons with disabilities under this model. Of the total, 100 were working at the time of evaluation. They used a four-step service delivery process: 1) registration and screening, 2) development of personal plan, 3) reference of persons with disabilities to available services, such as 26 Two job fairs took place in 2013 and four in 2014. As of 2015, the job fairs were incorporated into their annual work plan and, as a result, no longer need support from VNAH. “Before the project with VNAH, persons with disabilities job seekers came to HEIC and received the counseling service, but HEIC staff did it without professional technique and adequate accessible infrastructure and equipment. I feel the USAID-funded project was very useful for my work. It showed me a systematic approach and equipped me with practical skills and a process to work with persons with disabilities. I interviewed workers with disabilities, job seekers and families. They also thought our service was very useful and thanked the project. “ — HEIC staff member Evaluation of USAID Vietnam Disabilities Programming 45 vocational training and physical rehabilitation and 4) provision of after-training support and monitoring when the person secures a job. When asked to rate the level of satisfaction with the activities, one informant indicated that he was “very satisfied” with the staff training in skills in vocational rehabilitation, counseling and job fairs, adding that access to U.S. experts was critical to the training’s success. He said he was “satisfied” with the vocational rehabilitation model but thought a study tour to the United States to see how the model was applied would have been helpful. He was also “satisfied” with the provision of an elevator to access the third floor and the availability of devices for staff to talk to people with hearing impairments. Key informants said a major limitation of the project was that it did not prepare participants to qualify for jobs in certain companies. After the vocational training, the participants’ skills were insufficient to obtain some jobs. Not enough persons with disabilities met the demands of large labor companies. Interviewees identified challenges to providing employment services to persons with disabilities at the current level and quality. The primary challenge was to secure enough resources to organize more mobile services in remote districts and to pay for staff travel and cell phone expenses. Another challenge was adapting the U.S. model to the Vietnamese context. When asked what challenges persons with disabilities face in finding employment, the key informants cited a lack of accessibility infrastructure at companies, a lack of accommodations for people coming from distant provinces and a lack of accessible transportation. They noted that many persons with disabilities were unable to attend vocational training courses or go to work because motorized wheelchairs were too expensive and public transportation remained inaccessible. Support to the ESC vocational rehabilitation model. USAID provided extensive support to the government’s ESCs. These centers, located in the project provinces, provided traditional employment services (e.g., information, counseling, job introduction and job fairs). The ESCs received assistance to meet the needs of job seekers with disabilities, including installation of ramps, availability of special skilled counselors and tailor-made service manuals and information sheets for day-to-day services. Monthly job fairs were initiated to facilitate convenient access for persons with disabilities. USAID supported the implementation of the vocational rehabilitation model, whereby assistance is provided to address physical and attitudinal disabilities that limit vocational opportunities, in the Hanoi ESC. Other ESCs received USAID support in staff training, materials development, equipment and, in some cases, service fees. Project staff provided follow-up support with employers and persons with disabilities at the workplaces. The team interviewed a senior staff member at ESC Danang. In 2003, ESC Danang was one of two centers in Vietnam to receive capacity building and support from VNAH to build a vocational rehabilitation model. The program went through three phases:  Phase 1 (2003–2006)27: ESC focused on counseling skills training for staff. The training gave basic skills (e.g., how to: talk to persons with disabilities to assess disabilities, conditions and situations; encourage persons with disabilities to speak out and become more confident; and change attitudes of employment counselors in working persons with disabilities).  Phase 2 (2006–2009): ESC established a garment company to include workers with disabilities. The company closed because of poor business performance. From this experience, the ESC Danang realized that the idea of setting up such a company was not effective, in part because persons with disabilities and family members perceived their employment as charity or special 27 Dự án Việt Mỹ in 2006 then expanded to 2007. Evaluation of USAID Vietnam Disabilities Programming 46 treatment. Additionally, the company had segregated employees with disabilities from their counterparts without disabilities. The board realized that segregation was not good for anyone.  Phase 3 (2009–2011): ESC refocused on employment counseling and job placement for job seekers with disabilities and incorporated the services into the center’s regular activities. In this phase, ESC tried to encourage persons with disabilities to make their own efforts and come for jobs instead of reaching out and offering jobs to them. A consultative group was formed and met every two months to discuss progress and results. A VNAH staff was on-site at the ESC to help and to monitor job seekers with disabilities during implementation. When asked to rate his level of satisfaction with the USAID assistance received, the informant said he was “very satisfied” with the training in counseling skills and vocational rehabilitation, implementation of the vocational rehabilitation model and the technical assistance received from VNAH. He said he was “dissatisfied” with the establishment of the company for workers with disabilities. The ESC tailored services for job seekers with disabilities. Staff analyzed job descriptions at specific companies and, when seeing that persons with disabilities were capable of doing most of the jobs, negotiated with companies to change job descriptions and introduce jobs for persons with disabilities. Although the informant was “very satisfied” with the vocational rehabilitation model the ESC adopted, he said it did not successfully link persons with disabilities to health centers for physical rehabilitation, have suitable job seekers with disabilities referred to ESC for employment services or link persons with disabilities to communities to promote social inclusion. Key informants stated that the ESC had two integrated (inclusive) job fairs per month. The fairs were easy for persons with disabilities to access, especially the mobile job fairs in the local communities. They said they believed that if DPOs introduced the job fairs to persons with disabilities, job seekers with disabilities would be more confident to attend the fairs. Employment support through DPOs. DPOs established and trained with USAID assistance help to provide vocational training and livelihood support via small USAID grants. DPOs also work with and through the USAID-supported BREC to connect job seekers who have disabilities to potential employment with BREC’s member companies. DPO Hanoi. A senior member from DPO Hanoi stated that the DPO proposed that the People’s Provincial Committee organize a soft loan scheme (livelihood project) and vocational training courses for persons with disabilities at Hanoi ESC and district-level vocational training centers. He said he was “very satisfied” with the livelihood project and training courses. DPO Hiep Duc District. A senior member of DPO Hiep Duc District (Quang Nam Province) stated that the DPO members were involved in vocational training via support from DAI/VNAH. Participants received allowances during the training. He said he believed the training was successful for the participants, but that the project needed to expand. Only 20 of 1,600 persons with disabilities in the community participated in the training. The DPO was exploring other options for DPO members to secure employment. DPO Danang. The team interviewed three individuals from DPO Danang. Members were involved in the CRS IT vocational training project and the PDSP livelihood project. Members stated that they recommended a young person with disabilities for the IT training. When he completed his training, he returned to the DPO for a job. Another member received vocational training from an ESC. At the time of evaluation, he was still looking for a job. People interviewed said the quality of the IT training did not provide enough practical experience and, as a result, members who participated in the training did not meet the company’s requirements. They also Evaluation of USAID Vietnam Disabilities Programming 47 identified problems from the workers with disabilities; their requirements for working conditions and salary were often too high. Others were “too sensitive” while communicating with managers and co￾workers. Support to Hoi An Association for Young Persons with Disabilities. The immediate past chair and the chair at the time of evaluation were interviewed. Members of the association participated in vocational training through the CRS activity. The key informants reported that of the 39 individuals trained, a number overcame barriers and succeeded. One woman opened a shop and eventually married and had two children. In another case, a person received a wheelchair and was able to go to school and learn to read, then started a small business that resulted in stable finances with valuable assets (e.g., a motorbike). The key informants stated that many of the vocational training participants became successful, and could then support their families. IT Training at Dong A University: The team interviewed five staff members at the University. Dong A University provided an information technology training program (ITTP) for persons with disabilities. Potential students were introduced through DPOs, CRS and public announcements. Students were selected through individual interviews and were accepted if they had graduated from a secondary school. The training program spanned six months, but students were not required to attend the entire time. They could attend select courses available at different times. Although the total course duration was six months, students with disabilities took more time to complete the training. In addition to ITTP, students with disabilities received soft skills training (e.g., how to prepare job applications). During the course, students are sent to companies that employ persons with disabilities for either short visits or to participate in an internship program in which students worked on simple tasks. After the training, students were placed in jobs through job fairs or ESC. USAID capacity-building support included financial assistance to cover the cost of training facilities, student stipends (living expenses) and tuition, outreach activities and expenses to place graduates with enterprises. Technical assistance was provided 1) to build capacity, through training and workshops, of teachers in inclusive education for persons with disabilities, 2) for information-sharing events with universities and schools where trainings were provided, 3) to organize job fairs and 4) to connect ESC to graduates with disabilities. All five staff members said they were satisfied with the support from CRS. Dong A University’s records showed that of 131 people enrolled for the training, 111 finished coursework and 68 of those received certificates. A total of 46 people secured stable jobs. The university included a follow-up component to the training. Employers said that some individuals with disabilities were limited by their physical health and, as a result, had difficulty following a normal work schedule. Others said the persons with disabilities were not able to produce reasonable incomes; they were not patient and readily quit their jobs. Still others said some persons with disabilities have a good attitude but limited capacity. The university intended to continue the training from 2015–2018, but with revisions. Staff planned to expand selection criteria to include unemployed persons with disabilities and their counterparts who were employed but needed more skills training, as well as those who were enrolled in other, non-IT universities. The course content would be more advanced and focused on market-oriented skills (Web design, graphics, e-marketing and office IT). Course duration would increase to nine months. Follow-up plans for each graduate would be developed to ensure graduates receive ongoing support. Private-sector partnerships for jobs. The team interviewed three people from the BREC. The Blue Ribbon Employer Council, established in 2007, resulted from the partnership between VNAH and the Vietnam Chamber of Commerce and Industry (VCCI), with funding support from USAID. The main Evaluation of USAID Vietnam Disabilities Programming 48 purpose of BREC is to promote the employment of persons with disabilities in Vietnam. BREC reports that it began with 25 members and, at the time of evaluation, boasted a membership base of 160 employers representing both international and national companies and organizations. At the time of evaluation, BREC was staffed and coordinated by VCCI and VNAH, chaired by the director of the Bureau of Employers’ Activities of VCCI and financed by USAID. Capacities developed in BREC through USAID assistance helped BREC support integrated job fairs through the ESCs. Job placement services were provided via the network of BREC members and through awareness campaigns to other companies. Other support included capacity building for BREC, advocacy for BREC to become an independent entity and awards ceremonies for outstanding workers with disabilities and employers. Key informants from other government agencies (at both central and province levels) commented that BREC was effective in awareness-raising for international employers to meet their corporate social responsibilities. At the end of 2014, BREC had approximately 250 member companies and employed about 2,000 persons with disabilities When DPO members were interviewed, key informants said that BREC works with DPOs and persons with disabilities to disseminate information about available jobs in their member companies and to share contact between DPOs and business members. Discussions were ongoing regarding the future of BREC. No business plan was in place to continue BREC post-USAID assistance in the council’s form at the time of evaluation. An employer’s perspective. The team interviewed the chief executive officer (senior officer) and the senior officer’s assistant from the Vietnam Business Outsourcing Service (VBPO), which has been a BREC member since 2011. VBPO, in Danang Province, employed more than 200 people, 20 percent to 30 percent of whom are persons with disabilities. His agency provided business-outsourcing services primarily to companies from Japan (90 percent). American and European companies were expanding and, as a result, the senior officer anticipated that he would employ more than 300 people by the next fiscal year. When asked about DPO Danang, he said they were involved in closed activities and did not consult or link with other organizations. He further asserted, “[DPO Danang] is crazy about protecting persons with disabilities without reasoning.” In his opinion, members were not integrated into the social life and practice. Consequently, the DPO made policy recommendations based on their needs and did not consult with other individuals with disabilities. When asked about BREC, he said he believes its key achievements included its ability to “make a big voice” about the employment of persons with disabilities. Additionally, he expressed many ideas on how to promote employment for persons with disabilities, saying BREC’s biggest challenge was its official legal entity. At the time of evaluation, BREC was not yet an independent organization and had not decided which model was best for it. BREC members were working on a solution, setting up a task force and preparing a general meeting to present a new solution to members. The meeting was scheduled for May 2015 and would be organized by BREC members without any external support. VBPO participated in an on-the-job training project from 2011–2014. The senior officer implemented four three-month courses, one of which was for persons with hearing impairments. The course eventually evolved into a six-month course to give enrollees enough time to learn the skills. The training Evaluation of USAID Vietnam Disabilities Programming 49 was tailored to the company’s employment needs. Since 2015, the company had implemented two training courses for six months, one for VNAH and one for CRS. The company accepted referrals from DPO Danang and other DPOs, CRS, VNAH and VBPO employees. The entry requirement was an IQ test and commitment. The level of commitment was determined through the interview. Trainees received cost-shared IT training and stipends. Soft skills training (e.g., teamwork, communication, work ethics, spending money, work discipline, working attitude, attire) was provided through another funding source. VBPO received training fees and later received qualified employees. The senior officer indicated that it had become more difficult to hire competent persons with disabilities because most of them were already employed. Training persons with disabilities presented several challenges. First, longer courses would encounter against budget constraints. More persons with disabilities were enrolling for each course, and they were seeking people with higher-level skills (e.g., accounting). An additional challenge was the attitude of trainees; he said that some individuals were unsure about employment and were not eager to learn. At the time of evaluation, 23 trainees had received jobs in his company. Several had been promoted to supervisor/manager positions. The senior officer noted that persons with disabilities exhibited improved skills and confidence. They had stable incomes. The company received many benefits from the project, including a low turnover rate for all employees. The senior officer noted that persons with disabilities worked hard, but tended to have more health problems than employees without disabilities. He noted no difference in productivity between employees with and without disabilities. Another benefit to the company was that it received awards for being an innovative IT company that hired persons with disabilities. When asked about the type of benefits he received from being compliant with national laws, the senior officer said this did not provide any direct benefits. However, he said customers — particularly Japanese companies — were appreciative of the company hiring persons with disabilities. Swiss customers highly appreciated the price and quality of work. Senior Official at VBPO Commenting on Training Needs Decent employment means ensuring a continuous career development path for workers with disabilities, which allows them to get better skills and be promoted. In the BPO business, continuous improvement is a must. Data entry is only a starting skill and is indeed relatively challenging for older workers. They easily fail the speed and error requirements, which are the most important key indicator to keep the business competitive. In VBPO, the oldest person to get training is 35 years old and the oldest employee is 40. As the workers age, it would be better for them and for the company that the workers improve their skills in order to be promoted to more advanced tasks, such as accounting or design. However, promotion to higher positions would require more training, which workers with disabilities are not always eager to do, but if workers remain with the simple jobs, they will lose productivity over time. As a result, the company will have to replace them or compromise competitiveness. -- Senior Official from VBPO Evaluation of USAID Vietnam Disabilities Programming 50 The senior officer planned to continue to train and hire persons with disabilities. However, several challenges remained. First, financial incentives provided by laws/policies were too low and too difficult to access. Second, not enough persons with disabilities with adequate hard and soft skills wanted training and jobs. Third, advanced training for existing employees with disabilities and new recruits would need to be longer and better equipped. For example, an individual would have to receive one more month of training to perform salary accounting in Japanese. To provide design services, the company would need more powerful computers for training, but the company did not have money to invest in the equipment. The senior official also noted that persons with disabilities were not always eager to take more training to be promoted or to find better employment. Therefore, they would block their own career paths and company development. Key challenges in securing full employment opportunity for persons with disabilities. The USAID support was viewed primarily in a positive manner. Some beneficiaries benefited from the project, while others continued to have difficulties securing and maintaining employment. Those who participated in apprenticeships appeared to gain better opportunities for long-term employment. To increase employment for persons with disabilities, the USAID Disability Support Program used three models: 1) match individuals with companies, 2) match people with small business employers within the local community and 3) train people through education programs. Each model was implemented in different provinces. Consequently, persons were offered services or supports based on the model that happened to be used in a particular geographic area, rather than selecting one of the three models that met their needs and interests. Each model had gaps. Matching persons with disabilities to companies required motivation from the employers and employees. This model required a series of consultations to make people feel confident to approach companies. Matching persons with disabilities to small-business employers required close follow-up throughout the training to ensure that trainees with disabilities received quality training. It also required support after the training to ensure that the trainees were fully prepared to become self-employed. A major gap for this model was the small number of employers in rural areas, and the number of employers who were interested in participating in the model was even smaller. Training persons with disabilities through education programs required a long-term commitment and financial support for tuition, transportation and educational materials. Follow-up was important to help individuals obtain jobs. When asked what challenges the central government faced to continue providing employment services to persons with disabilities, one senior government official said all government offices and public buildings were difficult for persons with disabilities to access, and some persons with disabilities were not confident enough to come to ESC for services. To overcome these challenges, the ESC staff member believed it was necessary to establish a network of firms and companies to recruit persons with disabilities. When asked what challenges persons with disabilities face in obtaining decent employment, the ESC staff member said companies are still unwilling to hire them because of the costs. For example, many companies are not interested in the policy by which their taxes would be reduced if 30 percent of their employees were persons with disabilities. The reason for the disinterest is that they would need to upgrade their infrastructure, and they believe it would not be cost-effective to make the changes. Evaluation of USAID Vietnam Disabilities Programming 51 “To do case management, you have to put yourself in the person’s situation. You have to consult them [and have them] do for themselves. Case managers make persons with disabilities more active, not dependent.” — Case Manager Hoa Vang District, Danang City Supporting Systems for Service Delivery Case Management VNAH introduced case management in 2009 and continued to implement the concept through 2011 in select provinces. In 2014, the USAID Disability Support Program, through PDSP, piloted case management in 11 districts in Danang, Dong Nai and Binh Dinh provinces. The Bureau of Labor, Invalids and Social Affairs (BOLISA), under the District People’s Committee (DPC), was responsible for the supervision, technical assistance and supervision. The PDSP case management model included three case managers (education, health, and social affairs) at each commune. Each case manager managed about 10 persons with disabilities as their caseloads. They identified persons with disabilities in their communes and the types of services and supports that they and their family members needed. They managed the disabilities cases after their normal working hours. Case managers did receive a small allowance. Several case managers said it was not enough to make phone calls to beneficiaries or to buy gas for their scooters to go to beneficiaries’ homes. Roles and responsibilities. Case management supervisors stated that their role was to supervise case managers at the commune level. Their responsibilities were to oversee activities initially supported by VNAH and later by DAI, provide technical assistance to case managers, generate a list of persons with disabilities who needed support, coordinate with DOLISA and VNAH staff for verification and approval of beneficiaries and generate reports on services provided. The supervisors monitored case managers through telephone calls, client home visits, record reviews in the Disability Information System (DIS) and monthly meetings with the case managers. Case managers said their role was to identify persons with disabilities within their communes and identify their needs. They were responsible for developing plans for support and updating information on the DIS. They conducted household visits and phoned families. Client selection criteria. In the absence of clear selection guidelines, client selection criteria varied from location to location. Thanh Phu Commune in Dong Nai, for example, “randomly selected”28 persons for case management, and thus its caseload included persons with mild and severe disabilities, while most other communes were inclined to include only people with “the potential to rehabilitate.” The DOLISA officer from Danang Province said the criteria were the level of disability, the person’s skill level and ability to learn and the types of services and support the persons needed. Case managers from Hoa Vang Commune People’s Committee said that people were selected for case management if they were able to earn a living and had the potential to develop a livelihood (livelihood program), had the potential to rehabilitate or recover (health/medical services) or had the ability and desire to pursue schooling (inclusive education). Severity was not a key selection criterion for case management. Case managers from Hoa Tho Tay Commune People’s Committee said they targeted three groups to participate in case management: children with disabilities; poor families or families in difficult economic situations; and persons with disabilities who had the potential to rehabilitate. 28 Added by the CPC vice chairperson as an additional comment to case manager interviews. Evaluation of USAID Vietnam Disabilities Programming 52 Case management training. Case management courses were offered to case manager supervisors and case managers. The training was divided into two sections: overview and process of case management. Case management supervisors received training in supervision. The overview section included topics such as concept, objectives, and principles of case management; tasks, roles, and knowledge, background, skills and instructions for case managers; and case management process. The process section included training on the three steps of the case management process: collection of information related to the needs of clients, planning and implementation. They also reviewed tools that case managers would need, such as listening, summarizing, empathizing and mobilizing resources. Training in supervision was provided to 428 case managers and supervisors in social work and assessment skills. Supervisors indicated that they attended five courses: case management; monitoring and supervision in the case management system; disability information system (DIS); social work; and physical rehabilitation. Each course was three to four days long. Courses included theory and roleplaying. Case mangers attended three courses (e.g., case management, social work, referral systems) as well as courses tailored to their area of expertise (e.g., planning and instructional design and newborn screening). Supervisors and case managers were “satisfied” to “very satisfied” with the courses (Hoa Vang, Danang, Bien Hoa, Dong Nai and Trang Dai, Dong Nai). Participants rated the courses as “relevant” and said they learned more about persons with disabilities and programs that can support them. For example, one health case manager said it was her first time learning about social work. After the course, she understood that persons with disabilities need social support, as well as physical therapy or corrective surgery. Additionally, participants — especially those with limited experience with social work — received crucial skills and knowledge to supervise case managers. Case managers from Dong Nai said it was difficult for DOLISA staff to attend because of their schedules. They also said topics repeated (e.g., laws and policies related to persons with disabilities). Supervisors from Dong Nai stated that they do not use what they learned in the training to supervise case managers. For example, one supervisor commented on a lack of time to review the quality of the client profile case-by-case. All participants said that the courses needed to provide participants with more practice, particularly field practice with persons with disabilities. Participants from Dong Nai specifically said they wanted more technical assistance from VNAH staff because they found the staff to be very helpful. A health case management supervisor from Dong Nai expressed a need for additional health staff courses because they do not have social work backgrounds. He also believed that training should provide different courses for different sectors. Moreover, some skills (e.g., rehabilitation) require long￾term courses. He said it was difficult to learn everything in a short course. Case managers and supervisors from Danang said they did not receive any follow-up training or resources after attending the courses. Their counterparts from Dong Nai said they received follow-up support from VNAH technical staff and found it to be very helpful. Participants from both provinces said that PDSP technical staff joined them in the field to conduct a home visit and develop a client profile. Hoa Tho Tay Commune case managers said they received support from the Social Work Center on how to make referrals for psychological counseling support. In addition to the short-term training for case managers and supervisors described above, PDSP supported the development of a social work training program at Dong A University in Danang. It will continue to train social workers and case managers for DOLISA and other departments in the future. Evaluation of USAID Vietnam Disabilities Programming 53 Implementation. The implementation of case management varied across communes. The main challenge was not having the resources to address the needs of persons with disabilities, particularly human resources. Community leaders and key informants from the health, education and labor sectors were satisfied with the idea of case management. But in their view, implementation was hindered by it being fully used to provide services with assistance from the DAI/VNAH project, and under management of the government system, which required multiple approval processes. As a result, beneficiaries faced long delays to receive services and supports because of time-consuming and complicated approval procedures imposed by DAI/VNAH. Moreover, some persons with disabilities in difficult economic situations could not receive services and supports since they were not selected to the case management program under the inclusion criteria. Supervisors and case managers from Dong Nai and Danang provinces stated that many challenges to case management existed. First, it was difficult to build trust with families. Some families needed several visits to build trust. Second, some of the forms that families needed to complete were complicated and the families needed assistance.29 Third, both case managers at the commune level and case management supervisors at the district level worked part-time. Fourth, large caseloads resulted in case managers frequently working after hours.30 Supervisors from both provinces pointed to an insufficient number of staff and said they needed additional case managers. Case managers who were school managers and health staff were doing multiple tasks in their work, and it was difficult to do these tasks within a normal workday. Supervisors were part-time, and it was difficult to visit the clients with disabilities, as well as their other clients. In the case of Trang Dai (Dong Nai), the commune is located over a large area, making it difficult for supervisors and case managers to meet with families on a regular basis. Case managers from Hoa Tho Tay Ward said three managers needed to go to visit families because caregivers were frequently not at home. The health case manager indicated that the referral system did not allow her to refer persons with disabilities directly to health services at the provincial level. Going through the district added time to the process and, as a result, a delay in treatment. Case managers from Danang stated that distributing 30 cases among three case managers was appropriate. However, an increase of persons with disabilities to the caseload would make it difficult for them to perform their duties. The district case management supervisors from Dong Nai and Danang indicated that compensation for case managers and supervisors should be increased. The amount at the time was too small to afford phone calls to beneficiaries. However, MOLISA (SPD) indicated no state budget line existed for case management and, if such a budget existed, government had made no clear indication about who would work as case management staff and what type of salary and benefits they would receive. In his opinion, case management would have to depend solely on local budgets. Improved capacities for case management. The vice chairperson of the DPC (Hoa Vang District) noted that he observed improvement in the capacity and skills of case managers. Although Danang government officials were satisfied with the system, they suggested ways to improve the system. Suggestions included more funding to meet the service and support needs of beneficiaries, more capacity building for case managers and an increase in staff. Several pointed to a large amount of money invested in case management that directly benefited only a few people (e.g., because of long waitlists). 29 When DAI staff were asked about the forms, they stated that they do not have forms for people to complete. 30 Case managers are responsible for a large number of persons without disabilities. Evaluation of USAID Vietnam Disabilities Programming 54 All case management supervisors and case managers interviewed believed that case management was an important tool to assist and support persons with disabilities. Additionally, they believed case management provided awareness to the community that resulted in an increase in acceptance of adults and children with disabilities. To properly conduct case management, however, was challenging. For example, the health case management supervisor from Dong Nai said, “The client profile was difficult to complete for health case managers. It is time-consuming and requires a lot of effort.” Supervisors and case managers made suggestions to improve case management. One case manager from Trang Dai said case managers should be able to close a case on a person who had no need for additional services and supports. Then the manager would be able to open another case. The practice of not closing cases was due, in part, either to a miscommunication or to the fact that the services were tied to a contract from USAID and not from the government. In general, the issue was about being able to open new cases, not close existing ones. According to the PDSP chief of party, PDSP was allotted a certain amount of money dedicated to each type of service or support. The money for each had to be used for that service or support. Therefore, people who were identified as beneficiaries for a particular service or support remained on the list until they received assistance. Consequently, case managers could close cases but could not add new people to their caseload because the project allowed only a certain number of beneficiaries for each service and support. Once a service or support was allocated, no one else could receive that funding, and funding from one service/support could not move to another. A case manager from Trang Dai discussed challenges in providing services and supports to persons with disabilities in general. She asserted that the unmet needs of persons with disabilities who were not part of the project remained large. In particular, she said some persons with disabilities have difficult economic situations but are not eligible for the government social welfare scheme. She proposed that this group should receive the same type of health insurance as persons with severe disabilities or poor families. She proposed that USAID expand the project to help more persons with disabilities. Supervisors and case managers expressed frustratation at how long it took to receive services and supports. One case manager in Thanh Phu Commune (Dong Nai) said only two wheelchairs had been delivered in two years. Others in Danang objected to the many layers of bureaucracy. For example, one person asserted that hiring a special architect to prepare for replacement or repair of a roof was unnecessary; no design is needed for the standard roof that people build. Beneficiary concerns with case management. Adults with disabilities and family members of children with disabilities who did not receive services and supports that they required were very dissatisfied with the case management system. The primary reason for their dissatisfaction was the long waiting period for certain services and support (i.e., corrective surgery, livelihood support, and housing rehabilitation/repair). People waited six to 12 months. One person with a disability said he requested a wheelchair in September 2014 and did not receive it until February 2015. In another case, a woman with a disability had cancer and requested repairs on her home. She lived with her husband and children. Commune officers said the request consisted of simple repairs and would have made the home more accessible. While waiting to receive the materials for the work, she passed away. Case management supervisors and case managers from both provinces also voiced dissatisfaction with how long beneficiaries have to wait for services. One case manager from Trang Dai said, for pending cases whose needs had been assessed and submitted, the project should disburse the loan as soon as Evaluation of USAID Vietnam Disabilities Programming 55 possible for livelihood support. Other supervisors and case managers also said the project should try to shorten the time between receiving authorization for a support and when it is actually delivered. Disability Information System (DIS) The purpose of the DIS is to serve as a key monitoring, reporting and planning tool. DIS is Web-based and designed to manage data on persons with disabilities at the provincial level. It is used at every level within the DOLISA system: commune, district and province, and by social and labor staff. In addition, other service providers (e.g., health workers and hospital professionals) can access the online system. The primary functions of the DIS are: • Data entry and management: Data from a household survey on disability, province-wide, in 2011 was used as baseline. Data on persons with disabilities at the individual level is entered when a new case opens and is updated by DOLISA staff at the commune level. • Data analysis and reporting. • Case management. USAID, via VNAH, provided technical and financial assistance to develop and operate the disability information system in Danang. Technical assistance included coding the software package, revising and upgrading the software package and direct assistance to users. Financial assistance included system setup at DOLISA, maintenance and training. Use of the DIS. Case managers used DIS to update information on their clients (i.e., individual support plans including the client’s needs and services provided to them). Physicians and rehabilitation therapists could also use the system to track clients. However, they noted that their frequency of using it was lower than that of DOLISA staff and case managers. All case management supervisors and case managers stated that they entered and updated information in the system and extracted two types of reports. One report was sent to the labor officer for the district-level focal point, and the other report was sent to the case manager’s relevant line bureau (at the district level) and department (at the provincial level). DOLISA staff highly valued the system, especially in data analysis for program planning and management purposes, according to key informants. In the past, when data was needed, it took several weeks to obtain it from the district. A program officer from SPD said DOLISA was satisfied with assistance from VNAH/DAI. However, it would have been more effective if DOLISA had played a more proactive role in developing and maintaining the system. For example, DOLISA staff had to inform VNAH/DAI if they wanted to correct bugs or errors in the system. VNAH technical staff either fixed any problems or contracted with an IT company. This process takes time, while DOLISA IT staff could have maintained the system in a more efficient and effective manner. Several DOLISA staff said they had some initial difficulties. In 2014, PDSP upgraded the software, trained provincial DOLISAs and provided regular technical assistance, instruction and monitoring and evaluation. DOLISA staff said the DIS was very helpful and easy to use, and provided accurate data for reporting. Before the DIS, DOLISA used a simple management tool. At the time of evaluation, all sectors were using the DIS. One case manager from Bien Hoa said the DIS was not functioning well. One reason is that the commune officers did not have time or equipment (a computer) to enter data. The data they did collect was sent to PDSP for entry into the system. Evaluation of USAID Vietnam Disabilities Programming 56 Annex 9 presents an accounting of performance indicator targets and results culled from various annual reports (Annex 8) of both IVD projects and PDSP. Of the 211 performance indicators listed in the activities’ monitoring and evaluation plans, only 90 were provided targets. Reporting documents show that of the 90, 48 were exceeded by greater than 10 percent, 13 achieved between 90 percent and 110 percent of the target, and 29 fell short, achieving less than 90 percent of the target. Several issues exist with the way performance indicators were developed, tracked and used. Less than 50 percent of listed performance indicators had targets and few had verified baseline numbers; thus, the use of these indicators to track progress is limited. On the whole, for indicators with targets, performance was fair; two-thirds were met or exceeded. However, no narrative exists to explain how the indicators may have been used for performance management. Sustainability of Program Interventions Where and when possible, activities to assist the GVN to better deliver support and services to persons with disabilities should be designed for integration into existing GVN structures and systems to help ensure the sustainability of program interventions. Collaboration should be focused on identifying and addressing viable institutional arrangements to ensure long-term affordable service delivery and on removing barriers to needed institutional change. Key results to date include: • While the improved legal framework secures the rights of persons with disabilities under the law, full implementation of the law will require sustained human and financial resources from the GVN to ensure that persons with disabilities can fully participate in society. • USAID assistance has expanded and strengthened medical and health services for persons with disabilities, but it remains unclear whether the GVN will fully sustain the overall level and quality of the services when the projects end. Specific medical and health care processes and procedures established through USAID assistance have become the standard for providing these services and will continue beyond the life of the program. • Teachers and school management personnel increased their capacity to provide inclusive education. However, teacher turnover is high and it is unclear if the GVN will develop the professional development programs required to train new teachers post-USAID assistance. • The conditions for enforcement and implementation of education policies and laws are limited, especially at the community level. Localities depend on the development projects and programs to lead the way in ensuring adequate implementation of inclusive education policies. • Current DPOs are quasi-governmental rather than civil society organizations. For civil society to be truly established in Vietnam, non-governmental organizations (NGOs) that are for and by persons with disabilities and whose primary purpose is advocating for the rights of persons with disabilities may provide an alternative and more sustainable approach. • DPOs of each province visited have expanded their resource base to include a diverse group of supporters to help ensure that their work will continue beyond the life of the program. Instead of being fully dependent on USAID funding, DPOs are able to mobilize resource from different partners, including government agencies. Key Question 4: Are the programs’ outcomes sustainable? What challenges to sustainability remain? Evaluation of USAID Vietnam Disabilities Programming 57 • A need exists to closely examine vocational training services to identify more effective and efficient strategies to make the services more affordable and thus sustainable. • A number of factors strongly suggest that without more substantial local government support (funding for staff, training, systems development and expansion, etc.), case management as now provided through USAID assistance cannot be sustained. The Legal Framework. As a result of USAID technical assistance and financial support, the GVN made great strides in establishing a strong legal foundation for the rights of persons with disabilities. The passage of CRPD demonstrates the GVN’s effort to align with international standards and best practices. Moreover, the large number of national laws and policies confirms the intention to follow through with securing and improving the rights and opportunities for person with disabilities in Vietnamese society. The sustainability of the legal framework is dependent on the government’s ability to implement, coordinate and monitor the new laws and policies at the local level. Key informants indicated a need to increase the number and capacity of human resources at the central and local levels of government. Additionally, the current gaps in the National Disability Law and the educational law and policies hinders the implementation of the initial intent of the laws and, as a result, impede the ability of persons with disabilities to participate fully in society. Civil Society Support. The USAID program helped establish a network of DPOs that function as service providers. USAID provided business plan and fundraising training, and key DPOs of each province visited have expanded their resource base to include a diverse group of supporters to ensure that their work will continue beyond the life of the program. Instead of being fully dependent on USAID funding, DPOs are able to mobilize resources from different partners, including government agencies. For example, Danang DPO received more than VND 280 million (USD $13,500) from DOLISA in 2014. Membership fees, though minimal, also help maintain DPO operations. However, the DPOs are quasi￾governmental organizations, rather than non-governmental civil society organizations. For civil society to be truly established in Vietnam, it is important to expand support and assistance to non-governmental organizations (NGOs) for and by persons with disabilities whose primary purpose is advocating for the rights of persons with disabilities. Health/Medical Services. USAID played an integral role in strengthening the key rehabilitation and public health (i.e., birth defect surveillance, newborn screening and community-based pre-conception counseling) services training and in operationalizing rehabilitation and public health services in hospitals. Key informants noted that now that processes and procedures have been established through USAID assistance, they have become the standard way of providing the public health services and will continue beyond the life of the program. Although the rehabilitation services will continue, the level of services may be reduced due to funding limitations, and lack of prioritization of rehabilitation services by hospitals. It is highly unlikely that all direct health services will continue. Although these services were provided by government service providers, they were for persons with disabilities who had health insurance only. Health insurance covers 70 percent of the service’s total cost and USAID assistance covers the other 30 percent. As a result, the services will continue only for people with health insurance who have the ability to pay any additional costs of the services. Expansion of health insurance coverage should be considered. One senior official stated that rehabilitation faces many difficulties. Human resources are limited, and some medical universities cannot afford rehabilitation faculty. Some specialties of rehabilitation do not even have teachers, such as speech therapists. Evaluation of USAID Vietnam Disabilities Programming 58 Key informants have noted that diagnostic health services supported by USAID such as birth defects surveillance (BDS) and newborn screening (NBS) have been so effective that they have set a new standard of service. This standard has become an integral and critical part of services provided in the USAID-supported facilities. Evaluation team members were informed that as a consequence, the Danang Hospital for Women and Children has arranged to continue these services after the project ends. Inclusive education. As noted, teachers and school management personnel increased their capacity to provide inclusive education. However, the schools were dependent on the USAID Disability Support Program and other development projects to a certain degree to carry out inclusive education, in terms of sustaining the capacities of teachers and administrators through training, coaching and mentoring; financial support for teacher allowances for those who work with children with disabilities; and counseling and other support for families to get children with disabilities registered and settled into the schools. It was unclear whether training and professional development for inclusive education could be sustained for the long term. Inclusive education was incorporated into the education policy, so the MOET in-service teacher training system would continue training teachers on inclusive education. However, the number of credits for inclusive education could be limited, the training contents rotated year by year (not necessarily including inclusive education every year) and a limited number of teachers from each school could be trained per year. So both the quality and coverage of teacher training were significant issues. This is critical, given the high turnover rate of teachers and school managers who received inclusive education training. The conditions for enforcement and implementation of education policies and laws were limited, especially at the community level. Localities depended on the development projects and programs to lead the way in ensuring adequate implementation of inclusive education policies. While the rights of and the services for inclusive education for persons with disabilities were increasingly understood by local officials and even ingrained into community culture, a concern remained that when USAID assistance ended, progress could be reversed. One small example was that the activity provided a model on accessibility for regular schools. Unfortunately, schools were unable to install ramps without support from CRS because they had not budgeted for the activity. Employment/Livelihood. Commune officers indicated that they planned to continue apprenticeships with funding from the government vocational training program. They said the vocational rehabilitation services would continue beyond the life of the program since it had been institutionalized and became a well-recognized activity. However, other key informants indicated a need to closely examine the services to identify more effective and efficient strategies to make them more affordable and sustainable. The Hanoi ESC planned to expand its model on employment and vocation rehabilitation to employment centers in Hai Duong, Danang, Dong Nai and Can Tho. The bureau had funding and training materials to replicate the model, but not have experts to demonstrate how to consult with persons with disabilities in job fairs. The bureau wanted to have training with an employer in the Hanoi ESC. In terms of private sector partnerships for jobs, BREC was staffed and coordinated by VCCI and VNAH, chaired by the director of the Bureau of Employers’ Activities of VCCI and financed by USAID. Discussions were ongoing about developing and funding a business plan to sustain its services post-IVD. Service Delivery and Management. Several provinces planned to continue to use the DIS. Provincial and district government key informants noted that Danang and Dong Nai DOLISAs had begun to put policies and procedures into place to continue the DIS. For instance, they assigned government staff to be responsible for maintaining system. The reporting procedure using data from PDSP was standardized. One case manager from Bien Hoa said the DIS was not always functioning well at the commune level. One reason was that the commune Evaluation of USAID Vietnam Disabilities Programming 59 “It will be challenging to maintain current work. There will be still [a] case management system, but it will be operated in a little different way. For example, it may require involvement from more sectors, not just only health, education and labor sectors. Currently, [the] district level has not been informed of plan to continue this work after the project ends.” — Case Management Supervisor Cam Le District officers did not have time or equipment (a computer) to enter data. What data did collect was sent to PDSP for system entry. Monitoring and evaluation of service delivery was a critical management function. The head of AMSA stated that the MOH did not have a budget for monitoring and evaluation of the NAP. Thus it was unlikely to continue in its form at the time of evaluation. Although key informants were optimistic about the continuation of the activity’s case management initiative, a number of factors firmly suggest that without stronger local government support (funding for staff, training, systems development and expansion, etc.), case management as provided through USAID assistance could not be sustained. Key informants gave several reasons: • The case management system was working out of the kindness of the case managers’ hearts. Although case managers received a small allowance, it was not enough to cover expenses, and many of them gave their own money to the families of persons with disabilities. • Case management was not a key function or responsibility for health and education case managers. Many were health staff at the commune health center, schoolmasters, teachers or staff at DOLISA. They also had other tasks (teaching students or implementing public health programs) that took them away from case management. Case managers’ job descriptions had no provisions to permit them to continue case management. • The Ministry of Home Affairs must assign a “case manager” or “social worker” for them to exist legitimately on a department’s payroll. New positions were limited and competitive across public service functions. • Case managers would need to be full-time staff at the district and commune levels and these positions were not budgeted. • It would be difficult to replicate the case management model without a national database on persons with disabilities to assist central, provincial and district government officials in their planning. (The DIS is not national.) If a database did exist, an official from MOLISA’s Social Protection Department (SPD) said the government would not know how to meet the needs of persons with disabilities and have a sufficient budget to meet the needs. • Case management was a pilot project. The purpose of any pilot project is to be implemented on a small, controlled scale to allow its full impact, benefits and weaknesses to be evaluated before implementation on a regional or nationwide basis. However, no clear documentation showed that this pilot study was set up in this manner. The evaluation component of this project did not appear to have enough rigor to assess the project accurately. Evaluation of USAID Vietnam Disabilities Programming 60 V. CONCLUSIONS AND RECOMMENDATIONS Conclusions Improving Lives and Opportunities USAID-provided direct support assistance has improved the lives of persons with disabilities. Across all types of USAID assistance, recipients experienced improvements in mental and physical health, the ability to live independently and improved financial status. Quality-of-life improvements varied significantly, related to the level and type of assistance provided. Beneficiaries receiving multiple or more frequent sustained services more often reported a better quality of life. USAID assistance served a substantial number of persons with disabilities: Data from project sources shows that 4,697 persons with disabilities received medical assistance, 6,967 benefited from inclusive education assistance and 5,984 received training or assistance to find jobs through USAID-supported activities.31 Legal/ Policy Sector  Persons with disabilities have benefited from the legal and policy environment developed and strengthened with USAID technical assistance. Members of DPOs are more aware of their rights under the law.  Public awareness activities increased the acceptance of persons with disabilities within the community and improved provision of services to them. However, knowledge of the law among persons with disabilities and government officials should be more extensive. Health/ Medical Sector  Those who received medical assistance were better able to move and travel, acquire new knowledge and care for themselves. On the whole, people were generally satisfied with medical services provided.  Some problems emerged in accessibility to and delivery of the services. The project may not have reached and assisted those with the greatest needs, who are under the poverty line but not classified as poor and do not have health insurance. In addition, beneficiaries noted significant delays of six to 12 months to receive certain services. Education Sector  Recipients of education assistance improved their learning capacities, were more confident and attended school more often. Particularly, USAID assistance for inclusive education resulted in the children developing a more positive attitude and behavior toward study after their educational experiences. Parents of children with disabilities who report improvement in their child’s life skills valued inclusive education assistance.  However, depending on the local resources, the enforcement and implementation of the policies to support children with disabilities varied, and the rate of children accessing services differed. Moreover, findings suggested that conditions contributing to the effective implementation of inclusive education included resources and capacity, multi-sector coordination of services, financial resources and timely planning. 31 Data source: Various reports from IVD-VNAH, IVD-CRS, PDSP-DAI as listed in Annex 9. Evaluation of USAID Vietnam Disabilities Programming 61 Employment/ Livelihood Sector  Employment and livelihood assistance helped persons with disabilities find jobs and perform their work better. Beneficiaries with disabilities had mixed opinions about the different models of training. They enjoyed social benefits that stemmed directly and indirectly from work they found as a result of USAID-supported training and other employment services. Some persons with disabilities successfully sustained employment, while others did not. However, they all had opportunities to improve their employment and their life with USAID assistance.  The school-to-work project enabled graduates to obtain skills they could use to find work and earn a living, making them and their parents more confident about work and securing employment in the future. Development of the Legal and Policy Environment  The GVN established an extensive and comprehensive framework of disability laws and policies to protect the rights of persons with disabilities. The passage of the National Law on People with Disabilities (2010), supported by USAID assistance, marked a significant advance toward improving the legal framework to ensure the rights and expand the opportunities of persons with disabilities.  USAID assistance has helped empower persons with disabilities to participate in and advocate for a legal system that better protects their rights and enables the delivery of services they need.  Vestiges of stigmatization and discrimination remain within the legal and policy framework and service delivery system that is implemented under the law. Gaps in the current legal framework are due, in part, to inconsistency between national disability laws and the CRPD.  Members of DPOs have a broader voice to advocate on their own behalf as a result of USAID assistance.  The national databases on adults and children with disabilities are inadequate to improve the government’s ability to formulate, implement and coordinate laws and policies.  Full implementation of the disability laws and policies would depend on allocation of sufficient resources by GVN. Although the nation has made great strides in supporting equal protection for persons with disabilities, the lack of sufficient funding and human resource hinders full implementation of the law. Moreover, gaps in the Law on Persons with Disabilities contradict basic principles established in CRPD. These contradictions impede the implementation of the law and, as a result, impede the ability of persons with disabilities to participate fully in society.  NCCD, which is responsible for coordinating implementation of laws and policies, has neither sufficient authority nor capacity to do so. So far, the monitoring and evaluation system for laws and policy implementation has not been established. Strengthening Critical Institutional Capacities Legal/ Policy Sector  The technical assistance provided by USAID strengthened critical legal frameworks by building policymaking capacities. USAID laid the foundation for the key ministries — including the MOH, the MOET and the MOLISA — to strengthen their capacities to serve and support persons with disabilities. Health/Medical Sector  Through technician training and equipment provision, the USAID programs facilitated improvement of the capacity to delivery physical rehabilitation services, treatment of children Evaluation of USAID Vietnam Disabilities Programming 62 with autism, district hospitals and public health services in birth defects surveillance and newborn screening, especially in Da Nang.  In addition, support for long term HR development in vocational rehabilitation was provided through curriculum development and testing at four medical universities in the country.  The capacity to deliver services for persons with disabilities remains limited due to challenges in investment in facilities/ equipment and limited local resources. Inclusive Education  USAID assistance has provided multiple services focused on inclusive education to expand access for children with disabilities. Capacity building for teachers and administrators has been highly valued and effective, but HR resources are too limited to fully implement inclusive education commitments.  The government accepted the principle of inclusive education and passed decisions to adopt inclusive education into schools.  However, the educational system continues to foster the two-tiered structure, in which inclusive schools and special or segregated schools will be maintained. Employment/Livelihood Sector  A variety of models were developed and implemented with USAID assistance to provide vocational and IT training and employment/ livelihood-supporting services through training institutions, companies and DPOs. The capacities of the training institutions and DPOs were improved mainly through learning by doing.  USAID assistance has helped strengthen capacities to deliver employment services to persons with disabilities through material development and training delivery to BOE staff in vocational rehabilitation. It also supported the vocational rehabilitation model in two ESCs, which then were adopted as the standard model to replicate within the public employment service system.  The capacities to provide employment/ livelihood-support services to persons with disabilities are limited by the lack of resources for making training facilities accessible and limited networking with potential employers. Civil Society Sector  Through technical assistance, training and learning by doing in vocational/livelihood programs, USAID expanded and strengthened the capacity of DPOs as service providers, as well as law and policy advocates and disseminators. Through these activities, persons with disabilities established their presence within the government and are now invited to provide public input into deliberations on policy issues. They receive modest levels of medical, education and employment support from community-based organizations they establish and manage. Service Delivery and Management  The case management project was a pilot project to examine the feasibility of implementing such a system in Vietnam. If successful, case management services could be implemented at a larger scale (nationally). Consequently, case management was implemented in specific districts within a small number of provinces. (Da Nang was an exception; the program was implemented in all districts and communes there.) USAID supported the case management program through both technical and financial assistance for direct service delivery. However, given limited resources and time constraints, the program could not reach all persons with disabilities in the areas it served. As a result, a large number of persons with disabilities, including those with the greatest Evaluation of USAID Vietnam Disabilities Programming 63 needs, might not have been included in the USAID programs and could not receive assistance services. Therefore, USAID did not improve the service delivery system per se.  However, the recipients and case managers at the district level reported numerous positive outcomes from the case management program. Sustainability of Program Interventions Investment in capacity building for human resources is likely to be sustained in most areas, except for the case management program. In the long term, the availability of adequate human and financial resources from GVN for implementation is the key issue for sustainability across all sectors. Specific issues in each sector are the following: Legal/Policy Sector  The GVN’s ability to implement, coordinate and monitor the new laws and policies at the local is insufficient. Health/Medical Sector  The process and procedures to operate the rehabilitation and public health services in hospitals have become standards and will continue after the project, but perhaps at a reduced level of services due to limited funding and prioritization by hospitals. Public health services such as birth defects surveillance (BDS) and newborn screening (NBS) have become a new standard and will continue in Da Nang.  Although many direct health and medical services that improved with USAID assistance are likely to continue, it is unclear whether provincial DOHs will maintain the mode of service delivery to persons with disabilities when the project ends, due to limited funding. Education Sector  Schools still depend on USAID and development projects to carry out inclusive education and support families to get children with disabilities acclimated in schools.  MOET can continue in-service training of teachers, but quality and coverage is a big issue.  The USAID-supported model for accessibility for regular schools is unlikely to be replicated due to a lack of budget. Employment/Livelihood Sector  Vocational rehabilitation services will continue beyond the life of the program, since it has been institutionalized and is a routine activity in the supported ESCs. Its effectiveness in helping persons with disabilities secure long-term employment is unclear.  Livelihood opportunities will continue due to local government commitment. Civil Society Development Sector  Many DPOs established through PDSP were established as service providers and most are successful and likely to sustain the services.  Key DPOs of each province have expanded their funding base and will continue to function after the funding from USAID stops.  DPOs are quasi-governmental, not civil society organizations. For civil society to be truly established in Vietnam, non-governmental organizations (NGOs) that are for and by persons Evaluation of USAID Vietnam Disabilities Programming 64 with disabilities and whose primary purpose is advocating for the rights of persons with disabilities may provide an alternative and more sustainable approach. Service Delivery and Management  Several provinces plan to continue to use the DIS, including Danang and Dong Nai.  A number of factors firmly suggest that without stronger local government support (funding for staff, training, systems development and expansion, etc.), case management as provided through USAID assistance cannot be sustained. Recommendations Legal Framework Sector 1. Support and assist policymakers with reviewing and revising current disability laws and policies to ensure that (a) the current process to identify persons with disabilities, as well as the type and severity of disabilities, is valid and accurate; (b) the commune officials who make disability determinations have the expertise, knowledge and tools to carry out their responsibilities; (c) persons with mental disabilities are adequately included in all disability laws and policies; (d) all persons with disabilities, regardless of type and severity of disability, have equal access to services and supports; and (e) Vietnam’s laws and policies align with the CRPD, as appropriate. 2. Continue to support capacity development on evidence-based laws and policy formation. 3. Expand activities to enhance awareness among officials, service providers, persons with disabilities and the general public regarding the rights of persons with disabilities. 4. Work with the GVN to identify and better enforce critical provisions of the laws and policies that support persons with disabilities. 5. Support GVN in implementing the CRPD by supporting and assisting the establishment of a committee that will review and monitor implementation. 6. Assist GVN in developing a comprehensive and reliable national database to collect, synthesize and disseminate information on adults and children with disabilities to improve the government’s ability to formulate, implement and coordinate laws and policies. 7. Encourage those in charge to ensure that an effective mechanism exists to coordinate action plans, facilitate equality for persons with disabilities in society and supervise the effective and comprehensive implementation of provisions in the laws for persons with disabilities. Health/Medical Sector 1. Work with the MOH to strengthen and implement the NAP on Rehabilitation by engaging all departments and agencies. 2. If direct service provision continues, incorporate the services into the government’s existing service system and include people with and without medical insurance. 3. Assess the feasibility and options of providing capacity-building activities and equipment to potential providers who are neither licensed nor eligible for health insurance reimbursement and may not continue services after the project ends, as the services would not generate revenue. 4. Strengthen rehabilitation capacities by expanding training courses to include longer-term certification programs, incorporating a monitoring system for quality and effectiveness. Evaluation of USAID Vietnam Disabilities Programming 65 5. Work with the GVN to improve accessibility of persons with disabilities to the existing government health services. Education Sector 1. Support GVN and other stakeholders to promote for establishment of inclusive education for all children with disabilities, as provided for in Article 24 of the CRPD, which states that governments “shall ensure an inclusive educational system at all levels.” 2. For the long-term, support GVN to develop a transitional plan to transform special or segregated schools into inclusive educational programs to ensure that all children with disabilities receive education within the mainstream general educational system. A comprehensive assessment on the nature of Vietnamese special or segregated schools in providing education services for children with disabilities will yield better understanding about the need of those in the Vietnam context. 3. Work with GVN to dedicate more resources to providing community-based services and support to children with disabilities. 4. Work with GVN to implement standards for making schools physically accessible and modified for students with different disabilities. 5. Make new investments into current teacher training to provide appropriate curricula and support for teaching children with intellectual and other disabilities. 6. Work with the GVN to improve accessibility of children with disabilities to inclusive schools. Employment/Livelihood Sector 1. Work with stakeholders to improve collaboration among MOLISA, provincial and private sector programs on vocational training for persons with disabilities. 2. Continue to work with employers and MOLISA to access the labor market demand and identify and target additional vocational training that better matches local labor market opportunities. If support for IT training continues, it should be strengthened by providing more practical experience matched to the needs of potential employers. Proper training needs analysis and counseling with both persons with disabilities and employers before and after training would also be critical for the success of IT training and inclusion of persons with disabilities. 3. Build career/employment counseling capacities at schools to facilitate a smooth transition to work for persons with disabilities. 4. Ensure that livelihood programs provide not only basic training and a minimum start-up seed grant, but also training in how to sustain the business, including accessing capital for the long￾term. 5. Work with the MOLISA, ESCs and DPOs to develop and implement additional job placement services for persons with disabilities, including a job board website, individual job referral support and job fairs better targeted to persons with disabilities. 6. Work with the MOLISA General Directorate of Vocational Training to strengthen and implement the NAP on Vocational Training for Persons with Disabilities. Evaluation of USAID Vietnam Disabilities Programming 66 Civil Society Sector 1. Build capacity for DPOs in disability advocacy. This type of capacity building refers to teaching strategies, shaping leadership and advocacy skills and providing the tools to influence decision￾making processes in public and social arenas to eliminate inequalities between persons with and without disabilities. The purpose of this type of capacity building is to achieve respect for diversity and difference and to contribute to cultural, social and political change for individuals’ full and active citizenship. 2. Support the expansion of disability civil society organizations. Identify organizations that are forming as an association and build members’ capacity around disability advocacy. 3. Ensure that the contractor that provides leadership and advocacy training is a civil society organization operated by and is for persons with disabilities, preferably a non-governmental organization (NGO) or international NGO (INGO) with a background in independent living or self-advocacy. Service Delivery and Management 1. Work with the MOLISA, in coordination with the DOLISA and Provincial People’s Committees (PPCs), to strengthen, better disseminate and — where needed — train local officials on the Circular on Case Management. 2. Work with the GVN to modify case management practices to make it easier to incorporate case management into the government’s service delivery system. 3. Promote international best practices in case management to better advocate for persons with disabilities and to better coordinate efforts across the full range of needed services. 4. Employ a social worker with experience in establishing and implementing case management within a national service delivery system. This individual can review the case management curriculum to ensure that it can be part of implementing the Circular on Case Management. 5. Continue to provide direct services, but include more planning prior to implementation. Work closely with USAID to apply procedures and policies that are more consistent with the local situation. Sustainability of Program Interventions 1. Support activities with practices that are inclusive and consistent with the CRPD to provide continuing guidance to the GVN on international best practices. 2. If pilot projects continue, strengthen implementing partners’ ability to monitor and evaluate progress and incorporate results into their work to ensure that the project moves forward. 3. Future disabilities programming should have clear, realistic, reportable performance indicators, targets and results. 4. For future pilot projects, USAID should develop a strategy to use the monitoring and evaluation data to engage the central government in discussions on how USAID could play a role in supporting the government. 5. Programs should better document lessons learned from the implementation of the activities to guide future implementation. Evaluation of USAID Vietnam Disabilities Programming 67 ANNEX 1: REFERENCES Catholic Relief Services (2006). Inclusion of Vietnamese with Disabilities (IVD) Annual Report # 1 (October 2005 – September 2006). USAID: Hanoi, Vietnam. Catholic Relief Services (2009). Inclusion of Vietnamese with Disabilities Progress Report # 10, Submitted July 31, 2009. USAID: Hanoi, Vietnam. Catholic Relief Services (2009). Inclusion of Vietnamese with Disabilities Progress Report # 11, Submitted October 27, 2009. USAID: Hanoi, Vietnam. Catholic Relief Services (2010). Inclusion of Vietnamese with Disabilities Progress Report # 12, Submitted January 29, 2010. USAID: Hanoi, Vietnam. Catholic Relief Services (2010). Inclusion of Vietnamese with Disabilities Progress Report # 13, Submitted April 30, 2010. USAID: Hanoi, Vietnam. Catholic Relief Services (2010). Inclusion of Vietnamese with Disabilities Progress Report # 14, Submitted July 30, 2010. USAID: Hanoi, Vietnam. Catholic Relief Services (2010). Inclusion of Vietnamese with Disabilities Progress Report # 15, Submitted October 21, 2010. USAID: Hanoi, Vietnam. Catholic Relief Services (2011). Annual Program Performance Report for USAID Vietnam, 2011. USAID: Hanoi, Vietnam. Catholic Relief Services (2012). Annual Program Performance Report for USAID Vietnam, 2012. USAID: Hanoi, Vietnam. Catholic Relief Services (2013). Inclusion of Vietnamese with Disabilities, FY 2013 Annual Performance Report. USAID: Hanoi, Vietnam. Catholic Relief Services (2014). Inclusion of Vietnamese with Disabilities Annual Report, 2014. USAID: Hanoi, Vietnam. DAI (2013). Disability Support Program, FY 2013 Q4 (July 1 – September 30, 2013) and Annual Progress Report. USAID: Hanoi, Vietnam. DAI (2014). Persons with Disabilities Support Program, FY 2014 Q4 (July 1 – September 30, 2014) and Annual Progress Report. USAID: Hanoi, Vietnam. DAI (2014). Disability Support Program, FY 2014 Q1 Progress Report. USAID: Hanoi, Vietnam. DAI (2014). Disability Support Program, FY 2014 Q2 Progress Report. USAID: Hanoi, Vietnam. International Labor Organization (2013). Inclusion of Persons with disabilities in Vietnam. Irish Aid: Hanoi, Vietnam. National Association of Social Workers (NASW) (2013). NASW Standards for Social Work Case Management. Author: Washington. Nguyen Thi Hoang Yen & Nguyen Thi Thu Huong (2013). The Policy on Education of the Disabled in Vietnam: The Gaps and Its Impact on the Persons with Disabilities. VNU Journal of Education Research, 29(2), 24-33. Evaluation of USAID Vietnam Disabilities Programming 68 Pawar, M. (2014). Social Work Practice with Local Communities in Developing Countries: Imperatives for Political Engagement. Sage Open, Vol. 4 (1), 1-11. Rosenthal E. (2009). The Rights of Children with Disabilities in Vietnam: Bringing Vietnam’s Laws into Compliance with the UN Convention on the Rights of Persons with Disabilities. Washington, DC: Disabilities Rights International. USAID (2014). Case Management Toolkit: A User’s Guide for Strengthening Case Management Services in Child Welfare. Author: Washington DC USAID (2013). Disability Projects Review Assessment and Analysis Report. Author: Hanoi Vietnam. USAID (2005). Vietnam Disability Situation Assessment and Program Review. Management Systems International: Washington, D.C. UNFPA (2010). Fact Sheet: Persons with disabilities in Vietnam: Evidence from the 2009 Census. Author: Hanoi Vietnam Vietnam Assistance for the Handicapped (2006). Quarterly Progress Report for USAID Vietnam, Reporting Period: 1 Jan – 31 Mar 2006. USAID: Hanoi, Vietnam. Vietnam Assistance for the Handicapped (2006). Quarterly Progress Report for USAID Vietnam, Reporting Period: 1 Apr – 30 Jun 2006. USAID: Hanoi, Vietnam. Vietnam Assistance for the Handicapped (2006). Annual Report, Reporting Period: Oct 01- September 30, 2006. USAID: Hanoi, Vietnam. Vietnam Assistance for the Handicapped (2007). Semi-Annual Performance Report to U.S. Agency for International Development (USAID), Reporting Period: 1 October 2006 – 30 March 2007. USAID: Hanoi, Vietnam. Vietnam Assistance for the Handicapped (2007). Semi-Annual Performance Report to U.S. Agency for International Development (USAID), Reporting Period: 1 April 2007 to 30 September 2007. USAID: Hanoi, Vietnam. Vietnam Assistance for the Handicapped (2008). Semi-Annual Progress Report Submitted to USAID Vietnam, Reporting Period: 1 October 2007 to 31 March 2008. USAID: Hanoi, Vietnam. Vietnam Assistance for the Handicapped (2008). Annual Report for USAID Vietnam, Reporting Period: April 1, 2008 to September 30, 2008. USAID: Hanoi, Vietnam. Vietnam Assistance for the Handicapped (2009). Performance Report, Reporting Period: January 01, 2009- March 31, 2009. USAID: Hanoi, Vietnam. Vietnam Assistance for the Handicapped (2009). Performance Report, Reporting Period: April 1 – June 30, 2009. USAID: Hanoi, Vietnam. Vietnam Assistance for the Handicapped (2009). Performance Report, Reporting Period: Oct 1, 2008- Sep 30, 2009. USAID: Hanoi, Vietnam. Vietnam Assistance for the Handicapped (2010). Performance Report, Reporting Period: October 1, 2009 – September 30, 2010. USAID: Hanoi, Vietnam. Evaluation of USAID Vietnam Disabilities Programming 69 Vietnam Assistance for the Handicapped (2011). Annual Program Performance Report for USAID Vietnam, 2011. USAID: Hanoi, Vietnam Vietnam Assistance for the Handicapped (2012). Annual Program Performance Report for USAID Vietnam, 2012. USAID: Hanoi, Vietnam. Vietnam Assistance for the Handicapped (2013). Inclusion of the Vietnamese with Disabilities – IVD, FY 2013 Progress Report. USAID: Hanoi, Vietnam. Vietnam Assistance for the Handicapped (2014). Inclusion of the Vietnamese with Disabilities – IVD, FY 2014 Progress Report. USAID: Hanoi, Vietnam. Evaluation of USAID Vietnam Disabilities Programming 70 ANNEX 2: EVALUATION SCOPE OF WORK I. Background Since 1989, THE U.S. Government has been improving the quality of life and supporting the inclusion of Vietnamese with disabilities into all aspects of society. USAID began implementing assistance in the area of disabilities in 2005 with the ongoing Inclusion of Vietnamese with Disabilities (IVD) program implemented by Catholic Relief Services (CRS) and Vietnam Assistance for the Handicapped (VNAH). Building upon this experience, USAID/Vietnam developed the Persons with Disabilities Support Program (PDSP) in 2012. PDSP is implemented by DAI and VHAH. The USAID disability portfolio consists of a wide range of activities, covering medical (prosthetic and orthopedic, physical rehabilitation), education (inclusive environment and special education), and social (support employment, vocational rehabilitation and the organizations of persons with disabilities) sectors. The portfolio is also engaged in the development and support of disabilities policies in Vietnam. USAID/Vietnam programs assist persons with disabilities, regardless of the cause of their disability. USAID’s disability programs address Development Objective 2 in the Country Development Cooperation Strategy, “Capacity Strengthened to Protect and Improve Health and Well-being”, and further, Immediate Result 2.3 “Expanded Opportunities for Vulnerable Populations”. Taken together, the programs have the primary objectives of: • Developing a stronger legal and policy framework on disability; • Delivering direct assistance to persons with disabilities; • Building institutional capacities in the health and education sectors to improve disabilities services delivery, • Ensuring a training model that improves the quality of life and equal opportunity of employment for persons with disabilities. II. Concept of the Disabilities Programming Performance Evaluation 2. Evaluation Purpose Given USAID already has an initial concept for future programming, the primary purpose would be to document the general “impact” and key contributions of the programs in building awareness, political will, the enabling environment, institutional capacity and operational. This will include identifying current opportunities and gaps in disability programming that can serve as a base of understanding for the ongoing work of GVN, program implementers, Disabled People’s Organization and other stakeholders. 3. Objectives a. Relevance – Understand the extent to which the interventions have been consistent with beneficiaries’ needs, GVN expectations and interests, and USAID policies. b. Effectiveness – Identify/explain the causality of interventions and the impacts they have had on improving the policy environment and on the delivery of disabilities services. c. Impact – Assess the benefits on the ground in the daily lives of persons with disabilities. Evaluation of USAID Vietnam Disabilities Programming 71 d. Sustainability – Evaluate progress in sustaining positive impacts of the interventions and determine whether there is opportunity to further enhance the quality and quantity of the benefits of the programming. 4. Key Evaluation Questions a. To what extent and in what ways have the programs improved the lives and opportunities for disabled persons in Vietnam? Have the planned benefits been delivered and received as perceived by beneficiaries? b. To what extent has the programming assistance helped to develop the GVN legal and policy environment, and public awareness and support, for disability services? Is the current level sufficiently robust to sustain the services or are major changes/improvements needed? c. In terms of improving service delivery, has disabilities programming across projects provided both an enabling environment and a supporting holistic systems approach in technical assistance to strengthen critical institutional capacities? d. Are the programs’ outcomes sustainable? What challenges in terms of sustainability remain? 5. Scope a. The evaluation should be based on key technical areas that cut across all programs, thus the scope should be sector-wide and include all three disabilities programs/IPs. 6. Methodology a. Literature review/Desk Phase Existing quantitative and qualitative data will be collected and analyzed in collaboration with USAID and the IPS. Programming documents as well as other documents shaping the wider strategy/policy framework should also be reviewed: i. Project documents (project proposal, log- frame, work-plan, implementation plan, financial plan) ii. Project quarterly and annual reports and other technical reports and plans iii. Quarterly reports on project implementation progress and achievements iv. Relevant field reports and documents v. M & E plan (includes M&E plan/documents and software(for quantitative data) vi. Other general and relevant materials as identified in the inception stage b. Field Visits Field work will include individual and focus group meetings with, and/or surveys of, relevant stakeholders, including beneficiaries, GVN partners, other NGOs working in the field, and IPs. 7. Deliverables a. The full performance evaluation report b. A summary on best practices and lessons learned (5-8 pages for more general consumption) c. Briefing PowerPoint to present to stakeholders Evaluation of USAID Vietnam Disabilities Programming 72 d. Workshop (or Conference) with GVN on “Successes and Progress on the Road to a Fully Realized Program of Disabilities Services in Vietnam”. III. Proposed Level of Effort and Schedule The evaluation will be conducted by a dedicated four-person core team. The core team will be supplemented with assistance and support provided from VEMSS full-time staff and by a local researcher. An interpreter will be added to the team as required (based on the needs of the team). The evaluation team’s core members will include: · Team Leader/International Expert · Technical Expert - Disabilities Programming · Two Evaluation/Technical Specialists In selecting evaluation team members we will ensure the team covers a broad range of experience as the evaluation is complex and multi-faceted. We will select at least two evaluation team members who have a very strong background in evaluation and analysis, as well as individuals with specific experience in disabilities programming including policy development, institutional capacity building and training. The range of additional experience required includes program management, gender analysis, and financial sustainability/economic analysis. Research support: We will hire one local researcher to collect and organize project data sets and conduct a desk study of project performance. This work will be done in February and will be available to the team when field work is begun in March. Logistics coordination and interpretation: Basic scheduling and coordination will be managed by VEMSS staff. Technical support and direction will be provided by the COP/DCOP and scheduling and logistics will be supported by the project’s Administrative Assistant. If required (depending on the team’s staffing mix) an interpreter may be added to travel with the team. The Evaluation Team may include/be accompanied by staff members from USAID/Vietnam as appropriate to observe interviews and field visits. A list of interviewees and key stakeholders will be provided by USAID prior to the assignment’s inception. Position Descriptions Team Leader. This position will be filled by an international evaluation specialist and serve as the primary manager of the evaluation team. This position is responsible for the design of the evaluation’s methodology, team management, and for producing draft and final evaluation reports. The Team Leader shall take specific responsibility for the analysis of the program’s progress towards quantitative and qualitative targets, factors for such performance, and the program’s benefits and impacts. S/he shall also lead the preparation and presentation of the key evaluation findings, conclusions and recommendations to the USAID/Vietnam team and key partners. The Team Leader must be a senior consultant with experience in leading and conducting USAID program evaluations and be comfortable with a range of qualitative and quantitative data collection and analysis techniques, and must have led at least three independent performance evaluations. S/he should have an advanced degree in Public Health, Social Work or another field related to disabilities programming Evaluation of USAID Vietnam Disabilities Programming 73 and should have at least 10 years senior level experience working in a developing country. The Team Leader should also have a good understanding of USAID funded project implementation, administration, financing, and management procedures. Excellent oral and written skills are required. Technical Expert - Disabilities Programming. This position will serve as a full team member and provide expert analysis of the program’s operations and performance. The Technical Expert should have a degree Public Health, Social Work or other field related to disabilities programming and at least 5-10 years of experience in disabilities programming. Knowledge of USAID programs and context is highly desirable. He/she should have a strong understanding of the disability movement and challenges facing programs that support persons with disabilities in Vietnam or Southeast Asia. He/she must have experience in program evaluation and knowledge in conducting surveys, key informant interviews and focus groups. Strong English language and writing skills are required. Experience and understanding either of policy development, capacity building and/or training initiatives, and approaches to strengthen government ownership will be an advantage. Two National Evaluation/Technical Specialists. The Evaluation/Technical Specialists will participate fully in all aspects of the evaluation. These positions require evaluation experience, knowledge of disabilities programs and some combination of specialized skills pertinent to the evaluation, including in the areas of policy development, gender analysis, institutional development, training, and financial sustainability/economic analysis. Each position requires a minimum of five years of relevant experience. II. Proposed Level of Effort and Schedule Level of Effort The following is a suggested level of effort for the evaluation, by position and task. Position Management/ Planning Doc Review Int’l Travel Initial Briefing - Meetings Methodology Development Data Collection Analysis Drafting Final Report Total Days VEMSS COP/ DCOP 5 6 2 2 8 3 3 2 31 Team Leader 1 3 4 2 2 20 4 6 4 46 Tech Expert 7 2 2 20 4 5 2 42 Local Spec. 1 3 2 2 20 2 3 32 Local Spec. 2 3 2 2 20 2 3 32 Researcher 5 5 5 15 Evaluation of USAID Vietnam Disabilities Programming 74 Tentative Schedule The proposed schedule is as follows: Date Activity December 15 Finalize Scope of Work, begin recruitment. February 15 Team recruited and employment agreements finalized. February 15- March 6 Basic documents collected and distributed; document review begins; VEMSS office conducts desk-study review and collects and organizes existing data; key stakeholders identified; meetings scheduled for January. March : 9-13 Team arrives/convenes in Hanoi and begins work: 1) team planning meeting is held to clarify task and responsibilities; 2) briefings held with USAID, IPs and GVN; 3) methodology finalized and data collection instruments developed. March: 16-20 Initial data collection begins in Hanoi and instruments reviewed and adjusted. The team will visit several key stakeholders in Hanoi for data collection, and possibly hold a focus group meeting. This will ensure the team has a chance to revise the methodology and data collection instruments, and will be operating from the same frame of reference, which will be important once the team splits. March: 23-April 3 Team splits to conduct field work (two teams). At a minimum field work will be conducted in four provinces. This will include visits to IP field offices and to GVN provincial, district and management centers and offices. April 6-10 Team reconvenes in Hanoi and begins to analyze data to formulate findings and conclusions. April: 13-17 Team drafts reports. A presentation is held with USAID and key stakeholders to review findings, conclusions and recommendations. Report is drafted after initial validation workshop. April 24 Final draft detailed report submitted to USAID. May 15 Report finalized and submitted following receipt of USAID comments. VEMSS team prepares summary presentation and schedules learning event with stakeholders. Evaluation of USAID Vietnam Disabilities Programming 75 ANNEX 3: DIFFERENCES BETWEEN QUALITATIVE AND QUANTITATIVE RESEARCH Qualitative Research Quantitative Research Objective / Purpose • To gain an understanding of underlying reasons and motivation. • To provide insights into the setting of a problem, generating ideas and/or hypotheses for later quantitative research. • To uncover prevalent trends in thought and opinion • To quantify data and generalize results from a sample to the population of interest • To measure the incidence of various views and opinions in a chosen sample • Sometimes followed by qualitative research which is used to explore some findings further Sample Usually a small number of non-representative cases. Respondents selected to fulfill a given quota. Usually a large number of cases representing the population of interest. Randomly selected respondents. Data & Data collection Data is a categorical measurement expressed not in terms of numbers, but by means of s natural language description. Data collection includes unstructured or semi-structured techniques (e.g. individual in-depth interviews or group discussions). Quantitative data is a numerical measurement expressed not by means of a natural language description, but rather in terms of numbers. Data collection includes structured techniques such as online questionnaires, or telephone interviews. Data analysis No statistical data analysis Quantitative statistical data analysis. Outcome Exploratory and/or investigative. Findings are not conclusive and cannot be used to make generalizations about the population of interest. Develop an initial understanding and sound base for further decision making Used to recommend a final course of action. Evaluation of USAID Vietnam Disabilities Programming 76 ANNEX 4: SUMMARY OF USAID SUPPORTED SITES Provinces Poverty Rate Disability Prevalence Key components/activities Implementing Partners Sub￾grantees Other Partners Stakeholders Adults (16- 59) Children (5- 15) Centre level NA NA Promoting participatory and coordinated in formulating disability policy and regulation affecting inclusive education/vocational training CRS Partners: MOLISA, MOH, MOET, ULSA Developing a National Strategy for Integrated and Comprehensive Disability Services to persons with disabilities VNAH MOLISA, MOH, NCCD, VDF, BREC, VCCI, University of Social Science and Humanity of Hanoi (USSH), social work faculties, Committee of Social Affairs (CSA) and Committee of Foreign Affairs at the NA (ratification of CRPD) Stakeholders: UNICEF; Iris Aid; Spanish Red Cross; Handicap Evaluation of USAID Vietnam Disabilities Programming 77 Provinces Poverty Rate Disability Prevalence Key components/activities Implementing Partners Sub￾grantees Other Partners Stakeholders Adults (16- 59) Children (5- 15) international; Prosthetic Outreach Foundation; Vietnam Blind Association; Hanoi*** 6.44 3.23 1.18 Establishment of DPO Capacity building VNAH Hanoi DPO Hoa Binh** 47.31 5.41 1.47 Providing education support for children with disabilities through improved inclusive education CRS DPOs DOLISA District PC Thai Binh* 18.95 6.48 1.48 Improving persons with disabilities’ opportunity to access meaningful employment and/or self￾employment CRS WCDO Increasing Inclusive Employment for persons with disabilities through inclusive business development VNAH Danang* 2.39 6.00 2.39 Rolling-out and providing support to social work case management DAI DOLISA DOET Improving quality and accessibility of persons with disabilities to DAI Nguyen Dinh Chieu Evaluation of USAID Vietnam Disabilities Programming 78 Provinces Poverty Rate Disability Prevalence Key components/activities Implementing Partners Sub￾grantees Other Partners Stakeholders Adults (16- 59) Children (5- 15) standardized care and Tuong Lai Special school; DPOs; Danang UMTP Strengthening public health system DAI DOH DHWC Hai Chau and Cam Le hospitals Increasing sustainability of disability programs DAI DOLISA Improving persons with disabilities’ opportunity to access meaningful employment and/or self￾employment CRS WCDO Hue** Increasing Inclusive Employment for persons with disabilities through inclusive business development VNAH Evaluation of USAID Vietnam Disabilities Programming 79 Binh Dinh* 16.68 5.36 1.47 Rolling-out and providing support to social work case management DAI DOLISA District PC Improving quality and accessibility of persons with disabilities to standardized care DAI Quy Nhon Orthopedic and rehab hospital; 5 rehab units DOLISA District PC Vocational training and livelihood support DAI Suc Song DPO; Ass for the disabled and orphans; Strengthening health system DAI Increasing Inclusive Employment for persons with disabilities through inclusive business development VNAH Quang Nam* 23.47 5.53 1.97 DIS and case management training DAI DOLISA District PC Increasing Inclusive Employment for persons with disabilities through inclusive business development VNAH Quang Ngai* 23.65 5.42 1.62 Improving persons with disabilities’ opportunity to access meaningful CRS WCPO Evaluation of USAID Vietnam Disabilities Programming 80 employment and/or self￾employment Binh Thuan** 21.44 5.9 1.65 Providing education support for children with disabilities through improved inclusive education CRS DOET Binh Phuoc*** 17.2 4.37 0.99 Increasing Inclusive Employment for persons with disabilities through inclusive business development VNAH Dong Nai** 11.73 5.86 1.50 Rolling-out and providing support to social work case management DAI DOLISA District PC Strengthening public health system DAI Rehabilitati on unit/Vinh Cuu Districts; Center for disabled children and orphans DOH Vocational training and livelihood support DAI Song Pho Center Increasing Inclusive Employment for persons with disabilities through inclusive business development VNAH Tay Ninh*** 11.78 3.7 0.77 Increasing Inclusive Employment VNAH DOLISA Evaluation of USAID Vietnam Disabilities Programming 81 for persons with disabilities through inclusive business development HCMC*** 2.94 2.89 1.65 Providing education support for children with disabilities through improved inclusive education CRS DOET * High Disability Prevalence; ** Medium Disability Prevalence; *** Low Disability Prevalence Evaluation of USAID Vietnam Disabilities Programming 82 ANNEX 5: KEY INFORMANTS USAID  Randolph Flay, Program Development Office Director, United States Agency for International Development in Vietnam (USAID)  Nguyen Thi Ha, M&E Specialist/Gender Advisor, United States Agency for International Development in Vietnam (USAID)  Le Ha Van, Contracting Officer's Representative, United States Agency for International Development in Vietnam (USAID)  Ezra Simon, Education Officer of Environmental and Social Development Office, United States Agency for International Development in Vietnam (USAID)  Mei Mei Peng, Education Officer of Environmental and Social Development Office, United States Agency for International Development in Vietnam (USAID)  Kyung Choe, Office of General Development Director, United States Agency for International Development in Vietnam (USAID) IMPLEMENTING PARTNERS Vietnam Assistance for the Handicapped  Tran Van Ca, President  Bui Van Toan, Country Director  Nguyen Thi Thu Huong, Chief of Party  Robert Horvath, Deputy Chief of Party  Ong Kek Hin (Michael), Social Work Specialist Catholic Relief Services  Snigdha Chakraoborty, Country Director  Dinh Thi Nguyet, Chief Of Party  Sanda Rihtman, Asia Sub-regional Representative  Nguyen Thi Thuy, Technical Advisor  Luu Thu Huong, Program Officer  Nguyen Thu Ha, Monitoring, Evaluation, Accountability and Learning (MEAL) Officer  Tieu Dinh Hoa, Database Officer  Nguyen Ngoc Anh, Program Assistant Development Alternatives Inc.  Mark Rasmuson, Chief of Party  Bui Van Toan – Deputy Chief of Party  Bui Tri, Program Officer  Phan Ngoc Anh, Finance Manager  Nguyen Thi Van Phuong, Team leader of Public Health Team  Phan Quoc Bao, Program Officer Evaluation of USAID Vietnam Disabilities Programming 83 GOVERNMENT OF VIETNAM (Central Level) National Assembly  Luong Phan Cu, Former Vice Chairperson of Committee on Social Affairs Ministry of Health  Tran Quy Tuong, Deputy Director of Medical Services Administration Bureau  Le Tuan Dong, Head of Division of Rehabilitation and Examination, Medical Services Administration Ministry of Labor, Invalids and Social Affairs  Ha Dinh Bon, Director, Department of Legal Affairs  Nguyen Van Hoi, Director, Department of Social Protection  Nguyen Ngoc Toan, Vice Director, Department of Social Protection  Tran Canh Tung, Officer, Department of Social Protection  Dinh Thi Thuy, Deputy Director, Vietnam National Coordinating Council for Disabilities  Ngo Xuan Lieu, Officer, Labor Market Division Ministry of Education and Training  Tran Thi Tham, Deputy Director, Department of Primary Education  Nguyen Trong Hoan, Deputy Director, Department of Secondary Education  Ly Quoc Huy, Officer, Department of Primary Education  Nguyen Thi Quy Suu, Officer, Department of Primary Education OTHER ORGANIZATIONS/INSTITUTION AT CENTRAL LEVEL  Dang Van Thanh, Acting General Secretary, Vietnam Federation on Disability (VFD)  Phung Quang Huy, Director, Blue Ribbon Employers Council (BREC)  Nguyen Hong Thuc, Director, Settlement Research Institute  Nguyen Thi Kim Hoa, Dean of Social Work Faculty, University of Social Sciences and Humanity  Tieu Thi Minh Huong, Vice Dean of Social Work Faculty, University of Labor and Social Affairs  Le Thuy, Lecturer of Social Work Faculty, University of Labor and Social Affairs  Do Van Trai, Lecturer of Social Work Faculty, University of Labor and Social Affairs  Nguyen Trong Tien, Dean of Social Work Faculty, Vietnam Youth Academy  Tran Van Minh, Dean of Physical Rehabilitation Department, Hanoi Medical University Hospital  Nguyen Dinh Lieu, Chairperson of the Association for the Support of Vietnamese Handicapped and Orphans PROVINCIAL AND DISTRICT LEVEL Hanoi City  Nguyen Toan Phong, Director, Hanoi Employment Introduction Center, Hanoi Department of Labor, Invalids and Social Affairs  Duong Thi Van, Chairperson, Hanoi Disabled People Association Danang City City Level  Nguyen Hung Hiep, Deputy Director, Danang City Department of Labor, Invalids and Social Affairs Evaluation of USAID Vietnam Disabilities Programming 84  Nguyen Thi Thu Huong, Head of Social Protection Division, Danang City Department of Labor, Invalids and Social Affairs  Nguyen Thi Bich Thoa, Officer, Social Protection Division, Danang City Department of Labor, Invalids and Social Affairs  Ho Anh Dat, Officer Social Protection Division, Danang City Department of Labor, Invalids and Social Affairs  Mr. Vinh, IT officer, Danang City Department of Labor, Invalids and Social Affairs  Trinh Xuan Vinh, Program Officer, Social Protection Division, Danang City Department of Labor, Invalids and Social Affairs  Nguyen Phao, Director of Employment Service Center, Danang City Department of Labor, Invalids and Social Affairs  Truong Thi Nhu Hoa, Director of Social Work Center, Danang City Department of Labor, Invalids and Social Affairs  Nguyen Minh Hung, Deputy Director of Danang Department of Education and Training  Ho Thi Cam Binh, Head of Primary Education Division, Danang Department of Education and Training  Phan Thi Thuan Nhi, Head of Pre-school Education Division, Danang Department of Education and Training  Pham Thi Hanh, Deputy Principal of Nguyen Dinh Chieu Special Education School  Nguyen Thi Tuong Vi, Doctor of Cardiovascular, Danang Mother and Children Hospital  Vo Thi Minh Giang, Technician, Danang Rehabilitation Hospital  Bui Van Anh, Head of Spinal Cord Department, Danang Rehabilitation Hospital  Ho Manh Canh, Officer of Pediatric department, Danang Rehabilitation Hospital  Do Trong Tu, Teacher of Physical Education, Nguyen Dinh Chieu Special Education School  Nguyen Mau Chau Suong, Teacher, Nguyen Dinh Chieu Special Education School  Tran Thi Huyen Suong, Teacher of Rehabilitation, Danang University of Pharmacy and Medical Technology  Nguyen Tran Thi Y Nhi, Technical staff of Rehabilitation, Danang University of Pharmacy and Medical Technology  Vu Ngoc Duy, Counsellor/social worker, Danang Social Work Center  Mai Duc Vu, Training and community development staff, Danang Social Work Center  Tran Manh Huy, CEO, VBPO Company  Nguyen Pham Thanh Phuong, CEO assistant, VBPO Company  Le Hien Hanh, Beneficiary  Nguyen Thi Anh Dao, President of Steering Committee, Dong A University  Huynh Thi Thien Ly, Head of Project Management Board (on Vocational Training for Persons with disabilities), Dong A University  Nguyen Quoc Vuong, Member of Project Management Board, Dong A University  Le Cong Dao, Class Manager, Dong A University  Nguyen Hoai Nam, University Facility Manager, Dong A University District and Commune Level  Nguyen Thi Hiep, Vice Chairperson of Hoa Vang People’s Committee  Dang Thap, Head of Hoa Vang District Bureau of Labor, Invalids and Social Affairs  Tran Thi Quy, M&E officer of Hoang Vang District Bureau of Labor, Invalids and Social Affairs Evaluation of USAID Vietnam Disabilities Programming 85  Nguyen Thi Thuy Vy, Officer of Hoa Vang District Bureau of Education and Training  Tan Tai, Beneficiary, Hoa Phong Commune, Hoa Vang District, Danang City  Tan Hau, Beneficiary, Hoa Phong Commune, Hoa Vang District, Danang City  Nguyen Thi Ngoc Anh, Teacher, An Phuoc Primary School, Hoa Phong Commune, Hoa Vang District  Tran Van Phi, Vice Chairperson of Cam Le District People’s Committee  Doan Van Hoa, Deputy Head of Cam Le District Bureau of Labor, Invalids and Social Affairs  Nguyen Thanh Long, Officer, Cam Le District Bureau of Labor, Invalids and Social Affairs  Trinh Thi Hong, Officer, Cam Le District Bureau of Education and Training  Nguyen Thi Hue, Head of Cam Le District Bureau of Health  Ong Thi Thuy, Vice Chairperson of Hoa Tho Tay Commune People’s Committee, Cam Le District  Nguyen Thi Diem Le, Case manager on education – Deputy Principal of primary school, Hoa Tho Tay Commune, Cam Le District  Le Thi Hang, Case manager on health, Hoa Tho Tay Commune, Cam Le District  Nguyen Hong Duc, Case manager on social affairs, Hoa Tho Tay Commune, Cam Le District  Dang Thanh Minh, Beneficiary, Hoa Tho Tay Commune, Cam Le District Quang Nam Province Province Level  Nguyen Huy, Head of Social Protection Division, Quang Nam Department of Labor, Invalids and Social Affairs  Nguyen Van Loc, Deputy head of Secondary education division, Quang Nam Department of Education and Training  Phan Ngoc Anh, Deputy head of Primary education division, Quang Nam Department of Education and Training  Nguyen Thi Hang, Officer of Pre-school education division, Quang Nam Department of Education and Training District and Commune Level  Nguyen Van Nam, Vice Chairperson of Hiep Duc District People’s Committee  Tran Van Ca, Vice Chairperson of Hiep Duc District Red Cross Organization  Nguyen Bay, Project Collaborator, Hiep Duc District  Phan Nhu Son, Head of Hiep Duc District Bureau of Labor, Invalids and Social Affairs  Tran Ngoc Thanh, Deputy Head of Hiep Duc District Bureau of Labor, Invalids and Social Affairs  Nguyen Ngoc Nam, Vice Chairperson of Hiep Duc District Association of Persons with disabilities  Nguyen Thi Huyen Trang, Beneficiary, Village 3, Group 3, Que Tho Commune, Hiep Duc District  Tran Dong, Head of Bureau of Labor, Invalids and Social Affairs, Hoi An City  Bui Thi Ly, Officer, Bureau of Invalids and Social Affairs, Hoi An City  Nguyen Van Duong, Deputy Head of Bureau of Education and Training, Hoi An City  Van Quy Chan, Officer, Bureau of Education and Training, Hoi An City  Nguyen Thi Tuy, Officer, Bureau of Education and Training, Hoi An City  Dang Ngoc Buu, Chairperson of Hoi An City Association of Young Persons with disabilities  Trinh Xuan Vinh, Former Chairperson of Hoi An City Association of Young Persons with disabilities Evaluation of USAID Vietnam Disabilities Programming 86  Nguyen Thi Hong, Caretaker of beneficiary, Hoi An City  Nguyen Thi Dong, Headmaster of Cam Thanh Primary School, Cam Thanh Commune, Hoi An City  Ly Thi Son Chau, Beneficiary, Cam Thanh Commune, Hoi An City  Ngo Van Dung, Caretaker of beneficiary, Cam Thanh Commune, Hoi An City  Huynh Thi Kim Thao, Deputy Head of Bureau of Education and Training, Hiep Duc District  Phan Thai Quoc Tan, Officer of Bureau of Education and Training, Hiep Duc District  Tran Van Thuong, Officer of Bureau of Education and Training, Hiep Duc District  Pham Van Ba, Vice Chairperson, Que Binh Commune People’ Committee  Vo Doan Viet, Beneficiary, Nhat Tay Village, Binh Lam Commune, Hiep Duc District  Nguyen Thi Tuyet Phuong, Beneficiary, Hoi An City  Nguyen Thi Thuan, Beneficiary, Tan An Township, Hiep Duc District  Ho Cong Du, Beneficiary, Binh Lam Commune, Hiep Duc District  Vo Thi Anh Minh, Headmaster of Binh Minh Kindergarten, Que Binh Commune, Hiep Duc District Dong Nai Province Province Level  Vo Ngoc Thach, Vice Director, Dong Nai Department of Education and Training  Nguyen Minh Kiem. Deputy head, Primary Education Division under Dong Nai Department of Education and Training  Hoang Thi Van Nga, Director, Dong Nai Center for Children with Disabilities  Tran Minh Tam, Vice Director, Dong Nai Center for Children with Disabilities  Pham Thi Bach Hue, Career Development Officer, Dong Nai Center for Children with Disabilities  Nguyen Huu Thanh, Director of Dong Nai Cocial and Children Protection Office  Vo Son Thu, Vice Director of Employment Service Center, Dong Nai Department of Labor, Invalids and Social Affairs  Lan Chi, Officer of Employment Service Center, Dong Nai Department of Labor, Invalids and Social Affairs  Ngo Thi Nhan, Vice President, Dong Nai Association of Orange Agent Victims District and Commune Level  Lam Van Tinh, Head of Bureau of Labor, Invalids and Social Affairs, Bien Hoa City  Tran Van Thuong, Officer of Bien Hoa City Medical Center  Nguyen Hoa Phong, Officer of Bureau of Education and Training, Bien Hoa City  Lam Van Tinh, Head of Bureau of Labor, Invalids and Social Affairs, Bien Hoa City  Nguyen Ngoc Giang Thoai, Officer of Bureau of Labor, Invalids and Social Affairs, Bien Hoa City  Mr. Tung, Officer of Bureau of Labor, Invalids and Social Affairs, Bien Hoa City  Tran Cong Thuong, Officer of Bien Hoa Health Center, Bien Hoa City  Hoang Tan Dat, Health worker, An Hoa Commune, Bien Hoa City  Nguyen Thi Duyen, Vice Chairperson of Commune People’s Committee, Thanh Phu Commune, Vinh Cuu District  Pham Ngoc Hanh, Officer in charge of Labor, Invalids and Social Affairs, Thanh Phu Commune, Vinh Cuu District  Le Quoc Thai, Health worker, Thanh Phu Commune, Vinh Cuu District Evaluation of USAID Vietnam Disabilities Programming 87  Nguyen Thi Kim Cuong, Vice Headmaster of Primary school, Thanh Phu Commune, Vinh Cuu District  Tran Thi Huyen, Vice Headmaster, Primary school in Trang Dai Ward, Bien Hoa City  Luu Thi Phuoc, Officer in charge of children’s affairs of Trang Dai Ward People’s Committee, Bien Hoa City  Le Thi Thu Thao, Beneficiary, Trang Dai Ward, Bien Hoa City  Nguyen Trong Hieu, Beneficiary, Trang Dai Ward, Bien Hoa City  Nguyen Thi Thu Huong, Beneficiary, Hoa Binh Ward, Bien Hoa City  Truong Ngoc Quang, Beneficiary, Hoa Binh Ward, Bien Hoa City  Nguyen Thi Hoang Oanh, Worker in Chang Shin Company, Bien Hoa City  Dinh Thi Bich Tram, Human Resource Manager, Unipax Company Limited, Bien Hoa City  Quach Vinh Tai, Deputy Head of Bureau of Labor, Invalids and Social Affairs, Vinh Cuu District  Tran Ngoc Huong, Officer of Bureau of Health, Vinh Cuu District  Tran Van Quang, Officer of Bureau of Labor, Invalids and Social Affairs, Vinh Cuu District  Pham Hung, Vice Chairperson, Commune People’s Committee  Luu Thi Phuoc, Commune officer in charge of Labor, invalid and social affairs  Tran Thi Huyen, Vice Headmaster of Primary School  Co Thi Cai, Beneficiary, Thanh Phu Commune, Vinh Cuu District  Le Ngoc Hai, Beneficiary, Thanh Phu Commune, Vinh Cuu District Evaluation of USAID Vietnam Disabilities Programming 88 ANNEX 6: EVALUATION SCHEDULE Week of: 9 – 13 March, 2015 Location: Hanoi Time Name of Person/Title/Agency Tuesday 10 March, 2015 15:00 – 17:00 USAID Officers  Le Ha Van – Contracting Officer's Representative, United States Agency for International Development in Vietnam (USAID)  Ezra Simon – Education Officer of Environmental and Social Development Office, United States Agency for International Development in Vietnam (USAID)  Mei Mei Peng – Education Officer of Environmental and Social Development Office, United States Agency for International Development in Vietnam (USAID)  Kyung Choe – Office of General Development Director, United States Agency for International Development in Vietnam (USAID) Wednesday 10 March, 2015 09:30 – 11:30 USAID Officers  Le Ha Van – Contracting Officer's Representative, United States Agency for International Development in Vietnam (USAID)  Kyung Choe – Office of General Development Director, United States Agency for International Development in Vietnam (USAID) Thursday11 March, 2015 09:00 – 11:00 Vietnam Assistance for the Handicapped  Tran Van Ca – President  Bui Van Toan – Country Director  Nguyen Thi Thu Huong – Chief of Party  Robert Horvath – Deputy Chief of Party  Ong Kek Hin (Michael) – Social Work Specialist Friday 12 March, 2015 09:00 – 11:00 Vietnam Assistance for the Handicapped  Nguyen Thi Thu Huong – Chief of Party  Ong Kek Hin (Michael) – Social Work Specialist Evaluation of USAID Vietnam Disabilities Programming 89 Week of: 16 - 20 March, 2015 Location: Hanoi Time Name of Person/Title/Agency Tuesday 17 March, 2015 9:00 – 12:00 Catholic Relief Services  Snigdha Chakraoborty – Country Director  Sanda Rihtman – Asia Sub-regional Representative  Nguyen Thi Thuy – Technical Advisor  Luu Thu Huong – Program Officer  Nguyen Thu Ha – Monitoring, Evaluation, Accountability and Learning (MEAL) Officer  Tieu Dinh Hoa – Database Officer  Nguyen Ngoc Anh – Program Assistant Thursday 19 March, 2015 14:00 – 16:00 DAI team in Danang (via skype call)  Mark Rasmuson, Chief of Party  Bui Van Toan – Deputy Chief of Party  Bui Tri – Program Officer  Phan Quoc Bao – Program Officer Week of: 23 – 27 March, 2015 Location: Hanoi Time Name of Person/Title/Agency Monday 23 March, 2015 10:30 – 12:00 Vietnam Chamber of Commerce and Industry (VCCI) – Blue Ribbon Employment Committee (BREC)  Mr. Phung Quang Huy – President of BREC 13:00 – 14:30 Hanoi Disabled People Association  Ms. Duong Thi Van – Chairperson 14:00 – 15:30 Vietnam National Coordinating Council for Disabilities (NCCD)  Ms. Dinh Thi Thuy – Deputy Director Tuesday 24 March, 2015 09:00 – 10:30 Department of Legal Affairs – Ministry of Labor, Invalids and Social Affairs  Mr. Ha Dinh Bon – Director 09:00 – 10:30 Faculty of Physical Rehabilitation - Hanoi Medical University  Mr. Tran Van Minh – Dean 10:00 -11:30 Faculty of Social work – Hanoi University of Social Sciences and Humanity Evaluation of USAID Vietnam Disabilities Programming 90  Ms. Nguyen Thi Kim Hoa – Dean 14:00 – 15:30 National Assembly - Committee on Social Affairs  Mr.Luong Phan Cu - Former Vice chairperson Thursday 26 March, 2015 09:00 – 10:30 Department of Social Protection – Ministry of Labor, Invalids and Social Affairs  Mr. Nguyen Van Hoi – Director  Mr. Nguyen Ngoc Toan – Vice Director  Mr. Tran Canh Tung – Officer 09:00 – 10:30 Hanoi Employment service center  Mr. Nguyen Toan Phong – Director 14:00 – 15:30 Vietnam Federation on Disabilities  Mr. Dang Van Thanh – Acting General Secretary Friday 27 March, 2015 09:30 – 11:00 Catholic Relief Services  Ms. Nguyen Thi Thanh Hien – National Program Officer on Vocational Training 14:00 – 16:30 Esoftflow Company  Ms. Nguyen Thuy Ha, HR Manager  Company staff 16:00 – 17:30 Department of Employment – Ministry of Labor, Invalids and Social Affairs  Mr. Ngo Xuan Lieu - Officer Week of: 30 March – 3 April, 2015 Location: Danang City Time Name of Person/Title/Agency Monday 30 March, 2015 9:30 – 11:30 Danang Department of Labor, Invalids, and Social Affairs  Mr. Nguyen Hung Hiep – Vice Director  Ms. Nguyen Thi Thu Huong – Head of Social Protection Division  Informatics Technology Officer  Ms. Truong Thi Nhu Hoa – Director of Social Work Center  Mr. Nguyen Phao – Director of Employment Service Center  Staff of DOLISA who receive training or assistance from the program 14:00-17:00 Danang Rehabilitation Hospital  Mr. Than Van Chin - Director  Hospital staff who receive training or assistance from the program (2 to 3 people) Evaluation of USAID Vietnam Disabilities Programming 91  persons with disabilities clients (2 clients) 14:00-17:00 Employment Services Center  Group of staff who participated in capacity building/training activities Tuesday 31 March, 2015 8:00 – 11:00 Danang Department of Education and Training  Mr. Nguyen Minh Hung – Vice Director  Ms. Binh – Head of Division of Primary Education  Special education officer 8:00 – 11:00 Nguyen Dinh Chieu Special Education School  Ms. Hanh – Deputy principal  Teachers who receive training or assistance from the program  School students with disability 14:00 – 17:00 Danang DPO  Mr. Truong Cong Nghiem – President  Staff who receive training or assistance from the program 14:00 – 17:00 Dong A University  Ms. Nguyen Thi Anh Dao – President  Ms. Huynh Thi Thien Ly – Project management board member  Mr. Nguyen Quoc Vuong – Project management board member  Mr. Le Cong Dao – Class management  Trainers who receive support from the program Wednesday 1 April, 2015 8:00 – 11:00 Cam Le District People Committee  Tran Van Phi – Vice chairperson of District People’s Committee  Nguyen Thanh Long – Case supervisor  Nguyen Thi Hue - Case supervisor  Trinh Thi Hong - Case supervisor 8:00 – 11:00 Hoa Vang District People Committee  Nguyen Thi Hiep - Vice chairperson of District People’s Committee  Dang Thap – Head of District Bureau of Labor, Invalids and Social Affairs  Nguyen Thi Hoa – Case supervisor  Tran Thi Quy – Case supervision Evaluation of USAID Vietnam Disabilities Programming 92  Nguyen Thi Thuy Vy – Case supervisor 13:30-16:30 Hoa Tho Tay Commune, Cam Le District  Vice Chairperson of Commune People’s Committee  Case managers  persons with disabilities being beneficiaries of the project 13:30-16:30 Hoa Phong Commune, Hoa Vang District  Vice Chairperson of Commune People’s Committee  Case managers  persons with disabilities being beneficiaries of the project Thursday 2 April, 2015 8:00 – 11:00 An Phuoc Primary School, Cam Le District  Headmaster  Teachers who receive training or assistance form the project  Students with disabilities 8:00 – 11:00 Hoa Bac commune in Hoa Vang District 13:30-16:30 Danang University of Medical Technology and Pharmacy  Dr. Le Van Nho- Vice Director  MSc. Cao Bich Thuy - Head of Rehabilitation department  Trainers who receive support from the program Friday 3 April, 2015 8:30 – 11:30 Danang Women and Children Hospital  Mr. Tran Dinh Vinh – Director  Hospital staff who receive training or assistance from the program 8:30 – 11:30 VBPO Company  Mr. Tran Manh Huy – Director  Staff who receive training or assistance from the program 13:30 – 16:30 Thien Tam Company  Ms. Đặng Thị Ngọc Ánh - Director  Staff who receive training or assistance from the program 13:30 – 16:30 Interview with DAI technical staff Evaluation of USAID Vietnam Disabilities Programming 93 Week of: 06-10 April, 2015 Location: Quang Nam Province Time Name of Person/Title/Agency Monday 06 April, 2015 9:00 – 11:00 Quang Nam DPO  Mr. Hua Quoc Dung – Chairperson  Standing Board members  persons with disabilities being project beneficiaries 13:30-16:00 Quang Nam Department of Education and Training  Mrs. Phan Ngoc Anh – Division of Primary education  Mr. Nguyen Van Loc – Division of Secondary education Tuesday, 07 April 2015 8:00 – 11:30 Hiep Duc District  Mr. Nguyen Van Nam – Vice Chairperson of District People Committee  Huynh Thi Kim Thao – Head of District Unit of Education and Training  Chairperson of DPO Hiep Duc  Nguyen Bay – Social worker  Mai Van Ca – Hiep Duc Red Cross Organization 13:30 – 17:00 Visit Binh Lam commune (World Concern model on VT)  Visit household of persons with disabilities who participated in Vocational training model of CRS Wednesday 08 April, 2015 Hiep Duc District (cont.) 8:30 – 11:30 Visit 2 households with home-based schooling model in CRS’ IE model, one in Tan An Township, one in Que Tho commune 13:30 – 17:30 Schools visit in Que Binh Commune, Hiep Duc District  Kindergarten Binh Minh  Tran Hung Dao Primary school Lower secondary school Thursday 09 April, 2015 Hoi An City 8:00 – 11:00 Hoi An City People Committee Evaluation of USAID Vietnam Disabilities Programming 94  Mr. Tran Dong – Head of Bureau of Labor, Invalids and Social Affairs  Mr. Nguyen Van Duong - Head of Division of Education and Training  Mrs. Nguyen Tuy Phuong  Mr. Trinh Xuan Vinh - DPO 13:30 – 17:30 Household visit in Hoi An Visit 4 households of persons with disabilities in Hoi An who participated in World Concern VT model in Hoi An City and Cam Thanh Commune Friday, 10 April 2015 8:00 – 11:00 Household visit in Hoi An Visit 4 households of persons with disabilities in Hoi An who participated in CRS’s IE model: Cam Thanh Primary school in Cam Thanh Commune Week of: 06-10 April, 2015 Location: Dong Nai Province Time Name of Person/Title/Agency Monday 06 April, 2015 8:30 – 11:30 Dong Nai Provincial Department of Labor, Invalids, and Social Affairs  Mr. Nguyen Van Thuoc – Director  Ms. Nguyen Thi Kieu Oanh – Vice Director  Mr. Nguyen Huu Thanh – Head of Social Protection Division  Mr. Vo Son Thu – Vice Director, Employment Service Center  Ms. Nguyen Lan Chi – Officer, Employment Service Center 13:30-16:00 Vinh Cuu District People’s Committee  Mr. Lam Van Tinh – Head of Bien Hoa City Bureau of Labor, Invalids and Social Affairs  Mr. Tran Cong Thuong, Case manager  Ms. Nguyen Ngoc Giang Thoai, Case manager  Mr. Vu Hoai Phong, Case manager 16:00-18:00 Home visits in Bien Hoa City  Ms. Nguyen Thi Thu Huong, Beneficiary  Mr. Truong Ngoc Quang, Beneficiary Tuesday, 07 April 2015 10:30 – 12:00 Dong Nai Children with Disabilities Center 10:30 – 12:00 Dong Nai Association of Agent Orange Victims  Ms. Ngo Thi Nhan, Officer Evaluation of USAID Vietnam Disabilities Programming 95  Ms. Nguyen Thi Hien, Officer 13:30 – 17:00 Trang Dai Ward People’s Committee in Bien Hoa City  Mr. Pham Hung – Vice Chairperson of Ward People’s Committee  Ms. Luu Thi Phuoc, Case manager  Ms. Tran Thi Huyen, Case manager  Ms. Vu Thi Mi, Case manager Home visit in Trang Dai Ward Wednesday 08 April, 2015 9:00 – 10:30 People’s Committee of Vinh Cuu District  District People’s Committee leaders  Mr. Quach Vinh Tai, Case manager  Mr. Huynh Van Quang, Case manager  Mr. Luu Tri Cuong, Case manager 10:30 – 12:30 Vinh Cuu District Hospital  Hospital Director  Ms. Tran Ngoc Huong, Case manager 14:00-16:30 Bien Hoa Health Center  Hospital leader  Mr. Hoang Tan Dang, Doctor  Mr. Tran Cong Thuong, Doctor Thursday 09 April, 2015 9:00 – 10:30 Visit Unipax Company 14:00-17:30 Thanh Phu Commune People’s Committee in Vinh Cuu District  Ms. Nguyen Thi Duyen – Vice Chairperson  Ms. Pham Ngoc Hanh, Case manager  Mr. Le Quoc Thai, Case manager  Ms. Nguyen Thi Kim Cuong, Case manager Home visits in Thanh Phu Commune Friday, 10 April 2015 8:30 – 11:30 Dong Nai Department of Education and Training 2:00 – 5:00 Home visits in Bien Hoa City Evaluation of USAID Vietnam Disabilities Programming 96 Week of: 13 – 17 April, 2015 Location: Hanoi Time Name of Person/Title/Agency Tuesday 14 April, 2015 14:00 – 16:00 Catholic Relief Services  Ms. Nguyet Dinh – Chief of Party  Ms. Snighna - Country manager Wednesday 15 April, 2015 14:00 – 15:30 Ministry of Education and Training- Steering Committee on Promotion of Access to and Quality of Inclusive Education for Children with Disabilities via IT  Ms. Tran Thi Tham - Deputy Head of Primary Education Department  Mr. Nguyen Trong Hoan - Deputy Head of Secondary Education Department  Mr. Ly Quoc Huy - Officer  Mr. Le Tran Tuan - Officer Thursday 16 April, 2015 14:00 – 15:30 University of Labor and Social Affairs - Faculty of Social Work  Ms. Tieu Minh Huong – Vice Dean  Mr. Do Van Trai – Lecturer  Ms. Le Thi Thuy – Lecturer Friday 17 April, 2015 14:00 – 15:30 Settlement Research Institution  Dr. Nguyen Hong Thuc - Director Evaluation of USAID Vietnam Disabilities Programming 97 ANNEX 7: INTERVIEW GUIDES INTERVIEW GUIDE FOR FORMER VICE PRESIDENT OF COMMITTEE ON SOCIAL AFFAIRS - NATIONAL ASSEMBLY 1. I am going to read a list of disability laws and policies to you. As I read each one, please tell me which one did you help to review? Name of Disability Laws and Policies Check Law on Disability (2011) Decree 28/2012/ND-CP provides guidance for a number of articles of the Law of Persons with Disability, emphasizes the importance of care and support for persons with disabilities (2012) The Labor Code. Section III of the Code contains regulations on the employment of persons with disabilities (2012) The National Action Plan to Support Persons with disabilities for 2012-2020 (2011) UN Conventions on the Rights of Persons with Disabilities (2014) 2. Please tell me how National Assembly participated in the process to develop laws and policies? 3. Were persons with disabilities involved in the review of the laws and policies? If yes, how? If no, why not? 4. Were women with disabilities involved in the review of the laws and policies? If yes, how? If no, why not? 5. Did you receive any assistance or support from international partners to draft and review the laws and policies? 6. Please tell me if USAID supported your Committee through its Implementing Partners VNAH or CRS, to review disability laws/policies? [Prompt: Examples include: Study tours, Legal analysis, In￾country training, Workshops, Public Awareness, Capacity building, Strengthening networks] 7. Please rate your level of satisfaction with the USAID funded activities that the Committee received. Type of USAID funded activities On a scale from 1 to 4, how satisfied are you with USAID’s assistance to support you in this activity? (Directions: Circle one response) Very Dissatisfied Dissatisfied Satisfied Very Satisfied 1 2 3 4 Evaluation of USAID Vietnam Disabilities Programming 98 8. Did the USAID funded activities helped to develop the GVN disability related legal and policy environment? [If yes, how? If no, why not?] 9. Did the USAID funded activities strengthen the Committee’s capacities to review disability related laws and policies? [If yes, how? If no, why not?] 10. Did the USAID funded activities helped to develop public awareness on disability issues? [If yes, how? If no, why not?] 11. Did the USAID funded activities helped improve disability services? [If yes, how? If no, why not?] 12. Did the passage of disability laws and policies improve the lives and opportunities for persons with disabilities? [If yes, how? If no, why not?] 13. Did the passage of disability laws and policies improve the lives and opportunities for girls and women with disabilities? [If yes, how? If no, why not?] 14. What are the three main challenges in monitoring the implementation of disability laws and policies? [Prompt: Within the government, within society) 15. What has to be done to overcome these challenges? [Prompt: How can the government ensure that the implementation of these laws and policies are sustained over time?] 16. Please identify 3 main challenges the Committee face to continue to review and monitor the implementation of the disability laws and policies? (e.g., capacity building, resources, political will, legal framework) 17. What has to be done to overcome these challenges? 18. Please identify 3 main challenges persons with disabilities continue to face? 19. What has to be done to overcome these challenges? 20. Please identify 3 main challenges girls and women with disabilities continue to face? 21. What has to be done to overcome these challenges? Evaluation of USAID Vietnam Disabilities Programming 99 INTERVIEW GUIDE FOR MINISTRY OF LABOR, INVALIDS AND SOCIAL AFFAIRS – NATIONAL COORDINATING COUNCIL ON DISABILITIES (NCCD) List name and position of the members who attended the interview Name Position 1. I am going to read a list of disability laws and policies to you. As I read each one, please tell me which ones your Committee helped to develop? Name of Disability Laws and Policies Check Law on Disability (2011) The Labor Code. Section III of the Code contains regulations on the employment of persons with disabilities (2012) The National Action Plan to Support Persons with disabilities for 2012-2020 (2011) UN Convention on the Rights of Persons with Disabilities (2014) 2. Please tell me how your Committee contributed to developing or implementing the laws or policies? 3. I am going to read a list of activities to you. As I read each one, please tell me which one, if any, did USAID support through its Implementing Partners VNAH or CRS, to develop or implement disability laws/policies? [Directions: Read the list to the person and tick all appropriate boxes] [Prompts: Examples include: Study tours, Legal analysis, In-country training, Workshops, Public Awareness, Capacity building, Strengthening networks] Type of Activities Directions: Tick if “Yes.” 3. Were persons with disabilities included in the activity? 4. Were women with disabilities included in the activity? Directions: Tick if “Yes.” Evaluation of USAID Vietnam Disabilities Programming 100 4. For the activities that were mentioned, which one was the most important activity in developing the laws/policies? 5. For the activities that were mentioned, which one was the most important activity in implementing the laws/policies? 6. Please rate your level of satisfaction with the USAID assistance that you received. Type of Assistance (Directions: Insert answers from Q3) 8. On a scale from 1 to 4, how satisfied are you with USAID’s assistance to support you in this activity? (Directions: Circle one response) Very Dissatisfied Dissatisfied Satisfied Very Satisfied 1 2 3 4 7. Please identify 3 main barriers in continuing your work? (e.g., capacity building, resources, political will, legal framework) 8. What has to be done to resolve these barriers? [Prompt: This is in the past] 9. Please identify 3 main barriers persons with disabilities continue to face? 10. What has to be done to resolve these barriers? [Prompt: This is in the past] 11. What needs to be done to resolve these barriers? [Prompt: What needs to be done now in the future?] 12. Please identify 3 main barriers girls and women with disabilities continue to face? 13. What has to be done to resolve these barriers? 14. What needs to be done to resolve these barriers? [Prompt: What needs to be done now in the future?] 15. What specific benefits does your organization receive from the current disability laws and policies? [List up to 3 benefits] 16. What benefits or opportunities do you think persons with disabilities receive the current disability laws and policies? 17. What benefits or opportunities do you think girls and women with disabilities receive from the laws and policies? Evaluation of USAID Vietnam Disabilities Programming 101 INTERVIEW GUIDE FOR MINISTRY OF LABOR, INVALIDS AND SOCIAL AFFAIRS - SOCIAL PROTECTION DEPARTMENT List name and position of the members who attended the interview Name Position 1. I am going to read a list of disability laws and policies to you. As I read each one, please tell me which one did you help to develop or implement? Name of Disability Laws and Policies Developed Implemented Law on Disability (2011) The National Action Plan to Support Persons with disabilities for 2012-2020 (2011) Decree 28/2012/ND-CP provides guidance for a number of articles of the Law of Persons with Disability, emphasizes the importance of care and support for persons with disabilities (2012) The Labor Code. Section III of the Code contains regulations on the employment of persons with disabilities (2012) Circular 26/12/TT-BLĐTBXH provides guidance for Decree 28/2012/CD-CP (2012) Inter-ministerial Circular 34 /2012/TTLT- BYT-BLĐTBXH provides guidance for determination of disability level by medical council (2011) UN Conventions on the Rights of Persons with Disabilities (2014) 2. Please tell me how Social Protection Department contributed to developing or implementing the laws or policies? 3. Please tell me if USAID supported the Social Protection Department through its Implementing Partners VNAH or CRS, to review disability laws/policies? [Prompt: Did they provide Study tours, Legal analysis, in-country trainings?) Evaluation of USAID Vietnam Disabilities Programming 102 Type of Activities Directions: Tick if “Yes.” 3. Were persons with disabilities included in the activity? [Directions: Tick if “Yes”] 4. Were women with disabilities included in the activity? [Directions: Tick if yes] 4. Please rate your level of satisfaction with the USAID assistance that you received. Type of Assistance (Directions: Insert answers from Q3) On a scale from 1 to 4, how satisfied are you with USAID’s assistance to support you in this activity? (Directions: Circle one response) Very Dissatisfied Dissatisfied Satisfied Very Satisfied 1 2 3 4 5. For the activities that were mentioned, which one was the most important activity in developing the laws/policies? 6. For the activities that were mentioned, which one was the most important activity in implementing the laws/policies? 7. What benefits or opportunities do you think persons with disabilities receive the current disability laws and policies? 8. What benefits or opportunities do you think girls and women with disabilities receive from the laws and policies? 9. What type of activities did USAID provide to Social Protection Department, through its Implementing Partners VNAH, CRS or DAI, to improve its work in the following areas: employment/ livelihood support, social inclusion? [Prompt: vocational rehabilitation model/ curriculum develop/ training, case management] Areas Yes? Describe the Type of Activity that USAID Supported? Employment/ Livelihood/Vocational Rehabilitation Case Management NCCD and Coordination Awareness-raising 10. Please rate your level of satisfaction with the USAID assistance that you receive Evaluation of USAID Vietnam Disabilities Programming 103 Type of Assistance (Directions: Insert answers from Q9) On a scale from 1 to 4, how satisfied are you with USAID’s assistance to support you in this activity? (Directions: Circle one response) Very Dissatisfied Dissatisfied Satisfied Very Satisfied 1 2 3 4 11. Does Social Protection Department have plans to replicate the model of vocational rehabilitation for persons with disabilities? Why/ why not? 12. If yes, where will the model be replicated? [PROMPT: At the national level? At the local level: Which provinces? Any replication in the districts?] 13. How will Social Protection Department replicate the model? [PROMPT: disseminate the material, train staff to replicate, finance to establish vocational rehabilitation room at local employment service centers, etc.] 14. What resources (finance, human resource, etc.) will you use to replicate it? 15. Does Social Protection Department have plans to replicate the model of case management for persons with disabilities? Why/ why not? 16. If yes, where will the model be replicated? [PROMPT: At the national level? At the local level: Which provinces? Any replication in the districts?] 17. How will Social Protection Department replicate the model? [PROMPT: disseminate the material, train staff to replicate, finance to establish vocational rehabilitation room at local employment service centers, etc.] 18. What resources (finance, human resource, etc.) will you use to replicate it? 19. Please identify three main challenges you face in replicating case management? 20. What has to be done to overcome these challenges? 21. Do think case management help improve the lives and opportunities for persons with disabilities and their family? [If yes, how? If no, why not?] 22. Please identify three main challenges persons with disabilities continue to face. 23. What has to be done to overcome these challenges? [PROMPT: What needs to be done now or in the future?] 24. Please identify three main challenges girls and women with disabilities continue to face. What has to be done to overcome these challenges? [PROMPT: What needs to be done now or in the future?] Evaluation of USAID Vietnam Disabilities Programming 104 INTERVIEW GUIDE FOR MINISTRY OF LABOR, INVALIDS AND SOCIAL AFFAIRS – DEPARTMENT OF LEGAL AFFAIRS List name and position of the members who attended the interview Name Position I am going to read a list of disability laws and policies to you. As I read each one, please tell me which one did you help to develop or implement? Disability Laws, Ordinances and Regulations in Vietnam  Law on People with Disabilities (2011)  Decree 28/2012/ND-CP provides guidance for a number of articles of the Law of Persons with Disability, emphasizes the importance of care and support for persons with disabilities (2012)  Circular 26/12/TT-BLĐTBXH provides guidance for Decree 28/2012/CD-CP (2012)  Inter-ministerial Circular 34 /2012/TTLT- BYT-BLĐTBXH provides guidance for determination of disability level by medical council (2011)  The Labor Code. Section III of the Code contains regulations on the employment of persons with disabilities (2012) and the guiding decree.  The National Action Plan to Support Persons with disabilities for 2012-2020 (2011)  UN Conventions on the Rights of Persons with Disabilities (2014) 1. Please tell me how your Department contributed to the drafting of these disability laws and policies? 2. What activities did USAID, through its Implementing Partners VNAH or CRS, assist your Department in drafting disability laws/policies and providing guidance to the implementation of the laws, ordinances, and regulations? [Prompt: Examples include: Study tours, Legal analysis, In-country training, Workshops, Public Awareness, Capacity building, Strengthening networks] Type of Activities Directions: Tick if “Yes.” 4. Were persons with disabilities included in the activity? [Directions: Tick if “Yes”] 5. Were women with disabilities included in the activity? [Directions: Tick if yes] Evaluation of USAID Vietnam Disabilities Programming 105 3. For the activities that were mentioned, which one was the most important activity in drafting disability laws and policies? 4. Please rate your level of satisfaction with the USAID assistance that your Department received. Type of Assistance (Directions: Insert answers from Q3) On a scale from 1 to 4, how satisfied are you with USAID’s assistance to support you in this activity? Directions: Circle one response) Very Dissatisfied Dissatisfied Satisfied Very Satisfied 1 2 3 4 5. Do you know if there is a monitoring and evaluation mechanism within the government to ensure that the disability laws and policies are implemented at the national and local levels? 6. How does the mechanism function? 7. What are the challenges in monitoring the implementation of disability laws and policies at the national and local levels? 8. What has to be done to overcome these challenges? 9. Please identify 3 main challenges for your organization to further improve disability laws and policies? (e.g., capacity building, resources, political will, legal framework) 10. What needs to be done to overcome these challenges? Evaluation of USAID Vietnam Disabilities Programming 106 INTERVIEW GUIDE FOR MINISTRY OF LABOR, INVALIDS AND SOCIAL AFFAIRS – BUREAU OF EMPLOYMENT List name and position of the members who attended the interview Name Position 1. Please tell me if USAID supported BOE through its Implementing Partners VNAH or DAI, to build your capacity in providing employment-related services to persons with disabilities? If yes, what were the supported activities? [Prompt: examples include training material development, TOT, training workshops, pilot model of vocational rehabilitation, job fair and job placement for workers with disabilities, etc.] 2. For the activities that were mentioned, which one was the most important activity to help you improve services for job seekers with disabilities? 3. Please rate your level of satisfaction with the USAID assistance that you received. Type of Assistance (Directions: Insert answers from Q1) On a scale from 1 to 4, how satisfied are you with USAID’s assistance to support you in this activity? (Directions: Circle one response) Very Dissatisfied Dissatisfied Satisfied Very Satisfied 1 2 3 4 4. Did your BOE participate in the development of the training material on vocational rehabilitation for persons with disabilities workers? If yes, please describe the process of material development and what your organization contributed in this process. [Prompt: needs assessment, drafting, pilot, revisions, feedback workshop, updates, approval, etc.) 5. What was the goal and objectives of the training material on vocational rehabilitation? Who was the targeted group? What was the key content of the training material? [Prompt: Major areas or major topics] 6. What year did your organization decide to formally implement the training material? Where was it implemented/ or planned to be implemented? [PROMPT: At the national level? At the local level: Which provinces? Any implementation in the districts?] 7. How many staff has been trained as of January 1, 2015? How many more do you plan to train this year? 8. Does BOE have plans to replicate the model of vocational rehabilitation for persons with disabilities? Why/ why not? 9. If yes, where will the model be replicated? [PROMPT: At the national level? At the local level: Which provinces? Any replication in the districts?] Evaluation of USAID Vietnam Disabilities Programming 107 10. How will you replicate the model? [PROMPT: disseminate the material, train staff to replicate, finance to establish vocational rehabilitation room at local employment service centers, etc.] What resources (finance, human resource, etc.) will BOE use to replicate it? 11. Does BOE have plans to replicate the job fair/ job placement service for job seekers with disabilities? Why/ why not? 12. If yes, where will the service be replicated? [PROMPT: At the national level? At the local level: Which provinces? Any replication in the districts?] 13. How will you replicate the service? [PROMPT: disseminate the material, train staff to replicate, finance to organize integrated job fairs/ persons with disabilities job placement service at local employment service centers, etc.] What resources (finance, human resource, etc.) will BOE use to replicate it? 14. Did the USAID funded activities help you improve employment services for persons with disabilities? If yes, how? If no, why not? 15. Please identify three main challenges you face to continue providing employment services to persons with disabilities? (PROMPT: e.g., capacity building, resources, political will, legal framework) 16. What has to be done to overcome these challenges? [PROMPT: What needs to be done now or in the future?] 17. Do you think BOE employment services help strengthen employment opportunities for persons with disabilities? [If yes, how? If no, why not?] 18. Please identify three main challenges persons with disabilities face in gaining decent employment? 19. What has to be done to overcome these challenges? [PROMPT: What needs to be done now or in the future?] 20. Please identify three main challenges women with disabilities face in gaining decent employment? 21. What has to be done to overcome these barriers? [PROMPT: What needs to be done now or in the future?] Evaluation of USAID Vietnam Disabilities Programming 108 INTERVIEW GUIDE FOR MINISTRY OF EDUCATION AND TRAINING - STEERING COMMITTEE FOR CHILDREN WITH DISABILITIES AND CHILDREN IN DIFFICULT SITUATIONS 1. What are your roles and responsibilities? 2. Number and percentage of children with disabilities at  Nursery:  Primary:  Junior Secondary:  Secondary: 3. What are your areas of interventions? EDUCATION LAWS AND POLICIES 3. What disability education and policies have you worked to develop and implement? 4. What is the specific support did USAID provide through CRS and VNAH for development or implementation (e.g., TA, finance to organize workshops, models)? 5. What Issuance of non-code degrees, circulars, and decisions have you worked to develop or guide implementation? Was monitoring and evaluation present during implementation? If yes, please describe. 6. Was participation of persons with disabilities or caregivers participate in policy formulation/ implementation? 7. Were you involved in the development or implementation of regulations on accreditation and qualifications for primary, junior secondary and secondary schools? 8. How to transfer to the next level of education? How to enroll on junior sec/ sec school? 9. Vocational training and career guidance for children with disabilities at junior secondary and secondary education level (regular and special education schools) INCLUSIVE EDUCATION 10. Inclusive education training for in-service teachers: Please describe (e.g., contents, length of training, which does it target, follow- up support). 11. How is it implemented at school (e.g., Reduced number of non-persons with disabilities students, allowance for teachers in charge of the class, technical support and performance evaluation)? CROSSCUTTING ISSUES: Evaluation of USAID Vietnam Disabilities Programming 109 12. Is the school accessible in regular schools? What is the status of accessibility being included in the National School Standards? 13. children with disabilities with mobility impairment accepted at regular school (CRS advocated policy/ decision in Quang Nam or central) ONLINE CURRICULUM 14. How can children be certified at end of level if only 2 subjects? HOME BASED EDUCATION FOR children with disabilities WITH SEVERE DISABILITIES 15. How often do teachers come to the home? How to transfer from home-based education to in￾school? How to get certified at end of level if only 2 subjects? Evaluation of USAID Vietnam Disabilities Programming 110 INTERVIEW GUIDE FOR THE BLUE RIBBON EMPLOYER COUNCIL (BREC) List name and position of the council members who attended the interview Name Position 1. I am going to read a list of disability laws and policies to you. As I read each one, please tell me which, did you help to develop or implement? Name of Disability Laws and Policies Developed Implemented Law on Disability (2011) The National Action Plan to Support Persons with disabilities for 2012-2020 (2011) Decree 28/2012/ND-CP provides guidance for a number of articles of the Law of Persons with Disability, emphasizes the importance of care and support for persons with disabilities (2012) The Labor Code. Section III of the Code contains regulations on the employment of persons with disabilities (2012) Circular 26/12/TT-BLĐTBXH provides guidance for Decree 28/2012/CD-CP (2012) Inter-ministerial Circular 34 /2012/TTLT- BYT-BLĐTBXH provides guidance for determination of disability level by medical council (2011) UN Conventions on the Rights of Persons with Disabilities (2014) 2. I am going to read a list of activities to you. As I read each one, please tell me which, if any, did USAID support through its implementing partners VNAH or CRS, to develop or implement disability laws/policies? [Directions: Read the list to the person and tick all appropriate boxes] Type of Activities Directions: Tick if “Yes.” 3. Were persons with disabilities included in the activity? [Directions: Tick if “Yes”] 4. Were women with disabilities included in the activity? [Directions: Tick if yes] Study tours Evaluation of USAID Vietnam Disabilities Programming 111 Legal analysis In-country training Workshops Public Awareness Capacity building Strengthening network Other: 3. For the activities that were mentioned, which one was the most important activity in developing the laws/policies? 4. For the activities that were mentioned, which one was the most important activity in implementing the laws/policies? 5. Please rate your level of satisfaction with the assistance that you received. Type of Assistance On a scale from 1 to 4, how satisfied are you with USAID’s assistance to support you in this activity? (Directions: Circle one response) Very Dissatisfied Dissatisfied Satisfied Very Satisfied Study tours 1 2 3 4 Legal analysis 1 2 3 4 In-country training 1 2 3 4 Workshops 1 2 3 4 Public Awareness 1 2 3 4 Capacity building 1 2 3 4 Strengthening network 1 2 3 4 Other: 1 2 3 4 6. Please identify 3 main barriers in doing your work? (e.g., capacity buildings, resources, political will, legal framework. 7. What has to be done to resolve these barriers? 8. (e.g., Legislative or regulatory changes, institutional changes, and technical support) Evaluation of USAID Vietnam Disabilities Programming 112 9. Please identify 3 main barriers persons with disabilities face in gaining decent employment? (e.g., capacity building, resources) 10. What has to be done to resolve these barriers? (e.g., Legislative or regulatory changes, institutional changes, and technical support) Evaluation of USAID Vietnam Disabilities Programming 113 INTERVIEW GUIDE FOR TOT CURRICULUM DEVELOPMENT AND IMPLEMENTATION Name of the University: Name of the person or persons interviewed: Name Position 1. What was the issue that the curriculum addressed? 2. Was there a development team? __________ YES __________ No [PROMPT: If yes, what types of people were on the team? If no, why not?] 3. Did you conduct a needs assessment? _____ YES _____ NO [PROMPT: If yes, how did you do it? What was the number of people who were interviewed? What provinces did they come from? What were their jobs? If no, why not?] 4. How did you develop the curriculum? (Did they pilot the curriculum?, testing on pilot students, revisions? Updates? Approval?) 5. Who was the targeted group? 6. What was the goal and objectives? 7. What was the key content of the curriculum? [PROMPT: Major areas or major topics] 8. What year was the curriculum formally implemented? YEAR: _________ 9. Where was it implemented? [PROMPT: In Hanoi? Which provinces? In the districts? Which colleges or universities, at the local level] 10. How many people were trained as of January 1, 2015? Courses Trainers Case Managers Students Social work with persons with disabilities BA MSSW level Social work with persons with disabilities at college level Field Place Model program Case Management for persons with disabilities Evaluation of USAID Vietnam Disabilities Programming 114 TOTAL NUMBER 11. How did you evaluate the curriculum? [Initially and ongoing] 12. Do you follow-up with students to see if they are applying what they learned? (If yes, how? What feedback have you received? How do you use the feedback to improve the curriculum) Type Check one Type of Feedback Received How you used the Feedback Yes No Students Trainers Case Managers 13. What steps were taken to institutionalize the curriculum? (at university-, provincial-, and district￾levels) 14. How do think the implementation of the curriculum benefited the daily lives of persons with disabilities and their family members? 15. How did the curriculum impact the delivery of disability services? 16. Is the curriculum part of a permanent course or a course? [PROMPT: IF YES, what college/university is it being taught? How often is it taught? IF NO, why? 17. Who, what, where are the resources to support the curriculum in the future? [PROMPT: If there were no resources allocated, why not?] 18. How would you improve and expand the implementation of the curriculum? [PROMPT: Would you add other topics? (If yes, what would be the topics?)? Would you add other groups of people (If yes, what groups)? Would you expand the geographic location of the training (If yes, what locations?] Evaluation of USAID Vietnam Disabilities Programming 115 INTERVIEW GUIDE FOR IMPLEMENTING PARTNERS Name of implementing partner: List name and position of the members who attended the interview Name Position 1. What activities have been implemented by [Implementing Partner’s (IP’s) name] for USAID in the past ten years? I am going to read a list of activities by sector. As I read each activity, tell me if the [IP’S name] has been implementing the activity. [Directions: Tick (x) the activity when the person says, “Yes.”] Sectors/Types of Activities Yes? LEGAL AND POLICY FRAMEWORK Policy Development (at Central-, provincial-, district- levels?) Type of Activity: Policy Implementation (at Central-, provincial-, district- levels?) Type of Activity: Policy Coordination (at Central-, provincial-, district- levels?) Type of Activity: Capacity Building/ Institutional Strengthening Type of Activity: HEALTH Direct Services Type of Activity: Evaluation of USAID Vietnam Disabilities Programming 116 Capacity Building/ Institutional Strengthening Type of Activity: Sectors/Types of Activities Yes? EDUCATION SECTOR Direct Services: Inclusive Education Type of Activity: Direct Services: Special Education Type of Activity: Capacity Building/ Institutional Strengthening (Inclusive and Special Education) Type of Activity: EMPLOYMENT/LIVELIHOOD SECTOR Type of Activity: Service: Capacity Building/ Institutional Strengthening Type of Activity: SOCIAL INCLUSION Capacity Building/Institutional Strengthening Type of Activity: LEGAL AND POLICY FRAMEWORK QUESTIONS Read: For the activities that were mentioned in the legal and policy framework, please answer the following questions: 2. How were persons with disabilities involved in policy development process? 3. How were women with disabilities involved in the policy development process? 4. How did persons with disabilities benefit from USAID assistance/support? Evaluation of USAID Vietnam Disabilities Programming 117 5. Which one of these activities was the most important activity in developing laws/policies? Why? 6. Which one of these activities was the least important activity in developing laws/policies? Why? 7. Which one of these activities was the most important activity in implementing laws/policies? 8. Which one of these activities was the least important activity in implementing laws/policies? Why? 9. What were the three main challenges in providing assistance/support for the development of laws/policies? 10. How did you overcome these challenges? 11. What were the three main challenges in providing assistance/support for the implementation of laws/policies? 12. How did you overcome these challenges? 13. What issues still need to be addressed in the legal/policy framework sector (development and implementation)? 14. What type of USAID assistance would help address these issues? HEALTH QUESTIONS Read: For the activities that were mentioned in the Health Sector, please answer the following questions: 15. How did persons with disabilities benefit from USAID assistance/support? 16. Which one of these activities was the most important activity in providing assistance/support? Why? 17. Which one of these activities was the least important activity in providing assistance/support? Why? 18. What were the three main challenges in providing assistance/support? 19. How did you overcome these challenges? 20. What were the three main challenges in providing assistance/support? 21. How did you overcome these challenges? 22. What issues still need to be addressed in the health sector? 23. What type of USAID assistance would help address these issues? EDUCATION QUESTIONS Read: For the activities that were mentioned in the Education Sector, please answer the following questions: 24. How did persons with disabilities benefit from USAID assistance/support? 25. Which one of these activities was the most important activity in providing assistance/support? Why? 26. Which one of these activities was the least important activity in providing assistance/support? Why? 27. What were the three main challenges in providing assistance/support? 28. How did you overcome these challenges? 29. What were the three main challenges in providing assistance/support? 30. How did you overcome these challenges? 31. What issues still need to be addressed in the education sector? 32. What type of USAID assistance would help address these issues? Evaluation of USAID Vietnam Disabilities Programming 118 EMPLOYMENT/LIVELIHOOD QUESTIONS Read: For the activities that were mentioned in the Employment/Livelihood Sector, please answer the following questions: 33. How did persons with disabilities benefit from USAID assistance/support? 34. Which one of these activities was the most important activity in providing assistance/support? Why? 35. Which one of these activities was the least important activity in providing assistance/support? Why? 36. What were the three main challenges in providing assistance/support? 37. How did you overcome these challenges? 38. What were the three main challenges in providing assistance/support? 39. How did you overcome these challenges? 40. What issues still need to be addressed in the employment/livelihood sector? 41. What type of USAID assistance would help address these issues? SOCIAL INCLUSION QUESTIONS Read: For the activities that were mentioned in the Social Inclusion Sector, please answer the following questions: 42. How did persons with disabilities benefit from USAID assistance/support? 43. Which one of these activities was the most important activity in providing assistance/support? Why? 44. Which one of these activities was the least important activity in providing assistance/support? Why? 45. What were the three main challenges in providing assistance/support? 46. How did you overcome these challenges? 47. What were the three main challenges in providing assistance/support? 48. How did you overcome these challenges? 49. What issues still need to be addressed in the Social Inclusion sector? What type of USAID assistance would help address these issues? Evaluation of USAID Vietnam Disabilities Programming 119 INTERVIEW GUIDE FOR PROVINCIAL DEPARTMENT OF LABOR, INVALIDS AND SOCIAL AFAIRS NAME OF PROVINCE: List name and position of the members who attended the interview Name Position 1. Please describe activities/services that DOLISA provides to persons with disabilities? 2. The total number of case managers/social workers? 3. What is the average number of clients in their caseload? 4. Of this number, what is the average number of clients with disabilities? 5. Please tell me if USAID supported DOLISA through its Implementing Partners VNAH or DAI in the legal/policy framework sector. [Directions: Read the name of the activity. Tick the activity (x) if the person says, “Yes.” Then ask what type of assistance they received and write their answers in the space provided.] LEGAL/ POLICY SECTOR Yes? Activity: Development (at provincial- & district- levels?) (Directions: If applicable, indicate the circular/decree) Type of Assistance: Activity: Implementation (at provincial- & district- levels?) (Directions: If applicable, indicate the circular/decree) Type of Assistance: Activity: Coordination (at provincial- & district- levels?) (Directions: If applicable, indicate the circular/decree) Type of Assistance: Activity: Capacity Building (Directions: If applicable, indicate the circular/decree) Type of Assistance: 6. Please tell me how DOLISA contributed to developing or implementing the laws or policies? 7. What are the 3 main challenges they face in policy development? 8. What needs to be done to overcome these challenges? 9. Please rate your level of satisfaction with the USAID assistance that you received for each activity. Evaluation of USAID Vietnam Disabilities Programming 120 Activity & Type of Assistance (Directions: Insert answers from Q1) On a scale from 1 to 4, how satisfied are you with USAID’s assistance to support you in this activity? (Directions: Circle one response) Very Dissatisfied Dissatisfied Satisfied Very Satisfied 1 2 3 4 10. Please tell and me if USAID supported DOLISA through its Implementing Partners VNAH or DAI in the social inclusion sector. [Directions: Read the name of the activity. Tick the activity (x) if the person says, “Yes.” Then ask what type of assistance they received and write their answers in the space provided.] SOCIAL INCLUSION SECTOR Yes? Activity: Case Management Type of Assistance: Activity: Public Awareness Type of Assistance: Activity: Capacity Building Type of Assistance: Training of social workers/case managers/ supervisors 11. How has case management benefited persons with disabilities? 12. How has the training of case manager benefited persons with disabilities? 13. How have public awareness activities benefited persons with disabilities? How would you improve case management training? 14. What are the three challenges in implementing case management? 15. What needs to be done to overcome these challenges? 16. What type of support/assistance do you need from MOLISA to implement case management? What recommendations do you have to improve and expand case management? Evaluation of USAID Vietnam Disabilities Programming 121 INTERVIEW GUIDE FOR HEAD OF SOCIAL PROTECTION DIVISION – PROVINCIAL DEPARTMENT OF LABOR, INVALIDS AND SOCIAL AFFAIRS NAME OF PROVINCE/DISTRICT: List name and position of the members who attended the interview Name Position 1. Please tell and me if USAID supported Social Protection Division through its Implementing Partners VNAH or DAI in the legal/policy framework sector. [Directions: Read the name of the activity. Tick the activity (x) if the person says, “Yes.” Then ask what type of assistance they received and write their answers in the space provided.] LEGAL/ POLICY SECTOR Yes? Activity: Development (at provincial- & district- levels?) (Directions: If applicable, indicate the circular/decree) Type of Assistance: Activity: Implementation (at provincial- & district- levels?) (Directions: If applicable, indicate the circular/decree) Type of Assistance: NCCD Activity: Coordination (at provincial- & district- levels?) (Directions: If applicable, indicate the circular/decree) Type of Assistance: NCCD Activity: Capacity Building (Directions: If applicable, indicate the circular/decree) Type of Assistance: 2. Please tell me how Social Protection Division contributed to developing or implementing the laws or policies? a. Were persons with disabilities included in these activities? Yes No How? b. Were women with disabilities included in these activities? Yes No How? c. Were parents of children with disabilities included in these activities? Yes No How? 3. Directions: If the person indicated that Social Protection Division did not work with NCCD GO TO Q4. If the person stated that Social Protection Division worked with NCCD, ask the following questions: Evaluation of USAID Vietnam Disabilities Programming 122 a. On a scale from 1 to 4, How satisfied were you with NCCD assisting Social Protection Division in developing policies at the local level? (Directions: Circle one response) Very Dissatisfied Dissatisfied Satisfied Very Satisfied 1 2 3 4 b. On a scale from 1 to 4, How satisfied were you with NCCD assisting Social Protection Division in implementing policies at the local level? (Directions: Circle one response) Very Dissatisfied Dissatisfied Satisfied Very Satisfied 1 2 3 4 c. On a scale from 1 to 4, How satisfied were you with NCCD assisting Social Protection Division in coordinating between the Central and local levels? (Circle one response) Very Dissatisfied Dissatisfied Satisfied Very Satisfied 1 2 3 4 4. What type of assistance do you receive from USAID, through VNAH, CRS, or DAI, to set up and implement the provincial action plan on disabilities? 5. On a scale from 1 to 4, How satisfied are you with USAID’s assistance? (Circle one response) Very Dissatisfied Dissatisfied Satisfied Very Satisfied Don’t know 1 2 3 4 9 6. What needs to be done to overcome these challenges? 7. What are the three main challenges you face in implementing the current laws and policies on disabilities? 8. What needs to be done to overcome these challenges? 9. For the types of assistance that were mentioned, which one was the most important in implementing laws/policies/decrees? 10. For types of assistance that were mentioned, which one was the least important in implementing laws/policies/decrees? 11. Please tell and me if USAID supported Social Protection Division through its Implementing Partners VNAH or DAI in the social inclusion sector. [Directions: Read the name of the activity. Tick the activity (x) if the person says, “Yes.” Then ask what type of assistance they received and write their answers in the space provided.] SOCIAL INCLUSION SECTOR Yes? Evaluation of USAID Vietnam Disabilities Programming 123 Activity: Case Management Type of Assistance: Activity: Public Awareness Type of Assistance: Activity: Capacity Building Type of Assistance: Other Activity: Type of Assistance: Other Activity: Type of Assistance: Training of social workers/case managers/ supervisors 12. Please rate your level of satisfaction with the USAID assistance that you received for each activity. Activity & Type of Assistance (Directions: Insert answers from Q7) On a scale from 1 to 4, how satisfied are you with USAID’s assistance to support you in this activity? (Directions: Circle one response) Very Dissatisfied Dissatisfied Satisfied Very Satisfied 1 2 3 4 13. For the types of assistance mentioned, which one was the most important in developing case management? 14. For the types of assistance mentioned, which one were the most important in employing public awareness activities? 15. For the types of assistance mentioned, which one was the most important in providing training to social workers/case managers/case supervisors? 16. For the types of assistance mentioned, which one was the least important in developing case management? 17. For the types of assistance mentioned, which one were the least important in employing public awareness activities? For the types of assistance mentioned, which one was the least important in providing training to social workers/case managers/case supervisors? Evaluation of USAID Vietnam Disabilities Programming 124 INTERVIEW GUIDE FOR PROVINCIAL DEPARTMENT OF LABOR, INVALIDS AND SOCIAL AFFAIRS - SOCIAL WORK CENTER List name and position of the members who attended the interview Name Position 1. Please describe your role and responsibilities as the director of the Social Welfare Center. 2. You participated in the TOT that was conducted by USAID, through DAI? 3. Please describe the training 4. Did they provide follow-up support/assistance to the trainers? If yes, how? 5. OVERALL, how relevant was the training to the Center’s work? Not Relevant at All Somewhat Relevant Relevant Very Relevant 1 2 3 4 7. OVERALL, how satisfied are you with the training? Very Dissatisfied Dissatisfied Satisfied Very Satisfied 1 2 3 4 8. How would you improve the training? 9. What are the three challenges in implementing case management? 10. What would it take to overcome these challenges? 11. How did your staff benefit from the training? (Ask for examples) 12. How did the training improve the service quality? What type of follow-up support or assistance does your staff receive from DAI? Evaluation of USAID Vietnam Disabilities Programming 125 INTERVIEW GUIDE FOR PROFESSIONALS WHO RECEIVED CASE MANAGERMENT/SOCIAL WORK TRAINING BACKGROUND INFORMATION 1. Name: 2. Job Title: 3. Job Function: 4. Do you work with persons with disabilities? If no, why? _____ YES GO TO Q5 _____ NO WHY DID YOU TAKE THE CASE MANAGEMENT/SOCIAL WORK COURSE? 5. Did you take the Case Management/Social Work Course: _____ YES GO TO Q5.a. _____ NO END OF INTERVIEW 5.a. If yes, Date of Course (MM/YYY): ___________________ 5.b. If yes, why did you take the course? 6. What did you like about the course (e.g., length, topics, instructor, usefulness)? 7. What did you dislike about the course (e.g., length, topics, instructor, usefulness)? 8. How can the course be improved? 9. What skills did you learn? (If the person says that she/he did not learn anything, ask why? END OF INTERVIEW) 10. For each skill: Ask the following questions: Evaluation of USAID Vietnam Disabilities Programming 126 SKILL ONE: 1. 1. How often do you use this skill on the job (Circle one) a. I don’t use the skill (GO TO Q2) b. Sometimes (1 to 2 times a week) (GO TO Q3) c. Regularly (At least once a day) (GO TO Q3) d. Frequently (Several times a day) (GO TO Q3) 2. 2. If skill is not used, why? (Circle one responses): a. It is not part of my job (Go to the next SKILL) b. It is too difficult (GO TO Q3) c. I forgot how to do it (GO TO Q3) 3. How would you rate your ability to employ this skill? (Circle the most appropriate responses): a. I am unable to practice it (GO TO Q4) b. I mastered the skill (GO TO Q5) c. I am an expert of the skill (GO TO Q5) 4. If you are unable to perform the skill, why? (Circle one) a. It is too difficult (Directions: Go to next SKILL) b. No one else uses it (Directions: Go to next SKILL) c. I forgot how to do it (Directions: Go to next) SKILL 5. Which statement is true? (Circle the most appropriate response): a. I was able to perform this skill immediately after the training. b. It took me two weeks of practicing before I learned c. It took me more than two weeks of practicing before I learned 6. Which statement is true: (Circle one): a. The skill is performed in the manner described during the training. Evaluation of USAID Vietnam Disabilities Programming 127 b. I revised the skill to meet my clients’ needs. How? 11. How has case management benefited your ability to provide services to persons with disabilities? 12. How has case management benefited the lives of persons with disabilities? 13. What have been the challenges in trying to manage services across departments? 14. What additional resources or assistance do you need to overcome these challenges? 15. Since the training, what type of follow-up assistance and support have you received from the trainers/DAI? (If No, END OF INTERVIEW) 16. How satisfied are you with the assistance or support? Very Dissatisfied Dissatisfied Satisfied Very Satisfied 1 2 3 4 17. If very dissatisfied or dissatisfied, how would you improve it? Evaluation of USAID Vietnam Disabilities Programming 128 INTERVIEW GUIDE FOR PROVINCIAL DEPARTMENT OF LABOR, INVALIDS AND SOCIAL AFFAIRS – EMPLOYMENT SERVICE CENTER List name and position of the members who attended the interview Name Position 1. Please tell me if USAID supported your center through its Implementing Partners VNAH, CRS or DAI, to provide employment-related services to persons with disabilities? If yes, what were the supported activities? [Prompt: examples include development of vocational rehabilitation model and materials, staff training, delivery of job counseling/ job fair/ job placement, follow up with persons with disabilities workers and employers, collection of persons with disabilities-friendly labor market information and recruitment data, etc.] 2. Were persons with disabilities and business involved in the development of the employment-related services and activities mentioned above? If yes, how? If no, why not? 3. Please rate your level of satisfaction with the USAID assistance that you received. Type of Assistance (Directions: Insert answers from Q1) On a scale from 1 to 4, how satisfied are you with USAID’s assistance to support you in this activity? (Directions: Circle one response) Very Dissatisfied Dissatisfied Satisfied Very Satisfied 1 2 3 4 4. Did your center participate in the development of the vocational rehabilitation model for persons with disabilities workers? If yes, please describe the process of development and what your center contributed in this process. [Prompt: needs assessment, design and set up of the facility, preparation of guideline/materials, staff training, pilot, evaluation, launching, etc.] 5. What was the goal and objectives of the model? Who was the targeted customer? What were the keys steps in the service delivery process? [Instruction: get copies of materials] SERVICE DELIVERY PROCESS: 6. What year did your center formally launch the model? Where was or is it implemented? [PROMPT: At the center? Any implementation in the districts or lower level?] 7. YEAR: __________________ 8. LOCATIONS: __________________ 9. How many persons with disabilities customers have been served through the model as of January 1, 2015? How many more do you expect to serve this year? Evaluation of USAID Vietnam Disabilities Programming 129 10. Who paid for the vocational rehabilitation service provided through the model? From what sources was the cost of service provision recovered? 11. Does your center have plans to continue the model of vocational rehabilitation for persons with disabilities? Why/ why not? 12. What resources (finance, human resource, etc.) will you use to continue the model? 13. What year did your center start providing job counseling/ job fairs/ job placement to persons with disabilities? Where was or is it implemented? [PROMPT: At the center? Any implementation in the districts or lower level?] How many persons with disabilities customers have been served as of January 1st, 2015? How many more do you expect to serve this year? Job counseling Job fair Job placement YEAR: _________ _________ _________ LOCATIONS: _________ _________ _________ # CUSTOMERS served by 2014________ _________ _________ # CUSTOMERS expected in 2015 _________ _________ _________ 14. Who paid for the job counseling/ job fairs/ job placement service? From what sources was the cost of service provision recovered? 15. Does your center have plans to continue providing the services mentioned above for persons with disabilities? Why/ why not? 16. What resources (finance, human resource, etc.) will you use to continue the services? 17. Have you put in place a relevant structure and assigned relevant staff to continue vocational rehabilitation and other employment-related services? If yes, what is the new structure? If no, why not? 18. How many persons with disabilities gained employment through the services mentioned before? 19. Did you follow up with the PDW workers and/ or employers after service provision? If yes, how? If no, why not? 20. How did you get labor market information and recruitment data and tailor it to persons with disabilities’ needs? How did you communicate it to persons with disabilities? 21. Did the USAID funded activities help you improve employment services for persons with disabilities? If yes, how? If no, why not? 22. Please identify three main challenges you face to continue providing employment services to persons with disabilities at the level and quality you are doing now? (PROMPT: e.g., capacity building, resources, political will, legal framework, company’s structure, etc.) 23. What has to be done to overcome these challenges? [PROMPT: What needs to be done now or in the future?] 24. Do think your improved services help strengthen employment opportunities for persons with disabilities? [If yes, how? If no, why not?] 25. Please identify three main challenges persons with disabilities continue to face in gaining decent employment? 26. What has to be done to overcome these challenges? [PROMPT: What needs to be done now or in the future?] 27. Please identify three main challenges women with disabilities face in gaining decent employment? What has to be done to overcome these challenges? [PROMPT: What needs to be done now or in the future?] Evaluation of USAID Vietnam Disabilities Programming 130 INTERVIEW GUIDE FOR DPOs AND PARENT ASSOCIATIONS 1. Number of member inn which number are persons with disabilities _____ Parents of persons with disabilities __________ 2. What specific disability groups does your organization represent? 3. Year of establishment: 4. What type of {VNAH/CRS/DAI} programs did you participate in? Please describe the support you received in each program [Directions: Tick all applicable boxes]  Establishment and/ or capacity building of your own organization:  Improvement of network and coordination in disability:  Disability laws/policy development:  Education for children with disabilities (children with disabilities):  Employment and vocational training/ rehabilitation for persons with disabilities:  persons with disabilities case management:  Other (specify)… 5. For each program, please score your level of satisfaction with its usefulness from 1 to 4, 1 being the lowest and 4 being the highest level. Very Dissatisfied Dissatisfied Satisfied Very Satisfied 1 2 3 4 6. Did you contribute anything in the program(s)? If yes, Please describe your contribution for each program. 7. What changed in your organization as the result of each program? Please describe. 8. What specific benefits did your organization get? Please describe. 9. What do you do differently for your members as the result of the programs? 10. What specific benefits did your members get as the result of the programs? Please describe in details. 11. What benefits/ opportunities do you think general persons with disabilities get as the result of the programs? Please describe in details. 12. What legislation framework formed your organization? 13. Can you please describe the process you went through to form and officially register your organization? Evaluation of USAID Vietnam Disabilities Programming 131 14. What were the main challenges you had in this process? 15. How was your organization involved in the formulation process of the new persons with disabilities￾related laws/ policies? 16. How satisfied are you that your organization’s opinions were addressed by the new [laws/ policies]? For each law/ policy, please score your level of satisfaction from 1 to 4, 1 being the lowest and 4 being the highest level. Very Dissatisfied Dissatisfied Satisfied Very Satisfied 1 2 3 4  persons with disabilities laws and sub-laws: Level of Satisfaction = __________  persons with disabilities-related articles in the Labor Code and TVET laws and sub-laws: Level of Satisfaction = _________  NAP: Level of Satisfaction = __________ 17. What influences do the new {laws/policies} have on your organization? What do you do to take advantage of them or to comply with them? a. persons with disabilities laws and sub-laws: b. persons with disabilities-related articles in the Labor Code and TVET laws and sub-laws: 18. National Action Plan (NAP): 19. What additional benefits can your organization have from the new {laws/ policies} compared to the previous one? What are the new challenges? 20. What additional benefits do you think your members can have from the new {laws/ policies} compared to the previous one? Are there any new barriers/ challenges to them? If yes, what are they? 21. What additional benefits do you think general persons with disabilities can have from the new {laws/ policies} compared to the previous one? Are there any new barriers/ challenges to them? If yes, what are they? 22. What laws/ policies other than the formulation and persons with disabilities-related laws/ policies mentioned above have influence on your organization? Your current operation: Your long-term sustainability: 23. How did the USAID supported programs help you develop your institutional capacity? 24. What kind of training/ technical assistance did they provide? Please describe in details for each area below:  Your charter and bylaws:  Your strategies [membership, product/service, PR, advocacy, fund-raising, etc.]:  Management and staff training  Policy advocacy:  Office operation and activity delivery (including any initial financial support):  Internal structures/ systems to serve members and persons with disabilities:  Networking and coordination with Government, private sector, service providers for persons with disabilities: 25. How satisfied are you that the technical assistance and support from USAID was holistic and effective to strengthen your critical institutional capacities to serve your members? Evaluation of USAID Vietnam Disabilities Programming 132 Very Dissatisfied Dissatisfied Satisfied Very Satisfied 1 2 3 4 26. How satisfied are you that the technical assistance and support from USAID was holistic and effective to serve persons with disabilities in general? Please score your level of satisfaction from 1 to 4, 1 being the lowest and 4 being the highest. Very Dissatisfied Dissatisfied Satisfied Very Satisfied 1 2 3 4 27. What did you do differently to serve your members and as the result of the capacity building support from USAID? Please describe in details. 28. What did you do differently to serve general persons with disabilities as the result of the capacity building support from USAID? Please describe in details. 29. Does your organization and your members use any persons with disabilities-related services from the USAID-supported centers/ DPOs? If yes, what service(s) in what centers? (Check all that apply)  Employment Service Center (Hanoi/ Danang/ Dong Nai)  Vocational training centers Hanoi/ Danang  DPOs Hanoi/ Danang/ Quang Nam  Hanoi College of Information Technology  Blue Ribbon Employer Council  Other Specify: ___________________________________________________ 30. Are you satisfied with the service(s)? For each service, please score your level of satisfaction from 1 to 4, 1 being the lowest and 4 being the highest. Very Dissatisfied Dissatisfied Satisfied Very Satisfied 1 2 3 4 31. What are you satisfied about the services? What do you want to be done differently? 32. Do you continue to use the service(s) now? Why? Evaluation of USAID Vietnam Disabilities Programming 133 33. Do you think general persons with disabilities are able to access such services/facilities in your areas? Why and why not? 34. Do you think your organization will continue its operation and service delivery without external support? Why or why not? [e.g., relevance, political commitment/ mandate, benefit/profitability etc.] 35. What is your budget for operation and service delivery last and this year? What sources do you get it from and approximately how much from each source?  Membership fee:  Service turnover:  Government support:  Other sources (specify): 36. What is your strategy to mobilize resources? 37. Does your organization has or has put in place a relevant system/ structure to provide the service(s)? If yes, please describe it. 38. Are the programs’ outcomes [name: capacity, service helped to set up, network…] sustainable? What is sustainable and what is not? If sustain, with any modification? 39. Do you plan to continue doing that (training, service, networking…)? What’s your strategy to make it sustainable? Why/ why not? 40. Do you think your staff/ managers have been adequately prepared to continue without USAID support? Please score your level of satisfaction from 1 to 4, 1 being the lowest and 4 being the highest level. 41. Do you have or have you put in place a relevant system/ structure to continue what you are doing in persons with disabilities-related employment and activities? 42. Please list the three main challenges you have to continue. Evaluation of USAID Vietnam Disabilities Programming 134 INTERVIEW GUIDE FOR PROVINCIAL DEPARTMENT OF EDUCATION AND TRAINING BACKGROUND 1. What support did you receive in inclusive and special education for children with disabilities from CRS/VNAH/DAI? 2. Number and percentage of children with disabilities in your province/city at  Nursery:  Primary:  Junior Secondary:  Secondary: IMPLEMENTATION OF EDUCATION LAWS AND POLICIES 3. What disability education and policies, particularly those for children with disabilities have you implemented? Any advantages and disadvantages? Comments 4. What is the specific support did USAID provide through CRS, DAI and VNAH for implementation (e.g., TA, finance to organize workshops, models)? 5. How to transfer to the next level of education? How to enroll on junior secondary school? 6. Vocational training and career guidance for children with disabilities at junior secondary and secondary education level (regular and special education schools) INCLUSIVE EDUCATION 7. Support for inclusive education training for in-service teachers: Please describe (e.g., contents, length of training, who does it target, follow- up support). 8. How is it implemented at school (e.g., Reduced number of non-persons with disabilities students, allowance for teachers in charge of the class, technical support and performance evaluation)? CROSSCUTTING ISSUES: 9. Is the school accessible in regular schools? What is the status of accessibility being included in the National School Standards? 10. children with disabilities with mobility impairment accepted at regular school (CRS advocated policy/ decision in Quang Nam or central) ONLINE CURRICULUM 11. How can children be certified at end of level if only 2 subjects? Evaluation of USAID Vietnam Disabilities Programming 135 12. Home-based education for children with disabilities with severe disabilities: How often do teachers come to the home? How to transfer from home-based education to in-school? How to get certified at end of level if only 2 subjects? CAPACITY BUILDING 13. Did you attend any training course on education for students with disabilities supported by CRS, DAI, and VNAH? 14. If yes, please describe the course (How long was it? What topics were covered? Did it cover theory? Did it provide opportunities to practice what you learned? Did persons with disabilities present?) 15. OVERALL, how relevant was the training to your work? Not Relevant at All Somewhat Relevant Relevant Very Relevant 1 2 3 4 16. OVERALL, how satisfied were you with the training? Very Dissatisfied Dissatisfied Satisfied Very Satisfied 1 2 3 4 17. How did you benefit from the training? (Ask for examples) 18. Do you have any recommendations or suggestions in how to improve the training? 19. Did you receive any follow-up resources, assistance or support after the training? If yes, please describe it. Was it effective? 20. Do you have any recommendations or suggestions on what additional resources or assistance that you need to be effective at your work? Evaluation of USAID Vietnam Disabilities Programming 136 INTERVIEW GUIDE FOR DIRECTOR OF REHABILITATION HOSPITAL 1. What is the total number of staff and what is their average number of people on their caseload: Average Average Number Type of Staff Number on Caseload Medical doctors _____ _____ Nurses _____ _____ Technicians _____ _____ 2. Did you send staff to training that was funded by USAID through DAI? _____ Yes _____ No (END OF INTERVIEW) 3. If yes, how many staff attended the training? Medical doctors _____ Nurses _____ Technicians _____ 4. How is the staff applying the training to their work? 5. OVERALL, how relevant was the training to the Center’s work? Not Relevant at All Somewhat Relevant Relevant Very Relevant 1 2 3 4 6. OVERALL, how satisfied are you with the training? Very Dissatisfied Dissatisfied Satisfied Very Satisfied 1 2 3 4 Evaluation of USAID Vietnam Disabilities Programming 137 7. How did your staff benefit from the training? (Ask for examples) 8. How did the service improve the service quality? 9. How has the staff integrated what they learned into their day-to-day work? (Ask for examples) 10. What type of follow-up support or assistance did the staff receive from DAI and DOLISA 11. What additional trainings related to disability issues does your staff need? 12. What additional assistance or support do you need to improve services? Evaluation of USAID Vietnam Disabilities Programming 138 INTERVIEW GUIDE FOR MEDICAL PERSONNEL WHO RECEIVED TRAINING BACKGROUND INFORMATION 1. Name: 2. Job Title: 3. Job Function: 4. Do you work with persons with disabilities? If no, why? _____ YES GO TO Q5 _____ NO WHY DID YOU TAKE THE CASE MANAGEMENT/SOCIAL WORK COURSE? 5. Did you take any training related to persons with disabilities? _____ YES GO TO Q5.a. _____ NO END OF INTERVIEW 5.a. If yes, Date of Course (MM/YYY): ____________________ 5.b. If yes, why did you take the course? 6. What did you like about the course (e.g., length, topics, instructor, usefulness)? 7. What did you dislike about the course (e.g., length, topics, instructor, usefulness)? 8. How can the course be improved? 9. What skills did you learn? (If the person says that she/he did not learn anything, ask why? END OF INTERVIEW) 10. For each skill: Ask the following questions: Evaluation of USAID Vietnam Disabilities Programming 139 SKILL ONE: 3. 1. How often do you use this skill on the job (Circle one) e. I don’t use the skill (GO TO Q2) f. Sometimes (1 to 2 times a week) (GO TO Q3) g. Regularly (At least once a day) (GO TO Q3) h. Frequently (Several times a day) (GO TO Q3) 4. 2. If skill is not used, why? (Circle one responses): d. It is not part of my job (Go to the next SKILL) e. It is too difficult (GO TO Q3) f. I forgot how to do it (GO TO Q3) 3. How would you rate your ability to employ this skill? (Circle the most appropriate responses): d. I am unable to practice it (GO TO Q4) e. I mastered the skill (GO TO Q5) f. I am an expert of the skill (GO TO Q5) 4. If you are unable to perform the skill, why? (Circle one) d. It is too difficult (Directions: Go to next SKILL) Evaluation of USAID Vietnam Disabilities Programming 140 e. No one else uses it (Directions: Go to next SKILL) f. I forgot how to do it (Directions: Go to next) SKILL 5. Which statement is true? (Circle the most appropriate response): d. I was able to perform this skill immediately after the training. e. It took me two weeks of practicing before I learned f. It took me more than two weeks of practicing before I learned 6. Which statement is true: (Circle one): c. The skill is performed in the manner described during the training. d. I revised the skill to meet my clients’ needs. How? 11. How has the training benefited your ability to provide services to persons with disabilities? 12. How has the training benefited the lives of persons with disabilities? 13. What have been the challenges in applying the training to your work? 14. What additional resources or assistance do you need to overcome these challenges? 15. Since the training, what type of follow-up assistance and support have you received from the trainers/DAI? (If No, END OF INTERVIEW) 16. How satisfied are you with the assistance or support? Very Dissatisfied Dissatisfied Satisfied Very Satisfied 1 2 3 4 17. If very dissatisfied or dissatisfied, how would you improve it? Evaluation of USAID Vietnam Disabilities Programming 141 INTERVIEW GUIDE FOR THE COMPANIES EMPLOYING persons with disabilities List name and position of the respondents who attended the interview Name Position 1. USAID supported a variety of activities through its Implementing Partners VNAH, CRS or DAI, to improve employment opportunities for persons with disabilities (persons with disabilities). I am going to read a list of services and activities. As I read one, tell me if your company used the service or participated in the activity. [Direction: Tick (x) the service/ activity when the person says, “Yes.”] Sectors/Types of Activities Yes? LEGAL AND POLICY FRAMEWORK Type of Activity: EMPLOYMENT/LIVELIHOOD SECTOR Direct Services Type of Activity: Capacity Building/ Institutional Strengthening Type of Activity: Other Service and Activity: 2. Please rate your level of satisfaction with the services/ activities that you used/ participated Type of Service/ Activity [Insert answers from Q1] On a scale from 1 to 4, how satisfied are you with USAID’s assistance to support you in this activity? (Directions: Circle one response) Very Dissatisfied Dissatisfied Satisfied Very Satisfied 1 2 3 4 3. Did you pay/ make a contribution in the services/ activities you participated in? If yes, can you please describe how much you paid or what you contributed? 4. What specific benefits did your company get from the USAID-supported services/ activities? Please describe in details. [PROMPT: For example: Evaluation of USAID Vietnam Disabilities Programming 142 Company labor force [For example, Size and turnover rate, persons with disabilities workers’ attitude/ loyalty, Non-persons with disabilities workers’ attitude/ loyalty, Corporate culture] Business performance [For example, Productivity, Sales/ level of order, Profit, Supplier network, Company reputation] Compliance [For example, Company corporate social responsibilities (CSR) policy, National laws and policies, Buyer’s labor standards, International labor standards] Other benefits if any (specify): 5. What specific benefits did persons with disabilities get from the USAID-supported services/ activities? Please describe in details. [PROMPT: Number of new persons with disabilities employees, income, working conditions, participation in company’s social activities] 6. How satisfied are you that your company’s opinions have been addressed by the current disability laws/ policies? Please score your level of satisfaction from 1 to 4, 1 being the lowest and 4 being the highest. [Directions: Circle one response.] Very Dissatisfied Dissatisfied Satisfied Very Satisfied 1 2 3 4 7. What benefits or opportunities does your company receive from the current disability laws and policies? [List up to three benefits] 8. What benefits or opportunities do you think persons with disabilities receive from the current disability laws and policies? 9. What benefits or opportunities do you think girls and women with disabilities receive from the current disability laws and policies? 10. Do you have plans to employ more persons with disabilities? Why/ why not? 11. What resources (finance, human resource, etc.) will you use to employ persons with disabilities and accommodate them at the workplace (including recruitment, training, payroll, other services/ support for swift inclusion of persons with disabilities workers)? 12. Please identify three main challenges you still face to continue employing persons with disabilities? (e.g., capacity building, resources, company’s policy, legal framework 13. What has to be done to overcome these challenges? 14. Please identify three main challenges persons with disabilities continue to face in gaining decent employment? (e.g., self/public awareness, capacity building, resources, legal framework) 15. What has to be done to overcome these challenges? 16. Please identify three main challenges girls and women with disabilities continue to face in gaining decent employment? (e.g., self/public awareness, capacity building, resources, legal framework) 17. What has to be done to overcome these challenges? Evaluation of USAID Vietnam Disabilities Programming 143 INTERVIEW GUIDE FOR DISTRICT BUREAU OF LABOR, INVALIDS AND SOCIAL AFAIRS NAME OF PROVINCE: List name and position of the members who attended the interview Name Position 1. Please describe activities/services that DOLISA provides to persons with disabilities? 2. Please tell me if USAID supported DOLISA through its Implementing Partners VNAH or DAI in the legal/policy framework sector. [Directions: Read the name of the activity. Tick the activity (x) if the person says, “Yes.” Then ask what type of assistance they received and write their answers in the space provided.] LEGAL/ POLICY SECTOR Yes? Activity: Development (at provincial- & district- levels?) (Directions: If applicable, indicate the circular/decree) Type of Assistance: Activity: Implementation (at provincial- & district- levels?) (Directions: If applicable, indicate the circular/decree) Type of Assistance: Type of Assistance: Activity: Coordination (at provincial- & district- levels?) (Directions: If applicable, indicate the circular/decree) Type of Assistance: Activity: Capacity Building (Directions: If applicable, indicate the circular/decree) Type of Assistance: 3. Please tell me how DOLISA contributed to developing or implementing the laws or policies? 4. What are the 3 main challenges they face in policy development? 5. What needs to be done to overcome these challenges? 6. Please rate your level of satisfaction with the USAID assistance that you received for each activity. Evaluation of USAID Vietnam Disabilities Programming 144 Activity & Type of Assistance (Directions: Insert answers from Q1) On a scale from 1 to 4, how satisfied are you with USAID’s assistance to support you in this activity? (Directions: Circle one response) Very Dissatisfied Dissatisfied Satisfied Very Satisfied 1 2 3 4 7. Please tell and me if USAID supported DOLISA through its Implementing Partners VNAH or DAI in the social inclusion sector. [Directions: Read the name of the activity. Tick the activity (x) if the person says, “Yes.” Then ask what type of assistance they received and write their answers in the space provided.] SOCIAL INCLUSION SECTOR Yes? Activity: Case Management Type of Assistance: Activity: Public Awareness Type of Assistance: Activity: Capacity Building Type of Assistance: Training of social workers/case managers/ supervisors 8. How has case management benefited persons with disabilities? 9. How has the training of case manager benefited persons with disabilities? 10. What are the three challenges in implementing case management? 11. What needs to be done to overcome these challenges? 12. What type of support/assistance do you need from MOLISA to implement case management? 13. What recommendations do you have to improve and expand case management? Evaluation of USAID Vietnam Disabilities Programming 145 INTERVIEW GUIDE FOR CASE MANAGEMENT SUPERVISORS 1. Did you attend any USAID training on case management and people disabilities? (PROMPT: DAI implemented the training) 2. If yes, please describe the course (How long was it? What topics were covered? Did it cover theory? Did it provide opportunities to practice what you learned? Did persons with disabilities present?) 3. OVERALL, how relevant was the training to your work? Not Relevant at All Somewhat Relevant Relevant Very Relevant 1 2 3 4 4. OVERALL, how satisfied were you with the training? Very Dissatisfied Dissatisfied Satisfied Very Satisfied 1 2 3 4 5. How did you benefit from the training? (Ask for examples) 6. Do you have any recommendations or suggestions in how to improve the training? 7. Did you receive any follow-up resources, assistance or support from DAI after the training? If yes, please describe it. Was it effective? 8. Do you have any recommendations or suggestions on what additional resources or assistance that you need to be effective at your work? 9. What are your roles and responsibilities in your agency and your case management project? 10. How do you supervise them? 11. Did they attend any case management training? 12. How did the case managers benefit from the training? (Ask for examples) 13. How did the training help the case manager in doing their work? 14. Did you receive any follow-up resources, assistance or support from DAI after the training? If yes, please describe it. Was it effective? 15. Do you have any recommendations or suggestions on what additional resources or assistance the case managers need to be effective at their work? 16. What are three main challenges in implementing case management in the province? 17. How do they collaborate with other sectors (health and education)? What challenges do you face? How do they use the Information System for clients with disabilities? 18. What are your plans to continue case management after the project ends? 19. After the project ends, how will individual case managers maintain their skills and knowledge? Evaluation of USAID Vietnam Disabilities Programming 146 INTERVIEW GUIDE FOR PROFESSIONALS WHO RECEIVED CASE MANAGERMENT/SOCIAL WORK TRAINING BACKGROUND INFORMATION 1. Name: 2. Job Title: 3. Job Function: 4. Do you work with persons with disabilities? If no, why? _____ YES GO TO Q5 _____ NO WHY DID YOU TAKE THE CASE MANAGEMENT/SOCIAL WORK COURSE? 5. Did you take the Case Management/Social Work Course: _____ YES GO TO Q5.a. _____ NO END OF INTERVIEW 5.a. If yes, Date of Course (MM/YYY): ____________________ 5.b. If yes, why did you take the course? 6. What did you like about the course (e.g., length, topics, instructor, usefulness)? 7. What did you dislike about the course (e.g., length, topics, instructor, usefulness)? 8. How can the course be improved? 9. What skills did you learn? (If the person says that she/he did not learn anything, ask why? END OF INTERVIEW) 10. For each skill: Ask the following questions: Evaluation of USAID Vietnam Disabilities Programming 147 SKILL ONE: 5. 1. How often do you use this skill on the job (Circle one) i. I don’t use the skill (GO TO Q2) j. Sometimes (1 to 2 times a week) (GO TO Q3) k. Regularly (At least once a day) (GO TO Q3) l. Frequently (Several times a day) (GO TO Q3) 6. 2. If skill is not used, why? (Circle one responses): g. It is not part of my job (Go to the next SKILL) h. It is too difficult (GO TO Q3) i. I forgot how to do it (GO TO Q3) 3. How would you rate your ability to employ this skill? (Circle the most appropriate responses): g. I am unable to practice it (GO TO Q4) h. I mastered the skill (GO TO Q5) i. I am an expert of the skill (GO TO Q5) 4. If you are unable to perform the skill, why? (Circle one) g. It is too difficult (Directions: Go to next SKILL) Evaluation of USAID Vietnam Disabilities Programming 148 h. No one else uses it (Directions: Go to next SKILL) i. I forgot how to do it (Directions: Go to next) SKILL 5. Which statement is true? (Circle the most appropriate response): g. I was able to perform this skill immediately after the training. h. It took me two weeks of practicing before I learned i. It took me more than two weeks of practicing before I learned 6. Which statement is true: (Circle one): e. The skill is performed in the manner described during the training. f. I revised the skill to meet my clients’ needs. How? 11. How has case management benefited your ability to provide services to persons with disabilities? 12. How has case management benefited the lives of persons with disabilities? 13. What have been the challenges in trying to manage services across departments? 14. What additional resources or assistance do you need to overcome these challenges? 15. Since the training, what type of follow-up assistance and support have you received from the trainers/DAI? (If No, END OF INTERVIEW) 16. How satisfied are you with the assistance or support? 17. If very dissatisfied or dissatisfied, how would you improve it? Evaluation of USAID Vietnam Disabilities Programming 149 INTERVIEW GUIDE FOR SCHOOL LEADERS List name and position of the members who attended the interview Name Position 1. Please describe the school to me. What grades, number of students with and without disabilities, number of teachers, etc. 2. Why did USAID pick this school? 3. VNAH, CRS, and DAI are Implementing Partners of USAID. Do you or your teachers participate in any trainings (e.g., case management or social inclusion), receive assistance or support for the school, or receive devices or equipment for the children from any of these agencies? Please list below 4. For each training or type of assistance and support, how satisfied are you? Type of Service/Program Very Dissatisfied Dissatisfied Satisfied Very Satisfied 1 2 3 4 5. How have the children benefited from the services/programs? 6. How does case management work at the school? 7. What challenges do you face in implementing case management 8. How does inclusive education work at the school? What challenges do you face in implementing inclusive education? Evaluation of USAID Vietnam Disabilities Programming 150 INTERVIEW GUIDE FOR SPECIAL EDUCATION TEACHER WHO RECEIVED TRAINING OR RECEIVED ASSISTANCE IN THE CLASSROOM TRAINING 1. Name: 2. Job title: 3a. What grade do you teach? 3b. How many children are in your classroom? 3c. How many children have disabilities? 4. Did you take the course on how to work with children with disabilities? _____ YES GO TO Q5 _____ NO GO TO THE NEXT SECTION ON ASSISTANCE IN THE CLASSROOM 5.a. If yes, Date of Course (MM/YYY): 5.b. If yes, why did you take the course? 6. What did you like about the course (e.g., length, topics, instructor, usefulness)? 7. What did you dislike about the course (e.g., length, topics, instructor, usefulness)? 8. How can the course be improved? 9. What skills did you learn? (If the person says that she/he did not learn anything, ask why? END OF INTERVIEW). 10. For each skill: Ask the following questions Evaluation of USAID Vietnam Disabilities Programming 151 SKILL ONE: 7. 1. How often do you use this skill on the job (Circle one) m. I don’t use the skill (GO TO Q2) n. Sometimes (1 to 2 times a week) (GO TO Q3) o. Regularly (At least once a day) (GO TO Q3) p. Frequently (Several times a day) (GO TO Q3) 8. 2. If skill is not used, why? (Circle one responses): j. It is not part of my job (Go to the next SKILL) k. It is too difficult (GO TO Q3) l. I forgot how to do it (GO TO Q3) 3. How would you rate your ability to employ this skill? (Circle the most appropriate responses): j. I am unable to practice it (GO TO Q4) k. I mastered the skill (GO TO Q5) l. I am an expert of the skill (GO TO Q5) 4. If you are unable to perform the skill, why? (Circle one) j. It is too difficult (Directions: Go to next SKILL) Evaluation of USAID Vietnam Disabilities Programming 152 k. No one else uses it (Directions: Go to next SKILL) l. I forgot how to do it (Directions: Go to next) SKILL 7. Which statement is true? (Circle the most appropriate response): j. I was able to perform this skill immediately after the training. k. It took me two weeks of practicing before I learned l. It took me more than two weeks of practicing before I learned 8. Which statement is true: (Circle one): g. The skill is performed in the manner described during the training. h. I revised the skill to meet my clients’ needs. How? 11. How has the additional training benefited your ability to provide services to persons with disabilities? 12. How has the additional training benefited the lives of persons with disabilities? 13. What have been the challenges in trying to implement what you learned? 14. What additional resources or assistance do you need to overcome these challenges? 15. Since the training, what type of follow-up assistance and support have you received from the trainers/DAI? (If No, END OF INTERVIEW) 16. How satisfied are you with the assistance or support? Very Dissatisfied Dissatisfied Satisfied Very Satisfied 1 2 3 4 17. If very dissatisfied or dissatisfied, how would you improve it? Evaluation of USAID Vietnam Disabilities Programming 153 GO TO THE NEXT SECTION B. TEACHERS WHO RECEIVED ASSISTANCE IN THE CLASSROOM 18. What type of USAID assistance and support does USAID provide through its Implementing Partners, CRS, DAI, or VNHA? 19. Please rate your level of satisfaction with the USAID assistance that you received. Type of Assistance (Directions: Insert answers from Q1) On a scale from 1 to 4, how satisfied are you with USAID’s assistance to support you in this activity? (Directions: Circle one response) Very Dissatisfied Dissatisfied Satisfied Very Satisfied 1 2 3 4 20. How has inclusive education benefited the children with and without disabilities? 21. How has inclusive education benefited you? 22. What additional assistance do they need? Evaluation of USAID Vietnam Disabilities Programming 154 INTERVIEW GUIDE FOR ADULTS WITH DISABILITIES OR FAMILY MEMBERS OF CHILDREN 1. What type of services/programs do you receive through USAID’s Implementing Partners, VNAH, DAI, or CRS? 2. For each service, ask: On a scale from 1 to 4, how satisfied are you with the service? Very Dissatisfied Dissatisfied Satisfied Very Satisfied 1 2 3 4 3. Why are you [state their level of satisfaction] 4. Please describe the step you went through to receive the service(s)/program(s). 5. Did you face any challenges while getting the service(s)/program(s)?  From a government agency (e.g., Central, provincial, district, People’s Committee)  From the service provider?  From USAID’s Implementing Partner?  Community?  Family members? 6. Were you able to overcome the challenges? Why or Why not? 7. How long did it take you to receive the service or enter the program? [e.g., Number of months or the person says that she or he is still waiting] 8. Do you believe the service or program meets your needs? If yes, why. If not, why not? 9. Do you believe the staff that provides the service or implement the program understands persons with disabilities? Why or why not? 10. What additional training, resources, or support does the staff need to be more effective in their work? 11. What changed for you, as the result of each service/program? Please describe. 12. What specific benefits did you receive? Please describe. 13. What specific benefits did your family receive? Please describe. 14. What specific benefits does the community receive because you receive the service or participate in the program? Please describe 15. What do you do differently as the result of the service or program? 16. What do your family members do differently as the result of the service or program? 17. What recommendations or suggestions do you have to improve the process of obtaining the service or being accepted into the program? 18. What recommendations or suggestions do you have to improve the service or program? 19. What additional services or programs does the government need to support to continue to assist and support persons with disabilities and their families? 20. What challenges do persons with disabilities still face? 21. What needs to be done to overcome these challenges? Evaluation of USAID Vietnam Disabilities Programming 155 ANNEX 8: DESK REVIEW AND INDEPENDENT DATA SOURCE REVIEW FORM Name of Team Member: ____________ Date of Review: ______________ Document/Data Source Citation: _______________________________________________ Directions: Insert information from document/data source into the relevant evaluation question. Make sure to include the page number where the information is located. 1. To what extent and in what ways have the programs improved the lives and opportunities for persons with disabilities in Vietnam? Have the planned benefits been delivered and received as perceived by beneficiaries? 2. To what extent has the programming assistance helped to develop the Government of Vietnam’s (GVN) legal and policy environment, public awareness, and support for disability services? 3. In terms of improving service delivery, has disabilities programming across projects provided both an enabling environment and a supporting systems-approach in technical assistance to strengthen critical institutional capacities? 4. Are the programs’ outcomes sustainable? What challenges in terms of sustainability remain? Evaluation of USAID Vietnam Disabilities Programming 156 ANNEX 9: SOURCES OF DATA FOR NUMBERS OF DIRECT BENEFICIARIES 1 Disability Support Program, FY 2013 Q4 (July 1 – September 30, 2013), and Annual Progress Report Prepared by DAI 2 Persons with Disabilities Support Program, FY 2014 Q4 (July 1 – September 30, 2014) and Annual Progress Report Prepared by DAI 3 PDSP Summary for USAID Evaluation Team 3.18.15 Prepared by DAI 4 Inclusion of Vietnamese with Disabilities (IVWD), Annual Report # 1, (October 2005 – September 2006) Prepared by CRS 5 Inclusion of Vietnamese with Disabilities (IVWD), Progress Report # 12, Submitted January 29, 2010 Prepared by CRS 6 Inclusion of Vietnamese with Disabilities (IVWD), Progress Report # 15, Submitted October 21, 2010 Prepared by CRS 7 Inclusion of Vietnamese with Disabilities, Annual Performance Report for USAID Vietnam, Reporting period: October 2010-September 2011 Prepared by CRS 8 Inclusion of Vietnamese with Disabilities, Annual Performance Report for USAID Vietnam, Reporting period: October 1st, 2010 to September 30th, 2011 Prepared by CRS 9 Inclusion of Vietnamese with Disabilities, FY 2013 Annual Performance Report, October 1st, 2012 to September 30th, 2013 Prepared by CRS 10 Inclusion of Vietnamese with Disabilities, Annual Report, October 1st, 2013 to September 30th, 2014 Prepared by CRS 11 Inclusion of Vietnamese with Disabilities, Annual Progress Report for USAID Viet Nam, Reporting period: October 01,2005 - September 30, 2006 Prepared by VNAH 12 Inclusion of Vietnamese with Disabilities, Semi-Annual Performance Report to U.S. Agency for International Development (USAID), Reporting Period: 1 October 2006 - 30 March 2007 Prepared by VNAH 13 Inclusion of Vietnamese with Disabilities, Semi-Annual Performance Report to U.S. Agency for International Development (USAID), Reporting Period: 1 April 2007 - 30 September 2007 Prepared by VNAH 14 Inclusion of Vietnamese with Disabilities, Semi-Annual Performance Report to U.S. Agency for International Development (USAID), Reporting Period: 1 October 2007 - 31 March 2008 Prepared by VNAH 15 Inclusion of Vietnamese with Disabilities, Performance report, Reporting period: Oct 1, 2008 - Sep 30, 2009 Prepared by VNAH Evaluation of USAID Vietnam Disabilities Programming 157 16 Inclusion of Vietnamese with Disabilities, Performance report, Reporting period: Oct 1, 2009 - Sep 30, 2010 Prepared by VNAH 17 Inclusion of Vietnamese with Disabilities, Annual Program Performance Report for USADI Vietnam, Reporting period: October 2010 - September 2011 Prepared by VNAH 18 Inclusion of Vietnamese with Disabilities, Annual Program Performance Report for USADI Vietnam, Reporting period: October 2011 - September 2012 Prepared by VNAH 19 Inclusion of the Vietnamese with Disabilities - IVWD, FY 2013 Progress Report, 1 Oct 2012 - 30 Sep 2013 Prepared by VNAH 20 Inclusion of the Vietnamese with Disabilities - IVWD, FY 2013 Progress Report, 1 Oct 2013 - 30 Sep 2014 Prepared by VNAH Evaluation of USAID Vietnam Disabilities Programming 158 ANNEX 10: PERFORMANCE INDICATORS: TARGETED AND ACHIEVED INCLUSION OF VIETNAMESE WITH DISABILITIES (IVD) – VNAH Note: - Green = >110% of target, Blue = within 90-110% of target, Red = <90% of target Abbreviation: - SP: Service providers - persons with disabilities: Persons with disabilities Intermediate result (IR) Major activities/events 2012 2013 2014 SP persons with disabilities SP persons with disabilities SP persons with disabilities Target Result Target Result Target Result Target Result Target Result Target Result Objective 1: To develop a National Strategy for Integrated and Comprehensive Disability I.R.1.1. A policy framework for integrated rehabilitation services for persons with disabilities is developed I.R.1.1.1. Action Plan on integrated rehabilitation services is developed 1.1.1.1. a. Conduct a comprehensive assessment of the current national rehabilitation system. N/A N/A N/A N/A N/A 376 N/A 0 0 0 0 0 Evaluation of USAID Vietnam Disabilities Programming 159 Intermediate result (IR) Major activities/events 2012 2013 2014 SP persons with disabilities SP persons with disabilities SP persons with disabilities Target Result Target Result Target Result Target Result Target Result Target Result 1.1.1.1.b . Gender Baseline Survey is conducted N/A N/A N/A N/A N/A 90 N/A 242 N/A N/A N/A N/A 1.1.1.2. Draft the national action plan on rehabilitation (2013-2015 and its vision to 2020) N/A N/A N/A N/A N/A 43 N/A 0 30 286 0 0 1.1.1.4. Adopt the disability information system/software into the CBR program. N/A N/A N/A N/A N/A 715 N/A 6 33 65 0 0 IR 1.1.2. Standard procedure and guidelines on integrated and comprehensive rehabilitation services are developed IR 1.1.2.1. Standard procedures on integrated and comprehensive rehabilitation services are developed 1.1.2.1.1. Develop standard procedures - Circular on Case Management for persons with disabilities and CRB. N/A N/A N/A N/A N/A 152 N/A 0 45 268 0 0 Evaluation of USAID Vietnam Disabilities Programming 160 Intermediate result (IR) Major activities/events 2012 2013 2014 SP persons with disabilities SP persons with disabilities SP persons with disabilities Target Result Target Result Target Result Target Result Target Result Target Result 1.1.2.1.2. Providing training of trainer on standard procedure for relevant service providers and enforcement agencies N/A N/A N/A N/A N/A N/A N/A N/A 0 0 0 0 IR 1.1.2. Standard procedure and guidelines on integrated and comprehensive rehabilitation services are developed IR 1.1.2.2. Vocational Rehabilitation Services guidelines are developed 1.1.2.2.1. Document and disseminate good practices to policy makers and service providers. N/A 287 N/A 86 N/A 0 N/A 0 0 0 0 0 1.1.2.2.2. Provide technical assistance for vocational rehabilitation service demonstration in Hanoi 80 131 30 19 N/A 86 N/A 412 9 105 54 232 Evaluation of USAID Vietnam Disabilities Programming 161 Intermediate result (IR) Major activities/events 2012 2013 2014 SP persons with disabilities SP persons with disabilities SP persons with disabilities Target Result Target Result Target Result Target Result Target Result Target Result 1.1.2.2.3. Develop guidelines on vocational rehabilitation services N/A N/A N/A N/A N/A 37 N/A 4 100 5 50 0 I.R.1.2. Human Resources in Social Work and Vocational Rehabilitation are developed. IR.1.2.1. Curricula and Teaching Methodology for Social Work and Vocational Rehabilitation with persons with disabilities are developed. 1.2.1.1. a. Develop LC and SC curriculum and training programs on social work with persons with disabilities with USSH for universities and ULSA/MOLISA for vocational training colleges. 60 60 N/A N/A N/A 131 N/A 9 60 73 0 N/A Evaluation of USAID Vietnam Disabilities Programming 162 Intermediate result (IR) Major activities/events 2012 2013 2014 SP persons with disabilities SP persons with disabilities SP persons with disabilities Target Result Target Result Target Result Target Result Target Result Target Result 1.2.1.1. b. Develop 03 national training programs on social work with SPD/ MOLISA for frontline workers N/A N/A N/A N/A N/A 178 N/A 8 40 139 0 0 1.2.1.1. c. Circular on case management is developed and approved. N/A N/A N/A N/A N/A 0 N/A 0 0 5 0 0 1.2.1.1 d. Develop a SW w persons with disabilities Concentration Framework for partnering colleges and universities. N/A N/A N/A N/A N/A 9 N/A 0 N/A N/A N/A N/A 1.2.1.2. Incorporate vocational rehabilitation into curriculum of medical schools and training N/A N/A N/A N/A N/A 42 N/A 10 20 8 0 0 Evaluation of USAID Vietnam Disabilities Programming 163 Intermediate result (IR) Major activities/events 2012 2013 2014 SP persons with disabilities SP persons with disabilities SP persons with disabilities Target Result Target Result Target Result Target Result Target Result Target Result programs of Vocational Training Centers I.R.1.2.2. Training for teachers and practitioners on Social Work and Vocational Rehabilitation is conducted 1.2.2.1.a. Provide training to teachers, trainers and service providers on Social Work with Disability N/A N/A N/A N/A N/A 24 N/A 1 40 18 0 66 1.2.2.1.b. Develop a model of professional field education for social work students (add￾on activity) N/A N/A N/A N/A N/A 2 N/A 0 N/A 17 N/A 57 1.2.2.2. Provide training to teachers, trainers and service providers on vocational N/A N/A N/A N/A N/A 1 N/A 0 315 473 33 23 Evaluation of USAID Vietnam Disabilities Programming 164 Intermediate result (IR) Major activities/events 2012 2013 2014 SP persons with disabilities SP persons with disabilities SP persons with disabilities Target Result Target Result Target Result Target Result Target Result Target Result rehabilitation. IR.1.3. Enforcement Mechanism is developed IR.1.3.1. A Penalty Decree is developed to ensure compliance of the Law on People with Disabilities 1.3.1.1. Develop a Penalty Decree N/A N/A N/A N/A N/A 168 N/A 29 0 0 0 0 1.3.1.2. Conduct training on the implementation guidelines of the Decree to relevant service providers and enforcement agencies N/A N/A N/A N/A N/A 0 N/A 0 80 87 0 0 1.3.1.3. Promote the ratification of the CRPD (add-on activity) N/A N/A N/A N/A N/A 49 N/A 2 N/A 32 N/A 0 IR.1.3. Enforcement Mechanism is developed IR1.3.2. Monitoring and Evaluation (M&E) by NCCD and the Evaluation of USAID Vietnam Disabilities Programming 165 Intermediate result (IR) Major activities/events 2012 2013 2014 SP persons with disabilities SP persons with disabilities SP persons with disabilities Target Result Target Result Target Result Target Result Target Result Target Result Vietnamese Federation on Disability is designed and implemented. 1.3.2.1. Develop an overall M&E framework and M&E guidelines for NCCD and VFD N/A N/A N/A N/A N/A 38 N/A 1 55 133 0 0 1.3.2.2. Provide training and technical assistance for NCCD and VFD to implement an M&E system. N/A N/A N/A N/A N/A 0 N/A 0 15 29 0 0 Objective 2: To Increase Inclusive Employment for persons with disabilities through inclusive business development IR2.1. The Blue Ribbon Employer Council (BREC) is Strengthened and Financially Secure to promote inclusive employment and inclusive business. 2.1.1. Expand and strengthen BREC through restructuring 50 54 N/A N/A N/A 257 N/A 23 30 178 0 12 Evaluation of USAID Vietnam Disabilities Programming 166 Intermediate result (IR) Major activities/events 2012 2013 2014 SP persons with disabilities SP persons with disabilities SP persons with disabilities Target Result Target Result Target Result Target Result Target Result Target Result 2.1.2. Generate revenue and promote BREC and its value added services to members N/A N/A N/A N/A N/A 37 N/A 7 0 5 0 1 IR2.2. Employment for persons with disabilities is generated through BREC activities and its partnership with Employment Service Centers (ESC) at project sites 2.2.1. BREC work with the ESC in Hanoi, Danang, and Dong Nai and HCMC to support persons with disabilities in obtaining jobs, and employers to find qualified job seekers. N/A N/A 200 78 N/A 847 N/A 1366 35 60 500 574 IR2.3. Businesses have increased understanding about inclusive business and adopt inclusive business practices. Evaluation of USAID Vietnam Disabilities Programming 167 Intermediate result (IR) Major activities/events 2012 2013 2014 SP persons with disabilities SP persons with disabilities SP persons with disabilities Target Result Target Result Target Result Target Result Target Result Target Result 2.3.1. To develop and implement guidelines to promote inclusive business. N/A N/A N/A N/A N/A 381 N/A 167 0 5 0 1 Evaluation of USAID Vietnam Disabilities Programming 168 INCLUSION OF VIETNAMESE WITH DISABILITIES (IVD) - CRS Note: - Green = >110% of target, Blue = within 90-110% of target, Red = <90% of target - CRS’s M&E plan is from performance reports. - For 2012 only, the performance report also include a detailed breakout of indicators Indicator 2012 2013 2014 Target Result Target Result Target Result Number of vulnerable people benefiting from USG-supported social services 250 205 270 282 655 197 Number of service providers trained who serve vulnerable persons 20 20 50 70 310 120 Number of USG-assisted organizations and/or service delivery systems strengthened who serve vulnerable populations 15 11 16 33 14 40 % of persons with disabilities have increased agreement with the concept that both men and women should have equal access to social-economic opportunities N/A N/A 45% 47.16% 45% N/A Number of ITTP students who have received GBV awareness training N/A N/A 200 113 350 167 Number of guidelines/guidance on ICT application approved by MOET N/A N/A N/A N/A 1 1 Percentage of ITTP graduates have jobs or paid internship 12 months after graduation N/A N/A N/A 32.35% N/A 37.23% Evaluation of USAID Vietnam Disabilities Programming 169 2012 Detailed Breakout of CRS Indicators Intermediate Result (IR) Indicator (s) 2012 Target Result IR 1.1: YWD benefit from a package of curricular and extra￾curricular activities that meets market needs implemented by ITTP partners. % of ITTP students demonstrating improved soft skills 80% 100% % of students passing final examination; data disaggregated by sex 90% 85% IR 1.2: ITTP partners in Hanoi and HCMC demonstrate sustainability. # of YWD enrolled in partner schools 12 months and 24 months after phasing out direct financial support; data disaggregated by sex N/A N/A IR 2.1: MOET policy makers are committed to integrate ICT into guidelines for implementation. # of action plans for piloting ICT solutions are drafted by MOET. 04 N/A IR 2.2: Effective assistive technologies that remove barriers for secondary and higher education for SWDs are developed and identified. # of guidelines are drafted by MOET (video lectures and text-to-speech) N/A N/A # of sets of assistive technologies DVDs and users' manuals used at piloted schools N/A N/A IR 2.3: ICT-based distance learning for children with disabilities with severe disabilities is piloted and documented by MOET. Model of distance learning is developed and put into the guidance. N/A N/A IR 2.4: SWD, parents, IE teachers, school managers and policy makers benefit from resources made available online. # of people who have downloaded materials from the IE online library. 50 N/A # of hits per month. 100 N/A % of visitors rating the online IE resource center as useful. 60% N/A Evaluation of USAID Vietnam Disabilities Programming 170 Output 1.1.1 ITTP established in Danang. % of trained service providers, including university teachers, managers, and government officials, demonstrate increased knowledge in providing appropriate training services for SWD. 80% 100% # of service providers, including university teachers, managers, and government officials are trained in inclusive education management techniques; data disaggregated by sex 20 20 Output 1.1.2 ITTP students benefit from quality training and support services. # of courses co-designed with businesses. N/A 0 # of students completed long term IT training 190 74 # of students completed from short term IT training 60 27 # of students participated in soft skills training; data disaggregated by sex 250 145 % of students trained in soft skills that demonstrate improved knowledge 80% N/A Output 1.1.3: Peer-support among ITTP graduates and current students is increased through participation in online alumni network. # of online alumni networks and providing peer-support for ITTP students and graduates. 02 02 # of alumni registered to use the alumni network forum; data disaggregated by sex N/A N/A # of new posts in the alumni forum 500 500 % of alumni reporting at least 1 interaction with another ITTP alumni each quarter; data disaggregated by sex 70% N/A Output 1.1.4: ITTP model is documented by ITTP partners and Evaluation of USAID Vietnam Disabilities Programming 171 shared with MOLISA and MOET. # of vocational training centers have incorporated good practices from ITTP model into their own training programs. N/A N/A # of people attended the ITTP model sharing workshop. N/A N/A Output 1.2.1: ITTP partner schools successfully raise funds from alternative sources. # of fundraising plans developed by ITTP partners 03 2 Amount of financial contribution ITTP partners raise from alternative sources in year 2013 and 2014 $160,000 N/A Output 2.1.2: MOET pilot adapted assessments. # of schools participate in piloting of adaptive assessments 04 N/A # of students participating in adaptive assessments pilots; data disaggregated by sex N/A N/A Output 2.2.1: Assistive technologies developed according to specification. # of assistive technologies developed to the standards of specification. 02 N/A Output 2.2.2: Assistive technologies piloted in selected schools. # of students participate in assistive technology pilot; data disaggregated by sex N/A N/A # of teachers trained in teaching assistive technologies; data disaggregated by sex N/A N/A Output 2.3.1: A model for distance learning is adopted from international best-practices. A model of distance learning is drafted based on international best-practices and approved by MOET for pilot. 01 N/A Output 2.3.2: MOET has capacity to implement pilot of distance learning model. # of teachers trained in supporting children with disabilities and parents in distance learning; data disaggregated by sex N/A N/A Evaluation of USAID Vietnam Disabilities Programming 172 # of educational institutions are able to run the pilot model according to technical design. N/A N/A # of education managers outside the pilot sites able and willing to implement the piloted model in their localities. N/A N/A Output 2.3.3: Parents have capacity to support their children to participate in distance learning at home. # of parents have home support schedule for their children learning; data disaggregated by sex N/A N/A # of parents effectively help their children with disabilities to complete the course N/A N/A # of children with disabilities trained in using distance learning portal; data disaggregated by sex N/A N/A # of children with disabilities received mini-laptops; data disaggregated by sex N/A N/A Output 2.4.1: Online IE resource center is developed. # of documents digitized and shared in the online IR resource center 50 N/A Output 2.4.2: Plan for maintaining online IE resource center is developed. MOET has a plan to maintain the online IE resource center after project ends N/A N/A Evaluation of USAID Vietnam Disabilities Programming 173 DISABILITY SUPPORT PROGRAM (DSP) - DAI Note: - Green = >110% of target, Blue = within 90-110% of target, Red = <90% of target Indicator/ Frequency of Reporting Description Unit of Measure/ Disaggregation Data Source/ Collection Method Project life targets Achievement To Date % of Project 3- Year Target Achieved to Date Objective Indicators 1. Number of PWDs receiving comprehensive disability support services through ISP/CM referral system (annually) Measures the extent to which PWDs are being reached and provided with comprehensive disability support services through the ISP/CM and referral system. PWDs visited by the joint team and received at least two services per year. The activities under IR 1 contribute to this objective level indicator Number Disaggregated by commune, district, gender of CWD/PWD, and type of disability, and type of services received (general categories) ISP/CM records; DIS data; and CWD/PWD service support records of different departments (DOH, DOLISA, DOET) Total: 1,500 Year 1: 0 Year 2: 0 Year 3: 1,500 Total: 458 Year 1: 0 Year 2: 237 Year 3: 221 30 % 2. Number of additional CWDs (ages 5-17) that are currently enrolled in school with PDSP support (annually) Measures the participation of CWDs in the educational system through IE, SE, or normal classroom instruction programs. The activities under Sub IR 2.2 contribute to this objective level indicator Number and % Disaggregated by commune, district, gender of CWD in and out of school, age group (5-9, 10-17), type of disability, and type of education program Baseline 2011 Danang Disability Survey; annual ISP/CM records, DIS; and DOET baseline data Total: 285 Year 1: 0 Year 2: 135 Year 3: 150 Total: 216 Year 1: 40 Year 2: 156 Year 3: 20 76 % Danang: 186 Dong Nai: 10 Tay Ninh: 20 3. Number of additional PWDs currently employed (including self-employed) Measures the extent to which DSP support for VT and employment service programs for PWDs is improving the Number and % Baseline 2011 Danang Disability Survey; DOLISA baseline data; PDSP project records Total: 781 Year 1: 50 Total: 546 Year 1: 15 70 % Evaluation of USAID Vietnam Disabilities Programming 174 Indicator/ Frequency of Reporting Description Unit of Measure/ Disaggregation Data Source/ Collection Method Project life targets Achievement To Date % of Project 3- Year Target Achieved to Date with PDSP support (annually) labor force participation of PWDs. The activities under Sub IR 2.3 contribute to this objective level indicator. Disaggregated by commune, district, gender of PWD, age group of PWD (18-30, 31—60), type of disability and work sector) on VT and employment placements of PWDs, DIS; Follow-up survey of PWD employment status in Danang City during Year 3. Year 2: 379 Year 3: 352 Year 2: 248 Year 3: 283 4. Number of PWDs who receive any type of direct assistance for disability services by PDSP supports (Annually) Basic measurement of availability and use of disability support services in Danang by PDSP intervention. The activities under IR 1 and IR 2 contribute to this objective level indicator. Number and % (Disaggregated by commune, district, gender, age groups (0- 17, 18+), type of disability, and type of support received) 2011 Danang Disability Survey Total: 8,000 Year 1: 1,275 Year 2: 4,139 Year 3: 2,586 Total: 8,560 Year 1: 1,532 Year 2: 5,078 Year 3: 1,950 107 % Danang: 4,209 Dong Nai: 1,954 Binh Dinh: 2,242 Quang Nam: 135 Tay Ninh: 20 5. Number of DPOs and parents association whose capacity has been strengthened with PDSP support (annually) Capacity strengthening to be measured by groups participation in training in financial and organizational management, advocacy, PWDs rights; vocational training provision, referral￾making, etc. and participation in Danang disability planning and implementation activities with other stakeholders. The activities under IR 2 Number (Disaggregated by district, type of group strengthened (i.e., DPO and parent/caregiver groups) and type of capacity building received) PDSP project M&E reports, DSP training records Project life Target: 11 Da Nang: 7 district DPOs, 1 Parent Association, Dong Nai: 1 DPO, 1 Parent Association Achievement: 9 PDSP supported capacity building for 7 district DPOs &1 Autism Parent Association in Danang, 1 DPO in Binh Dinh 81 % Evaluation of USAID Vietnam Disabilities Programming 175 Indicator/ Frequency of Reporting Description Unit of Measure/ Disaggregation Data Source/ Collection Method Project life targets Achievement To Date % of Project 3- Year Target Achieved to Date contribute to this objective level indicator. Binh Dinh: 1 DPO 6. Number of new GVN policies/plans developed to support disability programs in Danang (Annually) A measure of the extent to which the government is taking concrete action to support disability programs and PWDs rights, through support by the USAID-funded PDSP The activities under IR 1, 2 and 3 contribute to this objective level indicator. Number (Disaggregated by general categories (policy/plan/ budget) PDSP project M&E reports; reviews of government policies, documents, and planning meetings for disability support program; records of DSP technical support to GVN departments (DOLISA, DOH, DOET) Total: 3 Year 1: 0 Year 2: 2 Year 3: 1 Total 3: 1. DOLISA's guideline for DIS and case management system. 2. Decision in 2014 issued by Danang People’s Committee on approval of Disability Action Plan 3..DOET guideline on educational supports for children with disabilities. 100 % Already met the project life target 7. % of PWDs reporting satisfaction with the quality of disability services they receive (Annually) An effort to get a client/beneficiary scorecard on perceived quality and improvements in disability support services over the course of the PDSP project. The activities under IR 1 & 2 contribute to this objective level indicator. % (Disaggregated by district, gender of PWD; and type of disability) Baseline (Year 1) and Follow-up (Year 2 and Year 3) stratified surveys (minimal sample size) Year 1: NA Year 2: 90% Year 3: 95% Year 1: NA Year 2: 93% Year 3: 98% 103 % The summary report is attached Evaluation of USAID Vietnam Disabilities Programming 176 Indicator/ Frequency of Reporting Description Unit of Measure/ Disaggregation Data Source/ Collection Method Project life targets Achievement To Date % of Project 3- Year Target Achieved to Date 8. Number of service providers of all types who have received training from PDSP (with PDSP support) (Quarterly) A measure of the aggregate reach of PDSP supported training and capacity building activities for disability support service providers. The activities under IR 1, 2 & 3 contribute to this objective level indicator. Number (Disaggregated by gender, district, and type of training received) PDSP project training records for all groups receiving DSP￾supported training : DOLISA, DOH and DOET, NGOs, DPOs,, VT/ Employment agencies Total project life: 2,718 Year 1: 518 Year 2: 1,500 Year 3: 700 Total for project life is 2,718 and the updated is 3,264 Year 1: 663 Year 2: 2,156 Year 3: 445 120 % PDSP already reached 3 year target 9. Number of PWDs who receive any type of disability service by PDSP support (Quarterly) Basic measurement of availability and use of disability support services in Danang by PDSP intervention. The activities under IR 1 & 2 contribute to this objective level indicator. Number (Disaggregated by commune, district, gender, age groups (0- 17, 18+), type of disability, and type of support received) DSP M&E report; monthly ISP/CM report, Total project life: 10,000 individuals Year 1: 1,747 Year 2: 5,317 Year 3: 2,936 The accumulative achievement is 9,482 Year 1: 2,115 Year 2: 5,343 Year 3: 2,024 95 % Danang: 4,569 Dong Nai: 2,320 Binh Dinh: 2,458 Quang Nam: 135 10. Number of target population receiving USG-funded Gender￾Based Violence awareness training (Annually) An indicator of the impact of gender-based violence awareness issues included in PDSP-supported disability trainings. The activities under IR 1 & 2 contribute to this objective level indicator. Number (Disaggregated by gender, district, target group (i.e., PWDs, service providers, and/or government disability program managers) PDSP M&E report, DSP project training records for all groups receiving GBV training Total project life: 489 Year 1: 0 Year 2: 289 Year 3: 200 The updated result is 246 Year 1: 0 Year 2: 211 Year 3: 35 50 % More GBV/GE trainings for social workers and PWDs will be conducted in Evaluation of USAID Vietnam Disabilities Programming 177 Indicator/ Frequency of Reporting Description Unit of Measure/ Disaggregation Data Source/ Collection Method Project life targets Achievement To Date % of Project 3- Year Target Achieved to Date next quarter. 11. 11. % improvement in knowledge of key concepts and topics of training on social work/case management for case managers, inclusive education for teachers, rehabilitation medicine for doctors and PTs, and PWD daily care for care givers (Annually) Basic measurement of knowledge improvement of participants after training. The activities under IR 2 contribute to this objective level indicator. % (Disaggregated by district, gender; and type of service providers) Data Source: Pre￾training/Post-training test of knowledge of key concepts and topics of training. 90% Percentage of knowledge for following specific trainings: case management for case managers/supervisors (68% - 93%), Autism spectrum (64% - 82%), early detection for case managers (32%-96%). The average score for post-test is 90.3% 100% 12.% of follow-up sample of trainees on case management/social work, inclusive education and medical rehabilitation who retain knowledge of key concepts and topics of training after six months following the training (Annually) Measurement of PDSP trainees’ knowledge retained six months after PDSP training. The activities under IR 2 contribute to this objective level indicator. % (Disaggregated by district, gender; and type of service providers) Follow-up survey for a random sub-sample of trainees, using same questionnaire as in the pre-training post training test, with additional follow-up questions Year 1 & 2: TBD Year 3: 70% 90% for case management training and 72,7% for autism training (Attached are summary reports for training follow up assessment). The average % for knowledge retention is 81% 116 % PDSP will conduct other training follow up (pediatric training for doctors and PTs; and social worker training for case managers) in next quarters. Evaluation of USAID Vietnam Disabilities Programming 178 Indicator/ Frequency of Reporting Description Unit of Measure/ Disaggregation Data Source/ Collection Method Project life targets Achievement To Date % of Project 3- Year Target Achieved to Date  13. % of PDSP trainees on case management/social work, inclusive education and medical rehabilitation currently using the information and/or skills learned in the training in their current job (six months after the training was completed)  (Annually) Measurement of PDSP trainees’ use of knowledge & skills that they learned from PDSP in their current job. The activities under IR 2 contribute to this objective level indicator. % (Disaggregated by district, gender; and type of service providers) Follow-up survey for a random sub-sample of trainees, using same questionnaire as in the pre-training post training test, with additional follow-up questions and observe their skills/actual performance. Year 3: 80% (TBD in the previous plan) 93% for case management trainees and 75% (doctors) and 63% (PTs) for autism training. The average percentage of current use of the knowledge is 77% 96 % PDSP will conduct other training follow up (pediatric training for doctors and PTs; social worker training for case managers) in next quarters.  14. Number of disability services for PWDs as a result of case management and referral system (Annually) Measures the numbers of services PWDs receive through the case management and referral mechanism. The activities under IR 1 & 2 contribute to this objective level indicator. Number PDSP M&E report; monthly ISP/CM report, Total project life: 2,300 Year 1: 0 Year 2: 300 Year 3: 2,000 Total: 1,411 Year 1: 0 Year 2: 554 Year 3: 857 61 % (1,235 in Da Nang and 176 in Dong Nai) 15. Number of USG assisted organizations or service delivery systems strengthened  who serve vulnerable populations  (Annually) This indicator measures the number of local organizations whose capacity or service delivery will be strengthened by PDSP support. The organizational capacity is defined in terms of the quantity and quality of support services they deliver to persons with disabilities. The activities under IR 1, 2 & 3 contribute to this objective level indicator. Number (Disaggregated by type of organization) PDSP report Total: 34 Year 1: NA Year 2: 29 Year 3: 5 Total: 33 Year 1: NA Year 2: 29 Year 3: 4 99 % Evaluation of USAID Vietnam Disabilities Programming 179 Indicator/ Frequency of Reporting Description Unit of Measure/ Disaggregation Data Source/ Collection Method Project Life Targets Achievement to date Notes & Difference between updated achievement and project life target IR 1: Effective disability case management/social work and a comprehensive referral system implemented in Danang using an approach that is sustainable and replicable to other hot spots. Sub IR 1.1 A force of professional social workers and case managers developed Evaluation of USAID Vietnam Disabilities Programming 180 Indicator/ Frequency of Reporting Description Unit of Measure/ Disaggregation Data Source/ Collection Method Project Life Targets Achievement to date Notes & Difference between updated achievement and project life target 1.1.1 1.1.1 Number of communal workers and district supervisors who successfully complete disability social work/case management/CBR training and refresher training supported by PDSP (Annually) Communal CMs and district, city CM supervisors are social workers, health workers and school teachers are an important segment of the disability support workforce. Supporting the development of DOLISA social workers and the social work capacity skills in DOH and DOET inclusive education and special education staff and teachers contributes to the development of a comprehensive integrated support and referral system for PWDs and their families and caregivers. Number Disaggregated by city/province, district, commune, gender of trainee; and department/ organizational affiliation of Trainee (e.g., DOLISA,DOH, DOET, NGO) Project training reports Life of Project Total: 339 Year 1: 189 (Danang) Year 2: 100 (Bien Hoa) Year 3: 50 (Phu Cat) The updated achievement is 426 Year 1: 189 in Danang. Year 2&3: 132 in Dong Nai and 105 in Binh Dinh 125 % PDSP already reached 3 year target Evaluation of USAID Vietnam Disabilities Programming 181 Indicator/ Frequency of Reporting Description Unit of Measure/ Disaggregation Data Source/ Collection Method Project Life Targets Achievement to date Notes & Difference between updated achievement and project life target 1.1.2 1.1.2 Number of trainers certified as Master trainers on SW and CM with PWDs through TOT, who are SW lecturers and SW professionals at local institutions (Annually) Training of trainers on social work, case management with disability and referral system Number count Disaggregated by city/province, gender of trainee; and University /organizational affiliation of Trainee Project training reports, reports from DOLISA, ULSA, NUSSA Life of Project Total: 15 Year 1: 15 Year 2: 15 Year 3: 0 Total: 24 Year 1: 24 Year 2: 24 Year 3: 0 160 % The target achieved and activities were completed in year 2. 1.1.3 Number of communes having communal workers trained in SW and CM in field of disability, and CBR skills. (Annually) Measures the geographic coverage of CM work capacity for disability support services Number count Disaggregated by city/province Project training reports and DOLISA District Social Work Service Center data Project life total: 75 Danang: 56 Dong Nai: 10 Binh Dinh: 9 The updated achievement is 131, including 56/56 (all 7 districts of Da Nang), 42/42 (in Bien Hoa and Vinh Cuu district of Dong Nai), 33/33 (in two districts of Phu cat and Hoai An, Binh Dinh province) 174 % PDSP already reached 3 year target Evaluation of USAID Vietnam Disabilities Programming 182 Indicator/ Frequency of Reporting Description Unit of Measure/ Disaggregation Data Source/ Collection Method Project Life Targets Achievement to date Notes & Difference between updated achievement and project life target 1.1.4 Local capacity for training commune and district disability SWs/CMs established at a local certificate￾granting institution that is supported by local government Measures the institutionalization of comprehensive disability support program Number count Disaggregated by city/province Documentation of official certificate￾granting program in SW/CM training established at local university Life of project Target: 1 1 (Dong A University) 100 % PDSP already reached 3 year target Evaluation of USAID Vietnam Disabilities Programming 183 Indicator/ Frequency of Reporting Description Unit of Measure/ Disaggregation Data Source/ Collection Method Project Life Targets Achievement to date Notes & Difference between updated achievement and project life target Sub IR 1.2 A multidisciplinary referral system developed and functioning 1.2.1 Number of Districts with teams of trained CM supervisors for a multidisciplinary disability referral system (Annually) Measures the geographic coverage of CM supervisors capacity for disability support services across districts and demonstrates interaction with the city- level Social Work Service Center and service providers. Number Project training reports and DOLISA District Social Work Service Center data Life of project target: 9 7 in Danang, 1 in Binh Dinh, 1 in Dong Nai, The actual achievement: 11 7 in Danang 2 in Dong Nai 2 in Binh Dinh 122 % PDSP reached 3 years target. Dong Nai authority also implemented this system by their own budget in three additional districts (Long Thanh, Trang Bom and Xuan Loc) 1.2.2 Number of Referral made where PWDs referred acting on the referral by the case managers (Quarterly) Measures the numbers of referrals that are made for any type of disability support services, and the number of clients who act on the referrals and then receive the referred services Number Disaggregated by district, commune, type of referral, gender of referred client Project training reports, DOLISA District Social Work Service Center data, service delivery site data Total: 3,228 Year 1: 0 Year 2: 228 Year 3: 3,000 Total: 1,411 Year 1: 0 Year 2: 554 Year 3: 857 44 % Evaluation of USAID Vietnam Disabilities Programming 184 Indicator/ Frequency of Reporting Description Unit of Measure/ Disaggregation Data Source/ Collection Method Project Life Targets Achievement to date Notes & Difference between updated achievement and project life target Sub IR 1.3. Supporting policies for case management and referral developed and promoted IR 2: Improved quality and access of specialized services for PWD Sub IR 2.1 Specialized quality medical rehabilitation services are available and accessible to PWD 2.1.1 Number of medical doctors receiving advanced training at district and city levels (and refresher training) (Annually) Measures number of doctors with improved skills in providing services to PWD. Number Disaggregated by district, gender of doctor Project training reports, DOH data Life of Project Target: 24 total doctors Year 1: 24 Year 2: 24 Year 3: 0 Total: 30 Year 1: 24 Year 2: 30 Year 3: 30 125 % PDSP already met 3 year target 2.1.2 Number of physical therapists (PTs) at district and city levels receiving advanced training and refresher training. (Annually) Measures number of PTs with improved skill sets to conduct therapy to PWD. Number Disaggregated by district, gender of PT Project training reports Life of Project Target: 40 PTs Year 1: 40 Year 2: 40 Year 3: 40 Total : 41 PTs Year 1: 40 Year 2: 41 Year 3: 41 102 % PDSP already met 3 years target Evaluation of USAID Vietnam Disabilities Programming 185 Indicator/ Frequency of Reporting Description Unit of Measure/ Disaggregation Data Source/ Collection Method Project Life Targets Achievement to date Notes & Difference between updated achievement and project life target 2.1.3 Number of CM/SW receiving basic CBR training (and refresher training) (Annually) Measures number of CM/SW with improved skill sets to conduct basic rehab services. Number Disaggregated by district, commune, gender of worker, type of worker Project training reports See IR.1 100 % 2.1.4 Number of medical rehabilitation doctors and senior PTs receiving TOT training (Annually) Measures number of doctors and senior PTs certified as Master Trainers. Number Disaggregated by district, gender of doctor or senior PT Project training reports Life of Project Total: 15 Year 1: 0 Year 2: 15 Year 3: 15 Total: 19 Year 1: 0 Year 2: 15 Year 3: 19 127 % PDSP already met 3 year target by end of year 2 (15 PTs and doctors). Evaluation of USAID Vietnam Disabilities Programming 186 Indicator/ Frequency of Reporting Description Unit of Measure/ Disaggregation Data Source/ Collection Method Project Life Targets Achievement to date Notes & Difference between updated achievement and project life target 2.1.5 Number of selective caregivers/parents provided training in basic therapy (Annually) Measures number of caregivers or parents able to provide basic PT to PWD in household. Number Disaggregated by caregiver gender, type of training, district, commune Project training reports, data from computerized disability information system (DIS) Life of Project Total: 820 caregivers Year 1: 70 Year 2: 750 Year 3: 0 The actual number is 575 Year 1: 50 Year 2: 450 Year 3: 75 70 % 2.1.6 Number of doctors, PTs and nurses trained at commune/ward health clinics in early detection and intervention (Annually) Measures number of health professionals trained at 56 commune/ward health clinics in early detection and intervention Number Disaggregated by gender, commune, ward, professional position PDSP project, DOH records Life of Project Total: 159 Year 1: 0 Year 2: 59 Year 3: 100 Total: 167 Year 1: 0 Year 2: 59 Year 3: 108 105 % PDSP already met 3 year target. Evaluation of USAID Vietnam Disabilities Programming 187 Indicator/ Frequency of Reporting Description Unit of Measure/ Disaggregation Data Source/ Collection Method Project Life Targets Achievement to date Notes & Difference between updated achievement and project life target 2.1.7 Number of clinical examination provided for diagnosis confirmation with PDSP support (Annually) Measures the number of PWDs identified in the 2010/2011Danang Disability Survey receiving confirmation of diagnosis. Survey identified 5,530 needing confirmation. Number Disaggregated by type of disability, gender of PWD, and age category (adult PWD or CWD <18) DOH clinical reports, data computerized disability information system (DIS) Life of Project Target: 650 PWD Year 1: 300 Year 2: 300 Year 3: 50 Total: 3,233 PWD Year 1: 464 Year 2: 2,537 Year 3: 232 497 % PDSP already reached over 3 years target. Danang: 690 Dong Nai: 533 Binh Dinh: 2,010 Evaluation of USAID Vietnam Disabilities Programming 188 Indicator/ Frequency of Reporting Description Unit of Measure/ Disaggregation Data Source/ Collection Method Project Life Targets Achievement to date Notes & Difference between updated achievement and project life target 2.1.8 Number of PWDs receiving PDSP support for physical therapy (Annually) Number of PWDs receiving PT with project support. 3,897 PWDs identified in the 2010/2011 Danang Disability Survey as needing PT. Number Disaggregated by type of disability, gender of PWD, and age category (adult PWD or CWD <18), district, commune Data from computerized disability information system (DIS), case manager files, PDSP project data Life of Project Target: 450 PWD Year 1: 0 Year 2: 300 Year 3: 150 Total: 781 Year 1: 170 Year 2: 455 Year 3: 156 174 % Danang: 661 Dong Nai: 0 Binh Dinh: 120 2.1.9 Number of PWDs with speech disability receiving speech therapy (Annually) Number of PWDs receiving speech therapy with project support. 3,029 PWDs identified in the 2010/2011 Danang Disability Survey as needing speech therapy Number Disaggregated by gender of PWD, and age category (adult PWD or CWD <18), district, commune Data from computerized disability information system (DIS), case manager files, PDSP project data Life of Project Target:175 PWDs Year 1: 0 Year 2: 100 Year 3: 75 Total: 246 PWDs Year 1: 0 Year 2: 115 Year 3: 131 140 % (154 in Danang and 92 in Quang Nam) Evaluation of USAID Vietnam Disabilities Programming 189 Indicator/ Frequency of Reporting Description Unit of Measure/ Disaggregation Data Source/ Collection Method Project Life Targets Achievement to date Notes & Difference between updated achievement and project life target 2.1.10 Number of PWDs with hearing disabilities and in need of hearing aids who receive them with PDSP project support (Quarterly) Number of PWDs receiving hearing aids with project support. 2,772 PWDs identified in the 2010/2011 Danang Disability Survey as hearing aids. Number Disaggregated by gender of PWD, and age category (adult PWD or CWD <18), district, commune Data from computerized disability information system (DIS), case manager files, PDSP project data Life of project target is 125 PWDs Year 1: 50 Year 2: 75 Year 3: 0 Total: 260 Year 1: 17 Year 2: 148 Year 3: 95 208 % PDSP already met 3 year target. Danang: 87 Dong Nai: 125 Binh Dinh: 48 2.1.11 Number of PWDs in need of corrective surgery (e.g., mobility & cleft lip palate) who receive surgeries with PDSP project support (Annually) Number of PWDs receiving corrective surgery with project support. 404 PWDs identified in the 2010/2011 Danang Disability Survey as requiring some sort of corrective surgery Number Disaggregated by gender of PWD, and age category (adult PWD or CWD <18), district, commune, type of surgery Data from computerized disability information system (DIS), case manager files, PDSP project data, DOH clinic records Life of Project Target: 50 PWDs Year 1: 0 Year 2: 50 Year 3: 0 Total: 15 Year 1: 7 Year 2: 3 Year 3: 5 30 % Danang: 8 Dong Nai: 7 Evaluation of USAID Vietnam Disabilities Programming 190 Indicator/ Frequency of Reporting Description Unit of Measure/ Disaggregation Data Source/ Collection Method Project Life Targets Achievement to date Notes & Difference between updated achievement and project life target 2.1.12 Number of assistive devices provided to PWDs in need of such devices (might exclude eyewear) with PDSP project support (Quarterly) Number of PWDs receiving assistive devices with project support. 5,000 PWDs identified in the 2010/2011 Danang Disability Survey as requiring some sort of assistive device (might exclude eyewear). Number Disaggregated by gender of PWD, and age category (adult PWD or CWD <18), district, commune, type of device Data from computerized disability information system (DIS), case manager files, PDSP project data, DOH clinic records Life of Project Target: 270 PWDs Year 1: 120 Year 2: 150 Year 3: 0 Total: 1,171 PWDs Year 1: 237 Year 2: 748 Year 3: 186 434 % PDSP reached over 3 years target. Danang: 487 Dong Nai: 628 Quang Nam: 56 Evaluation of USAID Vietnam Disabilities Programming 191 Indicator/ Frequency of Reporting Description Unit of Measure/ Disaggregation Data Source/ Collection Method Project Life Targets Achievement to date Notes & Difference between updated achievement and project life target Sub IR 2.2 Increased school enrollment of children with disabilities in Danang by 20 % 2.2.1 Number of children with disabilities enrolled in school (See indicator 2 in the objective/impact indicator) (Annually) Measures the number increase from baseline of CWD school enrollment and impact of DSP Project on lives of PWD. Number and %; percentage points increase Disaggregated by commune, district, gender, , age group (5- 9; 10-17), type of disability Various sources including annual analysis of data from computerized disability information system (DIS); USAID-funded Danang 2011 Baseline Survey; DOLISA UNICEF 2011 Disability Survey in Bien Hoa; PDSP 2013 Baseline Disability Survey in Phu Cat and DOET reports Total: 285 Year 1: 0 Year 2: 135 Year 3: 150 Total: 216 Year 1: 40 Year 2: 156 Year 3: 20 76 % Danang: 186 Dong Nai: 10 Tay Ninh: 20 2.2.2 Number of additional resource rooms set up for inclusive education at selective schools in 7 districts of Danang city with PDSP support (Annually) Targets based on setting up 2 IE rooms for each of the six urban districts, and 3 IE resource rooms in selected schools with DSP project support) Number Disaggregated by commune and district PDSP project data and grantee reports Life of Project Target: 15 resource rooms Year 1: 5 Year 2: 10 Year 3: 0 Total: 5 Year 1: 0 Year 2: 5 Year 3: 0 33 % Evaluation of USAID Vietnam Disabilities Programming 192 Indicator/ Frequency of Reporting Description Unit of Measure/ Disaggregation Data Source/ Collection Method Project Life Targets Achievement to date Notes & Difference between updated achievement and project life target 2.2.3 Number of teachers trained and receiving refresher training in inclusive education with PDSP support. (Annually) Measures training received by teachers in inclusive education and special education Priority given to the 101 elementary schools. Number Disaggregated by teacher gender, commune, district PDSP project data, program data from UNICEF and FIDA Life of Project Target: 260 Year 1: 100 Year 2: 80 Year 3: 80 Life of Project Target: 320 Year 1: 116 Year 2: 124 Year 3: 80 123 % The project already met 3 year target 2.2.4 Number of school teachers trained in early detection and intervention with PDSP support (Annually) Measures number of teachers trained in early detection and intervention (and receive refresher training) Number Disaggregated by gender, commune, district PDSP project and grantee reports, DOH/ DOET records, data from computerized disability information system (DIS) Life of Project Target: 245 teachers Year 1: 35 Year 2: 210 Year 3: 0 Total: 271 teachers Year 1: 61 Year 2: 210 Year 3: 0 110 % The project already met 3 year target Evaluation of USAID Vietnam Disabilities Programming 193 Indicator/ Frequency of Reporting Description Unit of Measure/ Disaggregation Data Source/ Collection Method Project Life Targets Achievement to date Notes & Difference between updated achievement and project life target 2.2.5 Number of CWD who receive early intervention with project support (Annually) Number of children receiving early intervention based on 2% estimate of children from VNAH’s experience from Cam Le District 2011 pilot study Number Disaggregated by gender, commune, district, type of disability PDSP project and grantee reports, DOH/ DOET records, CDC team reports; data from computerized disability information system (DIS) Life of Project Target: 175 CWDs Year 1: 0 Year 2: 100 Year 3: 75 Total 99 CWDs Year 1: 0 Year 2: 24 Year 3: 75 57 % Danang: 99 Dong Nai: 0 Binh Dinh: 0 2.2.6 Number of CWD who receive educational support by PDSP (Annually) Measures number of CWD benefiting from educational support by the project intervention.. Number Disaggregated by gender, commune, district, type of disability PDSP project data, Data from computerized disability information system (DIS), and DOET baseline data, Life of Project Target: 525 CWDs Year 1: 300 Year 2: 150 Year 3: 75 Total: 1,134 Year 1: 451 Year 2: 120 Year 3: 563 216 % The project already met the Project life target Danang: 873 Dong Nai: 171 Binh Dinh: 90 Evaluation of USAID Vietnam Disabilities Programming 194 Indicator/ Frequency of Reporting Description Unit of Measure/ Disaggregation Data Source/ Collection Method Project Life Targets Achievement to date Notes & Difference between updated achievement and project life target Sub IR 2.3 Employment rate among PWD increased by 20 % from baseline 2.3.1 Number of additional adult PWDs receiving employment, including self￾employment, with project support (See the objective indicator Number 3) (Annually) Measures the number of PWD in need of employment support as obtaining some employment. Number of PWDs in need identified by Danang Disability Survey as needing support: 1,304 Number and % Disaggregated by gender of PWD, type of employment, commune, district PDSP project data, Data from computerized disability information system (DIS), and DOLISA baseline data, Total: 781 Year 1: 50 Year 2: 379 Year 3: 352 Total: 546 Year 1: 15 Year 2: 248 Year 3: 283 70 % Danang: 408 Dong Nai: 138 2.3.2 Number of PWDs receiving Vocational training (Annually) Measures number of PWD benefiting from vocational training. Disability Survey indicated that 572 PWD in need of this type of training. Number Disaggregated by gender of PWD, district, commune PDSP project data Data from computerized disability information system (DIS) Life of Project Total: 195 PWD Year 1: 25 Year 2: 100 Year 3: 70 Total: 331 PWD Year 1: 20 Year 2: 200 Year 3: 95 170 % The project already met the project life target Evaluation of USAID Vietnam Disabilities Programming 195 Indicator/ Frequency of Reporting Description Unit of Measure/ Disaggregation Data Source/ Collection Method Project Life Targets Achievement to date Notes & Difference between updated achievement and project life target 2.3.3 Number of eligible PWDs receiving Preferential Loan with PDSP support (Annually) Measures number of PWD receiving loans. Disability Survey indicated that 1,600 PWD in need of this type of training. Number Disaggregated by gender of PWD, district, commune PDSP project data, Data from computerized disability information system (DIS) Life of Project Target: 110 Year 1: 10 Year 2: 50 Year 3: 50 Total: 87 Year 1: 35 Year 2: 32 Year 3: 20 88 % Sub IR 2.4 Improved housing 2.4.1 Number of PWD households receiving home improvement, accessibility, WC facilities and water supply and sanitation with PDSP project support. Measures number of PWD households needing home improvement receiving improvements with project support. 963 households identified in Disability Survey Number Disaggregated by gender of PWD, district, commune PDSP project and grantee reports Life of Project Target: 150 households Year 1: 0 Year 2: 75 Year 3: 75 Total: 0 Year 1: 0 Year 2: 0 Year 3: 0 0 % Sub IR 2.5 Supportive policies in place and promoted Evaluation of USAID Vietnam Disabilities Programming 196 Indicator/ Frequency of Reporting Description Unit of Measure/ Disaggregation Data Source/ Collection Method Project Life Targets Achievement to date Notes & Difference between updated achievement and project life target 2.5.1 Number of PWDs indirectly benefiting from DSP support of DPOs/public events (Annually) Measure the number of PWDs who participate in the social event such as advocacy, sport events, disability day events, disability forum with government Number Disaggregated by gender, DPO name, type of support received through the DSP supported DPO PDSP monitoring data; DPO reports Total: 672 Year 1: 472 Year 2: 150 Year 3: 50 Total: 942 Year 1: 583 Year 2: 265 Year 3: 94 140 % The project already met 3 year target. Danang: 360 Dong Nai: 366 Binh Dinh: 216 2.5.2 Number of government sponsored disability program meetings at the City-wide and district-level that formally include DPO and/or PWD self-help groups in Da Nang and other provinces (Annually) Measures willingness of City and district officials to include participation of DPOs and self-help groups as full members of committees or working groups Number Disaggregated by district, city, type of activity Project data and grantee reports (DPOs), Social Work Service Center data Year 1: 2 Year 2: 6 Year 3: 12 Total: 17 Year 1: 2 in Dong Nai Year 2: 3 in Danang Year 3: 11 in Dong Nai & 1 in Da Nang 85 % Evaluation of USAID Vietnam Disabilities Programming 197 Indicator/ Frequency of Reporting Description Unit of Measure/ Disaggregation Data Source/ Collection Method Project Life Targets Achievement to date Notes & Difference between updated achievement and project life target 2.5.3 Number of directives, disability action plans, decisions or policy guidelines developed by People’s Committee of Danang that involves DPOs/PWDs in design, planning. (Annually) Measures willingness of City and district officials to include participation of DPOs and self-help groups as full members of committees or working groups Number Official government records of disability meetings and directives, policies, action plans made. PDSP project records Total: 3 Year 1: 0 Year 2: 2 Year 3: 1 Total 3: 1/ DOLISA's guideline for DIS and case management system. 2/ Decision issued by Danang People’s Committee on the approval of Da Nang disability action plan in 2014. 3/ DOET guideline on educational supports for children with disabilities. 100 % The project already met 3 year target. IR 3: Improved relevant public health services (population-based birth defects surveillance, post-natal newborn screening, pre-pregnancy services and cancer surveillance. Sub IR 3.1 Birth defects surveillance (Da Nang) Evaluation of USAID Vietnam Disabilities Programming 198 Indicator/ Frequency of Reporting Description Unit of Measure/ Disaggregation Data Source/ Collection Method Project Life Targets Achievement to date Notes & Difference between updated achievement and project life target 3.1.1 Number of doctors and nurses/midwives in commune/ward health clinics and in /district/city health clinics trained in birth defect identification (Quarterly) Measures number of health professionals trained in birth defect identification in commune/wards health clinics and in district health centers, and city hospitals. Number Disaggregated by gender, commune, ward, district, city professional position PDSP project, DOH records Total: 310 Year 1&2: 0 Year 3: 310 (110 doctors & 200 nurses/midwives) Total: 174 Year 1&2: 0 Year 3: (86 doctors and 88 nurse/midwives) 56 % 3.1.2 Number of birth defects identified. (Quarterly) Directly measures the performance of the birth defect surveillance system in the pilot district Number Disaggregated by gender, commune, ward, district DOH records Total: 40 Year 1&2: 0 Year 3: 40 Year 3: 21 53 % 3.1.3 Percentage of cases with BD information correctly completed in the baby’s medical record, including the new separate BD form (Annually) Measures the performance of the birth defect surveillance system in the pilot district Percentage DOH records Year 3: 80% 100%+ 125 % Evaluation of USAID Vietnam Disabilities Programming 199 Indicator/ Frequency of Reporting Description Unit of Measure/ Disaggregation Data Source/ Collection Method Project Life Targets Achievement to date Notes & Difference between updated achievement and project life target 3.1.4 Number of birth defect cases that are double counted. (Annually) Measure the performance and the correction of the data recording, reporting and management for this system Number and % of all BDSS birth defect cases identified as double counted Disaggregated by gender, commune, ward, district DOH records Year 3: Goal of having less than 10% of cases double counted by end of project, and 0% once BDSS is fully computerized with software to catch and correct double entries medications 4/21 cases (end of Feb) The activity just recently started and the number is too small for calculating % 3.1.5 Percentage of babies with BDs referred to treatment/follow up services by Cam Le case management system among the total BD cases detected in Cam Le (Annually) Measures the effective integration of birth defect system into the referral/case management system in Cam Le Percentage Disaggregated by gender, commune, ward in Cam Le district DOH records Year 3: 100% 100% 100 % Sub IR 3.2 Newborn screening (Da Nang) Evaluation of USAID Vietnam Disabilities Programming 200 Indicator/ Frequency of Reporting Description Unit of Measure/ Disaggregation Data Source/ Collection Method Project Life Targets Achievement to date Notes & Difference between updated achievement and project life target 3.2.1 Number of babies screened for hearing loss conducted by trained staff supported by PDSP. (Quarterly) Measures the number of newborn screening babies supported by PDSP training program Number Disaggregated by gender, commune, ward, district DOH records Total: 800 Year 1&2: 0 Year 3: 800 Total: 334 Year 1&2: 0 Year 3: 334 (end of Feb) 42 % 3.2.2 Number of babies/infants diagnosed for heart diseases conducted by trained staff supported by PDSP. (Quarterly) Measures the number of newborn screening babies supported by PDSP training program Number Disaggregated by gender, commune, ward, district DOH records Total: 480 Year 1&2: 0 Year 3: 480 Total: 217 Year 1&2: 0 Year 3: 217 (end of Feb) 45 % 3.2.3 Number of babies diagnosed/ treated for metabolism and hormonal diseases conducted by trained staff supported by PDSP. (Quarterly) Measures the number of newborn screening babies supported by PDSP training program Number Disaggregated by gender, commune, ward, district DOH records Total: 70 Year 1: 0 Year 2: 0 Year 3: 70 Total: 29 Year 1: 0 Year 2: 0 Year 3: 29 41 % Evaluation of USAID Vietnam Disabilities Programming 201 Indicator/ Frequency of Reporting Description Unit of Measure/ Disaggregation Data Source/ Collection Method Project Life Targets Achievement to date Notes & Difference between updated achievement and project life target 3.2.4 Number of women at antenatal unit of Children and Obstetric hospital receiving counseling on NBS conducted by trained staff supported by PDSP. (Quarterly) Measures the number of reproductive age women who receive NBS counseling at the hospital Number Disaggregated by commune, ward, district DOH records Total: 1,840 Year 1: 0 Year2: 800 Year 3: 1,040 Total: 833 Year 1: 0 Year2: 158 Year 3: 675 (end of Feb) 45 % 3.2.5 Number of women at postnatal unit of Children and Obstetric hospital receiving counseling on NBS conducted by trained staff supported by PDSP. (Quarterly) Measures the number of reproductive age women who receive NBS counseling at the hospital Number Disaggregated by, commune, ward, district DOH records Total: 10,400 Year 1: 0 Year2: 800 Year 3: 9,600 Total: 7,598 Year 1: 0 Year2: 1,329 Year 3: 6,269 73 % Evaluation of USAID Vietnam Disabilities Programming 202 Indicator/ Frequency of Reporting Description Unit of Measure/ Disaggregation Data Source/ Collection Method Project Life Targets Achievement to date Notes & Difference between updated achievement and project life target 3.2.6 Number of people who attended pre￾marriage counseling club to receive information on NBS and PCC through talks/events. (Quarterly) Measures the number of reproductive age female and male who receive NBS counseling in community Number Disaggregated by gender, commune, ward, district DOH records Total: 2,500 Year 1: 0 Year 2: 760 Year 3: 1,740 Total: 1,823 Year 1: 0 Year 2: 1,091 Year 3: 732 (end of Feb) 73 % 3.2.7 Percentage of trained counselors observed who provide correct information (Quarterly) Measures the performance of counselors who provide information on NBS with PDSP support Percentage DOH records Year2: 70% Year 3: 70% Year 2: 25 % Year 3: 66,7 % 95 % In year 3, the observation will be made in next quarter Evaluation of USAID Vietnam Disabilities Programming 203 Indicator/ Frequency of Reporting Description Unit of Measure/ Disaggregation Data Source/ Collection Method Project Life Targets Achievement to date Notes & Difference between updated achievement and project life target 3.2.8 Percentage of trained counselors observed who demonstrate good counseling skills (Quarterly) Measures the performance of counselors who demonstrate the good skills in communication with PDSP support Percentage DOH records Year2: 70% Year 3: 70% Year2: 25% Year 3: 66,7% 95 % In year 3, the observation will be made in next quarter Evaluation of USAID Vietnam Disabilities Programming 204 Indicator/ Frequency of Reporting Description Unit of Measure/ Disaggregation Data Source/ Collection Method Project Life Targets Achievement to date Notes & Difference between updated achievement and project life target Sub IR 3.3 Pre-pregnancy (preconception) services (Da Nang) 3.3.1 Number of nurse/doctor at commune or ward/district/city health clinics trained (and receive refresher trainings) in pre￾pregnancy health counseling for women supported by PDSP (Annually) Measures number of health workers trained in pre￾pregnancy counseling for adolescents and women in prevention of birth defects. Number Disaggregated by commune or ward, gender of nurse or health worker DOH records Total: 86 Year 1: 0 Year 2: 6 (ToT) Year 3: 80 Total: 85 Year 1: 0 Year 2: 6 (ToT) + 44 health educators Year 3: 35 (end of Feb) 99 % Evaluation of USAID Vietnam Disabilities Programming 205 Indicator/ Frequency of Reporting Description Unit of Measure/ Disaggregation Data Source/ Collection Method Project Life Targets Achievement to date Notes & Difference between updated achievement and project life target 3.3.2 Number of women (including adolescent females) who receive pre￾pregnancy counseling at service delivery point. (Quarterly) Assumes strategy and action plan for providing pre￾pregnancy counseling services for prevention of birth defects and other disabilities in Danang is developed by key stakeholders, following international best practices and including provisions high quality training in counseling and key messages Number Disaggregated by commune or ward, district, age group (Under age 20, ages 20 and over) DOH records Total: 2,500 Year 1: 0 Year 2: 150 Year 3: 2,340 Total: 598 Year 1: 0 Year 2: 232 Year 3: 366 (end of Feb) 24 % 3.3.3 Number of women who received information from “group talks” at community in Da Nang Measures the number of women including adolescent females who receive the communication program on Pre Conception Care Number Disaggregated by commune or ward, district, age group (Under age 20, ages 20 and over) DOH records Total: 12,540 Year 1: 0 Year 2: 8,700 Year 3: 3,840 Total: 10,129 Year 1: 0 Year 2: 8,700 Year 3: 1,429 81 % Evaluation of USAID Vietnam Disabilities Programming 206 Indicator/ Frequency of Reporting Description Unit of Measure/ Disaggregation Data Source/ Collection Method Project Life Targets Achievement to date Notes & Difference between updated achievement and project life target Sub IR 3.4 Cancer surveillance and prevention: this component was canceled IR 4: Expand implementation of the Program of Comprehensive and Integrated Support to People with Disabilities to selected areas. Sub IR 4.1 Assessments in Bien Hoa and Phu Cat for Expanded PWD Services 4.1.1 Disability assessment tool, and assessment training and implementation plan finalized (Annually) Measures assessment completion assessment and of implementation plan Number Disaggregated by geography DSP project reports Life of project target: 2 assessments and 2 PDSP Action Plans Year 1 : 2 Year 2 : 2 Year 3: 0 Two assessments and 2 PDSP Action Plans developed and completed 100 % The project already met 3 year target. Evaluation of USAID Vietnam Disabilities Programming 207 Indicator/ Frequency of Reporting Description Unit of Measure/ Disaggregation Data Source/ Collection Method Project Life Targets Achievement to date Notes & Difference between updated achievement and project life target 4.1.2 Baseline PWD assessments conducted in hot spots, resulting in comparable baseline indicators on disability prevalence, services, needs, and institutional and CBR support capacity in hot spots (Annually) Measures assessment completion assessment and of implementation plan Number Disaggregated by geography PDSP project reports Life of project target: 2 assessments and 2 PDSP Action Plans Year 1 : 2 Year 2 : 2 Year 3: 0 Two assessments and 2 PDSP Action Plans developed and completed 100 % The project already met 3 year target. Sub IR 4.2 Direct Assistance Provided to PWDs in Dong Nai, Binh Dinh and other provinces Dong Nai (Bien Hoa city and Vinh Cuu district) 4.2.1 Number of PWDs provided direct assistance support by PDSP in Dong Nai (Quarterly) PWD identified in need of direct assistance by DOLISA/UNICEF 2011 survey Number Disaggregated by gender, age group, type of disability and type of assistance needed and provided with DSP support PDSP monthly reports on direct assistance provided by PDSP F&A Officer Total : 2,000 PWD individuals Total: 1,954 Year 1 : 378 Year 2 : 1,514 Year 3: 62 97 % Evaluation of USAID Vietnam Disabilities Programming 208 Indicator/ Frequency of Reporting Description Unit of Measure/ Disaggregation Data Source/ Collection Method Project Life Targets Achievement to date Notes & Difference between updated achievement and project life target 4.2.2 Number of communes/wards with PWD social work, case management, and CBR activities supported by PDSP (Annually) Measures the geographic coverage of CM work capacity for disability support services Number Disaggregated by city/province DCOP and Objective 1 team leader Dong Nai: 10 The updated achievement is 42/42 (in Bien Hoa and Vinh Cuu district of Dong Nai 420 % 4.2.3 Number of SWs, case managers, CBR workers trained in Dong Nai (Annually) Communal CMs and district, city CM supervisors are social workers, health workers and school teachers are an important segment of the disability support workforce. Number Disaggregated by city/province, district, commune, gender of trainee; and department/ organizational affiliation of Trainee (e.g., DOLISA,DOH, DOET, NGO) DSP monthly program reports on DSP sponsored trainings Life of Project Total: 100 The updated achievement Year 2&3: 132 in Dong Nai 132 % Evaluation of USAID Vietnam Disabilities Programming 209 Indicator/ Frequency of Reporting Description Unit of Measure/ Disaggregation Data Source/ Collection Method Project Life Targets Achievement to date Notes & Difference between updated achievement and project life target 4.2.4 Number of rehabilitation units at commune health clinic in Bien Hoa City and Vinh Cuu District provided equipment support by PDSP (Annually) Measures the number of rehab clinics in Dong Nai receive rehab equipment form PDSP support Number Disaggregated by type of support provided for each clinic DSP monthly program reports Project life target was 2 Year 1: 0 Year 2: 2 Year3: 0 Total: 2 Year 1: 0 Year 2: 2 Year3: 0 100 % PDSP met the target by the end of year 2). Binh Dinh (Phu Cat and Hoai An District) 4.2.5 Number of social workers, case managers and CBR workers trained with DSP support (Phu Cat and Hoai An district) (Quarterly) Communal CMs and district, city CM supervisors are social workers, health workers and school teachers are an important segment of the disability support workforce. Number Disaggregated by type of provider trained DSP monthly program reports on DSP sponsored trainings Total: Year 3: 50 (Phu Cat) The updated achievement is 105 in Binh Dinh 210 % Evaluation of USAID Vietnam Disabilities Programming 210 Indicator/ Frequency of Reporting Description Unit of Measure/ Disaggregation Data Source/ Collection Method Project Life Targets Achievement to date Notes & Difference between updated achievement and project life target 4.2.6 Number of PWDs in Phu Cat and Hoai An receiving direct assistance from PDSP (Annually) Measure the number of PWDs who receive PDSP support services that include a variety of services in social, health and education such as educational support, school enrollment for CWDs, assistive devices, livelihood/job, vocational training … Number Disaggregated by gender, age group, type of disability and type of direct assistance received by PWDs with DSP support DCOP & DSP F&A officer Total project target: 1,200 individuals Total: 2,268 Year1: 0 Year 2: 2,085 Year 3: 183 189 % Other provinces (Tay Ninh, Quang Nam, Thai Binh, Binh Phuoc and Hue) 4.2.7 Number of PWDs in other provinces receiving direct assistance from PDSP (Annually) Measure the number of PWDs who receive PDSP support services that include a variety of services in social, health and education such as educational support, school enrollment for CWDs, assistive devices, livelihood/job, vocational training … Number Disaggregated by gender, age group, type of disability and type of direct assistance received by PWDs with DSP support PDSP monthly program reports Total project life: 800 individuals Total: 155 Year 1: NA Year 2: NA Year 3: 135 in Quang Nam, 20 in Tay Ninh 19 % 4.2.8 Number of PWDs in other provinces participating in the public events with PDSP support Measure the number of PWDs who participate in the social event such as advocacy, sport events, disability day events, disability forum with government Number PDSP monthly program reports Total project target: 200 individuals Total: 0 Year 1: NA Year 2: NA Year 3: 0 0 % Evaluation of USAID Vietnam Disabilities Programming 211 Indicator/ Frequency of Reporting Description Unit of Measure/ Disaggregation Data Source/ Collection Method Project Life Targets Achievement to date Notes & Difference between updated achievement and project life target Note: Depending on which activities are rolled out, based on needs of Dong Nai,, Binh Dinh and other provinces, indicators from Objectives 1–3 will be used to measure progress