MOVING WITHOUT LIMITS (AID-440-A-15-00006) Report on the Mid-term Evaluation Date: December, 2021 1 Thua Thien Hue Department of Health Quang Nam Department of Health Binh Dinh Department of Health 2021 LIEM NGUYEN, WESLEY PRYOR, THANH NGUYEN MOVING WITHOUT LIMITS 2015-2022 MID-TERM EVALUATION REPORT 2 Moving without Limits Project 2015-2022 Mid-term Evaluation Report Abbreviations 3 I. Background 5 1. Overview of the project...............................................................................................5 2. Purpose & Scope of work ...........................................................................................5 3. Evaluation methodology.............................................................................................6 II. Progress & Specific Achievements to date 10 1. Progress & achievement of project outputs...........................................................10 2. MWL’s contribution to rehabilitation in health system .........................................13 3. Feedback and project impact from perspective of the beneficiaries & stakeholders........................................................................................................................16 4. Achievement of project’s goal and objectives.......................................................17 III. The AT-D service delivery system of MWL 17 1. Steps in AT-D service delivery.................................................................................18 a. Initiative 18 b. Assessment 18 c. Typology & Selection 19 d. Authorization 19 e. Implementation 20 f. Management & Follow-up 20 2. Structure of the MWL’s AT-D service delivery system and its quality................20 a. The structure 20 b. Quality of the MWL’s AT-D service delivery system 22 c. Other observations and issues relating to the MWL system 23 3. Project sustainability.................................................................................................24 a. Sustainable factors 24 b. Challenges to project sustainability 24 IV. Conclusions 25 V. Recommendations 26 Appendixes 32 3 Abbreviations ADL Activities of Daily Living AP Assistive product AT Assistive technology AS Assistive solution BDHD Binh Dinh Health Department COVID-19 Coronavirus disease of 2019 DIS Disability Information System DNUMTP Da Nang University of Medical Technology & Pharmacy DOH Department of Health HHD Thua Thien Hue Health Department HMIS Health Management Information System HMU Hanoi Medical University HUMP Hue University of Medicines and Pharmacy IC International Center ILA Independent living aids KII Key informant interview MEL Monitoring, evaluation and learning MOH Ministry of Health OPD Organizations of Persons with Disabilities PRMGEP Physical Rehabilitation Medicines Graduate Education Program QNHD Quang Nam Health Department SME Subject Matter Expert UNCRPD United Nations Convention on the Rights of Persons with Disabilities USAID United States Agency for International Development WHO World Health Organization 4 Abstract The project: “Moving without Limits Project - 2015-2022” (abbreviated as MWL) is implemented by the International Center (IC) “to improve persons with disabilities’ access to quality assistive technologies or devices (AT-D) that enhance their independence in activities of daily living and their participation in social activities in USAID target provinces and districts of Thua Thien Hue, Quang Nam and Binh Dinh”. This mid-term evaluation found that the project is on-track, has achieved most of its output indicators, and is likely to achieve more than its commitment. Key project benefits: The project has made significant contributions to rehabilitation and the WHO’s six “Health Systems Building Blocks”. In particular, the project made valuable contributions to developing the AT-D service delivery system and health workforce for rehabilitation, which are sustainable and ensure long-term effects. Support towards coverage of assistive technologies under health insurance, development of Physical Medicines and Rehabilitation (PMR) with Hanoi Medical University, the National Strategy on Rehabilitation, National Assessment for the AT￾ADs needs and Priority List of AT-ADs with Ministry of Health will also ensure sustainable benefits. General recommendations: To extend the project’s successes, it should 1) expand current disability-specific emphasis and adopt twin-track approach); 2) encourage active participation of non-medical stakeholders, such as social workers and education services, to work alongside health professionals in AT-D service delivery to strengthen practical use of AT-D for social participation; 3) continue to adopt and strengthen the person-centred approach and participation of persons with disabilities in different steps of the AT-D service delivery; 4) expand target population to include not only persons with mobility impairment but also everyone with mobility difficulties to push for AT-D to be understood as a part of UHC rather than a disability-specific response; 5) apply multiple AT-D service delivery models to meet diverse needs and access patterns of persons with mobility disabilities, e.g. delivering AT-D at schools and work places. Other specific recommendations are also provided. 5 II. Background 1. Overview of the project The project “Moving without Limits in Thua Thien Hue, Quang Nam & Binh Dinh 2015-2022” (abbreviated as MWL) is being implementing by the International Center (IC) “to improve persons with disabilities’ access to quality assistive technologies or devices (AT-D) that enhance their independence in activities of daily living and their participation in social activities in USAID target provinces and districts of Thua Thien Hue, Quang Nam and Binh Dinh”. The project has three specific objectives: • To improve availability of quality AT-D for persons with mobility impairments • To improve capacity related to AT-D for the health sector workforce at national and sub-national levels • To improve advocacy for policies/regulations and information sharing related to AT-D. Key beneficiaries of the project include more than 10,000 persons with mobility difficulties who were assessed for mobility assistive device needs, among whom 7,000 were provided with assistive products. Additionally, rehabilitation facilities and health centres in 5 provinces have been equipped with rehabilitation equipment and trainings to improve AT-D service delivery capability. The project covers 22 districts in Thua Thien Hue, Quang Nam, and Hue, and 2 central hospitals in Hanoi and Da Nang between July 2015 and September 2022. It is financially supported by the United States Agency for International Development (USAID) Vietnam with a funding of USD 7.9 million. The project has three main phases: 2015-2017; 2017-2019; and 2019-2022. The project includes various stakeholders, including provincial Health Departments of Thua Thien Hue, Quang Nam and Hue, district health centres, commune health centres, Bach Mai hospital, Da Nang Rehabilitation Hospital, Assistive Technology Australia (ATA), Hue University of Medicine and Pharmacy (HUMP), Da Nang University of Medical Technology and Pharmacy (DNUMTP), Hanoi Medical University (HMU), AT-D manufacturers (i.e. PhaNa & Kien Tuong), and Social Protection Centres of Provincial Department of Labour and Social Affairs (DOLISA). 2. Purpose & Scope of work This mid-term evaluation was conducted as a part of the project monitoring, evaluation and learning (MEL). The evaluation objective is to describe the quality, 6 effectiveness and efficiency of assistive product (AP)1 services targeted by the project. Additional aims are to describe features of the project design and execution that have been the most effective and impactful in delivering better access and positive outcomes for AP users, and to describe how project gains can be sustained and integrated into the Vietnamese service environment. Specific objectives of the mid-term evaluation were: ● To assess project progress and specific achievements to date; ● To describe the main AP service provision steps (screening/identification, needs assessment, prescription, procurement, adaptation and fitting, user training, follow-up, maintenance and repair) that are applied in the project, and assess their quality (which includes accessibility, competency, coordination, efficiency, flexibility, and user influence); ● To assess the project’s impacts (on both persons with disabilities & rehabilitation in health system), sustainability; and ● To make recommendations for the next phase 3. Evaluation methodology Evaluation approach Both existing documents and information from key informants are used to address the evaluation aims. The evaluation is conducted in three linked parts, that are summarised in the image below. Figure 1: Diagram of study approach and steps. These steps are further described in Table 1 below. 1 In this report, AT-D and AP are interchangeable. Guidelines & Standards Project aims National standards 1 In-depth interviews Document review Report synthesis (qual / quant) 2 Data analysis Interpretation Recommendations by step 3 7 Table 1: Description of key steps Part Description Outputs 1 – Refine frameworks, question guides and method Identify, summarise and integrate information from a range of sources including WHO guidelines and standards, information from the project, Vietnamese standards to refine the provided evaluation framework, generate questions for key informant interviews (KII), and identify the right mix of stakeholders to consult ● Refined evaluation framework ● Target KII list ● Question guide 2 – Interviews and document synthesis Stakeholder interviews and document synthesis. The precise list of stakeholders was refined during phase one but included people from i) the project, ii) suppliers / providers / professionals and iii) consumers and representative groups. This part included a preliminary coding / interpretation step to determine whether additional interviews or consultations are required. This step also identified existing reports as far as possible. ● Description of KIIs (number, interviewee type) and main themes. 3 – Analysis & interpretation Findings were analysed to generate recommendations by synthesising information from reports and interviews using a framework comprised of the main service provision steps and quality indicators. Preliminary recommendations were prepared for feedback while a full report was being prepared. ● Summary list of recommendations by step ● Full report Preliminary Review framework The conceptual basis for the review considered the International Centre’s own framework based on a European study, evaluation methods, and AP sector-specific frameworks and our experience in the Vietnamese health, disability, rehabilitation and AP sectors. Understanding the quality of AP service provision required evaluating the overall project approach and its main impacts, challenges and goals. We conducted a rapid summary of existing project information to understand how its design has influenced AP service provision, and to make recommendations for improvement that take the goal of sustainable integration of the project activities into the Vietnamese context. 8 Table 2: Three review areas and possible/proposed main evaluation themes Review area Themes to evaluate Overview of the project ● A summary of the project overall, with focus on efforts to improve AP service provision ● What are the major achievements, emphasising strengthening AP service provision steps. o How did the program design and approach change over time? o What was challenging/simple to change and why were any delays experienced? o What user stories/testimony illustrates project successes and challenges effectively (for reporting and using findings to inform change) Focus on service delivery Framework based on main AP service provision steps, and 6 themes: i) Accessibility (& coverage), ii) competence, iii) coordination, iv) efficiency, v) flexibility, vi) user inputs. This framework draws on the provided framework, existing standards and guidelines (especially WHO & ISPO standards), and specific tools and frameworks to develop evaluation questions and interpret findings. The review emphasises fidelity with existing standards, guidelines and procedures in AP provision to highlight possible areas for improvement and to highlight reasons for poor uptake of standards. Focus on the future: Sustainability and integration ● What project features have been most impactful? ● What project features have been integrated into the Vietnamese context (public and private) effectively? What has led to that success and can it be replicated? ● What aspects are the most vulnerable if international support is withdrawn? Field visit & the key informant interviews A summary of project stakeholders, key informants and the interviews is provided in Table 3. 9 Table 3: Stakeholders, key informants and the interviews Stakeholder Key informant Issues of interest & Number of interview Provincial level Recipients of AT-D /DLA devices & relevant people ●Persons with mobility disabilities ●Caregivers of persons with disabilities ●Local authorities & social organizations ● R: Recipients of different types of AT-D & home accessibility devices, and caregivers ● I: Each province: visit 6-8 households at home (about 30-45 minutes each household). ● Q: For persons with disabilities & caregivers o Use of AT-D (how, frequency, purposes, difficulties in specific situations, etc.); Quality of AT-D; o Technical user guide; seeking help when having a problem, maintenance; Experience with re-assessment; o Impacts on independence in daily living activities & participation; o Awareness & knowledge of AT-D o Training received o Integration: source of information/referral. Rehabilitation services providing info about other relevant services Rehab facilities that received AT-D equipment ● AT Unit ● Workshops ● Transition houses ●Managers of facilities ●Trainers ●Trainees ●AT Unit staffs: doctors & technologists ● R & I: AT Unit in Bach Mai; Transition house in Binh Dinh; Workshop in Bach Mai, Hue, Binh Dinh ● PRI: List of AT-D equipment provided to each rehab facility ● Q: o Use of AT-D equipment (frequency), maintenance, 8 steps, lifetime; accessibility of the facility: project support & impact o Impacts of equipment to the facility o Characteristics of typical users; contribution /impacts to the beneficiaries; o Contribution of the AT Unit to the facility; strengths & challenges for long-term development of the Unit; o M&E system: medical records, follow-up or tracking system, o Assessing impacts on independent living & social inclusion o Devices made locally Provincial / District /Commune Health Centers ●Representatives ● R: Trainees: Short course; technicians 6 month; doctors 12 months; AD-T assessment; commune health trainee from the TOTs ● I: 1 group discussion (6-8 persons/group) or 3-4 individual interviews per group; and by level (commune, district, province) ● Q: Participation in the training; impacts on daily work; impacts on persons with disabilities; Public awareness of AT￾D ● Integration: sharing of information, referrals, decision￾making avoided duplications and gaps Others: PAVA, OPD ●Representatives ● I & Q: participation in project activities; inclusion support (if available) to help persons with mobility disabilities to participate in social activities. National level Subject Matter Expert (SME) ●SME members ● 1 group discussion with a few SME members (5-6 persons) IC ●Project director ●Managing & MEL staffs ● I: 1-2 representative who are the main contact points /in charge ● Q: o Evaluation of the standard procedures (AT/AD service provision; implementation) & Technical user guides; plan to use them o Physical Rehab Medicines Graduate Education Program of the HMU ● Potential contribution of the project to the National Rehabilitation Strategy 2021-2025, e.g. AD-T service delivery 10 Stakeholder Key informant Issues of interest & Number of interview framework, list for health insurance coverage, standard tech procedure for pricing AT-D services ● Q: Development of the Physical Rehab Medicines Graduate Education Program; Plan to use & integration into formal education; Potential impacts ● Q: Information in the software, medical records, target users, how to use, easiness to use, plan to use data from the software, potential to expand; link to DIS A general guide for the interviews is provided in Appendix 1. III. Progress & Specific Achievements to date 1. Progress & achievement of project outputs The project is on track and it has nearly completed all committed activities. Outputs have been exceeded in some activities. Key indicators of the project and its achievements are presented in Table 4. By September of 2021, for example, the project already completed assessment of mobility function and needs of mobility devices for more than 11 thousand people while it committed to only 10 thousand people. By the end of the project, it is projected that the project will provide the assessment for well more than 12 thousand people with mobility difficulties following the request from localities. The project aimed to provide AT-D to 7 thousand people with mobility difficulties, and it already provided AT-Ds to 6.7 thousand people. The number could be significantly higher given the current constraints for service delivery due to the COVID-19. On capacity building, the project committed to provide short-term training on AT-D to 1,360 health staffs and caregiver at community level, but it already provided the training to 2,114 people. Likewise, the project committed to provide short-term training on AT-D assessment and selection for 60 health workers at provincial and district health centres, but it also already provided training to more than two times of that number (168 people). For the long-term training for health workers on advance physiotherapy and rehabilitation, the project has actually provided training to 93 people instead of 40 as committed. Out of 93 doctors and technicians trained in general concepts of rehabilitation and International Classification of Functioning (ICF), 45 of them was selected and suggested by provincial department of health (DOH) to be trained and become Subject Matter Experts in AT/AD. By September 2021, the project has already provided short￾term trainings to 32 prosthetics and orthotics (P&O) technologists on fabrication and production of orthotic devices while it committed to only 20 trainees. Further examples of project achievements on key indicators can be found in Table 4. Upon USAID approval, the project also initiated new activities to improve its effectiveness. For instance, in the third phase of the project, the project digitized its monitoring and evaluation (M&E) which includes electronic records on AT-D users. While this is late, it is an important addition to the project and lessons can be learned from that for improvement of the disability information system (DIS). The e-M&E 11 system can be used to gain data and to conduct further assessment of the project impacts in the future once data is available. Table 4: Key indicators of achievement Key indicators of achievement Target/ Committed Achieved by Sep 2021 Target by the end Objective 1: Improved availability of quality AT-D # persons with disabilities received AT-D assessment (& got e-record on AT-D) 10,000 11,154 (112%) 12,554 (126%) # persons with disabilities provided with AT-D 7,000 6,699 (96%) 7,600 (109%) # family received accessible housing interventions 30 12 (40%) 42 (140%) # home modification/adaptation teams set up 3 0 3 # person-times received follow-up for AT-D maintenance & repair 7,000 6,797 (97%) 9,748 (13%) % persons with disabilities increased knowledge of AT-D use 85% 94% 85% # persons with disabilities & caregivers received training on AT-D use & maintenance 7,000 6,699 (96%) 7,600 (109%) Objective 2: Capacity building & System strengthening # rehab facilities at central & provincial level supported with AT-D equipment 5 4 5 # district rehab facilities supported with AT-D equipment & accessibility adaptation 5 6 9 # AT Units established at central health facilities 2 1 2 # AD produced locally by USAID’s innovation 105 127 (121%) 157 (149%) # short-term training on AT-D for commune health staffs & caregivers 1,360 2,114 (155%) 2,174 (160%) # short-term training on AT-D assessment & selection: provincial & district health 60 168 (280%) 168 (280%) 12 Key indicators of achievement Target/ Committed Achieved by Sep 2021 Target by the end # long-term training for service providers (6 & 12 months) on advance PT & rehab 40 93 (46+47) (233%) 93 (233%) # P&O technologists received short-term trainings in fabrication & production of orthotic devices 20 32 (160%) 32 (160%) # members of Subject Matter Experts (SME) at national level (core group) 20 20 (100%) 20 (100%) # SME core group in AT at provincial level 3 3 3 # Physical Rehab Medicines Graduate Education Program improved & piloted 1 0 1 # guides for AT-D service provider on service provision 2 0 2 # guides for AT-D user targeting persons with disabilities & caregivers 14 12 (86%) 14 (100%) # respite care model is set up and piloted 1 0 1 Objective 3: Policy Advocacy – TA provision In progress A national AT-D assessment report developed (with WHO) done A final national report An AT-D service delivery framework developed as a part of the National Rehabilitation Strategy 2021-2025 3 drafts of strategy developed Rehab Strategy is developed and approved A suggested list of AT-D for health insurance coverage developed A situational analysis is done A suggested list of AT developed Standard technical procedure for pricing AT-D developed A gender mainstreaming action plan developed for inclusion of gender sensitivity in the AT-D service delivery Gender gap report Training on gender for doctors and Gender mainstreamed into the AT/ADs service delivery 13 Key indicators of achievement Target/ Committed Achieved by Sep 2021 Target by the end technicians conducted 2. MWL’s contribution to rehabilitation in health system The project has positively impacted all six “building blocks” of the health system under the WHO framework: service delivery, health workforce, health information system, access to essential medicines, financing, and leadership/governane. 2 The project makes particularly valuable contributions to AT￾D service delivery and health workforce for rehabilitation. More specifically: Concerning accessibility, positive aspects include MWL’s efforts to bring services to clients, and drawing on user-inputs from 2019 to shape the intervention. Many impacts will be sustained through an emphasis on system change and capacity￾building, while other areas are at some risk (see sustainability section, below, for elaboration). However, there are opportunities to review activity focus, moving from a disability-only emphasis focusing on persons with long-term mobility impairments, to providing AT-D as a broader ‘Universal Health Coverage’ intervention. There are also opportunities to improve accessibility through greater emphasis on community knowledge of services, recognising that awareness of needs and available services are the critical first step in accessing health care, including AT-D services. The competence of the service and its staff was explored. Training provided through the project was received very positively by participants, and helps ensure long-term benefits. Participants requested additional training, which is a reflection on their quality and relevance, and represents an opportunity for further impacts. A specific area for further emphasis is P&O assessments: Doctors and technologists reported low-confidence in contributing to assessments, which could further strengthen the quality of services. Despite availability of quality management methods and generic service standards in O&P and other mobility AT-D, respondents noted that protocols in areas including AT-D information and user feedback were not implemented effectively. MWL is currently developing standards in these and other areas, which is positive. Overall, services coordination is a strength. When we explored how people and information move between levels of services and different professional stakeholders, we noted good coordination across and with levels, between stakeholders, and 2 WHO, 2010. Monitoring the building blocks of health systems: A handbook of indicators and their measurement strategies. ISBN 978 92 4 156405 2. Available online at https://apps.who.int/iris/bitstream/handle/10665/258734/9789241564052-eng.pdf; accessed December, 2021. 14 between the different service steps. Nonetheless, links between levels (province, district, commune) were less consistent than other areas, and an important opportunity for improving coordination. This evaluation provided scope for a modest exploration of service efficiency. MWL provides full payment for AT-D services, which is positive, but challenging to sustain, and does not reflect the opportunity for some clients to co-pay for services. There are opportunities to explore how alternative cost-recovery arrangements could be introduced in the context of local regulations and norms about health payments. In terms of systematic approaches to monitor quality and costs, within the context of the current project, this has been provided by IC, but will need to be systematized within service management arrangements in order to be sustained. An additional perspective on efficiency is the emphasis on AT-D with less emphasis on accessible housing (30 households in the current project) or other adaptations, or for personal assistants, which could potentially amplify the benefit of AT-D (and other services) at low cost. In addition, the recent transition of the AT-AD service delivery strategy from community-based approach to inpatient service approach, meaning to provide AT￾AD services in the hospital, can help ensure the sustainability of the project. to The project has several positive effects on flexibility, particularly ensuring more and better choices to beneficiaries compared with before the project. This is central to the aim of person-centred care. Within the scope of this evaluation, user feedback was positive and strongly suggested better engagement with clients and flexible services that met their needs, but further information about how that affected clients (and who might have missed out on those benefits) is beyond the scope of the current evaluation. On the other hand, user influence on the project overall and the systematic changes it has sought to introduce were limited. Representation of persons with disabilities in decision making and implementation of service changes could have been strengthened. Another perspective on flexibility is how MWL has aimed to support local P&O service provision to improve the responsiveness to client needs. While this is a positive overall, whether it is cost-effective and sustainable, and the transition to the AT-AD service from the community into the rehabilitation service center or hospitals that the MWL project is enforcing would contribute to the sustainability of the existing AT-Ad service provision model. MWL made contributions at different levels to the 10 priorities of action under the 2030 Rehabilitation Initiatives of the WHO. 3 Specifically: 1. Creating strong leadership and political support for rehabilitation at subnational, national and global levels: Collaboration with the MOH and provincial health departments in conducting project activities, capacity building for health workforce, establishment of the SMEs, development of standard protocols and guiding documents are among major activities that have contributed to the first priory of action at national and provincial level. 2. Strengthening rehabilitation planning and implementation at national and subnational levels, including within emergency preparedness and response: Capacity 3 WHO, 2017. Rehabilitation 2030: A Call for Action. Meeting report. Accessible online at https://www.who.int/disabilities/care/Rehab2030MeetingReport2.pdf?ua=1. 15 building for rehabilitation leaders and establishment of the SMEs were conducted to strengthen rehabilitation planning while capacity building for local health workers and provision of equipment to health facilities were conducted to strengthen rehabilitation implementation. The project has provided training for 93 doctors and technicians from three provinces of Thua Thien Hue, Quang Nam and Binh Dinh on the AT subject matters. Of those, 45 doctors and technicians were selected and trained to be experts in AT/AD service provision in the three provinces. In addition, a national expert group in AT/AD was formed and housed under Vietnam Association on Rehabilitation (VinaRehab) presenting doctors and policy makers from different regions of Vietnam. 3. Improving integration of rehabilitation into the health sector and strengthening intersectoral links to effectively and efficiently meet population needs: Establishment of AT units in hospitals, mobilization of local health workers in AT-D service delivery are major activities that have contributed to this priority of action. 4. Incorporating rehabilitation into universal health coverage: The MWL made important moves toward this priority of action through specific actions with the MOH to advocate for AT-D coverage under the national health insurance (more details were presented earlier under the health financing). 5. Building comprehensive rehabilitation service delivery models to progressively achieve equitable access to quality services, including assistive products, for all the population, including those in rural and remote areas: The MWL provides AT-Ds to everyone who need them in target areas. An action plan was developed specifically for girls & women with disabilities in AT-D service delivery. 6. Developing a strong, multidisciplinary rehabilitation workforce that is suitable for each country context and ensuring rehabilitation as a topic is included in all health workforce education efforts: Contribution to the 6th priority of action is seen through various capacity building activities and development of a graduate rehabilitation program. 7. Expanding financing for rehabilitation through appropriate mechanisms: As presented earlier, the project aims to cover AT-D and rehabilitation for people with mobility disabilities under health insurance. This was reported by local heath managers and workers as a sustainable financing mechanism for people with mobility disabilities given their limited affordability and lifetime of the project. 8. Collecting information relevant to rehabilitation to enhance health information systems, including system-level rehabilitation data and information on functioning using the International Classification of Functioning, Disability and Health (ICF): The MWL made direct contribution to this priority of action through establishment of e-records on AT-D and management software of people with mobility disabilities, which can be expanded and adopted by the MOH once it’s completed as a component under the national disability information system (DIS) or the health management information system (HMIS). 9. Building research capacity and expanding the availability of quality evidence for rehabilitation: The MWL joined with relevant USAID funded projects to collect evidence on AT-D situation and needs of people with mobility disabilities; it 16 also conducted another study on gender gaps in the use of AT-Ds. Availability of e-database on AT-D among persons with severe disabilities is expected to be part of the disability system and run by the health system from national and local levels to provide further quality evidence that can be used to improve AT-D service delivery 10. Establishing and strengthening networks and partnerships in rehabilitation, particularly between low-middle- and high-income countries: The MWL includes multiple local stakeholders on AT-D service delivery at national and sub￾national levels. The project also includes information exchange and sharing resources with other implementing partners working on disability of the USAID. In addition, the project collaborates with international partners, e.g. the Assistive Technology of Australia (ATA), RehabSkilled UK, and WHO, to conduct some activities. Availability of such a network and partnerships in AT-D service delivery under the project certainly made some contributes to the 10th priority of action. 3. Feedback and project impact from perspective of the beneficiaries & stakeholders Beneficiary and stakeholder feedback about the project and its perceived impact was mostly very positive. Persons with mobility disabilities who received AT-D assessment & AT-Ds and their caregiver or families were happy with the products and the project implementation. While positive attitudes about receiving free products is expected, some of recipients of the AT-Ds compared the MWL with their previous experience from other projects and reported two important differences: 1) ADs that they received from the MWL have better quality than similar ADs that they had before from other organizations and they came from trustworthy manufacturers; 2) the process of AD service delivery seems more professional with multiple steps and more tailoring to the needs of users, i.e. more person-centered, follow up after provision that they did not experience before. For example, a person with mobility disabilities reported that he received a wheelchair before but they (i.e. the donor) just came with local authorities to deliver the wheelchair and that’s all. Under the MWL, he got examined before getting the wheelchair; the wheelchair fits him well; and people (i.e. health workers) asked him about the wheelchair after that. So, he felt more care and more professionalism in the delivery of the wheelchair. Health professionals who received capacity building or training under the MWL also provided very positive feedback about the trainings. Doctors and technicians were trained for 12 months and 6 months respectively in rehabilitation, ICF and recently on AT/AD, OT and Multi-disciplinary approach in rehabilitation service provision by ATA and Rehab Skilled in the U.K. Reflection of the short-term trainees that the course was too short and they wanted to learn more suggests both need and success of the training. Some of the long-term trainees reported that their training was very challenging, but they valued the content and knowledge they had learned, 17 particularly for the ATA training courses for the AT/AD prescription and OT. Short￾term P&O related training would be the most challenging as it requires much longer time. Consequently, the interviewed assessors frequently reported that they were not comfortable with the P&O assessment and prescription. The interviewed AT-D assessors reported that having a P&O expert in the AT-D assessment team, as it was implemented in early stage of the project, was a good option when P&O assessment is needed. However, it was also noted that the proportion of people who need a P&O is significantly lower than that of those who need non-P&O or other AT-D devices. Further assessment is needed to identify an optimized delivery model where P&O are properly considered in the context of poor financial and human resources. Managers of provincial health & health facilities highly appreciated provision of rehabilitation equipment, training and technical support provided by IC. They reported that equipment, along with trained human resources, had improved quality of rehabilitation service delivery in their health facilities. More important, at least two health managers reported that they are well aware of the short-term nature of the project investment; hence, they included the project provided equipment as a part of their long-run planning on rehabilitation development in their health facilities. This uptake attitude is essential to assure effective use of the provided equipment and sustain the project impact. 4. Achievement of project’s goal and objectives A large range of relevant activities and their achievements as presented earlier certainly implied success of the MWL in achieving its three specific objectives on improving: 1) availability of quality AT-D; 2) capacity on AT-D service delivery; and 3) policy advocacy and information sharing. The MWL also has achieved its goal within its scope of work, and it has a great potential to achieve further successes if the scope of work is expanded. The MWL’s goal is “To improve persons with disabilities’ access to quality assistive technologies or device s that enhance their independence in activities of daily living and their participation in social activities in USAID target provinces and districts of Thua Thien Hue, Quang Nam and Binh Dinh”. Reflection of the project AT-D beneficiaries evidently showed that their access to quality AT-Ds has improved. Evidence on enhancement of independence in activities of daily living and participation in social activities was not clear; further analyses using the electronic AT-D database would shed lights on this issue. The predominant focus on around medical conditions of persons with mobility disabilities and lack of non-medical or social partners and OPDs in the project is a major constraint to achieve social participation. Expansion of the scope of work and improve active participation of social organizations and OPDs in AT-D service delivery is therefore essential to achieve this goal. IV. The AT-D service delivery system of MWL 18 1. Steps in AT-D service delivery This section describes the steps the project took in AT-D service delivery. In 1993 and 1994, the study by Horizontal European Activities in Rehabilitation Technology (HEART) collected information from 16 European countries and identified seven essential steps in AT-D service delivery: (1) Initiative – first contact; (2) Assessment – evaluation of needs; (3) Typology of the AT solution – choosing the appropriate type of AT-D; (4) Selection – selecting the specific device; (5) Authorization for financing – obtaining funding; (6) Delivery – getting the device to the user; (7) Management and follow up – continued support. 4 The position paper of the Association for the Advancement of Assistive Technology in Europe (AAATE) for its workshop in 2012 re-confirmed these seven steps with a slight change in wording, i.e. Typo was replaced by Selection of assistive solution – defining the individual AT programme.5 While using different terminologies, these are also the steps that were used by the project for AT-D service delivery. For each of the steps, using documentary evidence and stakeholder testimony, we describe critical and positive aspects of the process. a. Initiative Identifying the need for assistive products is an essential first step leading to positive change. The project has strengthened how the commune and village health workers contact potential AT-D users which has led to new needs being identified. There are potential gaps in how knowledge about AT-D and where it can be accessed has been developed in key consumer groups, especially persons with disabilities. Doing so would increase the role of users in making choices about when and how to access care and complement the more ‘top down’ arrangement currently in place. b. Assessment The project has built on existing arrangements for clinical assessment. This is consistent with the aim to integrate AT-D into health services by valuing the 4 Luc de Witte, Emily Steel, Shivani Gupta, Vinicius Delgado Ramos & Uta Roentgen (2018) “Assistive technology provision: towards an international framework for assuring availability and accessibility of affordable high-quality assistive technology”, Disability and Rehabilitation: Assistive Technology, 13:5, 467- 472, DOI: 10.1080/17483107.2018.1470264 5 AAATE/EASTIN (2012). Service Delivery Systems for Assistive Technology in Europe: Position Paper. Available online at https://aaate.net/wp-content/uploads/sites/12/2016/02/ATServiceDelivery_PositionPaper.pdf. 27pp. a) Initiative b) Assessment c) Typology & Selection d) Authorization e) Implementation f) Management & Follow-up 19 expertise of health providers and creating links between health services and AT-D supply. While persons with disabilities were the key informants during the assessment, caregivers were consulted during the assessment in case the persons with disabilities could not answer the questions, or especially for children with disabilities. However, testimony suggests consumer and family input to assessment (and the selection step that follows it) could be strengthened. For example, engaging users and families with health professionals to identify individual goals, living environment and occupations during AT-D provision and plans for other services and supports if they are needed, would enhance this step and increase user engagement in the process. The existing assessment form allows health professionals to do assessment and measurement of the rehabilitation needs based on the functional independence, however, the needs of the persons with disabilities are not addressed in a multi-disciplinary approach due to the lack of the technical expertise in this area. IC-MWL project is addressing this gap by deploying Assistive Technology Australia to provide AT training with application of multi-disciplinary approach in AT/AD prescription/assessment. c. Typology & Selection As described for step b, there are some gaps in how individual experience and choices inform the prescription choice of AT-D. This probably arises from factors including how professionals understand and use consumer knowledge in decision-making, a limited range of product options, and limitations in what consumers and families know about their options and how AT-D might help (or not) their goals. More information about the range of products and consumer knowledge would help improve this step. The limited range of product options is constrained by availability of quality AT-D in the market and quality control. Supporting quality control strengthening would help to increase the range of product options and indirectly improve this step. d. Authorization The project has been a good opportunity to formalize the authorization step. Currently, the project pays for AT-D in full. In our testimony, consistent with information from other contexts, recipients reported that they could not afford to pay for products. Some users choose to upgrade their products (for example, manual to electric wheelchair) at significant cost (VND 15 mil for electric wheelchairs), illustrating at least some demand for value-adding. This is a challenging situation for the local context. On the one hand, out-of-pocket costs are very high, representing significant risks to household budgets if products are not appropriate. On the other hand, value-adding (i.e. taking a financial contribution from consumers through the local service system) is not possible under current arrangements. Options for co-payment or appropriate support for decision making for personal AT-D purchasing have not been explored in the project. 20 e. Implementation The implementation step is crucial to test and adapt products and ensure any potential issues are addressed before the product is supplied to the end-user. The introduction of this step at commune health centers and homes of end-users is a very encouraging step towards integration of AT-D services within existing health services. There are opportunities to strengthen quality assurance processes at the implementation step. The process quality of provision varies from case to case. While some variation necessary to adapt to the specific situation, it introduces risks of missing key steps that might affect the value of the product or user safety. One key example is when family members collect products (rather than the end-users). Information about fitting adjustment, maintenance and safe use and follow-up were provided to family members and the end user at the handover session in the community. f. Management & Follow-up In the current model, follow-up is recommended at 1, 6 and 12 month periods. This is positive. At the moment, 100% of users were followed up, and whether follow-up is challenging for some groups (for example people with communication difficulties, from rural settings, etc). For simple products, online periodic verification was recommended by rehabilitation technologists as a way to reduce the need for face-to-face follow up, saving travel time and the time of users and providers. One area of service with opportunities for improvement is the systematic provision of accessories or consumable items associated with products. Examples include the rubber tips of crutches, wheelchair bearings and tyres, or straps and buckles that are important parts of many products. More emphasis on this sub￾component of follow-up might improve efficiency and minimize disruption to safe and effective product use. One suggested approach was a basic ‘hotline’ where users could get advice via phone, rather than making appointments. This might be a very practical solution. 2. Structure of the MWL’s AT-D service delivery system and its quality a. The structure From discussions with MWL’s stakeholders on their roles and activities, a summarized structure of the MWL’s AT-D service delivery system was outlined and presented in Figure 1. It includes all major stakeholders of the project, their activities and connections among them. 21 Figure 1: Structure of the MWL’s AT-D service delivery system 22 b. Quality of the MWL’s AT-D service delivery system Quality of the MWL’s AT-D service delivery system was assessed using six quality indicators that were recommended by the HEART study for quality assessment: accessibility, competence, coordination, efficiency, flexibility, and user influence.6 Concerning accessibility, positive aspects include MWL’s efforts to bring services to clients, and drawing on user-inputs from 2019 to shape the intervention. Many impacts will be sustained through an emphasis on system change and capacity￾building, to make sure the AT-AD services to be delivered at the rehab facilities while other areas are at some risk (see sustainability section, below, for elaboration). However, there are opportunities to review activity focus, moving from a disability￾only emphasis focusing on persons with long-term mobility impairments, to providing AT-D as a broader ‘Universal Health Coverage’ intervention. There are also opportunities to improve accessibility through greater emphasis on community knowledge of services, recognising that awareness of needs and available services are the critical first step in accessing health care, including AT-D services. The competence of the service and its staff was explored. Training provided through the project was received very positively by participants, and helps ensure long-term benefits. Participants requested additional training, which is a reflection on their quality and relevance, and represents an opportunity for further impacts. A specific area for further emphasis is P&O assessments: Doctors and technologists reported low-confidence in contributing to assessments, which could further strengthen the quality of services. Despite availability of quality management methods and generic service standards in O&P and other mobility AT-D, respondents noted that protocols in areas including AT-D information and user feedback were not implemented effectively. MWL is currently developing standards in these and other areas, which is positive. Overall, services coordination is a strength. When we explored how people and information move between levels of services and different professional stakeholders, we noted good coordination across and with levels, between stakeholders, and between the different service steps. Nonetheless, links between levels (province, district, commune) were less consistent than other areas, and an important opportunity for improving coordination. This evaluation provided scope for a modest exploration of service efficiency. MWL provides full payment for AT-D services, which is positive, but challenging to sustain, and does not reflect the opportunity for some clients to co-pay for services. There are opportunities to explore how alternative cost-recovery arrangements could 6 AAATE/EASTIN (2012): ibid. 23 be introduced in the context of local regulations and norms about health payments. In terms of systematic approaches to monitor quality and costs, within the context of the current project, this has been provided by IC, but will need to be systematized within service management arrangements in order to be sustained – with this regard, the project starts to shift the community services to the facility-based services, meaning to provide AT/AD services at the rehabilitation facilities where possible. An additional perspective on efficiency is the emphasis on AT-D with less emphasis on accessible housing (30 households in the current project) or other adaptations, or for personal assistants, which could potentially amplify the benefit of AT-D (and other services) at low cost. A small pilot on personal assistants and respite care was expected to start at the time of this evaluation. The project has several positive effects on flexibility, particularly ensuring more and better choices to beneficiaries compared with before the project. This is central to the aim of person-centred care. Within the scope of this evaluation, user feedback was positive and strongly suggested better engagement with clients and flexible services that met their needs, but further information about how that affected clients (and who might have missed out on those benefits) is beyond the scope of the current evaluation. On the other hand, user influence on the project overall and the systematic changes it has sought to introduce were limited. Representation of persons with disabilities in decision making and implementation of service changes could have been strengthened. Another perspective on flexibility is how MWL has aimed to support local P&O service provision to improve the responsiveness to client needs. While this is a positive overall, whether it is cost￾effective and sustainable (or if there are other issues like staff satisfaction, quality issues, etc) and we can see the project is shifting the services from community to the health service could be one step to sustain the project activities and model. c. Other observations and issues relating to the MWL system The project builds on existing health system and received great support from local health departments. Provincial Health Departments show great support and they delegate the provincial rehabilitation hospital to be focal point to implement project activities, as well as to provide technical advisory to the DOH regarding the technical aspects of the project. This ensure the project transfer of skills and knowledge to the local health professionals and its sustainable application of the project in the long run. The project at the moment is trying to restructure the service delivery approach, shifting the services from community to health facilities, and build it into the health system for three provinces, to ensure the sustainability of the project when the funding ends. Finally, we note that the M&E system, including its interface with DIS, was incomplete at the time of the evaluation, but still being developed. 24 3. Project sustainability a. Sustainable factors Evidence of AT-Ds that were distributed to persons with mobility disabilities will be used after the end of the project, but will need to be repaired and replaced over time. Human resources whose capacity was improved thanks to the project will be sustained after the end of the project. Through trainings and participation, project participants have improved their awareness of rehabilitation practices and systems. More importantly, the project improved awareness, built knowledge and changed behaviour of persons with mobility disabilities in using AT-Ds, and that momentum will continue after the end of the project. AT-D provided through the project will continue to be used in the short-medium term, with the likelihood repairs will extend the life of products and consumers will be known to service providers and supported to access replacements when required. Health facilities continue to use the provided rehabilitation equipment, and they are expected to be valuable for many years. Long term awareness and behaviour change will continue to positively benefit persons with disabilities and others who might benefit from AT-D in the medium or long term. There is better awareness of AT-D quality provision of individualized AT￾D. Project AT-D recipients have better awareness of how to use and maintain their product, which will continue to benefit them. Among local health & social support workers, there is improved awareness of AT-D and their provision. Features of the project intersect with and will continue to influence each of the 6 Health Systems ‘Building Blocks’: health workforce, service delivery models, AT Unit & AT service provision strengthening, AT information system, financing for AT￾D via health insurance, AT leadership & governance. b. Challenges to project sustainability Despite the sustainable effects of the project described above, there are some limitations and challenges for sustaining the impacts of the program. The most obvious concerns financing the features of the project that are not financed locally. In the project, assessments were conducted in homes at various locations. This was a positive aspect, reducing burden and cost to the consumers, but is unlikely to persist without additional financing allocated to that model. It was reported that assessments were conducted at the commune health centers for those who do not need assessment of physical environment of their place of living. The use of mobile technology, i.e. tele-assessment with assistance of a local volunteer, as observed in a few assessments would be a cost-effective alternative, but its effectiveness in comparison to in-person visits should be assessed further. 25 The current arrangements for full payment of AT-D from the provider is an obvious cost pressure. As no financial contribution can be taken from clients, there is no opportunity to either offset product costs or provide choice through ‘value-adding’ arrangements in which consumers can pay for more costly products (with greater functionality, design choices, brand choices etc). It is unclear where resources for maintenance and follow-up will come from after the project concludes. The project plans to close out the cases with support and hand￾over to commune health center for follow up when the project ends. In addition, information about repair and replacements will stay with the commune health center so that they can provide to persons with disabilities in need and or refer to a place of service availability. Resources include both costs of repairs or replacements and HR for follow up. Recommendations to improve the sustainability of this component include a technical phone hotline or simple technological solutions to target resources where they are needed the most. V. Conclusions In summary: • The project is on-track. • It nearly achieves or has achieved most of its output indicators, and likely to achieve more than its commitment. • The project has contributed to rehabilitation and health system strengthening through its contributions to the six building blocks of the health system, especially in AT-D service delivery and health workforce strengthening. It also aligns with the 10 priorities of action under the WHO’s 2030 Rehabilitation Initiatives. • The project received very positive feedbacks from the beneficiaries and stakeholders. • An AT-D service delivery model with 6 steps were established as a good initial model that has potentials to scale-up. As expected given its early stage of development, this model has limitations that are mainly related to cost￾effectiveness and project sustainability that could be improved over time. Some of these limitations were presented earlier and recommendations are being presented in the next section; further studies on the 6 steps of service delivery are needed to fully assess this issue. • Given its scope of work, the current AT-D service delivery system heavily focuses on people with severe mobility disabilities and medical side of service provision, which is crucial but it is not enough for participation and inclusion of people with mobility disabilities. Recommendations to expand the scope and gain further achievements in a long-run are being provided in the next section. • Project impact will be sustained given its support to not only direct service provision but also health system strengthening. The current free-for-service 26 financing mechanism is helpful for many people with mobility disabilities, but it is also a challenge to sustain the project and increase its coverage. VI. Recommendations Based on the presented findings from the evaluation, the following recommendations are proposed to development of the next phase of this project: Strategic recommendations Expand the current disability-specific approach and adopt twin-track approach, which includes both disability-specific and mainstreaming. By design the project focuses on providing AT-D to address physical and functional difficulties of the target population. Whether the project should expand to broader social inclusion is a strategic choice – but clearly, there are missed opportunities to integrate and align the clinical services with broader social inclusion strategies, including labour, education and other sectors. While it is very important to improve medical conditions to maintain individual’s functioning and independence, it is no less important for AT-D provision to facilitate participation of persons with disability to restore or enjoy their life, e.g. helping them to build or rebuild social connectedness to family and community, to get back to work or get a job that keep them happy, to join community activities as other people. Encourage active participation of non-medical stakeholders to work alongside health professionals. The current main stakeholders of the project are all in health sector at all levels, from national to commune level. Ministry of Labors, Invalid and Social Affairs (MOLISA) and its departments have limited involvement to the project. Their biggest role would be through limited support of AT-D provision at Social Protection Centres that are under their management. MOLISA and its department do not get involved to any of the six steps of AT-D service delivery. Ministry of Education and Training (MOET) and its departments are absent in the project. It was observed from a few sessions of AT-D assessment, typology and selection that they were conducted by health practitioners only; consequently, it’s not strange that these sessions were mainly about medical conditions and a little bit about functioning around the house. Discussion about job or employment opportunities, education, participation in cultural activities, sports, leisure times, and how quality of life of the persons with mobility disabilities would change with a new and/or improved AT-D were totally missing. It is strongly recommended that the project will move beyond provision of quality AT-D to their impacts. Active participation of non-medical partners like local social workers, socio-cultural committees, education services, and local community leaders in different steps of AT-D service provision (especially assessment, typology and selection, and implementation) are crucial and should be considered to reach that goal. This process requires not only encouragement of non-medical stakeholders to 27 participate in the project activities but also awareness raising and empowering them to support persons with mobility disabilities. Lean more on partnership approach (where users a key role) than the directive approach (where professionals decide and user has little to say) in AT-D service provision. Moving toward person-centred approach has been appreciated by the AT-D users who are the project beneficiaries when they compare AT-D provision under the project to their previous experience (e.g. users are much appreciated to have assessment before selection of AT-D, personalization of AT-D, and follow-up after the AT-D provision as they did not go through so many steps before). This direction should be pushed further to achieve even greater successes. Involve persons with mobility disabilities right from the first step of AT-D service provision, i.e. initiative, and throughout all of the steps is highly recommended. Currently, persons with mobility disabilities are asked to choose their AT-D (in the selection step), but that would be their biggest involvement to the AT-D service delivery. While bringing services to people with mobility disabilities is necessary, it is also important for persons with mobility disabilities to aware of their needs, to know where to make the first contact to address their needs, and to actively seek effective solutions to address their needs. It is arguable that persons with mobility disabilities is not capable of participate more in the AT-D service provision, but that also means awareness raising and empowerment of people with mobility disabilities is needed, and highly recommended in the next phase, to move toward person-centred and partnership approach. Ensure that the AT-D service delivery system is driven by people with mobility difficulties for people with mobility difficulties. This is actually just another angle of strengthening person-centred approach, which promotes participation of persons with disabilities as not only the beneficiaries but also service providers. This is particularly important for people with disabilities to achieve the moto “Nothing about us without us”, that has been used by Organizations of People with Disabilities (OPD). Encouraging people with disabilities to participate in project activities as a doctor or technologist, or developing their career in that path may take time to get an impact but it’s essential. Consultation with the OPDs in all steps of AT-D service delivery will be a feasible entry point. Expand target population to include not only mobility impaired people but also everyone with mobility functional difficulties and general population. This is crucial to push for AT-D to be understood as part of universal health coverage (UHC) rather than only and always a disability response. Along that line, awareness raising on AT-D is important for not only persons with mobility disabilities but also public. Apply multiple AT-D service delivery models to meet diverse needs of people with mobility difficulties. The project currently provides AT-D services at home and commune health centers. The project management board is planning to move the AT￾D service delivery sites to health facilities at district and provincial level. While that would improve quality of service delivery as more health professionals with different expertise can assess a client, it is a barrier for many persons with mobility disabilities 28 who cannot access the service at district or provincial level (because of mobility constraints, travel and indirect costs, availability of personal assistant, etc.); further studies on accessibility to health facilities is needed. In addition, it’s recommended to the project to deliver AT-D services at not only home and health facilities, but also schools, workplaces, cultural and leisure centers, sport venues and other places. Project-specific recommendations The following are recommendations that are directly relating to specific activities of the project: • Continue to support the health workforce given their positive feedback, and potential contribution to other similar AT-D provision programmes of the Government and USAID or other donors in the future, and sustainable long￾term contribution to health system strengthening. For the same reasons, scaling-up of this activity is also strongly recommended. In the meantime, continue to finalise the Physical Medicines and Rehabilitation Program would lead to a comprehensive strategy for building the national multi-disciplinary rehabilitation workforce. • Complete the e-medical records & electronic M&E system on AT-D and exploit them. This activity was commenced late in the project cycle but it is crucial to monitor project progress, efficiency and effectiveness of the investment. As this system is still absent in Vietnam and hence it will be useful for not only the project but also similar interventions in the future. • Explore inclusion of information on persons without disabilities in the AT-D database if possible to learn about needs and unmet needs of AT￾D among the not only persons with mobility disabilities but also population with mobility difficulties and general population. • Once the AT-D database is available, make use of that through researches on AT-D service delivery; use research findings for amending project activities, developing and revising approach and strategies to deliver AT-D, policy advocacy, and supporting evidence-based policy making. • Develop scenarios for each AT-D service delivery step to meet diverse needs of people with mobility difficulties while minimizing the cost and optimizing cost-effectiveness of the AT-D service delivery. While this cannot be done comprehensively in the scope of this evaluation, some observations and reflections were recorded: • For the initiative or the first contact: the project may want to empower people with mobility disability and encourage them to actively seek the service when needed; the service delivery system or local health workers only approach or bring the services to those who have severe mobility disabilities and cannot seek the service themselves. • For the assessment: the project is currently providing assessment at community, and planning to move toward provision of assessment at district health centres. It’s recommended to the project to apply multiple channels of assessment, e.g. continue to provide community- and 29 homes-based assessment to people with severe mobility disabilities if appropriate (e.g. some of them may need to be assessed at district health centres for proper health examination), while encourage and/or support other people to get assessment at district health centres. Development of simple tools for assessment by local health workers or self-assessment should also be considered for those with mild level of mobility disabilities. Applying tele-health for remote assessment with certain groups of people with mobility disabilities should also be considered for cost-effectiveness of the service delivery. The classification of people with mobility disabilities should be advised by not only a local health worker but also a social worker who know about living standards and social context of the family of the person with mobility disabilities. • Explore possibilities of decentralization of AT-D service provision for simple AT-Ds, e.g. build capacity and authorize local health and social workers to conduct typology and selection, authorization, implementation, management and follow-up of simple AT-Ds while centralize AT-D service provision for other (more sophisticated) AT-Ds. Development of clear protocols and guidebooks for service provision of simple AT-Ds should be a prerequisite of this process. Tele-health could be used for technical support and quality control (see next recommendation for further details). • Explore the use of technology (e.g. SMS and/or app). This is especially important under the current COVID-19 context where social distancing is encouraged to avoid spread of this communicable disease. In addition, distant communication could help to reduce travel cost and reach more people, especially those living in remote areas, more frequently at lower cost and more convenience. The wide spread geographically of the stakeholders (i.e. project and health managers locate in Hanoi and the 3 provinces; service providers locate in multiple location in the 3 provinces, Da Nang, Hanoi; AT-D producers mainly locate in Ho Chi Minh city; clients or AT-D users locate in the 3 provinces) makes this even more important for timely and cost-effective service delivery. Specific ideas that arise during the evaluation include: • Develop and pilot online AT-D service delivery models, for certain steps and/or AT-Ds. For instance, develop and pilot an online periodic verification model for simple AT-D. • Develop and pilot mixed-models (i.e. online & offline simultaneously depending on specific service and characteristics of the clients) to improve cost-effectiveness of service delivery. • Develop and pilot an e-platform for AT-D service delivery where clients, service providers, health managers and other stakeholders can initiate the six steps of AT-D service delivery online and then follow-up online or off-line due to the situation. This process can also be improved by the use of machine learning and artificial intelligence (AI). For instance, the e-platform allows a client to actively make the first contact with the service delivery system, i.e. the initiative, or make a request for maintenance online instead of passively waiting for the local health 30 workers to reach them. The e-platform also allow service providers to conduct the assessment and other delivery steps (i.e. typology and selection, implementation, management and follow-up) online with certain groups of clients or certain AT-Ds. Health managers could use the e-platform to seek information and monitor the progress online, and provide e-authorization. • Establish and pilot a hotline to follow-up those who receive an AT-D soon after they receive it (i.e. within 3 to 5 days) to provide timely technical support to AT-D users while a full-scale e-platform is not yet available. It is expected that this hotline could help to identify cases that need to follow up immediately (e.g. those whose family members received an AT-D for them and used it incorrectly as the family members did not guide them properly; or those who may need re-fitting or adjustment) to avoid ‘giving-up’ that AT-D before the first follow-up in 1 month. Note that the 1-month, 6-month and 1-year follow-ups are working well; hence, this 3-5 day should not be a replacement for 1- month follow-up but an addition to that. • Document success stories in AT-D service delivery to use as role models and advocacy. This is strongly recommended as simple life-changing stories of few individuals, especially among working-aged adults, that were heard during the intervention could be served as powerful tools for policy advocacy and effective influence to other people with mobility disabilities. • Clearly distinguish between decentralization of service provision and local production. Conduct an assessment to deeper explore feasibility, potential impacts and sustainability of local workshop and production. Interviews with experienced workshop managers showed that establishment of local workshops is not too difficult but maintaining them after external financial support is a great challenge due to various factors including financing regulations, human resources or capability of service providers, volume and affordability of those who need a P&D, and public awareness of AT-Ds. Self￾maintenance of AT-Ds via local resources, e.g. a motorbike repair service or a local mechanic or a local carpenter, are not common; Improving awareness and practice of both AT-D users and local resources on maintenance of AT-Ds via local resources should be promoted to improve lifetime and effective use of AT￾Ds. • Consider multidimensional disability and make sure that no one is left behind. The project already considered gender aspect of disability, and it is expected that other aspects, such as poverty, education, and employment, will also be included. This would involve AT-D service delivery in not only health facilities and communities but also other settings such as schools and workplaces. • Conduct a research to explore affordability, willingness to pay for AT-D, feasibility and applicability of other financing schemes, e.g. co-payment & service or full-payment for certain groups of people with mobility difficulties, for effectiveness and sustainability. • Continue to strengthen networking and expand collaboration with international partners. The project has multiple local and international partners 31 and this has been an important factor contributing to its success given the lack of resources and skills in AT-D service delivery locally. Insights from the ATA in AT-D assessment and selection sessions for example clearly brought new perspectives, that are broader than medical perspective and more align to the current international practices, to the local assessment team. Such practices and collaboration will be fruitful to not only the project but also development of the AT-D service delivery system in general. 32 Appendixes Appendix 1: General Interview Guide for mid-term evaluation ● Progress & achievements to date? o What have been completed? What were delayed or late? Why? o Impacts of late commencement & Covid-19? ● Remaining activities & forecast? o Any expected delay? o Any anticipated incompletion? o Mitigation measurement? ● Implementation issues? o Strengths, weakness, challenges & opportunities of this project? o Risks or potential risks? ● Project impacts to date? o What are the most significant changes you have experienced thanks to the project? o What are the most significant contributions of this project? o What do you like the most about this project? o What do you dislike the most about this project? ● Effectiveness? o Is there anything you would do differently? Or do better? o Any changes you may want in the last year? ● Efficiency? o Is there any activity/thing that you would like to add or remove? ● Sustainability of project impacts after the end of the project? o Activities that would be continued after the end of this project? o Materials that could be used after the end of this project? o Long-term impacts? o Potentials to mobilise private sectors for project activities? ● Coordination & collaboration? o With local & existing initiatives? o With local authorities and relevant stakeholders? ● Budget reallocation needs?