This publication was produced independently by Melissa Chiappetta, Denise Buchner, Salika Khoonbarthao, Thatsaphone Songbandith, Catherine Villada, Sharon Meged, and Kwabena Boampong of Social Impact, Inc., at the request of the United States Agency for International Development. PHOTO CREDIT: QLA FINAL REPORT USAID/LAOS OKARD MID-TERM EVALUATION July 30, 2021 i | EVALUATION FINAL REPORT USAID.GOV TABLE OF CONTENTS ACRONYM LIST III EXECUTIVE SUMMARY 1 INTRODUCTION 8 OKARD ACTIVITY INTRODUCTION 9 OKARD ACTIVITY OVERVIEW 9 OKARD ACTIVITY PARTNERS 11 EVALUATION OVERVIEW, PURPOSE, AND QUESTIONS 14 EVALUATION OVERVIEW 14 EVALUATION PURPOSE 14 EVALUATION QUESTIONS 14 EVALUATION METHODS AND LIMITATIONS 15 EVALUATION METHODS 15 LIMITATIONS 15 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS (FCR) 17 EVALUATION QUESTION 1 17 EVALUATION QUESTION 2 23 EVALUATION QUESTION 3 30 EVALUATION QUESTION 4 56 EVALUATION QUESTION 5 57 EVALUATION QUESTION 6 71 EVALUATION QUESTION 7 77 EVALUATION QUESTION 8 81 ANNEX I: EVALUATION METHODS 88 ANNEX 2: EVALUATION DESIGN MATRIX 93 ANNEX 3: CASE STUDY – STORY OF CHANGE AS A RESULT OF CBID ACTIVITIES 98 ANNEX 4: BIBLIOGRAPHY 103 ANNEX 5: DATA COLLECTION INSTRUMENTS 106 ANNEX 6: SCOPE OF WORK 175 ii | EVALUATION FINAL REPORT USAID.GOV LIST OF FIGURES AND TABLES Figures Figure 1: Okard Results Framework...........................................................................................................................10 Figure 2: Okard Management and Partnership Structure ......................................................................................12 Figure 3: Okard Activity Timeline................................................................................................................................77 Tables Table 1: Education Levels by Sex in Xayphouthong and Kham ............................................................................21 Table 2: Summary of Critical Assumptions, Validity, and Changes in Context ................................................29 Table 3: Okard Activity Annual Progress Report Results, Inception to Year 4, Quarter 2..........................32 Table 4: Okard Activity Components Successfully and Not Yet Successfully Implemented to Date .........53 Table 5: Difficulty of Functioning Scoring Matrix for Beneficiary Selection.......................................................58 Table 6: Affordability of Basic Needs Scoring Matrix for Beneficiary Selection...............................................59 Table 7: Type and Level of Difficulty of Screened and Selected CBID Beneficiaries by SES .........................60 Table 8: Screened versus Selected CBID Beneficiaries by Level of Difficulty, SES, and Cause of Impairment ..............................................................................................................................................................................................63 Table 9: Screened versus Selected CBID Beneficiaries by Level of Difficulty, Sex, and Cause of Impairment ..............................................................................................................................................................................................67 Table 10: GIDAP Indicators ..........................................................................................................................................72 iii | EVALUATION FINAL REPORT USAID.GOV ACRONYM LIST ADS Automated Directives System AfA Association for Autism AN Affordability of Needs AMELP Activity Monitoring, Evaluation, and Learning Plan ARMI Association for Rural Mobilization and Improvement AT Assistive Technology CBID Community-Based Inclusive Development CDAP Capacity Development Action Plan CMR Center for Medical Rehabilitation CPDTC Continuous Professional Development Training Center COPE Cooperative Orthotic and Prosthetic Enterprise CSO Civil Society Organization DHR Department of Health Care and Rehabilitation DID Disability Inclusive Development DIT Diversity and Inclusion Training DLSW Department of Labor and Social Welfare DMAS Disability Mainstreaming Advisory Service DPO Disabled People’s Organization EE Economic Empowerment EQ Evaluation Question ET Evaluation Team EQUI Evaluation, Quality, Utilization, and Impact FCR Findings, Conclusions, and Recommendations FGD Focus Group Discussion GESI Gender Equality and Social Inclusion GIDAP Gender and Inclusive Development Action Plan GoL Government of Lao PDR HI Humanity and Inclusion IE Inclusive Education IEC Inclusive Education Center IGA Income Generating Activity INGO International Non-Governmental Organization IRB Institutional Review Board ITT Indicator Tracking Table KII Key Informant Interview Lao PDR Lao People’s Democratic Republic LDPA Lao Disabled People Association LGBTQI Lesbian, Gay, Bisexual, Transgender, Questioning, and Intersex LWVF Leahy War Victims Fund M&E Monitoring and Evaluation MEL Monitoring, Evaluation, and Learning MHPSS Mental Health and Psychosocial Support MLSW Ministry of Labor and Social Welfare MOES Ministry of Education and Sport MOFA Ministry of Foreign Affairs MOH Ministry of Health MOU Memorandum of Understanding MTE Mid-Term Evaluation NCDE National Committee for Disabled People and Elderly iv | EVALUATION FINAL REPORT USAID.GOV NPA Non-Profit Association OPD Organization for Persons with Disabilities PD Project Director PIRS Performance Indicator Reference Sheets PLOCA Participatory Local Organisational Capacity Assessment PRC Provincial Rehabilitation Center QA Quality Assurance QLA Quality of Life Association RDMA Regional Development Mission for Asia RTI Research Triangle Institute SBCC Social and Behavior Change Communication SE Stakeholder Engagement SES Socio-Economic Status SHG Self-Help Groups SI Social Impact SOW Scope of Work STARS Systematic Assessment of Rehabilitation Situation SVK Savannakhet Province TEAM Training, Economic Empowerment, Assistive Technologies and Medical Rehabilitation Activity TOC Theory of Change TL Team Lead TVET Technical and Vocational Education and Training USAID United States Agency for International Development USG United States Government UXO Unexploded Ordinances VOT Victims of Torture Fund WCC Women-Centered Care WEI World Education Inc. WGSS Washington Group Short Set WHO World Health Organization WRA Weapons Removal and Abatement VTE Vientiane XKH Xieng Khouang Province XPT Xayphouthong District Y1 Year 1 1 | EVALUATION FINAL REPORT USAID.GOV EXECUTIVE SUMMARY INTRODUCTION Persons with disabilities in the Lao People’s Democratic Republic (Lao PDR) face multiple challenges that limit their ability to achieve their full potential in society. While overall development indicators for the country are improving, persons with disabilities continue to experience higher rates of poverty, poorer health outcomes, lower education levels, and are often excluded from equal participation in broader society.1 To support the Government of Lao PDR (GoL) in addressing inequity and the numerous challenges facing persons with disabilities in Laos, USAID awarded World Education, Inc. (WEI) a five￾year, $15 million-dollar cooperative agreement to implement the Okard Activity in October 2017. The USAID Okard Activity has three main components: Health, Economic Empowerment (EE), and Stakeholder Engagement (SE). In addressing the three components, WEI uses a two-tiered approach to integrate the systems level (Tier 1) and the individual/community level (Tier 2). WEI identified three critical assumptions in Year 3 that underpin its theory of change, namely: 1) the Lao operating environment is stable and conducive to the Activity; 2) GoL administrative requirements are requested and processed as expected; and 3) all needed stakeholders are willing and able to engage with the Activity as expected. WEI engages with multiple partners to implement the Okard Activity, including its main government partners in the Ministry of Labor and Social Welfare’s National Committee for Disabled People and Elderly (NCDE) and the Ministry of Health (MOH) Department of Healthcare and Rehabilitation (DHR) and Center for Medical Rehabilitation (CMR)—all of which have signed agreements or Memoranda of Understanding (MOU) to receive in-kind support to implement their own scopes of work under the activity. WEI also partners with Humanity & Inclusion (HI), another international non-governmental organization, and the following local sub-recipients: the Cooperative Orthotic and Prosthetic Enterprise (COPE), the Quality of Life Association (QLA), the Association for Rural Mobilization and Improvement (ARMI), the Disability Mainstreaming Advisory Service Center (DMAS), and the Association for Autism (AfA). EVALUATION OVERVIEW AND PURPOSE The United States Agency for International Development’s (USAID) Laos Mission contracted Social Impact (SI) to conduct a mid-term evaluation (MTE) of the USAID Okard Activity between February and July 2021. The purpose of this MTE is to provide feedback to USAID, the GoL, WEI, and its partners to review activity progress and systems to date with the objectives of guiding strategic direction and planning for the remainder of the activity in order to capitalize on the approaches, strategies, and systems that are working and improve the ones that are not. EVALUATION QUESTIONS The evaluation questions (EQs) for the MTE are as follows: 1. To what extent have the USAID Okard strategy and interventions been relevant to the goals it intends to reach? 2. Are the context and critical assumptions still valid? What have been key changes, if any, that affect the anticipated achievements positively and/or negatively? 2 | EVALUATION FINAL REPORT USAID.GOV 3. What components of the Activity have been successfully and/or unsuccessfully implemented according to the approach? Why? 4. To what extent has USAID Okard been effective in achieving its expected results, particularly in terms of building/strengthening the systems as described in the design document (system￾centered)? 5. To what extent has USAID Okard been effective in achieving its expected results, particularly in terms of teaching target beneficiaries? Which groups are underserved, if any, among all targeted groups, and why? (People-centered?) 6. To what extent has USAID Okard been effective in achieving its expected results, particularly in terms of gender equality and other inclusion? 7. Does USAID Okard have the appropriate management system and partnership structures to accomplish its objectives? 8. Do USAID Okard partners have the right systems and capacity to fulfill their scope of work? What systems and/or capacities are currently missing in order to fulfill the scope of work? EVALUATION METHODS AND LIMITATIONS To address the above EQs, the SI evaluation team (ET) conducted an extensive document review, 33 KIIs, 13 group interviews, and a quantitative analysis of community-based inclusive development (CBID) beneficiary data. In total, the ET spoke with a total of 76 respondents from USAID, WEI, HI, local Okard sub-recipients, GoL officials, and beneficiaries about their experiences with the Okard Activity. Amongst these respondents, 36 individuals were male, 34 were female, and eight were persons with a disability. In addition to other evaluation limitations discussed in the body of the report, given challenges to movement as a result of the COVID-19 pandemic, the ET had to conduct all interviews and group discussions remotely, which both meant a significant reduction in the originally planned KII and focus group discussion sample and may have resulted in some selection bias. FINDINGS AND CONCLUSIONS Given the length of findings and the fact that conclusions essentially summarize the key findings in response to each EQ, the ET has chosen to present conclusions only in this Executive Summary. They are presented by EQ below. EQ1 – Given evaluation limitations, the ET does not have enough evidence to confirm that the Okard strategy and interventions have been relevant to the activity’s intended goals. However, initial evidence suggests that the strategy and interventions have been relevant and appropriate, with the one possible exception of technical and vocational education and training (TVET) activities, which would benefit from either: 1) reconsideration regarding their relevance at the present time given the low levels of education amongst persons with disabilities in Laos, or 2) increased coordination and cooperation with the TVET Department within the Ministry of Education and Sport. While international standards suggest that supporting TVET for persons with disabilities helps to strengthen systems and ensure inclusion and opportunity for persons with disabilities, given the long list of priorities for USAID Okard, the Agency should consider whether the focus on TVET activities makes sense at this time or if TVET activities might better be pursued once there is increased demand for it amongst persons with disabilities. EQ2 – The context and critical assumptions do not appear to still be valid. Rather, the context has been greatly affected by COVID-19, unexpected administrative delays (especially the lengthy MOU process), and the capacity and willingness of all stakeholders to engage fully with the activity. The COVID-19 3 | EVALUATION FINAL REPORT USAID.GOV pandemic is not over in Laos, and it is likely to continue to have an effect on the Okard Activity. Further, administrative delays may be inevitable, as the MOH remains busy addressing the pandemic and working on other tasks. While not all stakeholders were willing and able to engage in the first three years of the activity, Okard has successfully transitioned to new partners and resolved issues with remaining partners such that all partners appear to be and indicate they are now willing and able to fully engage. The one exception to this rule may be with regards to the GoL, which WEI staff report have not yet been able to take ownership over activities and implement their agreed-upon scope of work as expected. EQ3 – The ET have summarized components successfully and not yet successfully implemented to date in the table below, focusing on the major activities outlined in the Results Framework. Please note that given that relatively few of the Activity targets have been met to date, as a result of the numerous challenges discussed under EQ2 findings above, the ET considered a component successful if good progress has been made since barriers were removed, as well as if group interview and KII respondents reported improved services, capacity, and well-being. Further, the ET has not classified some activities, such as self-help groups, for which WEI and HI report some progress, but the ET did not have enough independent evidence to verify. ACTIVITIES ON TRACK FOR ACHIEVEMENT OR ACHIEVED (BY COMPONENT) ACTIVITIES AT RISK OF NOT BEING ACHIEVED (BY COMPONENT) Health In-service training of health service providers on rehabilitation Long-term capacity and expertise in rehabilitation (by developing more rehabilitation specialists) Increased knowledge and awareness of mental health challenges and compassionate care amongst service providers Community-level expertise in mental health screening, and follow-on training for District and Provincial Health Officials to help solidify their knowledge Provision of rehabilitation services and AT to CBID beneficiaries Better coordination with DHR and CMR on the purchase of AT devices for CBID beneficiaries to ensure sustainability Strengthening and integration of the Lao rehabilitation system by ensuring a key lead department for rehabilitation and ensuring funding is available to sustain rehabilitation and AT services for persons with disabilities either through direct government support or improved National Health Insurance Economic Empowerment Accessibility audit for one TVET institution and plan for data collection Capacity development for TVET service providers, and a strengthened TVET system overall Provision of advice on inclusion and accessibility to international and local NGOs and private companies Sustainability of DMAS (findings and recommendations discussed under EQ8) Formalization of DMAS Employment of persons with disabilities in paid jobs Long-term systems developed to sustain the employment pipeline GoL officials, caregivers, and beneficiaries trained and reporting improved knowledge in key IGAs Delivery of actual IGA inputs, though these appear to be primed for delivery soon Removal of key barriers to IGAs One pending construction waiver and UXO clearance request 4 | EVALUATION FINAL REPORT USAID.GOV ACTIVITIES ON TRACK FOR ACHIEVEMENT OR ACHIEVED (BY COMPONENT) ACTIVITIES AT RISK OF NOT BEING ACHIEVED (BY COMPONENT) Stakeholder Engagement Capacity built amongst District and Provincial Government officials NCDE capacity building to improve coordination across the government for disability inclusion Support in the development and dissemination of the Disability Law and other related laws and strategies Development of Ministry-level policies and more specific policies for education, transportation, employment, and gender ARMI, COPE, DMAS, and QLA capacity increased Additional DPO capacity development not yet achieved (but planned) and further capacity development of DMAS and COPE pending; DMAS study tours have also not happened SBCC campaigns and improved service provider and community awareness Formalization of a DPO network to provide feedback on policies and strategies and advocate to service providers EQ4 – Many of the findings for EQ4 are detailed under EQ3 since EQ3 focuses on successes and challenges across the activity as a whole. As discussed under EQ3, the Okard Activity has contributed to the development and dissemination of policies focused on the rights of persons with disabilities in Laos, which has come a long way toward strengthening the system. Further, the activity has been very successful in the provision of trainings to increase the capacity of the GoL with regards to health, especially mental health and psychosocial support, and EE services for persons with disabilities. However, the Laos rehabilitation system is still in need of reform and capacity development (as recognized by the MOH in their 2013-2025 Health Sector Reform Strategy) as is the TVET system. Finally, some stakeholders note concerns over the sustainability of the CBID model after the activity ends, given funding challenges (to pay for CBID facilitators, for instance). EQ5 – WEI, HI, QLA, and ARMI staff do not appear to have targeted CBID beneficiaries exactly as described in the Okard Activity’s Cooperative Agreement. As such, while Okard has been successful in reaching some of its targeted beneficiaries, including some victims of UXO and, according to CBID facilitators, a higher percentage of the elderly—listed as a highly vulnerable group—it appears to not have served the full number of persons with disabilities stemming from UXO accidents. Additionally, while not underserved, women do not appear to have been prioritized as suggested in the cooperative agreement. Further, since Okard did not collect details on ethnicity or sexual orientation during screening (the latter due to sensitivities of including such questions), these criteria could not be taken into account in the targeting of beneficiaries. EQ6 – While progress is slow toward reaching overall targets for gender and inclusion within the Okard Activity, the activity has demonstrated success with regards to equal or proportional inclusion of women in all its activities for which indicator data are available. There is less evidence that other targets for inclusion are met, which is evidenced by proportional targets that are not met for children. Moreover, there is no data to demonstrate how the project is targeting different age groups, including young women and men, or people from different ethnic groups. While the project may not be a “gender project” and it may endeavor to be inclusive of individuals across the lifespan, there is value to understanding how the activity affects men, women, boys, girls, and people of different ethnicities across all age groups. Finally, 5 | EVALUATION FINAL REPORT USAID.GOV WEI has not yet made progress toward its targets for women-centered care (WCC), which is an important component of the gender and inclusion targets. EQ7 – While the Okard Activity has faced many challenges since its inception in October 2017, activity staff have navigated these challenges and are currently well placed to accomplish many activity goals. Currently, both GoL MOUs are signed and all the sub-recipients, including GoL partners, are fully on board, each with a defined scope of work and budget. And, most USAID waivers (to allow for construction and distribution of restricted commodities) are now in place to ensure the smooth implementation of activities moving forward. Okard staff communication with key government stakeholders has not always met with the needs of those stakeholders, which may be affecting the success and sustainability of the activity results. Further, the provision of in-kind support only to the GoL rather than direct financial support and daily subsistence allowances provided by Okard have been additional points of contention between Okard Activity staff and GoL officials. Finally, Okard Activity indicators and quarterly and annual reports are not set up to ensure Okard partners and USAID can track which activities are successful and which are not in support of adaptive management and consideration of Okard follow-on activities. Indicators and reports also do not allow both groups to track which groups are receiving services and which are underrepresented. EQ8 – While several partner challenges delayed finalization of the partnership framework until very recently, all partners have now signed their sub-agreements, the latest in June 2021. The limited evidence the ET was able to find suggest local Activity sub-recipients (namely, ARMI, QLA, DMAS, COPE, and AfA) have the systems and capacity to complete their activities under Okard. However, the following additional systems and capacity (“missing systems and capacity”) would help to ensure both smoother operations moving forward and the sustainability of activity interventions. • QLA needs support to improve its communication strategy for better relations with GoL counterparts and to ensure capacity and funding exists to support the sustainability of activities after Okard ends. • In order to become a sustainable part of the EE system for persons with disabilities in Laos, DMAS needs urgent support to identify and implement a business and sustainability plan. RECOMMENDATIONS A full list of recommendations is presented in the report by EQ, including details on how the ET suggests each recommendation might be implemented, by whom, and key considerations for implementation. The ET has listed here the key recommendations in priority order without that additional detail and with a note to each original recommendation number from the body of the report. • Recommendation 5: USAID should consider providing the Okard Activity a no-cost extension given the delays in implementing planned activities, many of which were out of the control of key Okard partners. • Recommendation 3: WEI should update its critical assumptions given the evidence that the first two critical assumptions have not held and are unlikely to in the last months of the activity. 6 | EVALUATION FINAL REPORT USAID.GOV • Recommendation 4: WEI monitoring, evaluation, and learning staff should update overall Okard indicator targets, in close collaboration with USAID, to reflect the dramatic shift in the context and delays in getting activities started. • Recommendation 1: USAID should work with WEI and HI to consider whether continuing to invest in TVET activities under Okard is the best use of the Activity’s limited remaining time, given the seeming low level of demand for TVET amongst persons with disabilities. If USAID, WEI, and HI decide to continue TVET support, they should prioritize creation of a strategy to increase demand and teacher training on disability-inclusive pedagogy. • Recommendation 2: USAID should take note of the issues and trends in feedback that arose from the MOU process under the Okard Activity and should share these notes with future implementing partners embarking on the MOU process to help them more effectively manage the path to MOU signing. • Recommendation 6: WEI should prioritize Okard activities focused on strengthening rehabilitation and mental health within the health system in Laos, in close collaboration with DHR and CMR. Key priorities should include supporting DHR and CMR to organize a functional rehabilitation committee and specific technical sub-committees, where the lead for rehabilitation and the roles and responsibilities for operationalizing the rehabilitation strategy could be decided and agreed upon, including drafting a plan for improving the referral system and access to mental health services as well as supporting the DHR and CMR in defining a realistic plan for funding rehabilitation, mental health services, and assistive technologies. • Recommendation 17: WEI and HI should work with USAID/Laos and the GoL ahead of selecting beneficiaries for subsequent rounds of CBID services to ensure the activity is targeting and prioritizing the groups USAID/Laos wishes to target. • Recommendation 10: WEI and HI should conduct a definitive needs assessment of CBID beneficiary needs for all cohorts to ensure any final waivers and unexploded ordnance clearance requests are complete and additional waivers and GoL clearance requests are not needed. • Recommendation 23: HI should prioritize its work with DMAS to ensure the sustainability of the organization after the activity ends. • Recommendation 19: Okard should consider analyzing and reporting more refined disaggregated data with regards to age cohorts of beneficiaries and should also collect data and report on other vulnerable groups. • Recommendation 20: All USAID Okard partners should approach the gender and inclusive component of the activity with more intention. The activity can continue to be generally inclusive while at the same time intentionally aware that it is interested in providing services to particular vulnerable populations and ensuring that strategies and systems are in place to appropriately meet the needs of these vulnerable populations. Further, WEI staff should work in close collaboration with DHR to focus efforts on starting the WCC activity. 7 | EVALUATION FINAL REPORT USAID.GOV • Recommendation 21: Okard Activity communications staff should develop a plan to share details about Okard management, systems, and protocols to key Okard Activity partners and GoL stakeholders. Specifically, WEI should communicate the key roles and responsibilities of all partners and should provide details on Okard management and partnership structures as well as financial and communication protocols regarding activities. The communication plan should also reference Decree 4000 for consistent guidance on daily subsistence allowances. 8 | EVALUATION FINAL REPORT USAID.GOV INTRODUCTION Persons with disabilities in the Lao People’s Democratic Republic (Lao PDR) face multiple challenges that limit their ability to achieve their full potential in society. While overall development indicators for the country are improving, persons with disabilities continue to experience higher rates of poverty, poorer health outcomes, and lower education levels, and are often excluded from equal participation in broader society.1 For instance, the 2020 baseline assessment for a community-based inclusive development (CBID) pilot project found that while 68 percent of persons with disabilities were working in the two pilot districts (a number that matches that of persons without disabilities), those with disabilities were much less likely to have been paid for that work (10 percent of persons with disabilities versus 39 percent of persons without disabilities). Disability affects people from all areas of the country, and of all ages, ethnic groups, socioeconomic backgrounds, and gender identities; however, persons with disabilities from certain disadvantaged groups in society – such as women, elderly, and ethnic minorities – face additional hardship.2 Recent USAID research has found that women with disabilities face more stigma and harassment than men with disabilities, and ethnic minority women are marginalized even further, reinforced by the way Lao society views and interprets gender roles. For example, women with disabilities are often subject to gender roles that restrict their movement outside of the home, compromising their access to services and their ability to fully engage in society.3 According to the 2015 Lao PDR Population and Household Census, persons with disabilities account for 2.8 percent of the population.4 While disabilities may occur at birth or result from an injury, illness, disease, or old age, many Lao people are victims of unexploded ordinances (UXO) that remain from the Vietnam War.5 Since the Vietnam War ended, at least 20,000 people in Lao PDR have been killed or injured by UXO, which still contaminate approximately one-third of the country. 6 In 2016, the Government of the United States (USG) and the Government of Lao PDR (GoL) announced a comprehensive partnership, deepening the United States’ support to address the impacts of UXO contamination, including a new contribution of $90 million by 2019.7 This mandate focused U.S. foreign assistance activities on UXO-related development issues, including UXO-related disabilities. However, while much higher than in other countries, the prevalence of persons becoming disabled as a result of UXO accidents in Lao is declining. Further, the 2015 Census reported that illness was the leading cause of disability (44.7%) followed by other causes that do not fit in one category (24.74%), congenital causes (18.85%), accidents (7.19%) including road traffic accidents, war (3.74%), and UXO accidents (0.68%). In recent years, the GoL has also given increased attention to the rights and needs of persons with disabilities. In 2008, the GoL signed the Convention on the Rights of persons with disabilities, followed by its own National Decree on the Rights of Persons with Disabilities in 2014, and the endorsement of the WHO 1 USAID/RDMA. USAID Lao PDR Gender Analysis on Disability, March 2018. 2 Ibid. 3 Ibid. 4 “Health information systems and disability in the Lao PDR: A qualitative study.” International Journal of Health Planning and Management 31.4: 446-58. 5 Lao Statistics Bureau (2005). Results of Population and Housing Census 2005 Census. 6 Legacies of War (2017). Leftover Unexploded Ordnance. 7 White House (2016). Joint Declaration between the United States of America and the Lao People’s Democratic Republic. https://obamawhitehouse.archives.gov/the-press-office/2016/09/06/joint-declaration-between-united-states-america-and-lao￾peoples. 9 | EVALUATION FINAL REPORT USAID.GOV Disability Inclusive Health and Rehabilitation Plan 2015-2020. Further, in November 2018, the Minister of Health signed the National Rehabilitation Medicine Strategy; in December 2018, the GoL passed the National Disability Law; and in October 2020, the Prime Minister signed the National Disability Policy, Strategy and Action plan.8 Numerous other GoL laws and policies on topics ranging from Technical and Vocational Education and Training (TVET) to labor make reference to persons with disabilities. OKARD ACTIVITY INTRODUCTION OKARD ACTIVITY OVERVIEW To support the GoL in addressing inequity and the numerous challenges facing persons with disabilities in that country, in October 2017 USAID awarded World Education, Inc. (WEI) a five-year, $15 million-dollar cooperative agreement to implement the Okard Activity. USAID Okard builds on the previous USAID effort that focused on disability inclusion—the Training, Economic Empowerment, Assistive Technologies and Medical Rehabilitation (TEAM) Activity, which was implemented from 2014 to 2017 in four conflict￾affected countries: Colombia, the Democratic Republic of the Congo, Lao PDR, and Ukraine. WEI implemented the Activity in Laos, and report that the Activity was essentially a granting mechanism that provided small grants to organizations working on rehabilitation training, the provision of assistive technologies (AT), economic empowerment, and Disabled People’s Organizations (DPO) strengthening efforts. The TEAM Activity reports having successfully trained 532 rehabilitation doctors and nurses, providing AT to 1,180 persons with disabilities, supporting 1,660 persons with disabilities through economic empowerment activities, providing rehabilitation services directly to 2,815 persons, strengthening 302 organizations, and conducting extensive community awareness activities. However, through this Activity, WEI learned that while the small initiatives were successful, they were a bit disjointed and, thus, did not lead to the large-scale change needed in Laos. WEI reports having integrated these lessons into the Okard Activity and taking a more systematic approach focused on developing more sustainable mechanisms to produce long-term change in the country. This shift in focus was also a result of USAID’s Notice of Funding Opportunity’s focus on transferring skills and systems to the host country government institutions and local partners to ensure the sustainability of outcomes. The overall goal of the USAID Okard Activity is to improve and sustain the independent living and functional ability of persons with disabilities and their households in Lao PDR, regardless of age, sex, gender expression, or ethnicity. The Activity began in September 2017 and is set to end in September 2022. USAID Okard is underpinned by the United Nations Convention on the Rights of Persons with Disabilities, the GoL’s National Disability Policy Strategy and Action Plan, and the National Rehabilitation Strategy and Action Plan (endorsed by the GoL in October 2018). The Activity works to foster a society where persons who have long-term physical, mental, intellectual, or sensory impairments can achieve their optimal level of functioning and enjoy equal, full, and effective participation and opportunities in society.9 8 WEI. USAID Okard Activity Y2 Annual Report, November 2019 and WEI. USAID Okard Activity Y3 Annual Report, January 2021. 9 Nossal Institute for Global Health/WEI. CBID Demonstration Model Assessment Baseline Survey Report, November 2020. 10 | EVALUATION FINAL REPORT USAID.GOV The USAID Okard Activity has three main components: Health, Economic Empowerment (EE), and Stakeholder Engagement (SE), as shown in Figure 1 below, taken from the Okard Activity’s Monitoring, Evaluation, and Learning Plan (AMELP). In addressing the three components, WEI uses a two-tiered approach to integrate the systems level (Tier 1) and the individual/community level (Tier 2). FIGURE 1: OKARD RESULTS FRAMEWORK The Activity’s theory of change (TOC) predicts that if USAID Okard implements activities in the three components of health, EE, and SE at both the systems level and community level that: 1) government and non-government stakeholders will have a better understanding about the reality and experience of being a person with a disability, and 2) service providers, families, and persons with disability will work together to complement each other in their roles in health, education, business, and in public and private central, provincial, and district institutions. The TOC reasons that the goals for the Activity will continue to improve and be achieved more effectively and sustained over the long term through the two-tiered, multi￾sectoral approach. Three critical assumptions underpin this TOC, namely: 1) the Lao operating environment (natural disasters, disease outbreaks, political situation, economic situation, ability of international non-governmental organizations (INGOs) and civil society organizations (CSOs) to operate in Lao PDR, GoL policies) is stable and conducive to the Activity; 2) GoL administrative requirements are requested and processed as expected; and 3) all needed stakeholders are willing and able to engage with the Activity as expected. 11 | EVALUATION FINAL REPORT USAID.GOV The systems-level work involves WEI and its key partner, Humanity and Inclusion (HI) supporting key ministries through capacity building, technical assistance, and in-kind resources (directly controlled and managed by WEI/HI) with the goal of creating a sustainable enabling environment for persons with disabilities. To achieve this goal, Okard hopes to build the ownership and capacity of the Department of Healthcare and Rehabilitation (DHR) and the Center for Medical Rehabilitation (CMR) within the Ministry of Health (MOH) to integrate physical and psychiatric rehabilitation within the wider healthcare system and expand its reach, scope, and quality at the central level, all the way down to the provincial and district hospital and household levels. It also includes interventions to build the capacity of the Ministry of Labor and Social Welfare (MLSW) across all levels of the government (from the central to the district levels) and the National Committee for Disabled People and Elderly (NCDE) to support more inclusive TVET through the Ministry of Education and Sports (MOES) TVET Department, and to support income￾generation opportunities for persons with disabilities. Under the SE component of the Activity, Okard also works to develop the concept of “disability mainstreaming” in Lao, supports awareness raising and communications events, supports policy review and adaptation for disability-inclusion efforts, and works to increase the capacity of DPOs at the national and local levels. It seeks to accomplish the latter by establishing and strengthening networking mechanisms among DPOs and other development stakeholders and facilitating direct engagement between the GoL and these representative bodies. In addition to the systems-level work, Okard also includes a community-level pilot called the Community￾Based Inclusive Development (CBID) Demonstration Model. This is an evidence-based model that seeks to catalyze and test the implementation of the National Disability Policy, Strategy, and Action Plan, and the National Rehabilitation Strategy and Action Plan at the community level. CBID facilitators use a case management approach to work directly with persons with disabilities to identify barriers to their optimal functioning and then work with them, their families, communities, local authorities, and relevant service providers to collectively understand and address these barriers and create more inclusive communities. OKARD ACTIVITY PARTNERS To meet the goals of the Okard Activity, WEI has designed a partnership and management structure that has engaged private and public partners, as shown in Figure 2. Each partner has defined activities organized under the three components (health, EE, SE). The Social Impact evaluation team developed the below organizational chart and following description based on information provided by WEI, especially the USAID Okard Sub-Recipients Profile, as well as through information provided in key informant interviews (KIIs) with WEI, HI, and USAID. 12 | EVALUATION FINAL REPORT USAID.GOV FIGURE 2: OKARD MANAGEMENT AND PARTNERSHIP STRUCTURE As shown above, WEI is the prime grantee for the cooperative agreement with USAID for the Okard Activity. WEI is ultimately responsible for agreement oversight, compliance with USAID rules and regulations, and meeting the objectives of the Activity. As the Activity prime, WEI supervises all sub￾recipients and also provides support to them. WEI also leads the health component for the Activity, overseeing in-kind support to: • DHR to update the governance, policies, and information related to rehabilitation medicine; increase financing for rehabilitation medicine; broaden integration of rehabilitation medicine into the health sector; develop a multidisciplinary rehabilitation medicine workforce through both pre￾service and in-service training; provide Mental Health and Psychosocial Support (MHPSS) Training; strengthen and expand the habilitation and rehabilitation services network; and collect rehabilitation medicine data and support research. • CMR to establish the Continuous Professional Development Training Center (CPDTC) and lead capacity building for the health and rehabilitation workforce. Both DHR and CMR receive funding through in-kind support, meaning that specific activities are funded directly by Okard, in line with the USAID policy prohibiting direct funding to government agencies. WEI also has a sub-grant with the Cooperative Orthotic and Prosthetic Enterprise (COPE), which is an independent organization working under a partnership agreement framework with CMR and having permission to operate and receive funding from international and local donors by the Ministry of Foreign Affairs (MOFA). Activities carried out by COPE support health and rehabilitation centers within Lao PDR to provide AT to persons with disabilities and establish best practices. COPE will also deliver training courses to build the capacity of rehabilitation personnel in order to provide AT and will work to build collaboration amongst AT donors and providers. Working in close collaboration with CMR and DHR, COPE will develop a priority list for AT targeting priority beneficiaries, establish product standards for 13 | EVALUATION FINAL REPORT USAID.GOV AT, and conduct a cost-calculation study of rehabilitation services to produce evidence that can be used to advocate to the MOH for necessary increases to the rehabilitation budget. In addition to its work under the health component of the Activity, WEI also oversees CBID activities as a whole, with a special focus on quality assurance support to the Quality of Life Association (QLA) in the Xieng Khouang (XKH) Province (QLA’s work is discussed in more detail below). Furthermore, WEI leads the social behavior change communication (SBCC) component of the Okard Activity, which raises awareness and acceptance in communities of persons with disabilities, and gender and inclusive development action plan (GIDAP) activities, focused on ensuring all persons with disabilities are able to benefit from Okard activities. Finally, they provide overall management oversight for the Activity, act as the main point of contact for USAID, and lead monitoring, evaluation, and learning (MEL). HI was pre-identified and approved by USAID as a sub-recipient at the design stage of the Okard Activity, and they were involved in the development of the technical application. Functionally, HI is considered a co-implementer of the Activity, according to the USAID Okard Sub-Recipient’s Profile in Brief provided to the ET by WEI although the legal relationship is a sub-recipient with their own budget. HI is the only sub-recipient who can sub-grant to others. It is responsible for driving the EE and SE components of the Activity, with oversight of the CBID Demonstration Model in Savannakhet Province (SVK), which is implemented by the Association for Rural Mobilization and Improvement (ARMI). ARMI’s activities are discussed more below. HI also has a sub-grant with Disability Mainstreaming Advisory Service (DMAS), a newly registered consulting firm (August 2019) established as a social enterprise to support persons with disabilities to access employment and develop a business community committed and equipped to provide non￾discriminative access to employment for all persons. DMAS receives technical and financial support from Okard under the economic empowerment component to: 1) develop a DMAS Centre; 2) engage universities, vocational schools, social and private enterprises, and business associations to develop and implement effective and inclusive technical and vocational training courses;10 3) engage with private sector and Lao vocational support entities to integrate persons with disabilities into existing channels for employment; and 4) review disability rights and equity training. With support from HI, DMAS works with the TVET Department to strengthen inclusive training environments for all people in Lao PDR. Originally, DMAS worked under an agreement signed by WEI due to the delayed signing of HI’s Memoranda of Understanding (MOUs). In November 2020, DMAS fell under the supervision of HI. Under the SE component, HI holds a sub-grant agreement with the Association for Autism (AfA) to 1) support capacity building for DPOs so they are able to follow the Lao Decree on Associations and to demonstrate transparent governance systems, sustain fundraising capacity, and comply with donors’ rule and regulations for accountability; and 2) to coordinate with DPOs to advocate to decision makers and service providers to protect and realize the rights and meet the needs of all persons with disabilities and their households. Finally, HI contributes technical capacity to CMR activities through their health and rehabilitation specialist. 10 Note that this piece of DMAS’ scope of work or SOW was added after HI decided not to engage the original partner that was planned to implement this component due to capacity issues discussed in the findings section below. 14 | EVALUATION FINAL REPORT USAID.GOV NCDE is a key government partner under the MLSW. Their role under Okard is to oversee all interventions. NCDE also holds MOUs with WEI and HI, as MOUs are required for any INGO working in Laos. NCDE’s MOU with HI lays out not only HI’s work but also eight NCDE activities under Okard, which includes a sub-budget for in-kind contributions from the Okard Activity to NCDE to undertake interventions at the central government level related to the implementation of the Lao law, policies, and strategy on disabilities as well as piloting of data collection on disabilities. Both HI and WEI have MOUs with NCDE. ARMI and QLA are the implementers of the CBID Demonstration Model in XKH and SVK. QLA is an XKH-based non-profit association (NPA) that had significant experience supporting UXO survivors and persons with disabilities prior to their engagement with Okard. With technical support from WEI, QLA implements the CBID Demonstration Model in 35 villages in Kham District. ARMI is a SVK-based NPA with experience related to livelihood improvement, agriculture promotion, and hygiene and sanitation promotion. Prior to their engagement with Okard, ARMI did not have experience supporting persons with disabilities. ARMI, with technical support from HI, implements the CBID Demonstration Model in 16 villages in Xayphouthng District. EVALUATION OVERVIEW, PURPOSE, AND QUESTIONS EVALUATION OVERVIEW The United States Agency for International Development’s (USAID) Laos Mission (USAID/Laos) contracted Social Impact (SI) to conduct a mid-term evaluation (MTE) of the USAID Okard Activity between February and July 2021. The MTE is a requirement of the USAID Okard Cooperative Agreement No. AID-486-A-17-00004. This evaluation will serve as a “pause-and-reflect” opportunity and help USAID Okard to review its progress and guide its mid-term re-design workshop. EVALUATION PURPOSE The purpose of the USAID Okard MTE is to provide feedback to USAID, the GoL, WEI, and its partners on Activity progress and systems to date. The objectives are to guide strategic direction and planning for the remainder of the Activity to capitalize on the approaches, strategies, and systems that are working and improve those that are not. The MTE will inform the Okard Activity’s mid-term re-design workshop. The primary audience for this MTE is USAID, WEI, their sub-partners, and the GoL, with secondary and tertiary audiences of other government and non-government organizations working on similar issues in Lao PDR and elsewhere, and the interested general public. EVALUATION QUESTIONS The evaluation questions (EQs) for the MTE are as follows: • EQ1 - To what extent have the USAID Okard strategy and interventions been relevant to the goals it intends to reach? • EQ2 - Are the context and critical assumptions still valid? What have been key changes, if any, that affect the anticipated achievements positively and/or negatively? 15 | EVALUATION FINAL REPORT USAID.GOV • EQ3 - What components of the Activity have been successfully and/or unsuccessfully implemented according to the approach? Why? • EQ4 - To what extent has USAID Okard been effective in achieving its expected results, particularly in terms of building/strengthening the systems as described in the design document (system-centered)? • EQ5 - To what extent has USAID Okard been effective in achieving its expected results, particularly in terms of reaching target beneficiaries? Which groups are underserved, if any, among all targeted groups, and why? (People-centered?) • EQ6 - To what extent has USAID Okard been effective in achieving its expected results, particularly in terms of gender equality and other inclusion? • EQ7 - Does USAID Okard have the appropriate management system and partnership structures to accomplish its objectives? • EQ8 - Do USAID Okard partners have the right systems and capacity to fulfill their scope of work? What systems and/or capacities are currently missing in order to fulfill the scope of work? EVALUATION METHODS AND LIMITATIONS EVALUATION METHODS To address the above EQs, the SI evaluation team (ET) worked collaboratively with USAID, WEI, and HI through all phases of the evaluation, which included bi-weekly meetings as well as findings and recommendations workshops. The ET used qualitative methods including a document review, KIIs, group interviews, and quantitative analysis of CBID beneficiary data. The ET ended up holding 33 KIIs and 13 group interviews, resulting in a total of 76 respondents from USAID, WEI, HI, sub-recipients, GoL officials, and beneficiaries about their experiences with the Okard Activity. Amongst these respondents, 36 individuals were male, 34 were female, and eight were persons with a disability. This was a significant reduction in the number of individuals targeted in SI’s evaluation design report, which was necessary due to increased rates of coronavirus disease (COVID-19) in Laos that resulted in lockdowns. In the design report, the ET planned to collect data from 21 KIIs and 22 focus group discussions (FGDs), which were anticipated to provide input from approximately 218 individuals representing a wide spectrum of perspectives. Given the lockdown, the ET had to shift from in-person data collection to remote, phone￾based data collection, which meant shifting a lot of FGDs to KIIs and group interviews. The limitations of these changes are discussed in detail below. A full description of the methods, data collection activities, sampling, and analysis can be found in Annex I. LIMITATIONS The results of this MTE should be understood within the context of numerous limitations, as follows: Limited Sample Size - Qualitative studies should never be generalized beyond the population from whom the data were collected. While every attempt was taken to collect data from a wide range of stakeholders and beneficiaries, the results presented in this final report reflect the sentiments of the 16 | EVALUATION FINAL REPORT USAID.GOV individuals from whom data were collected. The onset of COVID-19 lockdowns, which started near to the time when data collection began for this MTE, posed numerous other limitations. Most importantly, the change of data collection method from a mixture of FGDs and KIIs to only KIIs resulted in far fewer informants who contributed to these results. Of particular mention is the reduction of participants at the community level and also the number of health workers. With regards to beneficiaries, caregivers, and other community participants, the team had planned to connect with from 84 to 112 respondents, but because of the shift to KIIs rather than FGDs, this number was limited to 16 (14-19 percent of the intended sample and only representing about 4 percent of beneficiaries). Response Bias - Another common challenge with qualitative data collection is the risk of response bias, or the affinity for some respondents to answer in a way they think the facilitator prefers. The ET tried to mitigate this by designing data collection tools that asked non-biased and open-ended questions. Self-Report Bias - Given that this evaluation was conducted at one point in time rather than over time, the ET was unable to directly measure capacity increases or other changes resulting from the Okard Activity. Rather, the team relied on self-reported information regarding changes in capacity, which can increase bias. The consent statements and independent nature of the evaluation may have mitigated this effect to some extent. Remote Data Collection and Selection Bias - Remote data collection through WhatsApp and mobile phones presented numerous other limitations. Most importantly, the ET was limited to connecting with people who either had a mobile phone or could borrow one. This may have resulted in selection bias, by reducing the ability of the ET to connect with the most vulnerable people, including people who are very poor or live in a remote location where mobile phone coverage is poor or non-existent. As this MTE endeavored to understand how the Okard Activity had affected persons with disabilities, the remote data collection method also limited the types of people with whom the ET could connect with, as it was impossible to connect with people who had disabilities that would preclude talking over a phone, such as people with hearing impairment or cognitive disabilities. The ET attempted to mitigate this problem by connecting with caregivers of people who would not be able to speak on a phone. A major challenge with the remote interviews was that, in some cases—especially when communicating with caregivers and beneficiaries—the quality of connection was very poor. This resulted in communication challenges, poor capture of audio, and sometimes required second and third attempts by the ET to connect with respondents at a later time. This format also meant that the evaluation team leader was not able to participate in many of the interviews (since they were in the middle of the night US/Canada time). This coupled with the poor connection made it more difficult for the evaluation team to ask respondents follow-up questions. The KII-only format of data collection for this MTE also limited the ET’s ability to bring a wide variety of people (people of different ages, ethnicities, people from different locations) to a centralized location for FGDs. Instead, the ET was limited to selecting a few individuals who could be reached and necessitated a scaling down of informant types, particularly at the community level. To mitigate some of these limitations, the ET tried to select individuals with a wide variety of disabilities for KIIs so that a small glimpse from a wide spectrum of experiences could be included in the results. The ET also tried to select a gender￾balanced informant base. In some cases, however, the selected individual was not available, which required the ET to select another individual. 17 | EVALUATION FINAL REPORT USAID.GOV Remote interviews with some persons with disabilities, their caregivers, community members, village leaders, and some individuals at the provincial and district government levels were sometimes challenged by what might have been intimidation or unfamiliarity with being interviewed over the phone. Whereas during a face-to-face FGD, a facilitator might be able to use gestures or other non-verbal cues to encourage shy or quiet participants to share their perspectives, most of the data collection for this MTE was limited to verbal communication during KIIs. Lack of Independence in Quantitative Analysis – The ET requested quantitative CBID beneficiary data from the Okard Activity, but WEI and HI were unable to provide that data due to concerns over transparency in data use provided to beneficiaries during the consent process. As such, the ET had to shift to providing a series of data requests to WEI and HI, and WEI and HI had to largely conduct the analysis themselves. This limited the possible types of analysis the team was able to conduct and also means the ET cannot guarantee the data are free from bias. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS (FCR) FCRs are included by EQ below. A large bulk of the findings fall under EQ3, since that question asks about all successful and unsuccessful aspects of the Okard Activity to date. These findings overlap with findings across all of the other EQs but especially for EQs 4, 5, and 6, which are focused on Activity results. The ET has aimed to present findings only once in detail and then reference those findings, when necessary, under subsequent EQs. While recommendations are also presented by EQ, they sometimes take into consideration findings from other EQs. The ET has done its best to note this whenever this is the case. EVALUATION QUESTION 1 TO WHAT EXTENT HAVE THE USAID OKARD STRATEGY AND INTERVENTIONS BEEN RELEVANT TO THE GOALS IT INTENDS TO REACH? EQ1 FINDINGS Nine out of ten (90 percent) of the key informants (GoL officials and Okard sub-recipients) that were asked about the design of the Okard Activity said the Activity was well designed and relevant to the goals it intended to reach with regards to stakeholder engagement (SE), health, and economic empowerment (EE) for persons with disabilities. Further, after reviewing international and regional best practice, the external baseline evaluation found that, “USAID Okard appears to be following the basic tenants of international best practices in disability inclusive programming such as engaging with DPOs, community￾based rehabilitation, and gender integration.” While the Okard Activity has not reached all of its goals to date, the majority of the 76 key informants interviewed reported that this is largely because of changes to the Lao context that created unexpected barriers to the Activity and significantly delayed several Activity components, as described in the findings presented under EQ2 below. Despite these delays, the Activity appears to have made a lot of progress toward reaching its goals in the past year, as evidenced by the findings presented under EQs 3-6 below, which reference Okard quarterly and annual reports and outcomes reported from a small sample of Activity beneficiaries (specifically, 12 CBID beneficiaries and staff from each of the five local organizations who have benefited from the Activity). While the ET was unable to confirm the numbers presented in Okard’s quarterly and annual reports or collect information from a representative sample of CBID and other beneficiaries (such as those who have benefitted from resume building and employment support from the Okard Activity) due to the COVID-19 pandemic and 18 | EVALUATION FINAL REPORT USAID.GOV limitations in the timeline for the evaluation, Okard’s recent progress toward achieving its targets provides initial evidence that the design holds promise. The one key informant who said the design was not relevant to the goals it intends to reach was from the TVET Department of the Ministry of Education and Sports (MOES); it also said the TVET activities could have been designed better. The USAID Okard Cooperative Agreement laid out the following supports to the Laos TVET system: • Technical support for targeted TVET colleges using the Lao Disabled People Association (LDPA) DMAS to identify and address barriers to the inclusion and participation of persons with disabilities. This will be implemented over several years in a phased approach, to allow for possible changes to take hold. Phase 1 targets are to support one TVET college in Xieng Khouang and one in Savannakhet. • Create bridge and reference systems (synergies and network) between the schools for the deaf and blind to refer students from school to TVET. • Work with the Inclusive Education Center (IEC) and Basic Education Quality and Access in Lao PDR synergies and networking to refer children with disabilities who are finishing inclusive education curriculum to TVET. • Technical support for TVET service providers adapting the delivery of the curricula in a more disability inclusive and supportive manner. This will include creating links between Sikeud Vocational Training Center, TVET, and vocational training centers to use their experience to support training in disability inclusion. It will also support development of information about available vocational courses for persons with disabilities and the public using different mediums (e.g. pictures, leaflet, and radio). Information about skills needs of businesses from the Market Analysis will be shared, to encourage market-driven vocational subjects to be developed.” According to numerous Okard Activity staff, partners, and USAID, these interventions were delayed as a result of: • Delays in the signing of Okard’s Memorandum of Understanding (MOU) (discussed under EQ2 findings below) with the GoL (which is required for all international organizations operating in Laos); • Issues related to COVID-19 (also described under EQ2); • The need to shift to a new partner to implement the TVET supports from the LDPA to DMAS as a result of issues related to LDPA’s capacity and financial policies, which did not include adequate provisions for the segregation of duties, among other issues, according to USAID officials; and • Delays created by the turnover and availability of the HI technical staff responsible for overseeing the TVET activities, which meant HI was not able to coordinate informally with the TVET Department while waiting for the MOU to be signed. 19 | EVALUATION FINAL REPORT USAID.GOV Okard and partner staff also reported that there is little demand for TVET in SVK and XKH, with none of the CBID beneficiaries selecting to participate in TVET support. TVET Department officials further report that no persons with disabilities are currently enrolled in Laos TVET institutions, to the best of their knowledge. WEI staff report that there may actually be some students with disabilities enrolled in TVET institutions in Laos, but because the TVET Department is not required to collect disaggregated data on disability, students with minor disabilities may go unnoticed and not have their needs met. The ET was unable to externally confirm these claims due to limitations in evaluation resources. Regardless, it is clear that current demand for TVET amongst persons with disabilities is not high. According to studies external to this MTE, the main reason for this lack of demand is the limited supply of inclusive secondary schools in Laos. As of 2006, only 13 percent of students with disabilities enrolled in primary schools went on to attend secondary schools, and there were only 11 inclusive-education (IE) secondary schools as of 2007 in comparison with 381 primary IE schools, according to a Save the Children report.11 Nonetheless, Okard has done some work in the area of TVET support, and international standards suggest that ensuring TVET institutions are inclusive is a best practice that results in improved opportunities and livelihoods for persons with disabilities. Okard staff report that while demand does not appear to be high in the provinces, it may be higher in the cities, particularly in the capital—Vientiane. Further, sometimes supply must proceed demand. The DMAS Center, the establishment of which was part of Okard’s activities (specifically E2.1, described in more detail under EQ3 below), was established in October 2018 to provide advisory services on disability inclusion, particularly in accessibility to facilities, according to DMAS staff. However, according to WEI staff, their scope was later expanded to include the provision of support to TVET institutions after HI decided not to provide a sub-grant to LDPA. DMAS staff also reported that prior to working with the Okard Activity, they had formed a village-level group of persons with disabilities and were making products to sell online. They then heard that the Okard Activity was making grants to organizations interested in supporting businesses, institutions, and service providers with practical advice to make disability inclusivity comprehensible and achievable, so they applied for and received the grant. Despite starting up their activities, DMAS reports having only conducted one disability inclusion audit for a TVET service provider (as of May 2021), the Lao-Korea Vocational College. They said the reason for only conducting one audit was largely because the colleges had declined to participate despite having received a formal invitation from HI. They believe the reason the colleges declined was because the Okard Activity did not have funding as of July 2021 to support the colleges in addressing any weaknesses identified in the audits. While the ET was unable to confirm this due to COVID travel limitations, WEI did prepare a waiver request to USAID (note: as of July 2021, request approval was still pending, according to WEI staff) to allow them to provide construction support to TVET institutions to address any such weaknesses. It is important to note that the construction waiver was necessary because construction was not originally permitted in the Okard cooperative agreement. Despite all of this, TVET Department staff reported that while TVET institutions were built before the Law on Disability was passed and were not designed to be accessible, after the law was passed, building modifications were made to ensure accessibility for persons with disabilities in at least one TVET 11 The organization worked on IE from 1993 – 2009. 20 | EVALUATION FINAL REPORT USAID.GOV institution per province. They said, “building facilities are not necessary because we already have them.” Due to travel restrictions from COVID-19, the ET was unable to verify these claims. However, the TVET Department staff said while they do not need support with construction, they do need support with teacher training (already planned through Okard and recently started, according to HI staff), procurement of assistive technologies for the classroom, and scholarships for persons with disabilities from rural areas to attend TVET institutions in larger cities. They also suggested that teacher support might best be offered by providing teachers the opportunity to go on study tours to learn best practices from other countries who have inclusive TVET institutions and then having those teachers collectively design inclusive curricula with HI and DMAS staff. The reason the TVET Department staff member said they believe the Okard Activity design did not effectively address the needs of TVET institutions in Laos was because “We were not involved in Activity development. Okard Activity only invited us to take part in the questionnaire design and asked for information regarding students with disabilities in vocational institutions.” WEI staff dispute this claim, and provided evidence that, indeed, the Deputy Director of the TVET Department was consulted in June 2017, during the design of the Okard Activity, and that he recommended providing disability awareness and training to teachers on how to teach to persons with disabilities. However, they also said that since then, interactions with the department have mostly been focused on collecting data on persons with disabilities enrolled in TVET given delays in this activity (discussed above), which have meant the other planned activities are only now moving forward. The ET confirmed these statements through a review of meeting minutes from meetings between WEI, HI, and the TVET Department. HI and WEI also report having conducted an orientation workshop in October 2020 for TVET activities with the specific goals of examining barriers and difficulties for persons with disabilities to access TVET and to develop an action plan to address these barriers. Through this meeting, Okard and TVET staff as well as other key stakeholders developed action plans for each of four targeted TVET centers. These action plans included conducting accessibility audits, developing a referral system to get persons with disabilities enrolled in TVET centers, and training TVET Department staff and teachers on disability inclusion and disability inclusive pedagogy. All activities were meant to have been implemented by June 2021, but according to HI, activities have been delayed as a result of COVID-19. Despite these efforts, staff from HI and WEI reported that one thing they have been surprised about through the course of the Okard Activity is that there is very little interest on behalf of persons with disabilities in SVK and XKH in attending TVET institutions. In fact, none of the CBID beneficiaries expressed interest. They said one reason for this is that many of the persons with disabilities have a low￾level of education and, thus, are not eligible to attend such institutions. The ET confirmed this finding through analysis of Okard CBID data, which showed that fewer than 18 percent of Okard beneficiaries overall in those two provinces had completed secondary education or above, as shown in Table 1. Further, the data show the high level of inequality in education between men and women, with woman being more than 26 percent less likely to have achieved a primary education than men (60.3 percent of women reported not having received a primary education, compared with only 33.7 percent of men). 21 | EVALUATION FINAL REPORT USAID.GOV TABLE 1: EDUCATION LEVELS BY SEX IN XAYPHOUTHONG AND KHAM LEVEL OF EDUCATION GENDER TOTAL MALE FEMALE Xayphouthong No Education 63 39.4% 140 71.1% 203 56.9% Primary Education 65 40.6% 41 20.8% 106 29.7% Secondary Education 24 15.0% 13 6.6% 37 10.4% Higher Education 5 3.1% 2 1.0% 7 2.0% Not Applicable 3 1.9% 1 0.5% 4 1.1% Total 160 100.0% 197 100.0% 357 100.0% Kham No Education 109 31.1% 145 52.5% 254 40.6% Primary Education 144 41.1% 101 36.6% 245 39.1% Secondary Education 91 26.0% 27 9.8% 118 18.8% Higher Education 4 1.1% 3 1.1% 7 1.1% Not Applicable 2 0.6% 0 0.0% 2 0.3% Total 350 100.0% 276 100.0% 626 100.0% Total No Education 172 33.7% 285 60.3% 457 46.5% Primary Education 209 41.0% 142 30.0% 351 35.7% Secondary Education 115 22.5% 40 8.5% 155 15.8% Higher Education 9 1.8% 5 1.1% 14 1.4% Not Applicable 5 1.0% 1 0.2% 6 0.6% Total 510 100.0% 473 100.0% 983 100.0% Source: ET analysis of CBID beneficiary data tables provided by WEI in July 2021 Furthermore, TVET Department staff said very few persons with disabilities are enrolled in TVET institutions in Laos. They said that this is largely because the institutions are not yet inclusive. Finally, WEI staff said they still believe TVET activities are important for three reasons: 1) while persons with disabilities in rural provinces may not be prepared to attend such institutions, those from cities may be more likely to do so; 2) improved supply sometimes leads to increased demand given that persons with disabilities are disincentivized to enroll in TVET if they know it is not set up for them to learn; and 3) international standards rightfully identify the importance of inclusive TVET. In fact, the ET found that “effective access 22 | EVALUATION FINAL REPORT USAID.GOV to general technical and vocational guidance programs, placement services, and vocational and continuing training” is required under Article 27 of the United Nations Convention on the Rights of Persons with Disabilities, to which Lao PDR is a signatory. Further, Sustainable Development Goal 4 prioritizes inclusive and equitable quality education and lifelong opportunities for all, and the Laos National Action Plan for Persons with Disabilities has a target of increasing the number of persons with disabilities attending TVET institutes by 15 percent by 2030. While no other respondents reported concerns about the design of the Okard Activity, 72 of the 76 total key informant participants (95 percent) said that the roll-out of the Activity has not gone entirely according to plan. Findings regarding what has gone well and what has not are included under EQs 3-6. EQ1 CONCLUSIONS While the ET does not have enough evidence to confirm that the Okard strategy and interventions have been relevant to the Activity’s intended goals, initial evidence suggests that the strategy and interventions have been relevant and appropriate, possibly with the one exception of TVET activities, which would benefit from either: 1) reconsideration regarding their relevance at the present time given the low levels of education amongst persons with disabilities in Laos, or 2) increased coordination and cooperation with the TVET Department. While international standards suggest that supporting TVET for persons with disabilities helps to strengthen systems and ensure inclusion and opportunity for persons with disabilities, given the long list of priorities (discussed in more detail under other EQs below) for USAID Okard, USAID should consider whether the focus on TVET activities makes sense at this time or if TVET activities might better be pursued once there is increased demand for them amongst persons with disabilities. EQ1 RECOMMENDATIONS Recommendation 1: USAID should work with WEI and HI to consider whether continuing to invest in TVET activities under Okard is the best use of Okard’s limited remaining time given the seeming low level of demand. If USAID and Okard Activity staff decide to continue TVET activities, HI and DMAS should meet with the TVET Department to discuss next steps in the activity and to collectively make plans for future TVET activities. Together, the two groups should consider whether the planned intervention components are the best use of Okard funds with regards to TVET. Specifically, they should determine whether it makes sense to continue pursuing construction efforts related to accessibility of TVET institutions or whether they might shift funds set aside for construction to other activities or interventions. They should consider whether some of the Okard Activity’s efforts might be best placed in supporting the TVET Department to develop a demand-creation strategy for promotion of TVET for persons with disabilities. If TVET activities move forward and resources allow, HI might also continue their plans to train teachers on disability-inclusive pedagogy so that once demand is created, teachers are prepared. Finally, Okard MEL staff should suggest new targets for the TVET activity that might be feasible within Okard’s timeframe. Should USAID and Okard Activity staff decide not to continue TVET activities under Okard, USAID should still consider including TVET activities under future interventions focused on inclusivity but should do so with consideration of the relative demand for TVET amongst persons with disabilities in Laos and the need for persons with disabilities to also have access to quality primary and secondary education. 23 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION 2 ARE THE CONTEXT AND CRITICAL ASSUMPTIONS STILL VALID? WHAT HAVE BEEN KEY CHANGES, IF ANY, THAT AFFECT THE ANTICIPATED ACHIEVEMENTS POSITIVELY AND/OR NEGATIVELY? The Okard Activity staff listed three main critical assumptions in their TOC: 1) The Lao operating environment (natural disasters, disease outbreaks, political situation, economic situation, ability of INGOs and CSOs to operate in Laos, GoL policies) is stable and conducive for the Activity. 2) GoL administrative requirements are requested and processed as expected. 3) All needed stakeholders are willing and able to engage with the Activity as expected. EQ2 FINDINGS These assumptions have not held throughout the Activity lifecycle. The Okard Activity has been affected by COVID-19, delays in the signing of the MOU with the GoL, delays in finalizing the scope of work (SOW) for in-kind support to government partners, Okard partner CSO capacity and staffing challenges, and a lack of required background and expertise by Lao therapists interested in expanding their credentials, as discussed in more detail below. ASSUMPTION 1 According to Okard Activity staff, USAID, and officials from the GoL, Okard’s rehabilitation scholarship program was designed to provide 15 scholarships to Lao MOH nurses, physiotherapists, and doctors who wanted to upgrade their training and diplomas by pursuing a bachelors or Master’s degree in physical, occupational, or speech and language therapy, including residency training on rehabilitation medicine for doctors. It was negatively affected by the COVID-19 pandemic (a change in context associated with assumption #1 above), issues with the willingness and ability of MOH doctors and nurses to take part in the program (related to assumption #3 above), and (according to WEI specifically) delays on the side of the GoL with regards to making and implementing decisions about the scholarship program (related to assumption #2 above). The scholarships were announced in early 2019, but according to meeting minutes from a meeting between DHR, CMR, WEI, and other key stakeholders focused on discussing the scholarship, only ten persons applied, and only two of those met the qualifications (one was subsequently disqualified as he was over the age of 50, according to DHR). This may have been because they were the only ones who met the qualifications. This includes the need to be in the MOH quota, which WEI said means the person is in a fully paid, permanent MOH position. It is important to note that, according to WEI, not all GoL employees are “in quota,” meaning many are not paid a full salary. The World Bank reports that these staff include those who operate under a “95-percent” contract, which “provides 95 percent of the salary of a permanent position but no social benefits and is considered a position in waiting, virtually guaranteed of a permanent job”; volunteer staff who are paid according to tasks performed and administrative budgets; and contract workers who received a fixed salary well below the 95 percent level.12 Generally, WEI 12 World Bank. “Lao PDR Civil Service Pay and Compensation Review: Attracting and Motivating Civil Servants,” June 2010. Accessed at: https://documents1.worldbank.org/curated/en/256861468266176014/pdf/580180ESW0WHIT0on0February020110TOC.pdf 24 | EVALUATION FINAL REPORT USAID.GOV reports that staff need to work for a number of months or years before they advance to being “in quota,” and the new quota is extremely limited and only provided to essential health professionals. According to the Okard Year 2 (Y2) Annual Progress report, the one applicant was awarded a scholarship in July 2019 to attend Khon Khaen University in Thailand for a residency in physical medicine and rehabilitation. But by September, the University had notified the MOH that she was struggling with her adjustment (and the rigor and stress of the curriculum, according to Okard Activity staff) and was being treated for mental health and psychosocial issues. As such, she returned to Laos to be with her family and recover. At that time, Okard Activity staff, DHR, and the Department of Professional Health Education, which was also engaged in designing the scholarship program, took time to reflect on the limited number of applicants as well as whether future applicants would be prepared for the scholarship program based on their background. According to WEI staff, scholarship recipients need to have both English and Thai language capabilities; have taken anatomy and physiology classes; and have basic study, critical thinking, and coping skills. However, WEI reports that not many people in Laos meet all of these qualifications and want to study rehabilitation medicine. For that reason, DHR’s scope of work with Okard included the implementation of bridging courses to provide some of these skills to interested candidates, but according to WEI staff, these courses had not yet begun as of the time this report was written due to delays in DHR signing their agreement with WEI. To add to this challenge, the COVID-19 pandemic hit the world in early 2020, and, according to Okard Activity staff and the Y3 Annual Progress Report, restrictions on movement and border closures further delayed the scholarship program. Notes from a meeting between DHR, CMR, and WEI, among others, in March 2020, show that DHR decided to create a short-term taskforce, at WEI’s urging, to strategize on possible alternatives to the scholarships. This work has not yet been completed, and it is not clear why. WEI staff report that it is because DHR has not taken the initiative to push this task force forward (related to assumption #2 below). The COVID-19 pandemic also caused several other challenges for the Okard Activity. In fact, delays and obstacles that arose as a result of COVID-19 were mentioned more than 100 times across 32 of the 47 KIIs conducted for this MTE. According to stakeholders, COVID-19 affected the following: • Timely provision of some assistive devices in the CBID districts (mentioned by Provincial and District Government officials as well as Village Leaders); • Timely provision of income generation activities (IGA) inputs, such as goats, fish, chickens, and grocery store items in the CBID districts (mentioned by Central and District Government officials and CBID Facilitators); • Staffing delays, with some international Activity staff unable to get into Laos due to border closures (according to Okard Activity staff) and some local staff positions unable to be filled (the latter included marketing staff for DMAS, according to DMAS staff); • The signing of COPE’s cooperative agreement (which was eventually signed in April 2021) and was also delayed as a result of capacity and governance issues, described below, according to USAID, Okard Activity, and COPE staff; 25 | EVALUATION FINAL REPORT USAID.GOV • The training of trainers for CMR’s CPDTC, which will provide in-service training to doctors and nurses in Laos on essential rehabilitation. The training of trainers was meant to partially take place in Thailand and through international trainers travelling to Laos, but both were delayed according to CMR staff; • Finalization of the Decree on the Social Fund for persons with disabilities, which was delayed because review meetings were cancelled in the provinces and the capital, according to Central Government officials who report that the decree will create a savings fund to provide needed services and assistive products to persons with disabilities in Laos; and • The reluctance of some CBID beneficiaries to stay in the hospital during the pandemic, which may have affected the success of their treatment, according to Provincial Government officials. According to World Health Organization reports, the COVID-19 pandemic first reached Laos in March 2020, though cases remained low, with weekly average case rates under 20, until April 2021, when confirmed cases jumped up to 574. GoL Prime Minister orders suggest the country was in a complete lockdown from March 30 - May 3, 2020 and has remained closed except for those who have an “essential and urgent need” to enter, which must be approved by the GoL per guidance provided on the MOFA website.13 It then went through a second lockdown beginning on April 22, 2021, after cases spiked, according to the U.S. Embassy in Laos, which local news outlets said was expected to be lifted on July 19, 2021.14 A new PM order was issued in July extending the lockdown to August 3, 2021.15 The ET could not confirm this date, as it was before the date the draft evaluation report was due. Local news reports said that the GoL hopes to vaccinate 22 percent of the population in 2021 and then an additional 48 percent of the population (for a total of 70 percent vaccinated) in 2022. As such, the ET finds that there is significant potential that the COVID-19 pandemic will continue to influence the operating environment in Laos through next year. Another aspect of the Lao operating environment that has not met with assumptions is the issue of UXO land clearance. WEI reports that they did not realize the number of families that would not have a clearance certificate in the CBID communities, nor did they realize how difficult it would be to activate clearance teams. Once WEI asked the U.S. State Department activities if they could help with clearance, they learned that they would have to wait one or two years for that clearance to occur. UXO clearance was critical for families in Xieung Khouang wishing to engage in animal rearing or farming under the IGA component of the CBID Activity. According to the Okard Activity Y4, Q1 report: “Kham District in XKH is highly contaminated with UXO, and the vast majority of inhabitants do not have a certificate verifying that their land has been surveyed and cleared, including USAID 13 MOFA. “Travel advisory for entry and exit of Lao PDR during the implementation of measures to prevent, control, and respond to the COVID-19 pandemic,” June 2020. Accessed at: http://www.mofa.gov.la/index.php/statements/notices/3587- travel-advisory-for-entry-and-exit-of-lao-pdr-during-the-implementation-of-measures-to-prevent,-control-and-respond-to-the￾covid-19-pandemic. 14 U.S. Embassy in Laos. “Government Announces Lockdown of Vientiane Capital Province & Additional COVID-19,” April 2021. Accessed at: https://la.usembassy.gov/government-announces-lockdown-of-vientiane-capital-province-additional-covid-19/. 15 MOFA. “The NOTICE of Extending the implementation of the Measures on Containment, Prevention, and Comprehensive Response to the COVID-19 Pandemic from 20 July - 3 August 2021,” July 2021. Accessed at: http://www.mofa.gov.la/index.php/statements/notices/4254-the-notice-of-extending-tthe-implementation-of-the-measures-on￾containment,-prevention,-and-comprehensive-response-to-the-covid-19-pandemic-from-20-july-3-august-2021. 26 | EVALUATION FINAL REPORT USAID.GOV Okard project beneficiaries. WEI consulted UXO Laos and the Mines Advisory Group during Q1- Y4 and it was confirmed that IGA activities involving disturbing land could trigger UXO risk and, therefore, should be avoided. As a result, the USAID Okard team will need additional time to negotiate a suitable alternative livelihood plan and IGA asset provision with supported households. As this UXO issue is related to the implementation of the National Disability Law (article on equal access to livelihood) in Q2-Y4, NCDE will discuss with the National Regulatory Authority to envisage considering persons with disabilities in the prioritization system in UXO clearance.” Okard Activity staff reported after this quarterly report was issued that they successfully negotiated prioritization for 12 families in Kham District to have their land cleared, but since then, additional families requested support (WEI noted, at the time this report was published, that their request for clearance for those additional families was pending with the GoL, but they felt optimistic). ASSUMPTION 2 The ET also found that the second assumption: “GoL administrative requirements are requested and processed as expected,” did not always hold. Rather, according to USAID and Okard Activity staff and partners, there were significant delays in the signing of WEI’s MOU. According to the Okard Activity MOU Timeline Summary, created by WEI, after a lot of discussion over which ministry would oversee the MOU, WEI submitted the first draft of the MOU to NCDE in February 2018. Following feedback from the NCDE and the MOLSW, WEI submitted an additional eight drafts of the MOU before the MOFA eventually signed in April 2019 (one year and seven months into the Activity period of performance). The ET reviewed drafts of the MOU and notes from MOU feedback meetings with NCDE and other ministries and also discussed the delay with the ministries involved, USAID, and WEI. Through all of this evidence, the team identified some themes of feedback and processes that caused delays in the signing of the MOU: • Initial decision of which ministry would oversee the MOU, which took five months to negotiate. The final decision was for NCDE under the MOLSW to oversee the MOU with feedback and involvement from the MOH, especially DHR and CMR. • Misunderstanding and discontent on the side of various ministries within the GoL on the mechanism of support for the government. Many GoL officials noted concerns, both to the ET and through MOU meetings with WEI, with the fact that USAID was providing support through WEI, who was then providing in-kind support (meeting materials, per diems, transportation costs, and direct payment of activity costs, costs to suppliers, etc.) to the GoL rather than the Government being able to make decisions, manage funding directly, and procure items on their own. One major point of concern was the procurement of vehicles. GoL officials noted that Okard’s budget for government transportation would be more than enough to purchase two vehicles for the GoL. Yet, USAID restrictions meant that the government would, instead, just be provided with reimbursements for travel costs incurred under the Activity. • Government questions and misunderstanding about the cohesiveness of the Activity and over which Okard Activity partners would implement what, where and what each of the partners’ budgets would be. • A desire on the side of the GoL for all Okard Activity equipment to be handed over to the GoL at the end of the Activity, even for equipment purchased by sub-recipients, rather than to USAID. Ultimately, the two parties agreed to the following language, “When the project 27 | EVALUATION FINAL REPORT USAID.GOV does conclude and there is no continuation of the project, World Education will transfer the equipment and materials listed in section 11 to the Ministry of Labour and Social Welfare in order to distribute to concerned partners, dependent on donor approval and in accordance with MOFA processes.” • A desire for the GoL to be involved in all major Activity decisions, especially: − The selection of Okard sub-recipients and short-term technical assistance staff − The selection of the beneficiary districts and beneficiaries. NCDE and DHR wanted more information on how the beneficiary districts, especially Kham District, were selected and how direct and indirect beneficiaries would be selected. They also did not want the MOU to say that the Activity may expand to other provinces depending on the mid-term evaluation. Instead, they saw the Okard Activity as a pilot activity that should only be implemented in two provinces. Depending on the results of activities in the two provinces, the GoL will consider expanding the model to other districts in the same two provinces. • A delay in the official approval of the Activity from the Governor of SVK. • Misunderstanding by Okard staff and some GoL officials on how many INGOs could operate under one MOU. Eventually, in March 2019, Okard Activity staff learned that each INGO working in Laos needs their own MOU with the GoL. Activity staff said that this was a new requirement or interpretation by MOFA of the GoL MOU process that happened after the Okard MOU was in place. The evaluation team was unable to verify these claims. But, this meant that WEI needed to remove HI and all of its activities and budget from its MOU, and then HI had to initiate a new MOU process with the GoL. HI’s MOU was eventually signed in October 2020 (three years into the Activity period of performance). Further, WEI noted that they had to pursue a modification to their MOU given the split, and that modification was finally approved in March 2021 (though activities were allowed to proceed ahead of this given the signed MOU). • A desire to avoid any contingencies in the proposal. NCDE noted in one of the MOU review meetings that they did not want any “ifs” in the MOU. • Availability of the government to meet and discuss the MOU and their feedback and for the NCDE to coordinate and consolidate comments from DHR and CMR. WEI staff noted challenges with coordinating meetings with the government and with the government moving feedback forward. In a general discussion about Activity delays, all three major GoL partners— NCDE, DHR, and CMR—also noted that they were very busy and it is sometimes difficult to coordinate meetings with WEI and HI, who are also busy, which seems to support WEI’s comment about availability for meetings. These challenges meant that the overall Activity burn rate was reduced for the first two years of the Okard Activity, according to WEI’s Okard Context Monitoring Matrix. Further, both WEI and HI report the likelihood for remaining funds at the end of the Activity’s period of performance as a result of these delays. Additionally, with regards to assumption #2, Okard Activity Annual Progress Reports and staff reported that other GoL approvals also took longer than expected. For instance, the Okard Activity Y3 Annual 28 | EVALUATION FINAL REPORT USAID.GOV Progress report noted a delay in the CMR Director’s approval of CMR’s CPDTC SOW and in DHR’s signing of their SOW. Part of the reason for the delay in CMR signing their SOW, according to Okard Activity staff, was confusion on the part of CMR and DHR about USAID rules and regulations and a request from the CMR Director to add procurement of advanced rehabilitation equipment not directly related to the objective of the CPDTC in-kind support to the SOW. Okard Activity staff also reported that additional delays occurred as a result of the MOH’s attention being divided between the Okard Activity, COVID-19 issues, and other projects. Finally, it is clear that many of the activities included in the in-kind agreements signed by DHR, CMR, and NCDE have not been completed, but the ET was unable to pursue additional data collection to determine why this is the case after being provided with the signed in-kind sub-grant agreements in July 2021, about a week before the final MTE report was due. ASSUMPTION 3 Finally, with regards to assumption #3, during KIIs, USAID and Okard Activity Staff reported challenges with partner capabilities and ability to participate in the Activity, including the following: • A lack of a full HI team and turnover amongst the HI team from early 2019 to June 2020 (when HI staff report they had a full team and a turnover mitigation plan in place); • The capacity of the LDPA, which was supposed to implement a significant portion of the stakeholder engagement activities and oversee CBID activities in SVK, to take on a USAID grant, including issues with the segregation of duties and transparency of management and limited staff (only one person) at the provincial level. Eventually, according to Okard Activity Annual Progress Reports, in Y1, HI replaced LDPA with ARMI to implement CBID activities in SVK, and in Y2, they ended the arrangement with LDPA to implement SE activities. In Y3, HI encouraged AfA to apply for the SE subgrant, and HI signed a partnership agreement with AfA in June 2021. • COPE administrative capacity issues around governance and financial management, according to USAID. COPE was supposed to begin working with the Activity in the beginning, but due to these capacity issues and COVID-19 (the latter was only mentioned by COPE staff), their participation was delayed until their subgrant was signed in April 2021. • Turnover amongst government staff, including the transfer of the MOH Deputy Minister overseeing rehabilitation to the Lao Red Cross, and the upcoming retirement of the MOH focal point for rehabilitation. Nonetheless, most stakeholders, including all GoL stakeholders outside of the two noted above (NCDE, DHR, CMR, and Provincial and District Government officials), ARMI, QLA, DMAS, COPE, and HI (the latter two, post challenges identified above) reported that they were willing and able to participate in the Okard Activity. While other stakeholders did identify some capacity challenges with some of these partners (discussed in detail in findings under EQs 4, 7, and 8, below), none of these capacity constraints prevented the partners from participating in the Activity. CBID staff and local government staff report also that while CBID beneficiaries have been “willing and able to engage with the Activity,” that willingness is beginning to wain as a result of the delays in the provision of health and EE supports described above. Table 2 below presents a summary of all findings related to each of the critical assumptions. 29 | EVALUATION FINAL REPORT USAID.GOV TABLE 2: SUMMARY OF CRITICAL ASSUMPTIONS, VALIDITY, AND CHANGES IN CONTEXT EQ2 CONCLUSIONS The context and critical assumptions do not appear to still be valid. Rather, the context has been greatly affected by COVID-19, unexpected administrative delays (especially the lengthy MOU process), and the capacity and willingness of all stakeholders to engage fully with the Activity. The COVID-19 pandemic is not over in Laos, and it is likely to continue to have an effect on the Okard Activity. Further, administrative delays are likely to continue, as the MOH remains busy addressing the pandemic and working on other tasks. While not all stakeholders were willing and able to engage with the Activity in its first three years, Okard Activity staff have successfully transitioned to new partners and resolved issues with remaining partners such that all partners appear to now be willing and able to fully engage. The one exception to this may be with regards to the GoL, who WEI staff report have not yet been able to take ownership over activities and implement their agreed-upon SOW as expected. EQ2 RECOMMENDATIONS Recommendation 2: USAID should take note of the issues and trends in feedback that arose from the MOU process under the Okard Activity and should share these notes with future implementing partners embarking on the MOU process to help the partners more effectively manage the ASSUMPTION HELD? CHANGES IN CONTEXT THAT AFFECT RESULTS 1 - Lao operating environment is stable and conducive for the activity No COVID-19 caused delays and challenges related to IGA and assistive device supports, staffing delays, the signing of COPE’s agreement, and the training of trainers for the CMR CPDTC, among other things. The unexpected need to clear UXOs has proven a challenge, which has required additional negotiations with the GoL. 2 - GoL administrative requirements are requested and processed as expected No MOU signing proved to be a significant challenge that delayed activities by 1.5 years for WEI and a little more than 3 years for HI. The signing of the CMR and DHR SOWs was delayed as a result of several rounds of revisions and the need for Okard to explain USAID rules and regulations to the GoL. The GoL has been delayed in implementing some components of their SOW, including the scholarship component. 3 - All Needed stakeholders are willing and able to engage with the Activity as expected Partly The TVET intervention stalled partly because there was not demand amongst persons with disab in the CBID target districts for TVET, as discussed under findings for EQ1 above. Some HI staff were not available from early-2019 to mid-2020. However, HI’s staff is now at full capacity. Some partners dropped out or did not have the capacity to engage with the Activity. However, WEI and HI have identified new sub-recipients to replace the former ones. GoL partners have not been able to implement activities according to the original agreed timelines. Persons with disabilities and their families are self-motivated, but Activity delays and lengthy processes are challenging that motivation. All partner challenges have been addressed with the exception of perhaps the GoL challenges, and current partners (as of July 2021) are now ready and able to engage with the Activity. 30 | EVALUATION FINAL REPORT USAID.GOV path to MOU signing moving forward. Future IPs should work to ensure their first draft of their MOU includes clear details on which partners will implement which activities and clear processes for government oversight of activities that avoids any contingencies. Recommendation 3: WEI should update its critical assumptions given the evidence that the first two critical assumptions have not held and are unlikely to in the last 14-15 months of the Activity. The third critical assumption remains valid and likely does not need to be changed. The current language for the first assumption signals that the assumption has not held if there are changes to/emergence of natural disasters, disease outbreaks, the political situation, the economic situation, the operating environment for INGOs and CSOs, and GoL policies. This assumption should be adjusted to remove the assumption and signal that no disease outbreaks will occur. It could be changed to read that “no new disease outbreaks will occur,” but WEI should work with the USAID Okard Implementation Management Committee, which includes key government and subgrantee partners, to draft a COVID-19 disruption mitigation plan that addresses possible continued disruptions to Activity components and identifies scenarios to address those disruptions. The second assumption should be updated as well, in light of the ongoing delays in GoL approvals. WEI and its non-government sub-recipient partners should outline the major approvals they know will be needed by the GoL in the last year of the Activity, and should plan extra time in their Y5 workplan for these approvals. The current assumption signal that reads, “Unexpected or unusually lengthy administrative processes (eg. MoU, permissions, work planning, reporting, sub-grant approval)” probably still works, but the amount of time that triggers a note in the matrix related to this assumption should be reconsidered to be more in line with Okard Activity experience with the length of approvals to date. Recommendation 4: WEI and HI should update their overall indicator targets, in close collaboration with USAID, to reflect the dramatic shift in the context. Specifically, WEI and HI should meet with USAID to look at results to date, as presented in this report, and should reconsider targets that Okard is not on track to implement as a result of Activity challenges faced to date. Together, the parties should consider whether the recent removal of many barriers will allow activities to ramp up significantly or if some targets may need to be reduced. In doing so, USAID, WEI, and HI should be sure to consider revised assumptions and the likelihood that COVID-19 and GoL administrative delays may continue to affect some interventions. EVALUATION QUESTION 3 WHAT COMPONENTS OF THE ACTIVITY HAVE BEEN SUCCESSFULLY AND/OR UNSUCCESSFULLY IMPLEMENTED ACCORDING TO THE APPROACH? WHY? Findings for EQ3 are presented by Activity component (health, EE, and SE), and then a summary of what has been successfully and unsuccessfully implemented is included in conclusions section below. A copy of Okard’s Y4 Q2 monitoring results are included in Table 3: Okard Activity Annual Progress Report Results, Inception to Year 4, Quarter 2 below to provide context for the findings under EQ3. Given that WEI did not start implementation in earnest until April 2019 (one year and seven months into the period of performance), despite being in Y4 at the time of the publication of this report, the Okard Activity has only had about 2.5 years to implement activities to date (as of July 2021). As such, the assessment of Activity results might best be compared against Y2 targets. Further, since HI did not start implementing in earnest until October 2020 (when their MOU was signed), they have only had 1.75 years to implement activities. So, their results might best be compared against Y1 or Y2 targets. Even still, it is important to 31 | EVALUATION FINAL REPORT USAID.GOV note that findings on the success of the Activity to date should be considered with a note to the unusually extensive challenges the Activity has faced during implementation, as noted under EQ2 above. In addition to the findings and analysis provided in the section below, the ET also completed a case study of the CBID model as a whole. This case study can be found in Annex 3 and provides evidence of the clear potential of the CBID model. 32 | EVALUATION FINAL REPORT USAID.GOV TABLE 3: OKARD ACTIVITY ANNUAL PROGRESS REPORT RESULTS, INCEPTION TO YEAR 4, QUARTER 2 INDICATOR Y1 TARGET Y2 TARGET Y3 TARGET Y4 TARGET Y1-Y4Q2 CUMULATIVE RESULTS PERCENT OF Y1 TARGET ACHIEVED BY Y4Q2 PERCENT OF Y1 AND Y2 TARGETS ACHIEVED BY Y4Q2 PERCENT OF Y1, Y2, AND Y3 TARGETS ACHIEVED BY Y4Q2 OVERALL ACTIVITY (5-YEAR) TARGET PERCENT OF 5- YEAR TARGET ACHIEVED BY Y4Q2 Standard Indicators ES. 4-1 Number of vulnerable persons (female/male 0 4,010 10,770 11,860 8,035 N/A 200.37% 54.36% 29,324 27.40% child/adult) benefiting from USG-supported social services (compilation of PM1, HI-3, and EI-2) ES. 4-2 20 530 1,000 1,000 2,029 10145.00 % 368.91% 130.90% 3,150 64.41% Number of service providers (female/male) trained who serve vulnerable persons ES. 4-3 10 50 152 155 406 4060.00 % Number of USG 676.67% 191.51% 377 107.69% -assisted organizations and/or service delivery systems that serve vulnerable persons strengthened PM1 Number of targeted beneficiaries 0 3,047 8,431 8,732 7,273 N/A 238.69% 63.36% 21,532 33.78% (female/male child/adult) with increased knowledge/skills and awareness PM4 0 47 116 116 17 N/A 36.17% 10.43% 326 5.21% 33 | EVALUATION FINAL REPORT USAID.GOV INDICATOR Y1 TARGET Y2 TARGET Y3 TARGET Y4 TARGET Y1-Y4Q2 CUMULATIVE RESULTS PERCENT OF Y1 TARGET ACHIEVED BY Y4Q2 PERCENT OF Y1 AND Y2 TARGETS ACHIEVED BY Y4Q2 PERCENT OF Y1, Y2, AND Y3 TARGETS ACHIEVED BY Y4Q2 OVERALL ACTIVITY (5-YEAR) TARGET PERCENT OF 5- YEAR TARGET ACHIEVED BY Y4Q2 Number of targeted vulnerable populations gaining new or better employment as a result of USG assistance Component 1 - Health Indicators HI-1 0 0 Basic: 15 Intermediat e: 8 Advanced: 0 Basic: 5 Intermediat e: 12 Advanced: 6 0 N/A N/A 0.00% Could not locate Unknow Number of service delivery systems with n improved capacity to provide women￾centered care (discussed under EQ6 below) HI-2 0% 0% 60% 70% 0% N/A N/A 0.00% 70% 0.00% Percentage of people (female/male child/adult) receiving “people-centered care” from targeted health and rehabilitation facilities HI-3 0 865 2,120 2,900 712 N/A 82.31% 23.85% 7,140 9.97% Number of people (female/male child/adult) receiving health and related rehabilitation services (including rehabilitation, assistive products, medical treatment for UXO survivors, and MHPSS services) HI-4 0 525 1,347 1,949 17 N/A 3.24% 0.91% 4,346 0.39% 34 | EVALUATION FINAL REPORT USAID.GOV INDICATOR Y1 TARGET Y2 TARGET Y3 TARGET Y4 TARGET Y1-Y4Q2 CUMULATIVE RESULTS PERCENT OF Y1 TARGET ACHIEVED BY Y4Q2 PERCENT OF Y1 AND Y2 TARGETS ACHIEVED BY Y4Q2 PERCENT OF Y1, Y2, AND Y3 TARGETS ACHIEVED BY Y4Q2 OVERALL ACTIVITY (5-YEAR) TARGET PERCENT OF 5- YEAR TARGET ACHIEVED BY Y4Q2 Number of people (female/male child/adult) who demonstrate increase in function or wellbeing Component 2 – Economic Empowerment Indicators EI-1 Number of persons with disabilities 0 8 20 10 0 N/A 0.00% 0.00% 40 0.00% (male/female) who graduate from TVET centers EI-2 Number of persons with disabilities and their 0 98 228 228 50 N/A 51.02% 15.30% 652 7.67% households (male/female child/adult) who receive an IGA or complete job readiness EI-3 Number of persons with disabilities and their 0 36 87 87 0 N/A 0.00% 0.00% 246 0.00% household (female/male child/adult) with improved economic self-sufficiency Component 3 – Stakeholder Engagement Indicators SI-1 0 20% 13% 13% 0% N/A 0.00% 0.00% 59% 0.00% Percent of activities in the Disability Action Plan initiated and monitored by the NCDE 35 | EVALUATION FINAL REPORT USAID.GOV INDICATOR Y1 TARGET Y2 TARGET Y3 TARGET Y4 TARGET Y1-Y4Q2 CUMULATIVE RESULTS PERCENT OF Y1 TARGET ACHIEVED BY Y4Q2 PERCENT OF Y1 AND Y2 TARGETS ACHIEVED BY Y4Q2 PERCENT OF Y1, Y2, AND Y3 TARGETS ACHIEVED BY Y4Q2 OVERALL ACTIVITY (5-YEAR) TARGET PERCENT OF 5- YEAR TARGET ACHIEVED BY Y4Q2 SI-2 Number of participants 0 82 53 13 0 N/A 0.00% 0.00% 161 0.00% (male/female/child/adult) involved in government-led policy processes who self￾identify as a person with a disability SI-3 0 5 11 6 0 N/A 0.00% 0.00% 32 0.00% Number of operational criteria achieved by the MOH and NCDE-related committees Gender-Specific Indicators (discussed under EQ6 below) HI-1 0 0 Basic: 15 Intermediat e: 8 Advanced: 0 Basic: 5 Intermediat e: 12 Advanced: 6 0 N/A N/A 0.00% Could not locate Unknow Number of service delivery systems with n improved capacity to provide women- centered care (discussed under EQ6 below) GNDR-2 0% 50% 50% 50% 0% N/A 0.00% 0.00% 40% 0.00% Percent of female participants in USG￾assisted programs designed to increase access to productive economic resources (assets, credit, income, or employment) Source: ET reorganization of the USAID Okard Indicator Tracking Table, January 2021. 36 | EVALUATION FINAL REPORT USAID.GOV EQ3 FINDINGS Component 1: Health As discussed in the overview of Okard Activity partners above, WEI is responsible for implementing activities under the health component of the Activity. Results for each of the key activities—train health service providers, deliver appropriate assistive products, and promote rehabilitation—from the Okard Results Framework are described below. Note that most of Okard’s indicators, shown in Table 3 above, for Component 1 focus on overall results from this component. With regards to HI-1 and HI-2, women￾and people-centered care, Okard Activity staff report that they developed the tools to begin to measure these indicators in Y3 and planned to roll them out in Y4, which is the reason there are no results in this area yet. H1 - Train health service providers - Under this component of the Activity, WEI had plans to conduct a formative assessment of the rehabilitation sector, to provide in-service training to existing health professionals on screening and rehabilitation, to develop the CPDTC, to support pre-service curriculum revisions in order to include more robust content regarding MHPSS, and to increase capacity of rehabilitation service providers within Laos through a scholarship program for MOH doctors, physiotherapist, and nurses to study in Thailand and through a bridging activity to support the former. To date, WEI has successfully completed its rehabilitation sector formative assessment in the form of the Systematic Assessment of the Rehabilitation Situation (STARS) Report. They also report in their quarterly and annual reports that, after several months negotiating the CMR’s SOW, in December 2020, they signed an agreement with CMR to establish the CPDTC. The rollout of the CMR center was delayed as a result of administrative processes discussed under EQ2, however. Also CPDTC training has not yet commenced partly because of a lack of expert trainers to lead trainings in the center. Indeed, Okard staff, Central Government officials, and the STARS Report all state that there is only one doctor specialized in rehabilitation medicine in all of Laos, which makes finding people to train the core trainers (CMR staff) in the CPDTC difficult. Specialists have to be brought in from other countries to train the trainers for CPDTC. This is why the scholarship program, discussed under EQ2, is so important and needs to be reformed and a new method prioritized to increase the number of specialized doctors in the country who can also provide training to additional health workers through the CPDTC in the future. With regards to in-service training, according to the Y4 Q2 indicator tracking table (ITT), a great number of service providers have been trained between all Activity components. As shown in Table 3, under ES.4-2, the Okard Activity had trained service providers a total of 2,029 person times (noting that providers are counted more than once if trained under more than one set of curricula) as of the end of Y4 Q2 (more than 100 percent of Okard’s five-year target). Key informants interviewed corroborated this evidence, saying that WEI has successfully built the capacity of MOH health professionals across all levels of the government as well as ARMI and QLA staff with regards to screening, rehabilitation, mental health, and referrals. Specifically, the ET found that all seven of the hospital, District, and Provincial Government health officials interviewed reported improved knowledge from Okard health trainings, as depicted by the following quotes: 37 | EVALUATION FINAL REPORT USAID.GOV “Our hospital gets a better result [because of] the training. For example, in the past our staff have not received training. So, when we have a case of a person suffering from epilepsy, we can only transfer them to the provincial hospital. But presently, we can manage this problem since we got the training....I have gained new knowledge from those trainings, especially on persons with neurological disabilities (held in Kham District) and behavior change (held in Vientiane). After observing and engaging in Okard Activity, I now understand the characteristics of persons with neurological disabilities and the related treatment. I have developed an in-depth knowledge in this area, which I didn’t have prior to the training.” - Provincial Govt. Staff “The training has helped increase our capacity and broadened our horizons. In particular, we have learned how to monitor and assess patients from communities on their functioning and well￾being as well as how the caregiver should care for them, which is what is our center does, providing health services.” - Provincial Govt. Staff GoL officials and beneficiaries also reported that village health volunteers have received health trainings. According to Okard Annual Progress Reports, in Y2, WEI staff completed a translation of the World Health Organization’s Mental Health training package, and in Y3, they rolled out the MHPSS training for doctors and nurses in SVK and XKH. All health center staff interviewed reported improved knowledge as a result of this training, as demonstrated by the following quotes: “I have been trained on the communication skills between medical staff and patients. In addition, I have been trained on mental health diseases, such as hallucination, depression, childhood epilepsy, dementia, and mental health problems. I have improved knowledge.” – District Health Center staff “I never knew before that there are many mental health problem cases in Laos until I joined the project and we visited the villages. I never expected that there are many mental health problem cases in one village.” – District Health Center staff One District Health official reported that the District Hospital provided mental health services for 24 patients between late 2020 and May 2021 (the time of the interview) using the knowledge gained in the training. He also reported that the conditions of the patients improved after receiving treatment. Another District Health official noted that his hospital has improved patient relationships as a result of the training. District hospital staff have requested more training. Two of the five District Government Health Officers reported, unprompted, that they would like to receive more training on persons with disabilities, specifically physical therapy, use of assistive devices and exercise machines, and how to screen a person for disabilities. Further, one sub-grantee reported that while the Okard Activity works with district- and provincial-level health centers, the Activity has overlooked community health centers. The STARS Report corroborates this statement, finding that “rehabilitation at the district and community levels is very limited.” And both an Okard Activity staff member and one of the District Health Center officials said that a follow-up MHPSS training is needed. The District official noted that while his knowledge was improved, he still has to read from the training materials when meeting with patients and feels a refresher course would help to solidify knowledge gained. Finally, the Okard Y3 Annual Progress Report states that WEI has been working with the World Health Organization (WHO) to develop an online training for DHR and possibly Provincial Government officials on the WHO Rehabilitation 2030 Agenda and how to strengthen health systems by integrating rehabilitation. WEI had plans to adapt this training and roll it out in Y4. The Y3 Okard Activity Annual Progress Report and Y4 Q1 Report also describe WEI’s coordination 38 | EVALUATION FINAL REPORT USAID.GOV work with the WHO to create a strategy to build the GoL’s capacity to provide MHPSS interventions and to discuss the possibility of a Laos multilateral mental health working group that would facilitate continued MHPSS efforts, both of which are ongoing. H2 - Deliver appropriate assistive products – The STARS Report found that, “Rehabilitation centers provide a limited number of assistive products; this is supplemented by private pharmacies in larger cities.” Under activity 2, according to its Y4 Q1 report, WEI with support from COPE on some activities has plans to develop a list of priority AT in Laos per priority targeted beneficiaries, deliver targeted training courses to increase the capacity of rehabilitation staff to provide AT, organize AT courses through the CPDTC, and integrate rehabilitation and AT curriculum into degree and non-degree education institutions. However, in the same quarterly report, WEI reports that many of these activities have been delayed as a result of delays in signing DHR’s SOW, which WEI reports NCDE signed in Y4 Q1. Okard staff also mentioned that the delay in COPE’s sub-grant agreement also contributed to these delays. Nonetheless, WEI reports that it is working on developing a data collection instrument to create the list of priority AT. Additionally, some beneficiaries in CBID pilot districts have received assistive devices. In fact, in 24 out of the 46 KIIs, stakeholders report that wheelchairs, crutches, prosthesis, handrails, modified steps in their houses, hearing aids, assistive toilets, supportive beds/chairs, and other assistive devices have been distributed to persons with disabilities in the target communities. Six of the 12 beneficiaries and caregivers interviewed reported that they or their loved one had received an assistive device,16 and at least five of those beneficiaries/caregivers reported improved health as a result of the AT, as described in more detail below: “I think it did help improve their functioning. They can contribute to society as well as helping their family. From having no assistive devices or broken devices, they can now become active . . . Providing assistive devices is very helpful for them as the assistive device allows them to play an active role in society. They are able to attend a wedding ceremony, traditional event, and participate in income-generating activities.” – Hospital staff “For example, with hearing-assistive devices, persons who have hearing impairment have never used them before, but after they have been given the devices by the project, they now are able to hear. There are many devices that help to improve and change the lives of persons with disabilities to be better, after the project has been implemented. . . Before bed-bound patients had to stay in the same place because they lay on beds that could not be moved. However, after the project provided them with moveable beds, the patients now can move to other areas in their house freely as they want.” – District Govt. Staff “Another useful device is assistive toilet chair. Before, they needed to have other people to help them when they used a toilet. Since the project provided the chair, they can help themselves, and the other members in the family do not have to assist them every time. For those who received prosthetic legs, they can work to help their family.” – CBID Facilitator “What I like most about the project is the support on the reconfiguration wheelchair. Without the wheelchair, he could not go anywhere. In the past, after waking up, we brought him to lay 16 Note, not all 12 needed devices, which is why the number is not higher. 39 | EVALUATION FINAL REPORT USAID.GOV outside the room. After we have the wheelchair, wherever we go, we can bring him with us. And he is happier when he meets people.” – Caregiver Despite this success, some key informants reported concerns over the appropriateness, usability, and sustainability of ATs, as described in the quotes below (which also suggest that the government believes that increasing their participation in decisions about assistive devices would improve their appropriateness and sustainability): “The procurement of assistive devices is done by HI, but in the end, they’ll be unusable. We suggest that we should get the devices from the company that provides installation and maintenance services as well as a 5-year or 10-year warranty. However, [HI] has their preferred suppliers. We don’t want that because we’re the one who use the devices. We don’t want the device to be hanging there and nobody uses it. It’s only a 10-month program [left]. Once the project completed, the device will be unusable.” – Central Govt. Staff “If possible, we would like the health sector to be directly responsible for the assistive devices provision activities together [with the project and the MLSW]. Although the project is responsible for the procurement of the assistive devices, when handing over to the villages, the officers from the District Health Office or Provincial Health Department [should be involved]. This is to ensure that the assistive device is appropriate and meets the targeted person’s needs because persons with disabilities might get even worse due to improper use of equipment as it’s not appropriate to be used in the neighborhood.” – Provincial Govt. Staff “We received the wheelchair and an assistive tool to support him in the toilet, but it doesn’t come with a cover for him to rest his back on. Therefore, he can’t use it because he can’t balance himself. It’s not very useful.” – Caregiver “No, I don’t want the walk crutch/ The project would like to give it to us, but I don’t want it because we already have it at home—the triple walk crutch. It is not comfortable; my husband might fall down because of it.” – Caregiver Additionally, according to District Health Officers, persons with disabilities, and caregivers, beneficiaries have complained about the need to travel to the capital to get their assistive devices and have reported that sometimes those costs have not been covered. WEI reported that all transportation costs should have been covered by ARMI and QLA prior to shifting these activities to COPE management once their sub-grant was signed. However, there seems to have been a gap in these systems at some point, as two out of the six (one-third) beneficiaries and caregivers who reported traveling for rehabilitation services said their travel costs were not covered. Finally, some assistive devices have been delayed as a result of COVID-19, according to District Health officials, who report that this has been frustrating to some beneficiaries, as they expected to have already received the devices. WEI reports that this delay occurred because the Provincial Rehabilitation Centers (PRCs) closed or decreased services as a result of COVID-19. H3 - Promote rehabilitation as part of the continuum of health care – Under this activity, WEI plans to work with the GoL to revise health policies to advance the role of rehabilitation at each level of the system, conduct a cost calculation study to provide evidence to the MOH of the needed rehabilitation budget, progressively integrate rehabilitation services across levels, launch national information campaigns 40 | EVALUATION FINAL REPORT USAID.GOV to inform health professionals of the importance of rehabilitation, strengthen and expand the rehabilitation services network, support peer-to-peer counseling, and develop and conduct CBID training. While many of these activities have been delayed as a result of delays in the signing of the DHR’s SOW as well as issues with COVID-19,17 WEI has made progress on some of the policy work, establishment of the National Rehabilitation Committee, dissemination of the laws and policies, cost calculations, and training. WEI staff report having contributed to policy development under the TEAM Activity and finalizing some of that work, contributing to finalization the National Rehabilitation Medicine Strategy, which according to the Y4 Q1 report was endorsed by the GoL in October 2018. In the Y2 Annual Report, WEI staff report that: “As a result of consistent support from the USAID Okard team to the Department of Healthcare and Rehabilitation, the Minister of Health endorsed the National Rehabilitation Strategy and signed an agreement to create a National Rehabilitation Committee to coordinate the operationalization of the strategy.” The report further says that the Deptuy Minister of Health chaired the first National Rehabilitaiton Committee in that same year. In interviews, Central Government officials report that Okard has provided support in disseminating the National Rehabilitation Strategy countrywide, implementing the strategy, and training staff. As discussed above, District and Provincial Health Officers report increased capacity with regards to knowledge of Laos’ disability policies. Further, key stakeholders note that the CBID training and the provision of rehabilitation supports to persons with disabilities in the CBID pilot districts has been successful. While Okard only achieved about 10 percent of its overall target for number of people receiving health and rehabilitation services as of the end of Y4 Q2, according to Table 3 above (see HI-3), this is largely a result of the unexpected Activity barriers and delays. Also, WEI reports limited results for HI-4— increase in function or well-being for a total of 17. Staff said this is because they have only recently started to conduct case management discharge interviews, which assess the functional ability of beneficiaries and compare that functional ability with pre-intervention reports of functional ability, according to PIRS. Note that, overall, 321 beneficiaries were selected to receive CBID case management services, so this number has the potential to increase to that level, which even if achieved, is still below the Y2 Activity target. Corroborating findings of improved well-being, however, the ET found that some beneficiaries reported improved function and well-being. Specifically, at least 24 key informants noted increased or improved health and utilization of health services for persons with disabilities in Okard target districts. And, seven out of 12 beneficiaries/caregivers said they or their loved one had received rehabilitation services as a result of the Okard Activity. Five of those reported improved health and well-being as a result. The remaining five out of 12 said they had not received rehabilitation services. Some did not know why not; others said they did not want to go or they didn’t understand how to receive the services. These same respondents reported that the services often improved the quality of life for the beneficiaries, as demonstrated by the below quote: “After the training, we provided the physical therapy for him. He is getting better and stronger. He is able to walk to his aunt and uncle’s house in the neighborhood. He also got some medicine 17 It is important to note the DHR sits within the MOH, which has been very busy managing the pandemic, according to WEI staff. 41 | EVALUATION FINAL REPORT USAID.GOV . . . I am very appreciative of the support from the project for our family, especially in terms of helping to improve my son’s health and relieve the burden of his medical expenses, etc. We are very happy to see him become stronger and healthier.” – Caregiver Despite this progress, three of the four caregivers interviewed reported challenges, mostly related to their own time constraints, in providing physical therapy to their loved one, as described in more detail below: “When we were not busy, we provided him the physical therapy. However, we are busy sometimes. His mother has to go to the rice field. So, on days we’re all busy, he wouldn’t have physical therapy. It would not be the same as in hospital [where] he would have a fixed schedule to receive the physical therapy.” – Caregiver “I think it will be great for him to do the exercise every day. However, I am not available every day. My parents are very old. My husband and I have to work in the rice field. We come home exhausted every night, so we cannot give him the physical therapy.” – Caregiver “When we were at the Rehabilitation Center, they trained us on how to do massage. They said he does not need to take medicine. As I was pregnant, it was difficult for me to give him massage, and I don’t have time to do it.” – Caregiver “We did record the video to practice at home. But the mobile phone was broken; we lost the video; we did not practice because I don’t remember the way to do it.” – Caregiver Notwithstanding the progress made with regard to direct provision of services through the CBID Demonstration Model, the ET found that systemic challenges to rehabilitation remain. Specifically, there is not one clear department leading rehabilitation work in Laos. DHR staff report that CMR is not prepared to take over rehabilitation activities and training, saying, “The main rehabilitation activities should be implemented by the rehabilitation center. However, the activities cannot be implemented because most staff at the center do not understand their roles and responsibilities. The center would like to take over all the works regarding rehabilitation, [but it] is impossible to do that.” Further, CMR staff report that DHR should not be providing oversight over CMR activities, as CMR is a “vertical line department” and does not report to DHR. The STARS Report validates this finding, saying, “The absence of a specific focal point with clear and consistent terms of reference reduces efficiency and contributes to potential miscommunication.” Two issues further complicate rehabilitation services in Laos. The first is that, according to the STARS Report, both the MOH and the MLSW-NCDE manage rehabilitation centers, with little evidence of collaboration. The second is that transfers and referrals from hospitals to the rehabilitation center or vice￾versa and transfers from the village to district to province to central level are inefficient, according to five Provincial and Central Government officials. These officials said that the referrals are sometimes delayed by several months, negatively affecting the health and progress of patients. One official reported a case of a woman who had a stroke and received support in the hospital to help her learn to walk again. She was then referred to CMR to learn how to eat, continue her mobility progress, etc., but the center would not take her for a few months. One caregiver reported needing a referral letter for each separate visit to the rehabilitation center, as well as problems with scheduling appointments and getting the referral letter. The STARS Report also corroborated this, finding that, “rehabilitation at the district and community levels is very limited, and case management/referral pathways are underdeveloped.” These challenges threaten the sustainability of Okard rehabilitation interventions. 42 | EVALUATION FINAL REPORT USAID.GOV Another challenge noted with the sustainability of the Okard Activity results from capacity at the Provincial Government level. Provincial Government officials in XKH reported that QLA has not appropriately involved them in the provision of rehabilitation services through CBID. Three officials said they have not been invited to participate in home visits or data collection efforts, which threatens the quality of the results as well as their buy-in to results. Specifically, they noted that since the QLA team does not include doctors, they are concerned whether they are providing the right support to households. One Provincial Government official said: “The health unit was not always involved, so the implementation was irregular and not as smooth as it should be…The equipment handout was done by QLA without the government’s involvement, which is a problem and a challenge for us. Is it appropriate to donate such equipment? I think that our technical officers should be involved in the donation of all kinds of equipment as well as money. It would be best if this Activity is implemented by the government sector and monitored by the project team.” The same official also expressed concern over their inability to track rehabilitation services provided in Laos for reporting to the Central Government because they have not been included in these processes. “It is true that QLA submitted proposals to the district office and the provincial authority. Sometimes, the authority was not available but they went ahead with the implementation anyway. When the government cannot consistently participate, the monitoring and reporting becomes harder such as monitoring how many patients were sent to the center or patients’ report. Information cannot be collected for Kham District Health Office. Only QLA knows about the patients, which complicates the district’s work on chain reporting and data collection of patients. This should be improved by both the QLA and Kham District Health Office, as well as all relevant departments. If the government is not present, it won’t be convenient for the monitoring and reporting to the upper levels.” Okard Activity staff report that District and Provincial Government officials are invited to participate. WEI said they could not find any evidence of this lack of coordination, and that perhaps issues of Provincial Government officials feeling out of the loop may be an issue of communication between NCDE and MOH. They also reported that the AT provided by CBID staff are the kind that a family could buy at the local pharmacy, and more “complex” AT are not provided by CBID staff but at hospitals. A final set of issues that could constrain the sustainability of the rehabilitation work are the limitations of the National Health Insurance, which stakeholders report does not cover rehabilitation, AT, or mental health medication, resulting in persons with disabilities not getting treatment and hospitals being crowded with people with disabilities they cannot treat and insufficient prosthetics, other assistive devices, and psychiatric drugs. Demonstrating the challenge, one official said, “Even though, we have knowledge and skills for the [mental health] treatment, the lack of medicine makes it impossible for us to provide the treatment.” This issue combined with limited government budgets, as reported by GoL officials and Okard Activity staff, threaten the sustainability of rehabilitation service progress made by the Activity, especially through the CBID Demonstration Model. Component 2: Economic Empowerment As noted above, HI oversees activities under the EE component of Okard, with support from ARMI and QLA for CBID EE activities and DMAS for TVET and other employment support for persons with disabilities. They are responsible for three categories of activities under EE: the development and 43 | EVALUATION FINAL REPORT USAID.GOV implementation of TVET for persons with disabilities, the integration of persons with disabilities into existing channels of employment, and the creation of new IGA. Findings related to components of these activities that have been successfully and unsuccessfully implemented follow. E1 - Develop and implement effective and inclusive technical and vocational training courses – Under this activity, HI, implemented by DMAS, is meant to provide technical support for four targeted TVET colleges in SVK and Vientiane (VTE), create bridge and reference systems between the schools for the deaf and blind and TVET and between other secondary schools and TVET, provide technical support to TVET service providers to adapt TVET curricula to be more disability inclusive, and facilitate support for persons with disabilities to access education and workplace integration support. As a result of their delayed MOU, HI got a late start on these activities. Nonetheless, DMAS has conducted one disability audit for one TVET institution, as described above. Additional details about support for disability audits and services to targeted TVET institutions is included under EQ1 above, which explains that no CBID beneficiaries selected TVET as their option for EE, and the TVET Department reports no persons with disabilities are currently enrolled in TVET institutions. Okard Indicator EI-1 demonstrates these unexpected outcomes, showing that as of the end of Y3, Okard reported no persons with disabilities had graduated from TVET institutions. Outside of this, the only other direct support provided by Okard to TVET institutions to date is related to building TVET Department officials’ capabilities to collect data on persons with disabilities within their institutions. In addition to the work on TVET to date, HI reports in the Okard Y4 Q1 report that it has drafted the comprehensive inclusion assessment and action plan for ensuring bridging between the schools for the deaf and blind and TVET institutions, but bridging with other schools is on hold, pending a “restructure” of the IEC within MOES.18 The IEC is responsible for leading GoL efforts to ensure inclusive education in Laos at all levels. E2 - Integrate persons with disabilities into existing channels for employment – Under EE Activity 2, HI was tasked with supervising DMAS, which is in turn tasked with establishing partnerships with the leading job advertising companies to promote equal opportunity clauses for persons with disabilities and establishing a Business and Disability Network. HI, ARMI, and QLA are tasked with supporting CBID beneficiaries to develop livelihoods plans. As discussed above, DMAS has been successfully developed. In the Okard Activity’s Y4 Q1 report, HI reports that they have: • Finalized mapping tools and guidance for stakeholders to map platforms for job advertisements and placements, a first step in identifying with whom HI will partner to promote equal opportunity clauses for persons with disabilities. • Engaged two employers to advocate for inclusive employment (DMAS led this work, according to the report). The Y3 Annual Progress Report also says that DMAS has reached out to employers to ask about their interest in recruiting persons with disabilities. DMAS staff and the Y3 Annual 18 Note that the IEC was recently moved within the Ministry, and the Director of IEC position is currently open, with an Acting Director currently filling the role, according to WEI staff. 44 | EVALUATION FINAL REPORT USAID.GOV Report also describe some success in this area and note that they have targeted hotels, factories, and NGOs (such as Save the Children and Oxfam). • Through the CBID facilitators, worked with CBID beneficiaries to develop livelihood plans. CBID facilitators corroborated this finding, and some beneficiaries and caregivers also reported they had worked on livelihood plans. HI and DMAS, in KIIs and group interviews, also reported that DMAS has made further progress on helping persons with disabilities to find and secure employment. Providing additional evidence for EI-2 (number of persons with disabilities who complete job readiness or receive an IGA) results reported in the table above, DMAS reported that they supported 47 persons with disabilities in writing CVs and preparing for interviews, and 17 or 19 (they could not remember the exact number) had secured paid jobs as a result. DMAS reported that the remaining 28-30 who did not secure jobs either did not end up applying or could not compete against other job candidates with a four-year degree given that they had two-year TVET degrees. The Okard Activity did not report any progress toward EI-3, related to improved economic self-sufficiency, because those individuals have not yet met the threshold to be counted under EI-3, but it seems if 17 or 19 people secured jobs as a result of Okard, they are likely to have improved economic self-sufficiency. E3 - Provide Income Generation Opportunities – Activities under E3 are largely related to the CBID intervention. Under this activity, HI was meant to conduct a market analysis to identify market￾driven opportunities for training, IGAs, and employment, then help CBID beneficiaries to develop plans for IGA and gain the necessary technical and financial literacy skills to implement those IGA, and finally provide direct in-kind support to implement the activities. According to Okard Activity staff, Government officials, and partners, most CBID beneficiaries and/or caregivers selected agricultural activities or animal rearing for their IGA, a few selected grocery/retail services, and others were more diverse, selecting tailoring, hairdressing, mechanics, etc. Okard staff said they did not anticipate during the Okard Activity design phase that waivers would be required for construction of small animal shelters.19 Since there are many USAID regulations around use of land, procurement of animals, and small-scale construction—put in place, according to USAID, to ensure USAID activities do not cause any harm through spreading animal diseases, exacerbating climate change challenges, etc.—this meant that many of the IGA-support activities were delayed as WEI and HI sought to obtain waivers. Fortunately, as of the time of this report (July 2021), WEI and HI report that they had secured all necessary waivers except one final one for construction. However, COVID-19 conditions were further delaying the delivery of some animals and retail supplies to beneficiaries due to restrictions on movements and shipments. A further complication was the issue of UXO land clearance. WEI and HI report having to work with the GoL to see if they could prioritize land clearance efforts already planned for XKH ahead of efforts in other parts of the country. The GoL agreed and ultimately cleared the land for 12 beneficiaries in XKH, according to WEI staff. However, WEI reports that additional families have since requested IGA 19 They did not believe this level of small-scale construction met the level of construction restricted under the cooperative agreement, nor did they consider that it would be complicated to activate clearance teams and that there would be so many families that needed clearance before IGA could commence. 45 | EVALUATION FINAL REPORT USAID.GOV that will require additional clearances. They have gone back to the GoL to request more support in this area, and were awaiting a final response from the government at the time of this report. The result is that many of the beneficiaries/caregivers report they are awaiting animals, looms, and other IGA supports, and CBID and GoL officials report concerns around continued buy-in if those beneficiaries do not receive those supports soon. One official said, “The point that I am not satisfied is the project has provided supports to target groups slowly as there are too many procedures to follow. Therefore, local villagers have complained about it.” The ET found that nine out of 12 beneficiaries and caregivers interviewed report that they are waiting to receive their IGA supports,20 as depicted by type below: • Four are awaiting pigs – two of these have constructed fences or houses for the pigs • Two are awaiting goats – one of these constructed a house for the goat • One is awaiting silk – this caregiver reports having bought a loom but not receiving the silk21 • Two are awaiting fish (one of these reported that he will receive tools as well since he is also a mechanic) District Government officials report that some beneficiaries have passed away (due to old age) while awaiting services from the Activity. Arguably, WEI and HI should have anticipated these challenges since their cooperative agreement directly says: “Restricted Commodities. The recipient must obtain prior written approval of the Agreement Officer (AO) or comply with required procedures under an applicable waiver, as provided by the AO when procuring any of the following commodities: (i) Agricultural commodities, (ii) Motor vehicles, (iii) Pharmaceuticals, (iv) Pesticides, (v) Used equipment, (vi) U.S. Government-owned excess property, or (vii) Fertilizer.” However, it is possible Okard Activity staff did not expect IGA demand to require so much clearance of land, restricted construction, or restricted commodities. Part of the reason Okard may not have realized this is that they had not conducted a market analysis ahead of the Activity start. That analysis has now been completed, according to WEI, but it has not yet been disseminated, according to quarterly and annual reports. Nonetheless, the commodities restriction barrier has been removed as of the time of this report, as WEI reports it received the necessary waiver in early 2021, after submitting a first draft of the request in March 2020. The two District Government officials who mentioned that some people had passed away while awaiting results also report that the delays caused by the need to procure waivers and await reopening after 20 Note that the remaining three beneficiaries/caregivers report not requesting IGA supports since they do not have time to engage in those activities. 21 Note: silk is a restricted commodity that requires a waiver from USAID. 46 | EVALUATION FINAL REPORT USAID.GOV COVID-19 closes were exacerbated by the length of time it took the Okard Activity to collect data and select beneficiaries. One of those officials said: “The project will end soon, but there is no concrete/obvious result. I think the project has too many procedures, with about seven to eight steps. If it’s possible, I want the project to shorten the procedures like other projects that have been implemented. For example, when they visited Village A to collect the number of persons with disabilities, they should have also collected data for the needs assessment at the same time. After that, the project could have prepared budget plans for them based on their needs. As [another official] said, they have been to villages to collect data too many times. Sometimes they spent only 30 minutes or 1 hour and kept coming in and out. If it’s possible, I want them to directly ask potential beneficiaries what are types of disabilities they have and what supports do they want in one visit. Then, we can directly create a budget plan and provide funds accordingly. They do not have to keep asking beneficiaries for four to five times because they need supports anyway. To be frank, some local people are not willing to cooperate with the project now because the team has always come to collect data with them so many times [with limited results].” Despite these challenges, the ET found that the Okard Activity has been successful in achieving some initial results under CBID activities. In fact, even at the time of data collection, beneficiaries, community members, and District Government officials report that while those the ET talked to had not received their supports at the time of data collection in March and April 2021, they were aware of some people who had received services. Specifically, stakeholders noted that they knew goats had been provided to those beneficiaries requesting goats in Xayphouthong,22 and one stakeholder noted that they were aware of a man running a massage shop out of his house as a result of supports from the Okard Activity. The ET could not confirm these results due to limitations in resources. As shown under E1-2 in Table 3 above, only 50 out of the targeted 652 beneficiaries had received job￾readiness or IGA support as of the end of FY4 Q2 (the latest data provided to the team). However, HI reports that as of the time of this report, additional beneficiaries had received support, and the ET heard from stakeholders that a delivery service was waiting for the borders to be opened following the latest COVID-19 outbreak so that they could deliver 100 goats. Further, since Okard just received the commodities waiver in 2021 and land was also just cleared for 14 beneficiaries, it seems likely that the number of IGA supports provided will begin increasing rapidly, which will affect not only EI-2 but also EI￾3 (improved economic self-sufficiency), notwithstanding additional barriers as a result of COVID-19. Outside of direct provision of IGAs, stakeholders report that service providers and beneficiaries alike have been trained and their knowledge has increased. According to Government officials, CBID facilitators, caregivers, and beneficiaries, the Okard Activity has provided training on vegetable growing; animal rearing for pigs, goats, chickens, frogs, and fish; and community awareness and participation in supporting persons with disabilities. According to Okard Performance Indicator Reference Sheets (PIRS), Okard Activity results for training GoL officials and CBID facilitators in this area are captured under ES.4- 2, and results for training of beneficiaries and caregivers in this area are captured under PM1, as indicator EI-2 does not count a beneficiary in this area until that beneficiary has received the IGA asset (or gone through other non-IGA-related job training, such as training on drafting a CV or going through a job 22 Some informants said two households received the supports and some said ten; the ET could not confirm either of these numbers. 47 | EVALUATION FINAL REPORT USAID.GOV interview). Further, all four District and Provincial Labor and Social Welfare and Agricultural Office staff interviewed reported increased knowledge as a result of these trainings. Four out of the seven beneficiaries and caregivers who requested IGA support and were asked about technical vocational training said that they had received the training. Three of those reported increased knowledge around caring for animals, agriculture, and other such topics. The fourth beneficiary said while he remembers attending the training, his memory is poor and he could not recall the topics from the training. Additional details about CBID Demonstration Model outcomes are included in Annex 3, which contains a case study and story of change about CBID. Component 3: Stakeholder Engagement S1 - Build the capacity of GoL to coordinate and implement relevant policies – As described in the Okard Activity Partners section of this report, HI oversees all interventions under the SE component of the Activity. According to the Okard AMELP and the Sub-Recipients Profile, they are responsible for the following activities, among others, in close coordination with NCDE: capacity building of NCDE to coordinate disability interventions throughout the country; training of relevant ministries at the national, provincial, and district levels; dissemination of the Disability Law; study tours for GoL officials; a pilot activity for disability ID card registration, and a pilot activity related to development of a national database system for persons with disabilities. However, HI and NCDE could not begin their work until October 2020, when HI’s MOU was signed. As such, results in this area have largely been delayed. Nonetheless, the ET did find evidence that some S1 activities had successfully begun, despite Activity barriers. Most notably, WEI (prior to HI’s MOU signing) and HI appear to have made progress on capacity building and dissemination of the Disability Law (S1.4 and S1.5). In its Y4, Q1 report, WEI reports that NCDE conducted two National Disability Law dissemination workshops for GoL officials in December 2020: one in SVK and one in XKH. During these workshops, NCDE presented the National Disability Policy, Strategy, and Action Plan for persons with disabilities in addition to the Disability Monograph (completed previously with HI support and other funding) and other legislation related to persons with disabilities. According to the quarterly report, a total of 241 participants attended from across Laos’ six Southern Provinces and six Northern Provinces. WEI reported that participants from different ministries recognized the need to translate the National Disability Law into specific ministry policies and committed to work with NCDE to do so. Stakeholders confirmed this report, with nine of the 11 District and Provincial Government officials asked about the Disability Law demonstrating their knowledge of it, as shown in this quote from a Provincial Health official: “The Lao government has recently issued the Disability Law, which was published and distributed in 2019. The law states that the rights and duties of persons with disabilities are the inclusion of persons with disabilities in society, such as accessibility, freedom of speech, job creation, and vocational promotion for persons with disabilities. Currently, the law states that there is no discrimination against persons with disabilities. They have the right to believe, the right to move, and the right to access society. Persons with disabilities also have needs and the society should not discriminate and should not set up barriers against them. Persons with disabilities can work in all types of professions in society.” 48 | EVALUATION FINAL REPORT USAID.GOV Many of these district and provincial-level officials also reported participating in disseminating the law at the village level. As noted above, the dissemination efforts have highlighted key policy gaps, also noted by Okard staff, some of Okard’s sub-recipients, and Central Government officials: • Employment – In Laos the law promotes disability inclusion in education and employment in a broad sense. However, Thailand and other countries with more advanced policies have requirements around recruitment and employment for persons with disabilities. If you are a company or a factory with a certain number of employees, you have to employ a certain number of persons with disabilities, according to one grantee. • Transportation – NCDE needs to consider reduced fees for public transportation, according to one grantee and one Central Government official. • Decree on funding for persons with disabilities – The Central Government reports that this is about to be approved. • Gender – There is no policy on women with disabilities specifically, according to Okard staff. • Ministry-specific policies – Most GoL ministries still need support to translate disability law and policy into internal policies within their ministries, according to Okard staff. Overall, nearly all District and Provincial Government staff and hospital staff interviewed report improved knowledge, skills, and capacity overall, as described under HI above and through the following quote: “Prior to the training, my knowledge was very general but after the training, I think myself, the officials, and the persons with disability who have participated in the training have a better understanding. In addition, [the training] has helped build the capacity for public partners, the project, and persons with disabilities to understand about this work better.” – Provincial Govt. Staff According to Central Government officials, the study tours have not been possible as a result of COVID￾19. In its Y4 Q1 report, WEI reports that NCDE, with support from HI, is beginning work on the pilot activity for disability ID card registration and development of a national database system for persons with disabilities. The quarterly report says that HI is working with a consultant and NCDE to review and adapt the Okard disability screening and Modular Tools used to screen persons with disabilities and capture more information about their needs for the CBID pilot for use in the development of a broader disability management information system. Officials from NCDE confirmed that this work has begun. Finally, while capacity building for NCDE has been ongoing according to quarterly and annual reports, these activities have also largely been delayed, as evidenced by SI-1—percent of activities in the Disability Action Plan initiated and monitored by the NCDE. Key informants reported that while coordination is generally good between government ministries and departments working on disability inclusion efforts, there are specific areas where coordination could be improved: 49 | EVALUATION FINAL REPORT USAID.GOV • Coordination between MLSW-NCDE and DHR • Coordination between DHR and CMR • Better integration and improved consultations with the TVET Department on inclusion efforts Evidence of this lack of coordination also comes in a review of Y4 Q2 results from SI-3 (shown in Table 3 above)—Number of operational criteria achieved by the MOH and NCDE-related committees—under which, Okard reports a result of zero. S2 - Strengthen representative and supporting organizations to improve coordination and engagement – Under this activity, HI is tasked with capacity building for disability-focused organizations, empowering and equipping disability role models, and providing technical support to formalize networking and collaboration among local organizations. To date, the Okard Activity has successfully worked to build the capacity of ARMI, QLA, DMAS, and COPE. ARMI staff, who were largely new graduates with little to no experience, report increased knowledge on persons with disabilities, how to look after them and give advice to the caregivers to look after them, compassionate communication, how to be a good facilitator, how to work in the community, how to collect data, etc. QLA staff report having gained the same knowledge as well as skills on leadership, IGA, and peer-to-peer techniques. According to DMAS staff, the service’s capacity has also been increased. They report having increased capacity regarding their understanding and ability to follow USAID rules and regulations, disability inclusion, accessibility audits and reasonable accommodation, job analysis, and the Laos Disability Law. Finally, COPE noted that they have received training from Okard on issues related to administration and financial management. Though COPE did not report whether the organization’s capacity was increased, WEI and HI report that it was. The following quotes illustrate some of the success of the capacity building efforts: “We didn’t have much experience working for the community because we never worked in this field before. Therefore, the trainings have helped to increase our knowledge and capacity, which we can adjust or apply in the community and helps us to know how to talk, access, and work with the community.” – Grantee staff “For me, I have learned so many things, especially the techniques on how to communicate and observe when talking with persons with disabilities. I also have been trained on the family-centered communication approach.” – Grantee staff Despite this increased capacity, DMAS and COPE report that they would like to receive more training to ensure their organizations can be sustainable after the end of the Activity. In their Y4 Q1 report, WEI report that they worked with HI and Research Triangle Institute (RTI) to customize existing Participatory Local Organisational Capacity Assessment (PLOCA) tools, approaches, and methodologies for use with DPOs. They then piloted the tools in Y4 Q1 with ARMI, DMAS, and QLA. WEI reports that, as a result of these pilot workshops, each organization/sub-recipient developed their own Capacity Development Action Plan (CDAP) based on the capacity assessment findings. DMAS staff said they had planned a trip to India, Sri Lanka, or Nepal to learn from organizations similar to theirs, but these trips were cancelled as a result of COVID-19. Staff said they were especially eager to 50 | EVALUATION FINAL REPORT USAID.GOV learn how other organizations have earned business from the government and private sector. They also feel employee capacity is still lacking in some aspects, and they do not believe the limited remaining time under the Okard period of performance will be sufficient to ensure they are able to gain all of the necessary expertise. Outside of piloting the PLOCA and CDAP process with ARMI, COPE, DMAS, and QLA, HI and WEI had not yet rolled out training to more DPOs as of the time of this report (July 2021). However, their workplans showed plans to do so in the future. Nonetheless, given that there is only about one year remaining in the Activity period of performance, they will have little time to oversee implementation of the CDAP’s arising out of the PLOCA process. With regards to the disability role model process, WEI reports in its Y4 Q1 report that together with NCDE, HI, and other key stakeholders, they decided to consolidate trainings to DPOs and make a comprehensive plan to train them on the Disability Law, inclusion, monitoring and evaluation, and disability inclusive development (DID). The idea is the AfA will take over these efforts now that their sub-grant is in place. The AfA is also meant to take on the third component of S2 activities related to coordinating with DPOs in order to advocate to decision makers and service providers to protect and realize the rights and meet the needs of all persons with disabilities and their households. As noted above, LDPA was originally slated to take on this activity, but HI moved their subgrant to AfA following capacity concerns with LDPA. LDPA was originally slated to formalize the DPO network for these purposes, but according to HI staff, they have reduced the scope of AfA’s role to focus on simply building capacity of DPOs and coordinating DPO efforts rather than working to formalize a network. Part of the reason for this is that, according to HI staff, NCDE is uncertain whether AfA has the capacity to lead a formal network of DPOs, given that they have not been leading that group to date and are missing some key components of a typical formal network, such as a clear structure and roles and responsibilities within the group and a budget for coordination. For that reason, HI report that NCDE was not willing to issue a decree through legislation to formalize the AfA-led DPO network they had created called the Organization for Persons with Disabilities (OPD). As a result, OPD is staffed but not legally able to represent or receive government funding or perform any activity. Nonetheless, HI believes they can work with AfA to build their capacity, and they believe AfA has the capacity to complete their SOW related to building the capacity of other DPOs and coordinating their advocacy efforts. While not explicitly in AfA’s or HI’s SOW any longer, HI said that it is possible that after their capacity building efforts, they may be able to ask NCDE to reconsider approving a formal network of DPOs, but they said they need more than a year to build the capacity and confidence of AfA. Now that AfA has a signed agreement, progress in this area should proceed at a faster rate. S3 - Mobilize existing grassroots networks and empower individuals to engage in the community – HI and AfA are responsible for mobilizing this network to empower persons with disabilities and their communities to self-advocate and for raising awareness about persons with disabilities within communities, according to the Sub-Recipients Profile document. To date, HI successfully started the formation of self-help groups (SHGs) meant for persons with disabilities to provide support to one another, to empower each other, and to advocate on behalf of their needs and inclusion. Specifically, HI developed the SHG Handbook, dated July 2020, and according to HI and the Y4 Q1 report, has trained QLA and ARMI on the SHGs. The CBID facilitators then trained district 51 | EVALUATION FINAL REPORT USAID.GOV government staff, who agreed to establish ten SHGs in ten target villages and to start the implementation of SHG activities in Y4 Q1, which ended in December 2020. The ET confirmed that the training took place, with CBID facilitators and District Government officials reporting having been trained. However, none of the beneficiaries or caregivers the team spoke with had heard about or taken part in a SHG. HI reports that this is likely because the SHGs were just beginning to roll out as the ET was collecting information, which was just a few months after the training of GoL officials. Additionally, the Okard Activity appears to have made extensive progress on its social behavior change communication (SBCC) activities. According to the SBCC Strategy and Action Plan, finalized by WEI SBCC staff in Y2 Q2, the Okard Activity had two goals under its SBCC strategy: • Goal 1: Improve understanding on disability at all levels of society to reduce stigma and discrimination toward persons with disabilities, while increasing social inclusion, access, and utilization of health services and access and participation in employment. • Goal 2: Improve understanding on what rehabilitation is (by both health service providers and community members), including information on assistive products, and the benefits of rehabilitation to optimize functioning and reduce disability for persons with health conditions and thus contribute to increasing the demand for and use of rehabilitation services. According to Annual Reports, WEI rolled out the first training focused on Goal 1 during Y2 of the Activity through a workshop with CBID facilitators from QLA and ARMI. The purpose of the training was to help the facilitators understand their own attitudes toward disability as well as the potential attitudes they might encounter in the community and different approaches to behavior change. WEI and HI held an additional series of four-day SBCC Basic Training Workshops in Y3 Q4 for CBID facilitators, DMAS staff, and other stakeholders with the goal of creating a solid foundation of SBCC principles, tools, capacities, and knowledge for raising community awareness about persons with disabilities, what they are capable of, and the challenges they face. According to Y4 Quarterly Progress Reports, QLA and ARMI began rolling out SBCC Focus Group Discussions in communities in Y4. In the Y4 Q1 Report, Okard staff also report that DMAS created a social media campaign dedicated to fostering a stimulating discussion for participants to share dilemmas related to disability and offer perspectives, solutions, and examples. DMAS plans to update their social media pages regularly to engage the public in dialogue around disability inclusion. While during KIIs, most persons with disabilities and their caregivers report that there is no stigma around persons with disabilities, a few reported that there is. Community members, CBID facilitators, District and Provincial Government staff, and one Village Leader reported improved community awareness and acceptance of persons with disabilities in CBID communities as a result of Okard SBCC activities, as demonstrated through the following quotes: “The project has helped persons with disabilities and their families, and local people in communities to understand the rights of persons with disabilities. This is the most significant achievement that they have had, and I am proud of it.” – District Government There is one case that we have helped in the village. The authorities agreed to let a person with disabilities be the village chief, but the community did not accept him because they did not know. The awareness-raising activity enabled the community to accept the differences, and he got voted 52 | EVALUATION FINAL REPORT USAID.GOV to be the village chief. This is a small thing that changes the community’s perception of persons with disabilities.” – CBID Facilitator I see good results. Before, they didn’t care. For example, they only visited us if I visited them first. When I don’t visit them, they won’t visit me, whether disabled or not. After Okard Activity came, people approached each other, which was good. When a person with disabilities fell from the bed, all the villagers came to see her. They were afraid that she would not recover, afraid that she would die.” – Beneficiary At first, we did not know [who] are persons with disabilities, how they are different. The community also did not know. At first, they understood that persons with disabilities are only the amputee group, but after we provided information . . . they started to understand more” – CBID Facilitator In addition to the SBCC Basic Training Workshops, focus group discussions, social media campaigns, and the Quarterly and Annual Reports, the SBCC Coordinator has developed and submitted the SOW for the rehabilitation awareness work, including a short survey on understanding health professionals’ rehabilitation knowledge and practices to DHR for their review. The survey will be launched following DHR’s review and approval, which WEI reports has been delayed. Finally, under S3, DMAS reports and the Y4 Q1 report confirms that DMAS finalized a Diversity and Inclusion Training (DIT) package and has begun to engage with NGOs and private companies on implementing the package and advising on inclusivity. DMAS and HI report that DMAS has provided support to Saylomyen Restaurant, the Child Fund, the Lao Rugby Federation, Plan International, Oxfam, SNV (a Netherlands-based international development organization), Bualapha company, Lao World, and Itecc. As further evidence of the success of this activity and DMAS’ organizational capacity, several of these organizations have reached out to DMAS directly to procure this support. S4 - Engage persons with disabilities in policy design and implementation – USAID, WEI, and HI report that activities and results under activity S4, related to inclusion of DPOs and persons with disabilities in policy and strategy formation and discussions as well as between persons with disabilities and service providers, have been delayed as a result of the delays in HI’s MOU and the switch between using LDPA and AfA to support these activities. HI reports zero results under SI-2 (as shown in Table 3 above), with regard to the number of participants involved in government-led policy processes who self-identify as a person with a disability. This is likely because of the high threshold for including someone as a participant—namely, the need for the meeting or workshop to be at least three hours in length and for the participant to have been in attendance at least 50 percent of the time in multiple-day workshops/meetings. In KIIs and group discussions, the ET found that there are mixed reviews of whether persons with disabilities are currently included in policy discussions. Of the 14 KIIs/group interviews where this was discussed, about half of respondents said they were included, and half said they were not. Key groups identified as perhaps being left out include: • Those from rural areas • People with non-physical disabilities • More than one person representing a specific type of disability 53 | EVALUATION FINAL REPORT USAID.GOV The reason Okard may be reporting a zero for inclusion of persons with disabilities is because the network for including DPOs in policy discussions is not yet institutionalized. Nonetheless, many stakeholders report inclusion of persons with disabilities in policy discussions (that do not rise to the level of reporting under SI-2). EQ3 CONCLUSIONS The ET has summarized components successfully implemented to date and those unsuccessfully implemented to date in Table 4 below, focusing on the major activities outlined in the Results Framework presented in Figure 1 above. Please note that given that relatively few of the Activity targets have been met to date, as a result of the numerous challenges discussed under EQ2 findings above, the ET considered a component successful if good progress has been made since barriers were removed as well as if group interview and KII respondents reported improved services, capacity, and well-being. Further, the ET has not classified some activities, such as SHGs, for which WEI and HI report some progress, but the ET did not have enough independent evidence to verify that success. TABLE 4: OKARD ACTIVITY COMPONENTS SUCCESSFULLY AND NOT YET SUCCESSFULLY IMPLEMENTED TO DATE ACTIVITIES SUCCESSFULLY IMPLEMENTED BY COMPONENT ACTIVITIES NOT YET SUCCESSFULLY IMPLEMENTED Health H1 - In-service training of health service providers on rehabilitation H1 - Long-term capacity and expertise in rehabilitation (by developing more rehabilitation specialists) H1 - Increased knowledge and awareness of mental health challenges and compassionate care amongst service providers H1 - Community-level expertise in mental health screening, and follow-on training for District and Provincial Health Officials to help solidify their knowledge H2, H3, and CBID - Provision of rehabilitation services and AT to CBID beneficiaries H2 - Better coordination with DHR and CMR on the purchase of AT devices for CBID beneficiaries to ensure sustainability H3 - Strengthening and integration of the Lao rehabilitation system by ensuring a key lead department for rehabilitation and ensuring funding is available to sustain rehabilitation and AT services for persons with disabilities either through direct government support or improved National Health Insurance Economic Empowerment E1 - Accessibility audit for one TVET institution and plan for data collection E1 - Capacity development for TVET service providers, and a strengthened TVET system overall E2 - Provision of advice on inclusion and accessibility to international and local NGOs and private companies E2 - Sustainability of DMAS (findings and recommendations discussed under EQ8) E2 - Formalization of DMAS E2 - Employment of persons with disabilities in paid jobs E2 - Long-term systems developed to sustain the employment pipeline E4/CBID - GoL officials, caregivers, and beneficiaries trained and reporting improved knowledge in key IGAs E3/CBID - Delivery of actual IGA inputs, though these appear to be primed for delivery soon 54 | EVALUATION FINAL REPORT USAID.GOV ACTIVITIES SUCCESSFULLY IMPLEMENTED BY COMPONENT ACTIVITIES NOT YET SUCCESSFULLY IMPLEMENTED Other - Removal of key barriers to IGAs E3/CBID - One pending construction waiver and UXO clearance request Stakeholder Engagement S1 - Capacity built amongst District and Provincial Government officials S1 - NCDE capacity building to improve coordination across the government for disability inclusion S1 - Support in the development and dissemination of the Disability Law and other related laws and strategies S1/S4 - Development of Ministry-level policies and more specific policies for education, transportation, employment, and gender S2 - ARMI, COPE, DMAS, and QLA capacity increased S2 - Additional DPO capacity development not yet achieved (but planned) and further capacity development of DMAS and COPE pending; DMAS study tours have also not happened S3 - SBCC campaigns and improved service provider and community awareness S3/S4 - Formalization of a DPO network to provide feedback on policies and strategies and advocate to service providers EQ3 RECOMMENDATIONS Recommendation 5: USAID should consider providing the Okard Activity a no-cost extension. Numerous delays, many of which were out of the control of key Okard partners—such as the delayed MOUs, administrative delays on the side of the government, partner capacity and turnover issues, the need to navigate USAID rules and regulations and advocate for land clearance to enable IGA supports, and the COVID-19 lockdowns—seriously affected the ability of Okard to be where it should be at this mid-term of its funding cycle. However, the Activity is now clearly very well set up to achieve its targets, having received all necessary MOUs (the last received just nine months before this report was produced), recently finalized all partnership agreements (the last just finalized one month before report writing, in June 2021), and having recently received UXO clearance support and construction and restricted commodities waivers. Moreover, in the short time that the project has operated with all of its partnerships and approvals in place, important progress was made toward reaching Activity targets. The ET suggests that USAID and WEI negotiate the length of this no-cost extension to best meet the needs of both parties. It further recommends that WEI and HI should generate new final Activity targets based on an improved understanding of the current context in Laos. Recommendation 6: WEI should prioritize Okard Activities focused on strengthening rehabilitation and mental health within the health system in Laos in close collaboration with DHR and CMR. This should start by working with DHR and CMR to assess what level of availability they have to be involved in these processes currently given their competing demands with managing the COVID-19 pandemic, and then together articulating a new timeline for activities that takes into consideration COVID-19 barriers and lengthy approval processes. Key priorities should include: supporting DHR and CMR to organize a functional rehabilitation committee and specific technical sub￾committees, where the lead for rehabilitation and the roles and responsibilities for operationalizing the rehabilitation strategy could be decided and agreed upon, including drafting a plan for improving the referral system and access to mental health services and supporting the DHR and CMR in defining a realistic plan for funding rehabilitation, mental health services, and ATOne option Okard could suggest to 55 | EVALUATION FINAL REPORT USAID.GOV DHR and CMR for future funding is to consider an upgrade to the National Health Insurance plan to include rehabilitation, mental health, and AT services. Recommendation 7: 23 WEI and DHR should together reconsider the DHR Scholarship Activity, designed to increase capacity for more Lao therapists and rehabilitation experts. They should either explore whether there are other (non-MOH) doctors and nurses in Laos with the technical capacity, language skills, and interest to pursue advanced credentialing in the field and/or expatriate trainers from Thailand and other countries with more advanced rehabilitation systems. If WEI and DHR decide the first option is feasible, they may want to consider contingent scholarships that require recipients to work within the MOH for a period of time following the completion of their degrees. Once a decision on a path forward is made, WEI and DHR should work together to set a timeline for the implementation of the activity to ensure a clear, feasible, and timebound plan with targets is created to increase the number of rehabilitation experts in Laos in the coming years. Recommendation 8: USAID Okard should work with DHR and CMR when procuring assistive devices for CBID beneficiaries to collaboratively select providers who offer the best warrantees and service on the devices. Recommendation 9: HI and DMAS should prioritize their efforts to: 1) continue building demand amongst employers for recruiting persons with disabilities to their staff, and 2) identify and work with job advertising organizations to define a long-term plan for increased employment of persons with disabilities in Laos through the provision of equal opportunity clauses in employment announcements. Recommendation 10: WEI and HI should conduct a definitive needs assessment of CBID beneficiary needs for all beneficiary cohorts to ensure any final waiver and UXO clearance requests are complete and additional waivers and GoL clearance requests are not needed. Recommendation 11: WEI and HI should encourage the GoL to develop more specific ministry-level policies and procedures, action plans, and targets to support the National Disability Law and the National Disability Policy, Strategy, and Action Plan. Specifically, Okard should support the GoL in considering policies on employment requirements for companies, transportation fees, and women with disabilities, among others. WEI and HI staff might provide the GoL with suggested language and best practices from other countries for the GoL to consider in drafting policies. To ensure the Action Plans and new policies remain effective and continue to advance the goals of the Disability Law, Okard Activity staff might also support the GoL to consider setting realistic, timebound targets for results that support the plans over the next 10 or so years. Recommendation 12: WEI should continue to build the capacity of health centers and hospitals. Specifically, they should provide training to CBID district hospitals on physical therapy, use of assistive devices, and screening of persons with disability. They should also expand training to the community health centers in CBID target districts to support persons with disabilities, especially with regards to rehabilitation and mental health screening. 23 Recommendation stems from findings under both EQ2 and EQ3. 56 | EVALUATION FINAL REPORT USAID.GOV Recommendation 13: HI should work closely with the AfA and other DPOs to build their capacity and the structure of the Lao DPO network so that NCDE may feel more comfortable with establishing a formal network. Regardless of what level of network is established, HI should work with the NCDE to establish protocols for engaging DPOs in policy reviews and other relevant reviews of services to ensure the sustainability of inclusive policy and service development in Laos moving forward. Recommendation 14: When there are unexpected delays in the provision of IGA inputs, Okard staff should clearly communicate those delays and estimates of when beneficiaries will likely receive services. Also, CBID staff should not promise specific inputs until they know those inputs have been cleared by USAID to avoid cases where beneficiaries invest their own money and do not receive the promised complimentary support in a timely manner. Recommendation 15: Okard should work with DHR and CMR to identify alternatives to at￾home physical therapy and exercises for families who may not have the capacity to provide that support at home. Then, when determining that rehabilitation is needed, CBID facilitators should work with households to determine whether they will have the time needed to provide the services at home or if another option for physical therapy might prove more sustainable and beneficial for their family member with a disability. EVALUATION QUESTION 4 TO WHAT EXTENT HAS USAID OKARD BEEN EFFECTIVE IN ACHIEVING ITS EXPECTED RESULTS, PARTICULARLY IN TERMS OF: BUILDING/STRENGTHENING THE SYSTEMS AS DESCRIBED IN THE DESIGN DOCUMENT (SYSTEM-CENTERED)? EQ4 FINDINGS AND CONCLUSIONS Many of the findings for EQ4 are detailed under EQ3 above since EQ3 focuses on successes and challenges across the Activity as a whole. As discussed under EQ3, the Okard Activity has contributed to the development and dissemination of policies focused on the rights of persons with disabilities in Laos, which has come a long way to strengthen the system. Further, the Activity has been very successful in the provision of trainings to increase the capacity of the GoL with regards to health, especially mental health and psychosocial support, and EE services for persons with disabilities. However, the Laos rehabilitation system is still in need of reform and capacity development (as recognized by the MOH in their 2013-2025 Health Sector Reform Strategy, which touches on rehabilitation but does not call out all of the challenges identified in the STARS report and this MTE), as is the TVET system. In terms of sustainability of results from improved systems, key stakeholders interviewed report believing that several aspects will be sustainable, including: • Stronger policies • Community awareness • IGA activities, especially agricultural and animal activities (most believe these will be sustainable, but some have questions) 57 | EVALUATION FINAL REPORT USAID.GOV Nonetheless, stakeholders reported mixed reviews on the sustainability of capacity. While many stakeholders report that capacity will be sustained, at least two mentioned concerns noting that one-time trainings may not result in sustained capacity due to government turnover. Some District Government officials and one beneficiary report concerns with IGA sustainability, noting that many persons with disabilities are unable to care for animals. While the caregiver says they will care for the animals, the GoL officials worry whether they will have time and whether they can afford food for animals. Other stakeholders note that health insurance limitations could be an issue for sustainability of health work. Finally, some stakeholders report concerns over the sustainability of the CBID model (which has shown clear promise, as discussed under EQ3 above and in the Case Study on CBID from Annex 3) after the Activity ends given funding challenges (to pay for CBID facilitators, for instance). EQ4 RECOMMENDATIONS Recommendation 16: Given the success to date and immediate future potential of CBID activities, USAID should consider scaling up the CBID model under any follow-on activities focused on inclusion of persons with disabilities in Laos. However, in writing the SOW for such a follow-on activity, USAID should be careful to consider how CBID activities might be sustained by the GoL and its partners moving forward. This might include ensuring the SOW calls for the implementing partner or grantee to work with the GoL to build such activities into their budgets or to work with CSOs like QLA and ARMI to identify a strategic plan for funding their case work moving forward. Further, in writing the potential SOW, USAID should consider targeting of potential follow-on provinces and districts in line with Recommendations 17 and 18. EVALUATION QUESTION 5 TO WHAT EXTENT HAS USAID OKARD BEEN EFFECTIVE IN ACHIEVING ITS EXPECTED RESULTS, PARTICULARLY IN TERMS OF REACHING TARGET BENEFICIARIES? WHICH GROUPS ARE UNDERSERVED, IF ANY, AMONG ALL TARGETED GROUPS, AND WHY? PEOPLE-CENTERED? EQ5 FINDINGS The USAID Okard Activity Cooperative Agreement specifies the target group for the intervention of “women, men, girls, and boys with disabilities, and their households, prioritizing UXO victims, victims of war and conflict, and persons with disabilities who have mobility limitations.” The agreement also notes that: “WEI will be careful to limit project support for non-mobility impairments to those impairments that result from injuries from UXO, conflict, and war or other such trauma.” “Lesbian, gay, bisexual, transgender, questioning, and intersex (LGBTQI) people face significant discrimination in Lao society. Other groups vulnerable to discrimination include the elderly, women and children, and ethnic minorities. Persons with disabilities who identify as one or more of these vulnerable groups will be targeted for additional support and inclusion during USAID Okard.” The Gender and Inclusive Development Action Plan (GIDAP), produced by WEI following a gender and inclusivity assessment in Y1 and updated in Y4 (2021), further supported the finding that the elderly, 58 | EVALUATION FINAL REPORT USAID.GOV women and children, the LGBTQI population, and ethnic minorities are amongst the most vulnerable groups of persons with disabilities. Despite this clear target group listed in the Cooperative Agreement, the ET found that the CBID beneficiary selection process did not include cause of impairment as a key criterion. This may have been partly because of findings noted in the GIDAP that, “Most rehabilitation in Lao PDR has been targeted to war-related injury and may not be responsive to the complex health needs of a changing population (increase of non-communicable diseases and aging).” It may also have been a result of a realization by Okard Activity staff of the relative low prevalence of UXO victims amongst the target population. Whatever the reason, Okard Activity staff used two criteria for prioritization of beneficiaries. The first was level of difficulty, according to the Washington Group Short Set (WGSS) questions, which has become international best practice for screening for persons with disabilities. The WGSS includes six core functional domains to capture data on difficulties with seeing, hearing, walking, cognition, self-care, and communication. Each question asks about difficulties in doing the activity due to a health problem using the four-point Likert scale: “no difficulty,” “some difficulty,” “a lot of difficulty,” and “cannot do it at all.” According to the Okard Activity’s Prioritization Framework for Needs Assessment, Okard staff used these ratings to calculate a vulnerability score, called Difficulty in Functioning, which is scored as described in Table 5 below. They also collected data on affordability of basic needs (used as a proxy for socio￾economic status (SES) and assigned scores for affordability of needs (AN) based on the matrix presented in Table 6 below). TABLE 5: DIFFICULTY OF FUNCTIONING SCORING MATRIX FOR BENEFICIARY SELECTION ACCUMULATED RESPONSES ACROSS SIX DOMAINS SCORE Answered as “no difficulty” or ONLY one answered as “some difficulty” 0 Answered with multiple difficulties as “some difficulty” ONLY 1 Answered only one domain as: “a lot of difficulty”, and the rest are “no difficulty” 3 Answered only one domain as: “cannot do at all”, and the rest are “no difficulty” 6 Answered multiple difficulties with at least one domain as “a lot of difficulty” but none as “cannot do at all” 8 Answered multiple difficulties with at least one domain answer “Cannot do at all” 10 Source: USAID Okard Prioritization for Need Assessment document provided by WEI 59 | EVALUATION FINAL REPORT USAID.GOV TABLE 6: AFFORDABILITY OF BASIC NEEDS SCORING MATRIX FOR BENEFICIARY SELECTION GIVEN THE RESOURCES AVAILABLE IN YOUR HOUSEHOLD, CAN YOU MEET BASIC ___ NEEDS? WEIGHT ANSWER FROM SCREENING FORM AND ITS SCORE AS MUCH AS NEEDED MOST OF THE TIME SOMETIMES NOT AT ALL Food 3 0 2 5 10 Medicine 2 0 2 5 10 Education 1 0 2 5 10 Source: USAID Okard Prioritization for Need Assessment document provided by WEI Okard’s Prioritization Framework notes that the scores from these two matrices are used to calculate a vulnerability score using the following formula (with the eight and five representing the weights of each criterion, which prioritizes difficulty in functioning): 𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉 𝑆𝑆𝑆𝑆𝑆𝑆 𝑖𝑖 = (8𝐷𝐷𝐷𝐷𝑖𝑖 + 5𝐴𝐴𝐴𝐴𝑖𝑖) (8 + 5) WEI staff report that while they use the prioritization tool as a guide for beneficiary selection, other factors are also considered, including the capacity of the CBID facilitators to manage cases related to the timing of training they have received, the geographic location and efficiency of case management loads, and the need to balance case loads. The ET found that this set of criteria led to the selection of individuals presented in Table 7 below. Note that to create this table, the ET requested the databases of those screened for CBID participation and CBID beneficiaries from WEI. However, as discussed in the evaluation limitations section above, WEI was unable to provide the data due to concerns about having not included research as a possible use for the data in their consent statements to screened individuals. As such, the team relayed a data request to WEI for summary data on the number of screened and selected individuals meeting a certain set of criteria (such as level of difficulty-by-difficulty type, affordability of basic needs, sex, cause of impairment, etc.). To classify affordability of basic needs, the ET requested that WEI calculate total AN scores for all screened individuals and then use those scores to divide screened applicants into thirds. Ultimately, this meant that WEI classified AN scores according to the following criteria: • One-third with the Most Affordability = 0-4.15 • One-third with Median Affordability = 4.16-5.83 • One-third with Lowest Affordability = 5.84-10 The team then compared the data between screened and selected applicants and summarized across various categories to produce the tables presented in this section of the report. 60 | EVALUATION FINAL REPORT USAID.GOV TABLE 7: TYPE AND LEVEL OF DIFFICULTY OF SCREENED AND SELECTED CBID BENEFICIARIES BY SES DISTRICT SCREENED OR SELECTED MOBILITY DIFFICULTIES NON-MOBILITY DIFFICULTIES TOTAL NO DIFFICULTY SOME A LOT CANNOT SOME A LOT CANNOT One-Third of Screened Applicants with the Least Needs (Most Affordability) XPT Screened 2 19 93 36 15 39 17 221 Selected in First Round 0 2 38 19 1 10 9 79 Percent of Screened Population Selected 0% 11% 41% 53% 7% 26% 53% 36% Kham Screened 34 61 196 57 63 149 53 613 Selected in First Round 5 3 27 27 2 11 17 92 Percent of Screened Population Selected 15% 5% 14% 47% 3% 7% 32% 15% Total Percent of Screened Population Selected 14% 6% 22% 49% 4% 11% 37% 21% One-Third of Screened Applicants with Median Needs (Median Affordability) XKP Screened 6 10 44 17 19 32 15 143 Selected in First Round 0 1 15 5 0 6 5 32 Percent of Screened Population Selected 0% 10% 34% 29% 0% 19% 33% 22% 61 | EVALUATION FINAL REPORT USAID.GOV DISTRICT SCREENED OR SELECTED MOBILITY DIFFICULTIES NON-MOBILITY DIFFICULTIES TOTAL NO DIFFICULTY SOME A LOT CANNOT SOME A LOT CANNOT Kham Screened 2 9 74 12 13 45 12 167 Selected in First Round 1 1 41 6 2 15 7 73 Percent of Screened Population Selected 50% 11% 55% 50% 15% 33% 58% 44% Total Percent of Screened Population Selected 13% 11% 47% 38% 6% 27% 44% 34% One-Third of Screened Applicants with the Most Needs (Lowest Affordability) XPT Screened 7 9 19 15 17 24 6 97 Selected in First Round 0 3 4 8 3 2 2 22 Percent of Screened Population Selected 0% 33% 21% 53% 18% 8% 33% 23% Kham Screened 0 2 10 7 0 6 5 30 Selected in First Round 0 0 8 6 0 4 4 22 Percent of Screened Population Selected 0% 0% 80% 86% 0% 67% 80% 73% Total Percent of Screened Population Selected 0% 27% 41% 64% 18% 20% 55% 35% Source: ET analysis of WEI data on screened and selected CBID beneficiaries 62 | EVALUATION FINAL REPORT USAID.GOV As shown in Table 7 above, the ET found that the percentage of individuals selected was highest for those with the least affordability and highest level of difficulty in relation to mobility (with 64 percent of that population selected). However, not all of those with the least affordability and highest level of difficulty with relation to mobility challenges were selected. WEI staff report that this is likely because of other factors that were considered in the selection process, discussed above, namely CBID facilitator training, geographic location, and the need to balance case loads. Further, the categories for difficulties below show a summary of the highest level of difficulty a person reported across each of the six types of functions, and whether the person had one or multiple difficulties, they are presented together. As such, it could be that some of those who fall into the most needs category in Table 7 only had one reported difficulty while others had multiple difficulties. Further, according to the data collection tools provided to the ET, people screened were not asked questions about sexual orientation or ethnicity, despite the inclusion of these groups in the list of the most vulnerable in the Cooperative Agreement, which says they will be prioritized for services. Part of the reason for this, according to WEI staff, was that they determined that there were sensitivities around asking questions about sexual orientation. Indeed, the GIDAP provides some evidence of these concerns, saying, “We suspect health service providers have low levels of understanding on sexual orientation and gender identity, and LGBTI’s expectations are not met in terms of how they want to be treated when seeking health services, including prosthetic and physical therapy services.” On the other hand, while cause of impairment was not a key criterion for selection, data collection tools did include a question about this. As such, the ET was able to analyze whether persons with disabilities arising from UXO accidents received services more frequently than those with other causes of impairment. Results of that analysis are included in Table 8: Screened vs. Selected CBID Beneficiaries by Level of Difficulty, SES, and Cause of Impairment below. 63 | EVALUATION FINAL REPORT USAID.GOV TABLE 8: SCREENED VERSUS SELECTED CBID BENEFICIARIES BY LEVEL OF DIFFICULTY, SES, AND CAUSE OF IMPAIRMENT CAUSE OF IMPAIRMENT SCREENED APPLICANTS BY LEVEL OF DIFFICULTY SELECTED CBID BENEFICIARIES BY LEVEL OF DIFFICULTY PERCENT OF SCREENED APPLICANTS SELECTED BY LEVEL TOTAL % OF GROUP SELECTED NONE SOME A LOT CAN NOT DO NONE SOME A LOT CANNOT DO NONE SOME A LOT CANNOT DO One-Third of Screened Applicants with the Least Needs (Most Affordability) Born with it 0 22 88 76 0 3 30 46 N/A 14% 34% 61% 42% Disease 0 23 86 25 0 1 15 13 N/A 4% 17% 52% 22% UXO Accident 0 15 25 1 0 1 7 0 N/A 7% 28% 0% 20% War 0 11 43 5 0 1 4 1 N/A 9% 9% 20% 10% Old Age 0 8 41 4 0 0 3 1 N/A 0% 7% 25% 8% Other Accident 0 48 110 27 0 2 27 11 N/A 4% 25% 41% 22% 64 | EVALUATION FINAL REPORT USAID.GOV CAUSE OF IMPAIRMENT SCREENED APPLICANTS BY LEVEL OF DIFFICULTY SELECTED CBID BENEFICIARIES BY LEVEL OF DIFFICULTY PERCENT OF SCREENED APPLICANTS SELECTED BY LEVEL TOTAL % OF GROUP SELECTED NONE SOME A LOT CAN NOT DO NONE SOME A LOT CANNOT DO NONE SOME A LOT CANNOT DO N/A (no difficulty in functioning) 34 0 0 0 5 0 0 0 15% N/A N/A N/A 15% One-Third of Screened Applicants with Median Needs (Median Affordability) Born with it 0 7 33 16 0 1 23 14 N/A 14% 70% 88% 68% Disease 0 5 30 7 0 1 15 3 N/A 20% 50% 43% 45% UXO Accident 0 4 7 1 0 1 5 1 N/A 25% 71% 100% 58% War 0 0 16 2 0 0 6 1 N/A N/A 38% 50% 39% Old Age 0 1 12 2 0 0 2 1 N/A 0% 17% 50% 20% Other Accident 0 6 42 6 0 1 26 3 N/A 17% 62% 50% 56% N/A (no difficulty in functioning) 2 0 0 0 1 0 0 0 50% N/A N/A N/A 50% One-Third of Screened Applicants with the Most Needs (Least Affordability) Born with it 0 5 3 15 0 3 3 14 N/A 60% 100% 93% 87% Disease 0 1 3 0 0 1 2 0 N/A 100% 67% N/A 75% UXO Accident 0 0 3 1 0 0 2 1 N/A N/A 67% 100% 75% War 0 0 6 0 0 0 5 0 N/A N/A 83% N/A 83% Old Age 0 0 3 0 0 0 2 0 N/A N/A 67% N/A 67% 65 | EVALUATION FINAL REPORT USAID.GOV CAUSE OF IMPAIRMENT SCREENED APPLICANTS BY LEVEL OF DIFFICULTY SELECTED CBID BENEFICIARIES BY LEVEL OF DIFFICULTY PERCENT OF SCREENED APPLICANTS SELECTED BY LEVEL TOTAL % OF GROUP SELECTED NONE SOME A LOT CAN NOT DO NONE SOME A LOT CANNOT DO NONE SOME A LOT CANNOT DO Other Accident 0 2 4 6 0 2 4 5 N/A 100% 100% 83% 92% N/A (no difficulty in functioning) 0 0 0 0 0 0 0 0 N/A N/A N/A N/A 0% Source: ET analysis of WEI data on screened and selected CBID beneficiaries 66 | EVALUATION FINAL REPORT USAID.GOV Table 8 shows that while a significant percentage of those who are victims of UXO accidents were selected as CBID beneficiaries, the percentage of those selected who were born with their difficulties was higher. WEI staff confirmed that 57 UXO survivors were identified in screening, and 18 of those were prioritized for case management. Staff reported that the remaining 39 others are on the waitlist for services.24 This finding does not appear to meet the target population specified in the cooperative agreement, since it says that if non-UXO survivors are provided support, WEI should limit that support to “non-mobility impairments that result from injuries from UXO, conflict, and war or other such trauma.” WEI staff said the reason the selected group does not meet with target population goals is a result of what they found to be the reality of needs in the target districts, including the finding noted above from the GIDAP that UXO survivors are not always the most at-need group. However, the ET could not find any data to support this statement. Some UXO survivors listed, for instance, as having “a lot” of difficulty or being unable to complete certain functions appear not to have been selected. There could be logical reasons for this, but they are not documented in a way that the ET was able to review those reasons and deviations. Further, it appears as though six CBID beneficiaries with no difficulties (17 percent of all those screened who reported no difficulties) were selected to receive services. WEI reports that these people who received services but reported no functional difficulties had acute health conditions identified during screening, and staff deemed these conditions important to treat, as if left untreated they could lead to long-term impairment. The ET also analyzed results by gender, given the importance of gender in the cooperative agreement, the March 2018 USAID Lao PDR Gender Analysis on Disability, and USAID Automated Directives System (ADS). Results are presented in Table 9 below. 24 WEI also noted that not all 39 UXO survivors identified with functional difficulties are likely to need support. For instance, WEI said that of the 18 prioritized cases, only 11 needed support; the remaining 7 seven had no “flagged” needs and have been made passive in the case management system.25 Holzaepfel E., Anderberg L, Feenstra M, Seibold K. 2018. USAID Lao PDR gender analysis on Disability Gender analysis final report, Social Impact. 67 | EVALUATION FINAL REPORT USAID.GOV TABLE 9: SCREENED VERSUS SELECTED CBID BENEFICIARIES BY LEVEL OF DIFFICULTY, SEX, AND CAUSE OF IMPAIRMENT CAUSE OF IMPAIRMENT SCREENED APPLICANTS BY LEVEL OF DIFFICULTY SELECTED CBID BENEFICIARIES BY LEVEL OF DIFFICULTY PERCENT OF SCREENED APPLICANTS SELECTED BY LEVEL TOTAL PERCENT OF GROUP SELECTED NONE SOME A LOT CANNOT DO NONE SOME A LOT CANNOT DO NONE SOME A LOT CANNOT DO Male Born with it 0 16 67 40 0 3 29 20 N/A 19% 43% 50% 42% Disease 0 18 51 6 0 1 7 3 N/A 6% 14% 50% 15% UXO Accident 0 15 34 4 0 2 17 3 N/A 13% 50% 75% 42% War 0 8 51 3 0 1 9 0 N/A 13% 18% 0% 16% Old Age 0 5 36 6 0 2 13 4 N/A 40% 36% 67% 40% Other Accident 0 33 67 16 0 0 15 9 N/A 0% 22% 56% 21% N/A no difficulty in functioning 24 0 0 0 6 0 0 0 25% N/A N/A N/A 25% Total 24 95 306 75 6 9 90 39 25% 9% 29% 52% 29% Female Born with it 0 16 50 38 0 2 20 25 N/A 13% 40% 66% 45% Disease 0 9 51 20 0 0 8 7 N/A 0% 16% 35% 19% UXO Accident 0 4 5 0 0 0 1 0 N/A N/A 20% N/A 11% War 0 3 13 3 0 0 5 1 N/A 0% 38% 33% 32% Old Age 0 8 44 6 0 2 18 4 N/A 25% 41% 67% 41% 68 | EVALUATION FINAL REPORT USAID.GOV CAUSE OF IMPAIRMENT SCREENED APPLICANTS BY LEVEL OF DIFFICULTY SELECTED CBID BENEFICIARIES BY LEVEL OF DIFFICULTY PERCENT OF SCREENED APPLICANTS SELECTED BY LEVEL TOTAL PERCENT OF GROUP SELECTED NONE SOME A LOT CANNOT DO NONE SOME A LOT CANNOT DO NONE SOME A LOT CANNOT DO Other Accident 0 19 62 17 0 1 15 4 N/A 5% 24% 24% 20% N/A no difficulty in functioning 12 0 0 0 0 0 0 0 0% N/A N/A N/A 0% Total 12 59 225 84 0 5 67 41 0% 8% 30% 49% 30% Source: ET analysis of WEI data on screened and selected CBID beneficiaries 69 | EVALUATION FINAL REPORT USAID.GOV Table 9 shows that while males were much more likely to be survivors of UXOs than women, female UXO survivors were not nearly as likely to be selected as CBID beneficiaries (11 percent of female UXO survivors were selected versus 75 percent of male UXO survivors). Further, it shows that while the criteria listed in the Okard Cooperative Agreement said that female beneficiaries should be prioritized when possible, given the higher level of stigma associated with women with disabilities, women were only one percentage point more likely to be selected than male beneficiaries (with 30 percent of women selected and 29 percent of men). Outside of the analysis of CBID data conducted by the ET, the team also compiled feedback from key informants on the targeting of key groups. Key informants reported the following: • The selection criteria, according to CBID facilitators, led to the selection of a lot of elderly beneficiaries. Due to difficulties in accessing the CBID beneficiary data, the ET was unable to verify these reports. However, given that the elderly were one of the vulnerable groups identified in the cooperative agreement, this seems to be a positive unintended consequence of selection criteria. • Given low budgets and possible caseloads for ARMI and QLA, a lot of target beneficiaries are left out, according to CBID facilitators. WEI staff explain these findings by saying: “Even though it is a fact that the number of UXO survivors benefiting directly from USAID Okard funding is low compared to other beneficiaries (with health condition from other causes), this number could have been higher if the district chosen for piloting the CBID model in Savannakhet Province had been a district where a greater number of people with long-term disabilities as a result of UXO accidents are living. These districts are in the east of Savannakhet Province.” However, while the ET did find that the number of UXO survivors was low overall, Tables 8 and 9 above show that sometimes a low percentage of those UXO survivors that do exist and were screened were selected as beneficiaries. For instance, only one out of the five female survivors of UXO accidents who report having a lot of difficulty in at least one functional area were selected to receive services. Nonetheless, when asked whether they are concerned with the prioritization of UXO survivors for support under the Okard Activity, USAID said that this was not a major concern for them. Further, WEI staff report that these changes in the targeted population were done in the context, understanding, and with the agreement of USAID. However, USAID was not aware of these discussions, and the ET was unable to identify documentation of this agreement. This could be because the Activity Agreement Officer's Representative role shifted from one person to another during the course of the Activity. Nonetheless, there does seem to be at least some gap in communication between Okard Activity staff and USAID around which groups are currently targeted by the Activity. According to WEI staff, they did originally try to prioritize UXO survivors since: “The four sources of funding [for USAID Okard] are (I) Leahy War Victims Fund (LWVF); (II) Victims of Torture Fund (VOT) both from Bureau for Development, Democracy, and Innovation (DDI) USAID; (III) the Office of Weapons Removal and Abatement (WRA) in the U.S. State Department’s Bureau of Political-Military Affairs; and (IV) the Disability Program Fund in RDMA [Regional Development Mission for Asia] Bangkok. LWVF responds to the growing need for physical rehabilitation with activities that strengthen comprehensive, sustainable, quality physical 70 | EVALUATION FINAL REPORT USAID.GOV rehabilitation and assistive health technologies (such as wheelchairs, activities of daily living devices, and prosthetic and orthotic devices) in conflict-affected countries. VOT supports mental health and psychosocial support, and the Disability Program Fund RDMA supports disability￾inclusive development and programming. Finally, WRA primarily targets the assistance to victims of UXO accidents.” WEI suggests that to better target UXO survivors moving forward, additional districts selected for an expansion of the CBID Demonstration Model should be selected based on the population of UXO survivors in the district, noting that Xayphouthong District (XPT) did not have a high number of UXO survivors. WEI reports that HI was tasked with selecting the beneficiary district in SVK since they were leading CBID activities in that province. According to WEI staff, HI delegated that decision to their former partner (LDPA), which apparently selected XPT based on data on the prevalence of disability and ease of access given that the district is close to the town of Savannakhet. WEI said that “In retrospect, WEI considers that we might have more strongly urged the selection of a district more affected by UXO, but believed it was important to give ownership to HI in early phases of the project.” EQ5 CONCLUSIONS WEI, HI, QLA, and ARMI staff do not appear to have targeted CBID beneficiaries exactly as described in the Okard Activity’s Cooperative Agreement. As such, while Okard has been successful in reaching some of its targeted beneficiaries, including some victims of UXO and, according to CBID facilitators, a higher percentage of the elderly—listed as a highly vulnerable group—it appears to not have selected the full number of persons reporting functional difficulties stemming from UXO accidents for further needs assessment and case management in the first round of case management. Additionally, while not underserved, women do not appear to have been prioritized as suggested in the cooperative agreement. Further, since Okard did not collect details on ethnicity or sexual orientation during screening, these criteria could not be taken into account in the targeting of beneficiaries. EQ5 RECOMMENDATIONS Recommendation 17: WEI and HI should work with USAID/Laos and the GoL ahead of selecting beneficiaries for subsequent rounds of CBID services to ensure the Activity is targeting and prioritizing the groups USAID/Laos wishes to target. In these conversations, all parties should consider UXO survivors, women, different age groups, and any other priority vulnerable groups USAID identified. Together, both groups should discuss what selection criteria will be and what to do if there are too many or too few individuals who meet priority criteria as well as how to address people who are not disabled but have critical health conditions or other critical circumstances that fall outside of the agreed-upon selection criteria. The goal should be for the criteria and selection to be completely transparent. Recommendation 18: Should USAID plan a follow-on activity to Okard, they should require that the grantee selects districts and provinces for expansion of the CBID model based on the criteria USAID determines is most critical in the conversations conducted as a result of Recommendation 15. Further, they should require the grantee to collect, analyze, track, and regularly present data on all of the key target groups, including groups targeted based on ethnicity and any other targeted vulnerable groups to ensure targeting is on track. 71 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION 6 TO WHAT EXTENT HAS USAID OKARD BEEN EFFECTIVE IN ACHIEVING ITS EXPECTED RESULTS, PARTICULARLY IN TERMS OF GENDER EQUALITY AND OTHER INCLUSION? The USAID Okard 2021 GIDAP, developed by WEI, outlines priority issues and mechanisms for action to ensure that USAID Okard provides equal access for disadvantaged and diverse groups of persons with disabilities. The GIDAP is supported by the USAID Okard SBCC Strategy and Action Plan, which aims to increase knowledge and understanding of disability and rehabilitation and transform attitudes and practices that lead to social exclusion and improved health-seeking behaviors. Guiding evidence for the GIDAP derives from the USAID Lao PDR Gender Analysis on Disability report.25 While the GIDAP recognizes a comprehensive set of findings that affect persons with disabilities, it presents actions that are feasible to address as part of the USAID Okard Activity and Results Framework within the time and resources available to the Activity and disaggregates data for woman and children. The WEI Gender Coordinator outlines gender and inclusive development actions in the GIDAP as activities that: 1) relate to accessibility and usability of information and communication and activities that develop disaggregated indicators to measure and track the influence of sex, gender, and age; 2) health￾related rehabilitation services; 3) different types of mental health and psychological support services; and 4) vocational training and income generating employment. Within each general consideration, concrete actions and related activities are listed for implementation within the CBID Demonstration Model and the components of health, stakeholder engagement, and economic empowerment. The GIDAP includes a plan for measuring disaggregated and gender specific data on 13 of the 16 Activity￾wide indicators, with data presented from October 2018 to September 2020 shown in Table 10 below. 25 Holzaepfel E., Anderberg L, Feenstra M, Seibold K. 2018. USAID Lao PDR gender analysis on Disability Gender analysis final report, Social Impact. 72 | EVALUATION FINAL REPORT USAID.GOV TABLE 10: GIDAP INDICATORS INDICATOR # INDICATOR NAME TARGET PROGRESS (OCT/2018 – SEPT/2020) TOTAL TARGET FEMALE (40%) CHILD (15%) TOTAL PROGRESS FEMALE CHILD Standard Indicators ES.4.1 Number of vulnerable Persons (female/male child/adult) benefiting from USG-supported social services 29,324 11.730 (40%) 5,860 (30%) 3,981 1,910 (47.9%) 372 (9.3%) ES.4.2 Number of service providers (female/male) trained who serve vulnerable persons 3150 1,260 (40%) N/A 1,282* 587 (46.4%) N/A PM1 Number of targeted beneficiaries (female/male child/adult) with increased knowledge/skills and awareness 21,532 8,613 (40%) 4,306 (20%) 3,633 1,726 (41.4%) 327 (9%) PM4 Number of targeted vulnerable populations gaining new or better employment as a result of USG assistance 326 130 N/A Not yet reported Not yet reported Not yet reported Health Indicators HI-2 Percentage of people (female/male child/adult) receiving “people-centered care” from targeted health and rehabilitation facilities 70% 70% 60% Not yet reported Not yet reported Not yet reported HI-3 Number of people (female/male child/adult) receiving health and related rehabilitation services (including rehabilitation, assistive products, medical treatment for UXO survivors, and MHPSS services) 7,140 2,856 (40%) 2,856 (40%) 301 162 (53.5%) 45 (15%) HI-4 Number of people (female/male child/adult) who demonstrate increase in function or wellbeing 4,346 1,738 (40%) 1,738 (40%) Not yet reported Not yet reported Not yet reported 73 | EVALUATION FINAL REPORT USAID.GOV INDICATOR # INDICATOR NAME TARGET PROGRESS (OCT/2018 – SEPT/2020) TOTAL TARGET FEMALE (40%) CHILD (15%) TOTAL PROGRESS FEMALE CHILD Economic Empowerment Indicators EI-1 Number of persons with disabilities (male/female) who graduate from TVET centers 40 20 (20%) N/A Not yet reported Not yet reported NA EI-2 Number of persons with disabilities and their households (male/female child/adult) who receive an IGA or complete job readiness 652 260 (40%) 108 (30%) 47 22 (46%) 0 (0%) EI-3 Number of persons with disabilities and their household (female/male child/adult) with improved economic self-sufficiency 246* 98 (40%) 43 (40%) Not yet reported Not yet reported Not yet reported Stakeholder Engagement Indicators SI-2 Number of participants (male/female child/adult) involved in government led policy processes who self-identify as a person with a disability 161 81 (50%) 24 (15%) Not yet reported Not yet reported Not yet reported Gender-Specific Indicators HI-1 Number of service delivery systems with improved capacity to provide Women-Centered Care 23 N/A N/A N/A N/A Not yet reported GNDR-2 Percentage of female participants in USG assisted programs designed to produce increased access to productive economic resources 40% N/A N/A N/A N/A Not yet reported Source: WEI’s GIDAP Report, September 2020 74 | EVALUATION FINAL REPORT USAID.GOV The Year 4 workplan for the Okard Activity outlines activities for continued progress on gender and inclusive development as the Activity moves through its fourth year. Plans include a review of the recently completed Women-Centered Care (WCC) tool to determine strengths and areas needing improvement amongst health facilities to provide WCC. Other plans focus on continued efforts to ensure that key messaging and IEC materials are gender-sensitive and inclusive, a plan for coordinating with and providing recommendations to DHR to ensure priority is given to women in the selection of scholarship candidates, ensuring that female peer-to-peer counsellors are available in each CBID district, capacity development programs that will ensure that TVET lecturers are gender sensitive and gender inclusive, and continued activities to enhance USAID Okard, HI, and sub-recipient staff understanding of and application of gender equality and inclusive development. EQ6 FINDINGS Overall, the findings from this MTE indicate that where USAID Okard has started or implemented activities related to gender and inclusion, those activities positively reflect gender, and to a lesser extent, inclusion within the Activity. For example, amongst vulnerable persons who received USG-supported social services (ES.4.1) nearly half (48%) were female and 9 percent (n=372) were children. For this indicator, the goal is for 40 percent of targeted recipients to be female and 30 percent of targeted recipients to be children (or a household where children reside). Using this indicator as an example and considering only indicators where data were reported in the GIDAP, the Activity has more than met the proportional target goal for women but is falling behind for children (age disaggregation). While the GIDAP sets targets women and children, WEI reports that the Activity is cognizant that their ability to reach targets is partially dependent on the composition of the population. It is possible that the number of children with disabilities in the targeted villages is lower than expected, which would explain fewer identified children reported in the current GIDAP. It should be noted that the Y4 Q1 Progress Report and Y4 Q2 ITT provided updated indicator data, but these were not disaggregated by gender and age so are not possible to consider for this analysis of gender and inclusion. However, data from the Y4 Q1 Progress Report indicate that all indicators where data are presented in the 2021 GIDAP demonstrated increases for overall targets, often by as much as 100 percent (double). Notably, GIDAP age data is disaggregated only for children (0-17) and women (18+) with no disaggregation for young adults (school age boys and girls) or older adults (60+), which are age cohorts identified as at risk for additional vulnerabilities, as discussed under EQ5 above. While the available data suggests Okard has made good progress toward meeting targets for the proportional inclusion of females in various Okard indicator activities, progress toward ultimate targets for each activity is slow. Data in the 2021 GIDAP show that 39 percent (n=1,226) of targeted service providers were trained to serve vulnerable people, and nearly half of these were female (n=587; 46 percent). Results from other GIDAP indicators are not as close to meeting ultimate targets. Only 14 percent (n=3,981) of targeted beneficiaries had received USG-supported social services (ES.4.1), amongst whom 48 percent (n=1,910) were female and 9 percent (n=372) were children. Only 17 percent (n=3,633) of targeted beneficiaries demonstrated increased knowledge, skills, and awareness (PM1), amongst whom 41 percent (n=1,726) were female and 9 percent (n=327) were children. Moreover, as the project progressed through its fourth year, only 4 percent (n=301) of beneficiaries had received health and related rehabilitation services, assistive products, medical treatment for UXO survivors, or MHPSS services (HI￾3), amongst whom 54 percent (n=162) were female and 15 percent (n=45) were children. Finally, only 7 percent (n=47) of beneficiaries received an IGA or completed job readiness training (EI-2), amongst whom 75 | EVALUATION FINAL REPORT USAID.GOV 46 percent (n=22) were female. No child with a disability or households with a child with a disability received an IGA or completed job readiness training as reported in the 2021 GIDAP. For all other gender and inclusion indicators, no data were included in the 2021 GIDAP report. The slow progress toward gender and inclusion targets, as indicated in the 2021GIDAP, are explained by the significant barriers experienced in the first three years of Okard (described in detail under EQ2), including the delayed signing of the WEI MOU, the requirement for and further delayed MOU for HI, challenges signing cooperative agreements with sub-recipients, and changes to sub-recipients, as well as challenges encountered rolling out the economic empowerment activities due to USAID regulations and cooperative agreement limitations, and the COVID-19 pandemic. Amongst MTE key informants interviewed, nearly all reported that they felt the Okard Activity has been equally inclusive of all persons who are disabled, regardless of age, ethnicity, socio-economic status, or location (rural/urban). “We all receive support equally,” was a sentiment frequently expressed by key informants. Further, the composition of the CBID Demonstration Model teams demonstrates Okard’s commitment to inclusivity, with representation of women and persons with a disability on the QLA and ARMI teams. GoL officials and other health stakeholders all reported equal access for women to attend training opportunities. These individuals also felt that the Okard Activity had good processes for ensuring equal services are provided to women and other vulnerable individuals at the community level. One respondent, however, noted that Okard did not target people from very rural areas because the target villages are not very rural. In this way, Okard may have left out the most vulnerable people. Several other community respondents expressed concern over what they viewed as unfair distribution of services. These individuals noted that there were far more people who wanted or needed services than the Activity could provide. Even CBID facilitators noted this concern and explained that the CBID screening tool has an algorithm for prioritizing individuals who will be brought into case management. The limits for this algorithm are set to identify individuals with the most need (level of difficulty and economic hardship) and is limited by the number of individuals the Activity can support with the available funds and human resources. WEI reports that gender balance is well respected within the Okard Activity, but the Activity is not a “gender project.” The Activity, according to WEI, does not, “have a focus on gender, women, children, or specific health conditions,” but rather takes an “intersectional approach [that] considers multiple factors of vulnerability, including gender.” The Activity addresses priority needs of persons with disabilities, but there is no policy or strategy that specifies specific attention for women, although they are specified as equal. According to Okard Activity workplans, WEI is working on a component for WCC, as it is observed that women’s access to healthcare is constrained by mobility limitations in some cases (outside of the home), heavy domestic duties, and unpaid family labor.26 Similarly, WEI reports that their Gender and Inclusion Officer looks at the Activity with a very broad spectrum and looks at age across the lifespan, which includes children and older adults. While many GIDAP indicators have not yet been updated with data and progress toward targets is slow, the Okard Year 4 workplan sets out plans to move forward on gender and inclusion indicators. With the ramping up of EE activities within the CBID Demonstration Model, it is likely that many of the gender and inclusion indicators will also ramp up and move closer to reaching targets, although much work remains 26 GH Pro. “USAID Laos Health Strategy 2019-2023,” March 2019. Accessed at: https://www.usaid.gov/sites/default/files/documents/1861/USAID_Laos_Health_Strategy_2019-2023.pdf. 76 | EVALUATION FINAL REPORT USAID.GOV to be done. Similarly, scaling of health and other standard indicators may soon result in increased numbers of targeted beneficiaries included in these indicator tables. Indicator H-1 aims to build capacity in 23 service delivery systems (health facilities) for improved WCC. To date, no progress has been made toward achieving this target. The Year 4 workplan indicates that a WCC tool is developed and will be used to inform Okard on the strengths and needs amongst health facilities to provide WCC. The ability of Okard to make progress on this activity (WCC) within the timeframe of the Activity will be dependent on how streamlined the processes is for collecting data using the WCC tool and the capacity for disseminating results and efficiently engaging processes that would improve WCC in the 23 targeted health facilities. EQ6 CONCLUSIONS While progress is slow toward reaching overall targets for gender and inclusion within the Okard Activity, the activity has demonstrated success with regards to equal or proportional inclusion of women in all its activities for which indicator data are available. There is less evidence that other targets for inclusion are met, which is evidenced by proportional targets not met for children. Moreover, there is no data to demonstrate how the project is targeting different age groups, including young women and men, or people from different ethnic groups. While the project may not be a “gender project” and it may endeavor to be inclusive of individuals across the lifespan, there is value to understanding how the Activity affects men, women, boys, girls, and people of different ethnicities across all age groups. Finally, WEI has not yet made progress toward its targets for WCC, which is an important component of the gender and inclusion targets. EQ6 RECOMMENDATIONS Recommendation 19: Okard should consider analyzing and reporting more refined disaggregated data with regards to the age cohorts of beneficiaries and should also collect data and report on other vulnerable groups. Okard Activity MEL staff should include age￾disaggregated data in future quarterly and annual reports. While it is worthwhile to disaggregate data by “women,” “children,” and “men,” these disaggregates leave unknown many vulnerabilities. For example, older women and younger women may be more vulnerable than older or younger men and women. Moreover, girls may experience disability and its consequences differently than is the experience of boys. If possible, data should also be collected and reported for other vulnerability factors, including ethnicity. Recommendation 20: All USAID Okard partners should approach the gender and inclusive component of the Activity with more intention. The Activity can continue to be generally inclusive while at the same time intentionally aware that it is interested in providing services to particular vulnerable populations and ensuring that strategies and systems are in place to appropriately meet the needs of these vulnerable populations. Further, Okard should work in close collaboration with DHR to focus efforts on starting the WCC activity. There may be components of the WCC activity that can be lined up for implementation before the results from the WCC tool are disseminated. 77 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION 7 DOES USAID OKARD HAVE THE APPROPRIATE MANAGEMENT SYSTEM AND PARTNERSHIP STRUCTURES TO ACCOMPLISH ITS OBJECTIVES? An important contextualizing factor related to the management and partnership structure of Okard is the timeline by which Okard was able to establish its partnerships. While USAID awarded the Okard Activity to WEI in October 2017, the MOU between WEI (and HI at the time) and NCDE was not signed until May 2019, 18 months after the Activity began. As stated, the MOU between HI and NCDE was signed in October 2020, three years after Okard’s inception. Between October 2017 and the signing of the WEI MOU in May 2019, WEI was able to plan for future Okard Activities, including the baseline study, but the intervention could not start until after the MOU was signed and the baseline study completed. In April 2019, Okard signed sub-agreements with ARMI and QLA. However, HI was unable to supervise ARMI in the field until their MOU was signed in October 2020. During this time, field supervision of ARMI fell to WEI, whose MOU has provision for working in both SVK and XKH. Sub-recipient agreements were signed with CMR, DHR, and DMAS in late 2020, while sub-agreements with COPE and AfA were signed in April 2021, after work on this MTE began. Figure 3 illustrates the Okard timeline. Figure 3: Okard Activity Timeline EQ7 FINDINGS As illustrated in Figure 2 in the introduction, the partnership and management structure of the Okard Activity is complex. However, what Figure 2 does not illustrate is the complex environments in which the Activity is operating, namely the social and political context of Lao PDR as well as the bureaucratic structure of USAID, which imposes numerous limitations and regulations that Okard must adhere to. WEI, being the prime recipient of the Okard Activity funding, carries the responsibility for managing the partnership structure created by the Activity, while simultaneously needing to navigate the Lao context, USAID regulations, and ultimately holding responsibility for the results. Findings from this MTE reflect this complexity. Partnership Structure – In the USAID Okard Sub-Recipient Profile drafted by WEI for the MTE, WEI reports that it worked collaboratively with HI on the technical application for Okard funding and were meant to be equal partners. However, once the Activity was awarded, the legal structure required one organization to take the role as “prime” recipient while the other, legally, became a sub-recipient with the 78 | EVALUATION FINAL REPORT USAID.GOV ability to contract sub-sub recipients. Since the start of the Activity, WEI and HI have navigated this complex relationship. WEI understands that they are the prime recipient and responsible for the Activity, yet the narrative remains that the two organizations are equal partners, according to the Sub-Recipient Profile. HI has worked in the rehabilitation sector in Lao PDR since 1996 and has much institutional knowledge and experience to contribute to the Okard Activity. WEI also has a history of working the field of rehabilitation in Lao PDR, most importantly the successful implementation of the USAID-funded TEAM project in 2014-2017, described above. This imbalanced relationship whereby two partners want to be equal yet are not, is problematic. From the start of the Activity, WEI staff report that HI experienced challenges with staffing, most notably the key position of Senior Technical Advisor, which recruited a staff person who was both a technical advisor and a manager. HI filled this position numerous times with a person who was qualified to be a technical lead but could not handle the managerial responsibilities, according to WEI. The repeated resignations of the Senior Technical Advisor, which was key to the operation of the whole Activity, strained WEI according to WEI staff, who report they were ultimately responsible for the project results, and also frustrated HI who were required to fill positions that they understood were not tenable. Despite these challenges, WEI and HI both report that HI has had a full staff since June 2020, with a plan for mitigating turnover moving forward. Other data from this MTE further illustrate the unequal relationship between HI and WEI. For example, HI staff report that they provide reports and data to WEI but do not have input into reports that are submitted to USAID. On other occasions, HI reports it is not invited to events to showcase project results, which would reasonably be expected if HI were an equal partner. According to interview data from WEI and HI, the Okard Activity has also experienced numerous other partnership changes since the Activity began. During the approval process for the USAID Okard technical application, numerous sub-partners were pre-identified. Importantly, the LDPA was pre-identified to lead some of the activities under the stakeholder engagement component of the Activity. For more than a year, WEI and HI worked with LDPA to sign a cooperative agreement but the partnership was dropped due to management issues on the part of LDPA. Subsequently, AfA was identified as a strong organization with capacity to lead on stakeholder engagement, albeit with a much smaller portfolio of activities. Under LDPA, the stakeholder engagement component of the Activity consisted of 20 activities, which was reduced to just five activities for AfA due to the much smaller timeframe remaining for AfA to complete the work. AfA signed their sub-agreement in June 2021, with less than 1.5 years remaining in the Activity. COPE is another organization with strong ties to the rehabilitation sector in Lao PDR that was pre￾identified as a sub-recipient for USAID Okard during the technical application process, according to WEI. However, WEI reports that due to administrative issues around governance and financial management, Okard was unable to sign the cooperative agreement with COPE until April 2021. As Okard receives funding from USAID, they are bound to numerous regulations around management and governance structures that sub-partners must adhere to. As a result of Okard’s capacity building efforts and the installation of a new COPE leadership team in August 2020, COPE was able to address its weaknesses and become compliant with the USAID regulations for sub-partners. COPE is now able to contribute to the Okard Activity. The timeline illustrated in Figure 3, shows the long path that Okard has walked to get to its current partnership structure. Part of the delay was a result of slow GoL processes related to the signing of the MOU and the surprise news that HI would require its own MOU. Other delays were a result of partnerships that did not work out or took a long time to get to the point where approval was possible. 79 | EVALUATION FINAL REPORT USAID.GOV Moreover, the COVID-19 pandemic, at the time of the writing of this report, had caused two national lockdowns, which essentially stopped work on Okard. Funding Structure – Numerous KII respondents to this MTE, primarily from the government sector, expressed frustration with the funding structure of Okard. They felt that the GoL is not properly engaged in the partnership because of the in-kind funding structure that Okard is bound to by USAID regulations, which does not permit direct funding to government bodies. While the ET takes this finding into account, is also recognizes that Okard is working in a regulatory environment that they do not control. Some government respondents and beneficiaries also noted that different Okard partners were using different travel and per diem rates and that some beneficiaries had their transportation covered while others did not, which they felt was unfair. In some cases, government stakeholders reported being provided with travel allowances, while in other situations they chose to use their own personal vehicle, potentially because they did not want to use public transportation. Data from this MTE indicate that different partners and sub-partners working under the umbrella of Okard, of which WEI is the organization that is primarily responsible, might have interpreted the GoL-mandated Decree 4000, which enumerates appropriate travel and per diem rates, differently. Communications – Other concerns expressed by government partners related to poorly timed communication, whereby individuals are given little notice of meetings or training workshops. Other GoL officials noted that some meetings were scheduled during the harvest season, and that training venues and logistics were often not culturally appropriate. While WEI recognizes the issue of untimely communication, they explain that they do not control some aspects of communication because they are required to have communications approved and ultimately distributed by GoL agencies. These findings provide context to the challenging environment in which Okard operates. Management of Training Activities – Since September 2019, Okard has embarked on a rigorous training program with their sub-partners. Without exception, respondents to this MTE spoke about the multiple trainings they attended and the good learning that came from those trainings. ARMI, for example, reported that none of their team had previously worked with persons with disabilities but because of the training they felt very capable of communicating with families, helping beneficiaries access health care and economic empowerment activities, and supporting community activities that are changing how community members understand the rights of persons with disabilities. The training program for the CBID Demonstration Model is particularly notable because Okard clearly embarked on a program that would increase application of learned skills and improved knowledge and attitudes toward persons with disabilities amongst CBID staff. Over the course of several months, the Okard Activity pulled CBID staff into centralized weeks-long training with the intent to train them on a specific component of the CBID intervention, after which staff would return to the community to implement this component of the intervention. Once that component of the intervention was complete, the staff would return to the centralized location for training on the next component before returning to the community to complete the task. This training method is well thought out and has clearly helped CBID staff retain knowledge. Across stakeholder groups, all informants of this MTE spoke positively about this. Despite many delays, Okard has capitalized on their time to develop rigorous and effective training programs and has delivered these programs to their partners as they have come on board. AMELP – In reviewing Okard quarterly and annual reports, the ET struggled to interpret results against each of the indicators. In some cases, the ET found that some indicators were left out of the summary tables reported in the quarterly and annual reports. In others, the team found that findings from this evaluation did not directly map to indicators, not because of issues with data quality, but because the indicators often aggregate a lot of information, as described in the PIRS. For instance, indicators on 80 | EVALUATION FINAL REPORT USAID.GOV capacity building (e.g. ES.4-2: number of service providers trained and PMI: number of targeted beneficiaries with increased knowledge) aggregate capacity built across each of the three Activity components in the case of ES.4-2 and across TVET and rehabilitation medicine in the case of PMI. The same is true of indicators on service delivery systems, number of persons who receive IGAs or complete job-readiness training, and number of operational criteria achieved by MOH and NCDE-related committees. These are not disaggregated by activity that contributes to the results, which makes interpretation for program management purposes difficult. For instance, it would be impossible for USAID to know how many CBID beneficiaries had received IGA training and how many received IGA supports (since these beneficiaries are aggregated with those receiving job readiness training and subsequently becoming employed for at least a month under PM4), for tracking the success of CBID activities, based on these indicators. This explains why the ET found that the Activity has had more success in achieving outcomes than appears to be the case by a simple review of the ITT, which has high thresholds for counting results. Similarly, it will be impossible for USAID to determine which government organizations require additional support in achieving operational criteria moving forward, as all criteria are aggregated under SI￾3. Finally, while the DQA report notes that “All disaggregations are sufficiently collected and precise enough to measure the indicator,” the ET found that these disaggregations (e.g., by female, male, child, adult) are not reported in quarterly and annual reports, nor in the ITT, which limits USAID ability to regularly monitor the targeting of key groups. WEI staff report that USAID asked them to simplify disaggregation for reporting in the ITT. However, if this is true, USAID may want to reconsider this in light of the above findings and interest in tracking which Activity components are effective and which are not as well, as which groups, if any, may be underserved. EQ7 CONCLUSION While the Okard Activity has faced many challenges since its inception in October 2017, Activity staff have navigated these challenges and are currently well placed to accomplish many activity goals. Currently, both GoL MOUs are signed and all the sub-recipients, including GoL partners, are fully on board, each with a defined scope of work and budget. And, most USAID waivers (to allow for construction and distribution of restricted commodities) are now in place to ensure the smooth implementation of activities moving forward. With regards to management, Okard staff communication with key government stakeholders has not always met with the needs of those stakeholders, which may be affecting the success and sustainability of the activity results. Further, the provision of in-kind support only to the GoL rather than direct financial support and GoL Daily Subsistence Allowances (DSAs) provided by Okard have been additional points of contention between Okard Activity staff and GoL Officials. Finally, Okard Activity indicators and quarterly and annual reports are not set up to ensure Okard partners and USAID can track which activities are successful and which are not, to support adaptive management and consideration of Okard follow-on activities. Indicators and reports also do not allow both groups to track which groups are receiving services and which are underrepresented. EQ7 RECOMMENDATIONS Recommendation 21: Okard Activity communication staff should develop a plan to communicate details about Activity management, systems, and protocols to key Okard Activity partners and GoL stakeholders. The communication plan should include details on: 81 | EVALUATION FINAL REPORT USAID.GOV • The roles and responsibilities of WEI and HI. A frank discussion between WEI and HI is needed. It is in the best interest of the Activity for WEI and HI to agree on how to work together in a situation where HI understands that WEI needs to monitor activities because of its role as the prime recipient of the USAID funding while at the same time respecting HI’s autonomy and ability to manage this complex project. This activity should include an agreement regarding transparency in communication with external stakeholders and donors as well as an agreement regarding how each organization will be acknowledged in external publications. • Overall management and partnership structures as well as financial and communication protocols regarding Okard activities. A comprehensive communication plan targeted toward and shared with all partners may help resolve some conflict amongst stakeholders. The communication plan should reference Decree 4000 for consistent guidance on DSAs. Recommendation 22: Okard Activity MEL staff should work to specify additional disaggregates for indicators to ensure they are reported by activity and group. Further they should report these numbers for each indicator in each quarterly and annual report for ease of USAID tracking and Activity management. EVALUATION QUESTION 8 DO USAID OKARD PARTNERS HAVE THE RIGHT SYSTEMS AND CAPACITY TO FULFILL THEIR SCOPE OF WORK? WHAT SYSTEMS AND/OR CAPACITIES ARE CURRENTLY MISSING IN ORDER TO FULFILL THE SCOPE OF WORK? EQ8 FINDINGS Findings about partner capacity are listed by partner below. It is important to note that this EQ relies entirely on perceptions and reported capacity. Due to timeline constraints, SI was unable to conduct organizational capacity assessments for each Okard Activity partner. As such, the ET cannot make conclusive statements about whether partners have the skills and capacity to complete the work, but only whether they appear to. The partners in this chapter only include Okard direct sub-recipients (not in-kind sub-recipients). Though WEI consider their GoL counterparts as partners, they are in-kind sub-recipients, and given that a major component of the Activity is to increase the capacity of the GoL, the government’s systems, capacity, and challenges are discussed under EQ3 above. Other GoL capacity challenges are also presented under EQ2 above insofar as they have contributed to delays in approvals for Okard activities or have an effect on the Laos operating environment. HUMANITY AND INCLUSION (HI) HI Overview – Humanity & Inclusion (HI) has a direct sub-grant under WEI, and is the only sub-recipient with their own sub-recipient partners. According to HI staff, they are responsible for implementing the SE and EE components, and also contribute to the health component of the Okard Activity through their Health and Rehabilitation Specialist. HI oversees AfA and the efforts to build the capacity and organizational structure of DPOs. HI also oversees DMAS’ activities; according to HI and DMAS, HI has been working to build the capacity of DMAS 82 | EVALUATION FINAL REPORT USAID.GOV by providing training on disability policy and inclusion, administration, and other targeted training. According to the Sub-Recipients Profile in Brief, HI monitors and supports ARMI’s implementation of the CBID Demonstration Model in SVK Province and Xayphothong District. HI Systems – According to staff from USAID, WEI, and HI, and as documented in the USAID Okard MOU Timeline Summary produced by WEI, in August 2019 (nearly two years into the Activity), WEI and HI were informed that HI could not continue to work on the USAID Okard Activity without a specific MOU between HI and NCDE. WEI reports that the demand for the separate MOU caused HI to stop work (no travel, external meetings, or activities) and meant that many of the Activity’s interventions could not be implemented until the MOU was completed. Delays in the MOU signing between NCDE and HI continued to prevent HI from implementing many aspects of the Activity in Y3. According to the Sub￾Recipients Profile, HI’s MOU was finally signed in October 2020—three years and one month into the Activity period of performance. As described above, WEI and HI coordination and communication has been challenging. WEI staff said that the MOU split and delay meant WEI needed to manage HI activities for a little over a year while HI’s MOU was pending. WEI staff report that this overextended their staff, who also had to provide supervision for ARMI’s CBID activities in SVK in the period between the signing of WEI’s MOU in May 2019 and HI’s in October 2020. However, both organizations report the relationship has improved in the past six months as a result of new staff and improved coordination structures, such as the USAID Okard Executive Committee. HI Capacity – As described above, USAID and WEI report that up until June 2020, HI struggled to maintain a full staff, especially key personnel. WEI staff report that HI’s Senior Technical Advisor left a year and a half into the Activity, and a series of different people were brought in to fill the position, which resulted in periods of time when the position was unfilled. According to NCDE and DMAS, turnover also caused challenges with both organizations’ staff knowing with whom they should communicate and work, especially after HI’s MOU came through and activities transitioned back from WEI to HI. This added to existing challenges in working with NCDE on the SE component. Okard Activity staff report that NCDE lacks motivation and efficiency regarding their work on the SE component, and that HI must push NCDE in order to see progress. While the gaps and turnover in the various HI positions created a challenge for maintaining a cohesive and motivated HI team that could deliver timely and quality activities in the first three years of the Okard intervention, WEI and HI report that HI has maintained a full staff since June 2020, and also has implemented a staff retention plan to avoid similar challenges in the future. QUALITY OF LIFE ASSOCIATION (QLA) QLA Overview – QLA is an XKH-based local NPA that WEI helped establish in 2009 to support vulnerable populations including UXO survivors and persons with disabilities, according to the Sub￾Recipients Profile. QLA implements the CBID Demonstration Model in XKH, with significant technical support from WEI. The project target is 35 villages in the Kham District, XKH, according to the Sub￾Recipients Profile. QLA Systems and Capacity – QLA Staff report that they have sufficient personnel in the Kham District to accomplish CBID goals. They report that there are eight staff in total, including one team leader, one monitoring and evaluation (M&E) officer, and six facilitators. Four staff are responsible for facilitating 83 | EVALUATION FINAL REPORT USAID.GOV beneficiaries, and two staff are responsible for the SHG activity and IGA. Out of the eight staff, three are females and five are males, of which one is a UXO victim. QLA has a strong background in disability inclusion and is very capable at the provincial level, according to WEI staff, but they also report that its government diplomacy skills could be improved. This finding is corroborated by feedback from Provincial Government officials that they feel QLA has not adequately involved them in CBID home visits or data collection efforts (see EQ3, H3 for more details). WEI and HI have trained QLA staff in data collection, rehabilitation support, EE, and mental health, according to QLA staff. They reported that after their mental health training, they realized the great need for the provision of mental health services for persons with disabilities. As such, while they report they have sufficient CBID staff to take care of people experiencing mental health challenges, they said they would like to create a CBID team of persons with disabilities specifically trained on mental health needs. Currently, there are no QLA district-level staff who are mental health experts, but all staff have been given basic training on the subject. ASSOCIATION FOR RURAL MOBILIZATION AND IMPROVEMENT (ARMI) ARMI Overview –ARMI is an SVK-based NPA. ARMI implements the CBID Demonstration Model in SVK, with significant technical support from HI. The project target is 16 villages in the Xayphouthong District, SVK, according to the Y2 Annual Progress Report. ARMI Systems – ARMI appears to be well-prepared to continue their work. Okard staff report that ARMI has strong management protocols and government relations skills. This is supported by GoL officials who report a close relationship with ARMI and close coordination regarding Activity implementation. Additionally, ARMI reports that it received a certificate of appreciation from the Ministry of Home Affairs in recognition of their work. Staff said that this is the first time the Ministry has given such a certificate in the past ten years. ARMI Capacity – ARMI reports that their staff are inclusive—especially their seven technical staff, which includes four women and one person with a disability. While the staff have no background in disability inclusion, ARMI staff report that Okard has significantly increased their capacity. Staff report increased knowledge regarding care for persons with disabilities, providing support and advice to their caregivers, communicating compassionately, being a good facilitator, working in the community, and collecting data for the CBID Demonstration Model. ARMI and HI report that ARMI’s staffing model is sufficient to achieve its objectives and successfully implement the CBID Demonstration Model in the 16 target villages. According to ARMI staff, the number of target beneficiaries totals more than 400; thus, staff have divided the caseloads across different periods. In the first period, ARMI reports having worked with 90 families, with each staff member having successfully taken on caseloads of close to 20 families or persons each. The remaining 300+ cases will be covered following this first round. DISABILITY MAINSTREAMING ADVISORY SERVICE CENTER (DMAS) DMAS Overview – DMAS was registered in August 2019 as a business consulting firm with a social enterprise business model. It is run by a group of persons with disabilities, and WEI reports that DMAS has a vision for a community where all people, regardless of their functional ability, realize their optimal potential by accessing decent and fulfilling employment and for a business community committed and equipped to non-discriminatory access to employment for all. 84 | EVALUATION FINAL REPORT USAID.GOV According to the Sub-Recipients Profile, DMAS offers advisory services on disability inclusion and accessibility, as well as providing technical expertise and coaching to private and public businesses (factories, companies, coffee shops, restaurants, and shops), institutions (government, NPAs, INGOs), and public and private service providers (healthcare service providers, schools, financial institutions, TVETs). Okard Activity staff and quarterly and annual Activity reports explain that DMAS is responsible for many of the EE activities related to TVET and integration of persons with disabilities into existing employment opportunities. DMAS Systems – According to DMAS staff, the Center has been operating since October 2020 (as of the signing of HI’s MOU) as a result of Okard funding support, and is structured to enhance disability inclusion in Lao and diversify the organizations cooperating with the government sector. According to the USAID Okard cooperative agreement, DMAS was to register as a social enterprise; however, DMAS staff report that when they went to register with the GoL, there was no such option. Instead, DMAS is registered as a sole proprietorship, which requires them to earn an income to operate. According to DMAS staff, the Ministry of Industry and Commerce issued the registration certificate for DMAS, and the MLSW issued a specialized license for working with persons with disabilities, valid for two years and renewable. According to DMAS staff, COVID-19 and the delayed MOUs have ultimately delayed DMAS’ work. DMAS staff report also experiencing challenges due to the transition from reporting to WEI to reporting to HI, following HI’s MOU finalization. DMAS Capacity – DMAS reported that they currently have seven employees, all of whom have disabilities. Although they can manage current operations with existing staff, they would benefit from hiring a marketing person to help ensure sustainability by marketing their services. Staff report that additional training on how to be a good service provider would increase the impact of their work and improve their overall operation. In the Y4 Q1 report, Okard Activity staff report that they completed a Capacity Development Action Plan for DMAS, which they will begin to implement in the coming months. DMAS has proven successful at helping persons with disabilities secure employment, reporting having found employment for 17-19 out of 49 people trained on CV writing and interviewing. DMAS also reports having established an advisory group to provide employment suggestions for participants and support them in searching for jobs. Further, as discussed above, several outside organizations have reached out to hire DMAS to support accessibility audits of their buildings. DMAS has provided an accessibility audit for NGOs such as Oxfam, worked with SNV Netherlands Development Organization on disability inclusion in Savannakhet, and is currently working on a joint disability inclusion and accessibility plan for Plan International, Childfund, and Lao Rugby. Private sector providers are interested in collaborating with DMAS to improve accessibility as well. DMAS has discussed disability inclusiveness with the Boualapha Company, Lao World, and Itec, all of whom are willing to build parking spaces for persons with disabilities on their properties. DMAS reports that they are struggling to maintain sustainability, due to both funding issues (DMAS’ staff is entirely funded by the Okard Activity currently) and the fact that the period of performance is too short to allow them to build additional capacity internally for their staff and externally through improved marketing. However, DMAS’ sustainability plan was to save program income from the provision of paid accessibility audits and DITs. Nonetheless, the ET found that ADS 303.3.10.4 says that “If program income 85 | EVALUATION FINAL REPORT USAID.GOV is anticipated, the AO determines the most appropriate use and must state in the award how it will be used.” The Okard Cooperative Agreement says that “The Recipient must account for Program Income in accordance with 2 CFR 200.307. Program Income earned under this award will be added to the project.” 2 CFR 200 defines program income as gross income earned by a grantee that is directly generated by an activity under an award or earned as a result of the award during the period of performance. “Adding the income to the award” means that “program income may be added to the award, increasing the total amount of the award. Program income must be used for the purposes and under the conditions of the award” (read: within the period of performance). As such, it appears that DMAS needs to identify other possible income to support their core operations after the Okard Activity ends. HI has suggested that DMAS should explore grant options and other side￾services and/or products besides the core service on inclusion. DMAS reports that Okard is supporting DMAS in the process of developing a sustainability plan through the provision of a consultant to support DMAS in creating a business plan for 2022-2027. ASSOCIATION FOR AUTISM (AFA) AfA Overview – AfA represents the interests of persons with autism spectrum disorder in Lao PDR by promoting their rights, improving their access to appropriate services, and providing them and their families with education, therapy, and training to better equip them to live full and productive lives, according to AfA’s website. WEI and HI report that AfA is well respected by other DPOs. According to HI and Okard Activity documents, AfA received a sub-grant from HI, signed in June 2021, to lead SE activities related to DPO strengthening and establishing a network by which the GoL can engage with DPOs. Once HI determined that LDPA did not meet capacity requirements to received a sub-grant, they brought AfA on as a new partner to implement some of the SE activities, though (as noted above) not the full scope of original LDPA activities. AfA Systems and Capacity – AfA has not yet begun implementing activities, and MTE methods have not included a capacity assessment of AfA. Okard staff report confidence in AfA’s capabilities, and feel AfA is more prepared to receive Okard funds than LDPA was. WEI and HI similarly feel that AfA is a strong DPO prepared to aid in building capacity for DPOs, take over leadership, and improve management of the informal network of DPOs developed to advise the GoL and other key stakeholders on policies and inclusion efforts. The scope of AfA’s activities was reduced from the 18 activities assigned to LDPA to five, according to Okard staff, due to the challenging context in Lao and bureaucratic constraints. The late approval of the MOU and NCDE’s reluctance to formalize the network of DPOs that will work with them are two of the primary reasons for this reduction. COOPERATIVE ORTHOTIC AND PROSTHETIC ENTERPRISE (COPE) COPE Overview – According to the Sub-Recipient Profile, COPE is an independent organization working within a partnership agreement framework with CMR and with permission for the MOFA to operate and receive funding from international and local donors. WEI staff report that COPE signed a sub-recipient agreement with WEI in April 2021. As mentioned above, under this sub-grant, COPE will work to build the capacity of the rehabilitation centers and hospitals to provide appropriate assistive devices to persons with disabilities, coordinate efforts of other donors and providers working to provide assistive devices in Lao, and directly procure some assistive devices, according to the Okard Sub￾Recipients Profile drafted by WEI for the MTE team. 86 | EVALUATION FINAL REPORT USAID.GOV COPE Systems and Capacity – While COPE’s governance structure and management practices did not meet WEI and USAID’s standards at the beginning of the period of performance, WEI reports COPE’s willingness to change and grow and its increased capacity following the installation of a new leadership team in August 2020. As a result, WEI was able to work with COPE to establish the necessary systems and capacity to ensure they could receive a sub-grant under the Okard Activity. Further, having worked with COPE for technical purposes in the past, Okard staff and GoL officials report that COPE is well prepared to take on work related to providing assistive devices, such as prosthetics, crutches, and wheelchairs. COPE staff report that their team has experience working under U.S. rules and regulations, and has been providing assistive devices to persons with disabilities for more than 20 years, including support to the GoL for such services in the past. COPE staff reported implementation success in 2020, having served between 1,300 and 1,500 people nationwide for non-Okard projects. According to COPE staff, they have used mobile clinics to provide assistive devices to a greater number of persons with disabilities, including women and people in rural areas. The mobile clinics have encouraged more women with disabilities to use these services, as many of the doctors providing treatments are female and there are more private spaces. In 2020, COPE reports having served 1,048 people, in spite of COVID-19, almost half of whom were women. COPE has existing training courses that it reports it will use to increase the capacity of the government rehabilitation personnel to effectively provide assistive devices to persons with disabilities. These courses will include training on serving people with neurological disabilities, such as speech and physical therapy, training on the safe use of wheelchairs, and training on prosthetic limbs and assistive devices, in addition to others. Additionally, COPE staff said Okard provided COPE with training on administration and financial management to fill any gaps they may have in providing assistive devices or improving accessibility. COPE staff report that the organization is understaffed, with one in ten positions currently unfilled. This means that all staff are having to fundraise and conduct monitoring and evaluation, in addition to their other roles and responsibilities. Further, USAID reports that it would like to see COPE become a fully registered CSO, as this would help to ensure its sustainability and ability to fundraise. EQ8 CONCLUSIONS While several partner challenges delayed finalization of the partnership framework until very recently, all partners have now signed their sub-agreements, with the latest one signed in June 2021. The limited evidence the ET was able to find suggest local Activity sub-recipients (namely, ARMI, QLA, DMAS, COPE, and AfA) have the systems and capacity to complete their activities under Okard. However, the following additional systems and capacity (“missing systems and capacity”) would help to ensure both smoother operations moving forward and the sustainability of activity interventions. ● QLA needs support to improve its communication strategy to improve relations with GoL counterparts and ensure capacity and funding exists to support the sustainability of activities after Okard ends. ● In order to become a sustainable part of the EE system for persons with disabilities in Laos, DMAS needs urgent support to identify and implement a business and sustainability plan. EQ8 RECOMMENDATIONS Recommendation 23: HI should prioritize its work with DMAS to ensure the sustainability of the organization after the Activity ends. Specifically, HI should continue its work with DMAS to 87 | EVALUATION FINAL REPORT USAID.GOV develop a strategic plan for the organization and should ensure the plan includes actions focused on identifying funding support (which could come in the form of income) for operations that does not come in until Okard ends. It is important that any funding support identified cannot be considered program income, which means that it must come in after the end of the Okard Activity. Perhaps HI and DMAS can line up inclusivity audits to begin immediately after the end of Okard. Recommendation 24: WEI should provide QLA with training on communications and diplomacy skills to help improve their government relations moving forward. Specifically, WEI should work with QLA staff to better understand their approach to communication and coordination with the GoL and to identify potential areas for improvement. This may include extra focus on communicating more broadly with Provincial Government officials and perhaps considering direct communication with more stakeholders as opposed to relying on third parties to relay information on behalf of the Activity. Recommendation 25: WEI should identify what barriers are preventing COPE from registering as a CSO in Laos and should work with COPE staff to address those barriers ahead of Okard Activity end. 88 | EVALUATION FINAL REPORT USAID.GOV ANNEX I: EVALUATION METHODS EVALUATION STARTUP AND PLANNING On February 24, 2021, SI hosted a kickoff meeting with USAID/Lao PDR and WEI to discuss stakeholder priorities and clarify the scope of work, the EQs, MTE methodology, and timeline. USAID and SI then discussed questions regarding the background documents and technical approach. Following the meeting, SI organized a remote team planning meeting to review the timeline, discuss roles and responsibilities, lines of communication, and Evaluation, Quality, Utilization, and Impact (EQUI®) Quality Assurance (QA) practices. This process ensured the MTE met stakeholder needs and was realistic and feasible. On March 24th, 2021, SI hosted an in-brief meeting with USAID/Lao PDR before starting field-based data collection, which provided SI and USAID/Lao PDR with an opportunity to discuss any updates to the purpose, intended use, expectations, timeline, gender and social inclusion considerations, and fieldwork logistics for the evaluation. DESIGN REPORT AND DATA COLLECTION TOOLS The ET finalized the evaluation design matrix describing data sources, data collection methods, and data analysis methods to answer each EQ. SI’s Project Director (PD) and the initial team lead (TL) reviewed the deliverable against the EQUI® QA Checklist for Work Plans and Inception Reports to ensure quality. SI’s Gender Equality and Social Inclusion (GESI) Advisor participated in the development of the Work Plan and Evaluation Design Report and data collection instruments to ensure that the ET integrated gender and social inclusion considerations in the evaluation design. The ET submitted the evaluation design and instruments to SI’s in-house Institutional Review Board (IRB) for a full IRB ethical review, including a review of, informed consent, strategies for data collection with vulnerable populations, and plans for data management including confidential storage of data. The IRB process included considerations for data collection during the COVID-19 pandemic. Originally, the ET developed 19 qualitative data collection tools for this evaluation. These tools were developed for KIIs and focus group discussions (FGDs). As a result of COVID-19 lockdowns in Lao PDR, which were announced on April 21st, 2021, all data collection tools were revised for remote KII data collection, and some were combined. As a result, 22 data collection tools were finalized for data collection for this MTE. Originally, the ET planned to conduct KIIs in VTE and a combination of FGDs and KIIs at the provincial, district, and community level. As FGDs were no longer possible due to the COVID-19 lockdowns, FGDs at the district and community level were scaled down to KIIs and group interviews, which required the development of revised and additional KII tools. Moreover, after completion of group interviews with WEI, HI, and USAID, the ET decided to add one additional KII focused on mental health with an Okard Activity contractor. DATA COLLECTION The ET collection data for this MTE between March 30th and May 20th, 2021. Data collection started with in-depth group interviews with WEI (3 group interviews), HI (2 group interviews) and USAID (1 group interview). These interviews combined remote and face-to-face data collection. Where possible, a field￾based ET member participated in the group interview face-to-face while the TL participated remotely. The USAID group interview was entirely remote (field-based ET members and TL joined remotely) due to security issues. The third WEI group interview was fully remote (all participants joined remotely) due to COVID-19 lockdowns. These group interviews provided important contextual background for the ET as well as data to respond to the EQs. 89 | EVALUATION FINAL REPORT USAID.GOV The group interviews with WEI, HI, and USAID were led remotely by the initial TL. These interviews were conducted in English. The initial TL also attend the KIIs with ARMI and CMR remotely, which were led by the local ET members and conducted in Lao language with English translation provided to the TL. All other KIIs were led by the local ET team members, who were accompanied by a note taker. Notes from KIIs conducted in Lao language were transcribed into Lao Language and then translated into English. All group interviews and KIIs were audio recorded. During group interviews and KIIs, the note taker took notes, which were later checked and filled in with the audio recordings. A data collection pause was planned for April 8-20 due to the Lao New year but was extended to April 30th due to the need for the ET to revise data collection plans and tools due to the COVID-19 lockdown. In the design report, the ET planned to collect data from 21 KIIs and 22 FGDs, which were anticipated to provide input from approximately 218 individuals representing a wide spectrum of perspectives. Data from these KIIs and FGDs were to inform two case studies (illustrating one “story of change” for each of SVK and XKH). The COVID-19 lockdowns, which made it impossible for local ET team members to travel to SVK and XKH and conduct FGDs, required that the ET scale back data collection considerably. Most notably, the ET necessarily changed all FGDs to KIIs, which resulted in fewer respondents, particularly at the community level. In the end, data collection for the MTE included 45 KIIs, which captured perceptions from 70 individuals, 36 of whom were male and 34 of whom were female (See Table 11 below). These KIIs were conducted via Zoom, WhatsApp, and mobile phone. Due to far fewer community respondents, the ET, in collaboration with USAID, decided to scale back to just one case study encompassing a story of change reflective of the community level KIIs conducted for this MTE. This case study is presented in this final report. Data collection in VTE remained largely unchanged other than the remote format. Minor changes to VTE data collection included dropping one KII with AfA as the respondent became unavailable due to the COVID-19 lockdowns. As well, one additional mental health specialist KII was added to the VTE KIIs. Other KIIs in VTE included TVET, ARMI, COPE, DMAS, NCDE, DHR, and CMR. At the provincial level, one remote KII was held with three individuals representing the Department of Labor and Social Welfare (DLSW), Department Of Health and the Provincial Rehabilitation Center from Xiengkhuang Province; these individuals were all able to join by WhatsApp. Two KIIs were conducted in Savanakhet province, one with the DLSW and one with the Provincial Centre of Rehabilitation. KIIs at the district level also remained largely unchanged except for the change from FGD to KII format, which resulted in fewer respondents. In Kham district, one remote KII was held with the Kham Administration Office, one with the Kham DLSW, one with the Kham Agricultural office, and one with the Kham Health Office. The QLA head of office was also interviewed remotely from Kham district. A combined WhatsApp KII was held with the QLA CBID team lead and two facilitators from Kham district. Finally, one doctor from Kham district was interviewed to provide perspective on the roll out of mental health supports as part of the CBID activity. Similarly, in Xayphouthoung District, one interview was conducted with the Xayphouthoung Administration Office, one with the Xayphouthong Vice Head of Office, one with the Xayphouthong Health Office, and one with the Xayphouthong Agricultural Office. In Xayphouthong District, a combined KII using WhatsApp was held with the CBID team lead along with two facilitators. To provide a different perspective, one nurse from Xayphouthong District was interviewed for perspective related to mental health supports. The main changes to data collection at the district level was the change from FGD to KII data collection for mental health workers, which reduced the number of respondents from 16 to two and the combined KIIs with CBID team leaders and two facilitators in each district. In the original plan, the ET planned to conduct one KII with the CBID team leader followed by a FGD with eight facilitators, which would have resulted in 18 participants compared to the six who the ET was able to connect with. 90 | EVALUATION FINAL REPORT USAID.GOV In each district, the ET conducted 8 KIIs with beneficiaries. These beneficiaries included two women with a disability, two men with a disability, two caregivers of a person with a disability, one community member (non-disabled), and one village leader. In each district, the ET planned to connect with one younger woman/man (18-24 years) and one older woman/man (25+ years). Similarly, the ET planned to connect with one caregiver in each district who cared for child (0-18 years) and one caregiver who cared for an adult (18+years) with a disability. Community members were selected for a KII from a database (provided by WEI) of individuals who had attended an SBCC seminar but did not have a disability and did not care for a person with a disability (see sampling). The change to the KII data collection format resulted in far fewer respondents at the community level. Originally, the ET had planned to conduct only FGDs with individuals at the community level and with more diverse groups. Moreover, there was a plan to bring individuals with a range of disabilities and from outlying and remote areas. This all necessarily changed because of the COVID-19 lockdowns. Plans to gather data from individuals who were involved in self help groups were dropped to reduce the number of KIIs for this MTE and also because the self-help group activity was only recently started within the CBID activity. In June 2021, one additional round of data collection was completed to supplement data related to EQ1 (To what extent was the original design of the Okard Activity relevant to its goals of improving systems and access to healthcare and economic empowerment to persons with disabilities?) In this round of data collection, 10 short KIIs were conducted with individuals from WEI, HI, DHR, CMR, NCDE, TVET, ARMI, QLA, DMAS, the Xiengkhuang Province Rehabilitation Center, and the Savanakhet Provincial DLSW. SAMPLING All respondents contacted for VTE and provincial KIIs were purposively selected based on their engagement with the Okard Activity. At the district level, most respondents were purposively selected based on their engagement with the Okard Activity except for mental health workers. Individuals selected for mental health worker KIIs were selected based on their attendance at the Okard mental health training, their profession (doctor/nurse), and their sex. A list of individuals from each district who had attended the training was provided to the ET by WEI. In Xayphouthong district, one female nurse was selected for a mental health KII and in Kham district one male doctor was selected from the list provided. According to the original design, prior to the COVID-19 lockdowns, the ET had envisioned conducting two FGDs with mental health professionals, one in each district. The reduction from two FGDs (16 individuals) to two KIIs (two individuals) substantially reduced the ETs ability to gather data from a wide spectrum of viewpoints and experiences related to mental health. Potential respondents for community level KIIs were selected from a database (provided by WEI) of individuals who were registered as beneficiaries with the CBID activity in each district. The TL began the random selection by creating a clean spreadsheet that contained all the potential respondents (i.e., women with disabilities). A random number was then generated using the random number generator function in Excel. The TL then reviewed the individual randomly selected. In the case of women and men with disabilities, the TL checked that the individual had 1) received services from the CBID activity and 2) did not have a disability that would make it impossible to talk to the person on the phone (i.e., hearing disability, intellectual disability) and 3) was either 18-24 years or more than 25 years old. The TL continued the random selection until two appropriate individuals of each sex (men/women) age 18-24 years and 25+ years were selected. With regards to caregivers, the TL randomly selected one individual who cared for a child (<18 years) and another who cared for an adult (18+ years). In Kham district two male caregivers were selected while in Xayphouthoung district two female caregivers were selected. Community members (individuals who are not disabled) were selected from a database of individuals who had attended SBCC training offered by the CBID activity. In Kham district one female community member was selected while in Xayphouthong district one male community member was selected. Village leaders, both of whom were male, were selected from a list of village leaders provided by WEI. For each respondent type, two potential 91 | EVALUATION FINAL REPORT USAID.GOV alternative respondents were randomly selected to account for the possibility that the ET would be unable to connect with the first selected participant. At the district level, some deviations from the primarily selected sample occurred. In Kham district, one man with a disability was unable to speak on the phone. The ET member spoke to his caregiver. In Xayphouthung district, one caretaker who was identified as a caregiver for a child turned out to be a caretaker for an adult. The ET member continued with the interview. In both districts, the ET necessarily had to select from the alternative list on multiple occasions because they were unable to connect with the primarily selected community-level individual. In both districts, the TL was required to pull additional alternative potential respondents because none of the selected or alternative community member respondents were reachable. In Kham district this resulted in a KII with a male caregiver because no female caregiver could be contacted. In some cases, the ET was unable to connect with selected individuals because they did not have a phone, or the connection was bad. On three occasions, a potential respondent was able to use a village heads phone to conduct the KII. CONSENT AND COMPENSATION Informed verbal consent was obtained from every respondent prior to starting a group interview (prior to COVID-19 lockdowns) or KII. Prior to starting the group interview or KII the ET read the informed consent script to potential respondents and asked for their verbal consent. Due to their participation, village respondents were each provided 35,000 LAK (equivalent to daily wage). For the cases where the respondent borrowed the village head’s phone to participate in the KII, the village head was compensated with 105,000 LAK to account for the additional support. All other KIIs were initiated by the ET so no cost to respondents was incurred. All data collected for this MTE were stored on SIs password protected SharePoint drive. DATA ANALYSIS English group interview and KII transcripts were uploaded to Dedoose 8.3.47b for qualitative data analysis. A code book was developed based on the EQs for this MTE and indicators for the Okard Activity, upon which data collection tools were developed. A verification process was conducted whereby two members of the ET and one SI staff coded the same transcript. A review session was held to check for inter-rater reliability, clarify code definitions, and add new codes. Data were then analyzed using thematic data analysis strategies. Where necessary, new codes were added to the code book. Data were coded for themes within each EQ and then re-read for relationships within and across EQs. Finally, data were reduced to key findings and sub-findings. Recommendations and the case study emerged from these findings. TABLE 11: FINAL KII AND GROUP INTERVIEW (GI) SAMPLE FOR THE MTE Informant Sex Type of Male Female Disability Interview Vientiane City USAID Staff 2 4 GI WEI staff 1 8 GI HI Staff 6 3 GI TVET Department Officials 1 KII ARMI Staff 1 KII COPE Staff 1 KII 92 | EVALUATION FINAL REPORT USAID.GOV Informant Sex Type of Male Female Disability Interview DMAS Staff 0 0 1 KII NCDE Officials 1 KII DHR Officials 1 KII CMR Officials 1 KII Xiengkhouang Province Department of Labor and Social Welfare 1 KII Department of Health 1 KII Provincial Center of Rehabilitation 1 KII Kham District Admin Office 1 KII Labor and Social Welfare Office 1 KII QLA CBID + 2 facilitators 2 1 KII QLA head office 1 KII Health office 1 KII Health workers, mental health 1 KII Kham Village Women with disabilities 2 2 KII Men with disabilities 2 2 KII Caregivers of a person with a disability 2 KII Community member 1 KII Village Leader 1 KII Savannakhet Province Department of Labor and Social Welfare 1 KII Provincial center of Rehabilitation 1 1 KII Xaphouthoung District Admin office 1 KII Vice head of office 2 KII Health office 2 KII Agricultural office 1 KII ARMI CBID + 2 facilitators 1 2 KII Health workers, mental health 1 KII Xayphouthoung Village Women with disabilities 2 2 KII Men with disabilities 2 2 KII Caregivers of a person with a disability 2 KII Community member 1 KII Village Leader 1 KII Total 36 34 93 | EVALUATION FINAL REPORT USAID.GOV ANNEX 2: EVALUATION DESIGN MATRIX EVALUATION MATRIX – USAID OKARD MTE # Evaluation Question Data Collection Tools Data Source and Quantity Location of Data Collection Data Disaggregation Data Analysis Objective 1 1 To what extent have the USAID Okard strategy and interventions been relevant to the goals it intends to reach? KIIs FGDs USAID (1 KII) WEI (2KIIs) HI (1KII) Government Officials (4KIIs) Sub-recipients (5KIIs) CBID Facilitators (2 FGDs) Village leaders (4KIIs) Healthcare workers – Mental Health (2 FGDs) Healthcare workers – Rehabilitation (2 FGDs) Community members (16 FGDs) XKH SVK VTE Stakeholder type Sex Age Location Cause of impairment, where appropriate FCR matrix (updated weekly) Data coding Qualitative thematic data analysis strategies 2 Are the context and critical assumptions still valid? What have been key changes, if any, that affect the anticipated achievements positively and/or negatively? Desk review KIIs FGDs USAID (1 KIIs) WEI (2 KIIs) HI (KII) Government Officials (4 KIIs) Sub-recipients (5 KIIs) CBID Facilitators (2 FGDs) Village leaders (4KIIs) Healthcare workers – Mental Health (2 FGDs) Healthcare workers – Rehabilitation (2 FGDs) XKH SVK VTE Stakeholder type Sex Age Location Cause of impairment, where appropriate FCR matrix (updated weekly) Data coding Qualitative thematic data analysis strategies 94 | EVALUATION FINAL REPORT USAID.GOV EVALUATION MATRIX – USAID OKARD MTE # Evaluation Question Data Collection Tools Data Source and Quantity Location of Data Collection Data Disaggregation Data Analysis Community members (16 FGDs) 3 What components of the Activity have been successfully and/or unsuccessfully implemented according to the approach? Why? KIIs FGDs DHR (1 KII) NCDE (1 KII) CMR (1 KII) Department of Education and Sport, TVET division (1KII) Sub-recipients (5 KIIs) CBID Facilitators (2 KIIs) Village leaders (4 KIIs) Healthcare workers – Mental Health (2 FGDs) Healthcare workers – Rehabilitation (2 FGDs) Community members (16 FGDs) Operational data (TBD) XKH SVK VTE Stakeholder type Sex Age Location Cause of impairment, where appropriate FCR matrix (updated weekly) Data coding Qualitative thematic data analysis strategies Quantitative data, if available, will be analyzed for basic statistics using SPSS 4 To what extent has USAID Okard been effective in achieving its expected results, particularly in terms of building/strengthening the systems as described in the design document (system￾centered) KIIs FGDs DHR (1 KII) NCDE (1 KII) CMR (1 KII) Department of Education and Sport, TVET division (1KII) Sub-recipients (5KIIs) Operational data (TBD) VTE Stakeholder type FCR matrix (updated weekly) Data coding Qualitative thematic data analysis strategies Quantitative data, if available, will 95 | EVALUATION FINAL REPORT USAID.GOV EVALUATION MATRIX – USAID OKARD MTE # Evaluation Question Data Collection Tools Data Source and Quantity Location of Data Collection Data Disaggregation Data Analysis be analyzed for basic statistics using SPSS 5 To what extent has USAID Okard been effective in achieving its expected results, particularly in terms of reaching target beneficiaries. Which groups are underserved, if any, among all targeted groups, and why? (people-centered); KIIs FGDs DHR (1 KII) NCDE (1 KII) CMR (1 KII) Department of Education and Sport, TVET division (1KII) Sub-recipients (5 KIIs) CBID Facilitators (2 KIIs) Village leaders (4 KIIs) Healthcare workers – Mental Health (2 FGDs) Healthcare workers – Rehabilitation (2 FGDs) Community members (16 FGDs) Operational data (TBD) XKH SVK VTE Sex Age Location Cause of impairment, where appropriate FCR matrix (updated weekly) Data coding Qualitative thematic data analysis strategies Quantitative data, if available, will be analysed for basic statistics using SPSS 96 | EVALUATION FINAL REPORT USAID.GOV EVALUATION MATRIX – USAID OKARD MTE # Evaluation Question Data Collection Tools Data Source and Quantity Location of Data Collection Data Disaggregation Data Analysis 6 To what extent has USAID Okard been effective in achieving its expected results, particularly in terms of gender equality and other inclusion? KIIs FGDs DHR (1 KII) NCDE (1 KII) CMR (1 KII) Department of Education and Sport, TVET division (1KII) Sub-recipients (5 KIIs) CBID Facilitators (2 KIIs) Village leaders (4 KIIs) Healthcare workers – Mental Health (2 FGDs) Healthcare workers – Rehabilitation (2 FGDs) Community members (16 FGDs XKH SVK VTE Sex Age Location Cause of impairment, where appropriate FCR matrix (updated weekly) Data coding Qualitative thematic data analysis strategies Objective 2 7 Does USAID Okard have the appropriate management system and partnership structures to accomplish its objectives? KIIs FGDs DHR (1 KII) NCDE (1 KII) CMR (1 KII) Department of Education and Sport, TVET division (1KII) WEI (2 KIIs) HI (1 KII) Sub-recipients (5KIIs) VTE Stakeholder type FCR matrix (updated weekly) Data coding Qualitative thematic data analysis strategies 97 | EVALUATION FINAL REPORT USAID.GOV EVALUATION MATRIX – USAID OKARD MTE # Evaluation Question Data Collection Tools Data Source and Quantity Location of Data Collection Data Disaggregation Data Analysis 8 Do USAID Okard partners have the right systems and capacity to fulfill their scope of work? What systems and/or capacities are currently missing in order to fulfill the scope of work? KIIs FGDs WEI (1 KII) HI (1 KII) Sub-recipients (5KIIs) VTE Stakeholder type FCR matrix (updated weekly) Data coding Qualitative thematic data analysis strategies 98 | EVALUATION FINAL REPORT USAID.GOV ANNEX 3: CASE STUDY – STORY OF CHANGE AS A RESULT OF CBID ACTIVITIES Case study data were collected from 16 community stakeholders (9 males; 7 females), including 12 beneficiaries in Xayphouthong and Kham districts. Data were collected from women and men with disabilities, caregivers of persons with disabilities, village leaders, and community members. Care was taken to collect data from persons of a variety of ages and with different types of disabilities. However, due to the very small sample size, community-level data for this MTE were not analyzed based on sex, age or type of disability. Table 12 provides details of the individuals who informed this MTE case study. TABLE 12: COMMUNITY-LEVEL BENEFICIARY INFORMANTS FOR THE OKARD MTE # Type of interview District Age (person with disability) Type of disability Services received 1 Men with disability Xayphouthong 30 years Physical disability Treatment at rehab centre; goat house built; waiting for goat 2 Man with disability Xayphouthong 55 years Physical disability Rehabilitation suggested- did not go; Built pig house; waiting for pig 3 Woman with disability Xayphouthong 25 years Physical disability Received wheelchair; Waiting for goat. 4 Woman with disability Xayphouthong 52 years Memory and hearing problem Okard wanted to send to rehab but did not go; Difficult to travel. Built goat house; waiting for goat 5 Caregiver (F) Xayphouthong 40 years Deaf Okard wanted to train in hair dressing, mother did not allow; Raising pigs. Not received yet 6 Caregiver (F) Xayphouthong 9 years Physical & mental disability Referred to rehab centre. Training on massage. Difficult to follow up at home. Did not receive EE support 7 Community member (M) Xayphouthong N/A N/A NA 8 Village leader (M) Xayphouthong N/A N/A NA 9 Man with disability Kham 70 years Physical disability - leg Prosthetic leg and crutches; Chose pig farming; waiting 10 Man with disability Kham 50 years Physical disability Walking stick received; hasn’t used it. Mechanic equipment coming. 11 Woman with disability Kham 25 years Vision & hearing loss Okard suggested referral to rehab but family cannot accompany; Weaving presented as an IGA option but does not want it as daughters would need to do it. 99 | EVALUATION FINAL REPORT USAID.GOV # Type of interview District Age (person with disability) Type of disability Services received 12 Caretaker (M) Kham Child (not provided) Physical disability Wheelchair & toilet chair provided. Bought loom; waiting for silk. 13 Caretaker (F) Kham 19 years Physical disability Referred to rehab and received physiotherapy and medicine. Challenge to do physiotherapy at home. Fish arming but did not receive yet. 14 Caretaker (M) Kham 12 years Physical disability Received wheelchair & physiotherapy; too busy to do it; Fish farming selected but mother does not want it as she is busy in the field; Would like assistance with toilet and bathtub 15 Community member (F) Kham N/A N/A N/A 16 Village leader (M) Kham N/A N/A N/A In September 2019, CBID Demonstration Model Activities began in 35 villages in Kham district and 16 villages in Xayphouthong district. The Okard CBID Demonstration Model Activity aims to demonstrate how the lives of persons with disabilities can be improved and sustained over the long-term if support for health, economic empowerment, and stakeholder engagement is provided at both the systems level and the community level. The aim of this case study is to narrate a ‘story of change’ of how the Okard CBID Demonstration Model Activity has impacted the lives of community level stakeholders and beneficiaries. Community level informants, without exception, spoke about CBID staff who came to their homes and communities with the aim of improving the lives of persons with disabilities. Informants reported that CBID staff held community meetings and engaged in a process that identified persons with disabilities in targeted communities. Many informants recalled activities that were intended to inform community members, including community members without disabilities, about the rights and abilities of persons with disabilities. Importantly, many informants spoke about how the Okard Activity changed attitudes towards persons with disabilities in their communities. “Currently, the people without disabilities have more understanding [about persons with disabilities] since the project started. [Okard] organized a meeting to disseminate [information about] the rights of people with disabilities to villagers. They invited all the villagers to attend the meeting. So now [the villagers] have changed their attitudes [towards people with disabilities].” Most informants recognized that the Activity was unable to help every person with a disability, but the Activity had selected, “those with the greatest needs.” During data collection, no informant indicated that any specific vulnerable group, including women, ethnic minorities, persons who are very old or young, or live in remote areas were excluded from the Activity. Many informants reported that no project had ever before come to their village to help persons with disabilities, which made them grateful for the Okard Activity. Some informants reported that villagers were sometimes unsure of the Activity at first, partly because other foreign funded projects had previously come to their village and made promises that were 100 | EVALUATION FINAL REPORT USAID.GOV not kept or were not helpful. In this way, the Okard Activity had to gain the trust of villagers. One informant thought that villagers started to trust the Okard Activity after they began to see people getting help. Informants spoke about the lengthy and detailed data collection and screening tools that the CBID implementation teams used to collect data about beneficiaries and families. Beneficiaries had a mixed reaction to this lengthy data collection process. Amongst beneficiaries who received services after the data collection, they were mainly focused on speaking about the services they received. Amongst beneficiaries who had not yet received services, they were more likely to report that, “[the Okard Activity] collected information and then…. disappeared,” which was communicated with a sense of frustration. These data indicate that a reasonably quick roll out of the Activity is an important component of gaining the trust of villagers. One informant pointed out that one problem might have been that villagers did not, “understand the implementation process” and, “wanted help immediately.” Many of the beneficiaries who contributed to this MTE spoke about receiving services, most notably healthcare services. Many beneficiaries were received a referral to a rehabilitation centre. In most cases, the costs of this referral, including travel and accommodation, were provided by the Activity. However, in some cases, beneficiaries reported having to make their way to the rehabilitation centre or a district hospital at their own expense. Amongst beneficiaries who received financial assistance for travel and accommodation, all were grateful. Notably, even amongst non-disabled informants, such as community members and village leaders, the travel and accommodation coverage aspect of the Activity was recognized as generous and good. As one informant said, “I have never seen any project that really wants to help us like this. They really pay attention.” Many beneficiaries reported receiving assistive devices such as a wheelchair, a toilet seat, crutches, a walking stick, orthopedic shoes, and a prosthetic leg. Many beneficiaries also reported receiving instruction for physiotherapy or massage from the rehabilitation center. Unfortunately, amongst the respondents who were received instruction on physiotherapy or massage, none were able to follow-up consistently at home due to heavy time demands on family members who would be responsible for the follow-through. In some cases, beneficiaries received a referral to a rehabilitation center but did not want to go. Amongst these individuals, many reported apprehensions about leaving their home or community or not having a family member who could accompany them. Often, for people who live as subsistence farmers, it can be very challenging to send one or two family members away even for a few days as their labor in the field is needed for the survival of the rest of the family. For some people, even a fully funded trip to a rehabilitation center may not be tenable. Amongst beneficiaries who received assistive devices, nearly all spoke about the important improvement that the assistive device brought to their life or the life of their family member with a disability. One respondent reported that he is, “...better now. [He] can walk longer after [he] got the walking crutch and the shoes.” This responded reported that in the past walking for him was, “very difficult.” One young woman reported, “… [my life] is better now…. because I have the wheelchair.” Another father reported how the assistive products he received for his daughter has improved her quality of life: “The Okard Activity has given us a lot. Let me tell you about it. They gave us a wheelchair and a toilet seat…… The wheelchair helps alleviate difficulty for my child when she goes outside. When we went to Vientiane capital, they bought us a crutch to help with walking, a defecating seat, and a potty. These are all things that they helped us with.” Perhaps the most compelling story collected from beneficiaries during this MTE came from a grandfather who cares for his 12-year-old grandson. The grandfather reported that the child was, “born normal” but then developed many physical disabilities. Through the first 11 years of this boys’ life, the family took him 101 | EVALUATION FINAL REPORT USAID.GOV to many doctors, including traditional medicine doctors and spiritual healers. Over the years, the family spent more than three million Kip (about $310 USD) seeking a cure for this child. “After the project team came,” the grandfather reported, “…we no longer relied on other alternatives.” The Activity provided the child with a wheelchair that supports his collar bones and shoulders and is, “tailored for him”. They were also provided with a toilet seat. The remainder of this story is best expressed in the grandfathers’ words as he describes the life-changing outcomes for this child that resulted from support from the Okard Activity: “The… wheelchair was tailored for him. The team measured and collected all the data needed, like they were tailoring a shirt. They measured his legs and hips. After two weeks, they called us and took us to try it out. It took another whole day for us to adjust the wheelchair to be suitable for him, which required two engineers to help us.” “What I like most about the project is the support on the wheelchair. Without the wheelchair, he could not go anywhere. In the past, after waking up, we brought him to lay outside the room. After we had the wheelchair, wherever we go, we can bring him with us. And he is happier when he meets people…. [Sometimes] he talks to his friends or people in the same age as him passing by or shopping at his mother’s retail shop.” “The [Okard Activity] took excellent care of him. They provided the wheelchair. When he grows up, we would like the project to provide us another wheelchair with a rope which helps us to pull him back when he moves forward. In addition, the wheelchair should enable him to exercise his hand. The current one we have is the one without the hand exercising tool.” This story is a testament to many goals achieved by the Okard Activity. Clearly, the boy’s life is changed because of the wheelchair. Moreover, the family has learned where they can go to get services, which will end their long and costly journey through multiple other providers including traditional and spiritual healers. While the Okard Activity may not always be present in the community to directly provide services, this family will at least always know where they can receive the services to help their family member. Potentially, this family or families who live near this boy, will pass this message on to other families who may have a similar need and are looking for help. Nearly all respondents to this MTE also spoke about how the Okard Activity to assist persons with disabilities and their family members to engage in economic empowerment activities. Many individuals and family members reported that the Okard Activity was planning to provide them with pigs, chickens, goats as well as silk for weaving and in one case mechanics tools. While respondents were looking forward to receiving these assets, and many had already invested in building an animal shelter or purchased a weaving loom, none had received the asset yet. While some respondents were frustrated by the delay and potentially worried that the asset may never arrive, others were optimistic because they had heard that the assets had started to be delivered in other villages. Other respondents were aware that the current delays were caused by the COVID-19 lockdowns. One respondent spoke about confusion over the type of economic empowerment activities that could be provided by the Okard Activity and how this impacted beneficiaries: “What the project team said without any confirmation caused…confusion, and the villagers thought that the project gave them hope and then lied. For example, some people proposed pig farming. [But then] … the welfare team checked the field and found that the area had not been UXO checked. Then, the UXO team came and checked. But the field could not be used for pig farming because the UXO process did not proceed. Therefore, the project team should check and say [when] they are really sure [that the Activity can be completed].” 102 | EVALUATION FINAL REPORT USAID.GOV One respondent thought that if the Okard Activity, “follows the plan” it will be able to, “improve the self￾sufficiency” of persons with disabilities. Another respondent thought the Activity is, “helping persons with disabilities [by] providing an animal for these people to raise.” It is noteworthy that several beneficiaries reported that they did not want to participate in the economic empowerment activity that was available to them by the Okard Activity. In several cases, respondents thought that they were too busy working in their fields to take responsibility for a farm animal or engage in weaving. On one case, a woman recognized that she would be unable to participate in the activity herself and thought her family members were too busy. In another situation, the Activity suggested that a deaf woman receive hair dressing training, but the family was concerned about the woman going away from the village for the training and requested that she instead be given pigs to raise. This story of change will end by giving the last word to two beneficiaries who wanted to send a message to the Okard Activity. These sentiments expressed by beneficiaries sum up their experience and gratitude for the Okard Activity. “I just want to thank the Okard Activity for caring about me.” “I am very appreciative for the support from the [Okard Activity] for our family, especially helping to improve my son’s health and relieve the burden on his medical expenses. We are very happy to see him become stronger and healthier.” 103 | EVALUATION FINAL REPORT USAID.GOV ANNEX 4: BIBLIOGRAPHY 1. USAID. (n.d) Activity Theory of Change Model. 2. Association for Rural Mobilization and Improvement (ARMI). (2019). Final Technical Proposal for USAID Okard. 3. Cooperative Orthotic and Prosthetic Enterprise (COPE). (2021). Draft Procurement Transition Plan. 4. COPE. (2021). Final Technical Proposal for USAID Okard. 5. DMAS. (2019). Final Technical Proposal for USAID Okard. 6. Franck, Bernard, ed. (2018). Year 1 Annual Progress Report. 7. Franck, Bernard, ed. (2018). Year 1 Quarter 1 Progress Report. 8. Franck, Bernard, ed. (2018). Year 1 Quarter 2 Progress Report. 9. Franck, Bernard, ed. (2018). Year 1 Quarter 3 Progress Report. 10. Franck, Bernard, ed. (2019). Year 2 Annual Progress Report. 11. Franck, Bernard, ed. (2019). Year 2 Quarter 1 Progress Report. 12. Franck, Bernard, ed. (2019). Year 2 Quarter 2 Progress Report. 13. Franck, Bernard, ed. (2019). Year 2 Quarter 3 Progress Report. 14. Franck, Bernard, ed. (2020). Year 3 Annual Progress Report. 15. Franck, Bernard, ed. (2020). Year 3 Quarter 1 Progress Report. 16. Franck, Bernard, ed. (2020). Year 3 Quarter 2 Progress Report. 17. Franck, Bernard, ed. (2020). Year 3 Quarter 3 Progress Report. 18. Franck, Bernard, ed. (2021). Year 4 Quarter 1 Progress Report 19. Ministry Of Finance. (2018). Decree 4000: DSA Rates 20. Nossal Institute for Global Health; World Education Inc. (2020). Community Based Inclusive Development (CBID) Demonstration Model Assessment: Baseline Survey Report. 21. Nossal Institute for Global Health; World Education Inc. (2021). Community Based Inclusive Development (CBID) Case Management Comprehensive Need Assessment: CBID Modular Tool 22. Quality of Life Association (QLA). (2019). Final Technical Proposal for USAID Okard. 23. SEAMEO VOCTECH. (2020). TVET Country Profile: Lao P.D.R. 24. Schmoger, C. (2015). Improving the Inclusion of Persons with Disabilities in TVET in Lao PDR, 1st of February to 3rd of April 2015. 25. Social Impact. (2019). Formative Study Report USAID/Lao PDR Okard Baseline Evaluation. 26. Social Impact. (2018). USAID/Lao PDR Gender Analysis on Disability: Gender Analysis Final Report. 27. USAID ASIA. (2018). Okard Annual Progress Report: Period Y1 28. USAID ASIA. (2019). Okard Annual Progress Report: Period Y2 29. USAID ASIA. (2021). Okard Annual Progress Report: Period Y3 30. USAID Laos, World Education Inc. (2021). Okard Data Quality Assessment Final Report 31. USAID Okard. (n.d). USAID Okard MOU Timeline Summary 32. USAID Okard. (2019). Mental Health Intervention Visual 33. USAID Okard. (2019). ARMI Proposal 34. USAID Okard. (2019). QLA CBID Proposal 35. USAID Okard. (2019). DMAS Proposal 36. USAID Okard. (2019). Communications and Outreach Plan FY20. 37. USAID Okard. (2019). Handout: Basic Monitoring, Evaluation and Learning. 38. USAID Okard. (2019). Module MEL Tools. 39. USAID Okard. (2019). USAID Okard MEL tools training for QLA. 40. USAID Okard. (2020). Activity Monitoring, Evaluation, and Learning (AMELP). 41. USAID Okard. (2020). Community Awareness Raising and Screening. 42. USAID Okard. (2020). Outreach and Communication Plan FY21. 43. USAID Okard. (2020). PISR Note for Data Quality Assessment from Discussion with RDMA. 104 | EVALUATION FINAL REPORT USAID.GOV 44. USAID Okard. (2020). Prioritization for Needs Assessment. 45. USAID Okard. (2021). Annex 1 ITT USAID Okard (FY-21) (Excel Document). 46. USAID Okard. (2021). Environmental Mitigation and Monitoring Plan. 47. USAID Okard. (2021). Gender and Inclusive Development Action Plan (GIDAP). 48. USAID Okard. (2021). Mid-term EE Evaluation Brief. 49. USAID Okard. (2021, unpublished). Social and Behavior Change Communication (SBCC) Strategy and Action Plan: Reaching for a Better Life. 50. USAID Okard. (2021). COPE Proposal 51. USAID ASIA. (2021). Okard Annual Progress Report: Period Y3 52. USAID. (2016). USAID Evaluation Toolkit. 53. USAID. (2020). Memorandum of Understanding Between The Ministry of Labour and Social Welfare and Humanity Inclusion Concerning Okard Project Supporting Disability Inclusive Development in Lao PDR 54. USAID. (2018). USAID Okard: Annual Work Plan (Year 1) 55. USAID. (2018). USAID Okard: Annual Work Plan (Year 2) 56. USAID. (2019). USAID Okard: Annual Work Plan (Year 3) 57. USAID. (2020). USAID Okard: Annual Work Plan (Year 4) 58. USAID. (2020). Service Request Technical Proposal. 59. USAID. (2021). USAID Okard Social Behavior Change Communication Strategy and Action Plan: Reaching For A Better Life. 60. USAID, HI. (2020). Self-Help Groups 61. USAID, HI, WEI. (2021). Summary of Self-Help Group Activities 62. Winrock International. (2020). Per Diem Rate Letter 63. WEI. (n.d). Action Plan of Lao-Korea Institute for Skills Development Vientiane (Word Document). 64. WEI. (2020). National Policy-Strategty and Action Plan on Persons with Disabilities (Lao Version) 65. WEI. (n.d). Okard Master Plan Final (Internal PDF). 66. WEI. (n.d). Action Plan of Vientiane Skill Development Centre (Word Document). 67. WEI. (n.d). Action Plan of Xaysombath Technology College (Word Document). 68. WEI. (2016). Health Serctor Reform Strategy and Framework TILL 2025. 69. WEI. (2017). Meeting Notes: WE-HI-DSSA (Word Document). 70. WEI. (2017). Meeting Minutes: USAID Okard Activity with MPH, WEI, ansd HI (Word Document). 71. WEI. (2017). Meeting Minutes: USAID OKAD with NCDE, WEI, and HI (Word Document). 72. WEI. (2018). Meeting Minutes: Final MOU Review (Word Document). 73. WEI. (2018). Meeting Minutes: USAID Okard MOU NCDE (Word Document). 74. WEI. (2018). Meeting Minutes: USAID Okard MOU NCDE and DHR (Word Document). 75. WEI. (2018). Meeting Minutes (April): USAID Okard MOU NCDE, DHR, and WEI (Word Document). 76. WEI. (2018). Meeting Minutes: USAID Okard Gener POC Final (Word Document). 77. WEI ( 2018). Meting Minutes (May): USAID Okard MOU NCDE, DHR, and WEI (Word Document). 78. WEI (2018). Meeting Minutes: USAID Okard MOU NCDE, DHR, and WEI Final (Word Document). 79. WEI (2018). Meeting Minutes: MOU NCDE and WEI (Word Document). 80. WEI. (2018). Meeting Minutes: US Amb. Involvement NCDE and WEI (Word Document). 81. WEI. (2019). Meeting Minutes: Final MOU Review (Word Document). 82. WEI. (2019). Meeting Minutes: Okard MOU (Word Document). 83. WEI. (2019). Meeting Minutes: USAID Okard with TVET. (Word Document). 84. WEI. (2019). Meeting Minutes: USAID Okard with General Director (Word Document). 85. WEI. (2020). Meeting Minutes: DHR Scholarhsip (Word Document) 105 | EVALUATION FINAL REPORT USAID.GOV 86. WEI. (2020). National Policy-Strategty and Action Plan on Persons with Disabilities (Lao Version) 87. World Bank. (2010). Lao PDR Civil Service Pay and Compensation Review: Attractign and Motivating Civil Servants 88. World Education, Inc. (2020). Systematic Assessment of Rehabilitation Situation (STARS) in Lao People’s Democratic Republic (Lao PDR). 106 | EVALUATION FINAL REPORT USAID.GOV ANNEX 5: DATA COLLECTION INSTRUMENTS KII CONSENT FORM Hello. My name is ______________, and I am here on behalf of Social Impact who has been contracted by the U.S Agency for International Development (USAID) to conduct a mid-term evaluation on the Okard Activity. [This is _________, she/he will be taking notes today]. You have been selected to participate in this evaluation because of your knowledge of this Activity. Today’s discussion is expected to last approximately 90 minutes in English. We will ask you questions related to the Okard Activity’s overall progress at achieving programmatic results. We expect to speak with approximately 200 people as part of this evaluation. The information that you and others provide will be used to write a report. This report will be shared with USAID and other stakeholders for comment and will eventually be made public. However, your individual responses in this discussion will be kept in confidence by the ET. This means that only the ET—and nobody else—will have access to discussion notes that include any personal identifying information, such as your name. Although we will report our evaluation findings in aggregate in the report, your name, position, and other personally identifying information will not appear in any reporting. Discussion notes that do not contain any personal identifying information (such as your name) may be given to USAID, if requested. Your participation is voluntary. We do not anticipate that participating in this discussion will result in any risks or direct benefit to you. However, your inputs may lead to recommendations that benefit the Okard Activity—and, thereby, the general public. You do not have to participate or answer specific questions if you do not want to. Should you choose to participate, please know that you may change your mind at any point during our discussion and can stop the discussion at any time. Given the COVID-19 pandemic there are several reasons you may choose not to participate in the study. If you or someone in your household or workplace has been feeling sick including having a cough or high temperature in the past two weeks, we ask you not to participate for your safety and the safety of others. Also, we plan to use safety protocols when we talk such as sitting far apart, wearing masks, and trying to talk in open air spaces, but if you are worried that these measures are not possible or not enough protection, you may not want to participate. With your permission, I would like to audio record this interview. Even though we have ________ here taking notes, the audio recording will be used by our ET so we can more accurately understand your perspective. The audio recording will not be shared with anyone outside of our ET. Contact: If you have any concerns or questions about the evaluation, you may contact the Social Impact Institutional Review Board at “irb@socialimpact.com”, or the M&E Specialist, Salika Khoonbarthao at “skhoonbarthao@socialimpact.com”. OBTAIN CONSENT Do you agree to participate in this study? Yes / No Are you willing to be recorded? Yes / No 107 | EVALUATION FINAL REPORT USAID.GOV KII PROTOCOL: USAID EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE INTRODUCTION To be collected from participant(s): Stakeholder type, Location To begin, please tell me about your involvement with the USAID Okard Activity. 1. What is your role with the project? 2. How long have you been involved? ENGAGEMENT 3. How engaged is USAID with the Okard Activity? a. Are you satisfied with the level of engagement? Achievement of expected results – System level Achievement of results – Gender and Inclusion Relevance: Stakeholder Engagement GOVERNMENT ENGAGEMENT/POLICY 4. In your opinion, does the government of Lao PDR have the capacity to support an improved policy framework which would lead to improved access to services for persons with disabilities in Lao PDR? 5. Does the government of Lao PDR have the will to support an improved policy framework which would lead to improved access to services for persons with disabilities in Lao PDR? 6. Has the Okard Activity contributed to improved knowledge and/or skills within the government of Lao PDR with regards to disability inclusion? How so? SI-3: Operational criteria achieved by the MOH and NCDE strategy and action plan coordinating bodies Relevance: Stakeholder Engagement Achievement of results – Systems Level MANAGEMENT 7. What is your opinion regarding the management structure of the Okard Activity? 8. How effective has the Okard Activity been at managing sub-recipients? 9. In your opinion how is the relationship between HI and WEI working? 10. What have been some of the main management issues for the Okard Activities? ES4-3: USG assisted organizations and/or service delivery systems that serve vulnerable persons strengthened 108 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE Relevance: Stakeholder Engagement STAKEHOLDER ENGAGEMENT 11. Do you think that the activities that have been implemented or started by the Okard Activity regarding Stakeholder Engagement are effective? a. If so, are they sufficient? Relevance: Economic Empowerment ECONOMIC EMPOWERMENT 12. Do you think that the activities that have been implemented or started by the Okard Activity regarding Economic Empowerment are effective? a. If so, are they sufficient? EI-3: # of persons with disabilities and their households with improved economic self￾sufficiency Achievement of results – People Centered HEALTH 13. Do you think that the activities that have been implemented or started by the Okard Activity regarding Health are effective? a. If so, are they sufficient? HI-4: # of people who demonstrate increase in functioning or wellbeing Relevance: Economic Empowerment Health Stakeholder Engagement CBID 14. In your opinion, are the CBID projects progressing in the manner that you think they should for this stage of the Okard Activity? a. What have been the challenges? b. What needs to be done to improve the CBID component of the Okard Activity? HI-2: % of persons with disabilities and their households who receive “People Centered Care” UNEXPECTED OUTCOMES 15. Have there been any unexpected outcomes that have resulted from the Okard Activity? Achievement of expected results – System level SUSTAINABILITY 16. In your opinion, is there potential for the Okard Activity to be sustained at the conclusion of the activity? a. What could be done to improve sustainability? CONCLUSION 17. Thank you for taking the time to speak to me today about the USAID Okard Activity. I have finished asking all my question now. Before we finish, is there anything else you would like to share with 109 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE me? Perhaps there is something that I did not cover, and you would like to add? 110 | EVALUATION FINAL REPORT USAID.GOV KII PROTOCOL: WEI EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE INTRODUCTION To be collected from participant(s): Stakeholder type, Sex, Age, Location To begin, please tell me about your involvement with the USAID Okard project. - What is your role with the project? - How long have you been involved? Achievement of expected results – System level Achievement of results – Gender and Inclusion Relevance: Stakeholder Engagement GOVERNMENT ENGAGEMENT/POLICY I would like to understand how the USAID Okard Activity is currently situated within the policy framework of the Government of Lao PDR. - What policies are currently in place or in development with regards to rehabilitation or support for persons who are disabled in Lao PDR? - Is the National Rehabilitation Committee operational? -If no, what is holding it up and do you expect it to become operational soon? If so, when? -If yes, when did it become operational? Did USAID Okard contribute to the launching of this committee? If yes, how so? - What benefit does or would the National Rehabilitation Committee bring to the policy landscape of Lao PDR? - Has the government of Lao PDR developed or begun the process to develop policy that would specifically support women who are disabled in Lao PDR? Can you please explain? - Has the government of Lao PDR developed or begun the process to develop policy that would target inclusion of children who are disabled either through better social inclusion, integration into school, or improved access to health services? Can you explain? - In your opinion, is the policy framework currently in place in Lao PDR favorable for the sustainability of USAID Okard after the Activity is complete? - In your opinion, does the government of Lao PDR have the capacity or will to support an improved SI-1 Activities in the Disability Action Plan initiated and monitored by the NCDE. SI-2: Participants involved in government led policy processes who self-identify as persons with a disability SI-3: Operational criteria achieved by the MOH and NCDE strategy and action plan coordinating bodies 111 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE policy framework which would lead to improved access to services for persons with disabilities in Laos, including health, education, and economic opportunities? Why or why not? - Since the start of the Okard Activity, do you think that people within the Government of Lao PDR have improved knowledge or the technical skills they need to do the work necessary to create an improved policy environment with regards to supporting people who are disabled? If yes, do you think this improved knowledge or technical skills can be attributed to the USAID Okard Activity? If no, what needs to happen so that people within the Government of Lao PDR gain the knowledge and skills they need to develop a better policy environment that will support people who are disabled? - Are there any people who self-identify as a person with a disability who currently or have in the past been involved in any government led policy processes? -If yes, can you explain to me who these people were and what their role is/was? Were any of these people female? Was Okard involved in this process? How so? - To your knowledge are Disability Committees at the provincial and/or district/village level functional? Why or why not? What needs to happen to make these functional? What benefit would these committees bring to USAID Okard and/or the government/policy framework in Lao PDR. MOH AND NCDE STRATEGY AND ACTION PLAN/Disability Action Plan - Can you please explain to me the main elements of the MOH and NCDE strategy and/or the Disability action plan? Are these different documents? - Are these plans currently operational? If no, why not? What is needed for them to become functional? - Are there any criteria of the MOH and NCDE action plan or the Disability Action Plan that are already achieved or will soon be achieved? Which ones? 112 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE - What do you think are the most critical factors that could contribute to the successful coordination and implementation and monitoring of these plans? - Are there any activities in the MOH and NCDE Strategy and/or Action Plan or the Disability Action Plan that are or were initiated and monitored by the NCDE? -If yes, which ones? Can you describe them to me? -If no, what would need to happen for the NCDE to begin to initiate or monitor them MANAGEMENT - Can you please explain to me how the USAID Okard project is managed? - What is the management relationship between WEI and HI? - How are sub-recipients managed? - What are the major issues related to management within the USAID Okard project? - Is there anything else that the ET should understand about the management of the USAID Okard project? Components that were implemented successfully / not successfully Achievement of results – Systems Level Achievement of results – Gender and Inclusion HEALTH I would like to understand how USAID Okard has thus far contributed to increased utilization of health and related rehabilitation services for persons with disabilities and their households. I am also interested to know about supports that USAID Okard has provided for improved capacity amongst health workers in respect to mental health and rehabilitation. INCREASED NUMBERS AND CAPACIY OF HEALTHCARE AND REHABILITAITON WORKFORCE - What is USAID Okard currently doing or planning to do to increase the number and capacity of skilled workers in the rehabilitation workforce? - Please can you tell me about support that was provided to health workers to improve capacity to provide mental health or rehabilitation services - Does USAID Okard currently have plans that specifically target increasing the number of females working in the field of rehabilitation - ES4-2: Service providers trained who serve vulnerable persons ES4-3: USG assisted organizations and/or services delivery systems that serve vulnerable persons strengthened. H-1: Service delivery systems with improved 113 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE TRAINING OF HEALTH AND REHABILITATION SERVICE PROVIDERS, INCLUDING MENTAL HEALTH - Has USAID Okard completed any training with health, mental health, or rehabilitation professionals? - If no, why not? When is this expected to happen? - If yes, how much of the training is complete? What types of training were completed? What other training is planned for the future? Who did the training? -Was the training effective? Was there a training of the trainer component? What was this like? -Did training for service providers grow their capacity to provide mental health or mental health support interventions for persons who are disabled and their households? Was this training effective in your opinion? Did USAID Okard receive any feedback regarding this training? -Did training for health, mental health and rehabilitation service providers grow their capacity to utilize compassionate communication when engaging with persons who are disabled and/or their household members? How is this evident or not evident? In your opinion, what is needed to improve health, mental health, and rehabilitation service providers capacity to utilize compassionate communication? Is this something that can be reasonably accomplished during the lifespan of the USAID Okard Activity? -Did health, mental health, and rehabilitation staff receive training in people-centered care or women’s centered care? Would you say that this training was sufficient or effective? Why or why not? -Do you have any knowledge about how many health, mental health, and rehabilitation service providers who were trained were women? Was there an attempt to specifically engage women in this training? If so, how were women engaged? Do you have any knowledge about how this training impacted women specifically? capacity to provide women￾centered care. 114 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE - Please can you tell me about any scholarships that may have been provided for mental health or rehabilitation providers to receive training elsewhere, such as in Thailand. Please explain to me how many of the scholarships were awarded, to whom and if any person has already started the course – as well as any other information that you think may be useful to the ET. REHABILITAITON SERVICES AT ALL LEVELS OF THE HEATH CARE SYSTEM - My understanding is that USAID Okard plans to provide rehabilitation interventions at each level of the health care system. How far along in the process is USAID Okard towards meeting this goal? Do these rehabilitation services include mental health services? - Are interventions that should be available at each level of the health care system clearly defined? Why or why not? If no, what needs to happen to achieve this? Do they include mental health services? REFERRAL - Can you please explain to me how the referral system works in Lao PDR? - Are these referral pathways working? -If no, what are some of the challenges that currently exist with regards to referral for persons who are disabled and/or require assistive equipment or other specialized healthcare services, inc? -How is USAID Okard addressing or planning to address challenges related to referrals for persons who are disabled in Lao PDR? -To date, has USAID Okard had any success with regards to improving the referral system related to rehabilitation in Lao PDR? PROCUREMENT - Currently, in Lao PDR, are there regulatory structures or mechanisms in place for the procurement of assistive products? -If yes, are these structures working? Who is leading this process? What has been the 115 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE contribution of USAID Okard? What could improve these structures? -If no, what needs to happen so that these regulatory structures can be put into place? What is USAID Okards plan to enable this process? HEALTH INFORMATION - Is routine data related to disability / rehabilitation currently collected within the DHIS2 platform? -If no, is there a plan to integrate the collection of rehabilitation data within the DHIS2 platform? How is USAID Okard contributing to this process? Do you expect that this will be completed before the end of the USAID Okard Activity? -If yes, what type of rehabilitation data is currently collected in the DHSI2 platform? Is this sufficient? How is this data disaggregated? Is data collected specifically for women? To date, has there been any functional use of this data by the MOH or USAID Okard? If yes, how so? Components that were implemented successfully / not successfully Relevance: Stakeholder Engagement Relevance: Assumption Health Relevance: Economic Empowerment SUB RECIEPIENTS I would like to understand better the process that USAID Okard went through to select sub-recipient grantees. - How were grantees selected? - Can you tell me about the process? Did it take longer than expected? What were the challenges? How did these challenges impact the USAID Okard activity? - When were the sub-recipient grants signed? To what extent has work on the sub-grants started? - Briefly, can you please tell me a little bit about each of the sub-grantees? I am interested to know about how long about they signed the sub-agreement, the main objectives of the project, things they have been able to accomplish thus far and whatever else you think might be beneficial to the ET. FUNDING FOR SUB-RECEIPIENTS - How has USAID Okard supported sub-recipients and manage USAID funding? Has this been successful? What are the challenges? SUPPORT FOR PROJECT IMPLEMENTATION ES4-1: Vulnerable persons benefiting from USG reported social services ES4-2: Service providers trained who serve vulnerable persons ES4-3: USG assisted organizations and/or service delivery systems that serve vulnerable 116 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE Achievement of results – Systems Level Achievement of results – People Centered Achievement of results – Gender and Inclusion - To what extent is USAID Okard supporting sub￾grantees in the implementation of their projects? - What plan does USAID Okard have to foster the knowledge and skills necessary for sub-grantees to manage their projects independently? TRAINING FOR SUB-RECEIPIENTS - Has any training for sub-recipients already taken place? If yes, what types? When did these trainings happen? - How has USAID Okard assessed the capacity needs of sub-recipients? Do your feel that USAID Okard has a good understanding of what these capacity needs are and has realistic plans in place to meet these needs? What are the needs that have already been identified? What other needs are there? - Were sub-recipients trained on accessible communication guidelines? -If no, is there a plan for this? When will it happen? -If yes, how effective was this training, in your opinion? Have you seen any information or communication materials developed by sub￾partners? Is this material developed using accessible communication guidelines? - Did sub-recipients receive diversity and inclusivity training, including gender specific training? -If yes, was it effective, in your opinion? What more is needed or planned with regards to training sub-recipients regarding the specific needs of women who are disabled? -If no, is there a plan for training specific to gender and inclusivity for sub-recipients? - Did sub-recipients receive training regarding USAID rules and regulations? Was this training effective, in your opinion? Was it useful to sub-recipients? COMMUNITY BASED INCLUSIVE DEVELOPMENT (CBID) - I am interested to understand the extent to which the CBID projects are underway. Can you explain this to me? persons strengthened HI: Service delivery systems with improved capacity to provide women￾centered care. H2: Persons with disabilities and their households who received “people centered care” H3: People receiving health and related rehabilitation services 117 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE - To what extent do the CBID facilitator teams represent diversity. Is there ate least one woman and one person with a disability on each team? What needs to be done to increase diversity amongst these teams? - To what extent has the CBID projects been able to support persons who are disabled and their households in the process of becoming more economically self-sufficient? - Are there any success stories that you are aware of? -How is data on this aspect of the project being captured? - What needs to be done to improve this aspect of the project? - To what extent has the CBID projects been successful in providing mental health support to persons who are disabled and their family members? -Who is providing this mental health support? How were they trained? Is the training sufficient? What could be improved? - To what extent has the CBID projects been able to support persons who are disabled and their household members in the process of accessing health services? -Has access to assistive products or other necessary medical treatments improved since start of the CBID projects? Do you know of any success stories? What are the challenges? What could be done to improve the ability of CBID projects to connect persons who are disabled and their household members with healthcare services and improve access to assistive devices? - Are the CBID project also focusing on the special needs of children who are disabled? If yes, what are the goals for the CBID projects with respect to children? Has any of this work already started? Achievement of expected results – System level TVET INSTITUTIONS I would like to ask you about your knowledge related to USAID Okard support to TVET institutions for inclusion of persons who are disabled. MOTIVATION EI-1: Persons with disabilities who graduate from TVET centers. 118 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE Achievement of results – Gender and Inclusion Relevance: Economic empowerment - In your opinion are TVET institutions motivated to improve opportunities for persons with disabilities within their institutions? Why or why not? -If no, what could be done to improve the motivation of TVET institutions to include persons who are disabled? TRAINING - Are you aware of any training that was provided to TVET staff by USAID Okard? - If yes, please explain to me what you know about this training? - In your opinion was this training sufficient? If no, what more is needed? - From your knowledge and experience, what were the most impactful approaches and techniques used in the training to build capacity of TVET staff to address stereotypical attitudes towards people who are disabled, reduce gender stereotypes, or create greater diversity in opportunities for people who are disabled within TVET programs? SCHOLARSHIPS - Can you tell me about scholarships that may have been awarded to vulnerable populations for participation in TVET programs? - In your opinion, is the USAID project working towards achieving the results – as should be expected for this stage of the project - as it was hoping to achieve with regards to improved inclusion of persons with disabilities in TVET programs? -Why or why not? EI-3: Persons with disabilities and their households with improved economic self￾sufficiency GNDR-2: Female participants in USG assisted programs designed to increase access to productive economic resources ES4-2: Service providers trained who serve vulnerable populations ES4-3: USG assisted organizations and/or services delivery systems that serve vulnerable persons strengthened. Components that were implemented successfully / not successfully INFORMATION AND COMMUNICATION I would like to ask you about information and communication that is developed by the USAID Okard Activity. Are you aware of any information or 119 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE Achievement of results – Gender and Inclusion communication that was developed and distributed by the USAID Okard Activity? If yes, - Can you describe the different information or communication products that you are aware of? - Who were the intended audiences for these information or communication products? - Do you know if persons with disabilities were consulted when designing these information and communication products? - In your opinion, were these products delivered in formats that meet the different needs of stakeholders? - Did any of the information and communication products use accessible design features, perhaps braille, large print, sign language, audio format, or subtitles? - Would you say that the information and communication products used accessible language? - Were any of the information and communication products that you are aware of targeted towards women or girls? If yes, -Can you describe these to me? - In your opinion, how effective were this information or communication products that targeted women and girls? - Were any of the information and communication products that you are aware of targeted towards youth or children? -Can you describe these to me? - In your opinion, how effective were this information or communication products that targeted youth or children? - In your opinion, did messages use simple and clear language? - What do you think could improve information and communication strategies currently employed by USAID Okard? 120 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE CONCLUSION - Thank-you for taking the time to speak to me today about the USAID Okard Activity. I have finished asking all my question now. Before we finish, is there anything else you would like to share with me? Perhaps there is something that I did not cover, and you would like to add? 121 | EVALUATION FINAL REPORT USAID.GOV KII PROTOCOL: HI EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE INTRODUCTION To be collected from participant(s): Stakeholder type, Sex, Age, Location To begin, please tell me about your involvement with the USAID Okard project. 1. What is your role with the project? 2. How long have you been involved? 3. Can you please tell us a little bit about the history of HI in Lao PDR? Achievement of expected results – System level Achievement of results – Gender and Inclusion Relevance: Stakeholder Engagement GOVERNMENT ENGAGEMENT/POLICY 4. In your opinion, does the government of Lao PDR have the capacity to support an improved policy framework which would lead to improved access to services for persons with disabilities in Lao PDR? 5. Relatedly, does the government of Lao PDR have the will to support an improved policy framework which would lead to improved access to services for persons with disabilities in Lao PDR? 6. Has the Okard Activity contributed to improved knowledge and/or skills within the government of Lao PDR with regards to disability inclusion? How so? 7. What is/or has been the involvement of persons who self-identify with a disability in government led processes related to disability inclusion? a. Probe: has this differed for men, women, youth? What about ethnic minorities? Rural/urban persons with disabilities? 8. What needs to happen to improve the policy framework for disability inclusion in Lao PDR? a. Probe: what are the considerations for men, women, youth? What about ethnic minorities? Rural/urban persons with disabilities? SI-2: Participants involved in government led policy processes who self-identify as persons with a disability SI-3: Operational criteria achieved by the MOH and NCDE strategy and action plan coordinating bodies MANAGEMENT 9. Please can you explain to me the management structure of the Okard Activity within HI? 10. How is staffing for the Okard Activity managed? a. How many staff? How are they assigned? What is the diversity? 122 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE b. Has there been any staffing challenges? Components that were implemented successfully / not successfully Relevance: Stakeholder Engagement Achievement of results – Systems Level RELATIONSHIPS AND SUB-RECIEPIENTS 11. My understanding is that HI required an MOU with the GoL before commencing work in the Okard Activity. Can you tell me about the process of getting this MOU? 12. Please can you tell me about the relationship between HI and USAID? a. What has been HIs involvement in getting waivers/agreements for project activities? b. Has HI received any support for working with USAID? 13. Please can you tell me about the relationship between HI and WE? 14. Please can you tell me about sub-recipients that are managed by HI? I am interested to know about sub-recipients who have already signed agreements or who will soon sign agreements. a. What are the strengths and limitations of these sub-recipients? ES4-3: USG assisted organizations and/or service delivery systems that serve vulnerable persons strengthened Components that were implemented successfully / not successfully Relevance: Stakeholder Engagement Relevance: Economic Empowerment Relevance: Stakeholder Engagement CBID I understand that HI is responsible for the Stakeholder Engagement and the Economic Empowerment (EE) components of the Okard Activity. Stakeholder Engagement 15. Please tell me about the Stakeholder Engagement Approach that HI is taking for the Okard Activity 16. What has been accomplished thus far with regards to Stakeholder Engagement within the Okard Activity? 17. Was any training conducted or is any further training planned or under development with regards to the Economic Empowerment component of the Okard Activity? a. Please tell me about the details of this training. 18. What still needs to happen with regards to the Stakeholder Engagement component of the Okard Activity. ES4-1: Vulnerable persons benefiting from USG reported social services ES4-2: Service providers trained who serve vulnerable persons ES4-3: USG assisted organizations and/or service delivery systems that serve vulnerable 123 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE Achievement of results – People Centered 19. In your opinion, what has worked well with regards to Stakeholder Engagement within the Okard Activity and what could be better? Economic Empowerment 20. Please tell me about Approach that HI is taking towards Economic Empowerment within the Okard Activity? 21. What has been accomplished thus far with regards to Economic Empowerment within the Okard Activity? 22. Was any training conducted or is any further training planned or under development with regards to the Economic Empowerment component of the Okard Activity? a. Please tell me about the details of this training. 23. What still needs to happen with regards to the Economic Empowerment component of the Okard Activity. 24. In your opinion, what has worked well with regards to Economic Empowerment within the Okard Activity and what could be better? 25. To what extent are TVET institutions included in the Economic Empowerment component of the Okard Activity? a. Is there a plan to increase involvement of TVET institutions within the Economic Empowerment component of the Okard Activity? persons strengthened HI: Service delivery systems with improved capacity to provide women￾centered care. H2: Persons with disabilities and their households who received “people centered care” Achievement of expected results – System level CONCLUSION 26. In your opinion, is the Okard Activity working toward achieving results as should be expected for this stage of the project with regards to Economic Empowerment and Stakeholder Engagement? 27. Thank-you for taking the time to speak to me today about the USAID Okard Activity. I have finished asking all my question now. Before we finish, is there anything else you would like to share with me? Perhaps there is something that I did not cover, and you would like to add? EI-3: Persons with disabilities and their households with improved economic self￾sufficiency EI-1: Persons with disabilities who graduate from TVET centers. 124 | EVALUATION FINAL REPORT USAID.GOV KII PROTOCOL: AFA EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE INTRODUCTION To be collected from participant(s): Stakeholder type, Location, Gender To begin, please tell me about your involvement with the USAID Okard Activity. 1. What is your role with the Okard Activity? 2. How long have you been involved? 3. How many staff will work for AFA under the Okard Activity? a. Is there enough staff to do the work that needs to be done? b. Does your team include women and persons with disabilities? Relevance: Stakeholder Engagement GOVERNMENT POLICY 4. Up to now, has AFA been invited to participate in government policy development regarding disability inclusion? 5. To your knowledge, are persons with disabilities, included in GOL policy development for disability inclusion? a. Has this differed for women, youth? What about ethnic minorities and people living in rural areas? SI-2: # of participants involved in government led policy process who self-identify as a person with a disability Relevance: Stakeholder Engagement Achievement of results – System Level ENGAGEMENT 6. Up to now, what has been the process of signing the sub-agreement with the Okard Activity? a. Are there any challenges? b. If yes, is the Okard Activity helping you overcome the challenges? How? 7. How would you describe the relationship between AFA and the Okard Activity? ES 4-3: # of USG assisted organizations and/or service delivery systems that serve vulnerable persons strengthened Relevance: Stakeholder Engagement SCOPE OF WORK 8. Can you please tell me about the work that AFA plans to do with regards to stakeholder engagement for the Okard Activity? ES 4-3: # of USG assisted organizations and/or service delivery systems that serve vulnerable 125 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE a. Do you feel that AFA has the capacity to complete this work in the time frame of the Okard Project? b. Do you anticipate any challenges? 9. How does AFA plan to develop “disability role models”? a. Do you feel that AFA has the capacity to complete this work in the time frame of the Okard Project? b. Do you anticipate any challenges c. What will be needed to make the plan for “disability role models” successful? 10. Can you please tell me about the work that AFA plans to do with regards to improved governance of organizations for persons with disabilities for the Okard Activity? a. Do you feel that AFA has the capacity to complete this work in the time frame of the Okard Project? b. Do you anticipate any challenges? 11. Can you explain to me the work AFA has done or plans to do with regards to developing a network of organizations for persons with disabilities in Lao PDR? a. Have there been any challenges with the development of this network? Please explain. 12. Does AFA plan to establish stakeholder engagement strategies that specifically target women or youth, people who are ethnic minorities or rural/urban persons with disabilities? TRAINING 13. Has AFA received any training from the Okard Activity? Please explain. If yes, a. Was this training relevant to AFA? b. Was it useful to the work AFA plans to do? c. Who was selected to participate in the training? d. Were male and female staff invited to attend the training? persons strengthened 126 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE If no, e. What types of training does AFA hope to receive from the Okard Activity? FUNDING 14. Is the funding AFA is set to receive for work with the Okard Activity sufficient? Why or why not? Achievement of expected results – System level SUSTAINABILITY 15. In your opinion, after the contribution provided by the Okard Activity to AFA, is there potential for the Okard Activity to be sustainable at the conclusion of the Activity?” ES 4-3: # of USG assisted organizations and/or service delivery systems that serve vulnerable persons strengthened CONCLUSION 16. Thank-you for taking the time to speak to me today about the Okard Activity. I have finished asking all my question now. Before we finish, is there anything else you would like to share with me? Perhaps there is something that I did not cover, and you would like to add? 127 | EVALUATION FINAL REPORT USAID.GOV KII PROTOCOL: ARMI EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE INTRODUCTION To be collected from participant(s): Stakeholder type, Location, Gender 1. Briefly, can you tell me a little about the history of ARMI and its work in Lao PDR? Please tell me about your involvement with the Okard Activity, specifically the CBID demonstration model. 2. What is your role with the CBID demonstration model? 3. How long have you been involved? Relevance: Stakeholder Engagement GOVERNMENT 4. Up to now, has ARMI been invited to participate in government policy development regarding disability inclusion? 5. To your knowledge, are persons with disabilities, including women, people who live rurally, people who are elderly or young, or people who are very poor included in GOL policy development for disability inclusion? a. If yes, how are people selected for inclusion? b. is this inclusion sufficient? 6. How would you describe the relationship between ARMI and government partners? a. Does ARMI feel accepted by government partners? b. Does ARMI feel respected by government partners? c. Is the relationship between ARMI and certain partners better than the relationship other partners? Please describe. 7. How would you describe the relationship between ARMI and HI? SI-2: # of participants involved in government led policy process who self-identify as a person with a disability Relevance: Stakeholder Engagement Achievement of results – System Level ENGAGEMENT AND MANAGEMENT 8. What was the process of signing the sub-agreement with the Okard Activity? a. Were there any challenges? b. If yes, did the Okard Activity help you overcome the challenges? How? ES 4-3: # of USG assisted organizations and/or service delivery systems that serve vulnerable 128 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE 9. Do you feel that you have enough staff to complete the work that needs to be done to successfully implement the CBID demonstration model? a. Does your team include women and persons with disabilities? 10. Were there any delays in starting the CBID demonstration model? Please explain. persons strengthened Relevance: Economic Empowerment Relevance: Health Achievement of results – People Centered Achievement of results – Gender and Inclusion SCOPE OF WORK 11. Can you please tell me about how the CBID demonstration model identifies persons with disabilities in CBID communities? a. What is working well with regards to this screening process? b. What are the challenges? c. Do you feel that the CBID demonstration model is able to adequately identify vulnerable people such as woman with disabilities or persons with disabilities who live in rural areas, are elderly or young, or very poor? d. Is there anything that should be done to improve this component of the CBID demonstration model? 12. How is the CBID demonstration model assessing the needs and function of persons with disabilities in CBID communities? a. How is the modular tool working? b. What could be improved about this component of the CBID demonstration model? 13. How does the CBID demonstration model prioritize the needs of persons with disabilities in CBID communities? a. Are there challenges related to prioritizing the needs of persons with disabilities? b. Do you feel that the CBID demonstration model equitably prioritizes women with disabilities, or persons with disabilities who live in rural areas, are elderly or young, or are very poor? 14. How is the CBID demonstration model meeting the mental health needs of persons with disabilities in CBID districts? HI-2: % of persons with disabilities and their households who receive “People Centered Care” HI-3: # of people receiving health and related rehabilitation services HI-4: # of people who demonstrate increase in functioning or wellbeing EI-2: # of persons with disabilities and their households who receive IGA or complete job readiness EI-3: # of persons with disabilities and their households with improved economic self￾sufficiency GNDR-2: % of female 129 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE a. Does the ARMI CBID team have sufficient resources to meet the mental health needs of persons with disabilities in CBID communities? b. Do you feel that the CBID demonstration model equitability addressed the mental health needs of all persons with disabilities who are identified as needing mental health service including women, people who live in rural areas, are elderly or young, or are very poor? c. What else is needed to support the mental health needs of persons with disabilities in CBID areas, if anything? d. Are there any examples of how mental health services to persons with disabilities in CBID demonstration model areas were met successfully? 15. Do you feel that persons with disabilities have better access to health care because of work performed by the CBID demonstration model? Why or why not? a. Do you feel that the CBID demonstration model equitability improves access to health care services for all persons with disabilities in CBID areas including women, people who live in rural areas, are elderly or young, or are very poor? b. How is the case management system for health working? c. What could be improved? 16. How is the CBID demonstration model handling referral of persons with disabilities in CBID communities? a. Is the referral process working? b. What are the challenges, if any? c. Have there been any successes? 17. How does ARMI monitor work done on the CBID demonstration model? a. Are there any challenges with monitoring project activities? b. Is the monitoring data used to improve the CBID activity? participants in USG assisted programs designed to increase access to productive economic resources PM1: # of targeted beneficiaries with increased knowledge and awareness 130 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE 18. What work is the CBID demonstration model doing to increase Economic Self-Sufficiency amongst persons with disabilities and their households in CBID communities? a. Does the ARMI CBID team have sufficient resources to meet the needs of persons with disabilities and their households with regards to economic self-sufficiency? b. Do you feel that the CBID demonstration model is able to provide equitable support for economic self-sufficiency to all persons with disabilities including women, people who live in rural areas, people who are elderly or young, or people who are very poor? c. What are the challenges? d. Have there been any successes? 19. To what extent has the CBID demonstration model implemented by ARMI engaged non-disabled community members in programming to increase awareness of disability inclusiveness? a. How effective have these interventions been? b. Have there been any successes? TRAINING 20. Has ARMI received any training from the Okard Activity? If yes, a. Please describe the types of training. b. Who was selected for this training? Were male and female staff or staff with disabilities given equal opportunity to participate? c. Was this training relevant to the work that ARMI is doing in CBID communities? d. Did the training help build the capacity of ARMI staff to be successful in their work implementing the CBID demonstration model? e. What other training would assist the ARMI staff to better implement the CBID demonstration model? FUNDING 21. Is the funding provided to ARMI by the Okard Activity, sufficient to implement the CBID demonstration model? 131 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE Achievement of expected results – System level SUSTAINABILITY 22. In your opinion, after the contribution provided by the Okard Activity to ARMI, is there potential for the CBID demonstration model to be sustainable at the conclusion of the Activity?” ES 4-3: # of USG assisted organizations and/or service delivery systems that serve vulnerable persons strengthened CONCLUSION 23. Thank-you for taking the time to speak to me today about the CBID demonstration model. I have finished asking all my question now. Before we finish, is there anything else you would like to share with me? Perhaps there is something that I did not cover, and you would like to add? 132 | EVALUATION FINAL REPORT USAID.GOV KII PROTOCOL: CMR EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE INTRODUCTION To be collected from participant(s): Stakeholder type, Location, Gender To begin, please tell me about your involvement with the USAID Okard Activity. 1. What is your role with the project? 2. How long have you been involved? Relevance: Stakeholder Engagement ENGAGEMENT 3. Are you satisfied with the level of engagement between CMR and the Okard Activity? Why or why not? Relevance: Health Achievement of results – System Level CONTINUOUS DEVELOPMENT TRAINING CENTER 4. Can you please tell me about the CMRs work on the Continuous Development Training Centre? 5. Which trainings have already taken place at this training center? 6. What trainings are planned for the future at this training center? 7. How does/has the Okard Activity support/ed your work at this training Center? a. Are you satisfied with this support? 8. What have been the successes and challenges with regards to your work at this training center? ES 4-2: # of service providers trained who serve vulnerable persons Relevance: Health HEALTH PROFESSIONAL SURVEY I understand that the CMR is engaged in developing a survey which will be distributed to health professionals 9. Who is this survey targeting? 10. What information do you hope to gather from the survey? 11. What is the status of the survey currently? 12. How will the information that you gather from this survey inform your work with regards to improving capacity for rehabilitation amongst health care professionals? 13. How has the Okard Activity supported the work on this survey ES 4-2: # of service providers trained who serve vulnerable persons 133 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE a. Are you satisfied with the support you have received for developing/ implementing this health professional survey? Relevance: Health REHABILITATION DEVICES 14. How is the CMR engaged in the procurement of assistive devices? 15. Has the CMR received support from the Okard Activity with regards to procurement of assistive devices? How so? 16. What is the relationship between CMR and COPE with regards to procurement of assistive devices? a. How satisfied are you with the partnership between CMR and COPE? HI-4: # of people who demonstrate increase in functioning or wellbeing FINANCIAL SUPPORT 17. Are you satisfied with how the Okard Activity financially supports the work within the CMR? Why or why not? Achievement of results – System Level SUSTAINABILITY 18. In your opinion, does the contribution of the Okard Activity to the Continuous Development Training Center have the potential to be sustainable at the conclusion of the Activity? USG assisted organizations and/or service delivery systems that serve vulnerable persons strengthened CONCLUSION 19. Thank-you for taking the time to speak to me today about the Okard Activity. I have finished asking all my question now. Before we finish, is there anything else you would like to share with me? Perhaps there is something that I did not cover, and you would like to add? 134 | EVALUATION FINAL REPORT USAID.GOV KII PROTOCOL: COPE EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE INTRODUCTION To be collected from participant(s): Stakeholder type, Location, Gender To begin, please tell me about your involvement with the USAID Okard Activity. 1. What is your role with the Okard Activity? 2. How long have you been involved? 3. How many staff will work for COPE under the Okard Activity? a. Is there enough staff to do the work that needs to be done? b. Are any COPE staff women or persons with disabilities? Relevance: Stakeholder Engagement GOVERNMENT 4. Up to now, has COPE been invited to participate in government policy development regarding disability inclusion? 5. To your knowledge, are persons with disabilities, including women, people who live rurally, people who are elderly or young, or people who are very poor included in GOL policy development for disability inclusion? a. If yes, how are people selected for inclusion? b. If yes, is this inclusion sufficient? 6. What is the relationship between COPE and the government partners (CMR, DHR)? a. Are there any challenges? b. Does COPE feel accepted by government partners? c. Does COPE feel respected by government partners? d. Is the relationship between COPE and certain partners better than with other partners? Please describe SI-2: # of participants involved in government led policy process who self-identify as a person with a disability Relevance: Stakeholder Engagement ENGAGEMENT 7. Up to now, what has been the process of signing the sub-agreement with the Okard Activity? a. Are there any challenges? ES 4-3: # of USG assisted organizations and/or service delivery systems 135 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE Achievement of results – System Level b. If yes, is the Okard Activity helping you overcome the challenges? How? 8. How would you describe the relationship between COPE and the Okard Activity? that serve vulnerable persons strengthened Relevance: Health Achievement of results – System Level Achievement of results – People Centered SCOPE OF WORK 9. Can you tell me about the work COPE plans to do with regards to supporting persons with disabilities to access assistive devices? a. Do you feel that COPE has the capacity to complete this work in the time frame of the Okard Activity? b. Do you feel that COPE is able to provide equal access to assistive devices for all persons with disabilities including women, people who live in rural areas, people who are elderly or young or people who are very poor? c. Do you anticipate any challenges? 10. Can you please tell me about the work that COPE plans to do with related to providing training courses to increase capacity of rehabilitation personnel to effectively provide assistive devices to persons with disabilities? a. Do you feel that COPE has the capacity to complete this work in the time frame of the Okard Activity? b. Do you know what types of rehabilitation personal will be targeted for these trainings? c. Is there a plan to specifically target the inclusion of women into these trainings? d. Do you anticipate any challenges? 11. Can you please tell me about the work COPE plans to do with regards to supporting CMR to procure P&O consumables? a. Do you feel that COPE has the capacity to complete this work in the time frame of the Okard Activity? b. Do you anticipate any challenges? 12. What other activities does COPE hope to achieve during the time frame of the Okard Activity? ES 4-1: # of vulnerable persons benefiting from USG￾reported social services ES 4-2: # of service providers trained who serve vulnerable persons HI-3: # of people receiving health and related rehabilitation services HI-4: # of people who demonstrate increase in functioning or wellbeing 136 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE a. Is there a plan to distribute a beneficiary survey? b. If yes, can you provide details regarding plans for this survey? TRAINING 13. Has COPE received any training from the Okard Activity? If yes, a. Please describe the types of training b. Who was selected for the training? Were male and female staff or staff with disabilities given equal opportunity to participate? c. Was this training relevant to COPE? d. Did the training help build the capacity of COPE staff to be successful in their work with the Okard Activity? e. What other training would assist the COPE staff to better implement the Okard Activity FUNDING 14. Is the funding COPE is set to receive for work with the Okard Activity sufficient? Why or why not? Achievement of expected results – System level SUSTAINABILITY 15. In your opinion, after the contribution provided by the Okard Activity to COPE, is there potential for the Okard Activity to be sustainable at the conclusion of the Activity?” ES 4-3: # of USG assisted organizations and/or service delivery systems that serve vulnerable persons strengthened CONCLUSION 16. Thank-you for taking the time to speak to me today about the Okard Activity. I have finished asking all my question now. Before we finish, is there anything else you would like to share with me? Perhaps there is something that I did not cover, and you would like to add? 137 | EVALUATION FINAL REPORT USAID.GOV KII PROTOCOL: DHR EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE INTRODUCTION To be collected from participant(s): Stakeholder type, Location, Gender To begin, please tell me about your involvement with the USAID Okard Activity. 1. What is your role with the project? 2. How long have you been involved? 3. Are there other staff within the DHR who support the Okard Activity? Relevance: Stakeholder Engagement ENGAGEMENT 4. How satisfied are you with the level of engagement between DHR and the Okard Activity? Please explain. Relevance: Stakeholder Engagement Achievement of results – System Level Achievement of results – Gender and Inclusion NATIONAL REHABILITATION STRATEGY 5. Can you please tell me about the DHRs work on the National Rehabilitation Strategy? 6. What are the mechanisms within the MOH to monitor the implementation of this strategy? 7. How does/has the Okard Activity support/ed the work on this strategy? a. Are you satisfied with this support? 8. Up to now what activities have been implemented? 9. Were women included in consultations regarding the development of the National Rehabilitation Strategy? 10. Were other vulnerable populations, including men and women with disabilities or persons with disabilities who live in rural areas, are elderly or young, or are very poor included in consultations regarding the development of the National Rehabilitation Strategy? 11. Were persons with disabilities, including women, included in consultations regarding the development of the National Rehabilitation Strategy? If yes, how so? 12. What have been the successes and challenges with regards to the implementation of the National Rehabilitation Strategy? SI-2: # of participants involved in government led policy process who self-identify as a person with a disability 138 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE 13. Did the STARS Baseline Report help you prioritize activities to strengthen rehabilitation within the health system in Lao PDR? a. If yes, did the STARS report help you understand the specific needs of sub-groups of persons with disabilities? Relevance: Health Achievement of results – System Level Achievement of results – People Centered INTEGRATION OF REHABILITATION INTO THE HEALTH SYSTEM 14. In your opinion, how effective has training provided by the Okard Activity been at increasing capacity of health providers to provide rehabilitation services in Lao PDR? 15. What has been the contribution by the Okard Activity to supporting the DHR to improve the availability of rehabilitation services at all levels of the health care system, including at the provincial and district level? a. As a result of the Okard Activity, have certain sub-groups of people such as men or women with disabilities, persons with disabilities who live in rural areas, people who are elderly or young, or people who are very poor benefited from improved access to rehabilitation services at all levels of the health care system? 16. What has been the contribution of the Okard Activity to supporting the DHR to procure assistive devices for persons with disabilities in Lao PDR? 17. What is needed to further integrate rehabilitation into the health system in Lao PDR? Scholarships My understanding is that the DHR should manage the scholarships for 15 students to attend training in Thailand. 18. What progress has been made with regards to disseminating these scholarships? 19. What are the challenges? 20. What needs to happen to make this component of the Okard Activity successful? Health Facility Assessment My understanding is that the DHR was engaged in the recent Health Facility Assessment. ES 4-2: # of service providers trained who serve vulnerable persons HI-3: # of people receiving health and related rehabilitation services 139 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE 21. How did the Okard Activity support work on this Health Facility Assessment? 22. How will the results from the Health Facility Assessment impact integration of rehabilitation services into the health system in Lao PDR? FINANCIAL AND TECHNICAL SUPPORT 23. Are you satisfied with how the Okard Activity financially supports the work within the DHR? Why or why not? Achievement of expected results – System level SUSTAINABILITY 24. In your opinion, does the contribution of the Okard Activity to the National Rehabilitation Strategy have the potential to be sustainable at the conclusion of the Activity? CONCLUSION 25. Thank-you for taking the time to speak to me today about the Okard Activity. I have finished asking all my question now. Before we finish, is there anything else you would like to share with me? Perhaps there is something that I did not cover, and you would like to add? 140 | EVALUATION FINAL REPORT USAID.GOV KII PROTOCOL: DMAS EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE INTRODUCTION To be collected from participant(s): Stakeholder type, Location, Gender To begin, please tell me about your involvement with the USAID Okard Activity. 1. What is your role with the Okard Activity? 2. How long have you been involved? 3. How many staff work for DMAS under the Okard Activity? a. Is there enough staff to do the work that needs to be done? b. Are any DMAS staff working on the Okard project women or persons with disabilities? Relevance: Stakeholder Engagement GOVERNMENT POLICY 4. Currently, is there government policy in place to support disability inclusion for persons with disabilities? If yes, a. Does this policy specifically address inclusive employment or accessibility for persons with disabilities? b. Has DMAS been invited to contribute to GoL policy for disability inclusion? c. To your knowledge, are persons with disabilities, including women, people who live in rural areas, people who are elderly or young, or people who are very poor included in GOL policy development for disability inclusion? i. If yes, is this inclusion sufficient? SI-2: # of participants involved in government led policy process who self-identify as a person with a disability Relevance: Stakeholder Engagement Achievement of results – System Level ENGAGEMENT AND MANAGEMENT 5. What was your process for starting the DMAS Center? a. How did the Okard Activity assist with starting the DMAS Center? 6. How was the process of signing the sub-agreement with the Okard Activity? a. Were there any challenges? b. If yes, did the Okard Activity help you overcome the challenges? How? ES 4-3: # of USG assisted organizations and/or service delivery systems that serve vulnerable persons strengthened 141 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE 7. How would you describe the relationship between DMAS and HI? 8. Do you feel that DMAS is accepted by Government Partners? 9. Do you feel that DMAS is respected by Government Partners? 10. Is the relationship between DMAS and certain partners better than the relationship with other partners? Please describe. Relevance: Economic Empowerment Achievement of results – People Centered Achievement of results – Gender and Inclusion INCLUSIVE EMPLOYMENT AND ACCESSIBILITY 11. Can you please tell me about the work that DMAS is doing with regards to inclusive employment? 12. Can you tell me about the work DMAS is doing with regards to accessibility for persons with disabilities? 13. Can you tell me about the work DMAS is doing with regards to support to TVET for improved inclusivity? 14. Does DMAS currently or plan in the future to establish inclusive employment strategies that specifically target women or youth, people who are ethnic minorities or rural/urban persons with disabilities? 15. Have there been any challenges related to the work DMAS is doing? a. What about compliance with USAID regulations? b. What other challenges are there? 16. Have there been any successes related to the work DMAS is doing? 17. Has the Okard Activity assisted the DMAS center to overcome any challenges? How so? TRAINING 18. What training has DMAS received from the Okard Activity to support your work? a. Please describe the types of training b. Who was selected for the training? Were male and female staff or staff with disabilities given equal opportunities to participate? c. Was this training relevant to DMAS? d. Did the training help build the capacity of DMAS staff to successfully implement the Okard Activity? GNDR-2: % of female participants in USG assisted programs designed to increase access to productive economic resources EI-3: # of persons with disabilities and their households with improved economic self￾sufficiency ES 4-2: # of service providers trained who serve vulnerable persons EI-1: # of persons with disabilities who graduate from TVET centers 142 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE e. What could be improved with regards to the training received by DMAS? FUNDING 19. Is the funding DMAS received for work with the Okard Activity sufficient? Why or why not? Achievement of expected results – System level SUSTAINABILITY 20. In your opinion, after the contribution provided by the Okard Activity to initiate DMAS, is there potential for DMAS to be sustainable at the conclusion of the Activity?” ES 4-3: # of USG assisted organizations and/or service delivery systems that serve vulnerable persons strengthened CONCLUSION 21. Thank-you for taking the time to speak to me today about the Okard Activity. I have finished asking all my question now. Before we finish, is there anything else you would like to share with me? Perhaps there is something that I did not cover, and you would like to add? 143 | EVALUATION FINAL REPORT USAID.GOV KII PROTOCOL: NCDE EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE INTRODUCTION To be collected from participant(s): Stakeholder Type, Location, Gender To begin, please tell me about your involvement with the USAID Okard Activity. 1. What is your role with the project? 2. How long have you been involved? Relevance: Stakeholder Engagement Achievement of results – System Level DISABILITY POLICY 3. Can you tell me about disability policy in Lao PDR? I am interested to understand about the National Disability Policy Strategy Action Plan and the Disability Law. 4. How has the Okard Activity supported the NCDE to develop or implement Disability Policy in Lao PDR? a. Are you satisfied with this support? 5. To what extent have persons with disabilities, including women, persons from rural areas, persons who are ethnic minorities, persons who are elderly or young or persons who are very poor been engaged in the process of developing Disability Policy in Lao PDR? 6. What challenges does the NCDE face with regards to developing or disseminating Disability Policy in Lao PDR? 7. What can be done to improve the development or dissemination of the Disability Policy in Lao PDR? SI-1: # of activities in the Disability Action Plan initiated and monitored by the NCDE. SI-3: # of operational criteria achieved by the MOH and NCDE strategy and action Plan coordinating bodies SI-2: # of participants involved in government led policy process who self-identify as a person with a disability Relevance: Stakeholder Engagement ENGAGEMENT 8. Can you please describe the level of engagement between the NCDE with the Okard Activity? a. Are you satisfied with this level of engagement? 144 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE 9. In your opinion, how well does the NCDE coordinate with other ministries to support the operationalization of disability policy in Lao PDR? a. Has the Okard Activity supported the NCDE to coordinate with other ministries to operationalize Disability Policy? How so? FUNDING 10. Are you satisfied with how the Okard Activity financially supports the work within the DHR? Why or why not? Relevance: Stakeholder Engagement Achievement of results – People Centered CBID DEMONSTRATION MODEL 11. What is the level of involvement of the NCDE with the CBID Demonstration Model? 12. What is your understanding of the activities that are currently underway or completed for the CBID Demonstration Model? 13. In your opinion, does the CBID Demonstration model contribute to disability policy system strengthening at the provincial and district levels? 14. How is the CBID demonstration model supporting persons with disabilities in communities? SUSTAINABILITY 15. In your opinion, does the contribution by the Okard Activity have the potential to be sustainable at the conclusion of the activity? USG assisted organizations and/or service delivery systems that serve vulnerable persons strengthened CONCLUSION 16. Thank-you for taking the time to speak to me today about the Okard Activity. I have finished asking all my question now. Before we finish, is there anything else you would like to share with me? Perhaps there is something that I did not cover, and you would like to add? 145 | EVALUATION FINAL REPORT USAID.GOV KII PROTOCOL: QLA EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE INTRODUCTION To be collected from participant(s): Stakeholder type, Location, Gender 1. Briefly, can you tell me a little about the history of QLA and its work in Lao PDR? Please tell me about your involvement with the Okard Activity, specifically the CBID demonstration model. 2. What is your role with the CBID demonstration model? 3. How long have you been involved? Relevance: Stakeholder Engagement GOVERNMENT 4. Up to now, has QLA been invited to participate in government policy development regarding disability inclusion? 5. To your knowledge, are persons with disabilities, including women, people who live rurally, people who are elderly or young, or people who very poor included in GOL policy development for disability inclusion? a. If yes, how are people selected for inclusion? b. is this inclusion sufficient? 6. How would you describe the relationship between QLA and government partners? a. Does QLA feel accepted by government partners? b. Does QLA feel respected by government partners? c. Is the relationship between QLA and certain partners better than the relationship with other partners? Please describe. 7. How would you describe the relationship between QLA and HI? SI-2: # of participants involved in government led policy process who self-identify as a person with a disability Relevance: Stakeholder Engagement Achievement of results – System Level ENGAGEMENT AND MANAGEMENT 8. What was the process of signing the sub-agreement with the Okard Activity? a. Were there any challenges? b. If yes, did the Okard Activity help you overcome the challenges? How? ES 4-3: # of USG assisted organizations and/or service delivery systems that serve vulnerable 146 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE 9. Do you feel that you have enough staff to complete the work that needs to be done to successfully implement the CBID demonstration model? a. Does your team include women and persons with disabilities? 10. Were there any delays in starting the CBID demonstration model? Please explain. persons strengthened Relevance: Economic Empowerment Relevance: Health Achievement of results – People Centered Achievement of results – Gender and Inclusion SCOPE OF WORK 11. Can you please tell me about how the CBID demonstration model identifies persons with disabilities in CBID communities? a. What is working well with regards to this screening process? b. What are the challenges? c. Do you feel that the CBID demonstration model is able to adequately identify vulnerable people such as woman with disabilities or persons with disabilities who live in rural areas, people who are elderly or young, or people who are very poor? d. Is there anything that should be done to improve this component of the CBID demonstration model? 12. How is the CBID demonstration model assessing the needs and function of persons with disabilities in CBID communities? a. How is the modular tool working? b. What could be improved about this component of the CBID demonstration model? 13. How does the CBID demonstration model prioritize the needs of persons with disabilities in CBID communities? a. Are there challenges related to prioritizing the needs of persons with disabilities? b. Do you feel that the CBID demonstration model equitably prioritizes women with disabilities, or persons with disabilities who live in rural areas, people who are elderly or young, or people who are very poor? HI-2: % of persons with disabilities and their households who receive “People Centered Care” HI-3: # of people receiving health and related rehabilitation services HI-4: # of people who demonstrate increase in functioning or wellbeing EI-2: # of persons with disabilities and their households who receive IGA or complete job readiness EI-3: # of persons with disabilities and their households with improved economic self￾sufficiency GNDR-2: % of female 147 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE 14. How is the CBID demonstration model meeting the mental health needs of persons with disabilities in CBID districts? a. Does the QLA CBID team have sufficient resources to meet the mental health needs of persons with disabilities in CBID communities? b. Do you feel that the CBID demonstration model equitability addressed the mental health needs of all persons with disabilities who are identified as needing mental health service including women, people who live in rural areas, people who are elderly or young, or people who are very poor? c. What else is needed to support the mental health needs of persons with disabilities in CBID areas, if anything? d. Are there any examples of how mental health services to persons with disabilities in CBID demonstration model areas were met successfully? 15. Do you feel that persons with disabilities have better access to health care because of work performed by the CBID demonstration model? Why or why not? a. Do you feel that the CBID demonstration model equitability improves access to health care services for all persons with disabilities in CBID areas including women, people who live in rural areas, people who are elderly or young, or people who are very poor? b. How is the case management system for health working? c. What could be improved? 16. How is the CBID demonstration model handling referral of persons with disabilities in CBID communities? a. Is the referral process working? b. What are the challenges, if any? c. Have there been any successes? 17. How does QLA monitor work done on the CBID demonstration model? a. Are there any challenges with monitoring project activities? participants in USG assisted programs designed to increase access to productive economic resources PM1: # of targeted beneficiaries with increased knowledge and awareness 148 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE b. Is the monitoring data used to improve the CBID activity? 18. What work is the CBID demonstration model doing to increase Economic Self-Sufficiency amongst persons with disabilities and their households in CBID communities? a. Does the QLA CBID team have sufficient resources to meet the needs of persons with disabilities and their households with regards to economic self-sufficiency? b. Do you feel that the CBID demonstration model is able to provide equitable support for economic self-sufficiency to all persons with disabilities including women, people who live in rural areas, people who are elderly or young, or people who are very poor? c. What are the challenges? d. Have there been any successes? 19. To what extent has the CBID demonstration model implemented by QLA engaged non-disabled community members in programming to increase awareness of disability inclusiveness? a. How effective have these interventions been? b. Have there been any successes? TRAINING 20. Has QLA received any training from the Okard Activity? If yes, a. Please describe the types of training. b. Who was selected for this training? Were male and female staff or staff with disabilities given equal opportunity to participate? c. Was this training relevant to the work that QLA is doing in CBID communities? d. Did the training help build the capacity of QLA staff to be successful in their work implementing the CBID demonstration model? e. What other training would assist the QLA staff to better implement the CBID demonstration model? 149 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE FUNDING 21. Is the funding provided to QLA by the Okard Activity, sufficient to implement the CBID demonstration model? Achievement of expected results – System level SUSTAINABILITY 22. In your opinion, after the contribution provided by the Okard Activity to QLA, is there potential for the CBID demonstration model to be sustainable at the conclusion of the Activity?” ES 4-3: # of USG assisted organizations and/or service delivery systems that serve vulnerable persons strengthened CONCLUSION 23. Thank-you for taking the time to speak to me today about the CBID demonstration model. I have finished asking all my question now. Before we finish, is there anything else you would like to share with me? Perhaps there is something that I did not cover, and you would like to add? 150 | EVALUATION FINAL REPORT USAID.GOV KII PROTOCOL: TVET EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE INTRODUCTION To be collected from participant(s): Stakeholder type, Location, Gender To begin, please tell me about your involvement with the USAID Okard Activity. 1. How have you been engaged with the Okard Activity? 2. What has been your level of engagement with the Okard Activity? Relevance: Stakeholder Engagement GOVERNMENT POLICY 3. To your knowledge, is there currently any government policy that supports inclusion of persons with disabilities into TVET? 4. Have you or anyone in your organization been invited to GoL policy discussions regarding inclusion of persons with disabilities into TVET? a. If so, who was included? Were any women or persons with disabilities included? SI-2: # of participants involved in government led policy process who self-identify as a person with a disability Relevance: Economic Empowerment Achievement of results – System Level Achievement of results – People Centered Achievement of results – Gender and Inclusion SCOPE OF WORK 5. From your experience, how well are persons with disabilities included in TVET? a. What are the challenges b. Have you had any successes? 6. How has the Okard Activity supported TVET to increase inclusiveness of TVET for persons with disabilities? Has any support received from the Okard Activity focused on improving inclusiveness for women with disabilities, or persons with disabilities who live in rural areas, persons who are ethnic minorities, or people who are very poor? 7. What else could the Okard Activity do to help TVET increase inclusiveness for persons with disabilities? TRAINING 8. Has this TVET institution received any training from the Okard Activity? If yes, EI-1: # of persons with disabilities who graduate from TVET centers GNDR-2: % of female participants in USG assisted programs designed to increase access to productive economic resources 151 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE a. Who was included in the training? b. Were any women or persons with disabilities included in the training? c. Was this training relevant to the TVET institution? How so? If no, d. What types of training would assist TVET institution to increase inclusiveness for persons with disabilities? CONCLUSION 9. Thank you for taking the time to speak to me today about the Okard Activity. I have finished asking all my question now. Before we finish, is there anything else you would like to share with me? Perhaps there is something that I did not cover, and you would like to add? 152 | EVALUATION FINAL REPORT USAID.GOV KII PROTOCOL: HEALTH WORKERS – FOCUS MENTAL HEALTH EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE INTRODUCTION To be collected from participant(s): Stakeholder Type, Location, Gender To begin, please tell me about your involvement with the USAID Okard Activity. 1. What is your role with the project? 2. How long have you been involved? Relevance: Stakeholder Engagement ENGAGEMENT 3. Can you please describe your knowledge of the CBID Demonstration Model? Relevance: Health Achievement of results – System Level TRAINING 1. Have you or anyone on your team received any training from the CBID Demonstration Model? a. Please describe the types of training. b. Who received the training? Were male and female staff or staff with disabilities given equal opportunities to participate in the training? c. Was this training relevant to your work? d. Did the training help build your capacity to provide rehabilitation services to persons with disabiliteis? 2. Did you receive any training specific to provision of mental health services to persons with disabilities? a. Did this training help you deliver mental health services to persons with disabilities? b. Did the training help you know how to assist vulnerable persons who require mental health services such as women and men with disabilities and persons with disabilities who are ethnic minorities, people who are elderly or young, people who live in rural areas or people who are very poor? c. Did the training help you be able to use compassionate communication skills? i. Can you provide an example? d. Have you had an opportunity to use the knowledge you learned from the training? Please explain. ES 4-2: # of service providers trained who serve vulnerable persons 153 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE Relevance: Stakeholder Engagement Achievement of results – People Centered Achievement of results – People Centered CBID DEMONSTRATION MODEL – MENTAL HEALTH 3. In your opinion, is the CBID Demonstration Model improving access to mental health services for persons with disabilities in CBID demonstration districts? a. How so? b. Can you provide an example of a success? c. What have been some of the challenges? d. Have these improvements to mental health services for persons with disabilities benefitted vulnerable people equally including women with disabilities and persons with disabilities and who are ethnic minorities, from rural areas, elderly or young. or very poor? 4. Do you feel that the Okard Activity has improved the capacity of your health facility or organization to incorporate disability inclusion within its operations? a. How so? b. Can you provide an example of how disability inclusion has improved in your facility or organization? c. What have been some of the challenges with regards to improving disability inclusion in your facility or organization? H-1: # of service delivery systems with improved capacity to provide women￾centered care HI-2: % of persons with disabilities and their households who receive “People Centered Care” Achievement of results – System Level SUSTAINABILITY 5. In your opinion, does the contribution by the Okard Activity have the potential to be sustainable at the conclusion of the activity? USG assisted organizations and/or service delivery systems that serve vulnerable persons strengthened CONCLUSION 6. Thank-you for taking the time to speak to me today about the Okard Activity. I have finished asking all my question now. Before we finish, is there anything else you would like to share with me? Perhaps there is something that I did not cover, and you would like to add? 154 | EVALUATION FINAL REPORT USAID.GOV KII PROTOCOL: CBID DEMONSTRATION MODEL EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE INTRODUCTION To be collected from participant(s): Stakeholder Type, Location, Gender To begin, please tell me about yourself. (Go around and have everyone introduce themselves) 1. Please tell me about the composition of your CBID team. a. Are there women on the team? b. Are there persons with disabilities on the team? Achievement of results – System Level TRAINING 2. Have you received any training for your work with the CBID Demonstration Model? a. Please describe the training. b. Was the training relevant to the work you are doing with the CBID Demonstration Model? c. Did the training build your capacity to do work on the CBID Demonstration Model? d. What other types of training would help you do your work on the CBID Demonstration Model? ES 4-2: # of service providers trained who serve vulnerable persons Relevance: Stakeholder Engagement Relevance: Economic Empowerment Relevance: Health CBID DEMONSTRATION MODEL 3. How is the CBID Demonstration Model identifying persons with disabilities in the communities where you are working? a. What is working well with regards to the identification process? b. What are the challenges? c. In your opinion, does the CBID demonstration model adequately identify vulnerable people such as women with disabilities or persons who are disabled and are ethnic minorities, elderly or young, very poor or from rural areas? HI-2: % of persons with disabilities and their households who receive “People Centered Care” HI-3: # of people receiving health and related rehabilitation services HI-4: # of people who demonstrate increase in 155 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE Achievement of results – People Centered Achievement of results – Gender and Inclusion 4. How is the CBID Demonstration Model assessing the needs and function of persons with disabilities in the communities where you are working? a. How is the modular tool working? b. What is working well with regards to assessing the needs and functions of persons with disabilities c. What could be improved? 5. How does the CBID Demonstration Model prioritize the needs of persons with disabilities in the communities where you are working? a. What works well? b. What could be improved? c. Do you feel that vulnerable populations such as women, ethnic minorities, the elderly or young, the very poor or individuals living in rural areas are equally prioritized for CBID interventions? 6. In your opinion, has the CBID Demonstration Model improved access to health care for persons with disabilities in your community? a. Can you provide an example of a success? b. What have been some of the challenges? 7. In your opinion, has the CBID Demonstration model improved access to mental health services for persons with disabilities in CBID demonstration districts? a. Can you provide an example of a success? b. What have been some of the challenges? 8. Has access to assistive devices improved due to inputs from the CBID Demonstration Model in this community? a. Can you provide an example of a success? b. What have been the challenges? 9. In your opinion, has the CBID Demonstration Model contributed to improvements in the referral system for rehabilitation for persons with disabilities in your community? a. Can you provide an example of a success? b. What have been the challenges? functioning or wellbeing EI-3: # of persons with disabilities and their households with improved economic self￾sufficiency GNDR-2: % of female participants in USG assisted programs designed to increase access to productive economic resources PM 4: # of targeted vulnerable populations gaining new or better employment as a result of USG assistance 156 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE 10. In your opinion, has the CBID Demonstration Model improved opportunities for economic self￾sufficiency for persons with disabilities in your community? a. Can you provide an example of a success? b. What have been some of the challenges? 11. In your opinion does the work done by the CBID demonstration model equally benefit all persons with disabilities including women with disabilities and persons with disabilities who are ethnic minorities, elderly or young, very poor or live rurally? a. How could the CBID Demonstration Model be more inclusive of all people who are disabled? Relevance: Stakeholder Engagement SOCIAL BEHAVIOUR CHANGE COMMUNICATION 12. In your opinion, has the CBID Demonstration model changed the lives of persons with disabilities in your community? 13. In your opinion, has the CBID Demonstration Model changed how non-disabled community members think about persons with disabilities? PM1: # of targeted beneficiaries with increased knowledge and awareness Achievement of results – People Centered OVERALL IMPRESSION 14. What is the most beneficial component of the CBID demonstration model, in your opinion? 15. What could be improved with the CBID Demonstration Model? CONCLUSION 16. Thank-you for taking the time to speak to me today about the CBID Demonstration. I have finished asking all my question now. Before we finish, is there anything else you would like to share with me? Perhaps there is something that I did not cover, and you would like to add? 157 | EVALUATION FINAL REPORT USAID.GOV KII PROTOCOL: COMMUNITY MEMBERS EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE INTRODUCTION To be collected from participant(s): Stakeholder Type, Location, Gender To begin, please tell me about yourself. (Go around and have everyone introduce themselves) Relevance: Stakeholder Engagement Relevance: Economic Empowerment Relevance: Health Achievement of results – People Centered Achievement of results – Gender and Inclusion CBID DEMONSTRATION MODEL Today I would like to talk to you about the CBID Demonstration Model that is doing work in your community. 1. Can you please tell me your knowledge of the CBID Demonstration Model? 2. In your opinion, has the CBID Demonstration Model improved access to health care for persons with disabilities in your community? a. Can you provide an example of a success? b. What have been some of the challenges? 3. In your opinion, has the CBID Demonstration model improved access to mental health services for persons with disabilities in CBID demonstration districts? a. Can you provide an example of a success? b. What have been some of the challenges? 4. Has access to assistive devices improved due to inputs from the CBID Demonstration Model in this community? a. Can you provide an example of a success? b. What have been the challenges? 5. In your opinion, has the CBID Demonstration Model contributed to improvements in the referral system for rehabilitation for persons with disabilities in your community? a. Can you provide an example of a success? b. What have been the challenges? 6. In your opinion, has the CBID Demonstration Model improved opportunities for economic self￾HI-2: % of persons with disabilities and their households who receive “People Centered Care” HI-3: # of people receiving health and related rehabilitation services HI-4: # of people who demonstrate increase in functioning or wellbeing EI-3: # of persons with disabilities and their households with improved economic self￾sufficiency GNDR-2: % of female participants in USG assisted programs designed to increase access to 158 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE sufficiency for persons with disabilities in your community? a. Can you provide an example of a success? b. What have been some of the challenges? 7. Are you aware of self-help groups that have started as a result of support from the CBID Demonstration model? a. What do you know about these self-help groups? b. Do you know anyone that has attended one of these self-help groups? c. If so, how has engagement with the self￾help groups supported persons with disabilities in your community? 8. In your opinion, does the CBID demonstration model equally benefit all persons with disabilities including vulnerable populations such as women with disabilities, and persons with disabilities who are ethnic minorities, elderly or young, very poor or from rural areas? productive economic resources PM 4: # of targeted vulnerable populations gaining new or better employment as a result of USG assistance Relevance: Stakeholder Engagement SOCIAL BEHAVIOUR CHANGE COMMUNICATION 9. In your opinion, has the CBID Demonstration model changed the lives of persons with disabilities in your community? 10. In your opinion, has the CBID Demonstration Model changed how non-disabled community members think about persons with disabilities? PM1: # of targeted beneficiaries with increased knowledge and awareness Achievement of results – People Centered OVERALL IMPRESSION 11. What is the most beneficial component of the CBID demonstration model, in your opinion? 12. What could be improved with the CBID Demonstration Model? CONCLUSION 13. Thank-you for taking the time to speak to me today about the CBID Demonstration. I have finished asking all my question now. Before we finish, is there anything else you would like to share with me? Perhaps there is something that I did not cover, and you would like to add? 159 | EVALUATION FINAL REPORT USAID.GOV KII PROTOCOL: KHAM: DISTRICT, ADMINISTRATION OFFICE; XAYPHOUTHONG: ADMINISTRATION OFFICE, VICE HEAD OF OFFICE EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE INTRODUCTION To be collected from participant(s): Stakeholder Type, Location, Gender To begin, please tell me about yourself. Relevance: Stakeholder Engagement Relevance: Economic Empowerment Relevance: Health Achievement of results – People Centered Achievement of results – Gender and Inclusion CBID DEMONSTRATION MODEL Today I would like to talk to you about the CBID Demonstration Model that is doing work in this district. 1. Can you please tell me your knowledge of the CBID Demonstration Model? 2. In your opinion, has the CBID Demonstration Model improved access to health care for persons with disabilities in your community? a. Can you provide an example of a success? b. What have been some of the challenges? 3. In your opinion, has the CBID Demonstration model improved access to mental health services for persons with disabilities in CBID demonstration districts? a. Can you provide an example of a success? b. What have been some of the challenges? 4. Has access to assistive devices improved due to inputs from the CBID Demonstration Model in this community? a. Can you provide an example of a success? b. What have been the challenges? c. ? 5. In your opinion, has the CBID Demonstration Model contributed to improvements in the referral system for rehabilitation for persons with disabilities in your community? a. Can you provide an example of a success? b. What have been the challenges? HI-2: % of persons with disabilities and their households who receive “People Centered Care” HI-3: # of people receiving health and related rehabilitation services HI-4: # of people who demonstrate increase in functioning or wellbeing EI-3: # of persons with disabilities and their households with improved economic self￾sufficiency GNDR-2: % of female participants in USG assisted programs designed to increase access to productive 160 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE 6. In your opinion, has the CBID Demonstration Model improved opportunities for economic self￾sufficiency for persons with disabilities in your community? a. Can you provide an example of a success? b. What have been some of the challenges? 7. In your opinion, does the CBID Demonstration model equally benefit all persons with disabilities including vulnerable populations including women with disabilities and persons with disabilities who are ethnic minorities, elderly or young, very poor or from rural areas? economic resources PM 4: # of targeted vulnerable populations gaining new or better employment as a result of USG assistance Relevance: Stakeholder Engagement SOCIAL BEHAVIOUR CHANGE COMMUNICATION 8. In your opinion, has the CBID Demonstration model changed the lives of persons with disabilities in your community? 9. In your opinion, has the CBID Demonstration Model changed how non-disabled community members think about persons with disabilities? PM1: # of targeted beneficiaries with increased knowledge and awareness Achievement of results – People Centered OVERALL IMPRESSION 10. What is the most beneficial component of the CBID demonstration model, in your opinion? 11. What could be improved with the CBID Demonstration Model? CONCLUSION 12. Thank-you for taking the time to speak to me today about the CBID Demonstration Model. I have finished asking all my question now. Before we finish, is there anything else you would like to share with me? Perhaps there is something that I did not cover, and you would like to add? 161 | EVALUATION FINAL REPORT USAID.GOV KII PROTOCOL: KHAM: HEALTH OFFICE; XAYPHOUTHONG: HEALTH OFFICE EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE INTRODUCTION To be collected from participant(s): Stakeholder Type, Location, Gender To begin, please tell me about your involvement with the CBID Demonstration Model. 1. What is your role with the CBID Demonstration Model? 2. How long have you been involved? Relevance: Stakeholder Engagement Achievement of results – System Level DISABILITY POLICY 3. Can you tell me about your knowledge regarding disability inclusion policy in Lao PDR? I am interested to understand your awareness of the National Disability Policy Strategy Action Plan and the Disability Law. a. Were you engaged in the development or dissemination of disability inclusion policy in Lao PDR? b. To your knowledge were men or women with disabilities, or persons with disabilities who are ethnic minorities, live in rural places, are elderly or young or who are very poor included in the development or dissemination of the disability inclusion policy in Lao PDR? c. In your opinion, how well does your department coordinate with other departments or ministries within the GOL to operationalize inclusion of persons with disabilities in Lao PDR? SI-1: # of activities in the Disability Action Plan initiated and monitored by the NCDE. SI-3: # of operational criteria achieved by the MOH and NCDE strategy and action Plan coordinating bodies SI-2: # of participants involved in government led policy process who self-identify as a person with a disability Relevance: Stakeholder Engagement ENGAGEMENT 4. Can you please describe the level of engagement between your department and the CBID Demonstration Model? a. Are you satisfied with this level of engagement? 162 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE 5. What could improve the level of engagement between your department and the CBID Demonstration Model? Relevance: Stakeholder Engagement TRAINING 6. Have you or anyone on your team received any training from the CBID Demonstration Model? a. Please describe the types of training. b. Who received the training? Were male and female staff or staff with disabilities given equal opportunities to participate in the training? c. Was this training relevant to your work? d. Did the training help build your capacity to do your work? ES 4-2: # of service providers trained who serve vulnerable persons Relevance: Stakeholder Engagement Relevance: Health Achievement of results – People Centered CBID DEMONSTRATION MODEL 7. In your opinion, is the CBID Demonstration Model improving access to health care for persons with disabilities in CBID demonstration districts? a. Can you provide an example of a success? b. What have been some of the challenges? 8. In your opinion, is the CBID Demonstration Model improving access to mental health services for persons with disabilities in CBID demonstration districts? a. Can you provide an example of a success? b. What have been some of the challenges? 9. Has access to assistive devices improved due to inputs from the CBID Demonstration Model? a. Can you provide an example of a success? b. What have been the challenges? 10. In your opinion, has the CBID Demonstration Model contributed to improvements in the referral system for rehabilitation in Lao PDR? a. Can you provide an example of a success? b. What have been the challenges? 11. In your opinion, does the CBID Demonstration Model equally benefit all persons with disabilities including women with disabilities and other vulnerable populations including persons with disabilities who are ethnic minorities, elderly or HI-2: % of persons with disabilities and their households who receive “People Centered Care” EI-3: # of persons with disabilities and their households with improved economic self￾sufficiency 163 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE young, persons with disabilities who are very poor or live in rural areas? SUSTAINABILITY 12. In your opinion, does the contribution by the CBID Demonstration Model have the potential to be sustainable at the conclusion of the activity? USG assisted organizations and/or service delivery systems that serve vulnerable persons strengthened CONCLUSION 13. Thank-you for taking the time to speak to me today about the CBID Demonstration Model. I have finished asking all my question now. Before we finish, is there anything else you would like to share with me? Perhaps there is something that I did not cover, and you would like to add? 164 | EVALUATION FINAL REPORT USAID.GOV KII PROTOCOL: PROVINCIAL DEPARTMENT LABOUR AND SOCIAL WELFARE; DEPARTMENT OF HEALTH; REHABILITATION CENTRE EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE INTRODUCTION To be collected from participant(s): Stakeholder Type, Location, Gender To begin, please tell me about your involvement with the CBID Demonstration Model. 1. What is your role with the project? 2. How long have you been involved? Relevance: Stakeholder Engagement Achievement of results – System Level DISABILITY POLICY 3. Can you tell me about your knowledge regarding disability inclusion policy in Lao PDR? I am interested to understand your awareness of the National Disability Policy Strategy Action Plan and the Disability Law. a. Were you engaged in the development or dissemination of disability inclusion policy in Lao PDR? b. To your knowledge were men or women with disabilities or disabled persons who are ethnic minorities, persons who live in rural places, persons who are elderly or young or persons who are very poor included in the development of dissemination of disability inclusion policy in Lao PDR? c. In your opinion, how well does your department coordinate with other departments or ministries within the GOL to operationalize inclusion of persons with disabilities in Lao PDR? SI-1: # of activities in the Disability Action Plan initiated and monitored by the NCDE. SI-3: # of operational criteria achieved by the MOH and NCDE strategy and action Plan coordinating bodies SI-2: # of participants involved in government led policy process who self-identify as a person with a disability Relevance: Stakeholder Engagement ENGAGEMENT 4. Can you please describe the level of engagement between your department and the CBID Demonstration Model? a. Are you satisfied with this level of engagement? 165 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE 5. What could improve the level of engagement between your department and the CBID Demonstration Model? Relevance: Stakeholder Engagement TRAINING 6. Have you or anyone on your team received any training from the CBID Demonstration Model? a. Please describe the types of training. b. Who received the training? Were male and female staff or staff with disabilities given equal opportunities to participate in the training? c. Was this training relevant to your work? d. Did the training help build your capacity to do your work? ES 4-2: # of service providers trained who serve vulnerable persons Relevance: Stakeholder Engagement Achievement of results – People Centered CBID DEMONSTRATION MODEL 7. In your opinion, is the CBID Demonstration Model improving access to health care for persons with disabilities in CBID demonstration districts? a. Can you provide an example of a success? b. What have been some of the challenges? 8. In your opinion, is the CBID Demonstration Model improving access to mental health services for persons with disabilities in CBID demonstration districts? a. Can you provide an example of a success? b. What have been some of the challenges? 9. Has access to assistive devices improved due to inputs from the CBID Demonstration Model? a. Can you provide an example of a success? b. What have been the challenges? 10. In your opinion, has the CBID Demonstration Model contributed to improvements in the referral system for rehabilitation in Lao PDR? a. Can you provide an example of a success? b. What have been the challenges? 11. In your opinion, is the CBID Demonstration Model improving opportunities for economic self￾sufficiency for persons with disabilities in CBID demonstration districts? a. Can you provide and example of a success? b. What have been some of the challenges? 12. In your opinion, is the CBID Demonstration Model equally benefiting all persons with disabilities including vulnerable populations such as women HI-2: % of persons with disabilities and their households who receive “People Centered Care” EI-3: # of persons with disabilities and their households with improved economic self￾sufficiency 166 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE with disabilities or persons with disabilities who are ethnic minorities, elderly or young, live in rural areas or are very poor? SUSTAINABILITY 13. In your opinion, does the contribution by the CBID Demonstration Model have the potential to be sustainable at the conclusion of the activity? USG assisted organizations and/or service delivery systems that serve vulnerable persons strengthened CONCLUSION 14. Thank-you for taking the time to speak to me today about the CBID Demonstration Model. I have finished asking all my question now. Before we finish, is there anything else you would like to share with me? Perhaps there is something that I did not cover, and you would like to add? 167 | EVALUATION FINAL REPORT USAID.GOV KII PROTOCOL: VILLAGE LEADERS EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE INTRODUCTION To be collected from participant(s): Stakeholder Type, Location, Gender To begin, please tell me about yourself. Relevance: Stakeholder Engagement Relevance: Economic Empowerment Relevance: Health Achievement of results – People Centered Achievement of results – Gender and Inclusion CBID DEMONSTRATION MODEL Today I would like to talk to you about the CBID Demonstration Model that is doing work in your community. 1. Can you please tell me your knowledge of the CBID Demonstration Model? 2. In your opinion, has the CBID Demonstration Model improved access to health care for persons with disabilities in your community? a. Can you provide an example of a success? b. What have been some of the challenges? 3. In your opinion, has the CBID Demonstration model improved access to mental health services for persons with disabilities in CBID demonstration districts? a. Can you provide an example of a success? b. What have been some of the challenges? 4. Has access to assistive devices improved due to inputs from the CBID Demonstration Model in this community? a. Can you provide an example of a success? b. What have been the challenges? c. ? 5. In your opinion, has the CBID Demonstration Model contributed to improvements in the referral system for rehabilitation for persons with disabilities in your community? a. Can you provide an example of a success? b. What have been the challenges? HI-2: % of persons with disabilities and their households who receive “People Centered Care” HI-3: # of people receiving health and related rehabilitation services HI-4: # of people who demonstrate increase in functioning or wellbeing EI-3: # of persons with disabilities and their households with improved economic self￾sufficiency GNDR-2: % of female participants in USG assisted programs designed to increase access to productive 168 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE 6. In your opinion, has the CBID Demonstration Model improved opportunities for economic self￾sufficiency for persons with disabilities in your community? a. Can you provide an example of a success? b. What have been some of the challenges? 7. In your opinion, does the CBID Demonstration model equally benefit all persons with disabilities including vulnerable populations including women with disabilities and persons with disabilities who are ethnic minorities, elderly or young, very poor or from rural areas? economic resources PM 4: # of targeted vulnerable populations gaining new or better employment as a result of USG assistance Relevance: Stakeholder Engagement SOCIAL BEHAVIOUR CHANGE COMMUNICATION 8. In your opinion, has the CBID Demonstration model changed the lives of persons with disabilities in your community? 9. In your opinion, has the CBID Demonstration Model changed how non-disabled community members think about persons with disabilities? PM1: # of targeted beneficiaries with increased knowledge and awareness Achievement of results – People Centered OVERALL IMPRESSION 10. What is the most beneficial component of the CBID demonstration model, in your opinion? 11. What could be improved with the CBID Demonstration Model? CONCLUSION 12. Thank-you for taking the time to speak to me today about the CBID Demonstration Model. I have finished asking all my question now. Before we finish, is there anything else you would like to share with me? Perhaps there is something that I did not cover, and you would like to add? 169 | EVALUATION FINAL REPORT USAID.GOV KII PROTOCOL: WOMEN, MEN, AND CAREGIVERS OF PERSONS WITH DISABILITIES EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE INTRODUCTION To be collected from participant(s): Stakeholder Type, Location, Gender To begin, please tell me about yourself. (Go around and have everyone introduce themselves) Relevance: Stakeholder Engagement Relevance: Economic Empowerment Relevance: Health Achievement of results – People Centered Achievement of results – Gender and Inclusion CBID DEMONSTRATION MODEL Today I would like to talk to you about the CBID Demonstration Model that is doing work in your community. 14. Can you please tell me about the process that the CBID Demonstration model used to identify persons with disabilities in your community? a. Do you feel that the process to identify persons with disabilities in your community was comprehensive? 15. What has the CBID Demonstration Model done in your community to improve access to health care for persons with disabilities in your community? a. Can you provide an example of a success? b. What have been some of the challenges? 16. In your opinion, has the CBID Demonstration model improved access to mental health services for persons with disabilities in CBID demonstration districts? a. Can you provide an example of a success? b. What have been some of the challenges? 17. Has access to assistive devices for persons with disabilities improved due to inputs from the CBID Demonstration Model in this community? a. Can you provide an example of a success? b. What have been the challenges? 18. In your opinion, has the CBID Demonstration Model contributed to improvements in the referral HI-2: % of persons with disabilities and their households who receive “People Centered Care” HI-3: # of people receiving health and related rehabilitation services HI-4: # of people who demonstrate increase in functioning or wellbeing EI-3: # of persons with disabilities and their households with improved economic self￾sufficiency GNDR-2: % of female participants in USG assisted programs designed to increase access to 170 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE system for rehabilitation for persons with disabilities in your community? a. Can you provide an example of a success? b. What have been the challenges? 19. In your opinion, has the CBID Demonstration Model improved opportunities for economic self￾sufficiency for persons with disabilities in your community? a. Can you provide an example of a success? b. What have been some of the challenges? 20. Are you aware of self-help groups that have started with support from the Okard Activity? a. If yes, can you tell me about these self-help groups? b. Do these self-help groups help persons with disabilities? If so, how? c. Are you personally involved, or do you know someone who is personally involved in a CBID Demonstration model self-help group? i. What has the experience been like for you or the person you know? 21. In your opinion, has the CBID demonstration model assisted children with disabilities in this community? d. If yes, can you tell me about how children have been assisted by the CBID demonstration model? e. Has support from the CBID demonstration model sufficiently helped children with disabilities in this community? 22. In your opinion, does the CBID demonstration model equally benefit all persons with disabilities including vulnerable populations such as women with disabilities or persons with disabilities who are ethnic minorities, elderly or young, very poor or from rural areas? productive economic resources PM 4: # of targeted vulnerable populations gaining new or better employment as a result of USG assistance Relevance: Stakeholder Engagement SOCIAL BEHAVIOUR CHANGE COMMUNICATION 23. In your opinion, has the CBID Demonstration model changed the lives of persons with disabilities in your community? PM1: # of targeted beneficiaries with increased 171 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE 24. In your opinion, has the CBID Demonstration Model changed how non-disabled community members think about persons with disabilities? knowledge and awareness Achievement of results – People Centered OVERALL IMPRESSION 25. What is the most beneficial component of the CBID demonstration model, in your opinion? 26. What could be improved with the CBID Demonstration Model? CONCLUSION 27. Thank you for taking the time to speak to me today about the CBID Demonstration Model. I have finished asking all my question now. Before we finish, is there anything else you would like to share with me? Perhaps there is something that I did not cover, and you would like to add? 172 | EVALUATION FINAL REPORT USAID.GOV KII PROTOCOL: KHAM: DISTRICT DEPARTMENT OF LABOUR AND SOCIAL WELFARE AGRICULTURE OFFICE; XAYPHOUTHONG: AGRICULTURAL OFFICE EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE INTRODUCTION To be collected from participant(s): Stakeholder Type, Location, Gender To begin, please tell me about your involvement with the CBID Demonstration Model. 1. What is your role with the CBID Demonstration Model? 2. How long have you been involved? Relevance: Stakeholder Engagement Achievement of results – System Level DISABILITY POLICY 3. Can you tell me about your knowledge regarding disability inclusion policy in Lao PDR? I am interested to understand your awareness of the National Disability Policy Strategy Action Plan and the Disability Law. a. Were you engaged in the development or dissemination of disability inclusion policy in Lao PDR? b. To your knowledge were men or women with disabilities, or persons with disabilities who are ethnic minorities, live in rural places, are elderly or young or who are very poor included in the development or dissemination of the disability inclusion policy in Lao PDR? c. In your opinion, how well does your department coordinate with other departments or ministries within the GOL to operationalize inclusion of persons with disabilities in Lao PDR? SI-1: # of activities in the Disability Action Plan initiated and monitored by the NCDE. SI-3: # of operational criteria achieved by the MOH and NCDE strategy and action Plan coordinating bodies SI-2: # of participants involved in government led policy process who self-identify as a person with a disability Relevance: Stakeholder Engagement ENGAGEMENT 4. Can you please describe the level of engagement between your department and the CBID Demonstration Model? 173 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE a. Are you satisfied with this level of engagement? 5. What could improve the level of engagement between your department and the CBID Demonstration Model? Relevance: Stakeholder Engagement TRAINING 6. Have you or anyone on your team received any training from the CBID Demonstration Model? a. Please describe the types of training. b. Who received the training? Were male and female staff or staff with disabilities given equal opportunities to participate in the training? c. Was this training relevant to your work? d. Did the training help build your capacity to do your work? ES 4-2: # of service providers trained who serve vulnerable persons Relevance: Stakeholder Engagement Relevance: Economic Empowerment Achievement of results – People Centered CBID DEMONSTRATION MODEL 7. In your opinion, is the CBID Demonstration Model improving opportunities for economic self￾sufficiency for persons with disabilities in CBID demonstration districts? 8. Has the CBID Demonstration Model helped persons with disabilities and/or their households get additional job-related training? a. If yes, what kinds of training did these people receive? b. Did this training help them get better employment? c. If no, is there a plan for employment related training within the CBID demonstration model? Please explain. 9. What have been some of the challenges related to improving economic self-sufficiency for persons with disabilities and/or their households through the CBID demonstration model? 10. Are there any successes that you are aware of with regards to the CBID Demonstration Model work towards improving economic self-sufficiency for persons with disabilities and/or their households? 11. What could the CBID Demonstration model do better to improve economic self-sufficiency for persons with disabilities and/or their households in this district? I would like to ask you about efforts by the CBID Demonstration model to help persons with disabilities EI-3: # of persons with disabilities and their households with improved economic self￾sufficiency 174 | EVALUATION FINAL REPORT USAID.GOV EVALUATION QUESTION QUESTIONS / SUB QUESTIONS INDICATOR REFERENCE engage in economic activities involving agriculture. (note to interviewer, the respondent may have addressed some of these questions above. Please check and fill in our knowledge but only asking questions not yet asked). 12. Specifically, what work has the CBID Demonstration model done to support persons with disabilities to engage in agricultural economic activities? a. Up to now has this work been successful? b. What have been some of the challenges? c. What have been some of the successes? d. What needs to be done to improve this component of the CBID Demonstration model? 13. In your opinion, does the CBID Demonstration Model equally benefit all persons with disabilities including women with disabilities and other vulnerable populations including persons with disabilities who are ethnic minorities, elderly or young, persons with disabilities who are very poor or live in rural areas? SUSTAINABILITY 14. In your opinion, does the work being done by the CBID Demonstration Model have the potential to be sustainable at the conclusion of the activity? USG assisted organizations and/or service delivery systems that serve vulnerable persons strengthened CONCLUSION 15. Thank-you for taking the time to speak to me today about the CBID Demonstration Model. I have finished asking all my question now. Before we finish, is there anything else you would like to share with me? Perhaps there is something that I did not cover, and you would like to add? 175 | EVALUATION FINAL REPORT USAID.GOV ANNEX 6: SCOPE OF WORK Service Request Technical Proposal SR 1.02 – USAID Okard Mid-term Evaluation Under “Monitoring, Evaluation and Learning Support to USAID Laos Country Office, Project, and Activities” (USAID/Laos Country Office MEL Support). Task Order No. 72048619F00007, Contract No. AID-486-I-14-00001 I. PURPOSE The purpose of the USAID Okard Mid-term Evaluation (MTE) is to review Activity progress and systems to date with the objectives of guiding strategic direction and planning for the remainder of the Activity to capitalize on the approaches that are working and improve the ones that are not. The specific objectives of the MTE are: • Objective 1: Critically assess the progress of the USAID Okard Activity from inception to date and analyze against expected results to identify potential adjustments to the strategy and interventions/services. • Objective 2: Assess effectiveness of the Activity’s management arrangements and partnerships; identify advantages, bottlenecks, and lessons learned to identify potential adjustments. The MTE is a requirement of the USAID Okard Cooperative Agreement No. AID-486-A-17-00004. This MTE will serve as a ‘pause and reflect’ opportunity and help USAID Okard to reflect its progress and guide its mid-term re-design workshop. II. BACKGROUND The USAID Okard (ໂອກາດ) Activity is guided by the philosophy of Disability Inclusive Development (DID) and the belief that no one should be left behind. USAID Okard is $15 million-dollar Activity implemented by World Education, Inc. (WEI) in Lao People’s Democratic Republic (PDR) from October 2017 to September 2022. USAID Okard’s goal is to improve and sustain the independent living and functional ability of persons with disabilities, regardless of factors such as age, sex, gender expression, ethnicity, and their households in Lao PDR. Activity interventions are based in XKH, Savannakhet Province (SVK), and Vientiane Capital (VTE). The Activity interventions are designed across three components and two tiers. The two-tiered approach integrates systems level (Tier 1) and individual/community level (Tier 2) interventions under three components of Health (Component 1), Economic Empowerment (Component 2), and Stakeholder Engagement (Component 3). The approach and more specifically the interventions of USAID Okard stem directly from the GoL National Disability Policy, Strategy and Action Plan and the National Rehabilitation Medicine Strategy and Action Plan. In addition to the three components, USAID Okard is piloting an overarching component called Community Based Inclusive Development (CBID) model that tests the cumulative effects on community disability inclusion and on optimal functioning and participation of persons with disabilities, when the three components are implemented simultaneously and cohesively in two pilot districts under the facilitation of a civil society organization. This component is based in one district of XKH and one district of SVK provinces. 176 | EVALUATION FINAL REPORT USAID.GOV The Activity faced initial delays in the signing of the memorandum of understanding (MOU) and sub￾agreement on specific scope of work with the GoL. Moreover, the COVID-19 pandemic has severely impacted the availability of government partners in implementing their scope of work related to Tier 1(system level). Evaluation Questions The evaluation questions for the MTE are as follows: Objective 1: • To what extent have the USAID Okard strategy and interventions been relevant to the goals it intends to reach? o Are the context and critical assumptions still valid? What have been key changes, if any, that affect the anticipated achievements positively and/or negatively? • What components of the Activity have been successfully and/or unsuccessfully implemented according to the approach? Why? • To what extent has USAID Okard been effective in achieving its expected results, particularly in terms of: o Building/strengthening the systems as described in the design document (system￾centered); o Reaching target beneficiaries. Which groups are underserved, if any, among all targeted groups, and why? (people-centered) o Gender equality and other inclusion? Objective 2: • Does USAID Okard have the appropriate management system and partnership structures to accomplish its objectives? • Do USAID Okard partners have the right systems and capacity to fulfill their scope of work? What systems and/or capacities are currently missing in order to fulfill the scope of work? III. APPROACH To address the above evaluation questions, Social Impact (SI) will lead a mid-term performance evaluation consisting of three phases: Start-up & Planning, Data Collection & Analysis, and Reporting & Learning. Given the interest in using this exercise for action-oriented future planning, learning and strategy, SI will work collaboratively with USAID and implementing partner (IP), World Education, through all phases of the evaluation. This will consist of holding regular check-ins with primary points of contact as agreed upon during the start-up phase and structuring the team’s deliverables to provide in-depth opportunity for the IP to engage with the evaluation’s findings through participatory workshops (see outline of workshops in Phase III below). Given the limited implementation to-date, the MTE will pay particular attention to how well the activity’s critical assumptions have held, in addition to how well the implementing partner has executed its implementation plan. Through this process, the team will be able to point to any necessary adjustments 177 | EVALUATION FINAL REPORT USAID.GOV needed to improve the theory of change and any weaknesses in implementation needed to ensure success in later years of the activity timeline. a. Methodology Phase 1 – Start-up & Planning The ET will launch the assessment with a kick-off meeting with USAID and the IP to further discuss stakeholder priorities and clarify the scope of work. Following this initial consultation, the ET will conduct the initial desk review and produce a Design Report detailing the team’s full methodology, proposed key informants, implementation schedule, and data collection tools. The Design Report will also outline roles and responsibilities of each involved party (SI, IP, USAID) and take into consideration the ethical and confidentiality issues and the context of the COVID-19 pandemic. Upon submission of the draft Design Report, the ET will present the design to USAID and the IP in an inbrief presentation. The inbrief provides an opportunity for question and answer with the ET about the Design Report and a collaborative discussion with all parties to finalize the details of the design and implementation plan. Phase II – Data Collection & Analysis The USAID Okard MTE will be conducted in Vientiane Capital, Savannakhet, and Xieng Khouang Provinces, with data collection among implementation partners including GoL Officials in central, provincial and district level, and non-government partners including Non-profit associations (NPAs) and private sector. The ET will rely on multiple methods to collect and analyze data that will inform their response to the evaluation questions and help situate findings in the country and activity context. The ET will first conduct a desk review of key project documents including workplans, annual and quarterly reports, the Activity Monitoring, Evaluation and Learning Plan, and supplementary reports as available. Based on the preliminary desk review and consultations, the ET will develop a Design Report detailing a full description of methods. Methods will include an analysis of secondary project data, including CBID case management data and USAID indicator data. The ET will collect additional qualitative data using methods such as key informant interviews, focus group discussions, and case studies (to be proposed in the Design Report). To analyze qualitative data, the ET will first identify and organize key themes in processed and coded data and then employ techniques such as comparative analysis to make sense of the data. Quantitative analysis is expected to consist of simple descriptive techniques to understand project data. Data analysis will focus on identifying findings related to the key evaluation questions. Phase III – Reporting & Learning Based on their analysis of data collected in Phase II, the ET will produce two key deliverables to communicate their findings, conclusions and recommendations for the remainder of the activity. The first deliverable will be a PowerPoint presentation which summarizes all findings and conclusions for each EQ and presents a set of proposed recommendations. The ET will deliver this presentation to stakeholders with the purpose of eliciting any clarifying information, correcting any factual errors and co-developing a final set of relevant, realistic recommendations that respond to the evaluation results. Based on USAID 178 | EVALUATION FINAL REPORT USAID.GOV and IP preference, SI proposes conducting the presentation in two sessions, with the first focused on validation of findings and the second focused on crafting recommendations. The audience for both sessions may include USAID land conflict resolution project, USAID Okard Executive Committee, USAID Okard Advisory Committee, among others as identified during the evaluation. The ET will prepare the second deliverable, the mid-term draft evaluation report, based on the analysis conducted and the adjustments and recommendations generated from the presentation. The draft report will be organized by evaluation question and will separately present evaluation findings (facts and evidence) along with the ET’s conclusions. The final section of the report will outline and describe action-oriented recommendations that speak to stakeholder’s needs and address the findings and conclusions presented. Upon receiving consolidated comments from USAID and the IP, the ET will review and adjust the draft report accordingly, providing a detailed response for each submitted comment. b. Team Composition The ET will consist of a Team Lead, a Local Evaluation Specialist, a Local Researcher/Technical Specialist, two notetakers, and an interpreter that can join the team as needed to provide interpretation from minority languages to Lao for the local research team or from Lao to English for the Team Leader. The ET will receive support from the SI management team (Project Director, Project Manager, and Project Assistant), who will provide quality assurance and management support throughout each phase of the assessment. SI will also enlist the support of a GESI Advisor to conduct targeted quality assurance reviews and support the team at key phases in the design and reporting to ensure the assessment meets USAID standards for GESI. IV. DELIVERABLES & TIMELINE The following table outlines the anticipated schedule of activities and deliverables for this service request. The final detailed timeline will be included in the Design Report after consultation with IP and USAID schedules during the kick-off meeting. Dates [Lao PDR] Activity or Deliverable February 23, 2021 Kick-off Meeting March 9, 2021 Design Report Submission March 11, 2021 Inbrief Presentation March 15-April 9, 2021 Data Collection TBD o/a April 26-28, 2021 Findings Validation & Recommendations Workshops May 14, 2021 Draft Report Submission May 28, 2021 USAID & IP Return Report Comments 179 | EVALUATION FINAL REPORT USAID.GOV Dates [Lao PDR] Activity or Deliverable June 11, 2021 Final Report Submission I. PERSONNEL The following personnel are proposed to complete the Service Request activities listed above: a. Team Lead – Denise Buchner Ms. Denise Buchner is a global health and development professional with over 15 years of experience in evaluation and research specializing in disability studies, health systems strengthening, and maternal, newborn, and child health. She has expertise in the management of monitoring, evaluation and research projects, including large and complex mixed-methods studies with donors such as the World Health Organization, USAID, and United Nation’s Children Fund. In addition to leading a wide range of health and disability-related evaluations around the world, Ms. Buchner is also familiar with disability in the Lao context. In 2007 she conducted an ethnographic study of social and cultural interpretations of illness and disability in Lao PDR. Her PhD thesis topic (2011) was also on disability in Lao PDR. Ms. Buchner holds a PhD in Global Health Research from the University of Calgary, Canada. b. Local Evaluation Specialist – Salika Khoonbarthao Ms. Salika Khoonbarthao is an experienced monitoring, evaluation, and data management specialist with over 10 years of experience conducting qualitative and quantitative field research. Ms. Khoonbarthao has served as fieldwork leader on numerous evaluations and specializes in the training of qualitative research teams. She has also led surveys for disability sector programs and focused on special needs education while getting her Master’s degree in Education and Early Childhood Studies at the University of Huddersfield in the United Kingdom. c. Local Technical Specialist – TBD The team is still recruiting for the second researcher position, which will ideally be filled by someone with a background in disability-related programming. The following SI staff are proposed to provide quality assurance and management oversight of the Assessment: a. Project Director – Catherine Villada b. Project Manager – TBD c. Project Assistant – Anderson Lanham d. GESI Advisor – Leah Ghoston 180 | EVALUATION FINAL REPORT USAID.GOV Level of Effort table Team Member Total Level of Effort Team Lead 48 Local Evaluation Specialist 32 Local Technical Specialist 32 Notetaker 1 18 Notetaker 2 18 Interpreter 15