MIDTERM EVALUATION OF THE PROGRAM TO STRENGTHEN SELF RELIANCE AND RESILIENCE OF TIBETAN COMMUNITIES IN SOUTH ASIA Final Evaluation Report February 2020 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Social Impact, Inc. The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. MIDTERM EVALUATION OF THE PROGRAM TO STRENGTHEN SELF RELIANCE AND RESILIENCE OF TIBETAN COMMUNITIES IN SOUTH ASIA FINAL EVALUATION REPORT February 2020 Asia Learning, Monitoring, and Evaluation (ALME) Activity Tasking TO50 AID-486-I-14-00001 / 72038619F00006 i | TASKING N050: TSRR EVALUATION USAID.GOV ABSTRACT This report covers the Midterm Performance Evaluation of the Program to Strengthen Self Reliance and Resilience of Tibetan Communities in South Asia (TSRR). TSRR is a five-year, $23 million activity funded by USAID aimed at improving the livelihoods of Tibetan refuges in India and Nepal through 11 Key Result Areas (KRAs) spanning livelihoods development, health, agriculture, education, capacity development, and arts and culture. The evaluation conducted by Social Impact near the start of the program’s forth year sought to understand 1) TSRR’s contribution to improved livelihoods; 2) the adoption of systems and processes introduced for increasing institutional capacity; 3) the effectiveness of the integrated multi-sector approach; and 4) potential areas of improvement for the program moving forward. To respond to these questions, SI undertook a participatory, mixed-methods evaluation that included desk review, key informant interviews (KIIs), focus group discussions (FGDs), and a quantitative survey in six Tibetan settlements in India and Nepal in November 2019. The evaluation concluded that TSRR made good progress across indicators in the first three years including improving livelihoods through increased access to finance, bolstering education systems, providing health insurance and services, and strengthening institutional capacity and cultural preservation. Despite covering a range of services, the evaluation noted that the program could improve targeting and integration of services by strengthening community participation and input. USAID.GOV TASKING N050: TSRR EVALUATION | ii TABLE OF CONTENTS EXECUTIVE SUMMARY................................................................................................................................................. IV METHODOLOGY ........................................................................................................................................................ IV FINDINGS ....................................................................................................................................................................... IV CONCLUSIONS .......................................................................................................................................................... IX RECOMMENDATIONS............................................................................................................................................... X 1. EVALUATION PURPOSE & EVALUATION QUESTIONS ............................................................................... 1 1.1 EVALUATION PURPOSE.................................................................................................................................. 2 1.2 EVALUATION QUESTIONS............................................................................................................................ 2 1.3 PROGRAM BACKGROUND........................................................................................................................... 3 2. EVALUATION METHODS & LIMITATIONS ........................................................................................................ 5 2.1 EVALUATION DESIGN AND METHODOLOGY..................................................................................... 5 3. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS.......................................................................... 9 3.1 FINDINGS ............................................................................................................................................................. 9 3.2 CONCLUSIONS................................................................................................................................................ 35 4. RECOMMENDATIONS ............................................................................................................................................ 40 4.1 EVALUATION QUESTION 1 RECOMMENDATIONS .......................................................................... 40 4.2 EVALUATION QUESTION 2 RECOMMENDATIONS .......................................................................... 41 4.3 EVALUATION QUESTION 3 RECOMMENDATIONS .......................................................................... 41 4.4 EVALUATION QUESTION 4 RECOMMENDATION............................................................................ 41 ANNEXES ......................................................................................................................................................................... 42 ANNEX I: EVALUATION STATEMENT OF WORK......................................................................................... 42 ANNEX II: EVALUATION DESIGN AND METHODOLOGY........................................................................ 48 ANNEX III: DATA COLLECTION INSTRUMENTS........................................................................................... 55 KII – IMPLEMENTING PARTNERS (INCLUDING LOCAL PARTNER NGOS) ......................................... 55 KII – CENTRAL TIBETAN ADMINISTRATION.................................................................................................. 58 KII – COMMUNITY LEADERS ................................................................................................................................. 61 KII – TED BENEFICIARIES (ENTREPRENEURS, BUSINESS OWNERS)........................................................ 64 KII – BENEFICIARIES ................................................................................................................................................... 67 ANNEX IV: SOURCES OF INFORMATION........................................................................................................ 76 ANNEX V: TSRR SETTLEMENTS IN INDIA AND NEPAL .............................................................................. 78 TABLE OF TABLES Table 1: Evaluation Questions........................................................................................................................................ 2 Table 2: Number of TSRR Communities by Community Type.............................................................................. 4 Table 3: Sampling Strategy by Data Collection Type and Location....................................................................... 6 Table 4: Qualitative Respondents by Type and Sex................................................................................................ 50 TABLE OF FIGURES Figure 1: Case Study - Micro Loan to Female Entrepreneur Enhances her Livelihood.................................. 12 Figure 2: Respondent's TSRR Model........................................................................................................................... 32 Figure 3: Locations of the TSRR Midterm Evaluation Sites in India and Nepal ................................................ 50 Figure 4: Percentage of Survey Respondents by Settlement Visited................................................................... 52 Figure 5: Survey Respondents' Household Income Levels..................................................................................... 52 Figure 6: Percentage of Survey Respondents by Age Group ................................................................................ 52 iii | TASKING N050: TSRR EVALUATION USAID.GOV ACRONYMS CCOCC Comprehensive Community Outreach and Coordinated Care CMRF Community-Managed Revolving Fund CTA Central Tibetan Administration DTCC Dharamsala Tibetan Career Center EGR Early Grade Reading EQ Evaluation Question ET Evaluation Team FGD Focus Group Discussion GBV Gender-Based Violence HBLCC Himalayan Buddhist Library and Cultural Center HIPA Himachal Institute of Public Administration HSYWE Himalayan Society for Youth and Women’s Empowerment KII Key Informant Interview KRA Key Result Area LWF Lodrik Welfare Fund MSME Micro, Small, or Medium Enterprise NBFC Non-Banking Financial Company NGO Non-Governmental Organization PNA Participatory Needs Assessment QPR Quarterly Progress Report SARD Social Resource Development Fund SBCC Social and Behaviorial Change Communications SI Social Impact SLF Snow Lion Foundation SO Strategic Objective TED Tibetan Entrepreneurship Development TIPA Tibetan Institute of Performing Arts TMS Tibetan Medicare System TOT Training of Trainers TSRR Tibetan Self Reliance and Resilience TTF The Tibet Fund US United States USAID United States Agency for International Development USAID.GOV TASKING N050: TSRR EVALUATION | iv EXECUTIVE SUMMARY The Program to Strengthen Self Reliance and Resilience of Tibetan Communities in South Asia (TSRR) is a five￾year, $23 million activity implemented by the Tibet Fund (TTF), working with the Central Tibetan Administration (CTA) in India and local implementing partners the Snow Lion Foundation (SLF), the Lodrik Welfare Fund (LWF), and the Himalayan Society for Youth and Women’s Empowerment (HSYWE), in Nepal. TSRR is multi-sectoral in nature, with 11 Key Result Areas (KRAs) spanning livelihoods development, health, agriculture, education, capacity development, and arts and culture. As TSRR enters its fourth year of implementation, the United States Agency for International Development (USAID) contracted Social Impact (SI) to conduct a midterm evaluation of the activity to: 1) assess the extent to which TSRR is contributing to improved livelihoods for Tibetan refugee communities in Nepal and India; 2) understand how effectively the systems and processes being introduced to Tibetan refugee communities for increasing their institutional capacity are being adopted; 3) understand if the integrated multi-sectoral approach is an effective model for TSRR’s theory of change; and 4) identify any aspects of TSRR that might need to be changed to maximize the likelihood of achieving planned outcomes. The audience for the evaluation report includes USAID/India, TTF and its local implementing partners, the CTA at the central and local levels, as well as TSRR beneficiaries and other community members invested in the activity. Findings are intended to inform implementation and management decisions over the remaining lifetime of TSRR, as well as other USAID programs of a similar nature. METHODOLOGY To respond to these questions, SI undertook a participatory, mixed-methods evaluation that included 43 key informant interviews (KIIs), six focus group discussions (FGDs), and a quantitative survey with 146 community members in six settlements during a 3-week field work period that took place from November 4–25, 2019. The evaluation was conducted in four phases, including desktop document review and work plan development, in-country data collection, analysis, and utilization/dissemination of findings. Field work was conducted in four communities in India: Dharamsala, Dehradun, Bylakuppe, and Hunsur, as well as two locations in Nepal: Kathmandu and Pokhara. Key stakeholder groups in the qualitative data collection include CTA officials at the central and local levels, TSRR beneficiaries from each of the various KRAs, and community leaders who have previously been engaged in supporting TSRR’s work. FINDINGS Evaluation Question (EQ)1: To what extent have the activities implemented so far contributed to improved livelihoods of the Tibetan refugee community? Within TSRR’s multi-sector approach, eight of the 11 KRAs1 target improved livelihoods: entrepreneurship, micro, and small enterprise development (KRA 1), workforce development (KRA 2), agriculture (KRA 3), basic education (KRA 4), higher education (KRA 5), community health (KRA 6), participatory community development (KRA 7), and women’s empowerment (KRA 8). The following section breaks down the evaluation’s findings by each of these KRAs. 1 Two KRAs target capacity building and are addressed under EQ2. v | TASKING N050: TSRR EVALUATION USAID.GOV KRA 1: ENTREPRENEURSHIP, MICRO, AND SMALL ENTERPRISE DEVELOPMENT Finding 1.1: Business development support (trainings) under Tibetan Entrepreneurship Development (TED) received mixed results as to their targeting and efficacy. Finding 1.2: Loans and access to financial services were the most-often cited benefit by KRA 1 stakeholders and noted future need of the activity. However, beneficiaries desire greater flexibility in loan terms, amounts, and timing of disbursal. Finding 1.3: TSRR has the potential to enhance integration at the community level as some are concerned benefits are too heavily concentrated among individuals (or households). Respondents had mixed reactions to the targeting of youth under 35 years old for support. While some saw this as a benefit in offering expanded livelihoods support for young people and potentially reducing migration, others were concerned that older community members who also needed support were not receiving it. Finding 1.4: Two innovations, the community-managed revolving funds (CMRF) and self-help groups, were noted as particularly successful, but appear to be concentrated in Nepal. These innovations have a lot of potential for growing businesses, and their ability to reach the “unbanked” population. In India, the group-based approach has not been given enough emphasis. The group-based approach could provide opportunity moving forward, including supporting village savings and loans activities, CMRF, and enhanced cooperative models for livelihoods activities at the community level. Finding 1.5: Despite TSRR’s stated approach of giving priority to female beneficiaries for micro, small and medium enterprises (MSME) grants, especially in winter sweater businesses and for economically disadvantaged applicants, the evaluation team (ET) observed uneven, non-standardized, implementation of this approach. Based on feedback provided from settlement officers and other CTA staff, beneficiary selection and conferment of priority status was given in some communities but not in others. It is important to note that very few respondents reported a perceived need to continue targeting women, as they did not feel that women were disadvantaged and therefore not a priority target (please see additional discussion of gender dynamics under Finding 8.3). KRA 2: WORKFORCE DEVELOPMENT Finding 2.1: KRA 2 is less developed than some other KRAs and has challenges in demonstrating its outcomes due to TSRR’s limited capacity to conduct follow-up and measure its impact. Finding 2.2: KRA 2 offers an area of potential growth and investment for TSRR based on respondents’ feedback that students who do not excel in their secondary studies generally pursue vocational education opportunities and that older generations with limited education feel they lack skills to enhance their livelihood opportunities. KRA 3: AGRICULTURE Finding 3.1: Technical assistance and provision of inputs were provided to increase farming yields and promote crop diversity, but a lack of integrated planning hindered effective utilization. Finding 3.2: Trainings, while increasing general knowledge, lack specificity and follow-up to enhance their impact. Farmers expressed challenges and limited support in implementing the new knowledge. Despite interest, the farmers could not get specific information about other crops like black pepper and coffee and/or crops that are not water-intensive. They expressed a need for additional follow-up, such USAID.GOV TASKING N050: TSRR EVALUATION | vi as mentoring support on use of correct technique to grow crops or the written materials to help them retain the information shared during the training. Finding 3.3: Agriculture is not a primary livelihood source for most respondents, with interest waning over time and among younger community members. KRA 4. BASIC EDUCATION STRENGTHENING Finding 4.1: Improvements in education in Tibetan schools, particularly reading, have been attributed to TSRR’s success in delivering teacher training, Tibetan reading materials, and infrastructure improvements such as libraries. Finding 4.2: Formal education presents an opportunity to share and enrich Tibetan culture, particularly given the emphasis on Tibetan language materials. Finding 4.3: The role of education extends beyond the classroom, including parental support for learning and counseling for students from difficult backgrounds to improve performance and retention. KRA 5. HIGHER EDUCATION STRENGTHENING Finding 5.1: Scholarships enabled more Tibetan students to access higher education opportunities, with strong completion rates among scholarship beneficiaries. Finding 5.2: Scholarships especially benefitted female youth, expanding their empowerment and professional opportunities. Finding 5.3: Timing of the scholarship funding is not well-coordinated with academic fee schedules, which places a burden on beneficiaries to arrange for postponement or wait for reimbursement of tuition and fees after USAID funds are received. Finding 5.4: Measuring outcomes of KRA 5 is challenging for TSRR administrators, including CTA staff, who have limited capacity to conduct long-term follow-up to measure impact of this activity. KRA 6. COMMUNITY HEALTH IMPROVEMENT Finding 6.1: Respondents felt that the cross-sectoral integration of Social and Behavioral Change Communications (SBCC) training in health and education has helped to improve health outcomes in local communities due to greater support to identify lifestyle and chronic health conditions. Finding 6.2: The Tibetan Medicare System (TMS) plays a critical role in facilitating access to secondary and tertiary health care for Tibetan communities in India and helps to reduce the financial burden of accessing health care. Finding 6.3: Comprehensive community outreach and coordinated care (CCOCC), which mobilizes health professionals to make home visits, has effectively introduced preventative health care into communities but its scope is constrained by staffing resources. Finding 6.4: While primary health care has improved under TSRR, greater specialization is lacking, especially for women’s services, such as maternal health. Respondents felt the need for care from specialized doctors such as ENT specialists and gynecologists, if not full time, then on a rotating or weekly basis. Some members suggested that the doctor at the clinic needs specialized training on areas specific to the needs of the community. vii | TASKING N050: TSRR EVALUATION USAID.GOV KRA 7. PARTICIPATORY COMMUNITY DEVELOPMENT Finding 7.1: The original design of the participatory needs assessment (PNA) was not effective and had to be overhauled in Year 2. TSRR was starting to roll out the revamped approach at the time of this evaluation. Finding 7.2: TSRR had mixed results regarding the equal inclusion of stakeholder voices in the PNA process. Although respondents reported that all stakeholders, including women, had been engaged, there were still individuals interviewed by the ET who had not been included in the PNA discussion (e.g., young people who were working to influence their communities, one school headmistress who had not been invited to the meeting despite expressing interest in participating). Finding 7.3: Beneficiary communities expressed a strong value and desire for a community-driven process to select projects for funding to minimize the top-down approach to development and promote the activities that meet the community’s needs. KRA 8. WOMEN’S EMPOWERMENT Finding 8.1: TSRR built the capacity of the Women’s Empowerment Desk at the CTA Central Office, including increased staff and visibly enhanced focus on women’s issues. However, this has not trickled down to the settlement level, where there were not dedicated staff focusing on women’s issues (although TSRR has reported plans to expand these efforts in FY2020). Finding 8.2: Gender training and sensitization workshops are changing mindsets, but slowly. Many respondents were resistant to the idea that females faced social challenges and limitations, but a few respondents indicated that this mindset has evolved through TSRR’s work. Finding 8.3: Community members were skeptical about the need for women’s empowerment support in Tibetan communities overall. EQ 2. To what extent are the systems and processes introduced for strengthening the capacity of Tibetan institutions being adopted and institutionalized by these organizations? The final three KRAs (9-11) support the strengthening of Tibetan institutions; these are presented in more detail below. KRA 9. INSTITUTIONAL CAPACITY DEVELOPMENT Findings: The beneficiaries from CTA’s central and local settlement offices of TSRR’s institutional capacity development activities regarded the trainings as helpful and offered specific examples of how it affected their core competencies like improved presentation skills, public speaking, and communications. However, this impact was felt less at the settlement levels, especially in offices with high turnover rates. KRA 10. LEADERSHIP DEVELOPMENT Findings: Beneficiaries of KRA 10 praised the quality of the leadership training, including skills development in decision-making, e-governance processes, legal guidance, and management skills. TSRR’s measurement of knowledge gains for KRAs 9 and 10 are largely based on self-reported measurements provided by training participants, and currently does not inform the relevant TSRR indicators. TSRR reported difficulties in quantifying the efficacy of this work and expressed interest in strengthening these measures in order to better understand the impact and lessons learned. USAID.GOV TASKING N050: TSRR EVALUATION | viii KRA 11: CULTURAL INSTITUTIONS AND RESILIENCE STRENGTHENING Finding 11.1: TSRR’s support to the Tibetan Institute of Performing Arts (TIPA) in India strengthened the organization’s capacity to offer quality arts programs, but the Himalayan Buddhist Library and Cultural Center (HBLCC) in Nepal needs continued support for development. Finding 11.2: Although support to artists and institutions provided encouragement to increase cultural vitality, the activity needs to expand its reach in Tibetan communities, especially in smaller villages, which generally lack arts and cultural institutions. Finding 11.3: The CTA is doing a good job to promote and preserve Tibetan culture, especially by integrating cultural opportunities into education, but more can be done to recognize how the Tibetan population, and thus its culture, is changing over time. While many of these things are already being done, such as designated days and events to celebrate Tibetan culture and promote Tibetan language, respondents indicated they felt more could be done to promote the sense of a strong national identity among refugee communities. Respondents’ suggestions include investing more in cultural and religious education and preserving Tibetan language and intercultural learning through translation of literature and academic publications into Tibetan language. Finding 11.4: Emigration, globalization, and assimilation are perceived by Tibetan communities as major threats to their increased cultural vitality and resilience. EQ 3. To what extent is the integrated multi-sectoral approach an effective model for responding to the theory of change put forward by the Program? Findings: While the multi-sectoral approach is an innovative model with great potential to offer a range of services to meet the holistic needs of a dynamic community, implementation for the first three years has been more a collection of services offered to different beneficiaries rather than an integrated model. Most beneficiaries had only accessed support from one KRA, with a few exceptions, such as KRA 1 loan beneficiaries who also received support from TMS or beneficiaries who had strong social networks that increased their awareness of available support. Respondents advocated for a paradigm shift within the strategic planning model that places Tibetan culture at the center of implementation, with all of the other KRAs emanating from this central focus. While those promoting this model do not suggest that the bulk of TSRR’s support go to cultural (i.e., artistic) endeavors, the holistic approach could be enhanced by this shift, in that cultural vitality is viewed as the core from which all the other areas are linked. EQ 4. What are the key aspects of the program that should be addressed at this stage to maximize outcomes over the rest of program implementation? Finding 1: Migration is a major challenge to the vitality of Tibetan settlements in that settlement populations are decreasing in number and increasing in the average age of its inhabitants. As young people go abroad to seek better livelihood opportunities, the vitality of these communities is diminished. Finding 2: As a government in exile, the CTA is dependent on donor funding to support its activities and services, which, a respondent noted, limits their ability to implement projects that are not directed by donors’ agendas. Respondents from the CTA felt that TSRR’s work to establish a Non-Banking Financial Company (NBFC) and develop this institution into a full-fledged bank offers the CTA an opportunity to break this donor dependency and was one of the most promising areas for sustainable impact. However, the path to commercial profitability for the envisioned bank remains unclear. ix | TASKING N050: TSRR EVALUATION USAID.GOV Finding 3: The limited scope of the multi-sectoral model as a strategic approach to participant recruitment, selection, and service provision has also limited its impact in promoting vital and holistic communities. Respondents encouraged TSRR to facilitate greater cooperation between the sectors at the central level in order to promote better communication and collaboration between departments. CONCLUSIONS EVALUATION QUESTION 1 Livelihoods: Overall, TSRR has made good progress in supporting livelihoods, including micro and small enterprise development and promoting access to enterprise finance, including the NBFC. While workforce development is one area of TSRR that has not been as well-developed, skills training is an important element in reaching a middle-aged labor force with limited education and providing engaging economic opportunities for young people with limited options. Investments in agriculture, including soil testing, fertilizer, and farmer’s training, have all produced measurable outputs. However, maximum growth has been constrained by poor timing and selection of crops and an aging population that is concerned about the role of agriculture for the next generation. Education: Data suggest that TSRR has performed extremely well in regard to achieving planned outputs and outcomes in basic and higher education sectors, including promoting Tibetan language reading, training teachers, and facilitating access for disadvantaged students. As with other KRAs, TSRR struggles to measure some long-term outcomes in their work under KRA 5 in higher education, such as increased employment rates for beneficiaries. Health: TSRR has made noted progress in the health sector, particularly through the SBCC and CCOCC programs. Both initiatives have helped improve the access and quality of primary healthcare and preventative measures by expanding the service provision and knowledge of local health care providers. Health insurance provided by TMS and expanded under TSRR has provided increased coverage and access, with 27,496 beneficiaries reported in TSRR’s Midline Report, and a 9 percent increase in the percentage of individuals with health insurance reported in TSRR’s FY19 Q4 report. TMS presents an opportunity for sustainable, non-donor, dependent growth in the future. However, beneficiaries noted the need for greater specialization, particularly in the area of women’s healthcare. Women’s Empowerment: TSRR’s development of the Women’s Empowerment Desk at the CTA is a prime and inarguable example of how the project has strengthened the capacity of the CTA. Gender training has also been a successful component, though the outputs are somewhat lower than targeted. EVALUATION QUESTION 2 Capacity Building: TSRR’s work in strengthening the capacity of the CTA has improved management, oversight, communications, and evidence-based leadership within the organization. While all outcome measures for the capacity development work are self-reported, supervisors reported observing changes in their staff and community members reported observing an improvement in the CTA’s functioning, specifically in the areas of enhanced communication with the community members, and in the capacity and productivity of staff. Arts and Cultural Vitality: Support for cultural vitality has been primarily achieved through financial donations to the TIPA in India and the HBLCC in Nepal, as well as grants to support Tibetan artists. The USAID.GOV TASKING N050: TSRR EVALUATION | x CTA’s work in cultural preservation was rated very highly in the community survey, and qualitative respondents praised their efforts, particularly in promoting the preservation and celebration of Tibetan language and customs. EVALUATION QUESTION 3 Multi-Sectoral Approach: The multi-sectoral approach is clear in the implementation of 11 KRAs across multiple countries and communities; however, the extent to which all of these elements function together as an integrated system is somewhat less evident. At the CTA level, there was coordination between departments, but this was not systemic. At the community level, there is more integration by the nature of service provision at the individual level but there is no formal selection process to ensure service provision to those most in need. EVALUATION QUESTION 4 Migration: In promoting Tibetan refugee communities that are resilient and have vitality, it is essential to address the issue of migration. While solving this issue is beyond the scope of TSRR’s mandate, the work that is being done through TSRR is attempting to create an environment that will alleviate the issue in a holistic, multi-sectoral and community-driven process. Banking: One activity that has potential to make a significant impact on the CTA’s organizational capacity is the development of the NBFC. Reducing the CTA’s dependency on donor investment and enabling the Tibetan refugee population greater access to banking and loan opportunities would be a significant achievement and legacy for the TSRR project, although the path to achieve commercial profitability is still unclear and may benefit from external consultation. Community Involvement: Strengthening the multi-sectoral approach at both the management and implementation levels would be a strategic investment that can build the types of vital, resilient communities that TSRR seeks to achieve. TSRR overall has achieved notable successes in supporting increased resilience and self-reliance for Tibetans in India and Nepal. However, the findings of this evaluation also identify some of the issues that TSRR has faced in its implementation. The Program continues to seek strategies to address these issues, including the immense challenge of implementing multi-sectoral programming across 11 KRAs to 52 sites in two countries. Oversight is held by the Settlement Officers, who manages this in addition to other duties. While local priorities are to be set through the PNA process, only half of the active communities have undertaken PNAs. TSRR acknowledges that the PNA process has had implementation difficulties and it is in the process of being revamped as the activity enters the fourth year of implementation. RECOMMENDATIONS EVALUATION QUESTION 1 RECOMMENDATIONS 1) Provide more legal support services to aspiring entrepreneurs for formal registration of businesses, which is required to own and operate a business, especially for those who do not have citizenship. 2) Revisit the targeting of entrepreneurship activities to focus on marginalized groups. Explore whether financial costs are most effective in their current investment structure (i.e., grants/loans to start-ups and MSMEs) or if additional investments in higher education and proactive business development and training strategies would promote the development of a stronger pool of aspiring entrepreneurs. xi | TASKING N050: TSRR EVALUATION USAID.GOV 3) TSRR should work in close collaboration with local implementing representatives identify and reach the most vulnerable, including women and youth, with livelihood development skills. 4) Explore opportunities to promote linkages between KRA 2 with other relevant activities including TED and other livelihoods development, as well as agriculture (KRA 3) and education (KRA 4). Seek to reach beneficiaries with under-employed adults who seek to develop their vocational skills. 5) TSRR should update analyses of high-growth sectors and explore potential areas to expand livelihoods development activities considering the changing labor market surrounding traditional handicrafts and waning interest in agriculture. While it is beyond the scope of this evaluation to conduct a full analysis of other potential high-growth sectors, this may be worth exploring. Respondents suggested TSRR should expand its investment in settlement-based factories, service sector businesses, animal husbandry, and a wider range of cottage industries. 6) Education appears to be a primary source of integration in the community, allowing for the introduction of health and cultural information as well as basic learning. The activity should assess how to improve community outreach through schools, especially among parents, to increase efficacy of learning materials (emphasizing the importance of reading at home) and reduce dropout rates. 7) TSRR should dedicate significant resources in the coming year to focus on implementing and expanding the newly revamped PNA process, to promote greater success in the next round of community engagement. TSRR should work closely with CTA staff at both the central and the settlement levels to ensure that all voices are included, and explore the potential to designate a local TSRR representative to assume support for direct management of implementation, including stronger female representation at the highest levels of decision-making authority. EVALUATION QUESTION 2 RECOMMENDATIONS 1) TSRR should explore conducting pre-post testing of outcomes from capacity-building efforts at the central and settlement levels in order to gain a stronger sense of the training’s longitudinal impact. EVALUATION QUESTION 3 RECOMMENDATIONS 1) TSRR and the CTA should continue to promote a multi-sectoral approach at both the central and community levels, including establishing regular inter-departmental meetings to share knowledge and promote collaboration within the Central CTA departments and at the secretary level. 2) TSRR should consider strategic ways to address respondents’ feedback to place Tibetan culture at the center of the project as a conceptual shift that can promote enhanced multi-sectoral thinking. TSRR should explore the possibility to establish a local TSRR representative in each community who is incentivized to manage the daily operations on the ground focused on implementing a holistic continuum of services that creates vitality and strengthens opportunities for all beneficiaries. EVALUATION QUESTION 4 RECOMMENDATION 1) TSRR and CTA should work to identify concrete strategies to successfully support development of the NBFC to become a fully operational banking center by the end of the project and promote access to finance for women and marginalized individuals, including those who lack access to traditional banking systems. 1 | TASKING TO50: TSRR ASSESSMENT USAID.GOV 1. EVALUATION PURPOSE & EVALUATION QUESTIONS In 1959, the government of the People’s Republic of China overran Tibet, sending some 150,000 Tibetans into exile. Many of these Tibetan refugees fled to Nepal and India: a 2009 census report published by the Central Tibetan Administration (CTA) reports a total of 128,000 Tibetans living outside of Tibet, including 94,203 refugees living in India and 13,514 in Nepal; another 1,298 live in Bhutan.2 More recent data included in TSRR’s Midline Report, citing WFIS 2019 data as its source, indicate that these numbers have since decreased, with 66,716 Tibetans living in India and 9,258 living in Nepal. In India, Tibetans are organized into 39 formal communities and “dozens of informal communities.”3 While in India, Tibetans have access to education, but they do not have citizenship rights, and cannot own land or property, because India is not a signatory to the 1951 Refugee Convention.4 The lack of rights and opportunities had led to a nearly 33 percent decrease in the Tibetan population in India (from 94,000 in 2009 to approximately 67,000 today). In Nepal, only a minority of Tibetans have Nepalese citizenship. Most live a stateless existence without residency rights and identity cards. Many Tibetan settlements are in earthquake-affected areas and are among the poorest residents of a largely rural and poor state.5 Tibetans in Nepal are also subject to arrest as a result of China’s ongoing influence in Nepal’s political and economic arenas, including recent discussions of a Nepal-China extradition treaty.6,7 Due to their lack of legal standing in India and Nepal, Tibetans in these countries remain poor and lack access to livelihood opportunities such as quality education and job skills training, financial support to develop small businesses, access to quality health care, and cultural development that enriches the vitality of a diaspora community through the shared experience of their heritage. Without this vibrancy in the Tibetan settlements, many young people migrate abroad to pursue better opportunities elsewhere. This leaves many elderly and infirm members in the communities, subsisting on remittances and small amounts of money made selling handicrafts or through community farming. In response to this situation, the United States Agency for International Development (USAID) awarded the five-year, $23 million Program to Strengthen Self Reliance and Resilience of Tibetan Communities in South Asia (TSRR) activity to the Tibet Fund (TTF) to strengthen the self-reliance and resilience of Tibetan communities in South Asia by “equipping them to thrive economically, become effective leaders, and maintaining the vitality of Tibetan communities and institutions while sustaining their unique identity and 2 Press Trust of India, “127,935 Tibetans living outside Tibet: Tibetan survey,” Hindustan Times, December 4, 2010, https://web.archive.org/web/20110927215516/http://www.hindustantimes.com/127935-Tibetans-living-outside-Tibet-Tibetan￾survey/Article1-634405.aspx. 3 Adhishek Madhukar and Rina Chandran, “Sixty years after Tibet, refugees in India get passports not property,” Reuters.com, June 21, 2017, https://www.reuters.com/article/us-india-refugees-tibet/sixty-years-after-fleeing-tibet-refugees-in-india-get￾passports-not-property-idUSKBN19D019. 4 Ibid. 5 Rani Singh, “Fears for Nepal’s ‘invisible’ Tibetan refugees,” BBC.com, April 27, 2015, https://www.bbc.com/news/world-asia￾32476346. 6 Gopal Sharma, “Nepal detains Tibetan refugees in crackdown as China’s influence grows,” Reuters.com, November 15, 2016, https://www.reuters.com/article/us-nepal-tibet-refugees-idUSKBN13A2BQ. 7 Meenakshi Ganguly, “Fears over an Nepal-China Extradition Treaty,” Human Rights Watch, October 11, 2019, https://www.hrw.org/news/2019/10/11/fears-over-nepal-china-extradition-treaty#. USAID.GOV TASKING N050: TSRR EVALUATION | 2 culture.”8 This five-year program (October 2016 – September 2021), entering its fourth year at the time of this evaluation, is multi-sectoral in its approach, with a wide range of activities including livelihoods and entrepreneurship development focusing on agricultural and handicrafts sectors, basic and higher education, health, gender, capacity development, and Tibetan arts and cultural preservation grants. TSRR envisioned a cross-cutting gender component to complement the other elements. This component incorporates a gender perspective to ensure women participate in and benefit equally from USAID support and strengthen CTA capacity to address key issues facing Tibetan women and girls, essentially seeking to change knowledge, beliefs, and perception on gender. TSRR’s wide range of focus is further spread out geographically, being implemented across India and Nepal by USAID/India and TTF. They work remotely through subgrants to local partner organizations including the Snow Lion Foundation (SLF), the Lodrik Welfare Fund (LWF), and the Himalayan Society for Youth and Women Empowerment (HSYWE). Overall, this ambitious and challenging program has many working pieces to be considered in this midterm evaluation, as described in the next section. 1.1 EVALUATION PURPOSE The purpose of this midterm evaluation is to: 1) assess the extent to which the TSRR activity is contributing to improved livelihoods for the Tibetan refugee communities in Nepal and India; 2) understand how effectively the systems and processes being introduced to Tibetan refugee communities to increase their institutional capacities are being adopted; 3) understand if the integrated multi-sectoral approach is an effective model for the TSRR activity’s theory of change; and 4) identify any aspects of the activity that might need to be changed to maximize the achievement of planned outcomes. The audience for the evaluation report includes USAID/India, TTF and its local implementing sub￾partners, and the CTA at the central and local levels, as well as TSRR beneficiaries and other community members invested in the activity. Findings are intended to inform implementation and management decisions over the remaining lifetime of TSRR, as well as other USAID programs of a similar nature. 1.2 EVALUATION QUESTIONS This performance evaluation seeks to answer the following evaluation questions (EQ): Table 1: Evaluation Questions EQ 1 To what extent have the activities implemented so far contributed to improved livelihoods of the Tibetan refugee community? EQ 2 To what extent are the systems and processes introduced for strengthening the capacity of Tibetan institutions being adopted and institutionalized by these organizations? EQ 3 To what extent is the integrated multi-sectoral approach an effective model for responding to the theory of change put forward by the Program? EQ 4 What are the key aspects of the program that should be addressed at this stage of the program, to maximize outcomes over the rest of program implementation? 8 Jamphel Shonu, “USAID Awards a Grant of USD 23 Million to Strengthen Self-Reliance and Resilience of Tibetan Communities in South Asia,” Central Tibetan Administration, October 3, 2016, https://tibet.net/2016/10/usaid-awards-a-grant-of-usd-23-million￾to-strengthen-self-reliance-and-resilience-of-tibetan-communities-in-south-asia/. 3 | TASKING TO50: TSRR ASSESSMENT USAID.GOV 1.3 PROGRAM BACKGROUND The TSRR Program has two main components: 1) Sustainable Livelihoods Improvement and 2) Strengthening Tibetan Institutions and Cultural Resilience. Component 1 has five Strategic Objectives (SO), including: • SO1: Increase household incomes and reduce underemployment, particularly among unemployed youth, through increased entrepreneurship and small enterprise growth, improved job skills, and increased value of marketable agriculture products • SO2: Improve quality of teaching and learning in Tibetan schools and increased access to higher education, contributing to increased productivity and improved livelihoods • SO3: Increase access to affordable quality health care and reduce the burden on impoverishing disease, contributing to increased productivity and improved livelihoods • SO4: Increase participation and ownership of local communities in the development process • SO5: Increase participation and empowerment of women in the development process Component 2 has two SOs, including: • SO6: Improve effectiveness, capacity, and sustainability of local institutions serving Tibetan communities • SO7: Increase cultural vitality and resilience of Tibetan communities In order to achieve these objectives, TSRR adopts an integrated multi-sectoral approach centering around 11 Key Result Areas (KRAs): • KRA1: Entrepreneurship, Micro, and Small Enterprise Development • KRA2: Workforce Development • KRA3: Agriculture Development • KRA4: Basic Education Strengthening • KRA5: Access to Higher Education • KRA6: Community Health Improvement • KRA7: Participatory Community Development • KRA8: Women’s Empowerment • KRA9: Institutional Capacity Development • KRA10: Leadership Development • KRA11: Cultural Institution and Resilience Strengthening The project is implemented in 52 Tibetan refugee settlements, including 39 in India and 12 in Nepal, but not every site has all 11 KRAs actively implemented in their community. Working in coordination with the CTA central office and local settlement officers in each community in which TSRR implements activities, TTF conducts a community-based participatory needs assessment (PNA) to determine what activities will be implemented at each site; PNAs have been conducted in 26 of the 52 currently-active settlement communities in which it works thus far. This process is intended to promote greater resiliency of the community and its members given local contexts, which can vary greatly. The sites are categorized as either agriculture-focused, handicraft-focused, or a cluster community (which has both); Table 2 below enumerates the TSRR communities by type of primary livelihood area. USAID.GOV TASKING N050: TSRR EVALUATION | 4 Table 2: Number of TSRR Communities by Community Type TYPE OF COMMUNITY NUMBER OF COMMUNITIES India Agriculture 15 Handcrafts 13 Cluster 11 Nepal Agriculture 4 Handcrafts 6 Cluster 2 The context in which these activities are implemented is an essential consideration in determining the best strategies for success, and this context can vary widely between sites, especially between India and Nepal where the political environments differ. While the status of the refugee populations in both India and Nepal is similar, largely because neither country is a signatory of the United Nations Convention and Protocols on Refugees, there are differences in the implementation contexts of the two countries. One primary difference is that USAID/Nepal does not fund any programs for Tibetan refugees in Nepal whereas USAID/India does so in India. This means that the activity is implemented Nepal without the oversight of USAID/Nepal, with activities facilitated by local implementing sub-partners (SLF, LWF, HSYWE). Most of these organizations noted their services are available to the broader Himalayan people and not just the Tibetan population. Thus, the activity in Nepal, is registered to cater to the Himalayan people, a term used to show cultural similarities between the Tibetan population and Nepali citizens in the mountainous region of Nepal. As such, beneficiaries of TSRR in Nepal also include Nepali citizens with Tibetan heritage and cultural similarities. Moreover, the Chinese government’s pressure to curb activities of the Tibetan population is felt more strongly in Nepal. The Government of Nepal does not allow public celebrations and mass gatherings of Tibetan population for fear that they protest the Chinese government. This situation has seriously curbed cultural rights in the Tibetan settlements. In India, public celebrations by Tibetan populations monitoring, but not banned. 5 | TASKING TO50: TSRR ASSESSMENT USAID.GOV 2. EVALUATION METHODS & LIMITATIONS 2.1 EVALUATION DESIGN AND METHODOLOGY In this evaluation, Social Impact (SI) undertook a participatory, mixed-methods evaluation design including: 43 key informant interviews (KIIs), 6 focus group discussions (FGDs), and a quantitative survey with 146 community members in the settlements visited by the evaluation team (ET) during the 3-week field work period that took place from November 4 – 25, 2019. The five-person ET included a Team Leader, a Senior M&E Specialist, an India Technical Specialist, and a Nepal Technical Specialist, as well as a Tibetan Research Assistant to provide translation/interpretation and insight into cultural perspectives. In some locations, an external interpreter was contracted to support work in that community. 2.1.1 PHASE ONE: DESKTOP DOCUMENT REVIEW AND DATA COLLECTION INSTRUMENT/WORKPLAN DEVELOPMENT The initial phase of this evaluation included a thorough review of TSRR’s quarterly and annual reports and annual work plans for the first two years of implementation.9 Using the information derived from these reports, the ET developed a workplan including research methodology and semi-structured qualitative data collection instruments (see Annex III: Data Collection Instruments) designed to solicit in￾depth knowledge from key stakeholders and beneficiaries on the activity’s implementation, adoption, and effectiveness. These tools were tailored for each respondent type, such as TSRR implementing partners including TTF and its local sub-partners, CTA, and community members, and were organized by EQ. The ET also developed the quantitative survey for community members to gather data on TSRR’s progress toward impact and outcome indicators. As part of this workplan development process, the ET also selected six sites to be visited during the field work period, starting from USAID’s request that Dharamsala, Bylakuppe, and Kathmandu be included as three of the sites. The ET’s process of additional site selection was initiated during the document review process by making note of implementation sites that were especially active and offered a wide range of activities to visit. While these criteria did mean that selection was skewed toward more￾developed implementation sites, it was crucial to visit robust sites that offered a wide range of activities and thus offered a productive data collection opportunity given the scope, time, and budget parameters of this evaluation. After this list was compiled, the ET analyzed the locations of each short-listed site, comparing which KRAs are being implemented there, as well as general logistical considerations such as length of time and mode of travel between sites. Based on this analysis, a list of possible sites was submitted to USAID and TTF for further advice. This resulted in the final selection of Dharamsala, Dehradun, Bylakuppe, and Hunsur in India; and Kathmandu and Pokhara in Nepal. The ET also used the background document review process to inform final selection and definition of the key stakeholder groups to be included in the data collection process: TSRR implementing partner staff including TTF, SLF, LWF and HSYWE; CTA staff at the central and local levels, both as 9 Additional project documents were provided later in the evaluation process and were reviewed at that time, but not considered for the workplan. USAID.GOV TASKING N050: TSRR EVALUATION | 6 implementing partners and as beneficiaries of TSRR support; community leaders; TSRR beneficiaries; and members of the general Tibetan community for the quantitative survey. 2.1.2 PHASE TWO: IN-COUNTRY DATA COLLECTION Once the workplan and design document had been developed, the Team Leader and Nepal Technical Specialist traveled to India to meet the rest of the team in Delhi to start the field work period. Initially, the ET met with USAID representatives to present and discuss the evaluation workplan and design plan at a formal in-brief session. Following this meeting, the ET met internally to review the methodology, discuss the workplan and design document, review the data collection instruments, and participate in an orientation session with the Tibetan Research Assistant to discuss cultural norms of Tibetan societies in India and Nepal. QUALITATIVE DATA Once the sites and targeted KRAs had been selected and respondent groups identified, TTF provided the ET with a full list of potential contacts from which to recruit TSRR beneficiaries for KIIs and FGDs. The ET initially tried to select respondents randomly, but due to challenges in contacting beneficiaries and gaining their agreement to participate (see Annex II: Evaluation Design and Methodology), the ET elected to have the Tibetan interpreters call down the full list in order until achieving the desired number of appointments. After the in-brief and training period, the research team split into two groups to cover the maximum number of sites within the allotted field visit period (see Annex II). In total, the two teams conducted 43 interviews with 57 key stakeholders, including 23 interviews with 28 TSRR beneficiaries, five interviews with TSRR implementing partners, and 12 interviews with representatives of the CTA. While the ET sought to achieve equal gender balance in recruiting KII respondents, the final sample had two-thirds male respondents, including more male CTA representatives and more male TSRR beneficiaries, which is somewhat representative of Tibetan society as a whole, which is made up of 55 percent males and 45 percent females.10 Table 3 outlines the sampling strategy by data collection type, disaggregated by sex. Table 3: Sampling Strategy by Data Collection Type and Location STAKEHOLDER GROUP # OF MEETINGS TOTAL RESPONDENTS MALES FEMALES KIIs TTF 2 4 2 2 CTA 12 15 9 6 Implementing Partners 3 4 3 1 Community Leaders 3 6 5 1 TSRR Beneficiaries 23 28 18 10 TOTAL KIIs 43 57 37 20 10 Central Tibetan Administration (2009) Demographic survey of Tibetans in exile 2009, p. 119, http://shodhganga.inflibnet.ac.in/jspui/bitstream/10603/121174/10/10_chapter 3.pdf. 7 | TASKING TO50: TSRR ASSESSMENT USAID.GOV STAKEHOLDER GROUP # OF MEETINGS TOTAL RESPONDENTS MALES FEMALES FGDs TSRR Beneficiaries 6 45 20 25 TOTAL QUALITATIVE 49 102 57 45 QUANTITATIVE SURVEY In addition to the qualitative data collection, the ET conducted a 16-question quantitative survey with 146 community members (not necessarily TSRR beneficiaries) identified through convenience sampling in the six sites visited. The ET worked in collaboration with the Settlement Office to identify areas within each settlement that would be promising and appropriate locations to approach Tibetan community members to respond to the survey; these included main market areas, near Buddhist temples and monasteries, residential areas, and in shops managed by Tibetans. Even working through Tibetan interpreters, the ET had challenges recruiting respondents for the quantitative survey. Community members were reluctant to participate in a random survey due to concerns about the political situation and privacy. Working in close collaboration with the Settlement Office (see Annex II for more details) did increase participation rates significantly, but it was still challenging for the ET to reach the desired number of completed surveys within the time allotted. Ultimately 146 individuals, of which 51 percent were male, were surveyed through the Tibetan interpreter, and their responses were captured in real time using the SurveyCTO online platform. As this sample size is not large enough to be considered statistically representative of the entire Tibetan community, it is important to consider how representative the survey respondents are in terms of demographic composition of the Tibetan population as a whole. The survey population included a much larger sample of respondents over the age of 50 (56 percent) compared to the average for the total population (18 percent).11 Respondents to the survey also reported much higher incomes on average, with over 50 percent reporting an average annual household income in local currency of over 240,000 rupees. Only 16.5 percent of baseline survey respondents reported an annual income of over 200,000.12 2.1.3 PHASE THREE: DATA ANALYSIS The ET analyzed the qualitative data gathered from KIIs and FGDs using qualitative data analysis software, MaxQDA. Key themes were identified and used to generate a coding structure, by which the ET systematically coded all qualitative data to ensure that analysis was conducted in an objective and comprehensive manner. The ET used content and comparative analyses to identify response categories and patterns and to reveal emergent themes and contextual factors. 11 Background on Tibetan Refugees, Economy and Industries. https://www.rochester.edu/tibetchallenge/background/#link_acc-1- 7-d. 12 Program to Strengthen Self-Reliance and Resilience of Tibetan Communities in South Asia (Baseline Report). The Tibet Fund. August 2019. USAID.GOV TASKING N050: TSRR EVALUATION | 8 Quantitative data were entered into SurveyCTO at the time of collection and were analyzed through that system’s online data collection system to tabulate responses, produce basic descriptive statistics, and disaggregate data by standard demographics including gender, location, age, etc. Finally, the ET triangulated all qualitative and quantitative data in order to validate findings across all data sources. These data were analyzed using SI’s Findings, Conclusions, and Recommendations matrix, based on the four EQs, which enabled the ET to systematically identify all findings across respondent groups and types, and to ensure that each EQ was answered thoroughly and ultimately documented in this evaluation report. The analysis process also considered the potential limitations and biases identified during the planning and data collection phases (see Annex II for more detail). 2.1.4 PHASE FOUR: UTILIZATION AND DISSEMINATION The ET facilitated an initial findings presentation in Delhi for USAID, TTF, and CTA staff to discuss the preliminary fieldwork findings. Based on the information shared in the debrief presentation, TTF shared a list of questions and requested clarifications that have been integrated into this evaluation report. This Evaluation Report is the key deliverable for the evaluation and presents de-identified aggregate level data informing the provision of evidence-based recommendations for the remainder of TSRR. It is intended to be a public document that will be disseminated to interested parties via USAID’s Development Experience Clearinghouse website in order to promote transparent and accountable development service provision. 9 | TASKING TO50: TSRR ASSESSMENT USAID.GOV 3. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 3.1 FINDINGS 3.1.1 EVALUATION QUESTION 1: TO WHAT EXTENT HAVE THE ACTIVITIES IMPLEMENTED SO FAR CONTRIBUTED TO IMPROVED LIVELIHOODS OF THE TIBETAN REFUGEE COMMUNITY? Within livelihood development, TSRR’s multi-sectoral approach contributed to eight KRAs, including: entrepreneurship, micro, and small enterprise development (KRA 1), workforce development (KRA 2), agriculture (KRA 3), basic education (KRA 4), higher education (KRA 5), community health (KRA 6), participatory community development (KRA 7), and women’s empowerment (KRA 8). The following section breaks down the evaluation’s findings by each of these KRAs. KRA 1: ENTREPRENEURSHIP, MICRO, AND SMALL ENTERPRISE DEVELOPMENT Under KRA 1, TSRR provides support in two components: the Tibetan Entrepreneurship Development (TED) initiative, and Access to Enterprise Finance. The TED initiative, launched with funding from the US Department of State in 2013, was continued under TSRR with the provision of business development services for entrepreneurs, piloting of services for existing Micro, Small and Medium Enterprises (MSMEs), and piloting of a new skills to enterprise service model for vocational trainees. Access to finance is supported through 1) the establishment of a Non-Banking Financial Company (NBFC) with the aim of providing savings and loan services to enterprises in India and 2) a feasibility study and the establishment of community managed revolving funds (CMRF) in Nepal.13 It is important to note that new TED activities in Nepal were halted at the end of FY18 (Quarterly Progress Report [QPR] 4 2018). One of CTA Social Resource Development Fund (SARD) respondent said this about the situation in Nepal: “TED hasn’t been very active because TTF has stopped the funding because they feel that the program hasn’t reached out to beneficiaries. TTF was expecting too much with the little investment they’d put in. Lots of conditions were put in the TED funding, [so there were] no registered business or rarely [registered]. They don’t see prospects here [because of the] legal situation here. People aspire to go abroad. Enough time was not given.” -KII Respondent TSRR contracted an external evaluation of TED activities in Nepal during the first half of the activity’s implementation and identified several significant challenges that the activity faced. These challenges include: difficulty in identifying entrepreneurship development facilitators and mentoring partners to support aspiring entrepreneurs, limited local partner management capacity, political challenges including difficulty in registering businesses owned by Tibetans in Nepal, and questions about the appropriateness of the TED training model for micro- and small enterprises in this context. Respondents in the TED evaluation also raised questions about the age limits of beneficiaries and recommended that older beneficiaries be included due to their more extensive professional experience and ability to register businesses, as older Tibetans in Nepal are more likely to have official status. Based on these challenges, 13 TSRR AID-386-G-16-00001 Quarterly Report FY19 Q4. USAID.GOV TASKING N050: TSRR EVALUATION | 10 TSRR elected to pause or discontinue TED activities in Nepal while decisions were made about how to better support aspiring entrepreneurs going forward given these findings. As part of the evaluation, the ET reviewed TSRR’s quarterly monitoring data, including outputs for KRA 1. Analysis of these data show ambitious, but hard to reach, targets. Data from the Q4 QPR indicate that targets regarding the provision of business development services to entrepreneurs and MSMEs were achieved in Year 1 but lagged in Year 2 (61 supported versus 140 target) before recovering in Year 3.14 One area that has seen marked success against targets is the total number of clients benefitting from financial services, which has been met or exceeded throughout the first three years of the activity. Student participation in entrepreneurship development activities shows a huge gap, achieving less than 65 percent of the target in Years 1 and 2. Other outcome-level data, such as the number of new enterprise start-ups linked to TED support, show initial success in Years 1 and 2 but target gaps (6 out of 20 outcomes) in Year 3. The findings below are reflective of these indicators. Respondents provided greater attention to, and input on, the role of financial services in livelihood development, and less of a focus on the role of business and development skills. This may be a result of fewer individuals receiving business development support (194 in Year 3) than individuals receiving financial services (2,300 in Year 3) but may also reflect their interests and needs. Finding 1.1: Business development support in the form of trainings under TED received mixed results as to its targeting and efficacy. Under TED, TSRR provided a variety of coaching, mentoring, and skills training opportunities. The ET spoke with both implementers and beneficiaries of these activities, as well as other community members who were aware of TSRR but had not participated. TED was lauded by an implementer as providing beneficiaries with “confidence, exposure, and awareness.” This statement was echoed by beneficiaries, one of whom received training on product quality, management, marketing and later receiving a loan noting it was good to have the training first as “it stays with you for a long time.” Another beneficiary who received a loan but was unable to attend training felt like she lost an opportunity to learn what her friends had learned about business management. As she lacked a formal education, she felt that such information would have benefitted her, teaching her how to grow her savings as well as keep track of how much money her business was making after she took the loan. While the evaluation primarily spoke to individuals who benefited from TED, the impact was not necessarily universal or equally beneficial to all. Regarding businesses, one MSME grantee noted “I could’ve done this without this support but from what I’ve heard from others, that this training should go on because others also want to have it.” Another, who believed the trainings to be successful, noted that almost all attendees were previous business owners, not new entrepreneurs.15 Over 70 percent of respondents from the community survey agreed or strongly agreed that they already “have adequate skills and knowledge to earn their livelihood.” Another beneficiary noted the training “was good, fine, but not highly effective or exceptional.” 14 TSRR AID-386-G-16-00001 Quarterly Report FY19 Q4 Annex 1. 15 It is important to note that we use “trainings” here in the broadest sense. The TSRR program has a large mix of activities and outreach from the community to the national level. 11 | TASKING TO50: TSRR ASSESSMENT USAID.GOV Given the breadth of knowledge, interest, and skills in the community, program targeting may play an important role. For activities that targeted entrepreneurs, one respondent cautioned “When we select them for the entrepreneurs’ program, we don’t understand their backgrounds and intentions.” While some are good candidates, others are seen as being “not serious” or as having family or other commitments that prohibit them from being successful. According to the same respondent, the activity received 25-30 applicants, primarily unemployed youth, leading the respondent to note the recruitment process requires improvement “so we have better candidates.” The current recruitment activity “trained 16 entrepreneurs, and 10 of them were successful in still running their business in a sustainable manner. Under the Skills to Enterprise, we trained maybe 20-25 unemployed youth, and three of them were successful.” However, these sentiments and figures are not reflected by all, with another respondent noting, “Success can be rounded up to new entrepreneurs under the program… [We established] 20 new entrepreneurs because of the program.” There is an opportunity to enhance recruitment and selection of beneficiaries, at least among new entrepreneurs, to promote participation rates and outcomes. Finding 1.2: Loans and access to financial services was the most cited benefit under KRA1 and future need for the activity was noted, in addition to desired greater flexibility. The loans and grants offered by TSRR were undoubtedly one area that had the highest profile and best reputation among respondents of all stakeholder types. People knew that the activity supported micro￾business development and were positive about the impact of this support. “People are more empowered. They can do their own business. Before they worked for others. Before they had to sell stuff on the road. With loans they feel empowered.” – Male KII Respondent, Settlement Officer The importance of loans and financial services was repeatedly raised across all levels of respondents and types of beneficiaries, even those targeted under other KRAs. Especially in Nepal, the legal status of most Tibetans, including the inability to register businesses, has made it difficult to secure access to finance. The program assists in fulfilling that gap with the establishment of the NBFC and the expansion of the CMRF. One KII respondent said, “[The] bank will be one of the biggest things to come out of the program.” Another respondent noted that “We have built a for-profit Tibetan financial institution… It’s the first of its kind.” According to TSRR Program staff, 3,000 businesses have been supported, with 45 percent female participation. Loans and grants under TSRR have provided beneficiaries with the capital to purchase inputs to grow their business in a way they previously viewed as infeasible. For example, one beneficiary noted, “The loan amount helped in providing me the choice to buy clothes from a wholesaler of my choice. With the loan amount, my business expanded and contributed to more selling and thus more income.” USAID.GOV TASKING N050: TSRR EVALUATION | 12 Figure 1: Case Study - Micro Loan to Female Entrepreneur Enhances her Livelihood Tenzin came from Tibet in 1997; she walked on foot for a month through the Himalayas to reach India. She has lived in Dharmasala since that. She got married after she arrived and had two children. She knew no one else in India and she did not get to speak with her family in Tibet until 2006. Her father passed away before he could come to see her in India, but her mother came to meet her grandchildren. Tenzin had little formal schooling, but in the late 1990s, she participated in a program by the Tibetan Women’s Association to learn tailoring, which she did until the birth of her first child. A year later, when she returned to work, she cooked and cleaned for a local association and provided most of the childcare while her husband worked the night shift as a waiter in a local restaurant. After five years, her daughter started school, which did not provide lunch for the children, so she had to leave her job to feed her child lunch. A friend suggested she start a home-based business so she could do childcare and work at the same time. She had saved 30,000 rupees for her children’s education, which she decided to use as the seed money to start a business selling jewelry that she makes herself and sells on a footpath near her home. Last year, the McLeod Small Business Association advertised a loan program from the CTA for small business development with a very low interest rate. Tenzin approached them and they helped her to complete the application. She took a loan for 100,000 rupees with 2,000 in interest in six months. She sold her existing jewelry to pay back the loan and then used her profit of about 40,000 rupees to expand her business into silver earrings and paid for her daughter’s tuition at Delhi University. She wishes she would have gotten this type of help earlier in her business development process because of the impact it had on her business and her family. This was the first time she had been offered an opportunity to take a business development loan; even if she had wanted to take one in the past, she did not have a guarantor. “Many of my friends also took this loan and I see how it transformed their lives; it really does help to support development of the Tibetan community. Many of my neighbors do not speak English or Hindi but they still manage to have a small business. When we see beggars in the road, we feel really proud that we do not have to do this because we have a livelihood.” In general, these individual statements and the case study are a strong reflection of respondents’ sentiments. However, it is important to note that they typically were coupled with suggestions for improvement. The data did not always differentiate between the types of financial support that was given, either through TED or other TSRR-supported loans and grants. While grants are more straight￾forward in their application, in that they do not require repayment and understanding of interest and terms, at least one settlement officer noted grants also present a disadvantage in that, “To make business bigger and better, loan is better. There is more ownership and responsibility.” Loans were also noted by CTA officials as an opportunity to move communities away from a “dependency mindset.” However, the terms of the loan, particularly regarding size and length of repayment, were noted as issues by several respondents. Suggestions including lengthening the payment period from 6 to 12 months to ensure repayment did not fall during the offseason and ensuring that the size of the loan 13 | TASKING TO50: TSRR ASSESSMENT USAID.GOV allowed for proper investment. Flexibility of loan products and financial institutions is noted further under Finding 1.4. Finding 1.3: TSRR has the potential to enhance integration at the community level, but some respondents are concerned benefits are too heavily concentrated among particular individuals and household units. Respondents had mixed reactions to the targeting of youth under 35 years old for support. While some saw this as a benefit in offering expanded livelihoods support for young people and potentially reducing migration, others were concerned that older community members who also needed support were not receiving it. Concerns about migration, particularly among youth, arose as a prominent issue during the evaluation. This report addresses it as a stand-alone component in Section 3.2.4 below. Communities place great emphasis on improving livelihoods in order to increase a sense of security. One CTA official notes there are “issues of migration – any chance they may have to go abroad or leave this [place], they will go. This is a challenge in all our programs, not just this one. It has been this way for a long time. We’re starting this program to reduce this migration rate, asking people to stay in the settlements and improve their livelihoods here.” While reducing migration is not a documented goal of the activity, it was mentioned without prompting in several interviews; some beneficiaries expressed skepticism that the activity would be successful even if it was. Given the propensity of individuals or households to migrate, several respondents advocated for a more holistic approach to livelihood development at the community, rather than individual, level. While some comments reflect a broader targeting approach, noting “The support should be accessible to [people of] all age groups and all Tibetans because right now the focus is more on young people,” most proposed community level investments are in infrastructure, especially cooperatives (bakery, dairy, sewing). This is especially relevant in Nepal where most Tibetans do not have citizenship, so they are forced to partner with someone who does have citizenship in order to register their business. At least seven respondents noted the role existing cooperatives or potential factories play in opening and diversifying livelihoods. Finding 1.4: The activity has an opportunity for further innovation and flexibility moving forward. There are several areas the activity and communities have had to innovate in order to maximize impact. Two innovations, the CMRF and self-help groups, were noted as particularly successful, but appear to be concentrated in Nepal. It is unclear if these innovations are due to the higher prevalence of formal barriers in Nepal or other contextual factors, however. CMRF closely mirrors village savings and loans in other parts of the world where participants invest a specific amount of money each month and take turns withdrawing loans, which are then paid back with interest, allowing the fund to grow and profit. Discussion with CTA staff and beneficiaries in Nepal noted this as a useful source for growing businesses, and its ability to reach the “unbanked” population, has given it special consideration under TSRR. The ability of some CMRFs to gain a profit on loan repayment may especially provide insight into any larger banking institution that the CTA is able to establish. However, the long-term impact of such activity is unclear and outside the scope of this USAID.GOV TASKING N050: TSRR EVALUATION | 14 evaluation to determine. Past research suggests that it improves financial inclusion and resilience to external shocks but does not have long term impacts on livelihood and household income.16 Another innovation that ties into the desire of many respondents for a more community-focused approach to livelihood improvements is the group approach. This approach mobilizes group savings to advance commercial agriculture for the community. A social mobilizer helps with mentoring support and identifying market linkages, and it is reasonable, though not specifically noted by respondents, that consolidation of products at the group level helps promote access to the commercial market. While not explicitly stated during interviews, one could anticipate that, if utilized correctly and with proper support and coordination, the group model could expand out from agriculture into areas of noodle making, baking, and tailoring. While financing and community groups were the primary areas noted for innovation in improving livelihoods, a few others were mentioned as well, including the diversification of livelihood programs from tailoring into other areas and higher utilization of technology platforms (social media, etc.).17 External competition from online retailers like Amazon and Flipkart was noted as negatively affecting business, especially in the past three to four years. Finding 1.5: Despite the grant agreement noting women entrepreneurs as a targeted group of the activity, the ET uncovered little evidence of this in practice, or the perceived need for this approach.18 KRA1 notes female entrepreneurs were targeted as beneficiaries under the India program. However, when most respondents were questioned as to how training or loans were provided under KRA1, they noted that gender was not a primary consideration. The main reasons presented for not following guidelines according to KII respondents were: 1) women are already outperforming men, especially in business and 2) loans are given at the household level, to either males or females, and not on an individual basis. When asked as part of the community survey whether they had the adequate skills and knowledge to earn their livelihood, 68 percent of females agreed or strongly agreed that they did, compared to 75 percent of males. Likewise, 51 percent of females noted sufficient savings for six months in the event that they lose their income versus 55 percent of males.19 Both findings suggest a level of equality among men and women, though it is unclear whether either gender is considering their individual financial security or that of their household. The QPR provided to the ET did not disaggregate outputs by gender so it is difficult to triangulate these statements with programmatic data. However, as noted under Finding 1.2, one respondent cited females as 38 percent of those receiving business loans. KRA 2: WORKFORCE DEVELOPMENT KRA 2, workforce development, includes support for the USAID-founded Dharamsala Tibetan Career Center (DTCC) in India. The DTCC provides vocational skills training, career guidance, and job 16 Dean, K., S. Beniamino, T. Bram, and C. Udry, “Impact of savings groups on the poor,” Proceedings of the National Academy of Sciences Mar 2017, 114 (12) 3079-3084; DOI: 10.1073/pnas.1611520114. 17 TTF has noted that workforce development training and financial services are available for a range of trades. However, perhaps due to the limited scope of this evaluation, those interviewed were primarily aware of livelihood activities in the garment (sweater) industry only. 18 TSRR AID-386-G-16-00001 FINAL Grant Agreement. 19 In both instances, the differences between men and women was not statistically significant, though these figures should be determined with caution given the non-random sampling of respondents. 15 | TASKING TO50: TSRR ASSESSMENT USAID.GOV placement services to Tibetan youth in India. Skills training courses topics include hairdressing, English language, web design, and other IT job preparedness areas. Under KRA 2, TSRR is also supporting the development of a workforce information system to capture and maintain data on the status of the workforce in Tibetan settlements. KRA 2 is not actively being implemented in Nepal. TSRR’s performance monitoring plan includes both output- and outcome-level indicators related to KRA 2, including the number of individuals completing United States (US) government-assisted workforce development programs, and the number of individuals with new or better employment following completion of US government-assisted workforce development programs. Figures from the Q4 QPR indicate that TSRR’s training outputs were strong in Year 1 (521 trainees versus 485 target) but declined somewhat in the subsequent years (359 trainees versus 460 target in 2018, and 226 trainees versus 305 target for 2019). Outcome-level data (number of individuals with new or better employment) indicate that Year 1 was on target and Year 2 exceeded its target (79 employed versus 60 target), but Year 3’s outcomes have been much lower than expected (35 employed versus 75 target). The final two years target even greater numbers. Finding 2.1: KRA 2 is less developed than some other KRAs, and experiences challenges in demonstrating its outcomes. Respondents felt that the activities under KRA 2 had strong outputs but that the outcomes were not to the level expected. As a CTA Department of Home KII respondent stated: “The outcome is not up to the mark … young people enroll and complete training but don’t go to work related to this field. Also, technical, computer courses, they’re helpful but not jobs oriented… compared to the outcome, it’s not that much.” Matching low-skilled laborers to high-growth markets and providing the necessary professional development to facilitate that transition is a challenging task that requires time to show demonstrable outcomes. As one implementing partner KII respondent characterized it: “The vocational training [has shown] much less progress; it has been slow and incremental.” Finding 2.2: KRA 2 offers an area of potential growth and investment for TSRR. As noted above, the targets for the final two years of TSRR are even higher than the first three years, and respondents felt that jobs-related skills development could be an area of expanded investment going forward, especially skills training for those with limited education and existing opportunities. Seventy￾two percent of survey respondents agreed that they have adequate skills and knowledge to earn a livelihood, meaning there is still about one in four community members who wish to increase their knowledge and skills to support increased livelihoods. KII/FGD respondents suggested that TSRR should expand its provision of skills development training courses to support community members who did not or would not obtain a university degree, including middle-aged labor force participants who were not educated in their youth, as well as young people who may wish to stay in the settlement communities but are not suitable candidates for higher education opportunities. “We Tibetans, we think education is for the elite, the people who get the best grades… There are so many students who are also interested in higher studies but don’t have the capacity … most average students don’t have their family’s support to choose higher studies … They start working at an early age: usually the girls are beauticians and the boys USAID.GOV TASKING N050: TSRR EVALUATION | 16 go into the army. Also, there are vocational education centers where they learn cooking, cultural arts preservation.” Female KII Respondent - KRA 5 Beneficiary Based on this feedback, respondents do feel that TSRR has a necessary role to play to meet the needs of youth seeking to join the labor force and promote enhanced livelihoods for older workers with limited knowledge and skills. KRA 3: AGRICULTURE Agriculture development interventions under TSRR include information and communication technology extension services; agriculture extension services such as soil health testing; trainings and exposure visits; development of the Organic Research and Training Center in Bylakuppe; establishment of new regional agriculture extension centers; and cash crop value addition. Additionally, in Nepal, agricultural development support is focused on the introduction of new cash crops, strengthening agricultural production, and supporting farm maintenance. TSRR’s performance monitoring data from the 2019 Q4 QPR show that the activity has struggled to consistently meet targets on trainings and adoption of improved technologies and management practices after the first year of the activity. In 2019, progress against targets was less than 50 percent for most indicators. Giving the timing of the evaluation, the ET did not have time to explore the factors behind the achievement gaps with the implementation team. However, TSRR documentation suggest that challenges in reaching targets, particularly in India, are partially due to the low-capacity of field-based agriculture extension officers and the limited management capacity at the Department of Home.20 Finding 3.1: Technical assistance and provision of inputs has been provided to increase farming yields and promote crop diversity, but a lack of integrated planning hindered effective utilization. For KRA3, the ET primarily interviewed beneficiaries in their communities. Beneficiaries were able to cite several benefits from the activity including soil testing, and provision of compost, fertilizer, fencing, and seedlings. A few beneficiaries were able to directly connect the receipt of these services to increased profits. “The soil testing helped me to know the quantity of manure to be used in the soil for a particular crop, how to use [the fertilizer] in correct proportion, etc. As a result, my yield increased and expenditure on manure decreased. My yield increased by around 40 percent while the expenses on manure decreased by half. This resulted in more profit from farming.” Male KII Respondent – KRA 3 Beneficiary However, other farmers noted that while they received the inputs and a report on soil quality, they were not well instructed on how to understand and utilize the report. In general a lack of integration hinders the effective utilization of the various types of support farmers are getting under the activity. For example, many FGD participants noted that water scarcity was a large issue in utilizing the inputs provided, particularly planning and maturing some of the tree crops. Four 20 FY 2019 Portfolio Review TSRR AID-386-G-00001 (October 1, 2018 to September 30, 2019). 17 | TASKING TO50: TSRR ASSESSMENT USAID.GOV respondents mentioned the issue of organic farming, which the CTA tried to introduce at the request of the community, however, there was limited market demand and thus it was ultimately unsuccessful: “Organic farming was a demand from the community; later these organic farming products were not linked to the market very well plus the yield was low too due to which farmers could not earn much and therefore they went back to the traditional farming techniques of using fertilizers and pesticides. They should have continued with organic farming and we could have worked toward finding solution for better market linkages.” Male KII Respondent, Community Leader Other respondents noted that while organic fertilizer had been provided, it reached farmers after planting was complete and therefore could not be used for that growing season and had to be saved for next year’s planting. Finding 3.2: Trainings, while increasing general knowledge, lack specificity and follow-up to expand their impact. Under TSRR, the recipients of trainings for farmers noted that training in plant cultivation and animal husbandry had been provided. Some beneficiaries could provide clear examples of how trainings improved their livelihoods, with one individual noting, “After the training I learned how to buy good quality cows, and milking techniques too… Milk production has increased by one liter for each cow.” However, participants also noted that the trainings were somewhat limited in scope. There was an interest to learn additional information about cash crops, like black pepper and coffee, and/or crops that were not water-intensive due to local shortages. In addition, even when information was viewed as relevant, farmers expressed a need for additional follow-up or the provision of written materials to help them retain the information. Finally, similar to Finding 3.1 above and despite the potential benefits, most respondents believe greater integration and planning, specifically through the provision of additional resources like loans for upgrading farm equipment or a viable water source to protect against drought, were necessary for future success. Finding 3.3: Agriculture is not a primary livelihood source for most respondents, with interest waning over time. In the community survey, only 10 percent of men and 6 percent of women noted that commercial farming was the primary source of income for their households, compared to approximately 40 percent for both groups reporting local or seasonal business. Additionally, those receiving income from farming were, on average, 10 years older than those receiving income from business. A 2017 Agriculture Assessment study notes that farming accounts for just 8.5 percent of aggregate household income.21 While these results should be interpreted with caution given the non-random selection of participants, including that the survey largely took place in public markets or trading centers, they are backed up by sentiments of FGD participants. Specifically, participants noted that farming was primarily done by the older generation despite a 2010 survey in which 52 percent of settlement youth reported an interest in agricultures.22 Many of those who initially expressed interest in participating in KRA 3 ended up withdrawing their candidacy. Farmers found it increasing challenging to meet the needs of their 21 Ingenus Strategy & Creative Research LLP: Transforming Tibetan Agriculture (16th October 2017). 22 Ibid. USAID.GOV TASKING N050: TSRR EVALUATION | 18 household, including paying for their children’s education, due to insufficient profit. These respondents noted huge challenges for the future of agriculture in their community, specifically in soil quality and irrigation, but most felt they were too old to find an alternative income source. KRA 4: BASIC EDUCATION STRENGTHENING Education was the highest rated area when community surveys respondents were asked, “If you were to receive additional support in one area, which would improve your daily life?” Thirty-seven percent of both male and female respondents put it first. In 2012, USAID began supporting basic education strengthening for Tibetan children with the Tibet Education Project. Under KRA 4, TSRR supported the continuation of strategies which began under the Project in FY17-18 while conducting a diagnostic assessment of next steps for TSRR.23 Key recommendations from the assessment included extending the existing Early Grade Reading (EGR) model, integrating ICT in classroom instruction, providing high quality STEM instructional training and materials, and improved training and professional opportunities, particularly among head teachers.24 Analysis of TSRR’s performance monitoring data from the Q4 QPR indicates that the majority of project targets have been met over the first three years, including the number of teachers accessing professional development (2,835 total) and the number of learners reached in reading programs funded by US government assistance (17,742 total).25 However, the number of teaching and learning materials provided was lower than anticipated for the first two years (167,134 total provided versus 482,850 targeted), but gained momentum in Year 3 (150,837 provided versus 114,100 targeted).26 Finding 4.1: Improvements in education in Tibetan schools, particularly reading, were attributed to TSRR’s delivery of teacher trainings, Tibetan reading materials, and infrastructure improvements. Participants found the professional development trainings and workshops for teachers were useful in many ways. During the interviews and FGDs, teachers recalled that trainings brought new perspectives, including the importance of continuous learning and the use of innovative, child-centered methods to make teaching more effective. Practical outputs of trainings included the introduction of peer observation and feedback, renovation and increased use of the library systems, and revisions in teaching methods and homework assignments to refocus on the student. “After the library workshop, we have renovated the library. Children now come to the library during free time… [whereas earlier] even the librarian also did not feel like coming to the library.” Female KII Respondent, KRA 4 Beneficiary Infrastructure improvements and the provision of new materials were believed by multiple respondents to be instrumental in creating an environment for learning, with one school administrator noting, “We also got smart class projector, furniture, science labs, shelves, windows, and doors etc. provided by USAID. This has been really helpful to create an environment where students can study in an encouraging and secure environment. My school has improved so much!” While valuable, most respondents also noted the importance of continuing education and mentoring support for teachers, especially as new teachers enter the system. In addition, while many of the school improvements and 23 TSRR AID-386-G-16-00001 Quarterly Report FY19 Q4. 24 Ibid. 25 TSRR AID-386-G-16-00001 Quarterly Report FY19 Q4 Annex 1. 26 Ibid. 19 | TASKING TO50: TSRR ASSESSMENT USAID.GOV materials were specifically tied to education, at least three respondents noted the importance of hygiene in creating a conducive learning environment. This may be especially important for encouraging enrollment of young girls, with one school administrator requesting incineration and sanitary supply vending machines. Finding 4.2: Formal education presents an opportunity to share and enrich Tibetan culture, particularly given the emphasis on Tibetan language materials. The introduction of EGR materials in the Tibetan language is has benefits beyond basic literacy skills. School administrators noted that access to the Tibetan language from an early grade will assist in promoting Tibetan culture. In Nepal, respondents see this as an opportunity to extend the Tibetan language and culture beyond the Tibetan community. Given the integration of Tibetans with the larger Himalayan population, they note that the classroom is more integrated and has some Nepalese and Himalayans of distant Tibetan dissent. One implementer said, “Only by getting everyone involved and supporting the Tibetan and Himalayan culture will it be able to thrive.” Finding 4.3: The role of education extends beyond the classroom, including parental support for learning objectives and counseling for students from difficult backgrounds to improve performance and retention. Interviews among parents and community leaders note that the value the community places on education has grown over time, especially as parents become more literate. One settlement officer even noted, “Ever since this intervention, there has been a change. People are empowered… Now even the small children are excited to read books. Even I was not reading books… now I read.” However, these sentiments were tempered by views from educators in the community. Teachers noted that parents’ understanding of EGR was low, but that that basic education to parents on concepts such as bedtime reading, as well as community engagement through play and drama, could help advance EGR techniques. In addition, teachers and school administrators expressed concerns with being able to handle personal and home issues that affect students’ ability to concentrate on education. In addition to traditional teacher training, respondents felt it was important that teachers receive counseling skills, or that school counselors were provided in order to support at-risk youth. KRA 5: HIGHER EDUCATION Under KRA 5, the TSRR program reports it has provided approximately 900 scholarships for students, some of which may have received a scholarship more than once, to pursue bachelor’s and post-graduate degrees at universities in India. This is about 85 percent achievement of the total target, 1,055 scholarships, for the first three years. Scholarships are awarded on the basis of academic merit. There are no special quotas for females but about 60 percent of the scholarship recipients are female. Student retention rates have been high, with a 80-90 percent completion rate for TSRR beneficiaries.27 In addition, TSRR provides test preparation coaching under this KRA for students preparing to take higher education entrance exams. Outputs for test preparation coaching have not been as high as targeted, with 1,151 beneficiaries reached versus a 1,753 target, taking into account double-counting of 27 Ibid. USAID.GOV TASKING N050: TSRR EVALUATION | 20 students who received scholarships for more than one year of studies. There are no outcome-level indicators for this activity, so there is currently no indication about the impact of the activity.28 Finding 5.1: Scholarships enabled more Tibetan students to access higher education opportunities. Respondents who were KRA 5 stakeholders had very high praise for the scholarship activity, feeling that increased students’ access to higher education opportunities. Students who were receiving scholarships indicated that, without this scholarship, they either would not be able to pursue a degree or their tuition would have imposed a significant financial burden on their families. Many respondents, even those benefitting from other types of TSRR support, testified to the growing importance that Tibetan culture places on higher education and post-secondary studies. This is in comparison to many of the older generations who were not able to benefit from such opportunities and thus had limited choices in their own livelihood pursuits. Therefore, the importance of quality educational opportunities for young people represents a measure of the Tibetan communities’ vitality and resilience, and granting the scholarships has a significance beyond the tuition payments. Finding 5.2: Scholarships especially benefited female youth and expanded their empowerment and professional opportunities. As mentioned above, respondents indicated that more females than males are benefitting from the scholarships. This is supported by TSRR’s monitoring records. Respondents felt the high rate of female beneficiaries was a good thing because it increased access to higher education and reduced the financial burden on families that would otherwise act as a barrier to female higher education participation. “I’ve never imagined that girls are being educated so much – there used to be so many boundaries and limitations, but these have been challenged because of these funds. Parents don’t have as many barriers to educate their females. Maybe it wasn’t the intention at the start, but it’s led to greater gender equality in the community because females are also studying so they have more say in their community.” Female KII respondent – KRA 5 Beneficiary Even among young beneficiaries themselves, the effect of financial support for young women to access a tertiary education has had a noticeable impact. Finding 5.3: Timing of USAID funding was not coordinated with academic payment calendars. One of the only critiques respondents had of the scholarship program is that the timing of when implementing partners received their funding did not coincide well with when students’ tuition and fees were due. Students had to find alternate arrangements, such negotiating with their schools or paying up front and getting reimbursed later in the year. This was illustrated by the following feedback from a female KRA 5 beneficiary: “The support came in late and I had to manage until it came, but I did. I spoke to the administration if I can pay the fees late. And they agreed. The money came in between semesters.” A CTA official concurred, noting, “The general approval comes in September, but we have to start scholarships in May-June. We can’t provide the money when they need it because the money arrives only in December or sometime like this.” This indicates a misalignment between activity 28 Ibid. 21 | TASKING TO50: TSRR ASSESSMENT USAID.GOV timelines and beneficiaries’ obligations. While this may also be affected by internal systems within the Department of Education and/or USAID systems, the ultimate reality is that students’ and their families’ capacity to make timely payments for school fees are affected. Implementing partners acknowledge this was an issue and indicate that steps have been taken to resolve it, although this was not able to be observed directly. Finding 5.4: Measuring outcomes of KRA 5 is challenging for TSRR administrators, including the CTA. As with the outcomes of the livelihood skills training activities implemented under other KRAs, one challenge that stakeholders mentioned in KRA 5 is the difficulty of measuring the outcomes of higher education investments. While CTA staff made strong efforts to track KRA 5 beneficiaries after they graduate to determine employment rates, this has been challenging. One Department of Education staff member said, “One challenge we face is that all students, especially those under TSRR scholarships, we try to contact them, check what they have been doing after their courses, but we can’t get a lot of responses from them, so we have a hard time tracking them after the scholarship period to find out the outcomes.” This challenge matches the difficulty that the ET team had in contacting TSRR beneficiaries overall, given disconnected phone numbers and limited responses. KRA6: COMMUNITY HEALTH IMPROVEMENT Under Community Health Improvement, TSRR supports various activities through the Tibetan Health System Capacity Strengthening Project, including the Tibetan Medicare System (TMS); tuberculosis care and management services; reproductive, maternal, newborn, child and adolescent health; social and behavioral change communication (SBCC); comprehensive community outreach and coordinated care (CCOCC); optimal service model support; health workforce strengthening; and a health information system. Prior to Year 3 of TSRR, this component was implemented under a separate USAID project. Starting in Year 3, TSRR was to continue and expand this component. In Year 3, the activity largely met or exceed targets for the expansion of TMS membership, health services, and trainings. Finding 6.1: The cross-sectoral integration of SBCC training in health and education helped improve health outcomes in local communities. TSRR administered SBCC trainings to schoolteachers, health unit nurses and staff, and other health professionals. Recipients of training noted that the trainings were helpful in learning innovative methods in health communication in order to induce behavioral change. “Previously whatever health education we used to do was just the lecture method. We would sit around in a large gathering and just teach… Now we incorporate games, videos, role plan, and different channels of communication… Before only old people showed up in the trainings but now young people attend.” Female KII Respondent The role of SBCC in reaching out to youth was emphasized by several respondents, including broadening the concept of adolescent health and integrating it with reproductive and maternal child health. Collaborating with schools provided an opportunity to introduce peer-based learning. While the outcomes of these models are beyond the scope of this evaluation to measure, several respondents noted they have seen individuals applying health improvement behaviors aimed at reducing diarrhea or USAID.GOV TASKING N050: TSRR EVALUATION | 22 hypertension in their own lives. Given that behavioral change is coupled with medicine distribution, some respondents attributed a drop in viral infections to TSRR’s intervention. Finding 6.2: TMS plays a critical role in access to the secondary and tertiary health care for Tibetan communities in India and helps reduce the financial burden of accessing healthcare. TMS, funded by USAID and implemented only in India, started prior to TSRR and has expanded under the activity. Under TSRR, the annual premium for vulnerable households has been subsidized, however, hospital staff noted that the subsidy will decrease over time to ensure sustainability. During interviews, most TMS beneficiaries reported benefitting from the insurance, which covers most diseases, medications, and surgeries. TMS beneficiary respondents had all received financial reimbursements for their medical claims, which has helped to reduce their out of pocket expenditure. At the same time, several noted that while registration was simple, reimbursement on claims could be time consuming, requiring respondents to take out loans to cover the costs in the interim. Additionally, while this evaluation focused on direct beneficiaries, respondents also noted that not everyone in the community was covered under the TMS. TSRR data reports current enrollment in TMS at 37 percent of the Tibetan population in India.29 Several reasons for non-coverage were provided, including a lack of awareness about the benefits of insurance, a feeling in the community that the benefits had been reduced over time, and coverage by other health schemes. One respondent also felt that the program lacked flexibility in comparison to private health insurance service providers. This individual suggested that TMS should have different premium amounts for different packages, some of which would allow a higher annual coverage and that individual coverage should be made available and that coverage should be less than the premium for family coverage.30 While noted as desirable by FGD respondents, Tibetan settlements in Nepal do not have access to TMS. Implementing partners provide financial assistance to help cover medical costs but most respondents noted they had to pay out of pocket. In Kathmandu, 50 percent of respondents noted additional support in healthcare as their primary future need, as compared to 24 percent in Pokhara and 23 percent overall. Finding 6.3: CCOCC has effectively introduced preventative health care into communities but is constrained by staffing resources. Recognizing the critical role of preventive healthcare, CCOCC mobilizes health professionals to make home visits, improving health care access to over 75,000 Tibetans in Nepal and India, which is approximately 87 percent of the population.31 Some FGD beneficiaries recalled that home visits had identified previously undetected health concerns, such as high blood sugar, and directed them to receive additional follow-up care. A CTA official mentioned that the program has made strides in early recognition of issues like high blood pressure by educating the community on early signs. Respondents said this service was critical, given that access to preventative medicine is relatively low among the Tibetan population and clinic referrals to more expensive hospitals can deter patients from seeking early 29 Program for Self Reliance and Resilience of Tibetan Communities in South Asia - Key Achievements FY17-19 Presentation. 30 The CTA notes that individual enrollment is feasible but only for households with one individual. Households or group living arrangements (e.g. monasteries) must enroll in the family plan. (https://tibetanhealth.org/tms-frequently-asked-questions/ Accessed: Feb 3, 2020). 31 Ibid. 23 | TASKING TO50: TSRR ASSESSMENT USAID.GOV care. However, interviews revealed that the program continues to face some challenges, such as resource constraints. In large and/or scattered settlements respondents noted that the number of nurses were insufficient to cover the number of households each month. In addition, while one health staffer noted being encouraged by the support offered by TSRR, they also stated that staff turnover continued to be an issue, which they attributed to low salaries. Turnover required additional training resources for replacements, making it difficult to “upgrade” the standards provided to the community. Finding 6.4: While primary health care has improved under TSRR, greater specialization was lacking, especially for women’s services. There are 38 settlement health facilities in India and 12 facilities in Nepal. Respondents indicated that these clinics/hospitals remain the first point of contact for diagnosis and treatment for the Tibetan community. Eighty-five percent of respondents in the community survey said they agreed that their primary health needs were met, though men were twice as likely to disagree than women—20 percent versus 10 percent. CTA officials noted that, under TSRR, support is given to these facilities, such as staff capacity building, provision of instruments, medicines, staff for community outreach, and the provision of the online health management information system. The majority of the beneficiaries interviewed noted that the facilities in their settlement clinic had increased over recent years. While recognizing these improvements, both community members and health professionals also noted that the basic care provided by settlement clinics was insufficient. Respondents advocated for more specialized care from professionals on a rotating or weekly basis, if not full time. One FGD participant stated: “Women in Tibetan communities are empowered, but when we go to the hospitals for delivery there are no gynecologists; specialized doctors for women’s health is a major requirement.” In addition, according to health care providers, the introduction of new tools, such as the Health Information System, provide underutilized opportunities to explore the data and identify settlement-specific diseases from which to build health care strategies. Finally, some respondents noted that medical centers should be better at incorporating traditional Tibetan medicine. KRA 7: PARTICIPATORY COMMUNITY DEVELOPMENT The PNA is a participatory process through which TSRR determines the needs of community beneficiaries and selects which activities to fund. The PNA process begins with a community meeting, managed by a trained facilitator, in which community members put forth ideas for potential projects. CTA and SARD analyze the proposed activities, taking into account technical and cost merit, and select the most promising ideas for funding under TSRR. In the Midline Report, the activity reports that 26 PNA meetings have been conducted, which exceeds the target of 23. Finding 7.1: The original design of the PNA was not effective and had to be overhauled in Year 2. The PNA process faced several challenges in the first two years of implementation, many of which are documented in TSRR’s periodic reports and annual planning documents. The FY19 Annual Work Plan includes the following analysis of PNA’s performance: “Implementation of this activity component has faced multiple challenges and achieved poor results in FY17-18. Problems stemming from low capacity caused long delays between initial community meetings and completion of feasibility analyses … Because some communities were not informed early enough that construction is not allowable, unmet USAID.GOV TASKING N050: TSRR EVALUATION | 24 expectations resulted when construction-related priorities were not approved for program support. In addition, household participation in PNAs and ownership of final plans were low overall, due to inexperience and limited capacity of implementation teams.”32 These challenges were also echoed by respondents during field visits, including this statement from one senior implementation manager: “Participatory community planning was a disaster.” Other respondents mentioned specific challenges that generally echoed the Work Plan’s conclusions quoted above, including non-allowance of construction projects, project ideas that lacked dispersion among the sectors (most were focused exclusively on livelihoods, with fewer suggestions focused on health or education), and conflicting political interests. Regarding political interests, most of the respondents who discussed this issue indicated that they felt that settlement officers had the final say on projects and sometimes made decisions based on their own interests, however, one respondent felt that the projects that were not selected did not match the CTA’s political priorities at the central level. As a result of these challenges, TSRR’s management team overhauled the PNA process for implementation in the latter half of 2019. TSRR’s FY2019 Portfolio Review document states that these changes to the PNA include: “decentralizing needs identification and prioritization to communities at the settlement level and increasing local participation and ownership of local development planning.”33 While this process had started at the time of this evaluation, it had not progressed to the point where it could be observed and evaluated for this study. The PNA will be a continuing issue for TSRR to monitor. Finding 7.2: Evidence of equal inclusion of stakeholder voices in the PNAs was mixed. While respondents who discussed the inclusion of diverse voices in the PNA process indicated that women, youth, and other marginalized groups were engaged in the process equally, there were no youth respondents available for KRA 7, and youth beneficiaries who were interviewed for other KRAs had not been engaged in a PNA. All of the available respondents from this KRA were men, but most of them believed that women’s interests and viewpoints were considered. However, in one settlement in Nepal, a female school administrator had not been invited to the discussion, which is a serious omission in considering the needs of the education sector in the strategic planning process. Finding 7.3: Beneficiary communities value and desire a community-driven process to select projects for funding. It is clear from the data gathered that community-level stakeholders are supportive of the participatory community planning process in general, despite the challenges. They recognize the need and benefit for the community to determine their own needs and receive tailored support to address them, as demonstrated by the following comment from one male Community Leader: “If the programs are not in line with the needs of the community, then they are not sustainable and are in fact meaningless. That is why the people demanded that their needs should be incorporated in project planning and implementation. That is how we shifted from a top-down to a bottom-up approach.” Based on this feedback, TSRR beneficiaries clearly value an opportunity to contribute to decision-making processes in their community development efforts. However, that not all voices were included in the 32 TSRR FY2019 Annual Work Plan, p. 54. 33 TSRR FY2019 Portfolio Review, p. 5. 25 | TASKING TO50: TSRR ASSESSMENT USAID.GOV process in the past, as only 60 percent of survey respondents indicated that they were aware of development projects happening in their settlements, and only half said they had been actively involved in the planning process. Enhanced communication and inclusion of all stakeholders will promote a more inclusive, and therefore representative, process. TSRR’s experience with PNAs in the first two years of implementation, the challenges they have faced, and the evolving approach they are undertaking to make the PNA more effective, are all lessons learned in community-driven development models and are an opportunity to establish best practice approaches for similar efforts in the future. KRA 8: WOMEN’S EMPOWERMENT Under KRA 8, TSRR implements a number of cross-cutting activities to promote women’s empowerment, including capacity building for the CTA’s Women’s Empowerment Desk, gender and leadership trainings, and awareness raising workshops that focus on gender-related issues such as gender-based violence (GBV). This component started out slowly in Year 1, reaching only 30 of the targeted 500 beneficiaries.34 However ,it has gathered momentum over the past two years and now reaches a reported 883 beneficiaries, some of which may be double counted.35 Finding 8.1: TSRR built the capacity of the Women’s Empowerment Desk at the CTA Central Office. Respondents cited TSRR’s work with the Women’s Empowerment Desk at the CTA Central Office as expanding the visibility and understanding of women’s issues at the community level, and promoting women’s empowerment as a focus area within the CTA’s work at a cross-sectoral level. “Since the USAID project started, we have more human resources; that is helping us build the capacity… Our desk is not only focused on women only but it focuses on overall gender issues. The knowledge about the desk is minimal in the settlement. That is where the challenge was and is. In the city itself, issues of gender, participation of women in the decision making, for all of that we needed support. In the last two years, we have done a lot. In the last two years, I feel we have moved somewhere.” Female KII Respondent – CTA Women’s Empowerment Desk As the respondent mentions, while there has been progress at the central level, this has not necessarily trickled down to the local level. Her observation matches the ET’s field observations, where the ET saw less focus on gender within the settlement offices overall. No staff dedicated to gender issues or women’s empowerment were observed at the local level. Finding 8.2: Gender Training and Sensitization Workshops are changing mindsets, but slowly. Beneficiaries reported that TSRR’s gender training was of good quality and had a positive impact on the development of the participants. Training participants indicated that their own awareness of GBV issues increased, and they are disseminating that knowledge to their community. As one female health clinic staff member who had taken GBV training in Nepal noted: “[I took] GBV training last year and I have given gender-based violence trainings in my community three to four times, including in schools … I gained a lot of awareness about 34 TSRR AID-386-G-16-00001 Quarterly Report FY19 Q4 Annex 1. 35 TSRR AID-386-G-16-00001 Quarterly Report FY19 Q4 Annex 1. USAID.GOV TASKING N050: TSRR EVALUATION | 26 this issue in my community and that people should intervene whenever they see these things starting between a wife and husband … Now I am able to view things that in the past, I wouldn’t have viewed as violence and to recognize them as such due to the training. I haven’t seen as much change within the community after the GBV training, but there has been some change.” Female KII Respondent – KRA 6 Health Clinic Staff Beneficiary Another respondent from India who had taken the Training of Trainers (TOT) on Gender- and Sexual￾Based Violence reported that her communication skills and capacity as a trainer had been strengthened by the experience: “I think I am more effective in the way of giving training. It made an immense difference on my skill so it made a difference in the community I am working in.” Finding 8.3: Community members were skeptical about the need for women’s empowerment support in Tibetan communities overall. The sense in the general community is that female empowerment is not an issue; a majority of respondents said that women do not face gender-based challenges in the Tibetan community. The only respondents who acknowledged any gender-based inequalities were either implementing partners or highly-educated beneficiaries. When asked specifically about gender-equality topics like GBV, some respondents acknowledged that these issues existed in their community but said generally women were considered to have fully equal rights and responsibilities in Tibetan society. While it is true that women have high standing and mostly have high freedom of movement in Tibetan culture, there are still biases to overcome. Many beneficiaries displayed strong resistance to the possibility that there were gender￾based concerns. When asked about this reticence, one respondent explained it as follows: “Listen to people and what they say about society; you’ll hear about domestic violence, sexual harassment, especially. But Tibetans are more silent about these things because they have more things to lose than to gain about expressing them. People are scared to have a bad image. People from India, an independent country, they have nothing to lose if they admit these things exist. In the US, your government will still function, you will still have water to drink. But for our people, especially the Tibetan Muslims, they will never express these things because they have too much to lose by sharing them.” Male KII Respondent – KRA 5 Beneficiary Despite these social misgivings, respondents directly involved in gender-based issues stressed the need for continued support for women’s empowerment activities, especially for men. The rate of male participation in gender-focused trainings has been much lower than female participation. Records indicate that the June 2019 TOT on Gender and Sexual Based Violence training had 17 participants and only one participant was male. While TSRR’s Annual Performance Data does not provide a breakdown of gender trainees by sex, upon further inquiry with TSRR the program reported that 36 percent of gender training beneficiaries are men, which indicates that the ratio of males to females in gender training can be increased. KRAs 9-11 work to support institutional capacity development and are explored under EQ 2. 27 | TASKING TO50: TSRR ASSESSMENT USAID.GOV 3.1.2 EVALUATION QUESTION 2: TO WHAT EXTENT ARE THE SYSTEMS AND PROCESSES INTRODUCED FOR STRENGTHENING THE CAPACITY OF TIBETAN INSTITUTIONS BEING ADOPTED AND INSTITUTIONALIZED BY THESE ORGANIZATIONS? TSRR’s work to support institutional capacity development was focused under KRAs 9 (Institutional Capacity Development), 10 (Leadership Development) and 11 (Cultural Institution and Resilience Training), with capacity development training for the CTA central office, including the SARD team, as well as CTA local staff in the settlement offices. KRA 9 provides training for CTA staff on program design and management, including monitoring and evaluation, as well as communication and workplace management, software, and other business-related skills. KRA 10 provides leadership training for mid￾and senior-level CTA staff, as well as those aspiring to advance to senior levels within the CTA. Training opportunities include 20 days of training at OP Jindal University in Sonipat, India, focused on leadership development and organizational and workplace conflict management; and a ten-day training at the Himachal Institute of Public Administration (HIPA) in Shimla, India. KRA 11 provides support to Tibetan cultural institutions. The next section presents the evaluation findings for each of these KRAs. At the time of this evaluation, TSRR did not conduct outcome-level testing of knowledge gains for these trainings, so the primary data demonstrating theses outcomes are TSRR’s self-reported data. KRA 9: INSTITUTIONAL CAPACITY DEVELOPMENT Findings: TSRR’s training beneficiaries at both the central and local settlement CTA offices all spoke highly of the Institutional Capacity Development training and could identify specific examples of how it affected their capacity development. Most respondents felt that their performance had improved over the last couple of years and only a few said they had observed no change. TSRR beneficiary respondents indicated that the CTA’s performance had improved over the past three years, specifically in communications and service provision. One FGD respondent in India stated: “The workshop on core competencies was very helpful and [included] many new learnings. Before, we never thought about the challenges we face in giving presentations and public speaking…from the training, we learned techniques to overcome those challenges. That definitely improved my efficiency in work, and my work performance also improved.” The quantitative survey also explored the extent to which community members felt that the capacity of the central and local CTA staff had changed. While the TSRR baseline report is not directly comparable to this evaluation’s quantitative survey, it is noteworthy that the CTA’s performance is rated higher in this midline survey than in the baseline. The baseline reported that “40 percent of Tibetans reported they are effectively supported by local Tibetan institutions, particularly the CTA.” The midterm survey conducted found that 54 percent of respondents felt the CTA provides adequate support for livelihoods development, 49 percent felt the CTA provides adequate support for welfare and other special needs, and 85 percent said the CTA does a good job to promote and preserve culture. These numbers indicate an upward trend in the level of satisfaction that Tibetans feel for the CTA’s services. However, it is worth noting that at the settlement level, the qualitative interviews revealed some variations in the effect of the capacity development work. For settlements in which the settlement officer had been in their job for less than six months, the benefits were not as apparent as in settlements in which the settlement officer had more longevity. This also applied to lower-level staff – accountants, in particular, seemed prone to high turnover. Most of the settlement officers the ET met with, including those who had been in their positions for more than a year, indicated that there was a need for more continuous training of their staff. USAID.GOV TASKING N050: TSRR EVALUATION | 28 KRA 10: LEADERSHIP DEVELOPMENT Findings: Respondents’ feedback for KRA 10 was similar in nature to that under KRA 9, with the three beneficiaries who had received this training praising the quality of the training and providing specific feedback about new skills they had applied on the job. One male Settlement Officer shared the following comparative analysis of the two leadership training activities: “I attended leadership training at OP Jindal Institute for 15 days and administrative training at HIPA for 10 days. At HIPA, I learned how to make decisions, how to connect with people and e-governance mechanisms to improve administration. HIPA was more useful because they taught us more practical issues than OP Jindal. Also, the facilitators at HIPA were very resourceful and the activities very informative.” Male KII Respondent – Settlement Officer KRA 10 Beneficiary KRA 11: CULTURAL INSTITUTION AND RESILIENCE STRENGTHENING KRA 11 was designed to achieve increased cultural vitality and resilience of the Tibetan communities. Major recipients of the activity support were cultural institutions in India and Nepal and full-time practitioners preserving and promoting Tibetan culture. Key activities under KRA 11 are Tibetan performing arts revitalization, supported through the establishment of the Tibetan museum, the creation of awards through the Tibetan Arts and Culture Fund, and support to the Himalayan Buddhist Library and Cultural Center (HBLCC). Finding 11.1: Support to the Tibetan Institute of Performing Arts (TIPA) in India supports the provision of quality arts programming and strengthening institutional sustainability, but the HBLCC in Nepal needs continued support. TIPA, established in 1959, is a mainstay of classical Tibetan arts, and its high self-sufficiency (it generated 92 percent of its own revenue at the start of TSRR, according to the QPR) was not surprising. Its presence in Dharamsala is strong, and a range of respondents felt that TSRR support has given them more opportunities to support their artists, organize cultural events, and revitalize performing arts in the community. HBLCC in Nepal, on the other hand, was established in 2017 and generated none of its own revenue at baseline. Nonetheless, work to preserve Tibetan-Himalayan Cultural heritage and strengthen ties between Tibetan-Himalayan people was appreciated by respondents. “The cultural center received funding to enhance this place. There is a museum here and important books are being translated. Under KRA 11, we have summer and winter camps for cultural networking. Under the trans-Himalayan studies, different sects discuss how Buddhism is to be transformed in the younger generation. Classes on Art and Mindfulness and inter-cultural learning have been very successful.” Female KII Respondent – KRA 11 Beneficiary However, respondents also expressed concern that TSRR support would be constrained given the political environment in which the HBLCC operates. Despite early assessments, further review of the operating environment and continued financial and technical support is considered necessary to make the HBLCC successful and sustainable after the activity. 29 | TASKING TO50: TSRR ASSESSMENT USAID.GOV Finding 11.2: Though support to artists and institutions provided impetus to increase cultural vitality, the activity needs to expand its reach in the Tibetan communities. TSRR helped increase cultural vitality and resilience through support to new and existing artists at TIPA, grants to full-time practitioners, and institutes promoting Tibetan culture and language. The activity’s performance indicators report a high percentage of artists’ believing their skills and capabilities improved after receiving US government-supported training. The support to artists at TIPA was aimed at revitalizing the Tibetan performance arts. Respondents, particularly in Dharamsala, felt that TIPA provided more opportunities and activities compared to before the support started. TSRR supported the creation of the Tibetan Arts and Culture Fund at the CTA Department of Religion and Culture to strengthen cultural resilience by inviting applications from Tibetans working in both traditional and contemporary arts and culture. A beneficiary of the Tibetan Arts and Culture Fund grant shared that their work product wouldn’t have been possible without TSRR funding and felt the activity has helped increase cultural resilience by preserving Tibetan heritage and practices. However, they also noted that their work would primarily appeal to academia, and not necessarily be accessible for the Tibetan community at large. Respondents noted that KRA 11 does not implement activities directly at the settlement or community level, although the TSRR program noted they do provide support for cultural activities (e.g. musical instruments in schools) in the community through the PNA process under KRA 7.36 Many settlements have local cultural institutions such as opera/music and dance associations and settlement officers play a significant role in organizing, celebrating, and observing different elements of Tibetan culture. Local cultural institutions in the settlements, school and colleges could be targeted by TSRR to expand its reach in the Tibetan communities. While there are some concerns that KRA 11 resources could be better deployed among multiple institutions and groups, rather than just a few, there are some encouraging outcomes of the activity considering the integration of multiple KRAs in increasing Tibetan cultural and vitality. In several of the finding sections above, the evaluation noted opportunities for the preservation of Tibetan dress (KRA 1), the teaching and expansion of the Tibetan language (KRA 4), and the potential integration of traditional medicine (KRA 6). Finding 11.3: CTA is doing a good job to promote and preserve Tibetan culture, especially by integrating cultural opportunities into education, but more can be done to recognize how the Tibetan population, and thus its culture, is changing over time. Eighty-four percent of community survey participants either agreed or strongly agreed that CTA is doing a good job promoting and preserving Tibetan culture, although this is largely independent of direct TSRR support. Similar sentiments were echoed by a range of beneficiaries during discussions and KIIs. A participant of an FGD in India stated, “CTA is doing good job with preserving Tibetan culture and it starts with schooling. Children are taught about Tibetan culture, practices, history, and dances.” FGD respondents in Nepal agreed that Tibetan culture will be strengthened through education, especially religious and cultural, and when native language is preserved. Respondents also added that Settlement Offices have played a key role in marking important cultural days and Tibetan practices in the 36 TSRR AID-386-G-16-00001 Quarterly Report FY19 Q4. USAID.GOV TASKING N050: TSRR EVALUATION | 30 community. Nevertheless, a scholarship beneficiary suggested that CTA should expand its view of culture beyond its traditional narrative. “We should have more diversity. The issue of Tibetans in exile – it’s undocumented. The CTA tries to preserve a particular kind of culture, not giving importance of the evolution of culture in our society… if we start paying attention to our reality, it is in exile, and the hardships we face.” Male KII Respondent – Scholarship Beneficiary Overall, respondents suggested that CTA invest more on cultural and religious education, preserving Tibetan language and intercultural learning through translation of literature into Tibetan languages that are comprehensible to everyone. Finding 11.4: Emigration, globalization, and assimilation are major challenges to increasing cultural vitality and resilience. As the world gets more connected through communication technology, and cultural beliefs and practices are exchanged at an accelerated rate, the Tibetan communities are evolving too. More Tibetans are migrating from the settlements to other countries in search of better lives. Respondents expressed concerns that as people migrate and scatter in different countries, there are greater chances of them assimilating to a new culture. Concerns regarding assimilation also remain in the settlements as younger generations create their identity in exile. A respondent stated, “after 20-30 years, Tibetan identity will be completely changed, getting married to Indians, Nepalese and we need to accept our difference.” Another respondent added that their generation was assimilating into the more dominant culture, but that teachers and families are trying to preserve their culture and language. A majority of the respondents expected migration from the settlements to continue. While some implementors noted that the activity tried to reduce migration, or at least provide economic opportunities to retain community members, several respondents expressed skepticism of this approach, especially for the younger generation. While one FGD respondent stated, “If the settlement develops, then the people will stay together and the culture will be strengthened,” others believe it is important for the CTA to recognize the unique challenges and changes being brought by the diaspora. 3.1.3 EVALUATION QUESTION 3: TO WHAT EXTENT IS THE INTEGRATED MULTI-SECTORAL APPROACH AN EFFECTIVE MODEL FOR RESPONDING TO THE THEORY OF CHANGE PUT FORWARD BY THE PROGRAM? Findings: Based on the activity’s Baseline Report, TSRR’s theory of change posits that if multi-sectoral support is provided to the Tibetan communities in South Asia – “combining strategies to increase household incomes and employment, improve basic education quality, increase access to higher education, and improve community health … integrated with community-based planning to maximize local participation and ownership of the development process” - then the self-reliance and resilience of these communities will be strengthened. The report also states, “A gender component incorporates women’s perspectives and participation in the design and implementation of all strategies. In parallel, the program aims to strengthen the key Tibetan institutions essential to ensuring the continuity of effective 31 | TASKING TO50: TSRR ASSESSMENT USAID.GOV program strategies and long-term impact… [and] a small grants fund focuses on Tibetan arts, language and culture”37 While TSRR’s multi-sectoral approach is an innovative and comprehensive approach to promoting the vitality of Tibetan communities, feedback shared by a few stakeholders, including top CTA officials, indicated that they felt the implementation approach thus far has been mostly a collection of services rather than an integrated strategy at all levels. The ET did identify some beneficiaries or families who were benefitting from support in more than one sector, but implementing partners indicated that this was not a systemic or targeted approach to recruitment and service provision within the community. Those beneficiaries who were receiving multi-sectoral support were among the minority of respondents. They often had better networks to learn about opportunities on their own, including one example of a family in which two members of the household work for the CTA, so they are aware of potential services. The three female children all have scholarships for post-secondary studies, the grandparents are on TMS and get a wide range of medical services, and the mother has received a loan to develop her sweater-selling business. While this level of participation is a model that illustrates the potential that a holistic approach can offer—targeting the most vulnerable members of a community with this range of services will support their enhanced self-reliance—it also suggests that only those with strong connections to the CTA are well-positioned to take advantage of the opportunities. The same approach was found within the capacity building work for Tibetan institutions: Respondents in the central CTA office indicated there are cross-sectoral collaborations happening between sectors where there are some natural synergies, but this is neither systemic nor mandated at the managerial level. While this approach does trickle down to the community level, there are some notable communities in which the settlement officer and his team have elected to adopt a different approach to beneficiary selection and holistic service provision that targets the most vulnerable community members with an array of services that meet their needs across sectors. A few respondents, including two CTA officials and one implementing partner representative, indicated that they felt TSRR’s strategic planning could better integrate Tibetan cultural vitality as the central point from which all KRAs emanate. They felt that as TSRR is currently conceptualized, the various KRAs are a collection of activities implemented under one umbrella, with education, health, and livelihoods as the primary area of focus, while other areas like gender and the arts appear to be an afterthought. The program description in the Final Grant Aid document outlines that “within each area of sectoral focus overarching strategies will be integrated with community-based planning and a cross-cutting gender component will be incorporated in design and implementation of all sustainable livelihood strategies.” Some respondents suggested that TSRR could expand its efforts to provide this cohesive set of interventions working together in a holistic manner to provide an array of services reaching the most vulnerable members of society, as described in the example of the beneficiary family provided above. The respondents felt that shifting the paradigm to focus on Tibetan culture as the center point would promote this holistic approach. In exploring implementation of the multi-sectoral approach, respondents were asked what element of TSRR tied together all the KRAs. Although TSRR is envisioned as an integrated multi-sectoral activity, one key senior CTA official felt it has not achieved this goal thus far, as 37 TSRR Program Baseline Report, p. 7. USAID.GOV TASKING N050: TSRR EVALUATION | 32 each KRA has diverged towards its sectoral goals. This response was echoed by other respondents, including implementing partners and beneficiaries. This is directly related to the issue of migration and to achieving the goal of enhancing the quality of life in the settlements in order to stem migration to other countries that may offer a better quality of life for Tibetan refugees. Since Tibetan culture is a shared element in all the settlements, the CTA official felt that culture should be the core from which all the KRAs emanate, the thread that tied together all the various sectors. In this revisioning, the culture component is not just a parallel program to strengthen key Tibetan institution, but something that will tie together the Tibetan community in exile at large in different countries as the CTA transforms into a transnational outfit with increased migration. This respondent proposed the following diagram to illustrate their idea. Figure 2: Respondent's TSRR Model This concept was echoed by other respondents, including an implementing partner representative, who shared the following comment: “I think sometimes we lack in sharing our knowledge between partners. I am not aware about what’s going on in other KRAs. We should hold some more meetings between partners to share what we are doing and to have a cross-sectoral approach. Especially in terms of arts and culture, because arts and culture is the main thing connecting all of us together. I see culture as the center of the whole project.” Male KII Respondent, Implementing Partner Another respondent, another CTA official, stated: “When you talk about culture, since the main objective of TSRR is to make the communities more self-reliant, when we have a multi-sectoral approach and see the results at the settlement level, it reduces the rate of migration to Western countries. When we do this, the people live together in a community and helps to preserve our culture. In this way TSRR is helping us to be more resilient.” Male KII Respondent - CTA Staff Representative 33 | TASKING TO50: TSRR ASSESSMENT USAID.GOV Thus, while there is clear evidence that TSRR works on the multi-sectoral level and encourages cross-sectoral collaboration, the comments and suggestions shared by respondents above suggest that beneficiaries and implementing partners feel more could be done to enhance this approach. 3.1.4 EVALUATION QUESTION 4: WHAT ARE THE KEY ASPECTS OF THE PROGRAM THAT SHOULD BE ADDRESSED AT THIS STAGE OF THE PROGRAM TO MAXIMIZE OUTCOMES OVER THE REST OF PROGRAM IMPLEMENTATION? Finding 1: Migration is a major challenge to the vitality of the Tibetan settlement communities in India and Nepal, and a major theme of the discussions that the ET had with respondents. One senior CTA official remarked that the past 15-20 years have been about migration and the CTA needs to reinvent in this context. Another CTA official explained that migration was an intentional and informed political decision in the early 1980s to boost the economy. This resulted in the Tibetan movement going global while providing exposure to youth and bringing remittances to the settlements. Respondents said that while a positive side of migration has been better livelihood for the migrants, as well as for households in the settlements, there is a growing sense of losing cultural unity as people are scattered all over the world and assimilating into their local dominant culture. The reasons for migration are not only economic but also reflect some Tibetans’ aspirations to get official citizenship or permanent residency in a favorable country. The lack of official documents curbs opportunities for many youths in India and Nepal. Although settlements in India have more favorable conditions for livelihoods improvement than Nepal, there are limitations on obtaining loans for the refugee population. In Nepal, the situation is even more challenging, as businesses cannot be legally registered and formal jobs cannot be held without citizenship documents. Respondents in Nepal were frustrated that Tibetan youths with good educations did not have the opportunity to prosper. Migration has affected the settlements in different ways. Populations in the settlements are shrinking, the number of students in Tibetan basic education schools is decreasing, settlements are turning into an hub of senior citizens, and changing social and technological dynamics are challenging the traditional ways of thinking. A CTA official stated that the administration had transformed into a transnational outfit and maintaining direct relationship with the diaspora group is the way of the future. However, community leaders and settlement officers also said that improved livelihood opportunities should be created within the settlements so that people would increase their economic resilience. Finding 2: The CTA is financially dependent on donor funds. As a government in exile, the CTA operates as an non-governmental organization (NGO) in India and therefore does not have tax revenues with which to fund activities and services for the Tibetan people; they are largely dependent on donor support. Developing the NBFC into a full financial institution would enable them to start producing revenue independently (as also discussed in EQ 1 above). TSRR and the CTA are working together to continue to expand the NBFC into a full-fledged banking institution, which senior CTA officials saw as having great promise if developed sufficiently. “When it comes to TSRR and reliance, since CTA doesn’t have income generating activities, it’s kind of an NGO working in India. To survive in the future, we need to have this USAID funding continue for a long time. Real structural issues prevent us from being USAID.GOV TASKING N050: TSRR EVALUATION | 34 self-reliant. … Non-profits do not have a scope to grow here. Given that, we started this banking. It is a full profit initiative. Incorporated separately, it gives out loans through USAID support. We are trying to aspire for a banking license so that we can accept deposits. It will be a full financial institution once it is up and running, we hope we generate some profits that goes back to CTA … The bank will be one of the biggest thing to come out of the program.” Male KII Respondent - Senior CTA Official The CTA officials interviewed indicated that they see development of the NBFC as an important step for the CTA so they can fund projects beyond those developed in response to a donor’s agenda, which can influence the scope and focus of development projects. However, the path to commercial profitability remains unclear and could benefit from additional guidance from external experts. In addition, respondents indicated that they believed having a bank that caters to Tibetan refugees could help to alleviate some of the barriers that Tibetans face in accessing finance for start-ups and business expansion projects. Just over half (54 percent) of survey respondents indicated that they have sufficient savings to cover their needs and the needs of their household if they lost their income for more than six months. Most of the settlements visited had a village savings scheme, but beneficiaries indicated they need better access to banking services that they felt the NBFC could fulfill, especially in Nepal, where Tibetans cannot access banking services like credit. Finding 3: The implementation strategy of the multi-sectoral approach is poorly integrated. As discussed in EQ 3 above, respondents encouraged TSRR to enhance the integrated multi-sectoral approach starting at the central level, down to local communities. While this is essential at the community level, they indicated that modeling the behavior starts at the management level, as evidenced by the following dialog between two CTA officials: Respondent 1: “Monthly meeting of secretaries (inter-departmental) would be very helpful … if they meet more on monthly basis than these things could be addressed.” Respondent 2: “We have program level inter-departmental meeting but there is no secretary level meeting. So that would be important.” -FGD Discussion of Male CTA Officials At the Settlement Office level, service provision is by nature more holistic as it is responding directly to the needs of the people in the community. Respondents reported that TSRR’s multi-sectoral approach enhanced their vitality, but this effect was more profound in those who had benefitted from more than one type of intervention, or their family members were benefitting from an array of services. One respondent in India reported: “In addition to apron weaving work, I am also a beneficiary of TMS health insurance. I received INR 30,000 from TMS for my treatment. It is good to have a multi-sectoral approach, and people have various needs.” This is one example that the ET found of TSRR’s array of services working together to promote resiliency in Tibetan refugee communities. There are others, including the scholarship beneficiary previously mentioned in EQ 3 whose family members were all benefitting from TSRR in one way or another, but this was not necessarily the norm and was not coordinated as part of TSRR’s selection process for beneficiaries. 35 | TASKING TO50: TSRR ASSESSMENT USAID.GOV While there was a general level of knowledge among some beneficiaries of the other opportunities available to them through TSRR, respondents felt there is an opportunity for the project to be more effective in reaching out to the most vulnerable beneficiaries and offering additional support. One prime example is the case study of the jewelry maker who had received one small loan, but her daughters did not have scholarships to go to university and her husband had not received support for his livelihood development. In some settlements, the Settlement Officer had chosen to focus beneficiary selection on vulnerable individuals, including female-headed households. 3.2 CONCLUSIONS 3.2.1 EVALUATION QUESTION 1 KRA 1: ENTREPRENEURSHIP, MICRO, AND SMALL ENTERPRISE DEVELOPMENT Overall, TSRR has made good progress in supporting the TED portfolio. Financial support to micro and small enterprise development contributed to expanding existing businesses and increasing incomes. Entrepreneurship development has been less successful, and there were concerns that the activity is not appropriately targeted and the number of high-quality applicants is insufficient. In access to enterprise finance, the activity has made good progress with the establishment of a NBFC, which could provide financial services, particularly access to loans, for Tibetans in India. This is viewed as a primary TSRR success. However, financial products are not accessible to all, and innovations to increase access, particularly using CMRFs in Nepal, have been deployed. Continued innovation, including the diversification of business, community-wide development initiatives, and access to technology to promote sales are all advocated by respondents in a quickly evolving market. KRA 2: WORKFORCE DEVELOPMENT Workforce development is one area of TSRR that has not been as well-developed as others, and outputs have been diminishing in recent quarters. There is a need in the community for livelihood skills training, with one in four survey respondents indicating they do not have adequate skills or knowledge to earn a livelihood. Skills training opportunities are seen as an important element for reaching a middle￾aged labor force with limited education, especially as the Tibetan population ages and also seeks to provide engaging economic opportunities for young people who are either joining the labor force or migrating abroad for better employment opportunities. KRA 3: AGRICULTURE DEVELOPMENT Investments in agriculture, including soil testing, fertilizer, and farmer training have all produced measurable outputs. Innovations, including community help groups have allowed individual farmers to build cooperatives and gain access to commercial markets. However, maximum growth has been constrained by poor planning (i.e., in the delivery of inputs), insufficient water supply for promoted crops (e.g. areca nut), and an aging population that is concerned about the role of agriculture (beyond subsistence farming) for the next generation. KRA 4: BASIC EDUCATION STRENGTHENING This KRA has performed very well in regards to achieving planned outputs and outcomes. Teachers clearly note an evolution in teaching techniques, and almost all community members interviewed can cite changes, particularly around EGR. Education is seen as an opportunity to pass on Tibetan culture USAID.GOV TASKING N050: TSRR EVALUATION | 36 both within Tibetan communities but also to a broader swath of the local population, especially in the Himalayan regions of Nepal. According to the survey, community members view education as a key need and opportunity for improvement. Due to their role in the community, some educators see themselves as counselors as well as teachers and advocate for greater resources with which to speak to the youth in their community. KRA 5: HIGHER EDUCATION Scholarships for Tibetan students to obtain university degrees in India have been a highly successful component of TSRR. The students interviewed were all very dedicated to their studies, appreciative of the support, and indicated that it would be very difficult, or impossible, to continue their studies without the financial support they received. The project has made significant investment in this area, with 900 students supported, and a significant proportion of those beneficiaries have been female. While this high rate of female participation was not mandated by the project’s recruitment guidelines, it has been a natural result of a greater number of female students being awarded the scholarships based on merit. This is an encouraging sign for women’s empowerment, and considered by one respondent to be a sign of greater equality for females in her community. One area for improvement within KRA 5 is coordinating the timing of TSRR funding to better match universities’ payment deadlines. Student beneficiaries and implementing partner representatives both reported that students are obliged to make arrangements with their university or pay out of pocket for their tuition and fees, and get reimbursed by TSRR later, which is a financial hardship for many. While TSRR and CTA have been working together to measure the outcomes of KRA 5, they have faced challenges in collecting longitudinal data on the employment rates of their scholarship beneficiaries. This is an area of interest to all implementing partners, including USAID, and is a challenge faced by many projects working with youth beneficiaries. KRA6: COMMUNITY HEALTH IMPROVEMENT TSRR has made notable progress through SBCC and community outreach through CCOCC. Both initiatives have helped improve access and visibility of primary healthcare and preventative measures. Health insurance provided by TMS and expanded under TSRR has provided increased coverage and access for beneficiaries and presents an opportunity for sustainable, non-donor-dependent growth in the future. Sustainability may instead be dependent on balancing annual premiums and subsidies for vulnerable populations, providing flexible plans, and increased community outreach and awareness plans. TMS is not available in Nepal, where healthcare is primarily paid out of pocket or subsidized through an implementing partner. While the activity has made marked improvements in primary care, beneficiaries noted the need for greater specialization. Specifically, women indicate healthcare as the second most critical need for livelihood improvement after education and note that most health clinics/hospitals lack gynecologists or doctors who specialize in women’s health. KRA 7: PARTICIPATORY COMMUNITY DEVELOPMENT Overall, the PNA process has faced a number of challenges that TSRR has sought to address. This issue has been well-documented in TSRR’s reports and strategic planning documents, and may continue to be a problem going forward, despite their best efforts to fix some of the challenges. Challenges arising from communication gaps or lack of understanding are easy to rectify; others such as low capacity, challenging political contexts, and lack of ownership or engagement, may not be as simple to solve. 37 | TASKING TO50: TSRR ASSESSMENT USAID.GOV Based on the field research, it is clear that beneficiaries desire a community-driven process to discuss their needs and priorities, and to agree on strategies to address them. However, there is evidence that not all stakeholders and voices are being included in those conversations, and in some cases, the process may be politically driven. As each community differs in its social and political context, TSRR’s challenge is to remain adaptive and provide strong oversight for the PNA process to ensure it operates effectively. KRA 8: WOMEN’S EMPOWERMENT TSRR’s development of the Women’s Empowerment Desk at the CTA is one prime and inarguable example of how the project has strengthened the capacity of the CTA. The Women’s Empowerment Desk is an important element in TSRR’s work, being a cross-sectoral body itself and thus similar in nature to TSRR. Given the gains that this office has made in the past couple of years, it shows great promise for being a sustainable contribution to the make-up of the CTA overall, and strengthening the organization’s focus on gender issues across departments. Gender training has also been a successful component, although the outputs are somewhat lower than originally targeted. Respondents were generally happy with the quality of the training and identified GBV as one area of greater awareness and intervention as a result of the training. 3.2.2 EVALUATION QUESTION 2 TSRR’s work in strengthening the capacity of the CTA includes a variety of training opportunities on topics that support better management, oversight, communications, and evidence-based leadership within the organization. Respondents who benefitted from this training had high praise for the experience, could provide examples of how they had applied it in their work, and expressed a desire for additional learning opportunities in the future. Supervisors also reported observing changes in their staff, and many community members reported seeing an improvement in the CTA’s functioning over the past few years. While all outcome measures for the capacity development work are self-reported, these qualitative data indicate that the project is having some impact on the CTA’s capacity. Support for cultural vitality has been primarily through financial support for TIPA in India and HBLCC in Nepal. In addition, grants are given to support Tibetan artists and to promote the preservation of Tibetan culture. The CTA’s work in cultural preservation was rated very highly both in the community survey and the qualitative data. Respondents praised CTA’s efforts to preserve Tibetan language and customs. However, issues of migration and globalization pose a challenge, especially among younger people, which is a major concern for those invested in the preservation of Tibetan culture and tradition. 3.2.3 EVALUATION QUESTION 3 The multi-sectoral approach is clearly evident in the implementation of 11 KRAs across multiple countries and communities. However, the extent to which all of these elements function together as an integrated system is somewhat less evident. The ET found examples of coordination between departments, especially education and health where there are natural synergies, but this was not systemic within the CTA at the central level. At the community level, there is more integration by nature of their service provision at the individual level. But there is no formal selection process to ensure service provision to those most in need. A USAID.GOV TASKING N050: TSRR EVALUATION | 38 multi-sectoral project offers an opportunity to provide an array of services on a holistic basis, and there are examples of this occurring, but respondents say it could be strengthened. Three respondents suggested that TSRR should place Tibetan culture at the center of the strategic approach, rather than confining it to the last KRA on the list. As understood by the ET, this did not necessarily mean that greater financial investment should go to the arts and cultural vitality component, but rather that the project should approach implementation with Tibetan culture as the central theme from which all activities emerge. For example, in the education sector it can be more provision of Tibetan language materials, in health it can be Tibetan alternative medicine; in TED it can be promotion of business linked to their culture and tradition such as barley ball making, apron-making etc. In addition, respondents indicated that TSRR’s implementation approach could better accommodate Tibetan socio￾cultural factors by facilitating trainings in Nepal rather than asking Tibetan refugees to go to India to participate in training activities, ensuring that training and materials are provided in Tibetan language, and acknowledging the importance of religion in Tibetan life while providing support (within USAID parameters where applicable), etc. 3.2.4 EVALUATION QUESTION 4 Migration is a central issue for Tibetan refugee communities. The development of these settlement communities, including livelihoods, women’s empowerment, health, education, and celebration of the cultural heritage, is seen as essential for encouraging young people to stay in their communities after graduation, to provide a lifestyle and community atmosphere that will support growing families and bring vibrancy to the settlement. While solving the issue of migration is beyond the scope of TSRR’s mandate, the work that is being done through this project is attempting to create an environment that will alleviate the issue in a holistic multi-sectoral and community-driven process. The CTA is highly dependent on donor funding for its operations. The creation of the NBFC is seen as having significant potential for reducing this dependence. Reducing the CTA’s dependency on donor investment and enabling the Tibetan refugee population greater access to banking and loan opportunities would be a significant achievement and legacy for the TSRR project. The lack of intentional coordination and integration among the KRAs is a key challenge for TSRR. Though instances of a holistic and multi-sectoral approach exist, those instances are generally not the result of intentional targeting. 3.2.5 OVERALL CONCLUSIONS TSRR overall has achieved notable successes in supporting increased resilience and self-reliance for Tibetans in India and Nepal, including: improving the provision of Tibetan language materials and increasing the quality of education in the Tibetan education system, supporting the provision of quality primary health care at the local level, supporting increased capacity of CTA staff at the central and local levels, providing scholarships to support Tibetan students’ higher education aspirations, supporting aspiring entrepreneurs to develop MSMEs, and promoting the development and archiving of Tibetan arts and culture. While increased livelihood is the primary goal of TSRR, evidence of this was observed at the individual level but was not directly observed at the community level, in part due to the many challenges and obstacles faced by Tibetan communities living in India and Nepal. 39 | TASKING TO50: TSRR ASSESSMENT USAID.GOV However, the findings of this evaluation also identify some of the issues that TSRR has faced in its implementation and continues to seek strategies to address, including the following: • There is evidence of poor coordination of support to meet local communities’ needs in the agricultural sector; • The PNA process has struggled to be implemented and some respondents felt the PNA process is politicized and not all voices are included in the decision-making process; • TSRR faces challenges in tracking the long-term effects of their interventions, especially collecting data within mobile communities of refugee beneficiaries; • Beneficiaries close to the activity recommend that culture be a more central focus of the implementation process; • The multi-sectoral approach is challenging to implement, to integrate interventions into a cohesive approach that reaches the most vulnerable populations with an array of services; • TSRR still strives to fully achieve the goal to nurture more vibrant settlement communities and combat migration trends. One of the main limitations of this evaluation is the limited scope of the field work: the ET visited six sites across two countries over three weeks within some of the world’s most rugged terrain. Travel was time consuming and the accessibility of remote village locations was a factor in site selection, as was the strong focus on collecting data from the CTA central offices in Dharamsala. TSRR is a vast and diverse activity with a wide range of services offered on a number of key sectors simultaneously. With a sample size of 159 qualitative respondents across 11 KRAs and 146 quantitative respondents to the community survey, each voice becomes a strong representative for the general experience of an average beneficiary. This limitation in the evaluation underscores the challenge that TSRR faces to implement the wide range of multi-sectoral programming across 11 KRAs to 52 sites in two countries. The model represents great potential and innovation in donor-funded programming, and TSRR benefits from a fairly flexible implementation approach that is crucial to ensure adaptability to individual site contexts. However, the level of coordination and intimate knowledge of each community’s needs required to meet TSRR’s ambitious goals necessitates a strong local presence in each of these communities to ensure that best practice standards are applied to local implementation contexts. While the ET was not able to visit most of TSRR’s implementation sites, the trends identified in the bulleted list of challenges above illustrate that this is still a challenge for TSRR. The current implementation model is that capacity development for the CTA is concentrated at the central level with additional training for Settlement Officers and their staff. The Settlement Officers are the equivalent of the local mayor of a community and have many responsibilities beyond management of TSRR. The local Settlement Offices are the main point of contact within a community and oversee general implementation, reporting back to the central CTA office and then to TTF, who archives and aggregates outputs and outcomes to report to USAID. Most of the monitoring and coordination comes from the TTF and CTA offices in Dharamsala (and in the case of Nepal, is disseminated to the three local sub-partners). This places a great deal of responsibility and control within the Settlement Office to identify implementation strategies that meet their communities’ needs; yet the PNA process has struggled to be implemented by TSRR’s own acknowledgement and is in the process of being revamped even as the activity enters the fourth year of implementation. USAID.GOV TASKING N050: TSRR EVALUATION | 40 4. RECOMMENDATIONS Based on these challenges, the ET’s overall conclusion is that the widespread range of KRAs in the multi￾sectoral approach, coupled with the broad range of implementation sites across 52 settlements in sometimes remote areas of India and Nepal, presents a huge communication and management challenge that suggests the need for increased oversight and management at the local levels to ensure that TSRR’s goal of promoting more resilient, self-reliant, and vibrant communities for Tibetans living in exile is achieved. Thus, the ET concludes that TSRR could benefit from the establishment of a local TSRR manager in each community to oversee implementation and focus on tracking outcomes at the local level. The activity may also benefit from a narrowing down of sectors to focus on the most successful activities. 4.1 EVALUATION QUESTION 1 RECOMMENDATIONS 1. TSRR should provide more legal support services for aspiring entrepreneurs for formal registration of the business, which is required to own and operate a business, especially for Tibetans who do not have citizenship status. 2. TSRR should target entrepreneurship activities on marginalized groups such as female-headed households or older beneficiaries with more business management experience. Explore whether financial costs are most effective in their current investment structure (I.e., grants/loans to start￾ups and MSMEs) or if additional investments in higher education and proactive business development and training strategies would promote the development of a stronger pool of aspiring entrepreneurs. 3. TSRR should work in close collaboration with local implementing representatives (currently the Settlement Officers and other community leaders) to identify and reach the most vulnerable, including women and youth, with livelihood skills. 4. TSRR should explore opportunities to promote linkages between KRA 2 and other livelihood activities such as TED, agriculture (KRA 3), and education (KRA 4). Consider implementing a pilot program to support vocational training opportunities for older beneficiaries (30-45 years old) who are unemployed/underemployed. 5. TSRR should conduct an updated analysis of high-growth sectors and explore potential areas to expand livelihoods development activities considering the changing labor market surrounding traditional handicrafts and waning interest in agriculture in Tibetan communities. This would help TSRR focus training efforts on sectors with the most potential for growth. If agriculture continues to be a key sector, the promotion of agricultural cooperatives should be considered to increase access to commercial markets. While it is beyond the scope of this evaluation to conduct a full analysis of other potential high-growth sectors, this may be worth exploring within TSRR’s support. Respondents suggested TSRR should expand its investment in settlement-based factories, service sector businesses, animal husbandry, and a wider range of cottage industries such as noodle production facilities. 6. Education appears to be a primary source of integration in the community, allowing for the introduction of health and cultural information as well as basic learning. TSRR should improve community outreach through the schools, especially among parents by emphasizing the importance of reading at home, to increase efficacy of learning materials and reduce dropout rates. 41 | TASKING TO50: TSRR ASSESSMENT USAID.GOV 7. TSRR should dedicate significant resources in the coming year to focus on implementing and expanding the newly revamped PNA process to promote greater success in the next round of community engagement. TSRR should collaborate closely with CTA staff at both the central and the settlement levels to ensure that all voices are included and explore the potential to designate a local TSRR representative to assume support for direct management of implementation, including stronger female representation at the highest levels of decision￾making authority within the settlement community. 4.2 EVALUATION QUESTION 2 RECOMMENDATIONS 1. TSRR should conduct pre-post testing of capacity-building outcomes (pre-post testing of skills and concepts, ongoing mentoring with iterative feedback to support implementation of new concepts) in order to gain a stronger sense of the training’s impact, especially longitudinally. 4.3 EVALUATION QUESTION 3 RECOMMENDATIONS 1. TSRR and the CTA should continue to promote a multi-sectoral approach at both the central and community levels, including establishing regular inter-departmental meetings to share knowledge and promote collaboration, including at the secretary level. 2. TSRR should consider strategic ways to address respondents’ feedback to place Tibetan language and culture at the center of the project as a conceptual shift that can promote enhanced multi-sectoral thinking. TSRR should explore the possibility to establish a local representative in each community who is incentivized to manage the daily operations on the ground and is focused on implementing a holistic continuum of service provision that creates vitality and strengthens opportunities for all. 4.4 EVALUATION QUESTION 4 RECOMMENDATION 1. TSRR and CTA should work to identify concrete strategies to successfully support development of the NBFC to become a fully operational banking center and promote access to finance for women and marginalized individuals, including those who lack access to traditional banking systems. USAID.GOV TASKING N050: TSRR EVALUATION | 42 ANNEXES ANNEX I: EVALUATION STATEMENT OF WORK Midterm Performance Evaluation of the Program to Strengthen Self Reliance and Resilience of Tibetan Communities in South Asia C.1 PROGRAM INFORMATION Activity/Project Name Program to Strengthen Self Reliance and Resilience of Tibetan Communities in South Asia (TSRR) Implementer The Tibet Fund (TTF) Cooperative Agreement/Contract # AID-386-G-16-00001 Total Estimated Ceiling of the Evaluated Project/Activity $23.00 Million Life of Project/Activity FY 2016- FY 2021 USAID/India Office Office of Social Sector Initiatives C.2 PURPOSE OF EVALUATION The purpose of this midterm performance evaluation of the “Program to Strengthen Self Reliance and Resilience of Tibetan Communities in South Asia (India and Nepal)” is to determine the extent to which progress is being made toward improving sustainable livelihoods improvement in the Tibetan communities, and strengthening of Tibetan institutions. The evaluation is being conducted after two and half years of program implementation. Findings from data generated through this evaluation, will be used by USAID/India, the Tibet Fund (TTF), and the Central Tibetan Administration (CTA), to inform implementation and management decisions over the rest of the implementation period. C.3 BACKGROUND The overall goal of the program is to strengthen the self-reliance and resilience of Tibetan communities in India and Nepal through sustainable livelihoods improvement and strengthened local institutions. Lack of economic opportunities in Tibetan refugee settlements and weak job competitiveness are causing growing underemployment, migration to cities, and increasingly emigration overseas, both among Tibetans born in exile and refugees who arrived from Tibet in recent years. A survey supported by The Tibet Fund found that over 25 percent of the refugee population subsists below or near the poverty threshold, supporting themselves by selling souvenirs, operating small restaurants, and engaging in minor trade, industry, and agriculture. The ability of the Tibetan exile population to pursue more productive livelihood strategies and achieve the economic self-reliance to sustain the vitality of their communities is a critical priority of the CTA. The program is implemented using an integrated multi-sector approach and has two main components: (I) Sustainable Livelihoods Improvement and (II) Strengthening Tibetan Institutions. Strategies to support improved sustainable livelihoods include Entrepreneurship, Micro and Small Enterprise Development, 43 | TASKING TO50: TSRR ASSESSMENT USAID.GOV Workforce Development, Agriculture Development, Basic Education Strengthening, Access to Higher Education, and Community Health Improvement. Over the last two years, the program has been working to achieve the following objectives through activities designed to improve livelihoods: • Increase household incomes and reduced underemployment, particularly among unemployed youth, through increased entrepreneurship and small enterprise growth, improved job skills, and increased value of marketable agriculture products; • Improve quality of teaching and learning in Tibetan schools and increased access to higher education, contributing to increased productivity and improved livelihoods; and • Increase access to affordable quality health care and reduce burden of impoverishing disease, contributing to increased productivity and improved livelihoods The second program component, which is to strengthen Tibetan Institutions, has the following objectives: • Improve effectiveness, capacity, and sustainability of local institutions serving Tibetan communities; and • Increase cultural vitality and resilience of Tibetan communities Strategies to achieve these objectives have been supported by activities in the following KRAs: • Entrepreneurship, Micro and Small Enterprise Development • Workforce Development • Agriculture Development • Basic Education Strengthening • Access to Higher Education • Community Health Improvement • Participatory Community Development • Women Empowerment • Institutional Capacity Development • Leadership Development • Cultural Institution and Resilience Strengthening DEVELOPMENT HYPOTHESIS: The program’s development hypothesis is encapsulated in the following statement: If livelihoods are improved in a sustainable manner, as measured by increased income and employment, improved education and health indicators, and increased participation of communities and women in the development process; and if the effectiveness, capacity, and sustainability of key institutions are strengthened; and if the cultural vitality of communities remains strong; then communities will become increasingly self-reliant and resilient. Critical assumptions underlying this hypothesis are that host government policies toward Tibetan communities in South Asia do not significantly change, no major macroeconomic shocks occur in the region, and the environment for economic and private sector development remains positive. C.4 EVALUATION QUESTIONS USAID.GOV TASKING N050: TSRR EVALUATION | 44 The evaluation should be designed to answer 4 overarching questions that focus on progress toward results, institutional capacity building and sustainability, and program design. These include the following: 1. To what extent has the activities implemented so far contributing toward improved livelihoods of the Tibetan refugee community? 2. To what extent are the systems and processes that are being introduced for strengthening the capacity of Tibetan institutions are being adopted and institutionalized by these organizations? 3. To what extent is the integrated multi-sectoral approach an effective model for responding to the theory of change put forward by the program? 4. What are the key aspects of the program that should be addressed at this stage of the program, to maximize outcomes over the rest of program implementation? C.5 EVALUATION DESIGN AND METHODOLOGY The evaluation is expected to use evidence from both quantitative and qualitative data that include primary and secondary data collection and analysis, to address the evaluation questions (EQs). Overall, it is expected that the methodology will be based on principles of participation, ownership, teamwork and learning, as follows: • Participation: to ensure that those affected by the program are able to voice their expectations, experience, learning points and insights; • Ownership: to ensure that CTA including the CTA Social and Resource Development Fund (SARD) and CTA Departments of Home, Education, and Health, USAID and other key stakeholders own the evaluation process (so it is not just something given to a consultant to accomplish); • Teamwork: to ensure the diversity of approaches and the seeking of consensus in the fundamental issues; • Learning: to ensure that CTA, USAID and other key stakeholders can identify and utilize the lessons learned and that the process contributes to identified program improvements over the rest of the implementation period. The evaluator will decide on the most rigorous and appropriate approaches for collecting quantitative and qualitative data. These may include but not limited to: • Document Reviews • Surveys/Questionnaires • KIIs with implementing partners and key stakeholders • FGDs • Case studies • Others It is important to note that a survey was conducted during phase 1 of the program 2015-2016, to establish baseline values for all impact and outcome indicators. The monitoring and evaluation plan of the program requires that midline surveys be conducted in the third year of the program to measure progress against outcome indicator targets. The evaluation team (ET) will need to include a section in the evaluation plan that clearly depicts how the survey will be conducted, the sample frame to be used, 45 | TASKING TO50: TSRR ASSESSMENT USAID.GOV sample size, quality assurance etc., to replicate the baseline survey. Out of 51 settlements in India and Nepal the evaluation will include four sampled settlements in India and two settlements in Nepal for the survey. The evaluation will address/ respond to gender-sensitive indicators, sex-dis-aggregated data and attention to gender inter-relations are required elements of USAID evaluations as highlighted in http://pdf.usaid.gov/pdf_docs/PA00K43P.pdf. From within, the ET or the parent organization, at-least one member must have expertise on gender￾related project evaluation to review the gender responsive checklist. DATA COLLECTION AND ANALYSIS: The ET will be accountable for ensuring data analysis methods are in line with best practices. For both quantitative and qualitative data, the ET will need to articulate methodologies for analyzing collected information, including any statistical software programs to be used. For qualitative data specifically, the ET will need to ensure that data are recorded and transcribed. C.6 EVALUATION QUESTIONS AND METHODOLOGY Evaluation Questions Data Sources Data Collection Method Data Analysis Method 1) To what extent have the activities implemented so far contributed to improved livelihoods of the Tibetan refugee community? 2. Activity beneficiaries 3. Activity Implementers (TTF and CTA) 4. USAID/India and USAID/Nepal 5. Community Members 6. KIIs (flow and Venn diagrams) 7. FGDs (matrix scoring) 8. Baseline and monitoring data review 9. Desk review Rapid Survey 10. Tally sheets 11. Thematic analyses 12. Outcome harvesting/mapping 13. Gender and comparative analysis 14. Analysis of survey data in Stata 2) To what extent are the systems and processes introduced for strengthening the capacity of Tibetan institutions being adopted and institutionalized by these organizations? 15. Activity Implementers (TTF and CTA) 16. Representatives of beneficiary institutions 17. USAID/India and USAID/Nepal 18. KIIs (flow and Venn diagrams) 19. Baseline and monitoring data review 20. Desk review 21. Tally sheets Thematic analyses 22. Outcome harvesting/mapping 23. Comparative analysis USAID.GOV TASKING N050: TSRR EVALUATION | 46 Evaluation Questions Data Sources Data Collection Method Data Analysis Method 3) To what extent is the integrated multi￾sectoral approach an effective model for responding to the theory of change put forward by the program? 24. Activity Implementers (TTF and CTA) 25. USAID/India and USAID/Nepal 26. KIIs (flow and Venn diagrams) 27. Baseline and monitoring data review 28. Desk review 29. Tally sheets 30. Outcome harvesting/mapping 31. Comparative analysis 4) What are the key aspects of the program that should be addressed at this stage of the program, to maximize outcomes over the rest of program implementation? 32. Activity beneficiaries Activity Implementers (TTF and CTA) 33. Representatives of beneficiary institutions USAID/India and USAID/Nepal 34. KIIs (flow and Venn diagrams) 35. FGDs (matrix scoring) Baseline and monitoring data review 36. Desk review 37. Tally sheets 38. Thematic analyses 39. Outcome harvesting/mapping 40. Gender and comparative analysis C.7 CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT • The evaluation report should represent a thoughtful, well-researched and well-organized effort to objectively evaluate what worked in the project, what did not, and why. • Evaluation reports shall address all EQs included in the scope of work. • The evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, EQs, ET composition, methodology, or timeline need to be agreed upon in writing by the Contracting Officer’s Representative. • Evaluation methodology shall be explained in detail, and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex in the final report. • Evaluation findings will assess outcomes and impact on males and females. • Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). • Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay or the compilation of people's opinions. Findings should be specific, concise and supported by strong quantitative or qualitative evidence. • Sources of information need to be properly identified and listed in an annex. • Recommendations need to be supported by a specific set of findings. • Recommendations should be action-oriented, practical, and specific, with defined responsibility for the action. C.8 EVALUATION MANAGEMENT 47 | TASKING TO50: TSRR ASSESSMENT USAID.GOV • The Contractor shall be responsible for obtaining visas and country clearances for travel for the consultants if required. • The Contractor shall be responsible for coordinating and facilitating assessment-related team planning meetings, field trips, interviews, and other meetings in conjunction with USAID and the Program team. • The Contractor shall be responsible for international and in-country logistics as applicable such as transportation, accommodations, communications, office support, etc. • The ET will receive support from USAID/India in selecting priority organizations and places to visit during the evaluation. The ET is expected to schedule interviews or other modes of data collection with key stakeholders, though USAID/India can assist in providing contact information. USAID.GOV TASKING N050: TSRR EVALUATION | 48 ANNEX II: EVALUATION DESIGN AND METHODOLOGY In this evaluation, SI undertook a participatory, mixed-methods evaluation design including: 45 KIIs, six FGDs, and a quantitative survey with 146 community members in the settlements visited by the ET during a three-week field work period that took place from November 4 – 25, 2019. The five-person ET included a Team Leader, a Senior M&E Specialist, an India Technical Specialist, and a Nepal Technical Specialist, as well as a Tibetan Research Assistant to provide translation/interpretation and insight into cultural perspectives. In some locations, an external interpreter was contracted to support work in that community. PHASE ONE: DESKTOP DOCUMENT REVIEW AND DATA COLLECTION INSTRUMENT/WORKPLAN DEVELOPMENT The initial phase of this evaluation included a thorough review of TSRR’s quarterly and annual reports and annual work plans for the first two years of implementation.38 Using the information derived from these reports, the ET developed a workplan including research methodology and semi-structured qualitative data collection instruments (see Annex III: Data Collection Instruments, for copies of data collection instruments) designed to solicit in-depth knowledge from key stakeholders and beneficiaries on the activity’s implementation, adoption, and effectiveness. These tools were tailored for each respondent type (e.g., USAID staff, CTA, community members) and were organized by EQ. The ET also developed the quantitative survey for community members to gather data on TSRR’s progress toward impact and outcome indicators. As part of this workplan development process, the ET also selected six sites to be visited during the field work period, starting from USAID’s request that Dharamsala, Bylakuppe, and Kathmandu be included as three of the sites selected. The ET’s process of additional site selection was initiated during the document review process by making note of implementation sites that were especially active and offered a wide range of activities to visit. While these criteria did mean that selection was skewed toward more-developed implementation sites, it was crucial to visit robust sites that offered a wide range of activities and thus offered a productive data collection opportunity given the scope, time and budget parameters, of this evaluation. After this list was compiled, the ET analyzed the locations of each short-listed site vis-à-vis which KRAs are being implemented there, as well as logistical considerations such as length of time and mode of travel between sites. Based on this analysis, a list of possible sites was submitted to USAID and TTF for further advice, which resulted in the final selection of the following six sites: Dharamsala, Dehradun, Bylakuppe, and Hunsur in India; and Kathmandu and Pokhara in Nepal. The ET also used the background document review process to inform final selection and definition of the key stakeholder groups to be included in the data collection process. These are: USAID staff supporting TSRR; CTA staff at the central and local levels (both as implementing partners and as beneficiaries of TSRR support); other implementing partner organizations; community leaders; TSRR beneficiaries; and community members for the quantitative survey. 38 Additional project documents were provided later in the evaluation process and were reviewed at that time, but not considered for the workplan. 49 | TASKING TO50: TSRR ASSESSMENT USAID.GOV PHASE TWO: IN-COUNTRY DATA COLLECTION Once the workplan and design document were developed, the Team Leader and Nepal Technical Specialist traveled to India to meet the rest of the team to start the field work period. Initially, the ET met with USAID representatives to present and discuss the evaluation workplan and design plan at a formal in-brief session. Following this meeting, the ET met internally to review the methodology, discuss any points in the workplan and design document that needed additional clarification, review the data collection instruments, and participate in an orientation session with the Tibetan Research Assistant to discuss cultural norms of Tibetan societies in India and Nepal. After meeting in Delhi for the in-brief and training period, the research team split into two groups to cover the maximum number of sites within the allotted field visit period. Each team consisted of one male and one female researcher, joined by an interpreter, in order to ensure gender representation in data collection and analysis, as well as to offer respondents an opportunity to speak with a researcher of their own sex if desired. QUALITATIVE DATA Once the sites had been selected and respondent groups identified, TTF provided the ET with a full list of potential contacts from which to recruit respondents for KIIs and FGDs within the TSRR beneficiary category. The ET initially tried to select respondents by random selection, but this proved problematic for several reasons. First, the ET had high refusal rates, including potential respondents who were reluctant to meet with ET members unless someone from the Tibetan community approached them first. This issue was resolved by having the interpreters set the meetings but resulted in many initial contacts refusing to participate. Second, the contact lists included a large proportion of telephone numbers that no longer reached the beneficiary we sought to contact or the beneficiary did not answer when called. In the end, the ET elected to have the Tibetan interpreters call down the full list in order until achieving the desired number of appointments, which frequently resulted in a large portion of the list being called. One team, consisting of the Team Leader and the Nepal Technical Specialist, first traveled to Dharamsala to gather data from TTF and the CTA central office staff and project beneficiaries in the McLeodgang settlement. This team then traveled to Nepal and collected data in the Jampaling and Tashi Palkhiel settlements in Pokhara; and the Boudha settlement in Kathmandu. The Team Leader also conducted interviews with beneficiaries in Delhi receiving scholarships to attend Indian universities. The second team, consisting of the Senior M&E Specialist and India Technical Specialist, traveled first to the Dekyiling settlement in Dehradun (Uttarakhand) before moving to southern India to visit the Dickyi Larsoe and Lugsung Samdupling settlements in Bylakuppe and Rabgyeling settlement in Hunsur. Figure 3 below provides a map of the Tibetan settlements in India and Nepal. The locations of the settlements visited by the ET are in red, whereas the other settlements are in blue. USAID.GOV TASKING N050: TSRR EVALUATION | 50 Figure 3: Locations of the TSRR Midterm Evaluation Sites in India and Nepal In total, the ET conducted 43 interviews with 57 key stakeholders, including 23 interviews with 28 TSRR beneficiaries, and 12 interviews with representatives of the CTA. While the ET sought to achieve equal gender balance in recruiting KII respondents, the final sample had two-thirds male respondents, including more CTA representatives and more male TSRR beneficiaries. Table 4 provides a disaggregation of respondents by type, including beneficiaries by KRA, as well as CTA and implementing partners and community leaders. Table 4: Qualitative Respondents by Type and Sex Respondent Type Total Males Females KRA 1 6 4 2 KRA 2 - - - KRA 3 20 10 10 KRA 4 15 6 9 KRA 5 4 2 2 KRA 6 20 9 11 KRA 7 3 3 - 51 | TASKING TO50: TSRR ASSESSMENT USAID.GOV KRA 8 2 1 1 KRA 9 1 1 - KRA 10 1 1 - KRA 11 1 1 - CTA 15 9 6 TTF/IP 8 5 3 Community Leaders 6 5 1 QUANTITATIVE SURVEY In addition to qualitative data collection, the evaluation conducted a 16-question quantitative survey with 146 community members (not necessarily TSRR beneficiaries) who were selected via convenience sampling in the six sites visited. The ET worked in collaboration with the Settlement Office to identify areas within each settlement that would be promising and appropriate locations to approach Tibetan community members to respond to the survey; these included main market areas, near Buddhist temples and monasteries, residential areas, and in shops managed by Tibetans. Even working through Tibetan interpreters, the ET experienced challenges recruiting respondents for the quantitative survey, including reluctance by community members to participate in a random survey due to concerns about the political situation and privacy. This issue was raised by the Settlement Officers and other CTA officials early in the data collection process, so the ET took a number of steps to promote participation rates, including working in close collaboration with the Settlement Office staff to notify the community that the survey was taking place and encouraging their cooperation, selecting locations for data collection recommended by the Settlement Officer, and using the Tibetan interpreters to approach potential respondents to explain the study and request their participation. Using these methods did increase participation rates significantly, but it was still challenging for the ET to reach the desired number of completed surveys within the time allotted. Ultimately 146 individuals, of which 51 percent were male, were surveyed through the Tibetan interpreter, and their responses were captured in real time using the SurveyCTO online platform. Figures 4-6 below provide disaggregated details about the survey respondents’ demographics. Figure 4 provides a breakdown of survey respondents by settlement, demonstrating that the largest percentage of respondents came from Dehradun (23 percent), with the smallest percentage of representation coming from respondents in Kathmandu (9 percent - reflecting the political challenges of collecting data in that location). Figure 5 disaggregates respondents’ reported household income levels, with the majority of respondents reporting between 20-30,000 rupees a month in income (23 percent). Figure 6 provides a breakdown of survey respondents by age category, with the greatest percentage of respondents falling in the 50-58 age range (21 percent), and the lowest percentage of respondents in their forties (12 percent). USAID.GOV TASKING N050: TSRR EVALUATION | 52 Figure 4: Percentage of Survey Respondents by Settlement Visited Figure 5: Survey Respondents' Household Income Levels Figure 6: Percentage of Survey Respondents by Age Group 53 | TASKING TO50: TSRR ASSESSMENT USAID.GOV PHASE THREE: DATA ANALYSIS At the end of the field-based data collection phase, the ET reconvened for compilation of data and analysis of findings. The ET analyzed the qualitative data gathered from KIIs and FGDs using qualitative data analysis software, MaxQDA. Key themes were identified and used to generate a coding structure, by which the ET systematically coded all qualitative data to ensure that analysis was conducted in an objective and comprehensive manner. The ET used content and comparative analyses to identify response categories and patterns and to elucidate emergent themes and contextual factors. Quantitative data were entered into SurveyCTO at the time of collection and were analyzed through that system’s online data collection system to tabulate responses, produce basic descriptive statistics, and disaggregate data according to standard demographics including gender, location, age, etc. Finally, the ET triangulated all qualitative and quantitative data in order to validate findings across all data sources. These data were analyzed using SI’s Findings, Conclusions, and Recommendations matrix, based on the four EQs, which enabled the ET to systematically identify all findings across respondent groups and types, and to ensure that each EQ is answered thoroughly by the ET, and ultimately documented in this evaluation report. PHASE FOUR: UTILIZATION AND DISSEMINATION The ET facilitated an initial findings presentation in Delhi for USAID, TTF, and CTA staff to discuss the preliminary findings of the field work prior to the Team Leader’s departure from India. Based on the information shared in the debrief presentation, TTF shared a list of questions and requested clarifications that have been integrated into this evaluation report. This Evaluation Report is the key deliverable for this evaluation, which presents de-identified aggregate level data informing the provision of evidence-based recommendations for the remainder of the activity. In keeping with SI’s commitment to utilization-focused evaluation, these recommendations identify the action, responsible party, and timeline for implementation. It is intended to be a public document that will be disseminated to interested parties via USAID’s Development Experience Clearinghouse website, to promote transparent and accountable development service provision. POTENTIAL LIMITATIONS AND BIASES REPRESENTATIVENESS: Both the qualitative and quantitative sample sizes are relatively small and cannot be considered representative of all Tibetan settlements or the broader population to a statistically significant degree. While sampling strategies tried to ensure that all KRAs were represented in the data collection, spreading out 45 interviews between 11 KRAs means that each voice captured becomes representative of the general experience to a much greater degree than in single-sector projects. In addition, selecting six out of 52 implementation sites to visit, and making those selections fit within the timeframe and budget of the evaluation, further restricts the representativeness of the data gathered. The sites visited in India represent two northern India settlements, of which there are at least a dozen more that were either smaller in size or more difficult to reach, and two agricultural sites in south-western India, but does not include any of the sites in the east, including the five sites in Arunachal Pradesh. USAID.GOV TASKING N050: TSRR EVALUATION | 54 RESPONSE BIAS: Response bias is the tendency of respondents to provide answers that are not completely truthful or candid, for any number of reasons. For example, with the knowledge that evaluation results can inform the future funding or direction of an activity, key informants may provide evaluators with positive feedback about the project because they want the activity to continue. The ET tried to mitigate this bias by emphasizing to participants a strong desire to hear about what has worked well, as well as what have been the challenges/weaknesses, and also making clear through the informed consent process that there are no direct benefits or repercussions for participating (or not) in the evaluation. SELECTION BIAS: Though not statistically representative, the purposive sampling for FGDs and KIIs ensures the evaluation includes a wide variety of relevant and informed viewpoints because stakeholders from each KRA are represented and the selection of sites included a variety of implementation contexts. While there is always a risk that project implementers could guide the ET to informants with more favorable experiences in the project, the ET has minimized this potential bias by requesting the full pool of beneficiary contacts for each type of activity, and sampling from that list, to ensure that respondents represent a diverse range of informants with varying levels of engagement with the project. The ET also used data triangulation to validate qualitative data. By comparing data provided in interviews against reviewed documents, for example, we can ensure that a single biased data point does not skew our analysis. RECALL BIAS: Recall bias is a common challenge in evaluations. One type of recall bias occurs when project beneficiaries unintentionally blend their experiences across multiple projects/programs into a composite memory. For example, if they have received services from multiple implementing partners, they may be unable to distinguish between programs to recall the specifics of the particular program of interest. The ET mitigates this risk by clearly explaining the purpose and background of the evaluation and the program being evaluated, including by giving the respondents the names of the local implementers and the timeframe in which it was implemented. INSTRUMENT BIAS: None of the data collection instruments were piloted, a limitation that did impact data collection and potentially, data quality. Some specific challenges that were faced in this regard include: the language of KII and FGD questions were too sophisticated or assumed familiarity with topics beyond the scope of some respondents and had to be modified or discarded during data collection efforts; quantitative survey questions were worded in ways that required interpretation in the field, specifically questions that included the concept of “households,” with so many Tibetans living abroad or in semi-transient situations. While the ET did maintain continuous communication among team members to discuss and find working adaptations to these challenges, application of the research instruments was semi-structured in nature. 55 | TASKING TO50: TSRR ASSESSMENT USAID.GOV ANNEX III: DATA COLLECTION INSTRUMENTS KII – IMPLEMENTING PARTNERS (INCLUDING LOCAL PARTNER NGOS) Date of Interview [DD/MM/YY]: |__|__|/|__|__|/|__|__| Name of Interviewer: _________________________________________________________ Name of Note-taker: _________________________________________________________ Name of Anyone Else Present: ________________________________________________ Time Interview Began [HH:MM]: |__|__|:|__|__| Time Interview Ended [HH:MM]: |__|__|:|__|__| INTERVIEWEES Gender Name(s): ________________________ Position(s): _______________________ M / F Name(s): ________________________ Position(s): _______________________ M / F Name(s): ________________________ Position(s): _______________________ M / F Name(s): ________________________ Position(s): _______________________ M / F MUST READ THE CONSENT STATEMENT AND GAIN CONSENT FROM ALL RESPONDENTS BEFORE PROCEEDING GENERAL 1. What is your role in your organization? 2. What roles do you and/or your office play in the Program to Strengthen Self Reliance and Resilience of Tibetan Communities in South Asia (TSRR)? a. Are there specific components of TSRR that you/your office focuses on more than others? If so, which components? EQ3: To what extent is the integrated multi-sectoral approach an effective model for responding to the theory of change put forward by the program? 3. TSRR integrates many activities (e.g. health, education, agriculture, employment, etc.). How well is this integrated approach working to achieve intended outcomes? 4. What is working well regarding the integrated nature of TSRR? 5. What are some of the challenges regarding the integration of different approaches/sectors under TSRR? USAID.GOV TASKING N050: TSRR EVALUATION | 56 a. Do all activities receive equal attention? If not, which activities are prioritized and why? Which activities receive less attention? EQ4: What are the key aspects of the program that should be addressed at this stage of the program, to maximize outcomes over the rest of program implementation? 6. Aside from the integrated nature of TSRR, are there other aspects that have been working well in terms of implementation? a. What are the strengths of how it is being implemented? 7. Has TSRR faced challenges in implementation? a. Challenges related to different implementing partners/organizations? b. Challenges regarding the target population/intended beneficiaries? c. Challenges working with the government or other external entities? 8. From your perspective, what could be done to improve TSRR’s implementation? EQ1: To what extent have the activities implemented so far contributed to improved livelihoods of the Tibetan refugee community? 9. What are the key achievements of TSRR thus far for refugee livelihoods? [Probes] a. incomes and employment? b. education, teaching, and learning? c. healthcare? d. community participation in development? e. women’s empowerment in development? 10. In your opinion, are there any areas where the program has not improved outcomes to the extent anticipated/desired? a. If so, which outcomes have been more difficult to achieve? Why? b. Are you aware of any unintended outcomes of TSRR – either positive or negative? If so, do you feel they have affected different types of people differently? (Probe for gender) 11. From your perspective, what are the primary challenges to achieving TSRR’s intended livelihood outcomes? EQ2: To what extent are the systems and processes introduced for strengthening the capacity of Tibetan institutions being adopted and institutionalized by these organizations? 12. From your perspective, what are the key achievements for strengthening Tibetan institutions through the TSRR program? a. How/to what extent has organizational effectiveness changed? b. How/to what extent has organizational capacity changed? c. How/to what extent has the sustainability of Tibetan institutions changed? 57 | TASKING TO50: TSRR ASSESSMENT USAID.GOV 13. What are TSRR’s key achievements for increasing cultural vitality and resilience? a. What does it mean to increase Tibetan vitality and resilience? b. In what ways has cultural vitality and resilience changed since the start of the program? Why? 14. What are the primary challenges to achieving TSRR’s objectives regarding Tibetan organizational capacity and cultural vitality and resilience? a. Challenges specific to improving organizational capacity and sustainability? b. Challenges specific to increasing cultural vitality and resilience? 15. Is there anything that the TSRR program could be doing differently to address some of these challenges? If so, what do you think should be done? 16. Are there any programmatic changes you would recommend that could help TSRR better achieve its intended outcomes? a. For improving refugee livelihoods (income, employment, education, health, etc.)? b. For improving intended outcomes related to organizational and cultural capacity? 17. Is there anything else you’d like to discuss with me? USAID.GOV TASKING N050: TSRR EVALUATION | 58 KII – CENTRAL TIBETAN ADMINISTRATION Date of Interview [DD/MM/YY]: |__|__|/|__|__|/|__|__| Name of Interviewer: _________________________________________________________ Name of Note-taker: _________________________________________________________ Name of Anyone Else Present: ________________________________________________ Time Interview Began [HH:MM]: |__|__|:|__|__| Time Interview Ended [HH:MM]: |__|__|:|__|__| INTERVIEWEES Gender Name(s): ________________________ Position(s): _______________________ M / F Name(s): ________________________ Position(s): _______________________ M / F Name(s): ________________________ Position(s): _______________________ M / F Name(s): ________________________ Position(s): _______________________ M / F MUST READ THE CONSENT STATEMENT AND GAIN CONSENT FROM ALL RESPONDENTS BEFORE PROCEEDING GENERAL 1. What is your role at CTA (include department)? 2. What roles do you and/or your department play in the Program to Strengthen Self Reliance and Resilience of Tibetan Communities in South Asia (TSRR)? a. Are there specific components of TSRR that you/your office focuses on more than others? If so, which components? EQ3: To what extent is the integrated multi-sectoral approach an effective model for responding to the theory of change put forward by the program? 3. TSRR integrates many activities (e.g. health, education, agriculture, employment, etc.). How well is this integrated approach working to achieve intended outcomes? 4. What is working well regarding the integrated nature of TSRR? 5. What are some of the challenges regarding the integration of different approaches/sectors under TSRR? 59 | TASKING TO50: TSRR ASSESSMENT USAID.GOV a. Do all activities receive equal attention? If not, which activities are prioritized and why? Which activities receive less attention? EQ4: What are the key aspects of the program that should be addressed at this stage of the program, to maximize outcomes over the rest of program implementation? 6. Aside from the integrated nature of the TSRR program, what has been working well in terms of execution? a. What are the strengths of how it is being implemented? 7. Has TSRR faced challenges in implementation? If so, what have they been? a. Challenges related to different implementing partners/organizations? b. Challenges regarding the target population/intended beneficiaries? c. Challenges working with the government or other external entities? d. Challenges with coordination within CTA or between CTA and TTF? 8. From your perspective, what could be done to improve TSRR’s implementation? EQ1: To what extent have the activities implemented so far contributed to improved livelihoods of the Tibetan refugee community? 9. What are the key achievements of TSRR thus far for refugee livelihoods? [Probes as relevant to individual’s job and focus area] a. incomes and employment? b. education, teaching, and learning? c. healthcare? d. community participation in development? e. women’s empowerment in development? 10. In your opinion, are there any areas where the program has not improved outcomes to the extent anticipated/desired? a. If so, which outcomes have been more difficult to achieve? Why? b. Are you aware of any unintended outcomes of TSRR – either positive or negative? If so, do you feel they have affected different types of people differently? (Probe for gender) 11. From your perspective, what are the primary challenges to achieving TSRR’s intended livelihood outcomes? EQ2: To what extent are the systems and processes introduced for strengthening the capacity of Tibetan institutions being adopted and institutionalized by these organizations? 12. From your perspective, what are the key achievements for strengthening Tibetan institutions, particularly CTA, through the TSRR program? a. How/to what extent has organizational effectiveness changed? b. How/to what extent has organizational capacity changed? c. How/to what extent has the sustainability of Tibetan institutions changed? USAID.GOV TASKING N050: TSRR EVALUATION | 60 13. Which trainings or activities aimed at capacity building have you taken part in? a. Which were the most effective and how? b. Which trainings should be repeated or have additional activities during the remainder of the program and why? 14. What are TSRR’s key achievements for increasing cultural vitality and resilience? a. What does it mean to increase Tibetan vitality and resilience? b. In what ways has cultural vitality and resilience changed since the start of the program? Why? 15. What are the primary challenges to achieving TSRR’s objectives regarding Tibetan organizational capacity and cultural vitality and resilience? a. Challenges specific to improving organizational capacity and sustainability? b. Challenges specific to increasing cultural vitality and resilience? 16. Is there anything that the TSRR program could be doing differently to address some of these challenges? If so, what do you think should be done? 17. Are there any programmatic changes you would recommend that could help TSRR better achieve its intended outcomes? a. For improving refugee livelihoods (income, employment, education, health, etc.)? b. For improving intended outcomes related to organizational and cultural capacity? 18. Is there anything else you’d like to discuss with me? 61 | TASKING TO50: TSRR ASSESSMENT USAID.GOV KII – COMMUNITY LEADERS Date of Interview [DD/MM/YY]: |__|__|/|__|__|/|__|__| Name of Interviewer: _________________________________________________________ Name of Note-taker: _________________________________________________________ Name of Anyone Else Present: ________________________________________________ Time Interview Began [HH:MM]: |__|__|:|__|__| Time Interview Ended [HH:MM]: |__|__|:|__|__| INTERVIEWEES Gender Name(s): ________________________ Position(s): _______________________ M / F Name(s): ________________________ Position(s): _______________________ M / F Name(s): ________________________ Position(s): _______________________ M / F Name(s): ________________________ Position(s): _______________________ M / F MUST READ THE CONSENT STATEMENT AND GAIN CONSENT FROM ALL RESPONDENTS BEFORE PROCEEDING GENERAL 1. What is your role in the community? 2. What activities of the Program to Strengthen Self Reliance and Resilience of Tibetan Communities in South Asia (TSRR) are being implemented in your community? a. What, if any, role do you play in the program’s implementation? EQ3: To what extent is the integrated multi-sectoral approach an effective model for responding to the theory of change put forward by the program? 3. TSRR integrates many activities (e.g. health, education, agriculture, employment, etc.). How well this integrated approach is working to achieve intended outcomes? 4. What is working well regarding the integrated nature of TSRR? 5. What are some of the challenges regarding the integration of different approaches/sectors under TSRR? a. Do all activities receive equal attention? If not, which activities are prioritized and why? Which activities receive less attention? USAID.GOV TASKING N050: TSRR EVALUATION | 62 b. Are there activities you believe TSRR should prioritize for your community over others and why? EQ4: What are the key aspects of the program that should be addressed at this stage of the program, to maximize outcomes over the rest of program implementation? 6. Aside from the integrated nature of TSRR, are there other aspects that have been working well in terms of implementation? a. What are the strengths of how it is being implemented? 7. Has TSRR faced challenges in implementation? If so, what have they been? a. Challenges related to different implementing partners/organizations? b. Challenges regarding the target population/intended beneficiaries? c. Challenges working with the government or other external entities? 8. From your perspective, what could be done to improve TSRR’s implementation? 9. Was your community involved in the planning process (needs assessment, etc.)? a. Were men and women equal participants in the process? EQ1: To what extent have the activities implemented so far contributed to improved livelihoods of the Tibetan refugee community? 10. What are the key achievements of TSRR thus far for refugee livelihoods? [Probes] a. incomes and employment? b. education, teaching, and learning? c. healthcare? d. community participation in development? e. women’s empowerment in development? 11. In your opinion, are there any areas where the program has not improved outcomes to the extent anticipated/desired? a. If so, which outcomes have been more difficult to achieve? Why? b. Are you aware of any unintended outcomes of TSRR – either positive or negative? If so, do you feel they have affected different types of people differently? (Probe for gender) 12. From your perspective, what are the primary challenges to achieving TSRR’s intended livelihood outcomes? EQ2: To what extent are the systems and processes introduced for strengthening the capacity of Tibetan institutions being adopted and institutionalized by these organizations? 13. From your perspective, what are the key achievements for strengthening Tibetan institutions through the TSRR program? a. How have your interactions changed with Tibetan institutions since the program’s start? b. How/to what extent has organizational effectiveness changed? 63 | TASKING TO50: TSRR ASSESSMENT USAID.GOV c. How/to what extent has organizational capacity changed? d. How/to what extent has the sustainability of Tibetan institutions changed? 14. What are TSRR’s key achievements for increasing cultural vitality and resilience? a. What does it mean to increase Tibetan vitality and resilience? b. In what ways has cultural vitality and resilience changed since the start of the program? Why? 15. What are the primary challenges to achieving TSRR’s objectives regarding Tibetan organizational capacity and cultural vitality and resilience? a. Challenges specific to improving organizational capacity and sustainability? b. Challenges specific to increasing cultural vitality and resilience? 16. Is there anything that the TSRR program could be doing differently to address some of these challenges? If so, what do you think should be done? 17. Are there any programmatic changes you would recommend that could help TSRR better achieve its intended outcomes? a. For improving intended outcomes related to refugee livelihoods (income, employment, education, health, etc.)? b. For improving intended outcomes related to organizational and cultural capacity? 18. Is there anything else you’d like to discuss with me? USAID.GOV TASKING N050: TSRR EVALUATION | 64 KII – TED BENEFICIARIES (ENTREPRENEURS, BUSINESS OWNERS) Date of Interview [DD/MM/YY]: |__|__|/|__|__|/|__|__| Name of Interviewer: _________________________________________________________ Name of Note-taker: _________________________________________________________ Name of Anyone Else Present: ________________________________________________ Time Interview Began [HH:MM]: |__|__|:|__|__| Time Interview Ended [HH:MM]: |__|__|:|__|__| INTERVIEWEES Gender Name(s): ________________________ Position(s): _______________________ M / F Name(s): ________________________ Position(s): _______________________ M / F Name(s): ________________________ Position(s): _______________________ M / F Name(s): ________________________ Position(s): _______________________ M / F MUST READ THE CONSENT STATEMENT AND GAIN CONSENT FROM ALL RESPONDENTS BEFORE PROCEEDING GENERAL 1. What type of business do you do? 2. What activities of the Program to Strengthen Self Reliance and Resilience of Tibetan Communities in South Asia (TSRR) have you taken part in? a. What trainings have you attended? b. What, if any, financial resources have you received? EQ1: To what extent have the activities implemented so far contributed to improved livelihoods of the Tibetan refugee community? 3. In what way has your business or have your business practices changed as a result of the program? a. From your perspective, how has the program helped you or others in your community achieve improvements in business outcomes? b. In your opinion, are there any areas where the program has not helped you achieve improvements in business outcomes? 65 | TASKING TO50: TSRR ASSESSMENT USAID.GOV 4. What has been the program’s impact on the livelihood of you and your family? What have been the key impacts of the program on your families… a. Financial Security? b. Health Outcomes? c. Educational Outcomes? 5. Who do you consider the primary beneficiaries of TSRR activities? a. Are men and women equal beneficiaries? 6. From your perspective, what are the primary challenges to achieving TSRR’s intended livelihood outcomes? EQ3: To what extent is the integrated multi-sectoral approach an effective model for responding to the theory of change put forward by the program? 7. TSRR integrates many activities (e.g. health, education, agriculture, employment, etc.). What are your thoughts on how well this integrated approach is working to achieve intended outcomes? a. What relationship, if any, do you see between these other activities (health, education, etc.) and your business vitality? 8. What is working well regarding the integrated nature of TSRR? 9. What are some of the challenges regarding the integration of different approaches/sectors under TSRR? a. Do all activities receive equal attention? If not, which activities are prioritized and why? Which activities receive less attention? EQ4: What are the key aspects of the program that should be addressed at this stage of the program, to maximize outcomes over the rest of program implementation? 10. Regarding business and workforce development, which parts of the program do you think are working well? a. What are the strengths of how it is being executed? 11. From your perspective, are there any areas in which the program hasn’t been as effective or could be improved in its efforts to address the needs of the business community? 12. From your perspective, what could be done to improve TSRR’s implementation? EQ2: To what extent are the systems and processes introduced for strengthening the capacity of Tibetan institutions being adopted and institutionalized by these organizations? 13. From your perspective, what are the key achievements for strengthening Tibetan institutions through the TSRR program? a. How have your interactions changed with Tibetan institutions since the program started? USAID.GOV TASKING N050: TSRR EVALUATION | 66 b. How sustainable do you think the programs/activities implemented by Tibetan institutions are after the program ends? 14. From your perspective, what are TSRR’s key achievements for increasing cultural vitality and resilience? a. From your perspective, what does it mean to increase Tibetan vitality and resilience? b. In what ways has cultural vitality and resilience changed since the start of the program? Why? 15. From your perspective, are there any programmatic changes you would recommend that could help TSRR better support business and entrepreneurship development? 16. Is there anything else you’d like to discuss with me? 67 | TASKING TO50: TSRR ASSESSMENT USAID.GOV KII – BENEFICIARIES Date of Interview [DD/MM/YY]: |__|__|/|__|__|/|__|__| Name of Interviewer: _________________________________________________________ Name of Note-taker: _________________________________________________________ Name of Anyone Else Present: ________________________________________________ Time Interview Began [HH:MM]: |__|__|:|__|__| Time Interview Ended [HH:MM]: |__|__|:|__|__| INTERVIEWEES Gender Name(s): ________________________ Position(s): _______________________ M / F Name(s): ________________________ Position(s): _______________________ M / F Name(s): ________________________ Position(s): _______________________ M / F Name(s): ________________________ Position(s): _______________________ M / F MUST READ THE CONSENT STATEMENT AND GAIN CONSENT FROM ALL RESPONDENTS BEFORE PROCEEDING EQ 1: To what extent have the activities implemented so far contributed to improved livelihoods of the Tibetan refugee community? 1. What components of the Program to Strengthen Self Reliance and Resilience of Tibetan Communities in South Asia (TSRR) have you participated in? [NOTE: select topics for questions 2-15 based on which TYPE of activity the respondent has participated in.] FARMING/SMALL AND MEDIUM ENTERPRISES DEVELOPMENT/ENTREPRENEURSHIP 2. In what ways did the program assist in improving your business/farming practices? Did that increase your revenues? 3. What challenges/barriers do you face in improving your business/crop? Did TSRR help you address these challenges? Is additional follow-up is required? (EQ4) EDUCATION USAID.GOV TASKING N050: TSRR EVALUATION | 68 4. Has access, quality and community perceptions of education in your community changed in the past three years? a. What do you think are the driving factors behind this change? 5. [If they don’t mention the program for #4]: Are you aware of any activities being conducted by TSRR to increase the quality of basic education in your community? What do you think has been the impact of these interventions? 6. Do people in your community face challenges in accessing higher education? If so, in what way? Is this different for men and women? HEALTH 7. Are you aware of any activities being conducted by TSRR to increase the quality of health care? 8. Has your access to health care changed in the past 3 years? [If yes] Has it improved or gotten worse? What do you think is the reason for this change? COMMUNITY PARTICIPATION 9. Have you taken part in any community development planning? If so, in what way? 10. Have you been part of a Participatory Needs Assessment (PNA)? If so, what was the goal of these planning sessions? 11. Do you feel like your voice is heard in these planning sessions? Do you think TSRR reflects your input? Are there ways you feel you have not been heard? a. Is everyone’s voice heard the same as yours? Are there some that are heard more/less? How so? b. [Probe]: Are men and women heard equally in the process? SELF-RELIANCE AND RESILIENCE 12. Do you feel like your income/work is enough to secure the livelihood of you and your family? If not, what additional support would you need? (EQ4) 13. If you were to lose your current source of income, would how long would your family be able to live at your current means before having to make changes? How would you cover expenses during that time (savings, loan, family support, government support, etc.)? CULTURE 14. From your perspective, what does it mean to increase Tibetan vitality and resilience? 15. In what ways has cultural vitality and resilience changed since the start of the program? Why? 69 | TASKING TO50: TSRR ASSESSMENT USAID.GOV EQ 3: To what extent is the integrated multi-sectoral approach an effective model for responding to the theory of change put forward by the program? 16. TSRR is working in a number of areas including business, agriculture, health, education and culture. From your perspective, how well are these different components working together? 17. Do you feel the program is performing better in some areas rather than others? Why do you think that is? EQ 2: To what extent are the systems and processes introduced for strengthening the capacity of Tibetan institutions being adopted and institutionalized by these organizations? 18. In what ways do you interact with the CTA? a. What programs/services do you participate in? b. Do you interact directly with CTA staff or do you interact with them indirectly through community leaders or other individuals/entities? 19. Have you noticed any changes in the way the CTA interacts with your community in the past 3 years? Do you consider this change an improvement or not? 20. What areas do you think the CTA is doing well as far as securing better livelihoods and preserving culture for your communities? Where can they improve? (EQ4) EQ 4: What are the key aspects of the program that should be addressed at this stage of the program, to maximize outcomes over the rest of program implementation? 21. In what ways do you think the program can improve? 22. Are you aware of any unintended outcomes of TSRR – either positive or negative? If so, do you feel they have affected different types of people differently? (Probe for gender) 23. Are there specific challenges that women in these settlements face (prompt: transportation, workload, training inclusion, role in business, education)? Do you think the program has helped women to address some of these challenges? What do you see as the challenges and opportunities facing women, and other vulnerable groups moving forward? USAID.GOV TASKING N050: TSRR EVALUATION | 70 FGD GUIDE – BENEFICIARIES Date of Focus Group [DD/MM/YY]: |__|__|/|__|__|/|__|__| Name of Interviewer: _________________________________________________________ Name of Note-taker: _________________________________________________________ Name of Anyone Else Present: ________________________________________________ Time Focus Group Began [HH:MM]: |__|__|:|__|__| Time Focus Group Ended [HH:MM]: |__|__|:|__|__| PARTICIPANTS Gender Name(s): ________________________ Position(s): _______________________ M / F Name(s): ________________________ Position(s): _______________________ M / F Name(s): ________________________ Position(s): _______________________ M / F Name(s): ________________________ Position(s): _______________________ M / F MUST READ THE CONSENT STATEMENT AND GAIN CONSENT FROM ALL RESPONDENTS BEFORE PROCEEDING EQ 1: To what extent have the activities implemented so far contributed to improved livelihoods of the Tibetan refugee community? 1. What components of the Program to Strengthen Self Reliance and Resilience of Tibetan Communities in South Asia (TSRR) have you participated in? 2. What were the benefits of participating in this program for you? EQ 3: To what extent is the integrated multi-sectoral approach an effective model for responding to the theory of change put forward by the program? 3. TSRR is working in a number of areas including business, agriculture, health, education and culture. From your perspective, how well are these different components working together? 4. Do you feel the program is performing better in some areas rather than others? Why do you think that is? EQ 2: To what extent are the systems and processes introduced for strengthening the capacity of Tibetan institutions being adopted and institutionalized by these organizations? 71 | TASKING TO50: TSRR ASSESSMENT USAID.GOV 5. In what ways do you interact with the CTA? a. What programs/services do you participate in? b. Do you interact directly with CTA staff or do you interact with them indirectly through community leaders or other individuals/entities? 6. Have you noticed any changes in the way the CTA interacts with your community in the past 3 years? Do you consider this change an improvement or not? 7. What areas do you think the CTA is doing well as far as securing better livelihoods and preserving culture for your communities? Where can they improve? (EQ4) EQ 4: What are the key aspects of the program that should be addressed at this stage of the program, to maximize outcomes over the rest of program implementation? 8. In what ways do you think the program can improve? 9. Are you aware of any unintended outcomes of TSRR – either positive or negative? If so, do you feel they have affected different types of people differently? (Probe for gender) 10. Are there specific challenges that women in these settlements face (prompt: transportation, workload, training inclusion, role in business, education)? Do you think the program has helped women to address some of these challenges? What do you see as the challenges and opportunities facing women, and other vulnerable groups moving forward? USAID.GOV TASKING N050: TSRR EVALUATION | 72 QUANTITATIVE SURVEY Consent Text (do not record any contact information) EQ Q# (Baseline Q#) Question Response Options Record Response (B1 Roster) Gender 1 = Male 2 = Female 3 = Prefer not to disclose |___| (B1 Roster) Age (do range if needed) 1 = <18 (Stop Interview) 2 = 18 – 29 3 = 30 – 39 4 = 40 – 49 5 = 50 – 59 6 = 60+ |___| EQ1 (A2) Classification of household by main income source 1 = Subsistence Farming 2 = Commercial Farming 3 = Local Business 4 = Seasonal Business 5 = Remittance 6 = Tourism 7 = Government (CTA/India/Nepal) 8 = Profession (specify) 9 = Other (specify) |___| If 7 circle: CTA / India / Nepal If 8 or 9: ________________ (A3) How many people live in your household? Household definition Number of Adults: |___|___| Number of Children: |___|___| EQ1 (C1) Can you tell us your average monthly household income? 0 = <5,000 1 = 5,001 – 10,000 2 = 10,001 – 20,000 3 = 20,001 – 30,000 4 = 30,001 – 40,000 5 = 40,001 – 50,000 6 = 50,001+ |___| EQ1 (C2) Can you tell us your average monthly household expenditure? 0 = <5,000 1 = 5,001 – 10,000 2 = 10,001 – 20,000 3 = 20,001 – 30,000 4 = 30,001 – 40,000 5 = 40,001 – 50,000 6 = 50,001+ |___| EQ1 (A1) I have adequate skills and knowledge to earn my livelihood SD = Strongly Disagree D = Disagree A = Agree SA = Strongly Agree |___| 73 | TASKING TO50: TSRR ASSESSMENT USAID.GOV EQ Q# (Baseline Q#) Question Response Options Record Response EQ1 (A5) In the event of losing my source of income, I have sufficient savings to cover my household expenditures for at least 6 months. SD = Strongly Disagree D = Disagree A = Agree SA = Strongly Agree |___| EQ1 (B1) I am confident that my child/children’s education (or the education of children in my community) will give them the knowledge and skills to become self￾reliant. SD = Strongly Disagree D = Disagree A = Agree SA = Strongly Agree |___| EQ1 (B4) My primary health care needs are adequately met. SD = Strongly Disagree D = Disagree A = Agree SA = Strongly Agree |___| EQ2 (D3) I receive adequate job skills training, loans, agriculture support, and/or other support for my livelihood from CTA. SD = Strongly Disagree D = Disagree A = Agree SA = Strongly Agree |___| EQ2 (D4) I receive adequate support for my welfare and other special needs from CTA. SD = Strongly Disagree D = Disagree A = Agree SA = Strongly Agree |___| EQ1/2 (new) CTA is doing a good job to promote and preserve Tibetan culture SD = Strongly Disagree D = Disagree A = Agree SA = Strongly Agree |___| EQ2 (E1) I am aware of development project(s) being planned or implemented in my community/settlement. 1 = Yes 0 = No |___| EQ2 (E2) I am actively involved in the planning process of community/ settlement development projects. SD = Strongly Disagree D = Disagree A = Agree SA = Strongly Agree |___| (new) If E1 = yes Who is implementing these development project(s)? Do not read aloud, select all that apply. 1 = USAID 2 = UNDP 3 = NGO (local or international) 4 = Government 5 = Other (Specify) |___|,|___|,|___| If 5, specify: ___________________ USAID.GOV TASKING N050: TSRR EVALUATION | 74 EQ Q# (Baseline Q#) Question Response Options Record Response (new) If E1 = yes What programmatic areas are these projects in? Do not read aloud, select all that apply. 1 = SME Development 2 = Workforce Development 3 = Agriculture 4 = Education 5 = Health 6 = Other (Specify) |___|,|___|,|___| If 6, specify: ___________________ EQ4 (new) Are you aware of programs being implemented that are funded by USAID (cross reference with question above) 1 = Yes 0 = No |___| EQ4 (new) If yes to Q above: How well do you think those programs are preforming? 1 = Very Well 2 = Well 3 = Some room for improvement 4 = A lot of room for improvement |___| EQ4 (new) Regardless of response to Q above: In what ways do you think the programs implemented by USAID can improve? 1 = More focus on a specific sector (health/education/etc.) 2 = More focus on promoting the rights of Tibetan people in India/Nepal 3 = More focus on including Tibetans in the planning process 4 = Other, specify |___| If other, specify: ___________________ EQ3 (new) If there are limited resources is it better for a program to address multiple areas of livelihood (education, health, agriculture) or should the program focus in one area? 1 = Address multiple areas 2 = One area |___| EQ3 (new) What sector do you think is the most important to your wellbeing? 1 = SME Development 2 = Workforce Development 3 = Agriculture 4 = Education 5 = Health 6 = Other (Specify) |___|,|___|,|___| If 6, specify: ___________________ 75 | TASKING TO50: TSRR ASSESSMENT USAID.GOV EQ Q# (Baseline Q#) Question Response Options Record Response EQ3 (new) Which area do you think others in your community would consider the most important? 1 = SME Development 2 = Workforce Development 3 = Agriculture 4 = Education 5 = Health 6 = Other (Specify) |___|,|___|,|___| If 6, specify: ___________________ USAID.GOV TASKING N050: TSRR EVALUATION | 76 ANNEX IV: SOURCES OF INFORMATION THE FOLLOWING BACKGROUND DOCUMENTS WERE CONSULTED AND REVIEWED BY THE ET: AMS Consulting (undated). Tibetan Health System Capacity Strengthening Project Baseline Survey of Tibetan Community in India and Nepal. Central Tibetan Administration (CTA) Department of Education (September 2017). Diagnostic Assessment of Education Strategies and Programs in the Department of Education, CTA. CETRA Study Team (September 2019). Diagnostic Assessment of the Tibetan Early Grade Reading (TEGR) Program. CTA Department of Education (September 2019) Scholarship Review. ERU Consultants (November 2017). Assessment of Education Strategies and Programs Incorporating the Suggestions of the Department of Education, CTA. Intellecap (May 2018). CTA – Department of Education Strategic Planning Report. Intellecap (November 2017). CTA – Department of Finance Strategic Planning Report. Intellecap (August 2018). CTA – Department of Health Strategic Planning Report. Intellecap (July 2018). CTA – Department of Home Strategic Planning Report. Klatzel, F. (October 2017). HLBCC Assessment: a study of options for cultural strengthening of Tibetan refugees and Himalayan people in Nepal. LI-BIRD (September 2017). Assessment of Agriculture Development Initiative in Selected Tibetan Settlements in Nepal. Micro-credit Ratings International Ltd. (M-CRIL) (January 2018). Diagnostic Assessment of the Tibetan Entrepreneurship Development Initiative. Rajesh, A., Gupta & Gupta (July 2018). List of Indicators to Establish Baseline Values for EGR Indicators. The Tibet Fund (TTF) (October 2017). Assessment of the Status of Women and Girls in Tibetan Communities and Schools. TTF (August 2019). Program to Strengthen the Self Reliance and Resilience of Tibetan Communities in South Asia (TSRR) Baseline Report. TTF. TSRR FY2017 Annual Work Plan. TTF. TSRR FY2018 Annual Work Plan. TTF. TSRR FY2019 Annual Work Plan. 77 | TASKING TO50: TSRR ASSESSMENT USAID.GOV TTF. TSRR FY2019 Portfolio Review. TTF. (January 2018). TSRR FY2018 Q1 Quarterly Progress Report (QPR), Period October 1 – December 30, 2017. TTF (April 2018). TSRR FY2018 Q2 QPR, Period January 1 – March 30, 2018. TTF (July 2018). TSRR FY2018 Q3 QPR, Period April 1 – June 30, 2018. TTF (October 2018). TSRR FY2018 Annual Report, Period October 1, 2017 – September 30, 2018. TTF (January 2019). TSRR FY2019 Q1 QPR, Period October 1 – December 30, 2018. TTF (April 2019). TSRR FY2019 Q2 QPR, Period January 1 – March 30, 2019. TTF (July 2019). TSRR FY2019 Q3 QPR, Period April 1 – June 30, 2019. TTF (October 2019). TSRR FY2019 Q4, QPR Period July 1 – September 30, 2019. USAID.GOV TASKING N050: TSRR EVALUATION | 78 ANNEX V: TSRR SETTLEMENTS IN INDIA AND NEPAL The following annex provides a list of the locations in which TSRR is operating. There are a total of 39 Tibetan Settlements in India and 12 settlements in Nepal, as shown in the tables below. The sites visited by the field work teams are in italics. INDIA 15 AGRICULTURE-BASED • Choephelling, Miao (Arunachal Pradesh) • Dhargyeling, Tezu (Arunachal Pradesh) • Dhondenling, Kollegal (Karnataka) • Dickyi Larsoe, Bylakuppe (Karnataka) • Doeguyulgyelling, Herbertpur • Doeguling, Mundgod (Karnataka) • Kunphenling, Ravangla (Sikkim) • Lugsung Samdupling, Bylakuppe (Karnataka) • Norgyeling, Bandhara (Maharashtra) • Phendeling, Mainpat (Chattsigarh) • Phuntsokling, (Odisha) • Rabgyeling, Hunsur (Karnataka) • Sonamling, Ladakh (Jammu and Kashmir) • Tenzingang, (Arunachal Pradesh) • Tuting (Arunachal Pradesh) 13 HANDICRAFT-BASED • Bir Dege, Bir (Himachal Pradesh) • Bir Chauntra, Bir (Himachal Pradesh) • Bir Tibetan Society, Bir (Himachal Pradesh) • Phuntsokling, Dalhousie (Himachal Pradesh) • Gapa Tibetan Society, Kumrao (Himachal Pradesh) • Kham Khatok, Sataun (Himachal Pradesh) • Paonta Choelsum, Paonta (Himachal Pradesh) • Sakya Tibetan Society, Puruwala (Himachal Pradesh) • TashiJong, Bir (Himachal Pradesh)chaodisha • Tashiling, Pandoh (Himachal Pradesh) • Tashiling, Sonada (West Bengal) • Tibetan Bonpo Foundation (Himachal Pradesh) • Yangchen Gatselling, Shimla (Himachal Pradesh) 11 CLUSTER COMMUNITIES • Bomdila (Arunachal Pradesh) • Darjeeling (West Bengal) • Dharamsala, Kangra Distt. (Himachal Pradesh) • Dhondupling, Clementown (Uttarakhand) • Gangtok (Sikkim) • Kalimpong (West Bengal) 79 | TASKING TO50: TSRR ASSESSMENT USAID.GOV • Lingtsang, Munduwala (Uttarakhand) • Palrabling, Kullu & Manali (Himachal Pradesh) • Samyeling, Delhi • Shillong (Mehgalaya) • Dekyiling, Dehradun (Uttarakhand) NEPAL 4 AGRICULTURE-BASED • Delekling, Solukhumbu • Dorpattan, Baglung • Jampaling, Lodrik, Pokhara • Namgyelling, Lotserok 6 HANDICRAFT-BASED • Paljorling, Lodrik, Pokhara • Gyegyeling, Rasuwa • Samdupling, Jwalakhel, Kathmandu • Tashi Palkhiel, Pokhara • Tashiling, Pokhara • Samphelling, Walung 2 CLUSTER COMMUNITIES • ChoeJor, Boudha, Kathmandu • Gyalsa-Phakshing, Kathmandu UNITED STATES AGENCY FOR INTERNATIONAL DEVELOPMENT 1300 PENNSYLVANIA AVENUE, NW WASHINGTON, DC 20523