February 27, 2015 – discussion draft This publication was produced for review by the United States Agency For International Development. It was prepared by Ernest Leonardo, Ha Phung Thi Ngan, Mildred Howard, Cuong Nguyen Dinh, Hoang Trang Vu, Tung Nguyen Duy, and Management Systems International. July 2015 This publication was produced for review by the United States Agency for International Development. It was prepared by Mildred Howard (Team Leader), Cuong Nguyen Dinh, Hoang Tran Vu, Tung Nguyen Duy and Management Systems International. Mid-Term Evaluation of the Sustainable Management of the HIV/AIDS Response and Transition to Technical Assistance (SMART TA) Activity in Vietnam Vietnam Evaluation, Monitoring and Survey Services (VEMSS) Project SMART TA Mid-Term Evaluation – Final Report Contracted under AID-440-C-14-00003 USAID/Vietnam Evaluation, Monitoring, and Survey Services (VEMSS) Cover photo courtesy of SMART TA DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Final Report iii TABLE OF CONTENTS LIST OF FIGURES AND TABLES ...................................................................................................................IV ACRONYMS ....................................................................................................................................................... V I. EXECUTIVE SUMMARY ........................................................................................................................ VII III. INTRODUCTION ........................................................................................................................................ 1 STATUS OF HIV/AIDS IN VIETNAM............................................................................................................................................... 1 THE NATIONAL RESPONSE............................................................................................................................................................... 3 IV. THE DEVELOPMENT CHALLENGE AND THE U.S. GOVERNMENT RESPONSE ........................ 4 V. SMART TA ACTIVITY DESCRIPTION ..................................................................................................... 5 VI. EVALUATION METHODS AND LIMITATIONS .................................................................................... 7 PURPOSE AND SCOPE OF THE EVALUATION ................................................................................................................................. 7 METHODS ............................................................................................................................................................................................. 7 TECHNICAL LIMITATIONS ................................................................................................................................................................. 8 VII.PERFORMANCE AND IMPLEMENTATION .........................................................................................10 INTRODUCTION.................................................................................................................................................................................10 FINDINGS............................................................................................................................................................................................11 CONCLUSIONS....................................................................................................................................................................................22 RECOMMENDATIONS........................................................................................................................................................................24 VIII. TRANSITION ....................................................................................................................................... 24 INTRODUCTION.................................................................................................................................................................................25 SMART TA’S TRANSITION ACTIVITIES AND INTENDED RESULTS.........................................................................................25 GVN READINESS, ACCEPTABILITY AND RECEPTIVENESS TO TRANSITION..........................................................................35 CONCLUSIONS....................................................................................................................................................................................38 RECOMMENDATIONS........................................................................................................................................................................39 IX. BUILDING SUSTAINABLE CAPACITY .................................................................................................. 42 INTRODUCTION.................................................................................................................................................................................42 CAPACITY-BUILDING ACTIVITIES AND INTENDED RESULTS...................................................................................................42 ARE SMART TA CAPACITY-BUILDING ACTIVITIES SUSTAINABLE? .......................................................................................46 CONCLUSIONS....................................................................................................................................................................................48 RECOMMENDATIONS........................................................................................................................................................................48 X. MANAGEMENT AND COORDINATION .............................................................................................. 48 INTRODUCTION.................................................................................................................................................................................50 CONCLUSIONS....................................................................................................................................................................................54 RECOMMENDATIONS........................................................................................................................................................................54 ANNEX A. SMART TA SERVICE MAP ........................................................................................................... 55 ANNEX B. EVALUATION SCOPE OF WORK ............................................................................................... 56 BACKGROUND INFORMATION........................................................................................................................................................57 ANNEX C. EVALUATION KEY QUESTIONS............................................................................................... 70 ANNEX D. PERSONS CONTACTED ..............................................................................................................71 ANNEX E. EVALUATION TOOLS ................................................................................................................. 76 ANNEX F. KEY INFORMANT QUESTIONNAIRE RESULTS ................................................................... 80 ANNEX G. EXIT SURVEY RESULTS ............................................................................................................. 84 ANNEX H. SMART TA 2013 PMP TARGETS AND RESULTS ...................................................................... 85 Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Final Report iv LIST OF FIGURES AND TABLES Figure 1: SMART TA’s Interactive Objectives.............................................................................................................. 5 Figure 2: HIV Service in Ho Chi Minh City by Type, by Site, by Donor .................................................................. 9 Figure 3: HIV Prevalence Trends Among PWIDs in Provinces with SMART TA Assistance ........................... 11 Figure 4: HIV Prevalence Trends Among PWIDs in Provinces with SMART TA AssistanceError! Bookmark not defined. Figure 5: HIV Prevalence Trends Among MSMs in Provinces with SMART TA Assistance ............................. 11 Figure 6: Estimated HIV Incidence .......................................................................................................................... 12 Figure 7: Proportion of KPs Reached by SMART ................................................Error! Bookmark not defined. Figure 8: Proportion of MMT Patients Enrolled at SMART TA Clinics ..........Error! Bookmark not defined. Figure 9: Numbers of ARV Patients and Death Rates ..........................................Error! Bookmark not defined. Figure 10: Counseling and Testing at SMART TA DSDs ......................................................................................... 14 Figure 11: Cascade Performance from May to September 2014 at SMART TA DSDs ....................................... 15 Figure 12: Results of Cascade Performance of Remote and Mountainous Model in Nghe An .......................... 16 Figure 13: Number of MMT Patients in SMART TA DSDs .................................................................................... 17 Figure 14: SMART TA Reporting Process ................................................................................................................... 21 Figure 15: TA Needs Assessment Criteria .................................................................................................................... 26 Figure 16: Bi-annual TA Needs Assessment Performance CriteriaART service delivery ..................................... 26 Figure 17: Level of Technical Transition for C&T sites ............................................................................................. 34 Figure 18: Numbers of MOH Staff Trained by SMART TA .................................................................................... 43 Table 1: HIV and AIDS Estimates, 2013 ....................................................................................................................... 1 Table 2: Active USAID HIV/AIDS Activities in 2015 ................................................................................................ 4 Table 3: TB Support at SMART TA OPCs .................................................................................................................. 19 Table 4: Reduction to Date in Annualized ROCs by Province ................................................................................. 28 Table 5: Reduction in Unit Expenditure for Care & Treatment Services ............................................................... 29 Table 6: Summary of Direct Service Delivery (DSD) Site Administrative Transitioning Progress .................... 30 Table 7: Transition Status at the End of SMART TA by Service Type ................................................................... 31 Table 8: Key Indicators to Classify Care & Treatment Sites ..................................................................................... 34 Table 9: Revenue Sources for the Dien Bien 2015-2020 Plan ................................................................................... 36 Table 10: Summary of Survey of SMART TA HIV Service Delivery Site Personnel ........................................... 44 Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Final Report v ACRONYMS AIDS Acquired Immune Deficiency Syndrome AHF AIDS Healthcare Foundation ART Antiretroviral Therapy ARV Antiretroviral C&T Care and Treatment CBO Community-Based Organization CBS Community-Based Supporters CCRD Center for Community Health Research and Development CDC Centers for Disease Control and Prevention CHAI Clinton Health Access Initiative CMD Curative Medicine Division of the Ministry of Health COHED Center for Community Health and Development CoPC Continuum of Prevention to Care CSO Civil Society Organization D28 Decision 28 DOD United States Department of Defense DOH Department of Health DOS United States Department of State DQA Data Quality Assessment DSD Direct Service Delivery EOA Enhanced Outreach Approach EOP End of Project FHI 360 Family Health International 360 FSW Female Sex Workers FY Fiscal Year GARP Global AIDS Response Progress GF Global Fund to Fight AIDS, Tuberculosis and Malaria GFATM Global Fund to Fight AIDS, Tuberculosis and Malaria GVN Government of Vietnam HAIVN The Partnership for Health Advancement in Vietnam HCMC Ho Chi Minh City HFG USAID’s Health Finance and Governance Project HID Health Insurance Department HIV Human Immunodeficiency Virus HIVQUAL National Care and Treatment Quality Improvement Program HR Human Resources HSS HIV Sentinel Surveillance HTC HIV Testing and Counseling IBBS Integrated Behavioral and Biological Survey IBSS Integrated Behavioral and Surveillance Survey IPT Isoniazid Preventive Therapy KP Key Populations Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Final Report vi KQ Key Question LGBT Lesbian, Gay, Bisexual, Transgender LIFE Center for the Promotion of Quality of Life LTFU Lost to Follow-Up M&E Monitoring and Evaluation MARP Most-at-Risk Populations MMT Methadone Maintenance Therapy MOH Ministry of Health MOH-DPF Ministry of Health – Department of Finance and Planning MOLISA Ministry of Labor, Invalids and Social Affairs MPI Ministry of Planning and Investment MSH Management Sciences for Health MSM Men Who Have Sex with Men NIHE National Institute of Hygiene and Epidemiology OGAC Office of the U.S. Global AIDS Coordinator OPC Outpatient Clinic PAC Provincial AIDS Center PATH Program for Appropriate Technology in Health PEPFAR President’s Emergency Plan for AIDS Relief PLHIV People Living with HIV PCMD Preventive Care Medicine Division of the Ministry of Health PPC Provincial People’s Committee PWID People Who Inject Drugs QI Quality Improvement ROC Recurring Operating Costs SAMHSA Substance Abuse and Mental Health Services Administration SCMS Supply Chain Management Systems SMART TA Sustainable Management of the HIV/AIDS Response and Transition to Technical Assistance SOP Standard Operating Procedure SOW Scope of Work STI Sexually Transmitted Infection T&TA Training and Technical Assistance TA Technical Assistance TACHSS Technical Assistance and Health System Strengthening TB Tuberculosis TOT Training of Trainers UNAIDS United Nations Program on HIV/AIDS UNFPA United Nations Population Fund USAID U.S. Agency for International Development USG U.S. Government VAAC Vietnam Administration for HIV/AIDS Control VCT Voluntary Testing and Counseling VEMSS Vietnam Evaluation, Monitoring and Survey Services VUSTA Vietnam Union of Science and Technology Associations WHO World Health Organization Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report vii I. EXECUTIVE SUMMARY Human immunodeficiency virus (HIV), the virus that causes acquired immune deficiency syndrome (AIDS), has become one of the world’s most serious health and development challenges. Vietnam is considered to be a low-HIV-prevalence country with an epidemic concentrated in particular high-risk populations, which include: people who inject drugs (PWID) and their regular sexual partners; female sex workers (FSW) and their clients; and men who have sex with men (MSM). According to estimates from the World Health Organization (WHO) and the Government of Vietnam (GVN), an estimated 258,524 people living with HIV (PLHIV)1 in 2013 amounted to an estimated HIV prevalence in the general population (aged 15–49 years) of 0.39 percent. The national mean prevalence is consistently below 0.4 percent and has been falling over the past decade, as has mean prevalence among pregnant women, currently estimated at 0.15 percent. No national generalized epidemic appears imminent; however, the persistent and shifting dynamics of the epidemic among high-risk populations in Vietnam requires continued attention to avoid the threat of HIV spreading from high-risk groups to the general population. Vietnam’s national response. The government’s National Strategy on HIV/AIDS Prevention and Control, 2010–2020, covers a full range of considerations required for a robust national HIV/AIDS response. In implementing its national strategy, the government has relied heavily on technical and funding support from international donors. As of 2013, the GVN estimated that international funding sources account for 70 percent of funding for HIV/AIDS programs and services. As a middle-income country, funding support from international donors is rapidly shrinking, and Vietnam is under increased pressure to transition to fiscal and technical ownership of its national HIV/AIDS programs and services. U.S. response. The Global Fund to Fight AIDS, Tuberculosis and Malaria (the Global Fund) and the U.S. Government’s (USG’s) President’s Emergency Plan for AIDS Relief (PEPFAR) provide the bulk of international support to Vietnam’s national HIV/AIDS program. With oversight by the Office of the U.S. Global AIDS Coordinator (OGAC), PEPFAR provides funds and coordinates the activities of five USG implementing agencies in Vietnam. These are: the U.S. Agency for International Development (USAID), the U.S. Centers for Disease Control and Prevention (CDC), Department of Defense (DOD), Substance Abuse and Mental Health Services Administration (SAMHSA) and Department of State (DOS). Each implementing agency has its own focus and network of working relationships with the GVN. Under the Sustainable Management of the HIV/AIDS Response and Transition to Technical Assistance (SMART TA) cooperative agreement implemented by FHI 360, PEPFAR/USAID provides technical assistance and support to the Ministry of Health (MOH), particularly the Vietnam Administration for HIV/AIDS Control (VAAC) at national, provincial, district and site levels. SMART TA Activity. SMART TA is a five-year (Oct. 1, 2011, to Sept. 30, 2016), $45 million activity with an overarching purpose to: 1. Directly contribute to and support the GVN’s capacity to provide quality and effective programs and services for prevention, treatment, care and eventual eradication of HIV/AIDS. 2. Consistent with planned incremental phase-down and eventual withdrawal of PEPFAR support to Vietnam, to facilitate, support and monitor a rational transition of financial, technical and programmatic responsibility for quality, effective HIV programs and services to the GVN. 1 Kato, M. et al, “Enhancing the Benefits of Antiretroviral Therapy in Vietnam: Towards Ending AIDS,” WHO, 2014. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report viii SMART TA is expected to support and strengthen the GVN’s national response by delivering results in three interactive objectives:  Deliver effective, quality HIV services within the Continuum of Prevention to Care (CoPC)2 ;  Strengthen the capacity of the GVN to sustain high-quality, effective HIV programs and services over the long term; and  Facilitate transition of financial, administrative and technical ownership of CoPC services to the GVN. SMART TA’s financial, technical and programmatic assistance is intended to help Vietnam move toward its goals to eliminate AIDS by reducing acquisition and transmission of HIV among key populations, reducing mortality of PLHIV and improving quality of life, and providing support for the creation and use of HIV￾related strategic information. Planned SMART TA activities include development of efficient and cost-effective core and service delivery models and technical approaches that can be replicated in medium- and low-resourced provinces while maintaining quality and results. SMART TA provides direct service delivery support, including training, technical assistance and some percentage of recurrent cost/salary to at peak in Year 2 115 CoPC direct service delivery (DSD) sites. It additionally provides direct service delivery training and technical assistance (without recurrent cost or salary support) to 42 TA-SDI sites. Annex A is a site map showing SMART TA’s past or current technical assistance presence in nine provinces. Evaluation Scope and Methodology: USAID requested this mid-term evaluation to determine the extent to which SMART TA has met its stated objectives, identify areas in need of mid-course corrections and gain useful insights for consideration in future programming. This evaluation focuses on SMART TA’s objectives and a set of related key questions as defined in the evaluation scope of work (SOW). Key questions address the following four thematic areas: 1. Performance and Implementation – The extent to which SMART TA is contributing to reduced acquisition and transmission of HIV, reduced morbidity and mortality and increased generation and use of strategic information focused on most-at-risk populations (MARPs). 2. Transition to Country Ownership – The extent to which SMART TA is contributing to the transitioning of financial, human resource, administration and technical ownership of HIV/AIDS programs and services to the GVN and other stakeholders (including pre-transition preparation, transition and post-transition support). 3. Building Sustainable Capacity – The extent to which SMART TA is contributing to strengthened long-term sustainability and institutionalization of local technical capacity to provide HIV/AIDS programs and services. 4. Management and Coordination – Ways in which SMART TA can become more efficient, effective and relevant in achieving its objectives, given the changing contextual dynamics in its operating environment. This evaluation used a mixed-methods approach, which is detailed in the main report. 2 The Continuum of Prevention to Care (CoPC) or the HIV care continuum — sometimes referred to as the HIV Treatment Cascade, a conceptual model widely used by international HIV/AIDS experts — defines the sequential stages of HIV/AIDS that people living with HIV go through, from initial diagnosis to achieving the goal of viral-load suppression. It also shows the proportion of individuals within a defined population who are engaged at each stage. This information helps health planners determine where and how to focus strategic interventions to improve results at each stage of the Cascade. Source: AIDS.gov. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report ix Principal Findings and Conclusions Performance and Implementation The MOH’s HIV Sentinel Surveillance (HSS) data show a downward trend in HIV prevalence in all SMART TA provinces, also indicating rates of acquisition and transmission of HIV are declining. Although no direct statistical correlation can be made between declining HIV prevalence and SMART TA’s activities, service providers at the site level in all provinces visited by the evaluation team stated that innovations such as targeted outreach techniques introduced by SMART TA had substantially increased their capacity to reach, test and treat individuals in their target areas and improved their confidence in being able to provide effective programs and services to their communities. SMART TA has been instrumental in launching and institutionalizing a methadone maintenance therapy (MMT) program that has proven highly effective in bringing addicts into the health care system and linking patients to HIV testing, counseling and treatment. The GVN has fully adopted the MMT program, and all SMART TA-supported provinces are implementing local government funding support and the regulatory framework that will allow MMT services to be integrated and co-located with other GVN-supported HIV/AIDS-related community services. The fully functional, GVN-owned MMT program, as observed by the evaluators in one province, includes a user-fee system and directly observed treatment protocols managed at a walk-up counter that is co-located with an antiretroviral (ARV) treatment counter and tuberculosis (TB) screening services. Glitches remain in budgetary frameworks in some provinces, in the ability of some facilities to access GVN-supplied methadone and, according to some MMT clients interviewed; some clients feel stress about their ability to pay the user fees. In spite of these challenges, the number of people who received MMT services has grown enormously, from fewer than 4,000 enrolled in the first year to more than 15,000 by the end of the program’s third year. As of September 2014, Dien Bien Province supported 5 MMT sites serving 1,384 patients and was intending to expand to 21 MMT stations at the commune level which could provide service to 4,300 people who inject drugs (PWID). Almost all MMT enrollees have been tested for HIV; of those found to be HIV-positive, almost all have started ARV treatment. This has a direct impact on reducing morbidity and mortality. As several MMT clients noted in interviews, the SMART TA-initiated MMT program has reduced their episodes of illness and improved their quality of life. Aware of the limitations of traditional outreach models, SMART TA developed and assisted provincial partners to implement an innovative enhanced outreach approach (EOA), incorporating community-based HIV interventions to reach, test, counsel and retain key populations. A variety of community groups and civil society organizations, including traditional peer educators and hamlet health workers receiving performance￾based incentives, deliver these interventions. Preliminary results have shown improved efficiency: 83 percent of individuals reached with EOA took an HIV test (compared to 39 percent with the traditional model) and 97 percent of newly identified HIV￾positive individuals were successfully referred to care and treatment services (compared to 40 percent reported nationwide). The data suggests that EOA is potentially a key element in the CoPC Cascade framework. Challenges with EOA that would need to be addressed to sustain and scale up performance￾based incentive systems relate to how the MOH would evaluate performance and provide rewards within its own compensation protocols for its own workers, and how grants or other donor programs would standardize and ensure the verifiability of results. SMART TA has worked with provincial AIDS centers (PACs) in Nghe An province to develop a “Reach, Test, Treat, Retain” model for rural and mountainous regions (Quy Chau and Que Phong). This model mobilizes the existing networks of hamlet health workers to conduct outreach to areas where HIV services are limited, arrange mobile HIV testing and counseling and provide follow-up for those who test positive. This program has increased outreach results enrollment in treatment programs for Nghe An. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report x HIV care and treatment (C&T), the largest component of the SMART TA assistance portfolio, is achieving its targets of getting HIV-positive individuals on ART and keeping them in treatment, an indirect indicator that SMART TA-supported sites are indeed reducing AIDS mortality. While the number of ARV patients increased from 2012 to 2014, the annual death rate at SMART TA-supported sites has declined from 2.8 percent to 2.1 percent, significantly lower than nationwide recorded AIDS death rates among ARV patients. The percentage of adults and children known to be alive and on treatment 12 months after starting ART was 86.4 percent in year three, achieving the 84.2 percent target. From client exit interviews, patients reported to the evaluation team that their quality of life has improved and they reported high satisfaction with the quality of care and treatment service they are receiving at SMART TA-supported outpatient clinics (OPCs). In addition, a recent TB-HIV integration pilot shows initial success in bringing TB screening to ARV patients, a group with high TB prevalence in Thai Binh and Ninh Binh. This achievement resulted from efforts made by SMART TA in close collaboration with local implementing partners to use a smart technical monitoring system and quality improvement to more holistically assess the overall health of individuals who are HIV-positive. SMART TA has expended great effort and contributed substantially to HIV/AIDS-related strategic information systems development, training and technical assistance. SMART TA helped strengthen critical national strategic information surveys, such as the MOH’s Vietnam HIV/AIDS Estimates and Projections; the VAAC Sentinel Surveillance Survey; the National Care and Treatment Quality Improvement Program (HIVQUAL); national data quality assessment (DQA) protocols; and an improved MMT reporting system. SMART TA support is ongoing, with minor adjustments required in: systems design; functionality; capacity to manage data, perform M&E and ensure data quality; and application of strategic information. Substantial technical capacity has been institutionalized and these systems are certain to continue beyond the life of SMART TA. However, the online database introduced by SMART TA, although aligned with PAC and VAAC reporting requirements, operates only in SMART TA-supported provinces. FHI 360’s internal online indicator management database - VNIS – has not been designed to be part of the GVN M&E system and has therefore not been institutionalized beyond its PEPFAR reporting function. SMART TA has made a difference in the effectiveness of programs and services at the site level with introduction of the Cascade measurement tool. Based on the Cascade conceptual model widely used by international HIV/AIDS experts, the Cascade analytical tool introduced by SMART TA measures the number of individuals who do or do not move through the CoPC. In introducing this internationally recognized strategic planning tool, SMART TA has helped managers at the site level use strategic data and information to more effectively plan and manage their work. The Cascade measurement tool is well￾entrenched and in use at SMART TA-supported sites with many examples of its effectiveness in improving strategic programming and results at the site level. VAAC and other stakeholders recognize the Cascade tool’s positive contribution to the national response, particularly for high-risk populations. Receptivity appears to exist for expanding the use of the Cascade measurement tool in high-priority sites beyond the SMART TA network; the feasibility of this should be further explored. Transition to Country Ownership Consistent with planned incremental phase-down and eventual withdrawal of PEPFAR support to Vietnam, SMART TA’s transition approach has been to facilitate transition of financial and technical capacity (including administrative and human resources components) of CoPC programs and services away from project support to country ownership. In the government’s view, a sustainable HIV response requires a focus on practical transition planning; strategic integration of HIV/AIDS with other health interventions and services; alternative means of financing HIV/AIDS services; and strengthened multi-sectoral coordination. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report xi To support the GVN, SMART TA is implementing a transition strategy focused on activities and processes to both reduce financial costs (via programmatic, administrative and human resources efficiencies to align recurring operating costs, or ROCs, with current and expected GVN budgets) and transfer technical knowledge to help ensure that the quality of care is maintained during and after transition. The metric to assess transition progress comes directly from the SMART TA cooperative agreement, which USAID modified in May 2013, requiring SMART TA to transition “40 percent of its Continuum of Prevention to Care (CoPC) interventions, partners and sites to the GVN and local partners with resources coming from the Government, other donor sources and efficiency gains.” In practice, the required reduction is based on the recurring operating costs in DSD sites. These costs are negotiated annually with the PACs and thus represent explicit GVN acceptance of SMART TA’s annual transition plans and targets. Whether this metric is a sufficiently direct and robust measure of the intended result (GVN ownership of the HIV response) can be addressed only in the context of the nature and level of development assistance expected by the GVN to provide an effective HIV service delivery program in Vietnam. Discussions with central and provincial-level GVN officials made it clear that the GVN will not maintain all existing services at all sites or in the same way that SMART TA provides them. GVN will not assume financial ownership dollar-for-dollar of SMART TA services. In particular, the GVN is not committed to picking up more costly dedicated standalone SMART TA DSD sites. Rather, the GVN focus will be on integrating HIV services into the existing health care system. GVN intends to achieve a Vietnamese solution to the country’s HIV/AIDS epidemic. Their expectation is for close collaboration with and technical assistance from SMART TA to prepare the system of SMART TA-funded DSD sites for GVN ownership. Reducing costs is the first step in doing so. In fact, for financial transition to be successful, the costs of delivering the services must be sustainable — that is, affordable to the GVN. SMART TA, in collaboration with the GVN, now does what the GVN will do when it assumes financial responsibility of the HIV response with its own limited resources: phase out inefficient DSD sites, transfer DSD sites to other donors who will continue to fund activities beyond the life of SMART TA, reduce management costs and rationalize service delivery. SMART TA is making substantial progress in meeting the ROC reduction requirement. As of January 2015, with more than 20 months remaining, the project has reduced ROCs by 37 percent; this is evidenced in part by approved reductions in the sub-agreements with the provinces and has been confirmed in field visits to the five provinces that documented ongoing and committed transitioning of staff and facility costs. Two critical HIV response long-term financing issues repeatedly came up in field interviews: how to reform the Vietnamese health insurance program, projected in some provinces to account for 70 percent of the HIV response; and how the GVN will fund methadone and ARV medications when the donor assistance ends. Without progress on both issues, financial transition of HIV services will not be sustainable. The TA Network developed by SMART TA provides technical assistance to ensure continued quality service delivery post-transition. The service is critical, appreciated by the GVN and seminal for long-term professional development of HIV service staff, but its sustainability is challenging. Though discussions about sustaining the TA Network are ongoing with VAAC, and several provinces have suggested that they will support the TA Network in the near term, there is at present no clear plan for either a permanent base of operations or a business model to sustain the network. In interviews, GVN officials at all levels demonstrated considerable understanding, support and commitment to SMART TA financial and technical transition. Particularly at the provincial level, the government collaborates readily with SMART TA on financial and technical transition. Joint decisions by GVN and SMART TA indicate a high level of acceptance of the SMART TA transition approach and activities. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report xii The GVN — including the VAAC and various other central-level officials — is concerned about a perceived lack of direct engagement with the U.S. Government (USG) and USAID. They do not understand USG’s intentions or timeframe on future HIV assistance commitments, which makes planning uncertain. Building Sustainable Capacity Interviews with national and provincial GVN officials and other HIV assistance stakeholders (including WHO, UNAIDS and CDC) and results of a survey of HIV practitioners show that SMART TA has provided significant training and technical assistance to individuals and service delivery sites to effectively operationalize best practices in the Vietnam HIV response. In particular, VAAC and PAC officials noted that the level of initial training was of major importance for upgrading skills and understanding how to best provide HIV outreach, counseling, testing, care and treatment. The capacity-building assistance was extensive for provincial and district technical-professional staff and, to a lesser degree, administrative staff. SMART TA did not undertake a rigorous baseline assessment or time-series competence improvement analysis to document the direct effects of its capacity-building approach and activities. When SMART TA carried out “push” assistance, all initiatives were considered new or almost new to the staff of all sites; therefore, SMART TA directed its “push” assistance to build a basic level of critical competencies. SMART TA uses its monitoring tools to assess the level of functionality of service delivery sites, regarding the results as evidence of outcomes of the program’s capacity-building assistance. While the protocol of measuring results — e.g., a district center health worker is trained in organizing HIV referrals and the health referral activity that follows is documented as being well done — is valid in terms of trying to go beyond outputs to assess outcomes, the level of assessment directly linked to capacity-building activities is not robust and not clearly reported. SMART TA has established and continues to build the capacities of its TA Network of training and technical assistance advisers, but as noted above, while preliminary planning is being undertaken, for now there is no strategy or plan for how this cadre will be organized or administered post-SMART TA. Thus, it remains unclear whether SMART TA will achieve the intended result of nurturing a long-term sustainable Vietnamese system of learning and training that can ensure quality HIV services. In coordination with the USAID Pathway Project, SMART TA has provided technical assistance to Vietnamese civil society organizations (CSOs) including the Center for Community Health and Development (COHED), the Center for Community Health Research and Development (CCRD) and Life-Center. The support was limited to basic training courses, sharing materials and planning/coordination meetings. These organizations expressed a need to access additional HIV community outreach technical training and to have more opportunities to dialogue with SMART T A about common issues and techniques to ensure stronger outreach performance. GVN officials at the national level noted that SMART TA did not adequately engage VAAC and MOH in designing and implementing the capacity-building programs. They suggested that future USAID programming could include training and technical assistance at the national-level VAAC and MOH. All PACs spoke of the complexity and lack of value in the PEPFAR-required M&E approach. PAC and district officials in Ho Chi Minh City (HCMC) and Dien Bien were particularly blunt with their concerns about what they view to be burdensome and unnecessary data requirements; they said the database system used to collect, track and report on service delivery was “too sophisticated” for their needs. The officials said the system was project-driven in its design and implementation, inconsistent with GVN systems, and will not be used when SMART TA ends. Some officials expressed concern in the field interviews that SMART TA did not provide sufficient basic management training to complement the technical assistance and supportive supervision focusing on management skill sets. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report xiii Management and Coordination SMART TA has a flexible project design, organizational structure and management strategies (such as the push-pull approach to identifying priorities, operations research and pilot studies) that allow it to stay informed about changing contextual dynamics and the need to shift emphasis/approaches to enhance its efficiency, effectiveness and relevance. More strategic operations research and pilot studies may be necessary. To effectively use its limited resources, SMART TA may need to prioritize studies that resonate with national policymakers and have the most potential for national application and scale-up. SMART TA should continue with and further strengthen their work to engage VAAC decision-makers and policymakers in participating in and disseminating national policy-informing operations research. If possible, details should be included in the VAAC/SMART TA sub￾agreements. These sub-agreements are being approved only three to four months into the GVN’s annual operating cycle, and VAAC believes that the annual plan approval (and the program in general) would benefit from closer, more routine communications and contact between VAAC and USAID. While the SMART TA sub￾agreement period of performance is aligned with the GVN budget cycle, the preparation of materials is not. Thus SMART TA should work with USAID, VAAC and the provinces to better coordinate the process of sub-agreement preparation. In addition, SMART TA should facilitate arrangements for USAID to hold regularly scheduled review meetings with VAAC, the PACs and SMART TA to foster common understanding and agreement on technical approaches and progress concerning SMART TA’s transition and assistance plans. Interviews in the field indicated that SMART TA made efforts to train and sensitize its staff and service providers on gender issues, stigma and discrimination and related issues, particularly those that hinder the quality of services. SMART TA also has taken steps to sensitize its internal staff to gender considerations in its administrative work, including design, monitoring and evaluation. Where relevant, the activity disaggregated indicators by gender and reported them appropriately. However, attention to gender-specific programming as outlined in various workplans did not appear to transfer to SMART TA’s training and technical assistance activities in provinces, districts and sites. For instance, one activity to conduct a pilot of a comprehensive community-based harm reduction model for FSWs to link vulnerable women to a variety of health and social support on HIV, sexual health, gender-based violence (GBV), and income generation was not implemented, in part due to delays by MOLISA. The evaluators concluded that in general, gender considerations have not been robust, in part because they have not been well defined. SMART TA has more recently developed a gender strategy to guide more pro-active programming going forward. Recommendations Performance and Implementation 1. GVN Support. SMART TA should work more closely with the VAAC and other relevant GVN institutions such as VUSTA, to address issues related to the complexity of the program, the technical assistance needed to sustain quality care and GVN’s capacity to replicate and scale up key performance innovations, while focusing on developing sustainable Vietnam solutions to the nation’s HIV epidemic. These all require closer communication with and stronger advocacy to Vietnam’s government at all levels. 2. AIDS Treatment Protocols. SMART TA should continue to work with VAAC and MOH to advocate for the adoption of a higher eligibility threshold treatment protocol of CD4+ from 350 to 500, as recommended by WHO and other in-country partners. This would help reduce the loss of patients who are enrolled in OPCs but not eligible for ARV treatment, and thus do not get the early Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report xiv treatment that could be most effective. MOH has recently circulated a letter suggesting that this standard be used in hospitals and clinics, but the protocol is not yet an official policy of MOH. 3. Annual Viral Load Testing. The lack of critical testing of viral loads for all AIDS patients has drawn concerns, per VAAC. OPCs can test only “for patients who meet clinical or immunological criteria for treatment failure.” The report notes: “A single viral load measurement at 12 months is the most accurate measurement of ART adherence and can be used to target additional adherence interventions to those patients with the greatest risk for treatment failure.”3 4. Integration into the MOH Curative Care Division. SMART TA should help to prepare care and treatment DSDs to be integrated into the curative division of MOH wherever appropriate. ART treatment in particular would benefit from integration into MOH’s infectious disease treatment centers, as staff could develop a more comprehensive understanding of HIV and work more holistically to provide treatment. 5. Enhanced Outreach Approach. While early results from the program look promising, more comprehensive evaluation with greater involvement of government stakeholders is needed to document costs and benefits of the EOA program. 6. Reach, Test, Treat and Retain. The decentralized “Reach, Test, Treat and Retain” initiative for mountainous and rural areas is a critical SMART TA activity. Where appropriate, it should be expanded. This may require substantial modification in job descriptions of hamlet health workers, as well as related legal documents (MOH guidelines or directives), to fully integrate the model into the current health system. 7. SOPs for Lost to Follow-Up (LTFU). SMART TA should give special attention to developing and disseminating explicit standard operating procedures (SOPs) to prevent loss to follow-up and re￾engage patients into care and treatment. This initiative should involve VAAC and provincial program managers, site staff and community-based support groups. 8. Strategic Information System. SMART TA should continue efforts to strengthen the national HIV/AIDS strategic information system. In particular, SMART TA should work with the VAAC to streamline the project-centric M&E system for transitioned DSD sites to be more consistent with and supportive of GVN data requirements and capacities. Transition to Country Ownership Recommendations on SMART TA’s Transition Activities Going Forward 1. Planning for Transition. Planning for transition at the site level is challenging and complex. SMART TA has worked closely in the field with the provinces and PACs and the activity collaborates with the VAAC on its annual work plan, but VAAC would like to see more engagement on transition issues. Periodic workshops with VAAC and provincial officials may be useful to highlight site transitioning directions, lessons learned and remaining challenges. 2. Expand Access to Health Insurance Expanding access to health insurance for HIV services is critical to long-term sustainability. Assistance can be provided to: 1) Identify and assist the transition of care and treatment DSD sites where appropriate to MOH’s curative division within which HIV services are reimbursable; 2) Extend initiatives to register eligible patients for health insurance coverage; and, 3) Support OPCs to become certified by MOH as meeting basic health care standards. 3 “Results from the Vietnam ART Cascade Completion Study,” Vietnam Administration of HIV/AIDS Control, March 2015 Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report xv 3. Transition Progress Reporting. SMART TA should work with USAID to standardize simpler transition reporting templates. Current reporting does not reflect the collaboration, integration and results seen in field visits. 4. Monitoring and Evaluation. Provincial officials have asked for assistance in developing M&E reporting protocols for DSD sites to be transitioned, either directly to the GVN or to technical assistance only status (DSD/TA), that are more consistent with existing GVN systems, and thus more sustainable beyond SMART TA. SMART TA will need to work closely with PEPFAR Vietnam to advocate for alignment of PEPFAR reporting for TA-SDI sites to the GVN reporting system. VAAC is already planning to strengthen its own M&E system across the central, provincial and district levels of reporting. SMART TA could set a new protocols that when DSD sites are transitioned to DSD/TA status, only the reports of the national system, which covers main indicators required by PEPFAR, would be used. It is also recommended that SMART TA work with both VAAC and the PACs to institutionalize and build capacity within the TA Network for specific M&E technical assistance. 5. Communication with GVN. SMART TA’s participation on technical working groups convened by VAAC is much appreciated, and the activity should provide its wide and deep perspective more directly into the policy dialogue. One way to do this would be organizing and hosting a conference to draw on lessons learned from the ongoing transition process. A Wider Role for USAID Engagement with the GVN on HIV Response Activities 1. Donor-Coordinated Roadmap on Future Support. In discussions, both central and provincial GVN officials agreed that it is challenging to prepare rational, comprehensive, effective plans to assume ownership of the HIV response when no coherent, consistent donor roadmap outlines donor intents and timelines for HIV assistance in Vietnam. VAAC commented that existing working groups are not up to the challenge of partnering with the GVN on critical long-term transition issues; it would like to see stronger USG leadership on developing the roadmap, and subsequently, closer collaboration with USAID to explore technical assistance post-SMART TA. 2. Health Insurance. As noted, financial sustainability for the HIV response will require a Vietnamese health insurance program that covers HIV services. Officials with VAAC, the Health Insurance Department (HID) and the Ministry of Planning and Investment (MPI) noted the need for continued technical assistance that focuses on the specifications and cost scenarios of alternative national health insurance benefits packages with basic HIV services coverage. USAID’s Health Finance and Governance (HFG) project focuses on this subject. Whatever mechanism USAID decides to use, such technical assistance should continue. The GVN goal of funding 50 percent of HIV direct service delivery by 2015 and 75 percent by 2020 is premised on a robust health insurance mechanism. 3. Provision of HIV Medications. While it is not clear when donor assistance to subsidize ARV medications and methadone will end, GVN will eventually need technical assistance to be able to source HIV medications at maximum cost advantages. 4. Technical Support to Strengthen Health Systems Associated with Integration and Decentralization of HIV Service Delivery. While much progress has been made in preparing DSD sites for full transition, MOH needs help to mainstream HIV response activities into its curative care division or more holistic joint prevention-care facilities (i.e., infectious or opportunistic disease centers). Technical assistance for dedicated HIV service sites could expand to the larger integrated system, both for technical and administrative capacity building. Pilot programs to demonstrate how a well-integrated center could enhance HIV service delivery would help GVN strengthen the overall health system and ensure a comprehensive HIV response. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report xvi 5. Sustaining the TA Network. To maintain the quality of service delivery post-transition, the TA Networks developed by SMART TA needs a permanent home. Several suggestions have surfaced on how to institutionalize the development/implementation of life-long professional development streams and certifications. The activity should clarify current proposals and select a path forward. 6. Community-Based Outreach. To scale up HIV testing and enhance links to care and treatment services, SMART TA should set up longer-term support to various community-based outreach efforts using a performance-based incentive approach where appropriate. 7. Continued Support for SMART TA Sites Not Transitioned by 2016. Significant investment in the 39 sites that will not transition by the end of SMART TA could be lost if continued support is not provided. At this point, it appears unlikely that GVN will pick up the ROCs. Areas for Expanded USG–GVN Engagement to Ensure Effective Transition 1. Provincial-Level Transition Roadmap. Transition of DSD sites has been largely piecemeal, reflecting different timelines for phase-out of donor assistance, different priorities within and among provinces and varying prevalence rates in the districts. VAAC working with each province to prepare a tentative five-year plan for HIV service delivery would guide donor efforts to plan future assistance. 2. Progress on Health Insurance. While USAID can provide input on making health insurance inclusive of HIV services, the GVN will have to step up the pace to avoid a drastic cutback in the HIV response. 3. Home for TA Networks, Other Professional Certifications. To sustain the quality of HIV Continuum of Prevention to Care (CoPC), establish long-term dedicated HIV professional certifications and professional capacity-building programs. Building Sustainable Capacity On Ongoing Capacity-Building Assistance 1. In interviews, health care workers exhibited a clear idea of the training they need. To be more responsive to the individual needs and interests of SMART TA contracted staff and MOH personnel, SMART TA should focus more attention on “pull” capacity assistance. 2. To better assess capacity-building activities’ success, impact and cost-effectiveness, SMART TA should directly survey assistance recipients on their knowledge retention and use of the specific skills learned. Assessment of service delivery is notable in the effort to measure outcomes, but this does not comprehensively capture the direct impact of training and technical assistance. 3. SMART TA should continue providing technical assistance to and coordinate with three previously engaged CSOs (COHED, CCRD and Life-Center) that are direct service providers in PEPFAR￾supported provinces. If resources allow, SMART TA should also support other CSOs for a more extensive network of sustainable CSO-based service providers as per the MOU between SMART TA and VUSTA. 4. SMART TA should collaborate with VAAC and MOH to host workshops or conferences for all HIV professionals to better disseminate best practices, on-the-ground learning and perspectives of successes and failures and required next steps in capacity-building assistance. 5. SMART TA should work with VAAC to develop a more streamlined reporting system that aligns with GVN needs and expectations as noted prior. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report xvii 6. SMART TA should work with VAAC to identify and implement a plan for the next 12 months for operational research that addresses issues raised in the field related to the effectiveness and efficiency of pilot models such as TB/HIV integration, the 3 in 1 model, the 2.0 treatment program, and the HIV/AIDS in closed settings (i.e., prisons) protocol. Such studies would be useful for PACs as they continue to plan for transition, and would be important for VAAC in its efforts to institutionalize and scale up the service models. 7. SMART TA should focus more effort on teaching site managers to use data from the M&E system to analyze issues and plan consequent interventions, and on basic administrative skill sets in financial management, strategic planning and public participation. On Sustainability 1. SMART TA should continue discussion with the GVN to identify and develop strategies and/or business plans to sustain local HIV related technical professional development capacities, as specified in its capacity-building objective. SMART TA could begin by working with VAAC to share the SMART TA knowledge base of best practices so VAAC can begin to serve as the clearinghouse of such information. SMART TA should share all of its capacity-building training materials, guidelines for implementing the service delivery models and operational innovations. 2. More importantly, SMART TA should continue to work with VAAC and other GVN institutions to develop strategies to sustain its TA Network and its knowledge base. SMART TA should take the lead in collaborating with VAAC, USAID and PEPFAR to assess alternatives business plans for viable institutional arrangements to ensure long-term HIV capacity building in Vietnam. Management and Coordination 1. SMART TA should continue with and further strengthen the Technical Assistance and Health System Strengthening (TAHSS) unit to prompt VAAC decision-makers and policymakers to participate in and disseminate national policy-informing operations research. If possible, details should be included in VAAC/SMART TA sub-agreements. 2. SMART TA should support PACs to carry out coordinated planning across donor-funded programs that align with GVN operating cycles. 3. SMART TA should encourage and facilitate arrangements for USAID to hold regularly scheduled review meetings or make provincial site reviews with VAAC and SMART TA. This can foster common understanding and agreement regarding technical approaches and progress on implementation of SMART TA’s transition and assistance plans. 4. SMART TA should revisit its March 2014 gender strategy to identify ways to strengthen this element in its program, and incorporate this strategy into its workplans, and do follow-up reporting. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 1 III. INTRODUCTION The United States Agency for International Development in Vietnam (USAID/Vietnam) requested this mid￾term evaluation of the FHI 360 Sustainable Management of the HIV/AIDS Response and Transition to Technical Assistance (SMART TA) activity, carried out by Management Systems International (MSI) under the Vietnam Evaluation, Monitoring and Survey Services (VEMSS) project. The evaluation was conducted from December 2014 to March 2015. Status of HIV/AIDS in Vietnam Vietnam is a low-HIV-prevalence country with an epidemic concentrated in a high-risk population, including people who inject drugs (PWID) and their regular sexual partners; female sex workers (FSWs) and their clients; and men who have sex with men (MSMs). In 2013, an estimated 258,5244 people living with HIV (PLHIV) in Vietnam amounted to an estimated HIV prevalence in the general population (aged 15–49 years) of less than 1 percent (0.39 percent).5 The national mean prevalence is consistently below 0.4 percent and has been falling during the past decade, as has the mean prevalence among pregnant women, currently estimated at 0.15 percent.6 In the judgment of the Government of Vietnam (GVN) and international monitoring partners such as UNAIDS, no national generalized epidemic is imminent.7 That said, the persistent epidemic in Vietnam demands vigilance to avoid the ever-present threat of HIV spreading from high-risk groups to the general population.8 Injecting drug use is the leading contributor to HIV transmission in Vietnam, further fueled through sexual transmission. Data from the 2009 HIV/STI Integrated Behavioral and Surveillance Survey (IBSS) Round II and the 2013 HIV Sentinel Surveillance (HSS)9 estimate that as many as 40 percent of the estimated 271,000 PWID (range: 100,000–335,000) are living with HIV. PWID are found throughout the country, but an estimated 80 percent of the drug-using population lives in 22 of Vietnam’s 63 provinces. Recent efforts to provide PWID with sterile injecting equipment, methadone and ART, among other HIV services, appear to be having an impact. According to HSS data, prevalence among PWID decreased steadily from 2004 to about 10.3 percent in 2013, falling below 11 percent for the first time since 1997. While prevalence among PWID is decreasing in many provinces, the epidemic is alarmingly high in others. According to HSS 2013, HIV prevalence among PWID is high in Thai Nguyen (34 percent), Lai Chau (27.7 percent), Hanoi (24 percent), Quang Ninh (22.4 percent) and Ho Chi Minh City, or HCMC (18.2 percent).10 4 Kato, M. et al, “Enhancing the Benefits of Antiretroviral Therapy in Vietnam: Towards Ending AIDS,” WHO, 2014. 5 Vietnam Administration for HIV/AIDS Control (VAAC), “Vietnam HIV/AIDS Estimates and Projections 2011–2015,” Ministry of Health, Hanoi, Vietnam, 2012. 6 Ibid. 7 Ibid. 8 The experiences of China in 2003, and more recently Pakistan, are relevant examples of breakthrough epidemics in “bridge populations.” See “Pakistan faces HIV/AIDS spreading from high-risk groups to general population,” News Medical, August 2011. 9 HSS sample size for 2013 was 10,379 in 50 provinces according to Vietnam Administration for HIV/AIDS Control (VAAC), “Vietnam HIV/AIDS Estimates and Projections 2011–2015,” Ministry of Health, Hanoi, Vietnam, 2012. 10 Sentinel Surveillance Survey 2013. VAAC, 2013. Table 1: HIV and AIDS Estimates, 2013 PLHIV 258,524 Adult HIV Prevalence 0.39% PWIDs with HIV 108,400 Women (15+) with HIV 71,000 Children with HIV 5,000 AIDS deaths (2012) 12,000 Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 2 Another mode of HIV/AIDS spread is transmission through sex workers. HIV prevalence among female sex workers (FSWs) began declining nationally in 2002; in HSS 2013, it reached 2.6 percent, a level not seen since 1998. However, some provinces remain disproportionately affected.11 HIV prevalence among FSW varies by province and exceeds 10 percent in Hanoi, Hai, Phong, Can Tho and HCMC. Evidence also indicates that street-based FSWs have a relatively higher HIV burden compared to venue-based FSWs. An estimated 3 percent to 8 percent of FSWs also inject drugs; among them, the HIV prevalence is an alarming 25–30 percent. While 97.1 percent of FSWs reported using condoms with their most recent clients,12 the rate is much lower among PLHIV: just 41.1 percent.13 In recent years, an HIV epidemic among MSM has garnered greater recognition as studies and surveillance about the behavior of MSM found the prevalence increasing. HSS data for MSM in 2013 found an average HIV prevalence of 3.7 percent.14 HIV prevalence among MSM appeared to be high in major cities. Available data indicated a growing epidemic in Hanoi and HCMC, with HIV prevalence estimated at up to 16 percent in these urban centers. Unprotected anal sex was the main route of transmission among MSM. A small group of injection-drug-using MSM had a high prevalence and cannot be overlooked. In eight provinces for which data were available, HIV prevalence among MSM who inject drugs was 6 percent, while mean prevalence among MSM who reported no history of injecting drugs was 1.8 percent. IBSS II (data from four provinces) found similarly discrepant results between injecting and non-injecting participants. MSM PWID were estimated at 7.2 percent of the total MSM survey population; most were in HCMC and Hanoi. HIV prevalence among MSM was greater than 10 percent in three of the four provinces surveyed (HCMC, Hanoi and Hai Phong) and as high as 20 percent among MSM who had not sold sex in Hanoi. The estimated MSM population ranges from 191,000 to 573,000. Data remain sparse for MSM in Vietnam and the true magnitude of the epidemic will become clear only after one or two more survey rounds covering a greater geographic area.15 However, some worrisome indications point to a subgroup of MSMs who may put female partners and their offspring at risk. A rise in reported cases of HIV-positive women, representing 32.5 percent of newly reported cases in 2013, likely reflects a slow but steady HIV transmission to women by men who engage in highly risky behaviors. Furthermore, children born to HIV-positive mothers are at considerably greater risk of infection. Among 1,045,005 pregnant women tested in antenatal clinics in 2011, 0.14 percent of them had positive HIV test results and only about half of them (an estimated 57 percent) received ARV prophylaxis.16 In 2013, an estimated 5,000 children in Vietnam were HIV-positive.17 Another somewhat hidden subgroup is people at risk for tuberculosis (TB), a leading cause of death for people with HIV worldwide. In 2013, approximately 13 percent of new TB cases occurred in people living with HIV. However, between 2004 and 2013, TB deaths in people with HIV declined by 33 percent, largely due to scaling up joint HIV/TB services.18 TB is the sixth-leading cause of death in Vietnam19 with a well￾documented link to HIV infectivity, necessitating cross-referenced HIV/AIDS prevention and treatment. 11 UNAIDS Vietnam, “Vietnam AIDS Response Progress Report, 2014,” National Committee for AIDS, Drug and Prostitution Prevention and Control, Hanoi, Vietnam, 2014. 12 Ibid. 13 Ibid. 14 Sentinel Surveillance Survey 2013. VAAC, 2013. 15 UNAIDS Vietnam, “Vietnam AIDS Response Progress Report, 2014,” National Committee for AIDS, Drug and Prostitution Prevention and Control, Hanoi, Vietnam, 2014. 16 Ibid. 17 “Epidemiological Fact Sheet on HIV and AIDS in Vietnam,” UNAIDS, 2012. 18 “The Global HIV/AIDS Epidemic,” Global Statistics, AIDS.gov, 2014. 19 “Top 10 Causes of Death in Vietnam,” Centers for Disease Control and Prevention (CDC) Factsheet, 2014. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 3 The National Response The Vietnamese government’s official statements and its response in the past 25 years to the country’s concentrated HIV epidemic recognize and embrace the threats and challenges. Since the first reported case of HIV infection, the government has been fully conscious of the disease’s multifaceted danger, regarding it as a direct threat to the country’s sustained economic and social development, as well as to its people’s health. 20 The country’s political leadership — the party, the government and social organizations at all levels, from central to local units — has demonstrated strong commitment to the fight against HIV/AIDS. Vietnam’s national response suggests that the epidemic has been and continues to be viewed as a shared responsibility and an interdisciplinary mission.21 The government’s National Strategy on HIV/AIDS Prevention and Control, 2010–2020, 22 covers a full range of considerations required for a robust national HIV/AIDS response. This includes guidelines related to: governance and leadership; the policy and legislative environment; multi-sectoral collaboration for HIV prevention and control, including 17 ministries and arms of government; an institutional framework for prevention, treatment, care and support programs and service delivery in hospitals and outpatient clinics in all provinces/districts; and a Vietnam-specific strategy for involvement of civil society. As noted in the Global AIDS Response Progress (GARP) report,23 Vietnam has made significant progress toward realizing a national goal of universal access to HIV prevention, treatment and care. In 2014, progress included: a major policy change and legislation that relaxes the punitive approach to sex workers; scaling up the WHO–UNAIDS “Treatment 2.0” that aims to achieve sustainable universal access to HIV/AIDS treatment; strengthening linkages between HIV services and maternal and child health and sexual and reproductive health; and developing and strengthening civil society organizations (CSOs) and community￾based organizations (CBOs). The government readily acknowledges and appreciates the technical and financial contributions by its international and bilateral development partners to the many policy and program advances in Vietnam’s national response over the years. At the same time, the health care system — at senior, provincial, and even at facility-levels — recognizes that major HIV donors are reducing their funding or withdrawing from the country. Faced with significant decreases in international resources, the government is taking identifiable steps — e.g., assuming financial ownership of transitioned direct service delivery (DSD) sites — to ensure that recent progress is not reversed. In the government’s view, a sustainable HIV response will require a focus on practical transition planning; strategic integration of HIV/AIDS with other health interventions and services; alternative means of financing HIV/AIDS services; and strengthened multi-sectoral coordination.24 In spite of these challenges, the nation aims to achieve the ambitious targets in the National Strategy on HIV/AIDS Prevention and Control in Vietnam by 2020 with a vision to 2030, consistent with the global targets set out in the Political Declaration of the United Nations General Assembly Special Session of 2011. Vietnam also supports the new UNAIDS global treatment target: 90-90-9025 by 2020, and the global goal to “End AIDS by 2030.” 26 20 Yamamoto, T. and Itoh, S., “Vietnam, Fighting a Rising Tide: The Response to AIDS in East Asia,” Japan Center for International Exchange, 2006, pp. 266-284. 21 Ibid. 22 Fifth Draft Strategy of the National Committee for AIDS, Drug and Prostitution Prevention and Control, August 2011. 23 UNAIDS Vietnam, “Vietnam AIDS Response Progress Report, 2014,” National Committee for AIDS, Drug and Prostitution Prevention and Control, Hanoi, Vietnam, 2014. 24 Ibid. 25 Refers to these targets: 90 percent of all people living with HIV will know their HIV status; 90 percent of all people with diagnosed HIV infection will receive sustained antiretroviral therapy; and 90 percent of all people receiving antiretroviral therapy will have durable viral suppression. 26 “Optimizing Vietnam’s HIV Response: An Investment Case,” National Assembly Task Force on HIV/AIDS, draft 2014. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 4 IV. THE DEVELOPMENT CHALLENGE AND THE U.S. GOVERNMENT RESPONSE HIV has become one of the world’s most serious health and development challenges. In 2003, U.S. President George W. Bush announced the President’s Emergency Plan for AIDS Relief (PEPFAR) — the largest international health initiative in history by a single nation to address a single disease. Under the leadership of the Office of the U.S. Global AIDS Coordinator (OGAC), U.S. Government (USG) agencies work collaboratively to implement PEPFAR under the direction of each country’s U.S. ambassador. Fifteen focus countries supported by PEPFAR collectively represent at least 50 percent of HIV infections worldwide.27 Vietnam became the 15th focus country for PEPFAR in June 2004. As of 2014, significant complementary international donor support for HIV/AIDS in Vietnam comes from the multilateral agencies, the Global Fund and PEPFAR. U.S. agencies implementing PEPFAR in Vietnam are the Centers for Disease Control and Prevention (CDC), USAID, the Department of Defense (DOD), Substance Abuse and Mental Health Services Administration (SAMHSA) and the Department of State (DOS). 28 Each agency has its own focus and network of working relationships with the GVN. Although they work with the same PEPFAR funding stream but through different modalities, CDC and USAID are closely aligned in their underlying objectives of providing technical assistance and support to and through the Ministry of Health (MOH), particularly VAAC. CDC works directly within the MOH, providing technical assistance at the national level and in 26 high-priority provinces, districts and facilities, including developing surveillance and strategic information systems; implementing evidence-based public health programs; strengthening laboratory systems; building public health workforce capacity; conducting monitoring and evaluation activities; and translating research into public health policy and practice. In contrast, USAID provides assistance through international and local grantees and contracting agencies that coordinate through VAAC to assist in selected high-prevalence provinces, districts and facilities that can benefit from more intense, hands-on training and technical assistance at site levels. While no direct overlap exists between individual sites and specific HIV response activities assisted by CDC and USAID implementing partners, there is working collaboration with CDC on various aspects of their assistance programs.29 USAID funded HIV response activities are shown in Table 2. Table 2: Active USAID HIV/AIDS Activities in 2015 Activity Name Implementer Start / End Date Sustainable Management of the HIV/AIDS Response and Transition to Technical Assistance (SMART TA) FHI 360 October 2011 / September 2016 Healthy Markets Activity Program for Appropriate Technology in Health (PATH) April 2014 / March 2019 Supply Chain Management Systems (SCMS) Management Sciences for Health (MSH) June 2009 / June 2015 Health Finance and Governance Activity Abt Associates Inc. October 2014 / September 2016 Coordinated U.N. Support Toward a Sustainable and Effective HIV Response Joint United Nations Program on HIV/AIDS (UNAIDS) October 2012 / September 2014 USAID Community HIV Link (1) Center for the Promotion of Quality of Life (LIFE) (2) Community Health and Development (COHED) (3) Center for Community Health and Development (CCRD) May 2014 / May 2016 27 “U.S. Government Cooperation with the Socialist Republic of Vietnam on HIV/AIDS,” Office of the U.S. Global AIDS Coordinator, 2004. 28 “PEPFAR–Vietnam,” U.S. Embassy, 2012. 29 For example, the SMART TA activity has collaborated with CDC on strategic information systems and joint development of a reference directory on HIV/AIDS operations research in Vietnam. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 5 V. SMART TA ACTIVITY DESCRIPTION SMART TA is a five-year (Oct. 1, 2011–Sept. 30, 2016), $45 million activity with an overarching purpose to: 1. Directly contribute to and support the GVN’s capacity to provide quality and effective programs and services for prevention, treatment, care and eventual eradication of HIV/AIDS, and assist with the recent GVN commitment to achieve global UNAIDS 90-90-90 targets by 2020.30 2. Consistent with planned incremental phase-down and eventual withdrawal of PEPFAR support to Vietnam, facilitate, support and monitor a rational transition to the GVN of financial, technical and programmatic responsibility for quality, effective HIV programs and services. SMART TA is expected to support and strengthen the GVN’s national response in three objectives:  Deliver quality HIV services within the Continuum of Prevention to Care (CoPC)31;  Strengthen the capacity of the GVN to sustain high-quality, effective HIV programs and services over the long term; and  Facilitate transition financial, administrative and technical ownership of CoPC services to the GVN. Figure 1 shows how the three SMART TA objectives must work in tandem to deliver results. SMART TA’s financial, technical and programmatic assistance is intended to help Vietnam move toward AIDS elimination goals by: reducing acquisition and transmission of HIV among key populations; reducing mortality of PLHIV and improving quality of life; and providing support to generate and use HIV-related strategic information. Planned SMART TA activities include development of efficient and cost-effective core and service delivery models and technical approaches that can be replicated in medium- and low-resourced provinces while maintaining quality and results. SMART TA provides direct service delivery support, including training, technical assistance and some percentage of recurrent cost/salary to what was at peak in Year 2 115 DSD sites. It provides training and technical assistance without recurrent cost or salary support to an additional 42 TA-SDI sites. Annex A is a site map showing SMART TA’s past and current assistance presence in nine designated provinces. Within the context of SMART TA, “transition” refers to the process of giving the GVN financial, technical and programmatic responsibility for high-quality and effective HIV service delivery. SMART TA activities include assistance to the GVN at appropriate levels in planning and implementing institutional changes that will result in incremental transition over five years to GVN and local partners assuming a specified percentage of financial and technical responsibility for CoPC programs and services. These transition plans are intended 30 Vietnam has pledged to have 105,000 people on ART and 80,000 individuals on methadone maintenance therapy (MMT) by 2015. See also footnote 22. 31The Continuum of Prevention to Care (CoPC) or the HIV care continuum — sometimes also referred to as the HIV treatment Cascade, a conceptual model widely employed by international HIV/AIDS experts — defines the sequential stages of HIV/AIDS for people living with HIV, from initial diagnosis to the goal of viral load suppression. It also shows the proportion of individuals within a defined population who are engaged at each stage. This helps health planners determine where and how to focus strategic interventions to improve results at each stage. Source: AIDS.gov. Figure 1: SMART TA’s Interactive Objectives Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 6 to be implemented based on a systematic assessment of the willingness, capabilities and readiness to sustain such functions and services. SMART TA views quality training and technical assistance (T&TA) as the bedrock of the HIV response, incorporating emerging science and knowledge to strengthen high-impact interventions; fostering the continuation of efforts to achieve results; encouraging changes in programming for greatest efficiency; and ensuring the quality of CoPC services over time. SMART TA’s capacity-building activities target individuals, sites and provincial and national levels of the health system using a dual “push” and “pull” process. This allows SMART TA to push emerging science and knowledge while being responsive and tailoring T&TA to the interests and needs of local planners, managers and service providers. SMART TA is not only expected to provide effective T&TA, but within the context of its transition objective, SMART TA is charged with promoting institutionalized T&TA to continue to strengthen HIV programs and services in Vietnam. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 7 VI. EVALUATION METHODS AND LIMITATIONS Purpose and Scope of the Evaluation This mid-term evaluation sought to determine the extent to which the SMART-TA activity has met the objectives stated in its cooperative agreement and related modifications. Based on the activity’s objectives, USAID requested that the evaluators describe SMART TA’s accomplishments and weaknesses, adherence to implementation plans, management and coordination functions, capacity building and knowledge/skills transfer and the outcome of its efforts to date to facilitate the transition of activity support to government ownership. USAID asked the evaluators to make recommendations for mid-course corrections and future programming. The team leader for this evaluation was Mildred Howard. Other team members are listed on the cover page. The evaluation was independently conducted. The evaluation scope of work (SOW) is included as Annex B. The full list of evaluation key questions (KQs) is in Annex C. Methods Being able to answer the evaluation’s KQs was a primary concern in formulating its conceptual framework, methodology and methods. The evaluation design involved construction of a matrix that analyzed each KQ to determine the specific data or information needed, where that piece of information would be obtained and what data analysis method would be used. This approach ensured continuous focus on all KQs throughout the data collection and analysis exercise. The evaluation used a mixed-methods approach, which included an extensive review of activity documentation and relevant literature, quantitative assessments involving secondary analysis of data and statistical reports from SMART TA activity files, qualitative assessments involving key informant interviews and a companion questionnaire and random interviews with clients as they exited service facilities. Both SMART TA and USAID gave input to identify key informants, ensuring that the evaluators held discussions with a core group of activity partners, collaborators and stakeholders. Key informants represented GVN and international agencies at the headquarters level as well as provincial policy, management and facility levels. A list of people interviewed individually or in groups is in Annex D. Samples of data collection instruments used for key informant interviews and exit interviews with target beneficiaries are in Annex E. Discussion of each method and relevant limitations follows. Desk Reviews: The evaluators initially received a limited number of core activity documents, supplemented with extensive activity documentation provided by SMART TA. The evaluation team participated in several SMART TA staff presentations, briefings and follow-ups, and activity staff generously provided file documentation requested by the evaluators. All evaluation team members were encouraged to familiarize themselves with all documentation, and a Dropbox account was established to facilitate sharing of documents and information. A drawback of the desk review is that it is extremely time-consuming. However, if carried out comprehensively by all evaluation team members, it adds invaluable information to the team’s understanding of the facts and issues. Key Informant Interviews: The evaluation team conducted face-to-face interviews with individuals and groups at the headquarters, provincial and facility levels. Group meetings, documented with meeting notes, and interviews (both group and individual) followed a standardized structured qualitative interview guideline aligned with key questions. A companion qualitative questionnaire (rating scale) was also used either before or after the interview session, giving evaluators additional qualitative input from key informants. Although Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 8 extremely useful for capturing essential perceptions and insights, the qualitative interview and questionnaire method generate subjective input that is vulnerable to influence by many factors, including the composition of participants. The objectivity of this qualitative information can be improved through verification, where possible, with quantitative inputs and triangulation. Exit Interviews: One-on-one conversations with randomly selected clients and beneficiaries were chosen instead of the standard focus group discussion because evaluators expected that participants would be more accepting of a private exit interview and it would be more conducive to disclosure. Many candidates were quite open to discussing personal details and experiences with the services they were receiving. Site Visits: Separate teams visited Hanoi, Dien Bien, Nghe An and Hai Phong provinces, and the full-time team members spent five days in Ho Chi Minh City and surrounding districts. With some variation, visits adhered to the administrative structure of the health system at the provincial level, usually including the Provincial Department of Health, the Provincial AIDS Committee and district health facilities where patients receive services. In some cases, these were freestanding service outlets and in other cases they were co-located in a hospital or other outpatient health facility. During site visits, the evaluation team used a standard protocol of meetings with management staff and service providers, while one or two evaluators separately conducted private exit interviews. Team Information Sharing, Data Synthesis and Triangulation: As a further, non-structured approach to refining and processing evaluation data and information, team members were required to fully use the qualitative interview instrument to take notes and document interviews and discussions. Team members shared the resulting notes with each other as part of their data synthesis process. At every step, from the initial document review to synthesizing final recommendations, the team fully incorporated each member’s and key informant’s knowledge and insights into formulating findings, conclusions and recommendations. The team held weekly wrap-up discussions and devoted a full week to drafting, team reviews, synthesis and integration of findings, conclusions and recommendations. Technical Limitations Questions of Outcome, Impact and Attribution: Several key questions are phrased to raise issues that enable evaluators to measure impact. Terms such as “extent” and “contributing” (i.e., attribution) connote measurement of impact that ideally require time-series data that could be attributed to project activity. But this information was not available, thus limiting the team’s ability to assess true activity impact. Instead, the evaluators used a mixed-methods data analysis approach, which presents several technical limitations in what conclusions can be drawn from the evidence. For example, for these purposes, quantitative data on incidence, with periodicity, would have been most appropriate to determine trends in HIV acquisition and transmission, but incidence data was not available. Instead, the team measured trends using available prevalence data spanning eight years. Results were supported by qualitative (anecdotal) information from provincial and facility-level staff during key informant interviews and use of a companion anonymous questionnaire. The questionnaire, however, as used in this evaluation is not statistically valid and it has an in-built bias.32 Figure 2 shows the various HIV service activities by delivery site and by donor for HCMC. Because data on reductions in prevalence are gathered at the provincial level, attributing results to a specific donor or site is not possible. At best, one could say that SMART TA provides about a third of the overall HIV response assistance in the province. Assuming the activity’s assistance achieves the same level of results as the assistance from GVN, Global Fund (GF), CDC, the Clinton Health Access Initiative (CHAI) or the AIDS 32 The tendency of an individual to respond to a question positively (or negatively), believing this is what is expected. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 9 Healthcare Foundation (AHF), then one-third of the results could be attributable to SMART TA. A rigorous assessment of SMART TA’s effectiveness compared to other donors is beyond the scope of this evaluation. Figure 2: HIV Service in Ho Chi Minh City by Type, by Site, by Donor Gender Issues: SMART TA activity documents and published literature reflect awareness of gender considerations. HIV/AIDS prevention and treatment programs in Vietnam are targeted to the conditions and consequences of affected populations regardless of patients’ gender. Because sexual behaviors and sexual exploitation are intrinsic to risk of infectivity, health planners and service providers tend to focus on risk factors, risk behaviors and patterns of acquisition/transmission in the most-at-risk populations (MARPS), rather than on gender. However, services are customized based on gender where warranted, e.g., special counseling for pregnant women or wives who are infected by their husbands. Within this evaluation, where possible and relevant, analysis has noted gender distinctions. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 10 VII. PERFORMANCE AND IMPLEMENTATION Introduction SMART TA is designed to contribute directly to the targets identified in the National Strategy on HIV/AIDS Prevention and Control in Vietnam and the Partnership Framework between the USG and the GVN for HIV/AIDS Prevention and Control. SMART TA also prioritizes programming that aligns to the recently strengthened national and provincial HIV Continuum of Prevention to Care (CoPC) frameworks. SMART TA supports a wide range of program activities within the CoPC. SMART TA provides DSD support to government implementers and technical assistance CSOs and community-based groups. At the beginning of Year 4, SMART TA had operationalized 38 care and treatment (C&T) direct service sites that provide a full range of medical care, including physical examinations, drug management and antiretroviral (ARV) medications distributed at no cost by the USAID Supply Chain Management Systems (SCMS) project. SMART TA has implemented effective referral procedures between outreach and testing sites and the C&T sites, and more recently between HIV/TB sites. This ensures routine TB screening in HIV outpatient clinics (OPCs) and HIV testing in TB clinics. An important aspect of all of SMART TA’s CoPC assistance is its enhanced strategic information system, which includes M&E data reporting, TA monitoring, service quality controls and improvement and operational research to identify issues requiring management attention. Prevention has focused on an enhanced outreach approach (EOA), incorporating community-based HIV interventions to reach, test, counsel and retain key populations and using traditional peer educators and hamlet health workers who receive performance-based incentives to deliver these interventions. Other prevention assistance includes: training and technical assistance in support of HIV counseling and testing; and the development and implementation of methadone maintenance therapy (MMT) plans as a point of entry to bring the primary at-risk cohort — people who inject drugs (PWID) — into the CoPC cascade. This evaluation assesses performance and implementation progress to date, and highlights program achievements and challenges related to the key questions presented in the evaluation workplan:  To what extent are the project’s activities, techniques and processes employed by SMART TA contributing to the intended results of reducing acquisition and transmission of HIV with focus on MARPs, as specified in its PMP?  To what extent are the project’s activities, techniques and processes employed by SMART TA contributing to the intended results of reducing morbidity and mortality of PLHIV and improving quality of life, as specified in its PMP?  To what extent are the project’s activities, techniques and processes employed by SMART TA contributing to the intended results of providing targeted support for the generation and use of HIV￾related strategic information, as specified in its PMP?  What challenges and lessons can be drawn for improving implementation in the next two years?  What are key recommendations for better designing future projects by USAID and GVN? A review of Year 3–2014 SMART TA PMP performance indicator targets and results (see Annex H) shows results for 20 of 44 indicators; 24 targets have not been set, due to ongoing unresolved discussions with local partners and changes in PEPFAR indicators at the global level. The project has achieved greater than 110 percent of the target for eight of 20 DSD and TA indicators reported, between 90 percent and 110 percent for nine targets and less than 90 percent for the remaining three targets. For these three, outreach protocols changed to focus exclusively on high-risk populations, reducing the number of people reached. Several spot checks of data did not identify any issues with data collection or reporting methods. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 11 0 20 40 60 80 100 2005 2009 2013 % HCMC Quang Ninh Nghe An Yen Bai Hanoi Hai Phong An Giang Can Tho Dien Bien 0 20 40 60 80 100 2005 2009 2013 % An Giang SSW Can Tho SSW Hanoi SSW Findings Reductions in Acquisition and Transmission of HIV among MARPs The latest data from the GVN HIV Integrated Behavioral and Biological Survey (IBBS)33 , shown in Figure 3, indicates declining HIV prevalence in all but one of the provinces where SMART TA assistance is provided. This is consistent with the reduction occurring during SMART TA’s predecessor project. HIV prevalence among PWIDs in HCMC, Hai Phong and Quang Ninh, where the PWID epidemic was most serious, has decreased significantly, from more than 60 percent in 2005 to 30 percent in 2013. No research to date explains the increased prevalence in Hanoi between 2009 and 2013. Figure 3: HIV Prevalence Trends among PWIDs in Provinces with SMART TA Assistance 33 NIHE, MOH, FHI 360 and CDC. HIV/STI Integrated Behavioral and Biological Surveillance Round III. Draft Report Version 12/15/2015. SMART TA Start Date 0 20 40 60 80 100 2005 2009 2013 % Can Tho Hanoi Hai Phong HCMC Figure 4: HIV Prevalence Trends among MSMs in Province with SMART TA Assistance s with SMART TA Assistance Figure 5: HIV Prevalence Trends among FSWs in Provinces with SMART TA Assistance Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 12 0 2,000 4,000 6,000 8,000 10,000 12,000 2010 2011 2012 2013 2014 2015 Number of New HIV Infections Smart TA (9 Provinces) Number of New HIV Infections Non-SMART TA Provinces As shown in Figures 4 and 5, the evaluation team also observed a decline in HIV prevalence among FSWs and MSM in provinces where SMART TA provides support.34 SMART TA works in HIV high-burden provinces, which jointly account for more than half of HIV/AIDS cases in the country. Therefore, the observed HIV prevalence in those provinces is much higher than the average in the non-SMART TA provinces. However, as shown in Figure 6, 35 the trend line in the provinces where SMART TA provides assistance shows a more substantial reduction in incidence than in the other provinces. This data suggests that the epidemic among key populations in SMART TA-assisted provinces is declining at a rate that can be characterized as having the epidemic under control. Discussions with PACs in key SMART TA provinces tell an encouraging story in that the direct correlation between declining trends in HIV prevalence in MSMs is probably connected to safer sexual habits. Consistent condom use among FSWs and MSMs remained at high levels (90 percent or more in most provinces). Percent of PWIDs who reported safe injection also increased in many SMART TA provinces in the past few years, from 60 percent to 82 percent in HCMC, 82 percent to 94 percent in Hanoi and 85 percent to 97 percent in Hai Phong Province.36 Figure 6: Estimated HIV Incidence37 Although correlation cannot be established statistically within the limits of this evaluation, service providers at one site reported to the evaluators that their team has shown generally increased effectiveness using approaches introduced by SMART TA (including outreach, methadone maintenance, therapy programs and HIV testing and counseling), contributing to reductions in both HIV incidence and prevalence.38 Overall, as shown in Figures 7 and 8, the number of individuals reached in the key populations (KPs) by SMART TA program in FY 2014 accounted for 10 percent of total estimated numbers of KP individuals in the nine provinces where SMART TA provides assistance (and one-fifth of the total of KPs reached by all implementing partners, according to VAAC). With 5,644 patients on treatment by the end of Year 3, participants in the MMT program supported by SMART TA accounted for almost half the total MMT patients in the nine provinces. 34 VAAC, NIHE. HIV Sentinel Surveillance, 2013. 35 VAAC and its international partners have produced incidence data through mathematical modeling of prevalence data. 36 NIHE, MOH, FHI360 and CDC. HIV/STI Integrated Behavioral and Biological Surveillance Round III. Draft Report Version 12/15/2015. 37 VAAC, UNAIDS, WHO. HIV/AIDS Estimates and Projection for 2015–2020. 38 Measurement of prevalence is a proxy measurement for acquisition/transmission. SMART TA Start Date Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 13 90% 10% # KP reached by SMART TA - supported program… Figure 7: Proportion of KPs Reached by SMART TA in SMART TA Provinces Reductions in AIDS Morbidity and Mortality The 16,458 patients receiving ARV treatment by SMART TA at the end of Year 3 have surpassed the PMP target of 16,000 and account for almost 43 percent of all ARV patients in the nine provinces where SMART TA works, and one-fifth of the total number of patients nationally. ARV prevented an estimated 11,00039 deaths in Vietnam; SMART TA contributed significantly to this tally through its identification and treatment assistance. Annual death rates among ARV patients who are enrolled at SMART TA supported clinics have declined since 2012 (from 2.8 percent to 2.1 percent).40 This is lower than the death rate estimated from a national evaluation (range: 11.6 percent after the first year to 17.8 percent after four years of treatment).41 No PMP indicator addresses a reduction in morbidity, understood as an unhealthy high viral load. Viral load testing is done only for patients who meet clinical criteria for treatment failure. Thus, morbidity rates for all AIDS patients have not been established as a baseline to measure reduction. Testing results in general do show that after one year of ART, viral loads drop significantly (to <100 cps/mL) for 90 percent of patients. SMART TA’s Technical Approach The evaluators found several technical developments that are likely to be programmatically linked to declining prevalence rates in SMART TA target provinces. 39 VAAC, UNAIDS, WHO. HIV/AIDS Vietnam estimates and projections until 2020. The 2013 round. 40 SMART TA Monitoring and Evaluation Data Base, 2014. 41 Nguyen DB, Do NT, Shiraishi RW, Le YN, Tran QH, et al. (2013). Outcomes of Antiretroviral Therapy in Vietnam: Results from a National Evaluation. PLoS ONE 8(2): e55750. doi:10.1371/journal.pone.0055750 Figure 8: Numbers of ARV Patients and Death Rates 47% 53% # MMT patients in 9 provinces Figure 8: Proportion of MMT Patients Enrolled at SMART TA Clinics Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 14 The CoPC Cascade Framework Methodology As consistently noted by GVN officials and other HIV stakeholders such as WHO, SMART TA introduced a useful management tool to help provincial and district managers strategize and strengthen their local programs. The CoPC Cascade is a system to monitor the number of at-risk individuals who receive the prevention and care services they need. It captures the steps necessary for everyone who needs HIV care to remain engaged in it — from an initial stage of being reached to getting tested for HIV, to being able to suppress the virus through treatment. The CoPC Cascade measurement tool (Cascade) helps provincial partners engage in timely analysis of readily available service data to pinpoint performance gaps and identify solutions to improve service delivery efficiency and effectiveness. The evaluators heard from many provincial and district managers that the Cascade tool is highly valued because:  It synthesizes existing data (mostly program reporting data) so users can find potential problems in CoPC Cascade. For example, the number of people who receive ARV services is significantly smaller than the number identified as HIV positive. The Cascade provides a complete and easily understood picture of what’s going on in the service environment (among key populations and within the broader community);  Its information can be used for multiple purposes, including program planning, health education and messaging, progress tracking and advocacy; and  It does not require advanced statistical techniques and data generation is not time-consuming. Improving Cascade Performance through Enhanced Outreach Approach (EOA) In Year 3, aware of the limitations of tradition outreach, SMART TA developed and assisted provincial partners in implementing an enhanced outreach approach. The EOA is designed to help local implementing partners better reach, test, treat and retain key populations in the HIV service system using locally available resources. In this model, outreach workers are commissioned to reach only high-risk individuals. The model was developed in consultation with multiple stakeholders, including local partners. By the end of Year 3, 22 sites in the nine SMART-TA supported provinces had implemented the model. Figure 10 shows the number of individuals tested and counseled in the first three years of SMART TA. While the number of people tested and counseled decreased significantly after EOA was introduced in Year 3, it may be that in Years 1 and 2 too many low-risk individuals entered the system. This is not an effective use of the activity’s limited resources. The Year 3 total, though lower in number, is more targeted to high-risk key populations and thus reflects a more efficient approach. As well, individuals referred to HIV testing and counseling through the EOA may not have gone to SMART TA HTC facilities and thus not reflected in the data below. Figure 9: Counseling and Testing at SMART TA DSD sites Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 15 “The approach has successfully mobilized available resources at sites. Reached KP individuals are able to get services at any service delivery points, not only SMART TA-supported ones.” — Hai Phong PAC Outreach Manager “SMART TA has provided comprehensive training and support on EOA implementation and management through the local network of master trainers.” — Dien Chau Outreach Staff “With this innovative model, both program managers and outreach workers are clear about target populations, hence enabling their ability to prioritize their work.” “The performance-based incentive system in EOA increases outreach workers’ motivation.” “Improvement of cost-effectiveness. This has been seen with the average numbers of KP individuals an outreach worker reached per month, and the reduction in cost per person reached.” “It is replicable for non-SMART TA-supported sites. Actually, discussions are now ongoing about the possibility of implementing EOA in GFATM (Global Fund on AIDS, TB and Malaria) sites.” — Hai Phong PAC Leaders Figure 10: Cascade Performance from May to September 2014 at SMART TA DSD sites SMART TA data from the Cascade assessment for the first six months of the EOA, shown in Figure 11, provides some evidence of the effectiveness of the new approach:  84 percent of KPs reached with EOA tested for HIV, compared to 39 percent under the traditional model, and  97 percent of identified HIV-positive cases were referred to care and treatment service (compared to nationwide figures of about 40 percent). Because the EOA is designed to reach the highest-risk key populations, it can more directly help prevent HIV. This is reflected in the fact that SMART TA’s reach of HIV-positive individuals was 6 percent in Years 1 and 2, and 5 percent in Year 3. The overall decline in HIV prevalence is attributable to various prevention activities, including the EOA protocol that includes a focus on people who are HIV￾negative, but practice high-risk behaviors. Central and provincial managers who spoke with the evaluators frequently mentioned the advantages and success of this approach. A sample of their comments is provided in the box to the right. One challenge the evaluators encountered is the coordination at sites with multiple outreach approaches. Different donor reward systems in cities or provinces where multiple donors provide financial and technical support may cause inefficiencies in managing the outreach Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 16 “It is a most effective model for rural and mountainous areas where HIV service is limited, and there are many barriers such as geographical distance, languages and different culture.” “The use of hamlet health workers ensures the sustainability of the model, since hamlet health workers are in the current health system.” “Supporting materials such as flipcharts and cards were well designed and useful.” — Nghe An Provincial and District Managers, 2015 program. A concern also exists about the sustainability, without external funding, of EOA and its commission reward protocol. The performance-based incentive system does not fit current government financial mechanisms and policy, which focus on general supplemental stipends for critical and potentially hazardous work related to infectious disease control. In addition, stakeholders still have a limited awareness of EOA. Given the short time of implementation, its effectiveness has not been well documented or shared widely. Increasing visibility would be one of the most important steps to mobilize local (government) resources for sustainability. The Reach, Test, Treat and Retain Remote and Mountainous Model An innovation that could be sustainable is the use of hamlet health workers for outreach to KPs. To address geographical and language barriers that prevent KPs in rural mountainous areas from accessing HIV services, SMART TA has worked with the PACs in Nghe An to develop a model to mobilize the existing network of hamlet health workers. Hamlet health workers are responsible for outreach in areas with limited HIV service. When high-risk individuals are identified, they are referred to HTC services and are entered into the reporting system via mobile technology. Identified HIV￾positive people are referred to care and treatment services. The workers have been testing mCare, a software that supports client tracking and management, to facilitate the process. This model of deploying hamlet health workers is in use in the two rural and mountainous districts of Nghe An, Que Phong and Quy Chau. Given the short period of implementation, there has not been sufficient time to systematically measure the effectiveness of the model. However, an initial cascade analysis shows notable effectiveness to date. Figure 12 shows, in red, the improved results in the two districts. Figure 11: Results of Cascade Performance of Remote and Mountainous Model in Nghe An Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 17 The use of hamlet health workers has great potential to more effectively reach KPs and deliver initial HIV services to them. However, several relatively minor operational issues should be addressed. One criticism noted in Nghe An was related to the introduction of mobile technology for patient tracking. Hamlet health workers use their mobile phones to send client information to the PAC IT system. When clients visit HIV testing and counseling (HTC), the system notifies health workers that their client had voluntary testing and counseling (VCT) service. The system also provides weekly notifications of clients who did not receive expected services. These procedures are also used when clients move to care and treatment. Several issues were identified with this technology. The technological capacity of local staff in mountainous areas is limited; despite training, some found the system difficult to use. When the evaluation team visited Nghe An, the mCare system was not working and PAC consequently had no record to provide the rewards. Also, the system requires good mobile service signal strength, which is not always available in remote areas. An alternative approach would be to use manual logbooks in areas where mobile technology cannot be implemented. Another issue is that for USAID compliance, the reward policy under the EOA requires that hamlet health workers travel to district offices monthly to submit claims and receive their rewards. Their regular paychecks are deposited into their accounts and they do not have to travel to district offices to submit timesheets or written evidence of their work. When donor assistance ends, EOA commission incentives are likely to revert to existing MOH supplemental payments, in the form of a general stipend, for work related to infectious diseases. However, for workers to qualify for this supplemental payment, their job descriptions should reflect the type and level of HIV services provided. Job descriptions can be revised at either the province or central level. SMART TA may need to assist with the job description revisions as part of the transition strategy. The Methadone Maintenance Therapy (MMT) Program All stakeholders interviewed who are familiar with SMART TA’s methadone maintenance therapy (MMT) program commented positively on the program’s scope, implementation and results. In Vietnam, where PWID are the most critical at-risk population for spreading HIV, MMT has served well as a point of entry to bring addicts into the health care system and link them to HIV testing, counseling and treatment. MMT is the front line of prevention. According to file documentation and reports, SMART TA has provided support, including technical assistance and direct service delivery, for MMT clinics in nine provinces. SMART TA has provided assistance to provinces and at the central level to set up the program, both functionally and financially. Figure 13 shows the enormous growth in demand for MMT services, from under 4,000 enrolled in the first year to 14,678 by the end of Year 3. Enrollment missed the target of 15, 454 by 776 cases because of delays in opening new sites with government funds. From April 15–June 16, 2014, a systematic review was conducted by 21 MMT clinic case managers of 5,795 clients, 1,155 of whom were HIV-positive. Case managers found and Figure 12: Number of MMT Patients in Vietnam Result = 14,678 Target = 15,454 Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 18 74.65% 74.97% 57.78% 52.60% 0.00% 20.00% 40.00% 60.00% 80.00% 100.00% Refer to HIV care HIV re-testing Integrated clinics Stand-alone clinics reached 161 (12.8 percent) of HIV-positive patients who had either not enrolled for care or had dropped out. This lost-to-follow-up (LTFU) cohort was offered individual counseling focused on the benefits of early treatment and retention in HIV care. This messaging stressed the need for periodic CD4 testing and early initiation of ARV therapy when symptoms or laboratory data indicate a need for medication. With intensive individual effort by case managers, 105 MMT HIV-positive patients, out of 161 LTFU cases, were enrolled or re-engaged in HIV care. This means that 65.2 percent of LTFU HIV-positive MMT patients were successfully linked to HIV care and treatment services. Among them, 16.8 percent (27 people) who had enrolled but never entered an HIV outpatient clinic or stopped using ARV returned for full ART care; the remaining 48.4 percent (78 people) had been lost during the pre-ART period and returned to outpatient clinics for CD4 cell count testing and further intervention. In addition, a data analysis compared the level of successful referral for MMT between integrated and stand￾alone clinics. An integrated clinic is a SMART TA facility that provides all services: methadone treatment, HIV testing, counseling and HIV care and treatment. A stand-alone clinic provides only methadone treatment. In terms of linking MMT patients to HIV care and retesting, Figure 15 shows a significantly higher success rate in referring MMT patients to HIV care and treatment: 74.65 percent (53 of 71 patients) of HIV￾positive MMT patients were linked to HIV care in integrated clinics, and only 57.78 percent (52 of 90 patients) in stand-alone facilities. Results improved significantly in the rate of HIV-negative MMT patients who returned for HIV testing every six months; 74.97 percent (1,219 patients) attend integrated clinics compared to 52.60 percent (1,031 patients) in stand-alone clinics (p value<0.001). To further assess the extent to which SMART TA may have helped reduce morbidity and mortality and improved the quality of life of at-risk people, this evaluation conducted interviews with enrolled patients. An exit survey (see Annex G) took place at several sites where MMT recipients go for services. The survey substantiates the perceptions expressed to the evaluators by service providers: MMT recipients experience substantial improvement in their health and quality of life by participating in the program. Figure 14: MMT Services Return of PWID to HIV Care Figure 15: Comparison HIV Linkage between MMT Integrated with HIV Care and Treatment and MMT Stand Alone Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 19 The survey was designed to be gender-sensitive, but no gender-related issues emerged. Female respondents consistently stated that they felt no difference in the attitude of clinic staff or quality of the care received, suggesting that SMART TA and partners in the provinces had done a good job of minimizing any gender bias in the program. A network of technical assistance providers, which included SMART TA staff and its partners, is being deployed to ensure the quality of MMT service. However, the long-term sustainability of the network is uncertain because it is not a part of the current health technical assistance system. There is some recent indication that the MMT mentorship program may be supported with provincial funding in HCMC and Hai Phong. The introduction of patient co-payments for MMT service has generated modest revenue without leading to a dropout problem; HCMC PAC noted no dropouts due to the cost. But some concern emerged about co-pay costs increasing from the initial low level ($.30/day), which may be a financial challenge for poor patients. HCMC PAC reported having a fee-waiver policy for poor patients. However, under current regulations, poverty certification entails a lengthy process and most poor patients are unable to complete it. HCMC does offer a waiver for patients who volunteer to serve in peer outreach activities. HIV Care and Treatment Services As noted, evidence shows that SMART TA care and treatment programs are effectively reducing AIDS-related mortality. Information collected through exit interviews at SMART TA DSD sites consistently suggests improvement in patients’ quality of life once they are on treatment. In general, interviewed patients reported great improvement in their quality of life, especially for both their physical and psychological health. (See text box for example comments.) SMART TA has supported delivery of effective, quality care and treatment services in 40 HIV or TB/HIV outpatient clinics. The quality of treatment services translates into lower mortality across all SMART TA DSD sites. The lower mortality rate is also due to efforts to address earlier ARV initiation. Currently, 86 percent of patients initiated ART at SMART TA OPCs within 30 days of being tested. SMART TA has made several efforts to address limited linkages between TB and HIV programs. SMART TA’s OPCs now perform TB screening for ART patients and help TB-diagnosed patients get treatment by linking them to locally provided TB services. Key achievements in TB assistance are shown in Table 3. In addition, SMART TA has implemented a TB–HIV integration and coordination program pilot in Thai Binh and Ninh Binh. TB is Indicators FY 2013-2014 Number of patients receiving ART and pre-ART 19,846 Percentage of clients screened for TB (%) 87.3% Number of patients who started new TB treatment 477 Number of clients currently on TB treatment 364 Number of patients who started new Isoniazid preventive therapy (IPT) 5,820 Number of clients currently on IPT 4,076 “Now I feel as healthy as I did before I got sick. I remember how bad I was before I got treatment. At that time, I was almost dead. Being on treatment not only makes me function well, but also makes me feel like a normal person in society, it rids me of my self-stigma and I am much more confident.” — Male patient, HCMC “I learned about my HIV status a month ago. I was in deep shock. I expected the process of treatment would be complicated. But everything happened in opposite ways. Clinic staff and doctors are friendly. Paperwork was reasonable and treatment started after only four weeks. This is the first week I am on treatment, but I feel very comfortable and not much worried about the future. I think I am getting better not only because of medication, but because of clinic staff’s attitude and counseling skills.” — Female patient, HCMC Table 3: TB Support at SMART TA OPCs Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 20 a major cause of AIDS-related death. A system of monitoring and quality improvement (QI) that is integrated with the National HIVQUAL program manages the quality of treatment services, using HIVQUAL indicators to identify the need for and provision of local QI coaching. Discussions with PACs and health center staff suggest that SMART technical C&T monitoring is, in general, an effective means to evaluate preparedness and identify TA needs for successful transition. SMART TA has seen some success its outreach and testing activities in reducing the percentage of patients who enroll for treatment with a low CD4+ count (<100 cells), which makes it more difficult for the treatment to be effective, by trying to identify and bring them into treatment earlier, when their counts are higher. Still, about one-third of patients started treatment with a low CD4+. One issue may be that the MOH ART treatment protocol provides treatment only for patients whose CD4+ counts are 350 cells or less, so some people with AIDS who have a CD4+ count higher than 350 may not enter the treatment program. Stigma and discrimination is another barrier to bringing patients into treatment earlier. During client exit interviews, many patients reported that they were jobless because their HIV status was disclosed once they entered treatment. Unlike in the MMT program, the evaluation team did not see evidence of an operational network of technical assistance providers for ARV treatment at all sites. In HCMC, an informal network established by the PAC exists, but its function and effectiveness were not clearly explained. Contributions to the Development and Use of Strategic Information Systems SMART TA has contributed to the development and use of strategic information systems at the national and site levels. National Strategic Information The evaluators reviewed SMART TA records and reports of its activities and achievements, and summarized SMART TA’s contributions to the development and use of strategic information systems:  Quality Improvement for National HIV/AIDS Surveillance System. With other partners, SMART TA helped strengthen epidemic and outcome/impact monitoring by supporting VAAC and NIHE in a training-of-trainers for staff from 31 provinces on the new HIV Sentinel Surveillance system. Also, SMART TA worked with PEPFAR partners to help NIHE develop the protocol and implementation of the IBBS Round III. Other strategic information activities in which SMART TA has been engaged include the National/Provincial HIV/AIDS Estimates and Projections, MARP size estimates and a number of operational research initiatives related to Cascade drop-offs and coordinated with VAAC. VAAC commented on the importance of SMART TA’s assistance in improving HIV/AIDS surveillance to better design and manage service delivery.  Improved Management Data Quality and Use. SMART TA worked with a variety of partners to improve programmatic data quality and use at the site, provincial and national levels. Among them: collaborating in a CDC–VAAC project with the Partnership for Health Advancement in Vietnam (HAIVN), Hanoi School of Public Health, Global Fund and WHO supported VAAC to generate 10 core-QI indicators for measuring care and treatment performance as part of the National Care and Treatment Quality Improvement Program (HIVQUAL); training workshops and coaching/mentoring at various PACs; development and piloting the national DQA tool; implementing an assessment of the national HIV reporting system (D28); and developing the VNIS360 online database at the site level to help the manage program-wide key indicators and facilitate analysis of triangulated data for QI purposes. The VNIS360 online database system is an internal project management system that would not be used by the GVN post-SMART TA. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 21 Facility-Based Strategic Information System In addition to the introduction of the Cascade framework, SMART TA has also established and implemented a comprehensive M&E system, required by PEPFAR, that includes a clear framework with specific indicator definitions, and data flow and data management procedures (see Figure 14). Guidelines for M&E specialists and the training package are well designed and provide essential knowledge and skills for M&E staff in various program areas (prevention outreach, MMT, HTC and care and treatment). Like other program areas of “pull” technical assistance, SMART TA uses a system that enables staff to identify and specify its own needs for assistance. Supplementing traditional data collection tools, SMART TA has introduced the use of new electronic technologies to help reduce staff time spent on data collection, and to enhance data integrity and validity. The “eLog,” “MMT Data Management and Reporting Package” and “mCare” are examples of SMART TA data management innovations introduced on a pilot basis in efforts to innovatively use technology in M&E activities. SMART TA also introduced staff at some sites to data feedback mechanisms by communicating back to a site once data were received and analyzed. Staff reported that they found the feedback helpful and timely, and that it provided SMART TA with an opening to strengthen and widely promote regular data quality assessments (DQAs). DQA processes and tools have been standardized and shared across partners. Site staff who were interviewed were all aware of DQA benefits. An Evaluation ‘Spot Check’ on Data Quality SMART TA routinely conducts its own data quality reviews as part of its performance assessment protocols, and has trained a large network of central level provincial level data managers in DQA. However, the evaluators additionally conducted a spot check on the quality of data in the information systems in use at SMART TA-supported sites to verify that data being reported by SMART TA into PEPFAR and GVN reporting systems is valid, and that the data accurately reflect user-generated data at those sites. A cursory review of data quality was conducted by comparing SMART TA headquarters files to raw data in record books at the site level. These spot checks, conducted at District 3 and District 8 facilities in Ho Chi Minh City, found the following:  Number of people who initiated ARV in the last quarter of FY 2014: In District 3, a small discrepancy was found. The actual count from the treatment registration logbook was 44, but the number reported to SMART TA was 46. No discrepancy was observed in District 8 records.  Number of patients who were on ARV treatment after 12 months: In District 3, the actual count and estimate was 81 percent; the reported figure was 88.7 percent. No discrepancy was observed in District 8 records.  Number of patients who are currently on MMT treatment: Variations were found in District 8 between the evaluators’ manual count and reported numbers. In July 2014, 294 were recorded, but 285 were reported; in August 2014, 305 were recorded, but 296 were reported; in September 2014, 305 were recorded, but 296 were reported. There was a change in PEPFAR reporting requirements from 3 to 6 months on MMT which could explain the data variance issues. Due to time constraints, the causes for the minor discrepancies observed were not investigated further. Results from the spot check suggest that the system at those sites produces reasonably accurate data. Figure 13: SMART TA Reporting Process Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 22 Challenges to SMART TA Assistance to Strategic Information Systems Although obviously useful at the provincial, district and site levels for strategic planning and management, the evaluation identified several concerns about strategic information use at SMART TA sites, including: • An advanced strategic information system, such as one developed by SMART TA, helps provide useful data and information for program design and management. However, the complexity of these systems requires significant investments in training and systems maintenance that may not be sustainable. • Few efforts have transpired to integrate the SMART TA project M&E system with the national AIDS reporting system. Local partners interviewed commented that the workload for data collection and reporting is excessive and too many reporting indicators go unused by GVN and would not be sustained when the donors leave. • Except for the many operational research initiatives whose findings have been widely circulated and much appreciated, no systematic documentation exists for innovative approaches and pilot programs as “strategic information” to use for planning and advocacy or transition. Evaluators found that it would be challenging to sustain many SI activities after SMART TA, if no solid strategic information capacities are improved for provincial partners. For example, while some positive feedback came in on performance-based outreach incentives, partners need more evidence on its effectiveness if such a model is to be integrated into the current MOH system. Without a stronger report on the effectiveness and challenges of scaling up a performance-based incentive mechanism, the GVN would not continue it post SMART TA. • Inconsistencies exist between guidelines and practices in data generation and reporting. SMART TA has supported partners in using an electronic database to manage routine data. Staff also use that database to generate periodic reports, which they could do by extracting numbers from the paper￾based logbook (as indicated in the M&E guidelines). This causes delays in updating paper-based logbooks and results in a disparity between reported numbers and the logbooks. • DSD staff capacity to conduct DQAs by themselves is currently limited. The evaluators asked selected staff to interpret data presented in the program data dashboard that SMART TA uses, but their ability to do so was weak. Graphics promoting data management were found at all sites. However, an M&E guideline was not available and ready for use at any site visited. Staff turnover is an issue, as new staff is not consistently trained for M&E as part of their orientation. • With its strong technical capacity, SMART TA could have taken a stronger leadership role in providing its partners with technical assistance in strategic information. For example: In 2013, SMART TA staff did not provide as much technical assistance to IBBS Round III as expected, in addition to financial support the activity provided for that study. Conclusions Overall Performance: The MOH’s HIV Sentinel Surveillance data show a downward trend in HIV prevalence in all provinces where SMART TA provides assistance. This is particularly true for HIV prevalence among PWIDs in HCMC, Hai Phong and Quang Ninh, where the PWID epidemic is most serious and highly prevalent. HIV prevalence in these areas decreased significantly, from 50 percent or higher in 2005 to 30–40 percent in 2013. This indicates that rates of acquisition and transmission of HIV are declining. Data from HIV/AIDS estimates reaffirmed this. Although no direct attribution can be made between declining HIV prevalence and SMART TA’s activities, epidemiological and key informant interview evidence shows that innovations such as targeted outreach techniques have been effective in identifying, testing and treating individuals in the target population. In terms of AIDS mortality, records showed declining death rates among 18,000 AIDS patients at SMART TA-supported OPCs, from 2.8 percent to 2.1 percent between 2012 and 2014. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 23 Methadone Maintenance Therapy (MMT): SMART TA has been instrumental in launching and institutionalizing an MMT program that has proven highly effective in bringing addicts into the health care system and linking patients to HIV testing, counseling and treatment. Demand for MMT services has grown enormously, from under 4,000 enrolled the first year to more than 15,000 nationally by the end of the program’s third year. Almost all MMT enrollees in SMART TA-supported sites have been tested for HIV and almost all of those who tested positive have started ARV treatment or been retained in treatment. SMART TA program data and information collected from key informant and stakeholder interviews during the evaluation indicates a positive association between the MMT program and an improved quality of life, and reduced enrollee morbidity and mortality. Enhanced Outreach Approach (EOA): The EOA is a practical approach to increase the effectiveness of outreach programs; however, it has been challenged by the fact that the built-in performance-based incentive mechanism may not be sustainable under the current government compensation systems. Although it is still early to make a conclusion, data suggests that the approach has led to an increase in the number of high-risk people who were willing to test for HIV and the number of newly identified HIV-positive people who were referred to treatment services. Nevertheless, the reward system, which includes performance-based incentives, may be difficult to sustain, according to several PAC officials who said it would be difficult for the MOH to administer a commission-based rewards system. While preliminarily efforts show a positive and promising outcome, further efforts are needed to ensure the sustainability of the Reach, Test, Treat and Retain Model. SMART TA has worked with the PAC in Nghe An Province to develop this model for the remote and mountainous regions, where HIV services are limited and existing networks of hamlet health workers are mobilized to conduct outreach. Hamlet workers arrange HIV testing and counseling and provide follow-up treatment referrals for those who test positive. This program has shown positive results in terms of increasing successful outreach to and identification of patients and increasing enrollment in treatment programs. Improving Care & Treatment: The SMART TA care and treatment program has contributed to a reduction in AIDS-related mortality and improvement of the quality of life of people with HIV and AIDS. However, an issue exists when PLHIV with CD4+ greater than 350 and presenting healthy are prohibited from starting treatment due to MOH protocols that require a CD4+ below 350. Many will drop out of the cascade as treatment is not forthcoming. Denying AIDS patients early treatment that may be most effective would not appear to be an effective approach to reducing mortality. Individually, patients expressed satisfaction with OPC services and noted a great change in their quality of life, which they attributed to the treatment service. While Strategic Information Systems have served SMART TA program management well, they are not consistent with national GVN M&E frameworks. As noted earlier, SMART TA has expended great effort and contributed substantially to HIV/AIDS SMART TA-related strategic information systems development, training and technical assistance. This work is ongoing and minor adjustments are required in systems design, functionality and capacity to manage data, perform M&E, ensure data quality and apply strategic information. SMART TA has also contributed to the development of a national strategic information system through its day-to-day technical support for VAAC and other government agencies. However, efforts to operate a SMART TA-related system and strengthen the current government system are not coordinated well. As a consequence, at the provincial and site levels, the strategic information system that serves the SMART TA program is not fully integrated into the local government system. A number of SMART TA-funded activities, such as the system to monitor site performance, may not be sustained after the program ends. Introduction of the Cascade Methodology Framework: A SMART TA initiative that has made a difference in the effectiveness of programs and services at the site level is the Cascade Methodology Framework, which has been a useful approach to monitor program performance. The Cascade tool measures the number of individuals who move — or don’t move — through the CoPC. Cascade has helped managers at the site level use strategic data and information to more effectively plan and manage their work. Since it requires minimum resource investment, this application will be likely be continued by local partners without SMART TA and other external support. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 24 Recommendations 1. GVN Support. SMART TA should work more closely with the VAAC and other relevant GVN institutions such as VUSTA, to address issues related to the complexity of the program, the technical assistance needed to sustain quality care and GVN’s capacity to replicate and scale up key performance innovations, while focusing on developing sustainable Vietnam solutions to the nation’s HIV epidemic. These all require closer communication with and stronger advocacy to Vietnam’s government at all levels. 2. AIDS Treatment Protocols. SMART TA should continue to work with VAAC and MOH to advocate for the adoption of a higher eligibility threshold treatment protocol of CD4+ from 350 to 500, as recommended by WHO and other in-country partners. This would help reduce the loss of patients who are enrolled in OPCs but not eligible for ARV treatment, and thus do not get the early treatment that could be most effective. MOH has recently circulated a letter suggesting that this standard be used in hospitals and clinics, but the protocol is not yet an official policy of MOH. 3. Annual Viral Load Testing. The lack of critical testing of viral loads for all AIDS patients has drawn concerns, per VAAC. OPCs can test only “for patients who meet clinical or immunological criteria for treatment failure.” The report notes: “A single viral load measurement at 12 months is the most accurate measurement of ART adherence and can be used to target additional adherence interventions to those patients with the greatest risk for treatment failure.”42 4. Integration into the MOH Curative Care Division. SMART TA should help to prepare care and treatment DSDs to be integrated into the curative division of the health sector wherever appropriate. ART treatment in particular would benefit from integration into MOH’s infectious disease treatment centers, as staff could develop a more comprehensive understanding of HIV and work more holistically to provide treatment. 5. Enhanced Outreach Approach. While early results from the program look promising, more comprehensive evaluation with greater involvement of government stakeholders is needed to document costs and benefits of the EOA program. 6. Reach, Test, Treat and Retain. The decentralized “Reach, Test, Treat and Retain” initiative for mountainous and rural areas is a critical SMART TA activity. Where appropriate, it should be expanded. This may require substantial modification in job descriptions of hamlet health workers, as well as related legal documents (MOH guidelines or directives), to fully integrate the model into the current health system. 7. SOPs for Lost to Follow-Up (LTFU). SMART TA should give special attention to developing and disseminating explicit standard operating procedures (SOPs) to prevent loss to follow-up and re￾engage patients into care and treatment. This initiative should involve VAAC and provincial program managers, site staff and community-based support groups. 8. Strategic Information System. SMART TA should continue efforts to strengthen the national HIV/AIDS strategic information system. In particular, SMART TA should work with the VAAC to streamline the project-centric M&E system for transitioned DSD sites to be more consistent with and supportive of GVN data requirements and capacities. 42 “Results from the Vietnam ART Cascade Completion Study,” Vietnam Administration of HIV/AIDS Control, March 2015 Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 25 VIII. TRANSITION Introduction “Transition is the process of fostering financial, technical and programmatic responsibility to the Government of Vietnam and designated civil society organizations for HIV service delivery that achieves quality technical performance standards.”43 This simple SMART TA definition of transition becomes a complex policy, operational, resource and tactical set of approaches, concerns, issues and responses, with myriad stakeholders and interlocking responsibilities, a small subset of which relates to USAID’s SMART TA program. In the GVN’s view, a sustainable HIV response requires a focus on practical transition planning; strategic integration of HIV/AIDS with other health interventions and services; alternative sustainable means of financing HIV/AIDS services; and strengthened multi-sectoral coordination. Consistent with the planned phase-down and eventual withdrawal of PEPFAR support to Vietnam, SMART TA’s transition approach has been to facilitate transition of financial and technical capacity of CoPC programs and services away from project support and to country ownership. SMART TA has developed and is implementing a “Transition Strategy” that outlines a rationale, collaboration protocols, detailed outcome metrics and benchmarks and techniques and processes for transitioning SMART TA’s direct service delivery (DSD) sites. This evaluation assesses progress to date and identifies challenges and opportunities in SMART TA’s strategic approach to transition SMART TA HIV/AIDS services to the GVN by addressing the key questions outlined in the approved inception report:  To what extent are the project’s activities, techniques and processes employed by SMART TA contributing to the intended results of transitioning financial, human resource, administration and technical ownership of HIV/AIDS services to the GVN and other stakeholders (including pre￾transition preparation, transition, and post-transition support)?  How are the readiness, acceptability and receptiveness of GVN and other stakeholders to the services transitioned by SMART TA? Underlying these questions is critical concern about the sustainability of SMART TA’s achievements beyond the life of project. Summary conclusions and recommendations for SMART TA, USAID and the GVN going forward to ensure an effective transition of SMART TA’s HIV services are presented at the end of this section. SMART TA’s Transition Activities and Intended Results SMART TA’s transition approach is focused on preparing individual DSD sites for transition and is designed to achieve: 1) financial cost reductions via programmatic, administrative and human resources efficiencies to align recurring operating costs (ROCs) with current and expected GVN budgets, and 2) the transfer of technical knowledge to help ensure that the quality of care is maintained during and after transition. 43 SMART TA Mid-Term Evaluation Briefing Package, page 7. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 26 Financial Transition Background SMART TA’s managed financial transition metric is derived from the cooperative agreement modification requiring a 40 percent transition of interventions, partners and sites with “resources coming from the Government, other donor sources and efficiency gains.” 44 SMART TA has construed the 40 percent transition requirement to mean a reduction in the financial allocations provided by SMART TA in the provincial sub-agreements that fund SMART TA DSDs. More specifically, the required reduction focuses on DSD ROCs — the routine direct service delivery expenses that fund CoPC services. ROCs covers staff compensation (which account for 90 percent of total ROCs) and office expenses (rent, utilities, and supplies). The advantage of this metric, a reduction in ROCs, is that the data are measurable, verifiable, precise and timely. ROCs are negotiated each year with provincial officials and are formally approved via the sub￾agreements; consequently, they represent explicit GVN acceptance of SMART TA’s annual transition plans and targets. Whether this metric is sufficient in directly and robustly measuring the intended result (GVN ownership of the HIV response) can be addressed only in the context of the nature and level of development assistance expected by the GVN to provide effective HIV service delivery. Reducing costs may not be the best measure of transitioning financial ownership of HIV service delivery to the GVN. However, discussions with central and provincial-level GVN officials made it clear that the GVN will not continue all existing services at all sites, or provide them the same way that SMART TA does. GVN will not assume dollar-for-dollar financial ownership of SMART TA services. In particular, the GVN is not committed to continuing more costly dedicated standalone SMART TA HIV DSD sites. Rather, it will focus on integrating HIV services into the existing health care system. GVN intends to achieve a Vietnam solution to the nation’s HIV/AIDS epidemic. Officials expect close collaboration with, and technical assistance from, SMART TA to prepare SMART T-funded DSD sites for GVN ownership. Reducing costs is the first step. For financial transition to be successful, the costs of delivering the services must be sustainable, i.e., affordable to the GVN; thus the SMART TA focus on reducing costs. SMART TA now does (in collaboration with the GVN) what the GVN ultimately must do as it assumes financial responsibility of the HIV response with its own limited resources: phase out inefficient DSD sites, transfer DSD sites to other donors who will continue to fund activities beyond the life of SMART TA, reduce management costs and rationalize service delivery. Within this context, SMART TA is implementing a number of financial transition activities, including: 1. Using strategic information to identify priorities, track progress and monitor results. The “SMART Technical Monitoring” tool and metrics are used to identify DSD sites that are suitable for a specific financial transition outcome, to target technical assistance resources to enhance “readiness” for transition where appropriate and to provide a means to track progress and monitor site performance post-transition. Sites are evaluated on their readiness for transition based on an assessment of different aspects of site performance and costs of the various CoPC interventions. 44 SMART TA Cooperative Agreement Modification #2 ART service delivery Figure 14: C&T TA Needs Assessment Criteria Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 27 2. Phasing out sites. As part of the GVN’s long-term HIV response planning, SMART TA works with the PACs to triage dedicated HIV sites and services based on service overlap, low HIV burden, or poor performance. 3. Transitioning DSDs to other donors. DSD sites are transitioned to other donor projects — the Global Fund or CDC — where they have a longer timeframe for transition or have a stronger relationship with the local government. This enables transition over a longer timeframe than SMART TA’s 2016 end date (Global Fund assistance is ongoing through 2017; CDC assistance is scheduled to end in 2018). This does not directly result in the transitioning of the site to GVN ownership according to SMART TA’s timeline, but it buys time and prevents a loss of service at sites for which the GVN may not be ready to accept responsibility. 4. Reducing ROCs unrelated to direct HIV service delivery In the scale-up of services to 115 DSD sites (Year 2), SMART TA initially provided for a full range of administrative and facility-related needs. To better align facility costs with expected GVN resources, SMART TA has begun to identify and reduce ROCs that are unrelated to direct service delivery or will not adversely affect service delivery quality. This includes certain management expenses, support for travel, overtime and positions that do not directly relate to HIV services and should be a routine GVN expenditure (e.g., janitorial staff). 5. Rationalizing service delivery. SMART TA’s efforts to improve affordability are focused on “rationalizing” services to achieve the highest impact for the least cost through the following interventions: • Internal Integration. SMART TA has emphasized the integration of three services (MMT, testing, and care and treatment) in one location, resulting in administrative and human resources savings and, through multi-tasking and the ability to provide immediate referrals, more effective holistic service. • Decentralization. In Dien Bien, SMART TA has worked with the PAC to introduce decentralized services reaching down to the communes and hamlets. Such services make it easier for patients to get direct care, and present cost savings compared to providing the service in a provincial hospital or dedicated HIV response center. • Enhanced Outreach. When introduced, the peer-driven incentive approach in some cases tripled the proportion of key populations who used HIV testing and counseling services. HCMC District 8 reports that its performance-based outreach has been successful in that most HIV￾positive PWIDs have now been identified, resulting in only 10 new cases a month — compared to 40 a month before the initiative launched. • Use of Mobile Technologies. Tracking patients through the Cascade via mobile technologies is more accurate and less costly. • Introduction of Co-Pay for MMT. SMART TA has worked with Hai Phong, Lao Cai and HCMC to introduce a co-pay for MMT patients. It is expected that the MMT co-pay will be implemented throughout Vietnam. • MMT as Prevention. The AIDS epidemic in Vietnam is largely driven by persons who inject drugs and share needles or have unprotected sex with multiple partners, or both. As an interface with the drug culture, MMT is designed to bring addicts into the health care system and link them to HIV testing, counseling and treatment. MMT is the front line of prevention. MMT centers are more easily transitioned to full GVN ownership than ART centers are, in part because MMT costs are significantly lower than ART over the care timeframe, and pilot co-pay Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 28 schemes are generating revenue that can help reduce the costs to the GVN. Because GVN is concerned about PWID and associated criminal activities of users, methadone treatment has become a high-priority crime avoidance activity. Treatment 2.0. Treatment 2.0 is a cost-effective protocol for stable patients, especially in remote locations, whereby the patient comes to the commune health stations every month for medication pick-up and comes to district out-patient clinics every six months instead of monthly. This protocol helps keep patients on treatment, as it saves time and transport costs. Though not currently part of the Treatment 2.0 protocol, viral load testing every six months should be considered to ensure drug management is adjusted if needed. Treatment 2.0 makes transition more affordable. Initially introduced in Dien Bien, Treatment 2.0 is being incorporated in pilot decentralization initiatives in seven northwestern provinces. 6. Transitioning DSDs to GVN Ownership. SMART TA works closely with the GVN to transition DSD sites so GVN assumes full financial, administration and technical responsibilities. All of the financial transition interventions noted above are implemented in close collaboration with the GVN, particularly with provincial officials. Findings on Financial Transition Findings on financial transition incorporate administrative and human resources, and focus on reducing costs to make service delivery more sustainable. Reducing Costs to Meet the Cooperative Agreement Transition Requirement SMART TA is making substantial progress in meeting the requirement of reducing supported ROCs by 40 percent by the end of the project. This is evidenced in part by approved reductions in the sub-agreements with the provinces and confirmed in field visits to the five provinces, which documented specific ongoing and committed transitioning of staff and facility costs. Table 4: Reduction to Date in Annualized ROCs by Province Province FY12 Annualized ROC January 1 2015 Annualized ROC Staffing Office Expenses Total Staffing Office Expenses Total % ROC reduced An Giang 83,843 8,907 92,750 65,492 4,889 70,381 24% Bac Giang 13,400 403 13,803 13,073 621 13,694 1% Can Tho 96,443 8,477 104,920 43,482 3,092 46,574 56% Dien Bien 147,269 16,461 163,730 123,474 11,533 135,007 18% Ha Noi 215,627 27,747 243,374 154,016 12,325 166,341 32% Hai Phong 183,657 22,315 205,972 61,007 6,861 67,868 67% HCMC 337,068 33,365 370,433 240,247 13,487 253,734 32% Lao Cai 43,150 5,475 48,625 33,920 5,195 39,115 20% Nghe An 58,146 5,237 63,383 38,430 2,626 41,056 35% Ninh Binh - - 7,200 1,468 8,668 Quang Ninh 149,493 11,486 160,979 98,428 7,833 106,261 34% Thai Binh 17,489 1,176 18,665 23,137 2,174 25,311 -36% Da Nang 10,498 1,079 11,577 - Khanh Hoa 39,137 5,460 44,597 - TOTAL 1,395,221 147,588 1,542,809 901,906 72,104 974,010 37% Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 29 Table 4 shows that SMART TA has already reduced ROCs by 37 percent. The reduction has been achieved through unit cost efficiencies (see Table 5), along with phasing out and transitioning sites to other donors, as permitted in the cooperative agreement modification. The largest percentage reductions were in Hai Phong and Can Tho, where there is a commitment to the full ownership of the HIV response. The variances by province relate to specific situations. Thai Binh support has increased, as the new sub￾agreement includes support for a pilot TB/HIV site. Both Hai Phong and Can Tho have had the most successful reduction in ROCs to date, but these are well-off provinces with strong PAC leadership. Both provinces were initially identified as rapid transition provinces by PEPFAR based on their criteria. Both Can Tho and Hai Phong proactively took charge on the reduction of ROCs by moving staff/tasks into GVN structures/systems. Hai Phong has been the first province to introduce the MMT co-pay model as well. Da Nang and Khanh Hoa were fully transitioned (site phase-out) in Year 2 of SMART TA; thus no ROCs appear for 2015. Other site transitions, committed by the end of Year 4 but yet to be implemented (including all MMTs), suggest that SMART TA will significantly surpass the 40 percent requirement for the reduction of ROCs. Between Year 1 and Year 2, the rationalization of HIV services resulted in a 15 percent median reduction of unit expenditures for care and treatment services. Table 5 shows that while unit cost expenditures have been reduced for both integrated and stand-alone sites, SMART TA stand-alone sites are, on average, 40 percent more expensive to operate. Table 5: Reduction in Unit Expenditure for Care & Treatment Services Progress in Administrative Transitioning of Direct Service Delivery Sites Table 6 is a schedule of recent and projected administrative transitioning of individual DSD sites to GVN or other donors, according to the four outcomes defined below: 1. Not Transitioned. DSDs continue to receive direct financial investment for ROCs and technical assistance from SMART to the end of the project. 2. Phased Out. All financial, technical and programmatic assistance to a site, partner or intervention is eliminated and the services are ended. Phase-out is selected as a transition option when service overlap, poor performance or low HIV burden exists. SMART TA and the GVN jointly decide whether to phase out a site, partner or intervention. 3. Partially Transitioned/Graduation. Sites are administered by the GVN and do not receive direct financial investment in the form of ROCs, but do receive technical assistance (TA-SDI) support from SMART TA or designated local TA providers. 4. Fully Transitioned/Sustainable. Sites are administered by the GVN and are classified as “sustainable” when ROCs are eliminated and SMART TA monitoring shows that the site continues to achieve quality technical performance standards post-transition. In sustainable sites, project-based financial and technical assistance have both been transferred to local ownership. These sites are no longer classified as TA-SDI sites. Unit Expenditure Year 1 Unit Expenditure Year 2 % Change Integrated Sites $54 $46 -15% Standalone Sites $77 $65 -16% Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 30 Table 6: Summary of Direct Service Delivery (DSD) Site Administrative Transitioning Progress Transition Status Care and Treatment Prevention MMT HTC TOTAL Beginning October 2013 38 22 20 35 115 Year 3 Activity DSD/Not Transitioned 38 22 12 35 107 Phased Out 0 0 0 0 0 Partially Transitioned 0 0 8 0 8 Fully Transitioned/Sustainable 0 0 0 0 0 Sites as of October 2014 38 22 12 35 107 Year 4 Activity* DSD/Not Transitioned 28 15 6 21 70 Phased Out 1 5 0 10 16 Partially Transitioned 9 2 6 4 21 Fully Transitioned/Sustainable 0 0 0 0 0 Sites as of October 2015 28 15 6 21 70 Year 5 Activity ** DSD/Not Transitioned 21 8 0 10 39 Phased Out 0 4 0 6 10 Partially Transitioned 7 3 6 5 21 Fully Transitioned/Sustainable*** 4 0 6 2 12 Non-Transitioned Sites 21 8 0 10 39 * Projections for Year 4 are based on agreements with the respective PACs. ** Projections for Year 5 depend on current assessments of GVN interests and may change. *** Counted from Partially Transitioned sites in Year 4. The projections are a work in progress and represent current thinking related to ongoing SMART TA deliberations with the VAAC and PACs. They were confirmed in part in the field visits to the five provinces and in discussions with VAAC. In terms of the number of sites transitioned (including phase-outs), SMART TA will near the 40 percent requirement by the end of Year 4, when 45 of 115 sites (39 percent) will have been phased out or transitioned. According to projections, SMART TA will surpass the 40 percent reduction goal by September 2016, phasing out or transitioning 76 of 115 sites, or 66 percent. The breakout by type of service sites to be phased out or transitioned (full elimination of ROCs) is as follows:  45 percent care and treatment sites transitioned,  64 percent of prevention sites transitioned,  100 percent of MMT sites transitioned, and  71 percent of HTC sites transitioned. As discussed in the technical transition section, SMART TA will continue to monitor service quality at sites that are partially or fully transitioned every six months through the end of the activity. Table 7 identifies the transition status by type of service site at the end of SMART TA. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 31 Table 7: Transition Status at the End of SMART TA by Service Type Transition Status Care and Treatment Prevention MMT HTC TOTAL DSD/Not Transitioned 21 8 0 10 39 Phased Out 1 9 0 16 26 Partially Transitioned 12 5 14 7 38 Fully Transitioned/Sustainable 4 0 6 2 12 TOTAL 38 22 20 35 115* Note: the total of 115 sites represents the peak number which occurred at the end of Year 2. A total of 26 direct service sites (of 115 DSD sites) are planned for phase-out by the end of SMART TA. HIV testing and outreach services in overlapping or low-prevalence locations are expected to be consolidated into general health care facilities without any significant inconvenience or reduction in service quality. If a care and support site will be phased out in an overlapping district, services will not be affected. Discussions with central and provincial officials support a finding that the GVN is committed to taking ownership of all MMT sites by the end of SMART TA; these services are relatively low-cost and are seen as effective in enticing the primary at-risk cohort of PWID to come in for HIV testing, referral, counseling and treatment, thus achieving containment. Field visits to Hai Phong and Hanoi found that MMT negotiated transition plans are robust. HCMC District 8 reports that methadone co-pay instituted in collaboration with SMART TA has been successful; not a single patient has discontinued treatment, and it has generated modest revenue (the cost to the patient is $.30 per day). Officials attribute the co-pay success rate in HCMC to SMART TA efforts to prepare patients for the co-pay requirement, and continued counseling/education on the value and benefits of MMT. In Hai Phong, the rate of patients dropping out of MMT after co-pay was instituted is the same as before co-pay, suggesting that people are not dropping treatment because of the co-pay. Use of a co-pay reduces the financial burden on the GVN to deliver MMT services and thus supports the GVN’s commitment to take ownership of all MMT sites by the end of 2015. Discussions with SMART TA, the VAAC and the PACs visited identified various reasons why 39 DSDs will not be transitioned by the end of SMART TA (as currently projected). Prevention services do not have a natural home in the MOH health care system. In HCMC, these services are reportedly provided by community-based organizations whose members receive a modest stipend from the PAC and by volunteers. Testing services are provided by MOH in its provincial hospitals or district health centers or district hospitals. HIV advocates are concerned that transitioning testing services to such facilities will be less effective due to issues of confidentiality and the stigma associated with HIV. Also, individuals who get tested at a hospital may not be quickly referred for HIV treatment services. Discussions to resolve these issues are ongoing. Issues with transitioning the remaining 21 care and treatment sites are the level of local priority and cost of the services provided. Poorer provinces and districts may simply not have the resources to assume the cost or may not give dedicated HIV response facilities a high priority relative to other health care needs. A major issue is the availability of HR openings for HIV personnel. Due to competition for the limited number of openings across all service agencies, the province or district must rank HIV personnel against all other service needs to be able to transition staff to the DOH payroll. Transitioning Human Resources As noted, ROCs largely reflect staffing costs. In Year 3, the number of staff transitioned – no longer funded by SMART TA – (333) surpassed the number of project contracted staff (292), evidence of significant transition progress. As of Sept. 30, 2014, 213 of these transitioned SMART TA contracted staff have been Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 32 picked up by the GVN either in permanent payroll status or on contract. The total of 333 transitioned also includes a number of community-based outreach workers moved from monthly stipends to PBIs. In collaboration with the GVN, more than 100 former SMART TA contracted staff have been dropped from service as part of the effort to consolidate services. This included a number of full-time outreach workers who may be replaced by part-time CBO staff or volunteers, and home-based care personnel who are no longer permitted to be on MOH payroll per a recent MOH directive. Going forward, home-based care workers may be provided by the Ministry of Labor, Invalids and Social Affairs (MOLISA) or various local community care or volunteer associations. An HCMC District 8 health official noted that home-based care was in greater demand earlier in the program when, unfortunately, patients were late getting treatment so the disease could not be contained and care was needed for dying patients. As most patients are now stabilized on ART, the need for home caregivers is lessening. In Dien Bien, GVN funds 50 percent of HIV staff; all donors fund the remaining 50 percent. By the end of 2015, the PAC has committed to integrating all SMART TA contract staff into the DOH. This is part of its 2015–2020 plan, which calls for all HIV services to be GVN-funded by 2020 within the MOH/DOH standard preventive care medicine division. In HCMC, the District 8 Center reported that nine of 24 SMART TA contracted staff have been placed on the DOH payroll, seven have been dropped and the remaining eight will be picked up on MOH contract by the end of 2015. PACs in Dien Bien and HCMC identified several challenges in migrating critical SMART TA contract staff to the DOH payroll. Fitting staff into Ministry of Internal Affairs HR caps is difficult; all provincial departments vie for a limited number of openings. Also, HR policies, protocols and systems do not exist to ensure that appropriate HIV HR capacities are institutionalized. Standards of performance are not yet defined for HIV, and accreditation is not endorsed in practice. Another concern of local health officials relates to the allowances (top-ups) provided for HIV staff; although these are important for recruiting qualified care givers, donors will eliminate them. The Global Fund has informed the GVN that their allowances will be reduced by 50 percent in 2015, and altogether in 2016. SMART TA intends to rationalize allowances by the end of the project to align either with the Global Fund or CDC-VAAC. This may result in the loss of skilled technicians. To the extent possible given the timeframe, training of replacements will be necessary to avoid a drop in service care. Financial Sustainability Two critical HIV response long-term financing issues were repeatedly raised in almost every interview conducted: 1) how to reform the Vietnamese health insurance program to include HIV services as part of the general benefits package of eligible services; and 2) how will the GVN provide methadone and ARV when the current donor assistance ends. Without progress on both of these issues, the financial transition of HIV services will not be sustainable. Though it is not within the current SMART TA agenda, health insurance has been identified as the key financing mechanism for future HIV service delivery. For instance, the 2015–2020 transition plan for Dien Bien Province projects that health insurance will cover 70 percent of total costs. Other visited provinces projected a similar reliance, especially for C&T services. A number of issues need to be resolved before Vietnamese health insurance can sustain the HIV response at the level envisioned in Dien Bien. The MOH-Health Insurance Department (HID) noted that for HIV services to be reimbursed, individual patients must be eligible and registered, the DSD itself must be certified by MOH as meeting basic health care standards and the specific service must be included in the list of benefits covered. The service also must be delivered within the MOH’s Curative Medicine Division (CMD), whose services are generally covered by health insurance. Most SMART TA DSDs, particularly the OPCs, are housed within the MOH Preventive Medicine Division (PCMD), whose services are not covered under existing health insurance directives. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 33 Dien Bien PAC noted that 97 percent of its patients are poor or near poor and thus are eligible, but they must register for a health insurance card to have their treatment covered by insurance. The PAC is working with SMART TA support to help poor and near-poor patients register. HIV care and treatment services have been relocated to the Infectious Disease Center within the Dien Bien provincial hospital, within the CMD, and thus HIV services are covered by health insurance to the extent that they are explicitly covered in the CMD benefits packages. Coverage remains unclear and informally applied, but the PAC reported that testing, health exams and drug management services are being partially reimbursed by insurance. SMART TA did pilot a program in Ho Chi Minh City and An Giang province to test options, methods, results and issues with applying for health insurance for PLHIV. SMART TA worked with poor and near poor patients to obtain insurance cards. In An Giang, the HIV services were provided at an OPC in a district general hospital within the MOH Curative Division; thus a full range of services (counseling, health check, testing and medicine) were covered by health insurance. In HCMC, the services were provided either at an OPC in the preventive care division, and thus not covered by insurance, or at a Commune Health Station (CHS), which combines curative and prevention services. For those who registered for health insurance, basic services including counseling and a health examination were covered at the CHS. However, many patients were concerned about possible stigma and discrimination if treated close to home and preferred to go to district or provincial OPCs, which are with the preventive care division and thus not covered by insurance.45 MOH guidance encourages district health centers to merge with district hospitals so district health centers can serve both a treatment and prevention function, which could be supported through health insurance reimbursements if MOH–HID issues an inclusive health benefits package to cover basic HIV services. It is probably beyond the scope of SMART TA to transition all HIV OPC services to the CMD, as has been done under the pilot program in HCMC and An Giang, and in the support of initiative in Dien Bien to locate HIV services within the IDC. However, SMART TA can help support the GVN in integrating care and treatment services to GCN treatment facilities, as these services are reimbursable by health insurance. No current official timeline exists for phasing out the donor subsidy of methadone and ARV medications. The GVN is aware that the day will come when they will have to find the resources to procure these medications. ARV funding in 2013 (from donors and the GVN) was about $17 million.46 While the GVN hopes to be able to recover a significant share of the cost of methadone, stakeholders do not believe that life￾saving drugs such as ARV, which disproportionately affects the poor and near poor, can or should be borne by the patients. Interviews with representatives of the National Assembly, VAAC and the MOH note their concern that the GVN is not sufficiently prepared to tender efficiently to get the medications needed at the level of discount that Global Fund and PEPFAR can achieve, and have stated that technical assistance is needed. Transition of Technical Knowledge Background SMART TA’s transfer of technical knowledge begins with its cyclical process to assess individual sites to determine the level of required technical assistance to ensure effective service delivery post-transition. Technical assistance is provided during and post-transition via the activity’s TA Network. Table 8 identifies eight key indicators initially developed to classify C&T sites for transition preparation and target capacity-building assistance during and post-transition of DSD sites. 45 Health Insurance for People Living with HIV in Ho Chi Minh City and An Giang: Results, Lessons Learnt and Recommendations 46 Vietnam National HIV Health Sector Programme Review integrating Mid-Term Review of the National Target Programme on HIV/AIDS Prevention and Control 2012-2015, page 68. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 34 Table 8: Key Indicators* to Classify Care & Treatment Sites * Note: To better align with the GVN reporting system, indicators #1, 6 and 7 were dropped after field testing. As noted in the previous chapter, the level of initial training was of major importance for operationalizing services and can be credited with substantially upgrading skills and understanding of HIV services. The capacity-building assistance was extensive for both provincial and district technical-professional staff and, to a lesser degree, administrative staff, as discussed in more detail under Building Sustainable Capacity. SMART TA is now engaged in developing a TA Network of trainers at provincial levels. TA Network providers are tasked with routine capacity-building in all aspects of CoPC, carrying out M&E responsibilities and responding to GVN TA requests. As individual DSD sites are identified for transition, the TA Networks will work to ensure continued technical performance and partner readiness. The TA networks are made up of GVN employees. Practicing HIV administrators expressed a desire for capacity-building assistance — training and on-the-job TA — in basic management techniques. While this is not a major focus of SMART TA capacity-building efforts, its absence could affect transition results. Findings on Technical Transition Discussions in the field and with key stakeholders in Hanoi have highlighted an essential aspect of SMART TA’s transition strategy — SMART TA’s knowledge base of HIV service delivery is high, it is being disseminated in the provinces and local health and HIV officials recognize the value of and appreciate SMART TA’s evidence-based approach to HIV service delivery. For instance, Dien Bien and HCMC PACs specifically noted the importance of SMART TA’s assessment methodology, including the indicators in Table 8, to assess whether the level of care provided is achieving effective results. SMART TA’s C&T Monitoring Round 1 (Figure 17) highlights the level of technical transition for C&T sites early in the transition process; follow-up data to measure progress is not expected until June 2015. The red horizontal line is the minimum standard of the indicator ID1, “Proportion of patients who are still alive and retained in care after 12 months of registration,” and the purple line is the minimum standard of the indicator ID2, “Proportion of patients who are still alive and on treatment after 12 months on ART.” ID2 is the most Indicator Minimum Standard 1. % of patients who are alive and retained in care after 12 months of registration ≥80% 2. % of patients who still alive and retained in treatment after 12 months of ART initiation ≥85% 3. % of client registered at the OPC in last 6 months and tested for CD4 within 15 days of enrolment ≥75% 4. % of pre-ART patients visiting the OPC regularly and per appointment with doctor ≥75% 5. % of ARV patients visiting the OPC for medication ≥85% 6. % of eligible patients initiating ART within 15 days during the past 6 months ≥65% 7. % of clients who underwent early warning indicator (EWI) screening for LTFU in past 6 months ≥80% 8. % of clients on ART for 12 consecutive months with suppressed viral loads <1000 cps/ mL ≥85% Figure 16: Level of Technical Transition for C&T sites Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 35 critical indicator. Most sites meet or are close to achieving the standard. Robust monitoring such as this helps to ensure that high technical standards are maintained during the transition process. Field visits suggest that SMART TA has helped to create an ethos of learning and using information in the PACs and district health centers to transition services. The HCMC District 8 team noted that service delivery will be only as effective as the knowledge management system, which provides the critical data needed for sound decision-making. Their understanding of and interest in the various tools introduced by SMART TA — Cascade analysis, Access to Care Information System (ACIS), the “SMART Technical Monitoring,” the HIV/QUAL quality performance assessment tool and the eLog data entry system — are evidence of progress in building a data-driven culture to better deliver HIV services. The TA Network providers, master trainers and other trained TA consultants are critical to ensuring high￾quality service delivery post SMART TA, but at the moment there are only nascent plans to sustain the network. While MOH has noted its intent to move in the direction of setting standards for required long￾term professional development of HIV staff, currently no required certifications or license renewal standards exist to provide the incentives to pay for the TA Network’s consultant services. Until such standards and certifications are institutionalized, there is no leverage. Hanoi and HCMC Medical Schools, or the Hanoi School for Public Health, would be natural homes for continuing HIV education programs and certifications, but none currently have a business plan to provide such services. The PACs and district health centers visited perceived the existing SMART TA monitoring and evaluation protocols and data needs to be burdensome and costly. As described by the Dien Bien and HCMC PACs, such requirements are driven by the project (SMART TA) and donor (PEPFAR) and ultimately will not be supported by the GVN. While SMART TA has worked to integrate project reporting into the provincial reporting system and supported PACs on data collection and analysis, report preparation, data feedback and data use for quality improvement, a simpler system overall would better align with GVN expectations. GVN Readiness, Acceptability and Receptiveness to Transition The level of GVN commitment to financial and technical transition is critical to the success, and thus the sustainability, of SMART TA’s transition activities. To determine whether achievements made through SMART TA are sustainable beyond the life of project, it is necessary to assess both GVN readiness to collaborate with SMART TA on implementing the transition strategy, and the larger GVN commitment to assume ownership of the entire Vietnam HIV response. GVN Readiness for and Commitment to Transitioning of SMART TA HIV Services Findings The GVN will assume financial responsibility for HIV services if DSD sites are cost-effective, prioritized to focus on basic care and treatment, supported by continuing technical assistance and prepared for full integration into MOH’s administrative structures and systems. PACs in four of the provinces visited (Dien Bien, Hai Phong, Hanoi and HCMC) have prepared, and in several cases received PPC approval for, detailed transition plans drafted in collaboration with SMART TA to achieve the results noted above. GVN budget projections at the provincial level are consistent with a sustained GVN financial commitment to SMART TA services, at least in terms of meeting the goal of a 40 percent transition of sites and services. The GVN has not budgeted to cover all of SMART TA’s current costs dollar for dollar, but will fund HIV services at transitioned sites within their own cost parameters. Pilot programs on integration and decentralization of services are robust and well-partnered with the GVN. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 36 The collaboration in Dien Bien with the PAC and the Provincial Peoples Committee (PPC) is a case in point. Early and frequent collaboration between SMART TA and the PAC led to a clear understanding of the achievements of SMART TA (reduced prevalence) and support for integration as the best and perhaps only way to provide more efficient (and thus affordable) and sustainable services. With significant SMART TA assistance, the PAC prepared its 2015–2020 HIV response transition plan, which was presented and approved by the PPC. While providing for a somewhat more basic set of HIV services, the plan nonetheless proposes fully integrating SMART TA services into existing MOH systems and structures in provincial, district and commune health care hospitals/clinics/stations. The PAC and PPC strongly believe that integration better reflects GVN cost models and will, by necessity, become the norm. The detailed budget for the five-year plan, prepared largely on the basis of SMART TA-provided costs estimates and impact scenarios, is shown in Table 9. Table 9: Revenue Sources for the Dien Bien 2015-2020 Plan Revenue Source of Dien Bien 2015-2020 Plan % of Projected Budget Direct Dien Bien Province 8% General GVN-MOH 3% Continuing donors support (PEPFAR and GF) 5% GVN health insurance 70% Co-pay (largely for methadone) 14% Other, including volunteer and private sector support 2% Of particular note, Dien Bien is projecting that GVN health insurance will provide 70 percent of the resources to operate the provincial HIV response by 2020. Dien Bien has already located its primary HIV care and treatment facility in the Infectious Disease Center at the provincial hospital, which is within MOH’s Curative Medicine Division; thus costs are reimbursable under GVN health insurance. Most (97 percent) Dien Bien patients are poor or near poor and thus are eligible for health insurance. While insurance does not cover all treatment services, efforts to include more HIV services within the covered benefits packages of treatment services are underway. Thus, the 70 percent target, though optimistic, is feasible if the GVN advances the agenda for health insurance. Other provinces visited also stressed the view that health insurance would become the primary mechanisms for funding HIV services. While many challenges must be addressed before HIV services are afforded health insurance coverage, the efforts in Dien Bien and HCMC point to progress and a continuing commitment to make this happen. PACs in Hanoi, Ho Chi Minh City (HCMC) and Haiphong have also worked closely with SMART TA to prepare and propose phased transition plans. The HCMC People’s Committee directly supports the PAC and its transition plan both financially and in approving the addition of transitioned staff onto the provincial DOH payroll, which is limited by caps set by the Ministry of Internal Affairs. As noted earlier, it is a challenge to fit transitioned staff within the provincial cap; any given year may see only a handful of openings, which all provincial departments compete to fill. Nonetheless, in District 8 Health Center 9, SMART TA staff have already been transitioned to the HCMC payroll. The remaining HIV response staff identified for retention are not able to fit under the cap and become permanent civil service employees by SMART TA’s end in 2016, but instead will be contracted as MOH employees without a civil service permanent status. This is a workable albeit not ideal arrangement, but is a serious demonstration of the GVN’s commitment to transition. GVN receptivity to financial transition is enhanced when HIV services are streamlined to be more affordable. Officials at both the central (VAAC, MPI, MOH-DPF, MOH-Health Insurance and National Assembly Committee on AIDS) and provincial (PACs, DOH) levels of government commented on the importance of Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 37 SMART TA’s overarching priority of rationalizing the HIV response cost structure through integration, consolidation and phase-outs to better align transitioned services with expected GVN financial commitments. VAAC specifically noted the importance of phasing out sites when services overlap, performance is poor or HIV burden is low. Several PACs spoke of the potential to actually improve HIV service delivery through integration as staff learn to multi-task and better synergize service delivery. A further demonstration of the GVN receptiveness to transition has been the extensive level of engagement and collaboration with SMART TA at the provincial level on transition planning and decision-making. VAAC officials spoke highly of the efforts, which in their view are the only pathway to sustaining the HIV services currently delivered by SMART TA. VAAC leadership also expressed a high level of satisfaction with the SMART TA approach and processes on transition collaboration, evidenced in part by the VAAC collaboration on, and approval of, an annual partnership agreement. VAAC would like to see more sharing of SMART TA transition protocols among all stakeholders, indirect evidence of their acceptance and support for the activities and processes of SMART TA transition efforts. VAAC officials did express concern with a perceived lack of USAID and PEPFAR engagement on key transition issues. Specifically, they expressed a desire to collaborate in understanding the options, and in designing future HIV assistance programs. GVN Commitment to Sustainable Ownership of the Vietnam HIV Response Background “The evidence is overwhelming: If we fail to invest in the HIV response in Vietnam, there will be a resurgence of the HIV epidemic, resulting in the illness and death of large numbers of people and rapidly increasing costs for the public health care system. Vietnam cannot afford to do nothing. Indeed, more money will have to be spent, whether it is now or later. Investing wisely now will have major positive impacts upon Vietnam’s response to HIV.”47 Based on the MOH’s targeted goal48 of allocating GVN funding for 50 percent of total HIV costs by 2015 and 75 percent by 2020, the VAAC estimates that a gap of $27 million will exist by the end of the SMART TA project.49 It is beyond the scope of this evaluation to assess the overall GVN commitment to HIV response in Vietnam. One can say that the “Investment Case”50 produced by UNAIDS and accepted by the VAAC is a good working document on which to continue the needed public dialogue about the costs, impact, priorities, guiding principles and future direction of GVN support to achieve its 2030 goal of 90/90/90.51 The “Investment Case” builds several scenarios of cost-services-impacts to help define potential avenues of success toward containing and potentially eliminating the AIDS epidemic. Understanding that donor and GVN resources will be limited, it sets clear and reasonable GVN priorities going forward: 1. Bring to scale evidence-based and comprehensive harm reduction for key populations 2. Scale up HIV testing and treatment, including “treatment as prevention” for key populations 3. Focus on key populations in high-burden areas 4. Sustain financing, including increasing the domestic budget and the role of health insurance 5. Integrate and decentralize HIV service delivery systems 6. Provide a sufficient supply of ARV drugs and methadone 47 Optimizing Vietnam’s HIV Response: An Investment Case, UNAIDS, September, 2014 48 Ministry of Health Project on Sustainable Financing for HIV/AIDS Prevention and Control Activities 2013-2020 49 Optimizing Vietnam’s HIV Response: An Investment Case, UNAIDS, September, 2014 50 Ibid. 51 90 percent of HIV+ people identified, 90 percent of those in treatment, 90 percent in treatment show reduced viral loads. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 38 Discussions with VAAC and the PACs visited confirm that the GVN, particularly at the provincial level of government, has been supportive and proactive in addressing the first three priorities. SMART TA’s transition approach recognizes the need to focus services on key populations and this is understood and accepted by the GVN. Budget projections are consistent with a sustained GVN financial commitment to SMART TA services, at least in terms of meeting the goal of a 40 percent transition of sites and services. The GVN has not made explicit when and to what extent it will provide funding to continue services now provided at SMART TA DSD sites that will not be transitioned by the end of SMART TA in 2016. Given the lack of a clear donor projection of how/when its funding will be phased out, it is not in the GVN’s interest to commit its resources at this time. GVN, mostly at the provincial level, is effectively supporting SMART TA transition and is projected to exceed the 40 percent requirement as noted in the Financial Transition section. Findings on GVN Commitment to Sustainable Ownership of the HIV Response No overall, donor-agreed roadmap exists for transitioning HIV assistance; nor do explicit projections of cost reduction phasing or any indication of what level and type of assistance can be expected when current HIV response projects come to an end. It is thus not surprising that SMART TA has chosen to focus financial transition collaboration where it could be most effective — at the provincial level and on operational issues. GVN officials, including the chair of the Health Committee of the National Assembly, the director of VAAC, and PAC leadership in the provinces visited, consistently presented a message that they understand that donor assistance will one day end and at that time the GVN will be prepared to provide basic HIV services that are more integrated within existing MOH facilities and less costly than SMART TA stand-alone OPCs, but fully capable of delivering an effective HIV response. SMART TA has contributed to the national dialogue on Vietnam’s HIV response via technical working groups convened by the VAAC, and through this process provided significant data and perspective in the development of the “Investment Case” policy document produced by UNAIDS. This document outlines a number of scenarios as to the costs, impact, priorities, guiding principles and future direction of GVN support to achieve its 2030 goal of 90/90/90.52 The document presents a pathway going forward for a sustainable GVN ownership of the HIV response in Vietnam. Conclusions Conclusions about the process and progress of transition are provided below, in reference to the two key questions noted at the beginning of this chapter: How Do SMART TA Activities and Processes Contribute Intended Transition Results SMART TA is implementing an effective transition of HIV service delivery and is on target to surpass the cooperative agreement requirement of transitioning 40 percent of sites, interventions and partners by the close of the project in September 2016. At the operational level, GVN accepts SMART TA’s transition approach to rationalize service delivery and provide targeted technical assistance based on individual DSD assessments to ensure that service sites are cost-effective and technically prepared for transition. This is a requirement for sustainability, and is showing early success. Integration, streamlining and consolidation are key to aligning service costs with GVN financial commitments and expectations. Integrated services are more affordable than dedicated standalone donor￾funded DSD sites and the only practical structural system into which HIV services can be mainstreamed. 52 90 percent of HIV+ people identified, 90 percent of those in treatment, 90 percent in treatment show reduced viral loads. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 39 Technical assistance to ensure continued quality service delivery post-transition is being provided via the provincial TA Networks supported by SMART TA, which do not have an endorsed home post-SMART TA. The service is critical, appreciated by the GVN, and seminal for long-term professional development of HIV services and staff. The sustainability of the TA Networks must be addressed going forward. SMART TA has a strong collaborative working relationship with the GVN, particularly at the provincial level, in developing and implementing transition plans. SMART TA assistance in preparing mid-term HIV service delivery plans and policy briefs such as the Investment Case, which focuses on transition and sustainability, is critical in putting key financial and technical issues on the table before decision-makers as part of the needed public dialogue on GVN’s long-term HIV response capacities and commitments. Financial sustainability of SMART TA services will require sustained and significant focus on developing an inclusive health insurance program, and on addressing the issue of the cost of medications. The issue of what to do with the projected 39 DSD sites that will not be transitioned is unresolved. Community-based outreach programs are not included within the MOH. Alternative solutions should be pursued. Issues related to HTC sites need to be resolved as part of the general integration of HIV services. Care and treatment sites not slated for transition are at risk at closing down, reducing HIV services in those communities. Continued support to enable a longer time frame may be needed. GVN Readiness and Receptiveness to Transition GVN officials at all levels demonstrated considerable understanding, support and commitment to SMART TA financial and technical transition as evidenced in projected transitioning of SMART DSD sites that will exceed the basic 40 percent requirement. At the provincial level, GVN engages readily in the collaboration on SMART TA financial and technical transition. Decisions jointly made by GVN and SMART TA indicate a high level of acceptance of the SMART TA transition approach and activities. The GVN — the VAAC and various other central-level officials — is concerned about a perceived lack of engagement directly with the USG/USAID. They do not understand USG’s intentions or timeframe on future HIV assistance commitments, which makes planning uncertain. On the broader question of long-term GVN commitment to and ownership of the Vietnam HIV response, GVN recognizes the need to prioritize its limited resources to achieve maximum impact and the need to develop long-term sustainable financial mechanisms to support basic care and treatment programs. Recommendations Recommendations going forward are presented below both on the wider perspective of future USAID programming and collaboration with the GVN and the major HIV donors, PEPFAR and the Global Fund, and specific to the work of SMART TA for the remaining 18 months of the project. Recommendations on SMART TA’s Transition Activities Going Forward 1. Planning for Transition. Planning for transition at the site level is a challenging and complex undertaking. While SMART TA has worked closely in the field with the provinces and PACs, and though SMART TA collaborates with the VAAC on its annual work plan, VAAC would like to see more engagement on transition issues. It may be useful to have periodic workshops with key VAAC and provincial officials to highlight directions, lessons learned and challenges remaining to site transitioning. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 40 “I also want to make clear that PEPFAR will not transition responsibility for its assistance to host governments without a well-defined and mutually-negotiated plan in place regardless of the context… As we move forward, PEPFAR is deeply committed to working hand-in-hand with all our partner countries to support their gradual assumption of greater responsibility for their national responses.” — Ambassador Berx, U.S. Global AIDS Coordinator, April 11, 2014 2. Expand Access to Health Insurance Expanding access to health insurance for HIV services is critical to long-term sustainability. Assistance can be provided to: 1) Identify and assist the transition of care and treatment DSD sites where appropriate to MOH’s curative division within which HIV services are reimbursable; 2) Extend initiatives to register eligible patients for health insurance coverage; and, 3) Support OPCs to become certified by MOH as meeting basic health care standards. 3. Transition Progress Reporting. SMART TA should work with USAID to standardize new and simpler transition reporting templates. Current reporting does not reflect the high level of collaboration, integration and results seen in field visits. 4. Monitoring and Evaluation. Provincial officials have asked for assistance in developing M&E reporting protocols for DSD sites to be transitioned, either directly to the GVN or to technical assistance only status (DSD/TA), that are more consistent with existing GVN systems, and thus more sustainable beyond SMART TA. SMART TA will need to work closely with PEPFAR Vietnam to advocate for alignment of PEPFAR reporting for TA-SDI sites to the GVN reporting system. VAAC is already planning to strengthen its own M&E system across the central, provincial and district levels of reporting. SMART TA could set a new protocols that when DSD sites are transitioned to DSD/TA status, only the reports of the national system, which covers main indicators required by PEPFAR, would be used. It is also recommended that SMART TA work with both VAAC and the PACs to institutionalize and build capacity within the TA Network for specific M&E technical assistance. 5. Communication with GVN. While SMART TA participation on technical working groups convened by VAAC is much appreciated, SMART TA could/should provide its wide and deep perspective more directly into the policy dialogue, perhaps by organizing/hosting a conference to draw on lessons learned from the ongoing transition process. A Wider Role for USAID Engagement with the GVN on HIV Response Activities At the end of SMART TA as currently planned, 39 direct service delivery sites will not have been phased out or partially/fully transitioned. What is to be done with these sites? While that question is important, a broader issue is in play: To what extent can USAID work with the GVN and the major donors to provide critical development technical assistance to help the GVN assume full ownership of the HIV response program in Vietnam? The Investment Case lays out the level of financial commitment and technical priorities required if the GVN is to achieve its 90/90/90 goals for AIDS elimination by 2030. What role can/should USAID play in helping the GVN meet these obligations? Discussions with various stakeholders suggest the following priorities for USAID consideration. 1. Donor-Coordinated Roadmap on Future Support. In discussions, both central and provincial GVN officials agreed that it is challenging to prepare rational, comprehensive, effective plans to assume ownership of the HIV response when no coherent, consistent donor roadmap outlines donor intents and timelines for HIV assistance in Vietnam. VAAC commented that existing working groups are not up to the challenge of partnering with the GVN on critical long-term transition issues; it would like to see stronger USG leadership on developing the roadmap, and subsequently, closer collaboration with USAID to explore technical assistance post-SMART TA. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 41 2. Health Insurance. As noted earlier, financial sustainability for the HIV response will require a Vietnamese health insurance program that is inclusive of HIV services. In interviews with VAAC, HID and MPI, GVN noted the need for continued technical assistance focused on the specifications and cost scenarios of alternative national health insurance benefits packages that include some basic level of HIV services. USAID’s Health Finance and Governance (HFG) project focuses on this subject. In whatever mechanism USAID deems appropriate going forward, it is critically important that such technical assistance continue. The GVN goal of funding 50 percent of HIV direct service delivery by 2015, and 75 percent by 2020, is premised on a robust health insurance mechanism. 3. Provision of HIV Medications. While it is not clear when donor assistance to subsidize ARV medications and methadone will end, there is an understanding that the GVN will eventually need technical assistance to assume the tendering responsibility and be able to source HIV medications to achieve maximum cost advantages. 4. Technical Support to Strengthen Health Systems Associated with Integration and Decentralization of HIV Service Delivery. While much progress has been made in preparing DSD sites for full transition, MOH needs help to mainstream HIV response activities into its curative care division or more holistic joint prevention-care facilities (i.e., infectious or opportunistic disease centers). Technical assistance for dedicated HIV service sites could expand to the larger integrated system, both for technical and administrative capacity building. Pilot programs to demonstrate how a well-integrated center could enhance HIV service delivery would help GVN strengthen the overall health system and ensure a comprehensive HIV response. 5. Sustaining the TA Network. To maintain the quality of service delivery post-transition, the TA Networks developed by SMART TA needs a permanent home. Several suggestions have surfaced on how to institutionalize the development/implementation of life-long professional development streams and certifications. The activity should clarify current proposals and select a path forward. 6. Community-Based Outreach. To scale up HIV testing and enhance links to care and treatment services, SMART TA should set up longer-term support to various community-based outreach efforts using a performance-based incentive approach where appropriate. 7. Continued Support for SMART TA Sites Not Transitioned by 2016. Significant investment in the 39 sites that will not transition by the end of SMART TA could be lost if continued support is not provided. At this point, it appears unlikely that GVN will pick up the ROCs. Areas for Expanded USG–GVN Engagement to Ensure Effective Transition 1. Provincial-Level Transition Roadmap. Transition of DSD sites has been largely piecemeal, reflecting different timelines for phase-out of donor assistance, different priorities within and among provinces and varying prevalence rates in the districts. VAAC working with each province to prepare a tentative five-year plan for HIV service delivery would guide donor efforts to plan future assistance. 2. Progress on Health Insurance. While USAID can provide input on making health insurance inclusive of HIV services, the GVN will have to step up the pace to avoid a drastic cutback in the HIV response. 3. Home for TA Networks, Other Professional Certifications. To sustain the quality of HIV Continuum of Prevention to Care (CoPC), establish long-term dedicated HIV professional certifications and professional capacity-building programs. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 42 IX. BUILDING SUSTAINABLE CAPACITY Introduction SMART TA was launched as Vietnam entered a new of phase of HIV/AIDS response that focused on scale￾up and new and better models of service delivery targeted to high-risk populations. Even though Vietnam is classified as a lower-middle income country, it faces many challenges in marshalling the technical, administrative and financial resources to sustain an effective HIV service delivery capacity. When SMART TA was conceived, the GVN HIV/AIDS response had neither a fully effective organizational structure nor a robust internal capacity to build skill sets of HIV service delivery and administrative staff. Effective monitoring and evaluation systems did not exist, nor specific HIV analytical tools, to measure progress and identify challenges to service delivery management.53 HIV response personnel were largely part￾time, had variable HIV knowledge and had variable operational experienced in counseling, diagnosis, care or treatment of people living with HIV/AIDS. Coordination between the sectors in building program capacities for planning and implementation was weak and fragmented and the capacity for supervising service delivery in state agencies was issue-laden.54 The internal capacity-building activities that did exist lacked focus and coordination.55 In this context, SMART TA was designed to include a major component of building sustainable HIV/AIDS response capacities. This evaluation addresses key issues that directly relate to the underlying purpose of SMART TA’s capacity-building efforts, including following:  To what extent are the activities, techniques and processes employed by SMART TA contributing to the intended results of strengthening local technical capacity of HIV/AIDS services?  How sustainable are the programs/program components? To what extent have activities designed for long-term sustainability been institutionalized? Findings are provided on the adaptability and GVN adoption of SMART TA’s capacity-building initiatives. Recommendations address ways to enhance local and national ownership and future commitment to continued implementation of good practices. Capacity-Building Activities and Intended Results Background The SMART TA ‘Push and Pull’ Approach SMART TA’s capacity-building initiatives were designed to be integral to achieving the program’s goals of introducing/scaling up new and better service delivery models, in transitioning those models to GVN ownership and in sustaining GVN service delivery at high quality. The primary SMART TA capacity-building objective is to strengthen GVN technical capacity, but considerable effort has also gone into transferring tools and processes for effective management. SMART TA assists national, provincial, district GVN and civil society organizations (CSOs) to understand and more effectively program their HIV response (and other health services) across the Continuum of Prevention and Care (CoPC) response. SMART TA technical assistance attempts to ensure that strategies are evidence-based, of good quality and locally feasible to ensure sustainability. 53 Chu Quoc An, Health System with Tasks in HIV/AIDS Control. 2012 54 CCRD Study on Organization of HIV/AIDS Response System in Vietnam, 2010 55 Situation analysis in the National Strategic HIV/AIDS Plan 2010-2020 Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 43 “Push and pull” terminology is used by SMART TA to distinguish between two approaches in the development of HIV capacities. “Push” is defined as a as a process that builds capacities by targeting assistance to best practices, new technologies and innovations as deemed necessary by the project. “Pull” is a process of facilitating access to technical information and skills that program implementers believe they need. SMART TA uses a combination of both methods to synergize capacity-building activities, helping ensure the maximum transfer and development of the knowledge needed to better manage, coordinate, deliver and monitor the HIV response. Innovative Program Methodologies and Models SMART TA’s CoPC “Cascade” analytical tool is a conceptual framework for identifying gaps in service delivery that influence prevention, care and treatment effectiveness. The Cascade model tracks how individuals move — or don’t move — through a health service continuum of care from testing to treatment. By identifying, monitoring and targeting assistance to reduce “leaks,” health service systems can improve performance and efficiency. The Cascade tool is one of many project innovations, which include the SMART technical monitoring system, to assess a site’s readiness for transition and monitor quality of post-transition service. These innovations include the eLog to simplify digital data entry in the field, mobile technologies to track outreach results and operational research to achieve greater understanding of patient behavior. All of these tools are designed to improve the effectiveness of the HIV response. Findings Figure 17: Numbers of MOH Staff Trained by SMART TA As shown in Figure 18, SMART TA has provided a significant amount of training to both HIV service personnel on the DOH payroll and SMART TA contracted staff. Training was largely technical, but administrators and managers received capacity-building assistance via “supportive supervision.” The training courses and TA visits by SMART TA staff were based on findings from the TA needs assessment activity. Effectiveness of SMART Capacity-Building Initiatives VAAC and PAC officials noted that the level of initial training was of major importance for operationalizing service and can be attributed with substantially upgrading skills and understanding how to best provide HIV outreach, counseling, testing and care and treatment. The capacity-building assistance was extensive for provincial and district technical-professional staff and, to a lesser degree, administrative staff. Discussions with PAC officials and a survey of staff in the field service sites visited, summarized below, document trainees’ perception that SMART TA’s capacity-building assistance has improved HIV service 0 500 1000 1500 2000 2500 3000 2012 2013 2014 Government staff SMART TA involved staff Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 44 delivery. For all questions, 90 percent or more of respondents agree or strongly agree that the assistance has helped them improve service delivery and meet the needs of HIV assistance recipients. Table 10: Summary of Survey of SMART TA HIV Service Delivery Site Personnel SMART TA did not undertake a rigorous baseline assessment or a time-series competence improvement analysis to document the direct effects of its capacity-building approach and activities. When the SMART TA carried out “push” assistance, all initiatives were considered new or almost new to the staff at all sites, so SMART TA directed its efforts to “push” assistance to build a basic level of critical competencies. SMART TA then used its program monitoring tools (see graphic below) to assess the level of functionality of service delivery sites and considers the results evidence of outcomes of the capacity-building assistance provided. While the protocol of measuring results (e.g., a district center health worker is trained in organizing HIV referrals and, following training, the health referral activity is documented as being well done) is valid in terms of trying to go beyond outputs to assess outcomes, the level of assessment directly linked to capacity-building activities is not robust and not clearly reported. Field interviews highlighted the fact that SMART TA worked closely with service sites to identify specific needs of the staff and consequently targeted training courses accordingly as part of their “pull” protocol. The process was useful for adjusting specific capacity-building assistance content and methodology in the training courses. In response to concerns that more and better “pull” capacity-building assistance was desired in the field, SMART TA noted that it has begun to tailor future capacity programming to be more responsive to field requests. GVN officials at the national level noted little engagement with VAAC, MOH or other ministries in the design and implementation of SMART TA’s capacity-building programs. They also suggested that training and technical assistance was needed at the national level VAAC and MOH said future USAID programming should consider providing such assistance. Questions # of Respondents Strongly Disagree % Disagree Neutral Agree Strongly Agree % 1 2 3 4 5 1. SMART TA project has introduced activities that help to reduce the acquisition and transmission of HIV 92 0% 2% 0% 8% 90% 2. SMART TA project has assisted my program in generating and using HIV-related data and information to improve use and quality of services 89 0% 4% 4% 26% 66% 3. SMART TA innovations in the areas of outreach, HIV Counseling and Testing, and Care and Treatment are useful and effective. 90 0% 0% 1% 21% 78% 4. SMART TA is providing the range and quality of assistance to build the GVN technical capacities needed to ensure transition. 85 0% 0% 7% 33% 60% 5. SMART TA project has been directly involved with increasing the technical skills or capacity of local providers or managers. 91 0% 3% 3% 26% 68% 6. SMART TA services are managed to meet the needs of local recipients. 92 0% 2% 5% 26% 67% 7. SMART TA has improved the coordination of services at the local level. 92 1% 4% 4% 24% 67% Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 45 PAC and district health center officials conveyed that SMART TA did not provide sufficient basic management training to complement the technical assistance and supportive supervision focused on management skills development. Capacity of service site managers in dealing with issues during the transition process is in need of improvement. The site managers are now dealing passively with key issues, e.g., reducing human resources and other recurring costs while maintaining service quality. They also expressed an interest in learning how to train staff to multi-task better. Evidence of adaptability and replication of methodologies and innovations The Cascade model of assessing CoPC progress and problems has been readily operationalized at the service sites visited, and as noted by several DSD managers, it is a critical tool for improving the effectiveness of HIV service delivery. The director of the District 8 Health Center in HCMC noted the importance and “highly effective” results of the Cascade analytical approach, including the operational research that provides an in-depth perspective on why patients drop out of care at the different stages of the Cascade. The director also made special note of the importance of SMART TA assistance on: mobile (phone) tracking systems; the MMT website (which averages 10,000 hits per month) implemented to raise awareness and provide a confidential setting for sharing information and answering questions; the development of human resources job descriptions as part of the effort to institutionalize HIV-sensitive health care approaches, which VAAC suggests can be replicated in other provinces; and the training provided to improve HIV counseling capacities. While the SMART TA-piloted mobile tracking and referral system was noted to be of high benefit in urban areas, interviewees in Dien Bien noted that many health workers did not have mobile phones or were reluctant to use the system. When they tried, the system often failed and referrals dropped. SMART TA has not set up effective channels for sharing its training materials and innovations with other institutions and stakeholders. As discussed with interviewees at the Hanoi School of Public Health and Hanoi Medical University, SMART TA helped develop curriculum and materials for pre-service training but no collaboration has taken place among these organizations to adapt materials for post-service capacity assistance. HAIVN and CDC are currently mandated to oversee this work. Many officials within the GVN and other outside stakeholders spoke positively of the innovative use of research on operational issues to improve understanding of different approaches to HIV service delivery. HCMC PAC and district officials said they would like to see specific capacity-building assistance to improve the competence of the GVN in-house staff in designing and conducting studies to conduct evidence-based assessments of interventions. While they spoke of a keen interest in learning these skills, it was not clear if the PAC or district centers would be able to dedicate the resources to maintain this capacity. All PACs spoke of the complexity and lack of value of the PEPFAR-required M&E approach. PAC and district officials in HCMC and Dien Bien were particularly blunt with their concerns about what they view to be burdensome and unnecessary data requirements, saying the database system used to collect, track and report on service delivery was “too sophisticated” for their needs. The officials commented that the system was project-driven in its design and implementation and inconsistent with GVN systems, and would not be used when SMART TA ends. District health staff in HCMC specifically noted the importance of electronic patient records for managing care innovated by SMART TA and said they would like to have more training and technical assistance in this area. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 46 Are SMART TA Capacity-Building Activities Sustainable? Background SMART TA has supported the establishment of nine provincial Technical Assistance Networks (TA Networks) of trainers as their primary approach in sustaining its capacity-building assistance initiatives. As well, SMART TA helped to operationalize the national MMT mentorship system. SMART TA planned and followed strategic steps to establish the network, including:  Assessment of current TA systems and needs at the provincial and site levels;  Developing plans for further strengthening and systematizing TA methods and activities including procedures for routine evaluation and adapting TA methods to local situations and culture in Vietnam;  Training-of-trainer programs at the provincial and local levels;  Knowledge management services fully developed to support the TA system and its TA providers; and  Training, mentoring and coaching of TA staff. TA Network members are required to reach a relevant level of knowledge and experience on health programs and HIV service delivery systems in Vietnam. The recruitment of TA members took into consideration their interest and skills in mentoring and training. The TA Networks comprise health professionals including doctors, public health officials and other related positions in government. Findings Institutionalizing Capacity-Building Assistance The provincial TA teams included members from different professional agencies at both the provincial and district levels who had knowledge and interest in providing training and mentoring. However, VAAC commented that the TA Network quality varies among provinces related to both the individual competence of the TA members and the environment of learning (local officials’ interest in using the assistance). Provinces do approve specific use of the TA Network to provide technical assistance via the network. PAC officials interviewed spoke of tentative acceptance of the approach. To date, 120 individuals have participated in the TA Networks, 56 of whom have been trained as master trainers. These TA members are from both GVN institutions and CSOs, and are specialists in all CoPC technical areas, management and M&E. Several service delivery sites have been become “study sites," where partnering provinces can visit and learn, and this experience is considered valuable and effective. MOH is moving toward standards for required long-term professional development of HIV staff. If such standards are set and certifications and license renewals are required, demand from service providers could surface for marketable training and professional development by TA Network members that could sustain their operations. But until such standards/certifications are institutionalized, no external incentive exists for service providers to maintain or improve their skills. Several officials in MOH noted that Hanoi and HCMC Medical Schools, and/or the Hanoi School for Public Health, would be natural homes for continuing education programs and certifications, but no business plan exists to do so. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 47 In the MOU signed with the VAAC, SMART TA committed to providing support to institutionalize capacity-building activities the central level, as follows:  Support improving capacity of the HIV/AIDS education and training system by developing training materials and training-of-trainers courses to be shared with the GVN and disseminated to all providers.  Standardize the procedures and guidelines for preventing and control of HIV/AIDS  Establish and improve capacity of the network/system of learning and training in the country.  Support participation in workshop, training courses and dialogues  Support the GVN with knowledge and techniques to conduct research and studies. VAAC commented that while SMART TA has implemented activities to address the above, most of the knowledge base developed to build capacities is still project-focused and project-maintained. Little has been directly transferred to the GVN for long-term sustainability. While SMART TA contributed significant technical support to the process of developing national HIV service guidelines, VAAC commented that the level of direct support to the central government to build its own technical capacities has not met expectations. This may be explained in part by the USAID award for the Leadership-Management-Governance project, which supplanted SMART TA and is supporting the GVN at the central level. Evidence suggests that training (including training curriculum and materials) is being absorbed by local partners, and being used in Hanoi Medical University and PACs. However, the possibility of institutionalization of all capacity-building strategies and training curricula — a critical element of SMART TA’s capacity-building objective — is still unclear. VAAC and MOH officials noted that ensuring that HIV services are provided post-transition at a high level of care and treatment requires a sustainable organizational structure for professional development of individuals’ skill sets and institutional capacities. CSOs interviewed noted that at the start of their projects, SMART TA was sharing information that helped them improve their outreach efforts, but more recently collaboration has been inexplicably restricted, both by USAID and by SMART TA. As noted elsewhere in this report, SMART TA’s project-oriented M&E reporting requirements, designed to meet complex PEPFAR standards, are not consistent with GVN needs and expectations. GVN made it clear consistently and in numerous interview comments that this element is not sustainable. SMART TA and PEPFAR recognize the importance of this issue and are working with technical committees to conceptualize a more streamlined and sustainable system for the post-SMART TA GVN HIV response. Evidence of Sustainable Learning Discussions with, and the survey of, local HIV health services staff noted the value of the training provided by SMART TA, commenting specifically that the overall effort to train, coach, mentor and provide “supportive supervision” improved their institutions by establishing more standardized norms and processes, thus strengthening their confidence and skill in performing their jobs. Learners from several sites noted improved capacities to coordinate with other staff, provide better support and care to patients and better administer activities. Pharmacists shared that the training gave them a much better understanding of new MOH regulations, helping them to improve drug management and reporting. Counselors noted they gained new skills in communicating with patients. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 48 The District 8 Center in HCMC noted an example of sustainable learning where staff helped design and conduct research to identify why PLHIVs did not enroll in the treatment program, or dropped out after enrollment. The results of the study were used to tailor messages that have since been disseminated in other provinces (see photo). An important example of sustainable learning related to program administration was participating PACs’ collaboration with SMART TA to develop long-term service delivery plans, including staffing levels, budgets and service delivery protocols (e.g., integrating, decentralization, Treatment 2.0, etc.). SMART TA provided input and technical assistance on how to prepare and present these plans; these skill sets continue to be used in the provinces. The Provincial Peoples Committee approved the 2015–2020 Dien Bien HIV Response Plan, underscoring the importance of such collaboration. Conclusions Interviews with GVN officials at the national and provincial levels and other HIV assistance stakeholders, including WHO, UNAIDS and CDC, and the results of the survey of HIV practitioners provide evidence that SMART TA has afforded significant training and technical assistance to individuals and service delivery sites to effectively operationalize best practices in the Vietnam HIV response. SMART TA has not comprehensively measured and reported the direct specific results of individual capacity￾building activities. Toward sustaining SMART TA capacity-building activities, the activity has established and continues to build the capacities of its TA Networks of training and technical assistance advisers, but the strategy or plan to organize or administer this post-SMART TA is nascent. It remains unclear whether SMART TA will achieve the intended result of nurturing a long-term sustainable Vietnamese system of learning and training that can ensure quality HIV services. Recommendations On Ongoing Capacity-Building Assistance 1. In interviews, health care workers exhibited a clear idea of the training they need. To be more responsive to the individual needs and interests of SMART TA contracted staff and MOH personnel, SMART TA should focus more attention on “pull” capacity assistance. 2. To better assess capacity-building activities’ success, impact and cost-effectiveness, SMART TA should directly survey assistance recipients on their knowledge retention and use of the specific skills learned. Assessment of service delivery is notable in the effort to measure outcomes, but this does not comprehensively capture the direct impact of training and technical assistance. 3. SMART TA should continue providing technical assistance to and coordinate with three previously engaged CSOs (COHED, CCRD and Life-Center) that are direct service providers in PEPFAR￾supported provinces. If resources allow, SMART TA should also support other CSOs for a more extensive network of sustainable CSO-based service providers as per the MOU between SMART TA and VUSTA. 4. SMART TA should collaborate with VAAC and MOH to host workshops or conferences for all HIV professionals to better disseminate best practices, on-the-ground learning and perspectives of successes and failures and required next steps in capacity-building assistance. 5. SMART TA should work with VAAC to develop a more streamlined reporting system that aligns with GVN needs and expectations as noted prior. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 49 6. SMART TA should work with VAAC to identify and implement a plan for the next 12 months for operational research that addresses issues raised in the field related to the effectiveness and efficiency of pilot models such as TB/HIV integration, the 3 in 1 model, the 2.0 treatment program, and the HIV/AIDS in closed settings (i.e., prisons) protocol. Such studies would be useful for PACs as they continue to plan for transition, and would be important for VAAC in its efforts to institutionalize and scale up the service models. 7. SMART TA should focus more effort on teaching site managers to use data from the M&E system to analyze issues and plan consequent interventions, and on basic administrative skill sets in financial management, strategic planning and public participation. On Sustainability 1. SMART TA should continue discussion with the GVN to identify and develop strategies and/or business plans to sustain local HIV related technical professional development capacities, as specified in its capacity-building objective. SMART TA could begin by working with VAAC to share the SMART TA knowledge base of best practices so VAAC can begin to serve as the clearinghouse of such information. SMART TA should share all of its capacity-building training materials, guidelines for implementing the service delivery models and operational innovations. 2. More importantly, SMART TA should continue to work with VAAC and other GVN institutions to develop strategies to sustain its TA Network and its knowledge base. SMART TA should take the lead in collaborating with VAAC, USAID and PEPFAR to assess alternatives and business plans for viable institutional arrangements to ensure long-term HIV capacity building in Vietnam. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 50 X. MANAGEMENT AND COORDINATION Introduction The nature of the epidemic and the national response to it continue to evolve, and the contextual dynamics of the environment in which SMART TA operates are ever-changing. For example, at the inception of SMART TA, PWIDs were a primary target among the most-at-risk population. Due to a variety of factors, HIV prevalence among PWIDs is decreasing. At the same time, concern is increasing among epidemiologists and health planners that infectivity among MSMs is rising and that MSMs may be a transmission “bridge” to the general population.56 Also, at inception, SMART TA’s technical approach in support of the national HIV/AIDS response involved concentration in each of its three interactive objectives (high-quality services, capacity building and transition). Given the changing dynamics in SMART TA’s contextual environment, the key evaluation questions below attempt to assess SMART TA’s design and approaches to management and implementation of its program to ensure relevance in meeting program objectives. Key Questions  How can the project design, management and implementation become more efficient, effective and relevant toward achieving project objectives in connection with the changing contextual dynamics?  How effective has the collaboration/coordination within project components and with other stakeholders been in maximizing efforts and achieving greater results?  How do the programs integrate gender equality and female empowerment in strategic planning, programming, project design and implementation, and monitoring and evaluation? The evaluators used a combination of review and examination of SMART TA documents and report, and in￾depth interviews with SMART TA’s management team to gain insights into the activity’s management approaches, results and areas that might benefit from improvements. All told, the evaluation team and the SMART TA management team spent more than 20 hours together in formal and informal presentations, group question-and-answer discussion sessions and one-on-one “data mining” with SMART TA’s strategic information unit. Findings Design, Management and Implementation Design-wise, SMART TA’s three interactive objectives are process-oriented and allow for shifts in approach or emphasis that improve results without changing the objective itself. SMART TA’s work plans and annual reports indicate that the activity is engaged in a continuous process of internal monitoring and adjusting approaches, processes and staffing. SMART TA has an internal organizational structure made of technical teams that concentrate vertically on developments, training and technical assistance in a specific programmatic area of concentration. Examples include Strategic Information Systems or Injecting Drug Use. As SMART TA’s management team told evaluators, and further evidenced by a review of SMART TA’s internal planning documents, the activity has given considerable attention over the past three years to adjusting its technical approaches, activities and internal project staffing levels and configurations to improve 56 “Epidemiological Fact Sheet on HIV and AIDS in Vietnam,” UNAIDS, 2012 Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 51 efficiency, effectiveness and relevance of its program. In Year 2, by its own assessment, SMART TA found that its HIV prevention activities and responsibilities were fragmented across several teams and saw a clear, emerging need to support combined prevention programs and interventions under one functional unit. Accordingly, during Year 2, SMART TA formed a new team, the Technical Assistance and Health System Strengthening (TACHSS) unit, to coordinate these activities across all SMART TA technical teams and to lead many activities that had been identified to begin in Year 3. The formation of the TACHSS team was expected to reduce some of the management and administrative burden from the other internal technical teams and allow them to enhance and strengthen the TA they provide to provinces and national institutions. The TACHSS team was set up to serve as a point of first contact for provinces, help them access SMART TA services and help ensure understanding and responsiveness to their needs, situations and requests. The TACHSS team also coordinates the pull and push TA system.57 In key informant interviews at the provincial and site levels, respondents consistently said it was easy to request and receive assistance from SMART TA through their provincial coordinator, suggesting that the activity’s approach of centralized coordination of technical assistance through the TACHSS team helps provide timely and relevant responses to requests from the sites. SMART TA has also instituted an internal performance management system that identifies primary and secondary responsibilities for each deliverable and major task listed in the SMART TA annual work plan, and integrates these responsibilities for team leaders and staff members. Additionally, SMART TA has provided training on management and personal coaching on supervision and related skills (including gender sensitization) for all associate directors. The use of operations research activities is also an important management strategy in SMART TA because these activities build strong technical collaboration and relevance at the provincial, district and site levels, and have frequently provided solutions to service delivery or outreach problems that were then implemented by authority of a PAC. SMART TA identifies and initiates priority research based on opportunities it identifies, as well as in response to programmatic problems and the need for information from the provincial, district or site levels. To date, SMART TA has conducted more than 24 different operations research studies and pilots, including:  Care and treatment studies at the provincial level;  Major care and treatment studies, including on the ART Cascade at the national level;  Methadone maintenance therapy pilots and studies at the national and provincial levels; and  Prevention and enhanced outreach approach studies and pilots at the national and provincial levels. Operations research activities are also important given the fact that operations research is among the most critical technical contribution SMART TA makes to the national HIV/AIDS response as noted by GVN and international HIV officials. Local officials commented that operations research needs to be more focused and selective. In particular, more attention should be directed at exploring innovative approaches and pilot programs that could inform planning and advocacy for successful and cost-effective transition (from external funding to GVN budget support for HIV/AIDS programs and services). To ensure that GVN decision-makers and policymakers are engaged in the results of operations research and pilots, SMART TA’s TACHSS team is in the process of summarizing study results for presentation to VAAC, and plans to provide technical assistance to VAAC and provinces to disseminate good models at a national workshop. 57 See a discussion of the “Push and Pull” approach in Chapter VII – Building Sustainable Capacity Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 52 Although SMART TA described plans for engaging VAAC at the beginning of formulating and piloting initiatives, the evaluators were also told of at least one instance where VAAC expressed no interest (that is, saw no relevance) in and declined to be involved in provincial operations or pilot studies. In the interest of effective use of limited resources, SMART TA may need to prioritize and give preference to studies that resonate with national policymakers and have most potential for national applications and scale-up. SMART TA Coordination This evaluation considered whether SMART TA had been effective in collaborating/coordinating with other stakeholders. Coordination emerged as an area of SMART TA performance that showed strengths as well as weaknesses. The evaluators talked with key informants, including PEPFAR agency partners, other international partners, the central government, VAAC as a sub-grantee and CBOs. In general, SMART TA was considered to be a technical leader and effective PEPFAR implementer. Based on file documentation as well as key informant interviews, in the first two years of the activity, SMART TA was highly visible as a coordinator and collaborator, particularly on strategic information systems developments, at the central level. As SMART TA’s work, by agreement with USAID, became more focused in provinces, districts and at the site level, the impression is that SMART TA exercised less initiative as a central-level collaborator. Following is the gist of discussions with various key informants: Interagency partners. SMART TA was seen as technically strong, particularly on operations research, strategic information systems development, use of the CoPC Cascade at the site level as a monitoring, evaluation and strategic planning tool, and on the enhanced outreach approach innovations. One agency partner with whom the evaluators talked said it would be helpful to have more frequent information-sharing and technical collaboration, but that informal collaboration appeared to be discouraged in preference for discussions and collaboration in more formal settings with USAID present. Another agency partner noted that SMART TA collaboration was limited and that particularly in recent times, it appeared that SMART TA was providing assistance on use of its VNIS360 information systems, but was not helping provinces on PEPFAR reporting as much it should have been. Other international partners. As evidenced by its internal work plans and reports, SMART TA was active on technical collaboration and support, particularly on strategic information systems development58 in its first two years. More than one key informant reported that SMART TA participated occasionally in donor forums and technical working groups and provided technical input when requested to do so. GVN.59 The SMART TA activity was known to all GVN representatives contacted, but generally there was no routine collaboration. The representative from the Ministry of Planning and Investment felt that, aside from VAAC, not much need existed for ongoing central-level oversight of SMART TA’s activities since all of its work was at the province level and below, and because no project financing went directly through government. Other central agencies. SMART TA collaborated with the Hanoi School of Public Health and several other partners, such as CDC and WHO, to support VAAC in developing the National Care and Treatment Improvement Program (HIVQUAL) indicators. In addition, SMART TA provided sub-grants to the National Institute of Hygiene and Epidemiology (NHIE) for research assistance. VAAC. SMART TA has a MOU with VAAC for all activities carried out in provinces, districts and sites. Annual site- and district-level transition plans are developed collaboratively with SMART TA, aggregated at provincial levels and then reviewed jointly with VAAC to serve as a basis for the more general SMART TA/VAAC annual plan, which is sent to USAID for approval. While VAAC expressed satisfaction with USAID support and the work being done by SMART TA in provinces, it had two concerns about the sub- 58 See Section V for more details on SMART TA’s work and collaborations on strategic information systems. 59 National Assembly, Ministry of Planning and Investment, Ministry of Labor, Invalids and Social Affairs. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 53 agreement planning process. Specifically, the GVN’s fiscal year does not coincide with the USG’s, so the budget cycles do not align. This results in VAAC/SMART TA MOUs and budgets not being approved until three or four months into the GVN’s operating cycle, which tends to slow down or even halt implementation of annual transition plans. VAAC believes that the annual plan approval, the program in general and efficient use of PEPFAR/USAID funds would all be improved if it had closer, more routine communications and contact with USAID. CBOs. SMART TA worked closely with the USAID-funded Pathways activity during its first year or two to provide training and technical assistance to Pathways’ CBOs in outreach strategies and working with peer educators. Since the Pathways activity closed, SMART TA no longer maintains close working ties with the three CBOs involved in the USAID-supported Community Links activity. The evaluators talked with key informants from each of the three CBOs. Each had positive things to say about the earlier collaboration and the training they received from SMART TA, but did not describe any significant current collaboration. In fact, an inherent conflict may have developed between SMART TA and the CBOs around performance￾based incentives to peer educators. Core funding for CBOs is based on per capita productivity of its peer educators. Recently, SMART TA has piloted performance-based incentive schemes that might be misinterpreted by CBOs as being in competition for productivity. SMART TA’s pilots will likely be used to inform GVN, including VAAC and the Vietnam Union of Science and Technology Associations (VUSTA), to take policy steps to engage CBOs and CSOs in community-based HIV/AIDS outreach. Gender Considerations The evaluators reviewed SMART TA documents and reports and talked with key informants to get some indication of the extent to which SMART identifies and uses opportunities to advocate for and operationalize gender considerations. Review of SMART TA work plans indicated that issues of gender are considered in the design of policies and programs and on any occasion that includes opportunities to advocate for the reduction of stigma/discrimination. SMART TA reported that it had developed a protocol to assess issues relevant to lesbian, gay, bisexual and transgender (LGBT) people, women, and other vulnerable populations in accessing services, service barriers and other gender-related issues in SMART TA-supported sites. Interviews in the field noted that SMART TA did make efforts to train and sensitize its staff and service providers on gender issues, stigma and discrimination and related issues, particularly those that are barriers to or negatively affect the quality of services. Although not extensive, SMART TA also has taken some steps to sensitize its internal staff to gender considerations in various aspects of its administrative work, including design, monitoring and evaluation. Where relevant, indicators were observed to be disaggregated by gender and reported appropriately. However, beyond that, attention to gender-specific programming as outlined in various workplans did not appear to transfer to SMART TA’s training and technical assistance activities in provinces, districts and sites. . For instance, one activity to conduct a pilot of a comprehensive community-based harm reduction model for FSWs to link vulnerable women to a variety of health and social support on HIV, sexual health, gender-based violence (GBV), and income generation was not implemented, in part due to delays by MOLISA. The evaluators concluded that in general, gender considerations have not been robust, in part because they have not been well defined. SMART TA has more recently developed a gender strategy to guide more pro-active programming going forward. As noted by several informants at the provincial and facility levels, HIV/AIDS prevention and treatment strategies and services in the Vietnamese context are targeted to the conditions and consequences of gender inequities and gender-related stigma and discrimination. Because sexual behaviors and sexual exploitation are Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 54 intrinsic to risk of infectivity, health planners and service providers tend to focus on risk factors, risk behaviors, and patterns of acquisition/transmission in most-at-risk populations (MARPS), rather than gender analysis or enumeration. Conclusions SMART TA’s three interactive objectives are process-oriented and sufficiently flexible to allow for shifts in approach or emphasis that could enhance SMART TA’s efficiency, effectiveness and relevance in meeting its objectives. SMART TA has an organizational structure and management strategies (such as the push-pull approach to identifying priorities, operations research and pilot studies) that allows it to stay informed about changing contextual dynamics and the need to shift emphasis/approaches that will enhance its efficiency, effectiveness and relevance. In the interest of effective use of limited resources, SMART TA may need to prioritize and give preference to operations research studies and pilots that resonate with national policymakers and have the greatest potential for national applications and scale-up. VAAC/SMART TA sub-agreements are not being approved until three or four months into the GVN’s annual operating cycle, and VAAC believes that the annual plan approval (and the program in general), would benefit from closer, more routine communications and contact with USAID. Gender considerations are not robust. Given that gender is so integral to HIV/AIDS programs and services, more attention should be paid to this technical area. Recommendations 1. SMART TA should continue with and further strengthen the Technical Assistance and Health System Strengthening (TAHSS) unit to prompt VAAC decision-makers and policymakers to participate in and disseminate national policy-informing operations research. If possible, details should be included in VAAC/SMART TA sub-agreements. 2. SMART TA should support PACs to carry out coordinated planning across donor-funded programs that align with GVN operating cycles. 3. SMART TA should encourage and facilitate arrangements for USAID to hold regularly scheduled review meetings or make provincial site reviews with VAAC and SMART TA. This can foster common understanding and agreement regarding technical approaches and progress on implementation of SMART TA’s transition and assistance plans. 4. SMART TA should revisit its March 2014 gender strategy to identify ways to strengthen this element in its program, and incorporate this strategy into its workplans, and do follow-up reporting. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 55 ANNEX A. SMART TA SERVICE MAP Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 56 ANNEX B. EVALUATION SCOPE OF WORK STATEMENT OF WORK TITLE Mid-term Evaluation of the Sustainable Management of the HIV/AIDS Response and Transition to Technical Assistance (SMART TA) Project in Vietnam. PURPOSE The United States Agency for International Development (USAID), in Vietnam, Office of Health, requires technical support to conduct a mid-term evaluation of USAID’s Sustainable Management of the HIV/AIDS Response and Transition to Technical Assistance (SMART TA) Project Vietnam (October 2011 – September 2016), as detailed in this Statement of Work. The evaluation is estimated to take about 12 weeks from Sept 1, 2014 to Nov 30, 2014. This mid-evaluation will primarily determine to what extent the USAID-funded program is meeting the stated objectives articulated in the Cooperative Agreement and related Modifications, the Performance Management Plan, and provide an overview of the major accomplishments, performance accountability, and implementation weaknesses, challenges or lessons learned to improve project performance. Specifically the Evaluation will serve:  To evaluate the progress made by SMART TA toward intended results and change as described in the Performance Management Plan (PMP) and describe the context and what other possible factors (such as service delivery programs by other donors or actors, advocacy and system strengthening by a variety of actors, etc.) that are also contributing to the observed changes.  To determine whether the activities, technical mix of interventions (as specified in SMART TA annual workplan and its M&E) are appropriate to achieve the intended results and/or change (including planned outputs, outcomes and impact) within the five-year performance period and to recommend modifications to project activities or priorities, as necessary, to address implementation issues, apply lessons learned, or capitalize on new opportunities, especially taking into account any weaknesses/deficiencies in program design.  To determine whether achievements made through SMART TA are sustainable beyond the life of project and, if not, recommend techniques and/or processes that would ensure sustainability of the gains made. The evaluation will thus provide USAID/Vietnam, the Government of Vietnam, and other in-country stakeholders with objective information on what has been achieved to date, what is working and why, as well as what is not working. The information will inform decisions to make appropriate modifications. INTRODUCTION The United States Agency for International Development Vietnam (USAID/Vietnam) is one of five U.S. Government (USG) agencies implementing HIV/AIDS activities in Vietnam under the U.S. President’s Emergency Plan for AIDS Relief (PEPFAR). PEPFAR Vietnam includes USAID, the Centers for Disease Control and Prevention (CDC), Substance Abuse and Mental Health Services Administration (SAMHSA), Department of Defense (DOD) and Department of State (DOS). The FY 2011 budget request for HIV/AIDS activities was $102 million, and the Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 57 USG budget allocations for HIV/AIDS activities in Vietnam is decreasing annually over the past few years ($89M in FY 2012, $69M in FY2013 and $65M in 2014). USAID is the largest USG PEPFAR implementing agency with an anticipated budget of approximately $50 million in FY 2011, $40M in FY2012, $40M in FY2013, and $40M in FY2014. In 2008, legislation was passed reauthorizing and expanding the USG’s commitment to PEPFAR for five additional years, from 2009-2013. A key overarching goal is the transition from an emergency response to the promotion of sustainable country programs. This is especially critical in Vietnam, as PEPFAR funding will decrease during the life of this project. In efforts to transition PEPFAR in Vietnam from an emergency response to sustainable country programs, USAID/Vietnam supports a project, Sustainable Management of the HIV/AIDS Response and Transition to Technical Assistance (SMART TA), which aims to provide critical services within the HIV/AIDS continuum of prevention to care (CoPC), transition selected PEPFAR-supported services to Government of Vietnam (GVN) management and financial support, and establish stronger technical collaboration to build the capacity of the GVN and other stakeholders to scale-up and provide quality services for People Living with HIV/AIDS (PLHIV) and Most-At-Risk Populations (MARPs). The timeframe for this project is from October 2011 to September 2016. The USAID SMART TA project has three main objectives: 1. Deliver quality HIV services within the Continuum of Prevention to Care (CoPC) - SMART TA will develop efficient and cost-effective core and supplementary packages of services that can be replicated in medium and low resourced provinces and ensure quality across implementing programs while transitioning ownership of key services to GVN. 2. Transition financial, administrative and technical ownership of CoPC services -SMART TA will incrementally transition a minimum of 40 percent of financial and technical responsibility for CoPC programs to GVN and local partners over five years, as feasible, based on a systematic assessment of the willingness, capabilities and readiness to sustain such functions and services. This transition will be managed in accordance with an approved timetable. Throughout the life of the agreement, SMART TA will provide the MOH, line ministries, PACs and CSOs with expertise and oversight to guarantee the success of this transition. 3. Strengthen technical capacity and country ownership - SMART TA will strengthen the institutional capacity and develop the human capital of targeted GVN and civil society organizations to manage and sustain the HIV response and support transition strategies. TA, capacity building measures and QI systems will be increasingly led and delivered by local organizations and providers. Effectiveness and Efficiency (“E2”): SMART TA will apply a range of economic evaluative techniques to maximize the cost-effectiveness of core and supplementary services including efficiency evaluation, affordability of scale-up and post-transition sustainability, cost-effectiveness analyses and modeling of costs and outputs or impacts. Background Information Vietnam HIV/AIDS context Vietnam’s HIV epidemic is concentrated among specific populations engaging in high-risk behaviors and in particular geographic locations where those risk groups are clustered. In Vietnam, the national HIV prevalence rates remain relatively low at 0.43 percent for age 15-49. However, rates are higher in urban centers including Ho Chi Minh City (HCMC) and Hai Phong (both over 1 percent). Higher HIV prevalence is concentrated among people who inject drugs (PWID) and their regular sexual partners, male and female sex workers (SW) and their clients, and men who have sex with men (MSM). Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 58 Collectively these groups are referred to as key populations (KPs). Those that spread HIV from one KP group to another, or from one MARP group to a vulnerable group (spouses, infants), are known as bridging populations. Injecting drug use continues to be the main behavior fueling the spread of HIV in Vietnam. The epidemic is particularly acute and worsening in select provinces, with IDU prevalence rates highest in HCMC (48 percent), Hai Phong (48 percent), Dien Bien (56 percent), and Quanh Ninh (56 percent). HIV prevalence is also high among female sex workers (SW), including both street-based and venue-based, averaging 16 percent in hotspot cities such as HCMC, Hanoi, Hai Phong, and Can Tho. Another emerging key and often invisible driver of HIV transmission is unprotected anal sex among MSM with data indicating a growing HIV epidemic among MSM in Hanoi and HCMC. There are additional groups or sub-groups (such as female persons who inject drugs, male sex workers and male clients of sex workers, and the regular sexual partners of these key groups) among whom HIV risk is higher than in the general population and responses are limited. Overlapping risk practices amplify HIV transmission risks for SW and MSM who also inject drugs; for example, infection rates of over 40 percent have been found among injecting FSW. While the major mode of HIV transmission in Vietnam is through the sharing of contaminated needles and other equipment among PWID, HIV transmission through heterosexual intercourse is increasing due to the spread of HIV from MARPs, including infected PWID and clients of sex workers, to their spouses or other regular partners. Although men accounted for 73.2 percent of all reported cases of HIV in 2009, the proportion of female PLHIV appears to be increasing, which has implications for service design. Response by the Government of Vietnam The GVN has taken important steps to deliver an effective response to the HIV epidemic in Vietnam. The Vietnam Administration for Preventive Medicine and HIV/AIDS Control, established in 2003, and the Vietnam Administration for HIV/AIDS Control (VAAC), established in 2005, coordinate the National Strategy on HIV/AIDS Prevention and Control in Vietnam, which was approved in 2004 and includes a vision to 2020. The overall goals of the current national HIV/AIDS strategy are to control HIV prevalence among the general population to below 0.3 percent by 2010, with no further increase after 2010; and to reduce the adverse impacts of HIV on socio-economic development. The national strategy outlines a three-pronged approach: 1) prevention of HIV transmission among MARPs and the general population; 2) care and treatment for people living with HIV; and 3) improvement in management and monitoring of the systems through which necessary services are delivered. An increased emphasis on prevention programs and increased availability of treatment and testing services in 2008 and 2009, targeting both the general population and KP, created a rapid increase in the number of people that know their HIV status and are accessing HIV prevention, care, and support services. Prevention programs implemented among KP included outreach, behavior change communication, condom promotion and distribution, needle and syringe programs and a national pilot methadone maintenance therapy (MMT) program for PWID. Coverage for HIV testing and counseling has been substantially increased over the past five years, through static and mobile VCT services as well as provider-initiated testing and counseling (PITC) in medical settings where people with elevated risk factors for HIV are typically seen (such as TB and STI screening and treatment clinics). Vietnam has been making tremendous efforts to increase the availability of HIV care and treatment services for PLHIV: from an initial 50 patients in 2003, 43,000 patients are now receiving anti-retroviral treatment (ART). Continued achievement will be derived from a synergy of the GVN ministries and civil society participation, including the critical involvement of PLHIV and KP groups. Affordability and sustainability will be vital challenges as the GVN takes on an increasing responsibility for financing treatment and care services. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 59 International support for national response With the arrival of PEPFAR in Vietnam, HIV/AIDS domestic and international expenditures increased rapidly, more than doubling between 2006 and 2008 from approximately $50 million to $108.7 million. Prevention, care and treatment programs receive the majority of the funding. Despite these increases in spending over time, the share of funds contributed by the national and local governments has remained the same since 2006, at approximately $8 million each year Vietnam’s primary donor for HIV/AIDS programs is the United States Government, through PEPFAR, which comprises over 85 percent of the total HIV/AIDS funding. The Global Fund for AIDS, TB and Malaria (GFATM), Asian Development Bank (ADB), United Kingdom’s Department for International Development (DfID), World Bank (WB), AusAid, and the Clinton Foundation are key donors supporting Vietnam’s HIV/AIDS response. The joint WB/DfID program will end in December 2012. The GFATM has disbursed a total of $27.5 million in funds since 2003 to support HIV programs. As Vietnam has recently achieved lower middle income country status (defined by the World Bank as GNI per capita of $996), donors are generally shifting their priorities away from HIV/AIDS in Vietnam. However, in 2010, the GFATM approved a $23.2 million Round 8 grant and a $104 million Round 9 grant designed to increase access to HIV/AIDS prevention, treatment and care and support and strengthen government and civil society capacity to administer these programs. USAID Vietnam Country Development Cooperative Strategy In January 2014, USAID Vietnam announced it 5 year Country Development Cooperative Strategy (CDCS) with the goal of Vietnam’s continued transformation into a responsible, more inclusive partner. The Goal Statement is informed by Vietnam’s growing partnership with the United States in key areas such as security and economic cooperation, while acknowledging the challenges the country faces to be a responsible international player due to limitations in its system of governance that limit participation and also overlook opportunities to be more inclusive of vulnerable populations who have not benefitted from economic growth. The goal also recognizes that Vietnam is in a period of great change and transformation, having achieved Middle-Income Country (MIC) status last year. The international community expects Vietnam to take on a greater share of the responsibility and financing for socio-economic development and continue the process of reform. Within the CDCS, health sector was included as one of its development objective “Capacity strengthened to protect and improve health and well-being”. Strengthening host country capacity to provide for human health and wellbeing is integral to the USG‘s efforts in supporting Vietnam’s continued transformation into a responsible, more inclusive partner and ensure sustainability of development interventions. It further is consistent with the Paris Declaration on Aid Effectiveness, signed by more than 100 bilateral donors and developing countries, which states that the: “capacity to plan, manage , implement and account for results … is critical for achieving development objectives.” Vietnam needs develop the capacity of its people and institutions to respond to a wide range of development challenges, boost country ownership, and rely less on external resources to provide temporary, unsustainable solutions. Summary of the SMART TA project scope: The SMART TA project is designed to contribute directly to the targets identified in the National Strategy on HIV/AIDS Prevention and Control in Vietnam and the Partnership Framework between the Government of the United States of America and the Government of the Socialist Republic of Vietnam for HIV/AIDS Prevention and Control. SMART TA builds on FHI 360 Vietnam’s extensive experience in responding to the HIV epidemic, its proven strengths in capacity building, and the collective SMART TA team’s mature relationships with GVN and civil society partners. In collaboration with GVN, donors, civil society organizations (CSOs) and other stakeholders, SMART TA will achieve the following key results for each objective by Year 5: Objective 1: Deliver quality HIV services within the CoPC Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 60  Operationalize sustainable Continuum of prevention to care (CoPC) models for medium and low resourced provinces, with innovative, efficient, evidence-based approaches and key quality improvement (QI)/technical capacity building assistance provided by local institutions.  Support direct service provision, prior to transitioning USAID-financially supported implementation through SMART TA, for the following beneficiaries: o 32,338 female sex workers (FSWs), 5,000 male clients, 30,740 people who inject drugs (IDUs) and 17,751 men who have sex with men (MSM) reached with HIV prevention services in targeted PEPFAR provinces; o 5,035 IDUs received MMT across 20 sites, 5 of which will be fully integrated with HIV care and treatment services; o 39,120 PLHIV and family members received umbrella care, including 19,560 adults and children living with HIV enrolled in HIV care and treatment services across 33 CoPC sites, of which 16,300 received antiretroviral therapy (ART).  Quality-assured GVN- and partner-owned and operated services (financed by GVN and supported with technical assistance and QI by SMART TA) estimated to reach the following beneficiaries: o 114,400 MARPs (IDUs, FSWs and clients, MSM) reached with individual and/or small group-level prevention interventions; o 56,500 individuals received HTC and received their test results; o 18,700 adult and pediatric PLHIV received a minimum of one clinical service. Objective 2: Transition financial, administrative and technical ownership of CoPC services to the GVN and other stakeholders SMART TA will assist the GVN to transition core and supplementary packages of HIV prevention and care services in focus provinces and beyond. SMART TA will (a) assess the capacity of the GVN and CSOs to implement individualized CoPC interventions for each province and develop a national five-year capacity￾building plan; (b) develop cost-effective CoPC models and service packages that can be replicated using local resources; (c) integrate services that maximize existing resources and meet clients’ needs; (d) strengthen national, provincial and district CoPC referral networks; (e) strengthen data use for program planning and revision; and (f) ensure quality across implementing sites and implementing agencies. The transition of financial, administrative and technical responsibilities for the implementation of HIV CoPC programs supported by SMART TA will require national and provincial consensus-building, capacity assessment, standardization of models and service packages, development of individual provincial transition plans, technical support, and ongoing monitoring and quality assurance/improvement. SMART TA will work USAID, the Ministry of Health (MOH) and the Vietnam Administration for HIV/AIDS Control (VAAC), the Ministry of Labor, Invalids and Social Affairs (MOLISA), the Ministry of Planning and Investment (MPI), the Ministry of Finance (MF), Provincial AIDS Centers (PACs), CSOs, local community-based lead agency, PEPFAR and other donors to transition of minimum 40 percent of the current SMART TA implementation portfolio (financial, administrative and technical functions and direct support of CoPC programs and services ) to the GVN and relevant CSOs by Year 5. SMART TA will provide critical TA to GVN as needed throughout the process. SMART TA will work with the GVN at the national and provincial levels to undertake a phased transition process that includes pre-transition, transition, and post-transition phases. Objective 3: Strengthen technical capacity and country ownership to sustain quality HIV and AIDS services SMART TA will work with GVN, Pathways, PEPFAR and other key stakeholders to employ a five-year capacity building transition plan at the national and provincial levels. SMART TA will promote sustainable country ownership of HIV TA and capacity building through three major strategies:  Support GVN and CSOs with TA and capacity building to plan, implement, monitor and evaluate a coordinated and sustainable HIV response from national to district levels. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 61  Develop CoPC learning/demonstration programs/sites that provide skills-based training opportunities, observation of best practices, and areas for operational research.  Support CSOs to expand civil society engagement in the HIV response. STATEMENT OF WORK: The purpose of this evaluation is to hire an external evaluation team to conduct a mid-term evaluation of USAID’s Sustainable Management of the HIV/AIDS Response and Transition to Technical Assistance (SMART TA) Project Vietnam (October 2011 – September 2016). This mid-evaluation will primarily determine to what extent the USAID-funded program is meeting the stated objectives articulated in the Cooperative Agreement and related Modifications, the Performance Management Plan, and provide an overview of the major accomplishments, performance accountability, and implementation weaknesses, challenges or lessons learned to improve project performance and making recommendation for future design of USAID HIV/AIDS project.  To evaluate the progress made by SMART TA toward intended results and change as described in the Performance Management Plan (PMP) and describe the context and what other possible factors (such as service delivery programs by other donors or actors, advocacy and system strengthening by a variety of actors, etc.) that are also contributing to the observed changes.  To determine whether the activities, technical mix of interventions (as specified in SMART TA annual workplan and its M&E) are appropriate to achieve the intended results and/or change (including planned outputs, outcomes and impact) within the five-year performance period and to recommend modifications to project activities or priorities, as necessary, to address implementation issues, apply lessons learned, or capitalize on new opportunities, especially taking into account any weaknesses/deficiencies in program design.  To determine whether achievements made through SMART TA are sustainable beyond the life of project and, if not, recommend techniques and/or processes that would ensure sustainability of the gains made. The evaluation will thus provide USAID/Vietnam, the Government of Vietnam, and other in-country stakeholders with objective information on what has been achieved to date, what is working and why, as well as what is not working. The information will inform decisions to make appropriate modifications. Evaluation Questions:  Performance and implementation: o To what extent are the project’s activities, techniques, and processes employed by SMART TA contributing to the intended results of reducing acquisition and transmission of HIV with focus on MARPs as specified in its PMP? o To what extent are the project’s activities, techniques, and processes employed by SMART TA contributing to the intended results of reducing morbidity and mortality of PLHIV and improving quality of life as specified in its PMP? o To what extent are the project’s activities, techniques, and processes employed by SMART TA contributing to the intended results of providing targeted support for the generation and use of HIV-related strategic information as specified in its PMP? o To what extent are the project’s activities, techniques, and processes employed by SMART TA contributing to the changes at outcomes and/or impact level? o Do the project objectives and annual workplan contain the correct and appropriate pilots, innovates, technical interventions and directions to achieve the intended project results Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 62 (including planned outputs, outcomes and impact), and whether the right populations are being reached in the right place at the right time? o What challenges and lessons can be drawn for improving the implementation in the next 2 years? o What are key recommendations for better designing future project by USAID Vietnam and GVN?  Transition: o To what extent are the project’s activities, techniques, and processes employed by SMART TA contributing to the intended results of transitioning financial, human resource, administration and technical ownership of HIV/AIDS services to the GVN and other stakeholders (including pre-transition preparation, transition, and post-transition support)? o How are the readiness, acceptability and receptiveness of GVN and other stakeholders to the services transited by SMART TA? o What are key challenges and barriers as well as lessons can be drawn for better implementation of transitioning USAID supported services to GVN? o What recommendations can be provided?  Capacity building and Sustainability: o To what extend the activities, techniques, and processes employed by SMART TA contributing to the intended results of strengthen local technical capacity on HIV/AIDS services? o To what extent have activities designed for long term sustainability been institutionalized? o How sustainable are the programs/program components? o How replicable, adaptable/adoptable are the programs/program components? o How can local and national ownership and future commitment to continued implementation of good practices/lessons learned be enhanced?  Management and coordination: o How can the project design, management, and implementation become more efficient, effective and relevant toward achieving project objectives in connection with the changing contextual dynamics? o How effective has the collaboration/coordination within project components and with other stakeholders been in maximizing efforts and achieving greater results? o How do the programs integrate gender equality and female empowerment in strategic planning, programming, project design and implementation; and monitoring and evaluation? TEAM COMPOSITION: An evaluation team with a team leader shall carry out the evaluation. Additionally, local support will be hired as necessary for data collection/analysis process, translation and logistics purposes. a. (Key personnel) Team Leader must be an international evaluation specialist, a full-time position for the duration of the Evaluation. This person must serve as the primary point of contact between USAID and the Evaluation Team. S/he must be a senior consultant with experience in leading and conducting USAID health program evaluations (at least two external performance evaluations). S/he must also be an independent consultant and have an MPH or related post graduate degree in public health/epidemiology. S/he must have at least 10 years senior level experience working in health in a developing country, experience with HIV/AIDS and PEPFAR is desirable. S/he must have experience in conducting qualitative and quantitative evaluations/assessments. Excellent oral and written skills are required. The Team Leader must also have experience in leading evaluation teams and preparing high quality documents. S/he should also have a good understanding of USAID and PEPFAR funded project implementation, administration, financing, and management procedures. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 63 The Team Leader shall take specific responsibility for assessing and analyzing the Program’s progress toward quantitative and qualitative targets, factors for such performance, benefits /impact of the strategies. S/he will provide leadership for the team, finalize the evaluation design, coordinate activities, arrange meetings, consolidate individual input from team members, and coordinate the process of assembling the final findings and recommendations. S/he shall also lead the preparation and presentation of the key evaluation findings and recommendations to the USAID/Vietnam team and key partners. b. (Key personnel) International Technical Specialist: must have Medicine/Public Health/Epidemiology degree and at least 7-10 years of experience in management of, or consulting on HIV/AIDS. Knowledge on PEPFAR and USAID programs and context is desirable. He/she must have a proven background and experience in HIV/AIDS programs and a strong understanding of the challenges facing such programs in Vietnam or Southeast Asia. He/she must also have a good understanding of the relevant national and other donors programs in HIV/AIDS prevention and control. He/she must have experience in program evaluation and knowledge in conducting surveys, key informant interviews and focus groups. Strong English language and writing skills required. The Specialist shall be responsible for assessing the ability of the project to achieve outcomes in the HIV/AIDS/communication/policy components. S/he shall document key lessons learned and provide recommendations for modifications in approach, results, or activities. c. (Key personnel) Two (2) National Technical Specialists: must have Medicine/Public Health/Epidemiology degree and at least 5-10 years of experience in management of, or consulting on HIV/AIDS programs. They must have a proven background and experience in doing HIV/AIDS and/or health program evaluations, and a strong understanding of the challenges facing such programs in Vietnam or Southeast Asia. They must also have a good understanding of the relevant national & other donors programs in HIV/AIDS prevention and control. The Specialist shall be responsible for assessing the ability of the project to achieve outcomes in the HIV/AIDS/communication/policy components. S/he shall document key lessons learned and provide recommendations for modifications in approach, results, or activities. d. Interpreter/Logistics Coordinator: The Interpreter/Logistics Coordinator must be a local staff member for handling the travel related logistics and providing administrative support to the technical team members. The Logistics Coordinator shall also be responsible for setting up meetings with USAID and stakeholders. Required qualifications include:  Demonstrated ability to be resourceful and to successfully execute complex logistical coordination; ability to multi-task, work well in stressful environments and perform tasks independently with minimal supervision.  Capacity for effective time management and flexibility.  Must be able to interact effectively with a broad range of internal and external partners, including international organizations, host country government officials, and NGOs counterparts.  Must be fluent in both English and Vietnamese.  Proven ability to communicate clearly, concisely and effectively both orally and in writing. The Evaluation Team may include/be accompanied by staff members from USAID/Vietnam and/or USAID/Washington, as appropriate, to observe interviews and field visits. A list of interviewees and key stakeholders will be provided by USAID prior to the assignment’s inception. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 64 METHODOLOGY & DATA SOURCES The evaluation team is responsible for developing an appropriate methodology that responds to the evaluation tasks and answers the evaluation questions above. USAID/Vietnam expects that both quantitative and qualitative methodologies will be employed, including (but not limited to) team planning discussions and meeting(s), a desk review, key informant interviews, focus group discussion and site visits. The exact number of interviews and site visits will be finalized in collaboration with the evaluation team prior to the visit. The following are anticipated elements of the methodology: 1. Document and literature review: The evaluation team will conduct a comprehensive literature review of pertinent documents including studies and assessments regarding the HIV/AIDS service delivery in Vietnam, the GVN strategies and plans on HIV/AIDS, and USAID and program documents, including but not limited to: - Vietnam government policies, strategies and plans, including but not limited to the Vietnam National Strategy on HIV/AIDS till 2015 and vision to 2020, series of decisions related to HIV/AIDS prevention, care & treatment, M&E from MOH, MOLISA, OOG, etc. - Vietnam PEPFAR Partnership Framework. - SMART TA Cooperative Agreement, Modifications, annual workplan and sub-contracts. - SMART TA performance reports, performance monitoring plan (PMP) and reports, assessments and evaluations. - SMART TA research, survey, assessment reports and presentations - Portfolio review templates - Any other relevant documents 2. Key informant interviews: The Contractor shall, where appropriate, conduct qualitative, in-depth interviews with key stakeholders and partners. The Contractor shall conduct face-to-face interviews with informants. When it is not possible to meet with stakeholders in person, telephone interviews shall be conducted. Key informants include but not limited to the following: - SMART TA program staff (Chief of Party, Deputy Chief of Party, Technical Directors, others); - Key stakeholders at national level (MOH, VAAC, MOLISA, MoF, National Assembly, OOG, UN agencies) - Key stakeholders at provincial level (People committee, DoH, PAC) - USAID staff (Contracting Officer’s Representative (COR) for SMART TA) and any other staff) - USG agencies within PEPFAR Vietnam; Centers for Disease Control and Prevention (CDC); DoD; SAMSHA - Other implementing partners (local NGOs, INGO) 3. Review of Routine Site-level Data 4. Group Interviews / Focus group discussion: The Contractor shall conduct group interviews or group discussion with sub-populations of interest to the SMART TA program. Such populations include SMART TA program staff from the fields, users of services provided by SMART TA, PACs staff who working with SMART TA project in the field. 5. Mini-surveys: The Contractor shall conduct mini surveys among sub-populations of interest to the SMART TA program. Such populations include SMART TA program staff from the fields, users of services provided by SMART TA. (Please describe rationale and methodology - how do each of these methods (mini surveys, etc.) link back to the evaluation questions?) Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 65 6. Direct site visits and observations: The Contractor shall gather information and assessing data from a cross-section of the geographical areas in which the programmatic activities are being implemented. The following sites are suggested: - SMART TA offices situated in Hanoi and HCMC; - SMART TA implementation platforms located in the nine PEPFAR provinces of Vietnam; - SMART TA services delivery sites; - Ministry of Health, Ministry of Finance, MOLISA headquarters located in Hanoi; - Provincial Health Departments and Provincial AIDS Centers located in provinces; - Other U.S Government-supported and non-U.S Government-supported organizations implementing similar activities; - Communities benefiting from SMART TA services 7. Review of GIS/MFL Data (to gain a picture of program coverage and gaps in service delivery). 8. Evaluation Sampling Methodology: The Contractor shall describe sampling method/s and sample frame to be used in determining the populations of study for this evaluation. The Contractor shall be required to  Describe data capture tools,  Types of data to be collected, and  How data will be managed to generate the required report/s. The Evaluation Team may be accompanied by staff members from USAID/Vietnam, as appropriate, to observe interviews and field visits. A list of interviewees and key stakeholders will be provided by USAID/Vietnam prior to the assignment’s inception. 9. Data Quality Standards: The evaluation team shall ensure that the data they collect will clearly and adequately represent answers to the evaluation questions, sufficiently precise to present a fair picture of performance, and at an appropriate level of detail. All data should be disaggregated by sex when possible. 10. Data Limitations: Having the evaluation done of the comprehensive and large scale HIV/AIDS project like SMART TA within a relatively short timeline may jeopardize the required level of detail that is expected to give a clear picture of the program performance. The selection of interviewees, participants for group discussion, sites for visit may introduce some potential selection bias. Although it is strongly recommended to have representative samples for the evaluation, this objective is challenge and difficult for field work due to limited number of time and human resource of evaluation team. In addition, the wide coverage of project activities (both national and provincial levels) and big range of activities on the field will affect the sampling procedures. It is anticipated that some interviews/discussions may be conducted through translators to the international evaluation teams. As a result, some differences of the language might lead to not capturing the full intent or meaning offered by the interviewees. It is also anticipated that some interviews/discussions may be conducted in the presence of one or more outside observers, including project and USAID staff. As a result, the interview responses might be affected by the presence of these observers. These potential limitations may introduce information bias. USAID expects that all threats to validity be discussed and documented in the evaluation planning stage, including what will be done to minimize threats to validity, and detailed in the final report. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 66 11. Data Analysis Prior to the start of data collection, the evaluation team will develop and present, for USAID/Vietnam review and approval, a data analysis plan that details how focus group interviews will be transcribed and analyzed, how the qualitative data from the focus group discussions and in-depth interviews with the key informants and other stakeholders will be integrated with quantitative data from the different related documents to reach conclusions about the effectiveness and efficiency of the SMART TA project. The Mission expects the evaluation team to present strong quantitative and qualitative analysis, within data limitations, that clearly addresses key issues found in the research questions. The Mission anticipates that the Evaluation Team will provide a more detailed explanation of the proposed methodology for carrying out the work. Data will be disaggregated, where possible, by gender to identify how program inputs are benefiting disadvantaged and advantaged groups. DELIVERABLES 1. Team Planning Meeting: A Team Planning Meeting (TPM) will be held in Vietnam at the outset of the evaluation. This meeting will allow USAID/Vietnam to discuss the purpose, expectations, and agenda of the assignment with the Evaluation Team. In addition, the team will:  Clarify team members’ roles and responsibilities  Review and develop final evaluation questions  Review and finalize the assignment timeline and share with USAID/Vietnam  Present and discuss data collection methods, instruments, tools and guidelines  Review and clarify any logistical and administrative procedures for the assignment. 2. Work Plan: During the Team Planning Meeting, the Team will prepare a detailed work plan which will include the methodologies to be used in the evaluation, timeline, and detailed Gantt chart. The work plan will be submitted to the USAID/VN/OH and USAID Evaluation Manager for approval no later than the 4th day of work. 3. Methodology Plan: A written methodology and data analysis plan (evaluation design, data analysis steps and detail, and operational work plan) will be prepared during the team planning meeting and discussed with USAID prior to implementation. 4. List of Interviewees and Schedule: USAID will provide the Evaluation Team with a stakeholder analysis that includes an initial list of interviewees, from which the Evaluation Team can work to create a more comprehensive list. Prior to starting data collection, the Evaluation Team shall provide USAID with a list of interviewees and a schedule for conducting the interviews. The Evaluation Team shall continue to share updated lists of interviewees and schedules as meetings/interviews take place and informants are added to/deleted from the schedule. 5. Data collection tools: Prior to starting fieldwork, the Evaluation Team shall share the data collection tools with the USAID Evaluation Program Manager for review, feedback and/or discussion and approval. 6. In-briefing and Mid-term brief with USAID: The Evaluation Team is expected to schedule and facilitate an in-briefing and mid-term briefing with USAID. At the in-brief, the Evaluation Team should have the list of interviewees and schedule prepared, along with the detailed Gantt chart that maps out the evaluation through the report drafting, feedback and final submission periods. At the mid-term brief, the Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 67 Evaluation Team shall provide USAID with a comprehensive status update on progress, challenges, and changes in scheduling/timeline. 7. Discussion of Preliminary Draft Evaluation Report: The Evaluation Team shall submit a preliminary draft of the report to the USAID Evaluation Manager, who shall provide preliminary comments prior to final Mission debriefing. This will facilitate preparation of a more final draft report that must be left with the Mission upon the evaluation team’s departure. 8. Debriefing with USAID: The team shall present the major findings of the evaluation to USAID/Vietnam through a PowerPoint presentation after submission of the draft report and before the team’s departure from country. The debriefing shall include a discussion of achievements and issues as well as recommendations for future activities designs and implementation. The team shall consider USAID/Vietnam comments and revise the draft report accordingly, as appropriate. 9. Debriefing with Partners: The team shall present the major finding of the evaluation to USAID partners (as appropriate and as defined by USAID) through a PowerPoint presentation prior to the team’s departure from country. The debriefing shall include a discussion of achievements and activities only, with no recommendations for future program. The team shall consider partner comments and revise the draft report accordingly, as appropriate. 10. Draft evaluation report: A draft report of the findings and recommendations shall be submitted to the USAID Evaluation Program Manager prior to the Team’s departure from Vietnam. The written report should clearly describe findings and conclusions. Recommendations for future programming must be addressed in a separate internal memo. USAID will provide written comments on the draft report within 10 working days of receiving the document. 11. Expanded Executive Summary: The team shall submit an expanded executive summary to accompany the final report that shall include a background summary on the evaluation purpose and methodology, and an overview of the main data points, findings, and conclusions. The expanded executive summary should be easy to read for wide distribution to local audiences and the partners are encouraged to look for creative presentation styles, formatting and means of dissemination. The expanded executive summary shall be submitted in English and Vietnam, in hard copy (50 copies) and electronically. The report must be disseminated within USAID and to stakeholders according to the dissemination plan. 12. Data Sets: All data instruments, data sets, presentations, meeting notes and final report for this evaluation must be presented to USAID on three (3) flash drives to the Evaluation Program Manager. All data on the flash drive must be in an unlocked, editable format. 13. Final Report: The Evaluation Team shall submit a final report that incorporates responses to Mission comments and suggestions no later than five working days after USAID/Vietnam provides written comments on the Team’s draft evaluation report (see above). This report should not exceed 30 pages in length (not including appendices, lists of contacts, etc.). The format shall include an executive summary, table of contents, glossary, methodology, findings, and conclusions. The report shall be submitted in English, electronically, and then disseminated within USAID/Vietnam for final approval. The report shall be disseminated within USAID and to stakeholders according to the dissemination plan developed by USAID. The final report should be a comprehensive analytical evidence-based evaluation report:  Detail and describe the progress made by SMART TA toward intended results and describe the context and what other possible factors (such as service delivery programs by other donors or actors, advocacy and system strengthening by a variety of actors, etc.) are also contributing to the observed changes.  Describe the effects, constraints, and lessons learned from USAID SMART TA project activities, innovates, pilots, technical approaches during the implementation of the project. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 68  Review current PEPFAR, USAID-funded program’ goals and objectives and their applicability in the context of GVN and other stakeholder objectives and activities, HIV/AIDS epidemiology in Vietnam, and the political context within Vietnam.  Evaluate level of coordination and collaboration between SMART TA project and other USAID funded HIV/AIDS projects, GVN, and other stakeholders.  Evaluate level of sustainability/replication/adaptation of SMART TA activities.  Provide recommendations, lessons, and modifications to project activities or priorities, as necessary, to address implementation issues, apply lessons learned, or capitalize on new opportunities, especially taking into account any weaknesses/deficiencies in program design.  Determine whether achievements made through SMART TA are sustainable beyond the life of project and, if not, recommend techniques and/or processes that would ensure sustainability of the gains made. The report shall follow USAID branding procedures. An acceptable report shall meet the following requirements as per USAID policy (please see: the USAID Evaluation Policy):  The evaluation report should represent a thoughtful, well-researched and well organized effort to objectively evaluate what worked in the project, what did not and why.  The evaluation report should address all evaluation questions included in the scope of work.  The evaluation report should include the scope of work as an Annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline shall be agreed upon in writing by the USAID Mission.  Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex to the final report.  Evaluation findings will assess outcomes and impacts using gender disaggregated data.  Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.).  Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions.  Findings should be specific, concise and supported by strong quantitative or qualitative evidence.  Sources of information need to be properly identified and listed in an Annex, including a list of all individuals interviewed.  Recommendations need to be supported by a specific set of findings. Recommendations should be action-oriented, practical and specific, with defined responsibility for the action. The annexes to the report shall include:  The Evaluation Scope of Work  Any “statements of differences” regarding significant unresolved difference of opinion by funders, implementers, and/or members of the evaluation team  All tools used in conducting the evaluation, such as questionnaires, checklists, survey instruments, and discussion guides  Sources of information, properly identified and listed  Disclosure of conflicts of interest forms for all evaluation team members, either attesting to a lack of conflict of interest or describing existing conflict of interest. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 69 All reports that mentioned above belong to USAID, not to the consultants or contractor, and use of any material in the report by the contractor is expressly prohibited. USAID will be responsible for dissemination of the final report. SCHEDULE The Contractor must plan for the evaluation to begin o/a early September 2014 and last approximately 12 weeks, of which 9-10 weeks will be in Vietnam. Additional time may be required for the Contractor to finalize the final report (if necessary). Below is an illustrative schedule: ACTIONS/DELIVERABLES– estimated LOEs DUE BY (o/a) Preparation & Review Literature – 7 days Prior to arrival Methodology & Work Plans (Draft) – 3 days Prior to arrival Begin in-country evaluation – Team Planning Meeting (TPM) 2 weeks after award Methodology/Work Plans (final) – 3 Days 2 weeks after award Site Visits/Data Collection – 5 weeks 7.5 weeks after award In-Briefing and Mid-term brief with USAID 5 weeks after award Data analysis – 5 days 9 weeks after award Initial Draft of Major Findings and Recommendations – 3.5 days 10 weeks after award Discussion of Preliminary Findings/Recommendations – 0.5 Day 10 weeks after award Debriefing with USAID and SMART TA 10 weeks after award Draft of Evaluation Report – 3 days 11 weeks after award Team Depart 11 weeks after award USAID Vietnam review of Draft Evaluation Report – 1 Week 11 weeks after award Final Report – 1 Week 12 weeks after award The Contractor is responsible for making all logistical arrangements for the team. All expatriate team members will need a valid visa prior to entering Vietnam. That includes full responsibility for, but not limited to, transportation, computers, organizing meetings and interviews, and implementing the proposed evaluation methodology. However, USAID/Vietnam can provide assistance if the Contractor has difficulty in arranging meetings with key stakeholders identified by USAID prior to the initiation of field work. Additionally, USAID can also make recommendations on in-country lodging, if needed. USAID/Vietnam personnel will be made available to the team for consultations regarding sources and technical issues, before and during the evaluation process. Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 70 ANNEX C. EVALUATION KEY QUESTIONS Performance and Implementation:  To what extent are the project’s activities, techniques, and processes employed by SMART TA contributing to the intended results of reducing morbidity and mortality of PLHIV and improving quality of life as specified in its PMP?  To what extent are the project’s activities, techniques, and processes employed by SMART TA contributing to the intended results of providing targeted support for the generation and use of HIV￾related strategic information as specified in its PMP?  To what extent are the project’s activities, techniques, and processes employed by SMART TA contributing to the changes at outcomes and/or impact level?  Do the project objectives and annual work plans contain the correct and appropriate pilots, innovations, technical interventions and directions to achieve the intended project results (including planned outputs, outcomes and impact), and are the right populations being reached in the right place at the right time?  What challenges and lessons can be drawn for improving the implementation in the next 2 years?  What are key recommendations for better designing future project by USAID Vietnam and GVN? Transition to Country Ownership:  To what extent are the project’s activities, techniques, and processes employed by SMART TA contributing to the intended results of transitioning financial, human resource, administration and technical ownership of HIV/AIDS services to the GVN and other stakeholders (including pre-transition preparation, transition, and post-transition support)?  How are the readiness, acceptability and receptiveness of GVN and other stakeholders to the services transited by SMART TA?  What are key challenges and barriers as well as lessons can be drawn for better implementation of transitioning USAID supported services to GVN?  What recommendations can be provided? Capacity Building for Sustainable Transition:  To what extent are the activities, techniques, and processes employed by SMART TA contributing to the intended results of strengthening local technical capacity on HIV/AIDS services?  To what extent have activities designed for long term sustainability been institutionalized?  How sustainable are the programs/program components?  How replicable, adaptable/adoptable are the programs/program components?  How can local and national ownership and future commitment to continued implementation of good practices/lessons learned be enhanced? Management and Coordination for Transition:  How can the project design, management, and implementation become more efficient, effective and relevant toward achieving project objectives in connection with the changing contextual dynamics?  How effective has the collaboration/coordination within project components and with other stakeholders been in maximizing efforts and achieving greater results?  How do the programs integrate gender equality and female empowerment in strategic planning, programming, project design and implementation; and monitoring and evaluation? Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 71 ANNEX D. PERSONS CONTACTED U.S. GOVERNMENT (USG) AGENCIES AND IMPLEMENTERS  Christopher T. Detwiler, The President’s Emergency Plan for AIDS Relief (PEPFAR), Country Coordinator  Gary West, Sustainable Management of the HIV/AIDS Response and Transition to Technical Assistance (SMART TA), Chief of Party  Nguyen Cam Anh, Office of Health - United States Agency for International Development in Vietnam (USAID)  Maria Francisco, Office of Health - United States Agency for International Development in Vietnam  Nguyen Thi Minh Ngoc, United States Agency for International Development in Vietnam  Nguyen Duong, United States Agency for International Development in Vietnam  John Eyres, United States Agency for International Development in Vietnam  Shirl Smith, United States Agency for International Development in Vietnam  Michelle McConnell, Center for Disease Control and Prevention (CDC)– Hanoi, Country Director  Marta Acker, Center for Disease Control and Prevention – Ho Chi Minh City, Associate Director  Kevin Mulvey, Substance Abuse and Mental Health Services Administration (SAMHSA), Drug Abuse Treatment Advisor  Satis Raj Pandey, Family Health International (FHI) 360, Acting Chief of Party  Caroline Francis, Family Health International (FHI) 360, Deputy Chief of Party  Nguyen Cuong Quoc, Family Health International (FHI) 360, Associate Director - Strategic Information  Dinh Thi Minh Thu, Family Health International (FHI) 360, Associate Director - Strategic Communications  Nguyen To Nhu, Family Health International (FHI) 360, Associate Director - Methadone Maintenance Treatment and Care & Treatment  Le Thi Ban, Family Health International (FHI) 360, Associate Director - HIV Prevention  Le Thi Thu Hien, Family Health International (FHI) 360, Associate Director – Technical Assistance Coordination and Health Systems Strengthening  Duong Thi Bich Phuong, Family Health International (FHI) 360, Associate Director - Finance  Phan Thu Phuong, Family Health International (FHI) 360, Technical Manager, Doctor  Pham Le Huy, Family Health International (FHI) 360, Technical Manager, Doctor  Nazzareno Todini, Health Finance Governance (HFG), Chief of Party  Kim Green, Program for Appropriate Technology in Health (PATH)-Healthy Markets, Chief of Party  Vu Ngoc Bao, Program for Appropriate Technology in Health (PATH)- Healthy Markets - Healthy Markets, Deputy Chief of Party  Ha Thuy Huong, Supply Chain Management Systems (SCMS), Country Director  Dao Thi Mai Hoa, Centre for Community Health and Development (COHED), Director  Dinh Thi Yen Nhi, Center for Community Health Research and Development (CCRD), Deputy Chief of Party  Dang Thu Trang, Center for Community Health Research and Development (CCRD), Senior Researcher  Pham Thai Hang, Center for Community Health Research and Development (CCRD), Deputy Director  Nguyen Nguyen Nhu Trang, Centre of Promotion of Quality of Life, Director  Hoang Thu Huong, Centre of Promotion of Quality of Life, Technical specialist  Nguyen Thanh Cong, Centre of Promotion of Quality of Life, SMART TA Project Officer, HCMC Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 72 INTERNATIONAL AGENCY PARTNERS  Kato Masaya. World Health Organization, Lead of HIV/AIDS Team GOVERNMENT OF VIETNAM (GVN) National Assembly  Nguyen Van Tien. Committee on Social Affairs, Vice Chairperson  Mistry of Health  Cao Ngoc Anh, Division on Payment Method Management - Department of Planning and Finance, Head of Division  Nguyen Thi Huong, Division on Payment Method Management - Department of Planning and Finance, Officer  Dang Trung Ha, Division on Payment Method Management - Department of Planning and Finance, Officer  Nguyen Nam Khanh, Division on Payment Method Management - Department of Planning and Finance, Officer  Nguyen Sao Mai, Division on Payment Method Management - Department of Planning and Finance, Officer  Nguyen Van Kham, Department of Insurance, Deputy Head of Department  Nguyen Minh Trung, Department of Insurance, Officer  Nguyen Quoc Toan, Department of Insurance, Officer  Ministry of Health, Vietnam Administration for AIDS Control (VAAC)  Bui Duc Duong, General Director  Pham Thi Thu Huong, Deputy General Director  Cao Thi Hue Chi, Department of Training, Scientific and International Cooperation,  Head of Department  Nguyen Thi Minh Tam, Harm Reduction Department, Head of Department  Do Thi Nhan, Care and Treatment Department, Head of Department  Cao Kim Thoa, Department of Communication & Community Mobilization, Deputy Head of Department  Vo Hai Son, M&E Department, Head of Department Ministry of Planning and Investment  Nguyen Tuong Son, Department of Labor, Culture and Social Affairs, Deputy head of Department  Ministry of Labor, Invalids and Social Affairs  Le Duc Hien, Deputy Director  Dr. Le Van Khanh, Department of Drug Use control Policy, Head of Department  Pham Ngoc Dung, Department of Sex-work control Policy, Head of Department  Vu Thi Hai Hoa, Department of Administration and human resource, Head of Department  Other central agencies  Hien Do, Hanoi School of Public Health, Head of HIV/AIDS Department  Nguyen Anh Tuan, National Institute of Hygiene and Epidemiology (NIHE), Head of HIV/AIDS Department PROVINCIAL AND DISTRICT LEVEL Ha Noi Provincial AIDS Committee /Centre for AIDS Prevention and Control (PAC)  Tran Quoc Tuan, Project Coordinator  Vo Thanh Thuy, M&E Officer Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 73 Out-patient Clinic Hoang Mai District  Nguyen Thi Ha, Director  Lai Minh Chau, Project Manager  Chu Thi Minh, Counselor Methadone Maintenance Therapy (MMT) Dong Da District  Bui Nguyen Hong, Director of MMT Clinic  Le Thi Huong, Counselor Nghe An PAC  Trinh Hung Tien, Vice Director  Luyen Van Trinh, Vice Director  Vu Sy Thang, Division of AIDS Care &Treatment, Head of Division cum Assistant to QTC Project  Nguyen Xuan Huong, Division of Communication, Head of Division cum Head of MMT Life gap Center  Luong Thi Hang, Division of Monitoring, Officer  Dang Thi Thu Binh, Coordinator of FHI 360 in Nghe An  Quy Chau District General Hospital  Dang Tan Minh, Director  Nguyen Manh Dung, Head of ARV/MMT Clinic  Nguyen Thi Thu Hoa, ARV/MMT Clinic, Admin  Lo Thi Tam, ARV/MMT Clinic, Counselor  Lan Van Duy, Care & Treatment Division, Staff  Quy Chau District Health Center  Nguyen Trong Khanh, VCT, Head of VCT  Cao Thi Huyen, Counselor  Vi Nam Dong, Counselor  Phan Xuan Duc, Counselor  Hoang Anh Trung, Counselor  Nguyen Thi Dinh, Deputy Director  Dien Chau District General Hospital  Dau Trong Quy, Out-patient Clinic, Director of the Clinic  Vu Thi Ha, Out-patient Clinic, VTC data controller  Pham Thi Nga, Out-patient Clinic, Methadone medicine distribution staff  Cao Thi Lien, Out-patient Clinic, Counselor/VTC data controller  Nguyen Thi Thu Hien, Out-patient Clinic, Counselor/Blood test technician  Phan Van Hieu, Out-patient Clinic, Public communicator  Pham Quoc Tuong, Out-patient Clinic, Volunteer PAC Hai Phong  Doan Thi Thu, Director  Nguyen Duy Hung, Vice Director  Dao Viet Tuan, Vice Director  Vu Thi Ha, Officer  Nguyen Thi Thanh Thuy, Officer  Nguyen Thi Nga, M&E officer  Hai Phong Department of Health  Pham Thu Xanh, Vice Director Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 74  Do Thi Minh Nguyet, Division of Medical Practice, Officer  Hai Phong Department of Planning & Investment  Hoang Thi Lien, Officer  Le Chan MMT  Dang Thi Nga, Counselor  Nguyen Xuan Hung, Staff  Nguyen Thi Thanh Thuy, M&E Division of Hai Phong PAC cum Methadone medicine distribution staff  Do Thi My Le, Counselor  Pham Thi Thanh Giang, Counselor  Dao Thi Hang Nga, Nurse  Pham Thi Thu Hang, Nurse  Do Thi Ha, Counselor  Hai An District General Hospital  Hoang Van Nhat, Director  Le Thi Thanh Phuong, Vice Director cum Head of Out-patient Clinic  Nguyen Thi Kim Tho, Admin/Data controller  Bui Thi Hong Ha, Counselor/ Blood Test Technician  Bui Thi Ha, Counselor Ho Chi Minh City PAC  Tieu Thi Thu Van, Director  Le Truong Giang, Deputy Director  Nguyen Thi Hue, Division of Preventative Intervention  Le Thanh Tung Nho, Officer  Nguyen Thi Kim Phuong, Officer  Huynh Tan Tien, M&E Officer  Tran Thi Truc Linh, Finance Officer  Luong Quoc Binh, Coordinator of SMART TA Project in HCMC  Nguyen Thi Thu Van, Project officer of SMART TA in HCMC/ Counselor  District 3 General Hospital (Out-patient Clinic)  Nguyen Thi Tuyet Mai, Pharmaceutical specialist  Pham Thi Be Ba, Pharmaceutical specialist  Nguyen Hoang Hai, Admin  Luu Nguyen Khoa, VCT data controller  Ho Thi Bao Yen, VCT data controller/ Counselor  Phan Chi Tin, Blood test technician  Vu Thi Khoan, Head of Department  Le Thi Hong, Deputy Head of Department  Binh Thanh Out-patient Clinic  Le Dien Trung, Division of Community Counselling, Head of Division and the Out-patient Clinic  Bui Thi Thuy Duong, Division of Community Counselling, VCT  Dang Ngoc Phuong, Division of Community Counselling, Counselor on Methadone  Bui Thi Thanh Nga, Staff  Ngo Thanh Binh, Division of Pharmacy, Staff  Tran Thi Thu Thao, MMT Clinic, Staff Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 75  Ngo Thi Anh Dong, Doctor  Nguyen Duy Thai, Vice Director  Center for Community Development Support, District 8  Nguyen Ngoc Thoa, Vice Director  Dang Minh Hien, Vice Director  Dang The He, Director  Pham Thi Trang, Nurse  Nguyen Thi Kim Loan, M&E officer  Dinh Thi Phuong, Nurse  Tran Thi Hong, Counselor  Truong Thanh Binh, Counselor  Nguyen Van Hai, Counselor  Pham Thanh Hieu, Head of Division Dien Bien PAC  Hoang Xuan Chien, Director  Tran Van Tho, Vice Director  Le Thi Luyen, M&E officer  Phin Thi Thuy, Staff  Nguyen Thien Huong, Staff  Nguyen Thi Kim Hoa, Head of Laboratory  Le Thi To Khuyen, Head of Care & Treatment Department  Nguyen Ngoc Hien, Project Coordinator  Dien Bien Provincial Department of Health  Trieu Dinh Thanh, Director  Dien Bien Provincial Department of Finance  Nguyen Van Hoan, Vice Director  Dien Bien Provincial Department of Planning & Investment  Luong Thi Huong, Deputy Head of Division of Planning  Dien Bien Provincial General Hospital - Out-patient Clinic  Tran Duc Nghia, Deputy Head of Department of Project Management  Luong Anh Duong, Coordinator  Hoang Thi Chuong, Director of MMT Clinic  Bui Thi Ut, Data controller  Cao Van Thang, Doctor  Vu Thi Ban, Pharmacist Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 76 ANNEX E. EVALUATION TOOLS Qualitative Interview (QI) for Key Informants Instructions to evaluators: (1) Confidentiality is important to some informants, but not others. Ascertain the individual’s or group’s preferences to be identified by name, or not, and proceed accordingly. (2) If agreed upon, obtain an attendance list showing name and title of key informant(s). (3) Evaluators should ensure that at least the following issues/questions are raised during individual or group discussions, and typed notes using this form should be filed electronically on a weekly basis with the evaluation team leader. PERFORMANCE AND IMPLEMENTATION 1. Please describe the interventions in your programs that help to reduce the acquisition and transmission of HIV. How has the SMART TA project supported those interventions? 2. How has SMART TA assisted you and the offices you work with in generating and using HIV-related data and information to improve use and quality of services? 3. Have you heard about (or are familiar with) SMART TA innovations in the areas of outreach, HIV Counseling and Testing, Methadone Maintenance Therapy, and Care and Treatment? What is your opinion about their usefulness and effectiveness of ____________? TRANSITION 4. To what extent is the GVN ownership and commitment to the transition of HIV/AIDS services in Vietnam evolving? 5. Is there sufficient interagency policy development and coordination to effectively guide the transition of SMART TA services delivery? 6. Is SMART TA providing the range and quality of assistance to build the GVN technical capacities needed to ensure transition from service delivery to technical assistance? Click here to enter text. 7. What other kinds of technical assistance could SMART TA provide to help GVN to achieve its 2020 HIV/AIDS strategy? CAPACITY BUILDING 8. How receptive do you believe local managers and service providers are to the types of program interventions SMART TA has introduced up to now? 9. Which SMART TA program components (list them) are replicable? Can be further institutionalized? 10. Can you provide some examples where the SMART TA project has been directly involved with increasing the technical skills or capacity of local providers or managers? Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 77 11. What approaches do you think SMART TA should take to increase the interest of service providers to adapt good practices? MANAGEMENT AND COORDINATION 12. How are the SMART TA services managed to meet the needs of local recipients? 13. What activities have SMART TA undertaken to improve the coordination of services at the local level? 14. How do you think gender issues have been addressed in SMART TA? ANY OTHER RELEVANT NOTES Qualitative Questionnaire (QQ) for Key Informants Please circle the number in the box that best describes the extent to which you agree or disagree with each of the statements below. On a scale of 1-5, 1 = “Strongly Disagree”, and 5 = “Strongly Agree”. If you are unsure, indicate this by a tick () in the “Don’t Know” box. 1. SMART TA project has introduced activities that help to reduce the acquisition and transmission of HIV. Don’t Know Strongly Disagree Disagree Somewhat Neutral Agree Somewhat Strongly Agree 1 2 3 4 5 2. SMART TA project has assisted my program in generating and using HIV-related data and information to improve use and quality of services. Don’t Know Strongly Disagree Disagree Somewhat Neutral Agree Somewhat Strongly Agree 1 2 3 4 5 3. SMART TA innovations in the areas of outreach, HIV Counseling and Testing, and Care and Treatment are useful and effective. Don’t Know Strongly Disagree Disagree Somewhat Neutral Agree Somewhat Strongly Agree 1 2 3 4 5 4. SMART TA is providing the range and quality of assistance to build the GVN technical capacities needed to ensure transition. Don’t Know Strongly Disagree Disagree Somewhat Neutral Agree Somewhat Strongly Agree 1 2 3 4 5 5. SMART TA project has been directly involved with increasing the technical skills or capacity of local providers or managers. Don’t Know Strongly Disagree Disagree Somewhat Neutral Agree Somewhat Strongly Agree 1 2 3 4 5 Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 78 6. SMART TA services are managed to meet the needs of local recipients. Don’t Know Strongly Disagree Disagree Somewhat Neutral Agree Somewhat Strongly Agree 1 2 3 4 5 7. SMART TA has improved the coordination of services at the local level. Don’t Know Strongly Disagree Disagree Somewhat Neutral Agree Somewhat Strongly Agree 1 2 3 4 5 8. SMART TA has appropriately addressed gender issues. Don’t Know Strongly Disagree Disagree Somewhat Neutral Agree Somewhat Strongly Agree 1 2 3 4 5 Exit Interview Protocol for Target Beneficiaries (Clients) 1. Recruitment sites: 12 service clients will be selected in HCMC and Hai Phong at:  ARV Outpatient Clinics  Methadone Maintenance Therapy Clinics  HIV counseling and testing 2. Selection criteria  Have had experience with the services at least 3 months or more (for MMT and ARV)  Male to Female ratio: 1:1  18 years old or older 3. Recruitment strategy: Service clients will be randomly approached after receiving services at sites and will be enrolled upon the selection criteria. 4. Data and information to collect: 4.1 General information Age Sex Economic condition Educational level Marital status Time of day services received Other notes: 4.2 Service access and use  What services do you often receive from this site?  Why did you choose this site for the services? Have you experienced any problems in accessing and using services?  How satisfied are you with the services you receive in terms of privacy, confidentiality, convenience (open hours, distance, facility arrangement), friendliness of the service providers Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 79  Do you have to pay for the services? Are the service costs affordable to you? Do you bear any additional costs for receiving services?  What do you think about the quality of services?  Do you find any difficulty in continuing the services?  Tell us 3 things you like most at this site?  Tell us 3 things you are dissatisfied with about services at this site?  Do you have any suggestions in improving the services at this site? 4.3 Quality of life  How has the programs (services) influenced your life in terms of physical health, economic, social, and emotional changes? How has the program (services) influenced your family in terms of economic, social, and emotional changes? Are you now feeling confident in joining social activities, working, learning, Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 80 ANNEX F. KEY INFORMANT QUESTIONNAIRE RESULTS Qualitative Questionnaire (QQ) for Key Informants On a scale of 1-5, 1 = “Strongly Disagree”, and 5 = “Strongly Agree”. If you are unsure, indicate this by circling (0) in the “Don’t Know” box. 1. SMART TA project has introduced activities that help to reduce the acquisition and transmission of HIV. (n= 92) 2. SMART TA project has assisted my program in generating and using HIV￾related data and information to improve use and quality of services. (n= 89) 0 2 0 8 90 0 20 40 60 80 100 1 2 3 4 5 % 0 3 4 26 66 0 20 40 60 80 1- Strongly disagree 2 3 4 5- Strongly agree % Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 81 3. SMART TA innovations in the areas of outreach, HIV Counseling and Testing, and Care and Treatment are useful and effective. (n= 90) 4. SMART TA is providing the range and quality of assistance to build the GVN technical capacities needed to ensure transition. (n= 85) 0 0 1 21 78 0 20 40 60 80 100 1- Strongly disagree 2 3 4 5- Strongly agree % 0 0 7 33 60 0 20 40 60 80 100 1- Strongly disagree 2 3 4 5- Strongly agree % Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 82 5. SMART TA project has been directly involved with increasing the technical skills or capacity of local providers or managers. (n= 91) 6. SMART TA services are managed to meet the needs of local recipients. (n= 92) 0 2 3 26 68 0 20 40 60 80 100 1- Strongly disagree 2 3 4 5- Strongly agree % 0 2 5 26 66 0 20 40 60 80 100 1- Strongly disagree 2 3 4 5- Strongly agree % Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 83 7. SMART TA has improved the coordination of services at the local level. (n= 92) 8. SMART TA has appropriately addressed gender issues. (n= 97) 1 4 4 24 66 0 20 40 60 80 100 1- Strongly disagree 2 3 4 5- Strongly agree % 1 7 24 37 30 0 20 40 60 80 100 1- Strongly disagree 2 3 4 5- Strongly agree % Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 84 ANNEX G. EXIT SURVEY RESULTS Objective: To collect candid, impromptu perspectives from recipients of MMT and related HIV services being provided at SMART TA-supported sites. Method: Evaluation team members randomly selected participants at service delivery points including ARV out-patient clinics, MMT clinics and HIV testing facility. Patients or clients were approached and invited for individual interviews after they completely received the services. There were totally 14 participants, of those 7 were female, enrolled in this qualitative data collection. Details on participants were given in table 1. Qualitative questionnaire with open questions, with focuses on client's quality of life and client's satisfaction, were used to guide the interview. Table 1: Clients participating service exit interviews Number of interviews Gender Client sources Dien Chau, Nghe An 2  2 female  2 ARV patients Le Chan, Hai Phong 4  2 male  2 female  1 HTC clients  3 MMT and ARV patients District 3, HCMC 4  2 male  2 female  4 ARV patients District 8, HCMC 4  3 male  1 female  3 ARV and MMT patients  1 ARV patients Limitation: The design is not either scientific qualitative or quantitative research so findings may not be representative or able to use for further inference. Results: 1) Participants were at different age: the youngest was 32 and the oldest was 57 years old. Participants spent different time lengths with the service program (from a month to 6 years, except HTC client who came for the service first time). All participants reported low incomes (from no income to 5 million VND, except one with 20 million). 2) Improvement in quality of life All patients reported that their quality of life have been greatly changed since they started receiving services (except for a HTC client, who was recently found with HIV positive). Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 85 ANNEX H. SMART TA 2013 PMP TARGETS AND RESULTS Key Performance Indicators Year 3 Target Year 3 Result Year 3 Target Year 3 Result DSD DSD TA TA 1. Number of key populations reached with individual and/or small group level preventive interventions that are based on evidence and/or meet the minimum standards required 45,100 30,330 NA NA 2. Number of people who inject drugs (PWID) on medication assisted therapy 4,580 1,941 10,874 12,737 3. Number of individuals who received results of their HIV Testing and Counselling (HTC) services for HIV 60,450 56,038 NA NA 4. Proportion of KPs reached by [EOA] community outreach workers who received HIV test results and post-test counselling 70% 80% NA NA 5. Proportion of KP individuals who received testing results and post-test counselling among HTC clients in province during reporting period 65% 73% NA NA 6. Proportion of newly identified HIV positive cases successfully referred from HTC to OPC 85.0% 97.0% NA NA 7. % HIV positivity rate among KPs 6.0% 5.0% NA NA 8. Number of adults & children newly enrolled on ART 2,500 2,717 450 93 9. Number of adults & children currently receiving ART 16,000 16,458 450 1705 10. Proportion of clients newly registered in last 6 months and tested for CD4 within 15 days of enrolment 75.0% 84.2% NA NA 11. CD4 level when initiating ARV treatment among PLHIV in the last 6 months 28.0% 35.7% NA NA 12. Percentage of adults and children known to be alive and on treatment 12 months after initiation of antiretroviral therapy 90.0% 86.4% 75.0% 84.2% 13. Percentage of PEPFAR-supported ART sites achieving a 85% ART retention rate NA NA NA NA 14. Proportion of PLHIV in HIV clinical care who were screened for TB symptoms at the last clinical visit NA NA NA NA 15. Number/type of priority push TA initiatives 5 5 NA NA 16. Number/ composition of local TA networks 80 81 NA NA 17. Number of provinces with local TA networks 9 9 NA NA 18. Number of TA sites supported by SMART TA NA NA NA NA 19. Proportion of SMART TA-supported provinces with annual TA plans NA NA NA NA 20. Proportion of routine capacity building and M&E events carried out by local TA network members NA NA NA NA 21. Proportion of targeted provinces undertaking cascade analyses of the HIV response and implementing service improvement plans based on these analyses 9 9 NA NA 22. Proportion of targeted SMART TA-supported provinces receiving push TA that adopt the intervention program approaches or integrate new technology NA NA NA NA Green = >110% of target, Blue = within 90-110% of target, Red = <90% of target Vietnam Evaluation, Monitoring and Survey Services | SMART TA Mid-Term Evaluation Report 86