INDONESIA MONITORING & EVALUATION SUPPORT PROJECT MIDTERM EVALUATION OF USAID/INDONESIA LINKAGES PROGRAM NOVEMBER 12, 2017 This publication was produced for review by the United States Agency for International Development. It was prepared by Management Systems International (MSI), A Tetra Tech Company. MIDTERM EVALUATION OF USAID/INDONESIA LINKAGES PROGRAM INDONESIA MONITORING & EVALUATION SUPPORT PROJECT Contracted under AID-497-C-16-00006 Evaluation Team Contributors: Jenne Roberts Ratna Soehoed Maria Au Ngoc Thi Minh Nguyen Ricky Andriansyah DISCLAIMER The authors’ views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States Government. CONTENTS EXECUTIVE SUMMARY...................................................................................................................1 PROJECT BACKGROUND............................................................................................................................................ 1 EVALUATION QUESTIONS, METHOD, AND LIMITATIONS ........................................................................... 2 FINDINGS AND CONCLUSIONS .............................................................................................................................. 2 PROJECT BACKGROUND.............................................................................................................3 EVALUATION PURPOSE ............................................................................................................................................... 5 EVALUATION QUESTIONS, METHOD, LIMITATIONS ........................................................5 LIMITATIONS .................................................................................................................................................................... 6 FINDINGS AND CONCLUSIONS ...............................................................................................7 RESULT AREA 1: INCREASING AVAILABILITY OF COMPREHENSIVE HIV SERVICES, ACROSS THE COPCT FOR KP................................................................................................................................................................ 7 RESULT AREA 2: ENHANCING AND SUSTAINING DEMAND FOR COMPREHENSIVE HIV SERVICES ACROSS THE COPCT AMONG KP.....................................................................................................18 RESULT AREA 3: STRENGTHENING SYSTEMS, ORGANIZATIONAL AND TECHNICAL CAPACITY........................................................................................................................................................................ 25 CONSOLIDATED RECOMMENDATIONS..............................................................................38 ANNEX I: EVALUATION METHODOLOGY AND TIMELINE...........................................44 METHODOLOGY AND APPROACH...................................................................................................................... 44 ANNEX II: DATA COLLECTION SCHEDULE........................................................................53 ANNEX III: DATA COLLECTION INSTRUMENTS ...............................................................60 INTERVIEW GUIDE FOR THE PROVINCIAL AIDS OFFICE .............................................................................60 ANNEX IV: SOURCES OF INFORMATION ............................................................................69 DOCUMENTS REVIEWED .......................................................................................................................................... 69 BIBLIOGRAPHY...............................................................................................................................76 USAID.GOV LINKAGES MIDTERM EVALUATION REPORT | ii ACRONYMS AEM Asian Epidemiological Model AIDS Acquired Immunodeficiency Syndrome ART Antiretroviral Therapy ARV Antiretrovirals (medicines) CBS Ccommunity-Based Ssupporters CHAI Clinton Health Access Initiative CoPTC Continuum of Prevention, Treatment and Care (for HIV) CSO Civil Society Organization DSD Direct Service Delivery EOA Enhanced Outreach Approach FSW Female Sex Workers FY Fiscal Year GFATM Global Fund to Fight AIDS Tuberculosis and Malaria GOI Government of Indonesia GWL-INA Indonesia Gay, Waria, and other MSM Network HBV Hepatitis B Virus HCT HIV Counseling and Testing HIV Human Immunodeficiency Virus IAC Indonesia AIDS Coalition IBBS Integrated Biological and Behavioral Survey IEC/BCC Information Education, Communication/Behavioral Change Communication IPPI Ikatan Perempuan Positif Indonesia (Indonesia Positive Female Organization) IPSD Intervensi Penurunan Stigma dan Diskriminasi (Stigma and discrimination education intervention) KP Key Populations (affected by HIV) KI Key Informants LINKAGES Linkages across the Continuum of HIV Services for Key Populations Affected by HIV Project LKB Layanan Komprehensif Berkelanjutan (Integrated Decentralized Continuum of Care Services) LTFU Lost to follow up MAJU Empowering Access to Justice MoH Ministry of Health MSM Men who Have Sex with Men NAC National AIDS Commission NSP Needle and Syringe Programs NU Nahdlatul Ulama (Indonesia Islam organization) OPSI Organisasi Perubahan Sosial Indonesia PHO Provincial Health Office PEPFAR President’s Emergency Plan for AIDS Relief PITC Provided-Initiated Testing and Counselling PKBI Perkumpulan Keluarga Berencana Indonesia (Indonesian Planned Parenthood Association) PLHIV People Living with HIV PMEP Program Monitoring and Evaluation Plan PWID People Who Inject Drugs REACH Rapidly Expanding Access for Care for HIV SBC Social and Behavior Change Communications SIHA Sistem Informasi HIV dan AIDS (Strategic Information HIV and AIDS) SOP Standard Operational Procedures STI Sexually Transmitted Infections SUFA Strategic Use of Anti-Retroviral medicines (Indonesian strategy to increase access to treatment for HIV and get people diagnosed with HIV onto treatment) TA-SDI Technical Assistance-Service Delivery Improvements TB Tuberculosis TG Transgender UNAIDS Joint United Nations Programme on HIV/AIDS UNICEF United Nations Children's Fund USA United States of America USAID United States Agency for International Development VCT Voluntary Counselling and Testing (for HIV) Waria Male-to-Female Transgender Person WHO World Health Organization YIM Yayasan Intermedika (Intermedika Foundation) 1 | LINKAGES MIDTERM EVALUATION REPORT USAID.GOV EXECUTIVE SUMMARY The USAID/Indonesia Health Technical Office has commissioned a midterm evaluation of its flagship HIV project, LINKAGES. The aims of this midterm evaluation are to promote accountability; assess project progress, outputs and any outcomes to date; and recommend ways to improve project performance across the three key result areas to enhance the impacts of the Government of Indonesia’s (GOI) HIV program. The findings are expected to be used by USAID, partners, and other stakeholders to promote understanding of the project’s strengths, progress and challenges and maximize the contribution LINKAGES makes to the national HIV program. This evaluation recommends ways to continue to maximize the impact of the project. The primary audience for this midterm evaluation is USAID/Indonesia, the in-country partners and GOI stakeholders. In addition, USAID will share the findings from this midterm evaluation with the Implementing Partner. PROJECT BACKGROUND The Linkages Across the Continuum of HIV Services for Key Populations Affected by HIV – or “LINKAGES” – is the United States Agency for International Development’s (USAID) first global project dedicated to mitigating the impact of HIV and AIDS among key populations most at risk of acquiring or transmitting HIV. LINKAGES aims to accelerate the ability of partner governments, civil society organizations serving key populations and the private sector to more effectively plan, deliver and optimize comprehensive, scaled HIV/AIDS prevention, care and treatment services that reduce HIV transmission among key populations and their sexual partners and improve the quality of life for those who are HIV positive. LINKAGES is a global mechanism led by FHI 360, in partnership with Pact, Intra Health International, and the University of North Carolina (UNC) in Chapel Hill, managed by an Agreement Officer Representative (AOR) in USAID/Washington. USAID/Indonesia has bought into the mechanism since 2015 and a local activity manager in Indonesia works closely with the AOR in Washington DC. LINKAGES Indonesia prioritizes programming that aligns with the United States President’s Emergency Plan for AIDS Relief (PEPFAR) 3.0’s five key agendas – impact, efficiency, sustainability, partnership and human rights – to achieve global 90-90-90 goals and an AIDS-free generation. The project also contributes directly to the targets identified in Indonesia’s National AIDS Commission’s (draft) National Strategy and Action Plan (SRAN), 2015-2019 and the Jakarta Fast Track Initiative. In Indonesia, FHI 360 partners with Pact to conduct a range of capacity building and technical assistance activities to reduce HIV transmission among key populations — sex workers, men who have sex with men, transgender women, and people who inject drugs — and to improve their enrollment and retention in care. In Papua, LINKAGES targets priority populations in two districts where HIV prevalence rates indicate there is a generalized epidemic. LINKAGES, congruent with the PEPFAR approach, requires implementers to shift investment to strategic, highest burden areas; to demonstrate progress towards epidemic control; and to support the Government of Indonesia (GOI) to assume greater financial ownership of the HIV response. The total investment over the project lifetime is expected to be $26,513,726. USAID.GOV LINKAGES MIDTERM EVALUATION REPORT | 2 EVALUATION QUESTIONS, METHOD, AND LIMITATIONS To guide this evaluation, USAID has identified the evaluation questions: • What progress has been made by LINKAGES in achieving its goals, objectives, and performance targets? • To what extent have project activities strengthened the capacity of CSO and GOI partners in achieving the goals and targets of Indonesia’s National HIV Program? • To what extent have project activities strengthened the capacity of national and sub-national (province and district) governments and other stakeholders in managing Indonesia’s National HIV Program? • To what extent are LINKAGES methodologies, interventions, and management setting the stage for the future sustainability, replication, adaptation, and adoption of project outputs and outcomes? • How effective has the collaboration and coordination with other internationally funded HIV programs been in maximizing efforts and achieving greater results? The evaluation was designed to comply with the USAID Evaluation Policy (2011 updated 2016) and the PEPFAR Evaluation Standards of Practice V2.0 (2015). The evaluation is consistent with USAID’s definition of a performance evaluation (p 3) and PEPFAR’s definition of a process evaluation (p3). A mixed methods evaluation was undertaken to answer the evaluation questions. Primary qualitative data collection methods included semi structured interviews, field visits and observation at implementation sites, and key informant and small group interviews and large group discussions. (See Annex III for the data collection instruments.) The evaluation team collected data from key informants selected using a purposive sample of respondents from government, a range of partner organizations, civil society organizations, and LINKAGES and USAID staff to gain their perspectives on the program. FINDINGS AND CONCLUSIONS The evaluation finds that after a challenging first two years, LINKAGES is on track to meet the project goals and objectives and is making substantial progress in all 3 result areas. LINKAGES has established a solid presence as a recognized TA provider to CSO and GOI. The project’s approach to innovation, accountability and quality improvement is driving results in a national program still struggling to set ambitious targets and create an enabling environment for a successful public health, rights-based approach. The project is addressing knowledge gaps and skills shortages and strengthening community and health systems and the critical transfer of clients between these systems for greater retention across the cascade. There is ample documentation of a thoughtful and opportunistic approach to analyzing gaps and bottlenecks in the national program, designing, developing, implementing and evaluating fit-for purpose responses, and improving the capacity to use data for decision making and to monitor progress. These and other measures have increased the national program’s ability to reach and test key populations at risk of HIV, and link people living with HIV to treatment and care. Project efforts are addressing the attitudes, skills and confidence of health professionals and the operating rules that limit access to services, human rights advocacy the role of violence in HIV vulnerability and access to services. Stigma and discrimination, fear, low self-worth, and limited health literacy or knowledge of how to reduce the impact of HIV remain huge barriers to demand creation. While there is no agreement on the kind of 3 | LINKAGES MIDTERM EVALUATION REPORT USAID.GOV differentiated approaches that will be most impactful in the national program leaders, the project is steadily increasing investment in customized and differentiated on and off –line social and behavior change interventions. The project recognizes the need to empower individuals and communities to take an active role in knowing their status, improve their health seeking behavior and increase the demand for and uptake of health services. Despite the inhibiting, social, legal and human rights environment, the project has been strengthening the capacity of CSO that are actively supporting networks of KP, including PLHIV. The project activities are measurably increasing management and technical capacity of the CSO. This enables organizations to bring KP together to discuss and build demand for new prevention approaches, such as Voluntary Male Medical Circumcision, treatment as prevention and PrEP. These organizations are raising awareness of the impact of criminalization, poor quality health care, restricted operating hours, inequity and making links with legal advocacy and human rights organizations. These activities represent possibly the most sustainable activities that can be undertaken by any project, as ultimately the strength and cohesion of KP networks, and their ability to demand high quality, accessible and effective services and influence policy is the best indicator of a potential of the GOI to reach the UNAIDS vision of 90-90-90. Demand creation activities in Papua are hampered by the complexity of the operating environment and limited understanding of health decision making and health seeking behavior. The project has been successful in bringing academics, technical experts, KP, service providers and decision makers together to make use of epidemiologic and cost data to improve performance and increase program impact and effectiveness. The project is supporting, facilitating and contributing to coordination and collaboration efforts to strengthen service systems that improve outcomes across the cascade. There are varying levels of commitment to ambitious target setting, and the evaluation recognizes that USAID and LINKAGES play a key role in ensuring managers at every level, across the GOI and CSO services, understand how they can play a role in setting and achieving targets. PROJECT BACKGROUND The Linkages Across the Continuum of HIV Services for Key Populations Affected by HIV – or “LINKAGES” – is the United States Agency for International Development’s (USAID) first global project dedicated to mitigating the impact of HIV and AIDS among key populations most at risk of acquiring or transmitting HIV. LINKAGES aims to accelerate the ability of partner governments, civil society organizations serving key populations (KP) and private sector providers to more effectively plan and implement services that reduce HIV transmission among key populations and their sexual partners and extend the lives of those already living with HIV. Home to the third largest number of people living with HIV (PLHIV) in Asia, Indonesia is critical for AIDS epidemic control in the region. Indonesia has been identified as a priority country and has adopted a Fast￾Track approach, which relies on front-loading investments and initiating efficient and innovative approaches over the next five years in order to reach critical HIV prevention and treatment targets. The five Jakarta municipalities have accordingly developed Fast-Track targets, as almost 17% of the total number of people living with HIV living in Indonesia are thought to be living or working in Jakarta. Despite the commitment to a Fast Track approach, Indonesia has been slow to adopt or scale up the policy changes and practices needed to reach ambitious targets. There is a need for a broad range of Technical Assistance (TA) to support innovation across the Continuum of HIV Services, but with few TA providers, LINKAGES has the potential to play a key role in the national program. USAID.GOV LINKAGES MIDTERM EVALUATION REPORT | 4 LINKAGES is led by FHI 360, in partnership with Pact, IntraHealth International, and the University of North Carolina (UNC) in Chapel Hill. It is a global mechanism managed by an Agreement Officer Representative (AOR) in USAID/Washington. USAID/Indonesia has bought into the mechanism since 2015 and has an activity manager on the ground to manage LINKAGES Indonesia in close collaboration with the AOR in Washington DC. LINKAGES Indonesia prioritizes programming that aligns with PEPFAR 3.0’s five key agendas – impact, efficiency, sustainability, partnership and human rights – to achieve global 90- 90-90 goals and an AIDS-free generation. The project also contributes directly to the targets identified in Indonesia’s National AIDS Commission’s (draft) National Strategy and Action Plan (SRAN), 2015-2019 and Jakarta Fast Track Initiative targets. In Indonesia, FHI 360 partners with Pact, to conduct a range of activities to increase capacity to reduce HIV transmission among key populations — sex workers (SW), men who have sex with men (MSM), transgender women (TG), and people who inject drugs (PWID) — and to improve their enrollment and retention in care. In Papua Province, LINKAGES targets priority populations in three districts where HIV prevalence rates indicate there is a generalised epidemic. Congruent with the PEPFAR approach, LINKAGES assist implementers to shift investments to strategic, highest burden areas; to demonstrate progress towards epidemic control; and to support the GOI to assume greater financial ownership of the HIV response. LINKAGES Indonesia interventions aim to support the national response to HIV and enhance the success of the Government of Indonesia’s (GOI) program by expanding the reach to key populations most at risk of acquiring or transmitting HIV, by promoting routine HIV testing and counseling, and by actively enrolling those with HIV into care and support interventions that enable them to remain in care. The program is starting from a low base in Indonesia, where approximately 90% of people living with HIV (PLHIV) are not enrolled or retained in treatment programs that succeed in suppressing their viral load. High lost to follow￾up (LTFU) rates are very common across the continuum in many settings, contributing to a significant and preventable burden of HIV morbidity and mortality. The project also provides technical assistance (TA) to Indonesian counterparts to enable them to use, develop, adapt and scale up evidence-based approaches to service provision; to help key populations mobilize and advocate for changes in laws and the conduct of police, health care workers and policymakers; and to work with governments to make programs sustainable for the long term. The project aims to achieve the following results: • Result 1: Increased availability of comprehensive prevention, care, and treatment services, including reliable coverage across the continuum of care for key populations. • Result 2: Demand for comprehensive prevention, care, and treatment services among key populations enhanced and sustained. • Result 3: Strengthened systems for planning, monitoring, evaluating, and assuring the quality of programs for key populations. The key elements of LINKAGES strategic and technical approach are: • Identifying key populations and locales and comprehensively assessing risk, • Diagnosing 'leaks' and revealing access barriers within the HIV services cascade, • Scaling up 'what works' while innovating to ensure the most strategic use of resources and access to newly emerging technologies, 5 | LINKAGES MIDTERM EVALUATION REPORT USAID.GOV • Addressing structural barriers and transforming local KP organizations, • Ensuring interventions are sustainable over the long term, and • Supporting the mainstreaming of human rights, gender and competency and capacity development. USAID Indonesia commenced the LINKAGES mechanism in May 2015. The total investment over the project lifetime, including central funding, is expected to be $26,513,726. The LINKAGES’ operating model emphasizes efforts directed at four inter-related foundational areas: (a) utilizing program data to populate the HIV cascade and identify key leaks across the continuum of HIV prevention and care (CoPCT) for key populations; (b) a decentralized implementation approach that brings together community- and facility-based partners and focuses on sustainable health system strengthening interventions; (c) a demand-driven technical assistance (TA) systems framework at national, provincial, district, and facility/organizational levels; and (d) a strong human rights focus that ensures key population representation and voice in HIV CoPCT policy and implementation. EVALUATION PURPOSE The purposes of this midterm evaluation are to promote accountability; assess project performance, outputs and any outcomes to date; and recommend ways to improve project performance in Indonesia across the three key result areas to enhance the impacts of the Government of Indonesia’s (GOI) HIV program. Findings are expected to be used by USAID, partners, and other stakeholders to promote understanding of the project’s strengths, progress and challenges. The primary audience for this midterm evaluation is USAID/Indonesia and GOI stakeholders. In addition, USAID will share the findings from this midterm evaluation with the Implementing Partner. This evaluation recommends ways to improve LINKAGES’s technical and management performance; suggestions for how the project can improve the provision of TA and provide ideas on how USAID can enhance the impacts and efficiency of the GOI’s HIV program. EVALUATION QUESTIONS, METHOD, LIMITATIONS To guide this evaluation, USAID has identified the following six evaluation questions: What progress has been made by LINKAGES in achieving its goals, objectives, and performance targets? What contributions has the project made in increasing HIV testing and ARV coverage in its priority sites? To what extent have project activities strengthened the capacity of CSO and GOI partners in achieving the goals and targets of Indonesia’s National HIV Program? How has the project improved collaboration between these actors? To what extent have project activities strengthened the capacity of national and sub-national (province and district) governments and other stakeholders in managing Indonesia’s National HIV Program? To what extent are LINKAGES methodologies, interventions, and management setting the stage for the future sustainability, replication, adaptation, and adoption of project outputs and outcomes? How effective has the collaboration and coordination with other HIV programs funded by USG (e.g., Procurement Supply Chain Management, Challenge TB) been in maximizing efforts and achieving greater USAID.GOV LINKAGES MIDTERM EVALUATION REPORT | 6 results? How effective has the collaboration and coordination with other internationally funded HIV programs been in maximizing efforts and achieving greater results? How effectively is the project being managed internally with consortium partners (subcontractors)? How effective is the coordination between LINKAGES and USAID/Indonesia, and between LINKAGES Indonesia and LINKAGES headquarters in Washington? This midterm performance evaluation provides an opportunity to support innovation and ongoing adaptation of the program, as it is implemented, in a complex and changing context and generate practical recommendations, to inform program management and improve impact in the remaining years of implementation. The evaluation design uses mixed–methods, and was co-designed by the Evaluation Team Leader, the commissioning clients and KP groups, representing those intended to benefit from the program. The evaluation was designed to comply with the USAID Evaluation Policy (2011 updated 2016) and the PEPFAR Evaluation Standards of Practice V2.0 (2015). The evaluation is consistent with USAID’s definition of a performance evaluation (p 3) and PEPFAR’s definition of a process evaluation (p3). Consistent with the principle of ensuring there is meaningful involvement in all aspects of the response to HIV by the KP, and the USAID Evaluation Policy, a co-design workshop was held on 8 August 2017 with a small group of people from Civil Society Organisations (CSO) working closely with or led by KP. During the workshop participants contributed to developing the methods, data collection instruments, selection of key informants and the evaluative criteria for the evaluation. For a report on the Workshop see Annex) The design outlined a mixed methods evaluation to answer the evaluation questions. For a full description of the Methodology, including data collection and analysis plan and the time line see Annex 1. The evaluation was conducted August – October 2017, half way through the project’s official implementation. Primary qualitative data collection methods included semi structured interviews, field visits and observation at implementation sites, key informant and small group interviews and large group discussions. The evaluation team collected data from key informants selected using a purposive sample of respondents from government, a range of partner organizations, civil society organizations, and LINKAGES and USAID staff to gain their perspectives on the program. (See Annex III for the data collection instruments.) Further data was collected through a review of key documents. And all data was triangulated against performance data submitted in compliance with PEPFAR guidelines LIMITATIONS Due to the large number and type of stakeholders, and the need to collect all primary data with a period of four weeks, it was not possible to interview all stakeholders. The number of stakeholders interviewed (139 people) was maximized by the evaluation team splitting into two sub-teams when possible during site visits, and meeting outside of usual business hours if necessary. There were a number of language barriers, as three members of the team are not fluent in Indonesian, and in Papua a number of informants spoke Indonesian as a second language. However, the impact was minimized through the use of an interpreter during interviews, particularly with CSO implementing partners and project beneficiaries. 7 | LINKAGES MIDTERM EVALUATION REPORT USAID.GOV FINDINGS AND CONCLUSIONS RESULT AREA 1: INCREASING AVAILABILITY OF COMPREHENSIVE HIV SERVICES, ACROSS THE COPCT FOR KP RELEVANT EVALUATION QUESTIONS What progress has been made by LINKAGES in achieving Result Area 1 performance targets? What contributions has the project made in increasing HIV testing and ARV coverage in its priority sites? To what extent have project activities strengthened the capacity of national and sub-national (province and district) governments and other stakeholders in managing Indonesia’s National HIV Program? During the evaluation co-design workshop, the intended beneficiaries said they would evaluate the program as having increased the availability of services if there were more services, they were accessible to and appropriate for KP and made them feel welcome. The project measures availability in terms of the number of KP reached, tested for HIV and enrolled in care. The LINKAGES project has partnered with the Provincial Health Offices, the District Health Offices and an increasing number of HIV treatment facilities in hospitals and primary health care centers and private clinics in Jakarta and Papua to expand the range and quality of comprehensive HIV services across the continuum of care. The project activities, explored in more detail below, are contributing to increasing HIV testing and treatment coverage by: • increasing reach and high yield testing • linking PLHIV to HIV care and treatment services • initiating timely treatment with anti-retroviral therapy (ART) • sustaining individuals on ART and ultimately suppressing viral loads (VL) The project currently has 19 partner organizations. The project is addressing the availability of comprehensive HIV services across the continuum of Prevention and Care and Treatment (CoPCT) for Key Populations (KP) through the provision of support for direct service delivery (DSD) and Technical Assistance for service delivery improvement (TA- SDI) to Government, Civil Society Organizations (CSO), including those funded by the Global Fund for AIDS, TB and Malaria (GFATM or the GF) and the private sector in the five municipal districts of Jakarta and three districts in Papua province (Jayapura, Wamena and Mimika). Efforts by the project to increase availability are being underpinned by a range of activities to improve data collection, management and utilization, and to strengthen the capacity of the GOI to set and targets. These activities support the national AIDS program, and allow for better monitoring, accountability and transparency. LINKAGES has developed the data collection and management tools necessary to include reach and test data to the cascade, as illustrated in Figure 1. Until recently the National HIV/AIDS Program in Jakarta and Papua had no system for capturing prevention data to illustrate reach among key populations and HIV testing results. The project has developed and introduced new data collection and curation systems and tools to address this gap, including a cascade generator. The project efforts have ensured that data related to reach and testing is combined with the treatment cascade data and presented in an accessible format. USAID.GOV LINKAGES MIDTERM EVALUATION REPORT | 8 The national program was not including viral load suppression in the cascade, as there is only limited, poor quality data available at present. However, LINKAGES consistently advocate for this, and although the bar contains no data at present, it serves as a reminder of the need to monitor treatment effectiveness. A user-friendly cascade generator has recently been introduced. It takes 6 data points from the vast collection available in SIHA and allows implementers and managers at site, District and provincial level to generate a dashboard showing the numbers tested, positivity yield and other key data. At a glance, the number of people from each KP that have tested can be identified. FIGURE 1: LINKAGES COPTC CASCADE OF HIV SERVICES FOR KEY POPULATIONS LINKAGES has initiated a number of activities to increase high-yield HIV testing efforts by CSO and GOI testing services through the introduction and implementation of diversified and decentralized HIV testing and screening services, including: • Strengthening fixed and mobile testing services through the Enhanced Outreach Approach • Developing and implementing the Community Outreach Management System (COMS) and the project’s client management databank (CMD) • Supporting the coordination and streamlining of mobile testing strategies at district levels • Simplifying pre-testing and work flow procedures through an online system (DOKLING) • Providing TA for feasibility research and advocacy for self-testing ENHANCED OUTREACH APPROACH AND COMMUNITY OUTREACH MONITORING SYSTEM (COMS): Prior to LINKAGES introducing the Enhanced Outreach Approach, HIV prevention activities in Jakarta were experiencing many of the problems traditional outreach approaches have recently been criticized for: failing to achieve sufficient coverage among KP at high risk of HIV infection, failing to convert a high percentage of those reached into HIV testing clients, and resulting in relatively low case finding 9 | LINKAGES MIDTERM EVALUATION REPORT USAID.GOV rates. Recent epidemiological modeling estimates that about 63% of new HIV infections in Indonesia are among KP, with incidence rising among MSM. Modeling cited in the most recent Global Fund funding request suggests that with current levels of intervention, overall HIV incidence will remain flat at 40,000- 50,000 new cases annually, until 2030, increasingly among MSM. The ineffectiveness of current approaches to produce a net decrease in HIV incidence for KP is thus a major driver of the epidemic in Indonesia. In a context of new evidence demonstrating the importance of early testing and treatment to achieve reductions in HIV incidence at the community level, it is critical that community-based outreach produces demonstrable improvements in the HIV cascade by testing high-risk individuals, supporting treatment initiation among those who are HIV positive, and supporting high levels of treatment adherence among ART patients. In response, in 2017 LINKAGES introduced the Enhanced Outreach Approach (EOA), which will be revised and further refined before the end of 2017 to be known as the ‘Enhancing’ Outreach Approach. EOA is a prevention and care outreach strategy that uses one outreach workforce (cooperating with peers in the community) to provide customized outreach services across three segmented KP and PLHIV groupings: (a) outreach for HIV negative and/or unknown HIV status KP individuals; (b) outreach for [unstable] PLHIV who are registered in HIV care and treatment services; and (c) outreach for PLHIV who are not registered in HIV care and treatment services (e.g. never registered or lost to follow up). This differs from the GF supported model, which uses one workforce for prevention outreach and another for supporting people once they are enrolled in treatment. The approach is further differentiated to increase reach among different key populations. LINKAGES was criticized by a development partner and a local CSO for ‘introducing a new system’ rather than ‘letting a new system develop organically’. This criticism seemed spurious, after all the national program is constantly introducing new systems, and the ideas are drawn from a wide range of contributors. Another CSO manager said that LINKAGES should not focus on reach and testing, or high yield testing, because the GOI will soon adopt a policy of “Test and Start’ in 21 districts, so this will find enough new cases.” Other key GOI HIV program staff talked about the need to reach non-key populations, such as transport workers, also using traditional outreach approaches. This criticism of the project’s efforts to innovate highlights the context in which LINKAGES is introducing systems that track the effectiveness of reaching KP: the evaluation team notes that neither LINKAGES nor the GOI national program have been able to embed a culture of innovation or a sense of urgency in increasing reach or high yield testing. Throughout the evaluation key informants argued for restricted operating days and times, or for no change to existing approaches (despite their lack of efficacy). During interviews with UN partners, Global Fund principal recipients and GOI program leaders the evaluation team repeatedly experienced discussions of reach and test turning into discussions on who gets to take credit for reaching the targets, (which indicates a limited understanding of commitment to Collective Impact) or how much pressure the staff are under if the CSO bring in too many clients. The project supported CSOs stated that although the project pushes them to reach ambitious targets, it has also given them to tools to increase their reach and monitor their yield. They were also willing to discuss the merits of differentiated approaches. In contrast, discussions with funders and the GOI centered on payment, credit for numbers reached and how difficult it would be to meet a target of 50% of KP. GOI health care providers at the puskesmas level repeatedly said that testing quotas/limits need to be set or clients should only come on specified days and times, because it creates too much work for the staff at them, especially lab staff. USAID.GOV LINKAGES MIDTERM EVALUATION REPORT | 10 CSO field staff working with MSM in Papua 1reported that it is very difficult to get men to test, and even the Outreach workers at times feel reluctant to ask other men if they know their status and would be willing to have a HIV test. Past experience has taught them to fear that if the result is positive the MSM will ‘blame them’ or go on to ‘hide from them’. The operating environment is very difficult, due to high levels of stigma and discrimination, and ignorance about HIV. All sites have limited testing hours, some puskesmas only offer HIV testing between 9 – 11 am on certain days, and so the CSO has encouraged the puskesmas to increase their opening hours and supported them to do mobile testing. The CSO and puskesmas staff report the MSM prefer community-based testing because it is perceived as being more private and friendly than going to the clinic. There are high levels of stigma and suspicion, and the Papuan MSM are reportedly especially hidden and reluctant to be identified. The CSO staff are actively involved in Papua Pelangi, a community-based organization that brings MSM together to strengthen the sense of community, break down barriers between Papuan and non-Papuan men and provide opportunities for peers to educate each other. The organization receives no outside financial support, but members have been acting as peer mobilisers for LINKAGES supported CSO. DATABASE ONLINE UNTUK MOBILE HIV KONSELING DAN TESTING (DOKLING) SYSTEM: After observing the ad-hoc, uncoordinated and cumbersome approach available to CSO when planning and reviewing the results of mobile HIV testing, LINKAGES identified a need to streamline and simplify the process. Voluntary testing at mobile HIV testing and counseling sites in places accessible and familiar to key populations were scheduled on an ad hoc basis, without systematic documentation that would allow CSO to monitor the effectiveness of their testing strategies. LINKAGES developed and introduced the Database Online untuk Mobile HIV KonseLing dan TestING (DOKLING) system. DOKLING is a mobile testing scheduling and tracking platform that enables CSO to systematically schedule mobile VCT visits, allows puskesmas testing staff to confirm their availability for scheduled testing and supports puskesmas to organize mobile VCT and rationalize human resources and testing supplies. DOKLING also allows Provincial and District Health Office personnel to review mobile testing data and to place services for greatest impact. LINKAGES developed the DOKLING user manual and facilitated numerous training sessions to show users how it works. Implementers from 44 puskesmas, 4 private clinics, 2 hospitals and 21 CSOs serving key populations were oriented on the DOKLING system. DOKLING was rolled out across five Jakarta districts on April 1, 2017. In Papua, LINKAGES clinical services Technical Assistance (TA) staff are working with the Papua Provincial Health Office to develop a mobile testing procedure which may incorporate components of DOKLING. Project data indicates that from April to August 2017, 97% of health facilities and 87% of CSO in Jakarta have utilized DOKLING to schedule 182 mobile testing visits where 701 key populations have tested for HIV throughout the city. The majority of key populations testing for HIV have been female sex workers (n=485), who make up 69% of the coverage total, with MSM trailing behind at 15% (n=22 persons). Extensive TA has been required to get DOKLING working smoothly. One internal DOKLING report notes “Both CSOs and puskesmas struggle with inputting data into DOKLING – particularly unique client identification codes (CSO) and anonymous service uptake information (puskesmas).” (Aulia Human, 2017) However, the evaluation team heard from many informants that the system was being widely used, glitches had been quickly resolved and the CSO and puskesmas were conducting a lot more community-based 1 (Jenne - Outreach with the CSO workers in Jayawijaya on 15.8.17) 11 | LINKAGES MIDTERM EVALUATION REPORT USAID.GOV testing together than ever before. Figure 2 below shows the uptake of DOKLING in Jakarta by health facilities and CSO, five months after the introduction of the system. FIGURE 1: DOKLING SYSTEM DEVELOPMENT In the past, CSO needed to send a formal letter to a health facility to request a mobile testing visit and wait to have their request approved. With DOKLING, CSO personnel now enter the system and click on their preferred testing date/time. The facility either approves or rejects the proposed visit, and an automatic email is provided to the requesting CSO. Most informants report the process is working smoothly. Only one CSO reported that they were not receiving timely responses from the testing staff at the puskesmas. When they investigated, they found that the staff at the center was new, had not been to the training and didn’t know they had to click the ‘approve/reject’ button to respond to the requested mobile testing. This was quickly and easily resolved. The District Health Offices have worked closely with LINKAGES to get the system in place and encourage site level staff to use it. Most significantly, DOKLING allows the user to track HIV yield at testing sites and use this information to reflect on results and plan future testing activities. DOKLING contains a geo-tag function that allows for the presentation of HIV yield data by sub district, which can facilitate programmatic prioritization or amplification, supply forecasting, and community-facility coordination. LINKAGES has trained partners to show them how to use this function. Several CSOs report using DOKLING to identify how many cases they are finding, so they can see which sites are really ‘hot’ hotspots, then scheduling their outreach, prevention and mobile testing so that they are returning often to high yield sites until the number of new cases identified drops to none or one. Sites where very few cases are identified are dropped back to be visited less often. Some CSO, for example Yayasan Pesona Jakarta,2 reported using DOKLING dashboards so Field Coordinators can be strategic in managing their community outreach workers and identify the HIV yield by site. 2 Focus Group Discussion with Yayasan Pesona Jakarta on (date) USAID.GOV LINKAGES MIDTERM EVALUATION REPORT | 12 LINKAGES will support efforts to analyze testing data and will further develop performance dashboards for enhanced data utilization. In addition to mobile testing coordination and tracking, DOKLING has the potential to be used to strengthen facility-facility service referrals for repeat/confirmatory testing and tracking viral load testing specimen transport. This capability can be easily added to the online system. TB/HIV SERVICE INTEGRATION: The project is improving TB/HIV service integration by: Developing procedures and expertise to strengthen capacity to incorporate TB screening into HIV outreach services; analyzing patient flow between TB and HIV services; finalizing TB/HIV clinical mentorship guidelines and other relevant Standard Operating Procedures and /communications; supporting TB/HIV district planning and strategy development; and advancing TB/HIV integration demonstration sites. The key informants noted the high level of cooperation between LINKAGES, Challenge TB, and other partners, and the momentum the project has brought to this work. Despite clear policy guidelines to test TB patients for HIV, insufficient systems are in place to underpin rolling out routine testing among all TB patients. Many key informants report that doctors are reluctant to offer testing, because HIV is highly stigmatized, and they are not confidant with gaining informed consent or delivering a positive test result. Additional capacity building is needed, and systems of accountability. The project has identified DOKLING as a potential short-term solution to accelerate HIV testing among TB clients at puskesmas where HIV screening/testing is not yet operational for TB patients. LINKING PLHIV TO HIV CARE AND TREATMENT SERVICES: LINKAGES activities to increase the number of people who test positive who are linked to treatment services and enrolled in care include: 1. Increasing the number of PLHIV receiving treatment through implementation of the SUFA strategy, decentralized ART provision, enhanced linkages, cooperation and integration of community-based organizations and health facilities 2. Supporting the decentralization of service delivery so that treatment can be accessed at an expanded number of puskesmas and satellite services 3. Strengthening cohort reporting/analysis to improve client retention 4. Introducing systems to track retention among ART clients 5. Strengthening TB/HIV service integration 6. Contributing TA to efforts to systematize viral load testing for 90-90-90 7. Supporting increased retention through preventing, detecting, and resolving loss to follow up (LTFU) issues. LINKAGES has placed time-limited personnel in each District Health Office to assist with the activities outlined above. Since 2014, GOI has implemented a ‘Test and Treat’ policy through its “Strategic Use of ARVs” (SUFA) initiative which mandates immediate treatment with ART for key populations, TB patients, and pregnant women in over 100 districts, including in Jakarta and Papua. However, the implementation and uptake of this policy has been slow, and results are poor, with best estimates suggesting only 12% of all PLHIV are receiving treatment. LINKAGES has identified and addressed a number of the factors that contribute to 13 | LINKAGES MIDTERM EVALUATION REPORT USAID.GOV these results, including confusion among service providers, health system inadequacies and doctor’s concerns re treatment readiness. Despite these efforts, there are ongoing challenges to enrolling and retaining PLHIV in HIV treatment. Confusion persists over who can access treatment immediately and when and how to start PLHIV on treatment. The high levels of stigma and discrimination inhibit many people from identifying as KP, and then they are not prioritized for treatment. PLHIV quickly become lost to the health system once they are denied treatment, and the clinics do not retain any recall systems. A review of LINKAGES documents related to clinical mentoring and interviews with key informants suggests that if there was a clear policy to test and treat all, this would simplify maters for clinicians and help expedite PLHIV from testing to treatment services. Many doctors spoke to the evaluation team about their concerns related to initiating newly diagnosed PLHIV on treatment, a process which typically takes 2 -4 weeks. There are a number of barriers to initiating newly diagnosed PLHIV onto anti-retroviral therapy which need to be addressed at a system level, including: • KP are unwilling to be labelled as a member of a stigmatized group and are therefore listed as ‘other’ and not eligible for immediate treatment under SUFA. • Identity cards and other documentation with proof of local registration are needed, and many KP don’t have them and cannot acquire them. • Health care providers have not been trained and/or lack confidence in their ability to prescribe the appropriate medications or manage side effects. • Health care providers assume the patients will have poor adherence and therefore ‘test’ their readiness by requiring patients to pay for and undergo tests, often off-site. • At each test (e.g., liver function test, or viral load or CD4 count test), the patient is required to show appropriate identification and citizen registration documents, and pay for the test, and return later for the results. • Testing treatment readiness can also involve requiring PLHIV to take 2 weeks’ worth of anti￾biotics and then report back to the clinic. • Patients are asked to sign declarations of commitment to treatment, which look like contracts and scare KP. • When patients move from one location to another, the ad hoc transfer process often results in them having to be tested again before they can access treatment. All of these things act as a deterrent and limit the availability of services. They also breed an environment of distrust and contribute to the large number of PLHIV lost to the treatment system soon after being diagnosed. Some doctors will refer patients on to local hospitals for initiating treatment if they are not confidant, while others will contact clinical mentors for guidance. Interviewees did however indicate that the recent introduction of targets is motivating them to get their patients on treatment and keep them enrolled at their puskesmas. The doctors have limited training in providing HIV treatment, and no access to ongoing professional development or continuing medical education. The project has responded by providing clinical mentoring, and telephone support provided by an experienced treatment doctor. USAID.GOV LINKAGES MIDTERM EVALUATION REPORT | 14 MONITORING QUALITY OF CARE FOR PLHIV: The Project is providing technical assistance to develop and roll out a viral load testing system. This involves working with program partners to conceptualize a viral load (VL) testing and targeting strategy and developing systems to collect, manage, report and use VL testing data. The project is also supporting development of VL specimen transportation system, a joint utilization plan for VL testing machines and promoting the benefits of VL testing among PLHIV. CONCLUSIONS LINKAGES has made considerable progress in achieving Result Area 1 performance targets, despite the many delays in establishing the project attributable to the need to secure higher level agreements between the US Government and the GOI. In a national program that needs to be improved, the LINKAGES approach of addressing shortfalls in coordination and collaboration, strengthening links and practices across the service system and improving the collection and use of data is both strategic and effective. At the commencement of LINKAGES in 2015, available data and program review results suggested that the national HIV program lacked the coverage and intervention effectiveness needed to make a major impact on the course of HIV in the country,3 and the quality and comprehensiveness of the national prevention package of services for key populations was suboptimal. The national program was not routinely collecting or analyzing reach and test data, and coverage was assessed semi-annually through sentinel site surveillance which indicated that only 18.5% of FSW and 23.4% of the estimated MSM4 in country had been reached5. No national data was available on the proportion of key or priority populations reached who went on to be tested for HIV. Although 120,677 people were recorded as ever having been enrolled on ART, only 63,066 PLHIV were receiving ART at the end of 2015, 6 which represented an estimated national ART coverage rate of 8%.7 Monitoring for treatment effectiveness was not on the radar, as viral load testing was not widely available. The evaluation finds that the project is on track to achieve key reach and test targets. As can be seen in Figure 3, the project is reaching the reach the targets for HIV prevention among KP and priority populations in Jakarta and Papua. Over 50 000 people have been tested in FY17, and over 3000 new PLHIV have been diagnosed. Problems persist in achieving satisfactory levels of treatment uptake, but by the end of quarter 3 the number of PLHIV on treatment had already exceeded targets. 3 2015 GFATM Concept Note 4 According to the 2013 UNAIDS Global AIDS Response Progress Reporting (GARPR) report and National AIDS Commission (NAC) program data 5 “reached” was defined as at least one contact per person per year (regardless of which component(s) of the prevention package was received) 6 2015 MoH Indonesia Quarter 4 HIV Program Report 7 2014 UNAIDS Global AIDS Response Progress Reporting (GARPR) Report 15 | LINKAGES MIDTERM EVALUATION REPORT USAID.GOV FIGURE 3: FY17 Q1-Q3 PERFORMANCE AGAINST ANNUAL TARGETS IN JAKARTA AND PAPUA The project has made significant contributions to increasing HIV testing and treatment coverage, however there is a long way to go to achieve the level of coverage necessary to halt the epidemic. The project has expanded testing options for KP and PP, advocated for self-testing, increased the focus on high-yield testing and increased the number of sites that are able to initiate and provide treatment through TA for decentralisation of HIV treatment. The evaluation finds that LINKAGES is playing a key role in expanding HIV program coverage and improving intervention effectiveness in Jakarta and Papua. The project applies a collective impact approach, working alongside a small group of TA providers to support the national program, and playing a leading role in improving the range and quality of services available across the CoPCT to key populations. All indications from the document review and discussions with key informants indicate that the project has been effective at influencing policy and practice and introducing new services and systems needed to optimize the prevention packages for KP. The project demonstrates a willingness to consistently innovate, introduce new strategies and demonstrate how prevention and testing methods can be rolled out at scale in Indonesia. This comes with some criticism, as the operating environment can be slow to innovate, and policy advances are not consistently supported with implementation plans or infrastructure. The project staff are to be commended for continuing to propose ambitious targets and lead change in the face of what could best be described as indifference among some key stakeholders. The evaluation concludes that LINKAGES activities are contributing to increasing the availability of services that reach and support testing among KP and PP. The introduction of the Enhanced Outreach Approach has occurred in the face of some resistance or complacency among national program partners but is helping CSO partners to strive to reach more KP. The project has increased the focus on reach and testing by adding these parameters to the cascade. Although not all LINKAGES reach, and test targets have been met, over the last few months testing participation rates have been increasing and targets have been introduced at the puskesmas level, resulting in immediate increases in testing numbers. Reaching female sex workers remains challenging and differentiated models should continue to be pursued to reach workers online. USAID.GOV LINKAGES MIDTERM EVALUATION REPORT | 16 Increases in testing numbers can be directly attributed to the use of DOKLING and the decentralization of HIV services. DOKLING has simplified the mobile testing scheduling process and is promoting a new level of transparency. Prior to the introduction of the Enhanced Outreach Approach and DOKLING, community-based testing efforts lacked focus, were uncoordinated and results could not be tracked. With DOKLING, the Provincial Health Office (PHO), District Health Offices (DHO), health facilities, and CSO can see at a glance: where and when mobile testing is taking place; who is requesting mobile testing services; how often mobile testing is being carried out; and which key populations are benefitting from mobile testing services. DOKLING is also facilitating data reporting and analysis, and the easy access to data is reported by key informants to be informing discussions between the DHO and puskesmas. Before the introduction of DOKLING, the PHO and DHO were unable to view mobile testing data, so it was under-utilized. Now data can be reviewed and analyzed by facility, district, time frame, and CSO, noting that all client information is protected by the systematic use of Unique Identification Codes (UIC). Several DHO report using DOKLING and informants stated that it has improved their knowledge of what is happening in their District and their understanding of where cases are being identified. The evaluation finds that DOKLING is well accepted, and recently several other provinces and districts have requested that access to the system. In Jakarta, for sustainability purposes, there is a plan to host DOKLING under PHO website as sub-domain once the system is stabilised. As users get familiar with DOKLING, they have requested more functionality and the project is looking at using DOKLING to support testing among TB patients. LINKAGES has also increased availability by encouraging extended facility operating hours and supporting sites to adopt other KP friendly approaches. The project activities have also provided the impetus for streamlining HIV diagnosis and enrollment procedures to enable Test and Start. Many barriers remain however, and the restricted operating hours needs further investigation. The push back against testing targets at some sites could potentially be alleviated by further investigation into the workload and work flow and patient to provider ratios. Introduction of the cascade generator represents a significant project achievement. The SIHA system collects vast amounts of data, most of which was never used, especially at the site or District Level Locally generated dashboards are opening up conversations at every level and have revolutionized the role of data in program monitoring and management. The dashboards provide an easy to read, compelling graphic representation of the key data. Across all levels of the health system, HIV service providers are learning how to generate cascades at their own site and District level, and how to use the data to review the results of community-based testing and identify the number of PLHIV lost to follow up across the cascade. This represents a significant shift in program planning and management, and more optimal use of data. Now that most sites can use the cascade generator, ongoing effort is needed to increase capacity to analyze, interpret and use the dashboard data. Capacity building should focus on building data use core competencies and capacity to use data to monitor performance. This is likely to contribute to a virtuous cycle in which improved data collection, analysis, availability, interpretation, and use continuously generate more demand for and sustained use of data, which leads to improved accountability and decision making. The evaluation team was able to witness how the use of the cascade generator is supporting the program to accumulate a critical mass of new understanding of the KP reached, tested and enrolled in care. The project can expect to see significant increases in availability of services for KP as this new understanding influences managers. 17 | LINKAGES MIDTERM EVALUATION REPORT USAID.GOV There are a number of challenges to further increasing test and reach results which require more or ongoing attention. The facility-CSO partnerships are being developed, and the MOU process is facilitating this, but the relationships are still variable in practice. Some facilities think they have too many partners, or their partners bring in too many people for testing and they are not happy about the workload this generates. Some community-based workers report frustration with the limited operating hours and find that people are reluctant to test if they have already been turned away because they went to the clinic on the wrong day. The regular coordination meetings facilitated by the project help to resolve some issues, but are not getting to the underlying, system problems. The LINKAGES project has been successful in focusing HIV program efforts on key populations in Jakarta and key and priority populations in Papua, but HIV testing rates among TB patients remain low. During the field visits the evaluators noted the absent or complicated procedures linking TB patients to HIV testing. Despite Indonesia recording over a million TB cases per year, the 2017 Joint External Monitoring of the TB program found that only 5% of the estimated HIV‐positive TB patients were detected or notified to the national TB and HIV programmes. Provider-initiated and opt-out systems that reach TB patients have good potential to increase testing numbers and find new cases. The project plans to work closely with the new Challenge TB project in the coming year and should consider including TB staff in capacity building to increase their confidence in offering HIV testing. This is likely to yield high positivity rates for limited outlay. Ongoing clinical mentoring is needed, but this should not be confused with capacity building for overall program improvement. The challenge is to empower and inspire health care providers to monitor their own effectiveness and take pride in their achievements, not rest once tasks are completed. The World Health Organization recommends routine monitoring of ART effectiveness using viral load (VL) testing to monitor treatment adherence and minimize failure. However, there is a lack of awareness among clinicians as to the long-term patient benefits and VL testing’s role in monitoring treatment adherence and efficacy and prolonging the use of first line regimens and the longevity of treatment programs. Even where clinicians and PLHIV are aware of the value of VL testing, the cost is preventing uptake. Barriers to monitoring treatment effectiveness need to be removed and access to testing increased. There are potential economies of scale and benefits to sharing viral load testing infrastructure with the TB program, especially in Papua, where every clinic and hospital visited by the evaluators had a GeneXpert machine sitting on their bench. RECOMMENDATIONS Recommendations to increase the availability of comprehensive HIV services, across the continuum, for KP and PP: 1. Continue to strengthen the capacity of service delivery sites to analyse their patient loads, testing strategies and collaboration efforts, and plan to use existing resources more strategically, in discussion with the DHO and Puskesmas Managers, to increase retention across the cascade. 2. Assist CSO to identify how changes in their overall intervention-mix and skill mix could create efficiencies and increase impact. USAID.GOV LINKAGES MIDTERM EVALUATION REPORT | 18 3. Collaborate with Challenge TB and other stakeholders to systematically increase HIV testing among TB patients by addressing the capacity, confidence and accountability of TB staff, adding functionality to DOKLING to support coordination and specimen tracking. 4. Continue to support puskesmas and CSO staff to work together to optimise community-based testing, increase HIV yield and support KP across the CoPCT. This will require capacity building to ensure health and CSO staff can analyse the dashboard data and refine their strategies based on results. 5. Bring together key stakeholders to remove the barriers to initiating newly diagnosed PLHIV onto anti-retro viral therapy and retaining them on treatment when they move locations to reduce the delay between diagnosis and treatment and high rates of lost to follow up. 6. System changes are required to free up places at private clinics for PLHIV without documentation, and move stable, treatment experienced patients to new clinics. This may require a staged approach, as the clinics are likely to be reluctant to refer patients on. Change in the distribution of patients has the potential to realize widespread benefits, as working with stable patients will increase the confidence and expertise of doctors in new treatment sites, as well as reducing travel time for patients. 7. Demonstrate interventions that assist the GOI to understand the benefits of and challenges to expanding access to and reporting on viral load testing. Contribute to the development of implementation plans designed to overcome infrastructure and program complexities and address the affordability constraints. RESULT AREA 2: ENHANCING AND SUSTAINING DEMAND FOR COMPREHENSIVE HIV SERVICES ACROSS THE COPCT AMONG KP The demand for HIV testing and treatment is low among KP and PP, especially in Papua. As discussed above in Result Area 1 (Enhanced Outreach Approach), many of those reached do not access HIV testing, and once diagnosed, many are lost to follow up and do not benefit from early access to HIV treatment and care. The LINKAGES project approach to demand creation is to raise awareness and test the feasibility of introducing innovative, evidence-based approaches and increasing the engagement space for KP. The demand creation activities adopted by LINKAGES to improve the enabling environment and empower key populations to access the continuum include: 1. Advancing innovative, evidence-based models and approaches to promote demand, for example: • Voluntary Male Medical Circumcision (VMMC) in Papua • Pre-Exposure Prophylaxis (PrEP) for MSM • Enhanced Outreach Approaches Supporting KP engagement through: • Social and behavior change (SBC) campaigns and communication and ICT • Addressing stigma and discrimination (S&D) in health care settings • Promoting human rights for KP • Developing violence prevention and response initiatives 19 | LINKAGES MIDTERM EVALUATION REPORT USAID.GOV The contexts in Papua and Jakarta are very different, and the project has adopted a range of approaches to strengthening and sustaining demand in two such different epidemics and environments. The project has mapped out key partners, (see Figure 3) including the National AIDS Program, KP Networks and advocacy organizations and primary health care centers where assessments and studies can be implemented, and new ideas tested. The project has enlisted Universities and creative and big data analytics companies to assist in the development of demand creation and monitoring activities. FIGURE 4: LINKAGES KEY PARTNERS IN RESULT AREA 2 (SUSTAINING AND ENHANCING DEMAND ACROSS THE COPCT) The project is creating demand for novel, evidence-based prevention approaches, most notably to date VMMC and PrEP, and the project advocates for treatment as prevention and accelerated implementation of the SUFA policy. There is an active champion of VMMC in the Provincial AIDS Office (PAC/KPA) in Papua, and the office has procured 6000 kits and done a pilot study to demonstrate the feasibility of VMMC. To capitalize on the interest in VMMC, LINKAGES planned and facilitated a review that brought together technical experts from WHO, PEPFAR/USAID, and the Ministry of Health to review VMMC interventions in Papua and to articulate key VMMC TA recommendations. The project is now working closely with USAID to confirm the type/extent of ongoing TA and to advocate for increased TA support from WHO and GFATM to roll out this prevention intervention. Since 2015, WHO has recommended that all people at substantial risk of HIV infection should be offered PrEP as part of comprehensive prevention options. (WHO, 2015) The GOI has not enshrined access to PrEP in policy or made it available. The high HIV prevalence among MSM and TG women Waria) in Jakarta (32% and 30%, respectively) (MoH Indonesia, 2015) indicate that the two KP are likely to benefit from PrEP if it were made widely available and uptake and use were high among those most at risk, including young people. The LINKAGES TA response includes a range of activities designed to: • strengthen understanding of PrEP and how it might be introduced in Indonesia USAID.GOV LINKAGES MIDTERM EVALUATION REPORT | 20 • generate evidence of the knowledge and acceptability of PrEP among MSM community, the HIV/AIDS/STI Sub-Directorate, health service providers and key stakeholders • advocate for the supply of ARV for PrEP • holding discussions with Gilead on registration of Truvada® (a requirement for introduction of generics), and facilitating the rapid introduction of generics LINKAGES sponsored participation in the regional PrEP consultation in Bangkok in January 2017 for representatives from the AIDS Research Center at Atma Jaya Catholic University and a screenwriter for a popular LGBT web series. Since then, LINKAGES has established a two-phase PrEP acceptability and feasibility implementation research trial, in partnership with The Ruang Carlo clinic and Atma Jaya University. They have facilitated discussions on PrEP and its value for MSM with local MSM and Waria. The study is now underway to explore existing knowledge, attitudes and beliefs regarding the use of oral PrEP as a prevention method among HIV-negative MSM and Waria, generate recommendations for effective service delivery, including interest in accessing PrEP, preferred modes of delivery and willingness to pay. Potential models of service delivery are being explored, including (VCT/STI center, CSO-based delivery, etc.) along with a ‘willingness to pay’ analysis. In 2017 LINKAGES has initiated and is supporting” Jakarta is PrEP’d”: An Implementation Pilot Study to deliver, evaluate and document community-driven, private clinic-based HIV PrEP services for MSM/TGW at high risk for HIV infection. The study will develop, implement and evaluate PrEP services and document the delivery and evaluation of PrEP services to inform future use. The Ruang Carlo Clinic, operated by the St Carolus Hospital and funded by LINKAGES, will host this PrEP pilot study and the MOH will provide ART and some reagent support. The project is enhancing demand by supporting KP engagement in the national HIV program, most notably by seeking to increase access to HIV and legal services and reducing the barriers KP experience in accessing services. SBC INTERVENTIONS TO SUPPORT UPTAKE OF HIV HEALTH AND LEGAL SERVICES In year 1 LINKAGES documented and developed understanding of how outreach and support services for KP were operating. This involved the development and utilization of the LINKAGES ‘outreach flow and thresholds assessment tool’. The tool is used to assess performance across the CoPCT cascade. It is divided into five sections, each of which outlines recommended technical performance expectations for identifying, reaching, and testing KP and diagnosing PLHIV, treating and retaining PLHIV on ART; and utilizing data and monitoring technical performance. In year 2, the project conducted reviews with eight USAID supported CSO working with all KP sub-populations. During the reviews the CSO staff shared their communications and data management tools and demonstrated or described strategies they use to identify and reach members of key populations, successfully refer them to HIV testing, and to support enrolment and retention in care and treatment services. The findings of the review were used to inform the Enhanced Outreach Approach (EOA) which is described in Result Area 1 (above). One CSO representative stated: “No one had ever looked so closely at what we do in the field: usually they just want to talk about how much outreach workers get paid, where we go and how many people we see. We had to really think about what we do so we could explain it during the review.” KP CSO in Jakarta 21 | LINKAGES MIDTERM EVALUATION REPORT USAID.GOV The findings from the in-depth reviews provided data on the type and quantity of SBC interventions already in play, and gaps. To improve communication through the outreach system, LINKAGES has worked closely with the PAC, the national VCT Counsellors Association, WHO, Carlo Clinic and others to provide training addressing areas of weakness identified during periodic assessment against the technical thresholds. Training focuses on increasing knowledge and communication skills among outreach workers to motivate and up skill them. Capacity building has provided participants with skills and knowledge, as well as tools, to improve performance. Key informants report that they appreciate the training and can see how it is relevant to their role, and understand how expanding their skills can result in opening up opportunities for KP and PP. SUPPORTING KP ENGAGEMENT The project is enhancing KP engagement by introducing Implementation Tools for people who use drugs. LINKAGES International Partners have developed a new practical guidance for collaborative interventions on implementing comprehensive HIV and HCV programs with people who use drugs (PWID). The PWID implementation tool has been disseminated to CSO and other key stakeholders. Nine out of 10 interventions cited in the guidelines are consistent with the Ministry of Health Decree No. 55/2015 regarding harm reduction programing in Indonesia; the exception is the use of naloxone following an overdose. The new guideline emphasizes the empowerment of the PWID community and their central involvement in policy and program planning and implementation. LINKAGES is adapting the new practical guidance to the Indonesian context and facilitating key stakeholders support from the MoH, UNAIDS, UNODC, WHO, Spiritia and the Indonesia Network of People Who Use Drugs (PKNI), who were also involved in developing the original UNODC Guidance. This work is in the early stages, and activities to empower PWID and prevent HIV infection among them will be conducted in FY-18. The project is enhancing demand by addressing stigma and discrimination (S&D) by health care workers. The level of S&D in health care settings is reportedly high and many instances were witnessed by the evaluation team. For example, a HIV-positive person receiving treatment through a puskesmas in Jayapura recounted to the evaluator how they were regularly advised by one of their health care providers to ‘stop being gay’. He has to see this health care worker every four weeks to get treatment. At an outreach site visit a young MSM said he would never go near a clinic for a HIV test, in case the workers reported him to his family or workplace, as had happened to his friend. Transgender women get referred to as ‘men’ or grouped with gay men by some service providers. Also, in Papua, one of the CSO Outreach workers disclosed during a Focus Group Discussion with the evaluation team that they were unable to understand why a man would want to be with a man, that he struggled with this, but he still delivers services to MSM, and welcomes the training provided by LINKAGES. In year 1 LINKAGES supported the Ministry of Health to update the National Training Module for Stigma and Discrimination Reduction Among Health Care Workers, following participation in the Asian Regional Consultation on Addressing HIV-Related Stigma and Discrimination in Healthcare Settings, held in May 2017 in Bangkok. Since then LINKAGES has adapted consortium partner IntraHealth’ S Health Care Worker KP Stigma and Discrimination Reduction Training Guide for Indonesia. The National S and D Facilitators have been introduced to the new materials and a training of trainers for selected GOI trainers has just been held in Jakarta. The project is currently working with IntraHealth and the Jakarta PHO to roll out this training. Training of Trainers in Papua will follow and the MOH is sourcing funds to roll out the training in other provinces. USAID.GOV LINKAGES MIDTERM EVALUATION REPORT | 22 LINKAGES is seeking to increase demand by protecting human rights for KP, documenting abuses and increasing access to legal services, most of whom have not focused on human rights for KP in the past. The project found that KP Networks have limited resources and are unable to devote consistent resources to identifying and documenting the human rights abuses and challenges that KP face when accessing health and other services. LINKAGES has supported the Indonesian AIDS Coalition (IAC) to hold a series of community working group meetings, develop an advocacy plan, and to appoint and support five legal aid workers (one for each of the five Jakarta districts). The workers have only recently been appointed. Their role is to forge links between KP organizations and mainstream legal and human rights services and monitor legal cases, in collaboration with the Asia Foundation. The interventions identified in the advocacy plan were included in the funding request recently approved by the GFATM and will be rolled out in 2018. Indonesia will receive $1.9 Million to strengthen community strengthening in 23 districts and $ 2 million and $ 2.7 million (catalytic funding) for reducing human rights barriers. SUPPORTING KP BY DEVELOPING VIOLENCE PREVENTION AND RESPONSE INITIATIVES PEPFAR and WHO guidance notes indicate that all KPs reached through LINKAGES programs should be screened for violence. However, screening can be detrimental if care and support services are not available. Accordingly, in Year 1 LINKAGES conducted an analysis of gender-based violence (GBV) service responses in the health system in Papua to identify appropriate services for survivors and inform the development of future programming. Rates of gender-based violence are reportedly high in Papua, and LINKAGES had already established relationships with two hospitals in Jayapura and other services prepared to work on improving the service system. Early investigations in Papua by LINKAGES identified many barriers such as the lack of a medical protocol to guide service delivery, few trained providers and long waiting times, and no regulations to enable the provision of HIV post exposure prophylaxis (PEP) for rape survivors. The DHO, hospital management and all level of health providers expressed support for addressing GBV within the health system. (Ramsey, GBV assessment and consultation trip report to Papua, 2016) LINKAGES has since worked with GOI stakeholders to create GBV protocols and standard operating procedures, develop a GBV clinical services training package for health care workers and conduct a training of providers at three facilities and supported quality assurance through “supportive supervision for GBV services”. The training covered the specific needs and experiences of KP. LINKAGES has since worked with the USAID funded Bersama project to work with communities on GBV awareness and prevention. Bersama is setting up an office in Jayapura, identifying local staff and partners and identifying intervention sites. Their work has built on advocacy and referral system work completed under an earlier project (KINERJA) and focused on strengthening the coordination between local and national level key government bodies, engaging target communities to change harmful gender norms which give rise to GBV and strengthening civil society organizations to better advocate for GBV services. (Ramsey M. , 2016) The future project efforts are expanding beyond GBV response to encompass the prevention of violence against all KP regardless of gender, and appropriate service responses. CONCLUSIONS The project has successfully included reach and test parameters into the HIV cascade, and this is focusing attention on the need to create demand for a range of HIV prevention interventions, in order to achieve testing targets. The project is the only TA provider actively working on differentiated social and behavior change communication interventions, across a range of platforms, especially in Jakarta The demand creation activities are designed to complement the decentralization activities that are making more 23 | LINKAGES MIDTERM EVALUATION REPORT USAID.GOV services available. The social and behavior change communication activities are facilitating the introduction of differentiated approaches, and expanding access to on-line outreach and personalized risk assessment, which is likely to be especially useful for reaching MSM and sex workers. The evaluation concludes that key Jakarta program stakeholders are reluctant to accept a differentiated support model for KP, and intensive activation will be needed. Fears that new systems will be embraced without evidence of their effectiveness need to be alleviated; however, LINKAGES should not back down from innovation. Demand creation for new outreach and support methods, online risk self-assessment tools and referral systems will need to be carefully monitored and the outcomes shared with Jakarta Fast track team members and others. LINKAGES needs to be prepared to scale up what works and let go of what doesn’t in a timely fashion, to make the best use of the final two years of the project. Jakarta studies with MSM are creating demand for PrEP, and there was a high level of awareness of PREP, the challenges with making it available and potential benefits among MSM groups. Discussion between the evaluators and DSD supported CSO indicate the TA received is enabling the staff to increase their HIV testing yields. They were proud of their efforts to reach targets and beginning to understand how to use their data to be more effective. There is some indication that demand for VMMC is growing in Papua, although it is not clear if this is occurring among the most at risk men, or if demand creation is strategically focused. The project has played a lead role in developing policy and plans related VMMC. While this activity is useful, it should not be seen as any sort of panacea for the generalized epidemic in Papua. The key barriers to demand creation, including low awareness of HIV and how to prevent it, religious decrees prohibiting the use of condoms, and high levels of stigma will require intensive attention. Treatment resistance is likely to be an issue in Papua over the medium to long term, as many PLHIV experience many treatment interruptions. Demand for health system strengthening and further decentralization of HIV services is low, and further demand creation activities targeting PLHIV and PP are needed. PEPFAR 3.0 lays out a data-driven approach that stresses evidence based interventions reaching the highest risk populations in areas of highest HIV incidence. This “epidemic control” model relies on systems that ensure KP move efficiently through the entire HIV continuum of prevention, care, and treatment (CoPCT) services. This is not the case in Indonesia, and there is limited investigation of the barriers faced, as providers seem to prefer to rely on untested assumptions and anecdotes. Tragically, health services turn people away if they present for testing outside prescribed times. They also fail to utilize testing opportunities to establish re-call systems that retain KP, have no tracked referral systems ensuring PLHIV are enrolled in care and have no systems to monitor routine causes of loss of engagement among KP across the CoPCT. Demand creation in settings such as TB treatment sites, prisons and drug treatment services also seem to be under-utilized. The national program supports re-engagement with HIV-negative KP at a policy level, but re-engagement is not monitored or easily tracked. SIHA can’t nuance the cohort data to show first versus repeat testers or identify actual yield among new clients. Better data could enhance understanding of demand gaps and what is working. The project is working with USAID to construct a cohort cascade to track movement across the system over time. The project could incorporate this work into the leadership development training, so that those who are genuinely interested in making sue of the data could be included in generating it and understand how it is constructed. USAID.GOV LINKAGES MIDTERM EVALUATION REPORT | 24 To achieve a seamless integration of interventions requires strong linkages to enhance and sustain demand. However, the links between program elements in Indonesia are not well established, the barriers to entry have not been sustainably addressed. The national program has not yet conducted an in-depth analysis of what is needed to ensure health services are well linked. LINKAGES is well placed to conduct patient journey maps and other studies that could inform program efforts in Papua to increase demand. Introducing reach and testing targets has gone some way to increasing demand, as CSO staff strive to reach their targets. In Papua the evaluation team visited CSO where leader boards were displayed, showing how many people each community-based worker had taken for HIV testing, and how many tested positive. This was a source of much pride, and the CSO staff were able to discuss their strategies for identifying new hot spots and making the best use of their time. However, they may be even more motivated if they understood how the targets are set. If they were more actively involved in setting District and their own targets, they may be more invested in achieving results, and a greater understanding of the local epidemic could foster a sense of urgency in getting PLHIV on treatment. The evaluators witnessed very high levels of compassion, dedication and motivation among the CSO staff, and solid leadership, but found that there are too many small CSO with limited reach, capacity, and potential for rapid scale up. Some economies of scale are needed to reach people in numbers sufficient to make a difference. CSO ‘back office’ and administrative functions could potentially be centralized for a number of CSO, freeing them up to increase their capacity to plan, implement and monitor high return strategies. RECOMMENDATIONS To enhance and sustain demand among KP and PP, across the Continuum of Care, the following actions are recommended: 1. To support decentralization and increase demand for services, investigate the mobility of ART patients and see how much demand in Jakarta and Jayapura is generated from people living outside the catchment area. Use this information to support Districts to rationalize and optimize patient loads and reduce travel for KP. a. Map home versus place of treatment and identify opportunities for high load clinics to transfer established patients out to new sites and mechanisms for doing this safely b. Study newly diagnosed PLHIV at selected facilities and follow this cohort to determine specifically how they can be retained in or lost to treatment systems c. Analysis of official registered place of residence (using information on NIK or JKN card) vs. location of ART initiation and continuation using existing patient-level SIHA data (offline site-level SIHA) at selected sites 2. Demand creation in Papua is hampered by poor coordination, and bottlenecks at key decision￾making points. Leadership is concentrated among the CSO and faith based organizations. Continue to explore above site-level opportunities to ensure changes in TA provision do not result in the active leaders in the response no longer being linked to GOI services, and not sufficiently resourced to continue to provide outreach and support services. 3. Demand creation is also hampered by limited knowledge of prevention and other programming options among the PP and other key stakeholders. Conduct studies that generate greater 25 | LINKAGES MIDTERM EVALUATION REPORT USAID.GOV understanding of the PP and their movements, health seeking behavior and patient journeys in Papua, to inform program planning and campaign design. 4. Continue to develop the links between KP networks, CSO and legal and Human Rights organizations, and build capacity to ensure case files and other data can be effectively used for advocacy and to gain local and international support for law reform and further protection of the rights of KP. 5. Build on the training to reduce stigma and discrimination in health care settings to reduce S&D among TB service providers, and other health care professional engaged in HIV testing and treatment. 6. Due to the resistance to introducing differentiated outreach and support services, they may need to be packaged and heavily promoted, impact monitored, and results widely shared to get buy in. Regardless of levels of support for new ideas, LINKAGES must continue to undertake this important work with TA partners. Wherever possible, timely cross-sectional cascades should be generated to assesses overall performance for each specified KP or subpopulation within each District to demonstrate successful interventions. 7. Long-term leadership and pressure for improvement in the national HIV program will come from the KP networks and CSO, therefore the project should invest in leadership development and capacity assessment and building for CSO. KP Network leaders must be supported to engage in coordination and planning forums, and to continue to develop the capacity to drive the national response. 8. Use DOKLING and other methods to strengthen facility-facility referral linkages and/or to organize VL specimen transport 9. Advocate for policy level approval to allow self –testing, online access to risk assessment and self￾referral for testing and monitor the impact this has on demand for testing and treatment. RESULT AREA 3: STRENGTHENING SYSTEMS, ORGANIZATIONAL AND TECHNICAL CAPACITY In Results Area 3, LINKAGES work across the PEPFAR efficiency, sustainability and partnership agendas to: • Deepen coordination, and advance planning for a joint implementation of the Jakarta Fast Track Roadmap and for a more comprehensive response to HIV in Papua. • Strengthening monitoring and evaluation (M&E) and strategic information (SI) systems that incorporate regular reporting, coordination and feedback mechanisms. • Advance technical, programmatic and financial performance for transitioning and sustainability approach that can be implemented at site and above-site levels. STRENGTHENING THE JAKARTA FAST TRACK INITIATIVE USAID/LINKAGES have partnered with UNAIDS and WHO to form the Fast Track Core Group Technical Assistance Team, collaborating with the Jakarta Provincial Health Office. LINKAGES also USAID.GOV LINKAGES MIDTERM EVALUATION REPORT | 26 provide financial and programmatic assistance. Over the last two years the strengthening activities have included: • Developing the Fast Track Road Map, including a shared set of Fast Track targets at the Provincial, District and site level, so that it is clear how many KP need to be reached and tested every month to meet the annual targets. • Developing one shared set of common progress measures across the cascade that all stakeholders work towards. • Developing one coordination body with responsibilities identified at site, district, and province levels. • Developing one shared One Fast Track action plan outlining specific actions to be taken at the Provincial and District levels. • Providing ideas and evidence and hosting or contributing to a range of forums to enable decision makers to focus on identifying, introducing and monitoring the impact of interventions. • Following up with templates and planning support at District, and more recently at site level, including providing health staff with access to the Action Plan template to work out how they will reach their targets, and the Monitoring Dashboards to assess their progress. • Developing a district-level technical performance dashboard template to be used by fast-track implementers. • Providing TA to the Provincial AIDS Commission to conduct a KP behavior survey. • Providing financial support and technical assistance to the Ministry of Health (MOH) to conduct Asian Epidemic Modelling (AEM), develop KP and PLHIV projected numbers over time and ensuring the data is used as a basis for an investment case analysis; and to conduct the 2017 Integrated Biological and Behavioral Surveillance (IBBS). • Provided technical, financial and logistical assistance to prepare the new funding request to the Global Fund for HIV, TB and Malaria for the period 2018-2020. This includes contributions to 10 GFATM sub-working groups; helping partners to articulate CoPTC intervention packages for high, medium and low burden HIV districts; and holding community consultations to inform the funding request. A District Manager from Jakarta said USAID funded initiatives often build on GOI priorities, and the input provided by the project helps with progressing how to implement desired program changes or make them more effective. For example, the National HIV Program had prioritized increasing access to treatment for PLHIV, and the TA provided by LINKAGES has allowed the program to scale up the number of puskesmas that can dispense ART and widen the coverage. The Ministry of Health HIV Program manager said that Fast Track and the National Program both seek to find and treat PLHIV through the national SUFA policy, but the key difference is that Jakarta has targets and regular coordination meetings where everyone uses the data, and this pushes people, and accelerates the response in Jakarta. She stated: “LINKAGES work with the facilities to generate solutions that can also be used outside Jakarta to accelerate the response, like developing DOKLING for mobile testing, which the facilities want to be involved in developing, because they are most familiar with the local setting.” CONCLUSIONS LINKAGES has provided extensive input into developing key foundational documents, processes and tools for progressing The Jakarta Fast Track Initiative (JFTI). The shared vision, with concrete targets, is now 27 | LINKAGES MIDTERM EVALUATION REPORT USAID.GOV articulated in Fast Track Road Map (August 2016), and the Jakarta Fast Track Action Plan (May 2017) and during regular planning and review forums. LINKAGES has played a central role in achieving this level of planning and consensus. The evaluation concludes that the project has played a key role in intensifying and maintaining the focus on the locations and populations most affected by HIV. Senior GOI informants responsible for managing HIV programs at the Province and District level in Jakarta spoke with evaluators about the important role that LINKAGES has played in building capacity, especially in relation to using data to support decision making, accelerating the roll out of treatment services into additional primary health care centers and ‘keeping the focus on KP, when we have many other things to think about in the program.’ The EOA is increasing reach, and helping to drive a focus on reaching KP and TB patients. Health Program Managers and puskesmas staff said the Fast Track Targets provide the impetus for increasing reach and testing efforts, and the DOKLING system was the central tool that has increased mobile testing, because now it is so easy to coordinate and schedule mobile testing. The contribution made by LINKAGES to the JFTI is highly appreciated by the GOI. A key management level stakeholder stated “We are getting good data now from puskesmas, with the support from LINKAGES to the health facilities as well as the District and provincial Health Office; it is much appreciated. M and E, data validation, and training for recording and reporting has been a priority, and we see the result in our improved records.” They went on to say they rely on LINKAGES for TA not only because the project funds activities the national budget can’t extend to, but also because they are more focused on the KP and provide technical expertise not available otherwise. GOI key informants also spoke about their concern for maritime and transport workers and noted the extent to which non-KP are affected and require interventions. This tendency to move the conversation away from the KP was noted during a range of interviews with GOI stakeholders and reinforced the need for LINKAGES to continue to have a presence and keep the focus on KP. The Fast Track Initiative Coordination Forums are critically important because they provide a regular opportunity to bring people together with an explicit focus on accelerating the response, optimizing the HIV prevention and care environment and overcoming the legal, social, environmental, and structural barriers that limit access to the full range of services across the HIV CoPCT. The project is centrally involved in securing multi-stakeholder engagement, introducing diversified strategies and maintaining a commitment to enabling KP, including PLHIV, to take on leadership roles. The project can best sustain the focus on KP by continuing to strengthen the KP leadership and secure their participation in decision making systems. The main concern raised repeatedly by stakeholders in Jakarta relates to the level of political leadership and commitment they can expect for the Fast Track Initiative in the future. Many people are wondering whether the incoming Governor will support the Fast Track Initiative to the same extent as the out-going Governor, who signed Jakarta up as a Fast Track City. In the coming year, LINKAGES will need to work closely with the other Fast Track partners to ensure the promised efforts to rid Jakarta of all sex work do not result a worsening of the epidemic and even greater difficulty in reaching sex workers. Actions designed to “act firmly against all places that violate regulations” (Wijaya, 2017) will require close monitoring, and advocacy based on important lessons learned from when the Jakarta administration shut down the brothel district of Kalijodo in North and West Jakarta in 2016. LINKAGES foundational activities USAID.GOV LINKAGES MIDTERM EVALUATION REPORT | 28 related to collating, cleaning, curating and using data will be critical. The project can continue to contribute by ensuring CSO have the skills, expertise and resources to continue to identify hot spots and understand sex work arrangements and migration patterns that contribute to risk. Background studies on KP mobility, barriers to obtaining services and preferred outreach and social and behavior change will be needed to inform ongoing program evolution in Jakarta. It is critical that LINKAGES take a bold and proactive stance in the Fast Track forums, and encourages participants to go beyond coordination and step up as leaders, willing to advocate for legal and policy reform and provide stewardship of the limited resources. LINKAGES has taken several important steps in addressing the limited leadership: by mapping stakeholders; coordinating a Workflow, Data Flow and Client Flow Workshop that supports decision making and through the Performance Against Fast Track Targets, 2016-2017 activity. To further strengthen leadership, the stakeholder mapping exercise could drill deeper to allow LINKAGES to understand what drives and motivates people in positions of influence to act (or not act), understand the alliances and networks that exist and what shapes them, identify opportunities for expanding the projects sphere of influence and leverage existing resources. Strengthening the capacity of GOI Fast Track coordinators can be time consuming and frustrating, as many GOI staff are re-assigned every two years and spend only a short part of their career working on HIV. Despite many capacity building activities, high staff turnover regularly drains the leadership potential of decision makers. The project may therefore need to focus more explicitly on building the capacity for and incentivizing leadership. Leaders will need to be made, and quickly, if they are to have a positive impact during the short period of their careers spent in HIV program management and coordination roles. This can occur at a number of levels, through leadership development programs, and other activities designed to assist people to see what needs to be done and then take action, overcome a pervasive sense of powerless witnessed by the evaluators and give people confidence to step up. For example, LINKAGES could invite potential leaders to co-design and co-lead specific events or activities with them; or remove the decision-making risk by providing detailed policy briefs that review the pros and cons of a range of actions; or easing people into leadership roles with increasing levels of responsibility and tailored support. TA implementation arrangements between LINKAGES, the other Fast Track Initiative partners and GFATM principal recipients started out as strategically ad hoc, and the process for initiating TA activities was not clearly defined. The LINKAGES Project Manager described how the project has repeatedly followed the formulae of helping the national program to see what they have (in terms of systems, data and resources), and then helped them build on it, make it more effective or user friendly, and monitor progress. This has worked very well, and it’s clear that LINKAGES has seized as well as created opportunities. The project offers training and capacity building, and participants come, but there is no commitment from the recipient agencies to allow people to make changes in the workplace based on what they are learning. Likewise, there are no consequences when participants fail to convert training into demonstrably increased capacity. Therefore, the project will move to a more strategic approach guided by a Scope of Work developed with the PHO and DHO, which proactively addresses the barriers to ensuring that capacity building efforts are transferring skills and changing practice. STRENGTHENING COORDINATION, CAPACITY, AND PLANNING IN PAPUA LINKAGES has supported Papua PHO to articulate annual 2017 targets for the Papua comprehensive response, using program and epidemiologic data as references. A series of workshops bringing together academics, epidemiologists and local program staff have been held to ensure the decision makers are fully 29 | LINKAGES MIDTERM EVALUATION REPORT USAID.GOV familiar with and in agreement over the Papua data. Capacity building is ongoing to familiarise local stakeholders with how population sizes are estimated, and targets are set. Prior to these workshops, decision makers did not have the information at their disposal to allow them to fully understand the epidemic in their province or allow them to monitor progress. The project has introduced the Cascade Generator and upskilling partners to generate and make use of Dashboards. Many HIV facilities have difficulty contributing timely data to SIHA, due to limited or unreliable infrastructure and staff. The project has placed time-limited personnel in several key GOI locations, and they are entering data. The increased availability of data and the use of dashboards is contributing to a more strategic and focused HIV program approach in Papua. The KPA District office in Jayawijaya report that coordination and program management is at an all-time low. They are suffering from an absence of leadership, senior staff and missed opportunities. For example, stigma and ignorance about HIV is a big barrier to increasing demand, and the District Office of Information and Communication could be engaged in addressing this, but they have not been coordinated into the HIV program. The Strategic Plan for HIV expired in 2014 and has not been updated and there has been no decree in 2017 appointing staff to key positions, and no budget allocation for the Provincial HIV Program. Staff turnover is quite high due to low salaries, and the KPA is seen as a stepping stone to securing another government position. The remaining KPA staff report that key positions are earmarked for local (Papuans) but many positions are unfilled, or the incumbents are challenged in fulfilling the duties of their roles. As a result of these challenges, coordination meetings are not occurring. The lack of coordination and planning mechanisms has implications locally and nationally: “We don’t know who is doing what, reporting is difficult. No one at management level is analyzing the reports. The services and NGO submit their reports but then nothing happens to them. Some services continue their work, like distributing condoms, but this hamper planning and progress.” It is difficult to assess the quality of treatment in Papua as there is only one Viral Load machine for HIV in the whole Papua province. The project is establishing VL testing technical thresholds and working with the Supply Chain and Challenge TB programs to expand the use of the TB program’s GeneXpert machines. CONCLUSIONS The project has bought together key experts and decision makers in coordination, planning and data review workshops. As a result, people in key positions have a better understanding of the epidemic and an appreciation of the need to coordinate and strengthen planning. Strategic planning for the HIV program has been completed in Mimika, and the other Districts are requesting TA to undertake planning. This is a critical activity, in light of the lack of leadership and isolated nature of the work context. Program sustainability is an issue, as Papua does not have a clear, comprehensive plan or accountability systems in Papua. There is limited ability to tap into existing resources and strategically allocate them, and many people in leadership positions have had limited exposure to comprehensive programming for HIV, so they are at a loss as to what to do or how to assess opportunity cost and suitability. However, willingness to implement seems high, if some direction were provided. The project could assist with above site level TA to bring stakeholders together to develop District Level Strategic Plans linked to work plans for implementing ARV Decentralization that address the very low ART coverage rate and high loss to follow-up rates. In the absence of functioning coordination mechanisms driven by the GOI, the project will need to play a lead role in developing these plans and mapping accountabilities. USAID.GOV LINKAGES MIDTERM EVALUATION REPORT | 30 The project has played a key role in increasing capacity to implement decentralization, which is bringing services closer to the community by enabling PLHIV to access treatment at the Puskesmas level, instead of only via key hospital sites. However further decentralization and scaling up access to services requires the involvement and commitment of all stakeholders and considerable effort to ensure the facilities and staff are ready to supply treatment. The decentralization process was done in several stages over a year due to the limited of facilities, infrastructure and human resources, and now there are systems in place for stocking the puskesmas with test kits and treatment drugs. During visits to sites the evaluation team observed signage advertising the availability of ART and SOP and guidelines for the service providers. Considerable clinical mentoring is being provided across the province, and all sites reported regular contact with the mentors. The Wamena puskesmas reported that although they started HIV testing in 2007 they still have limited capacity for testing priority populations bought in by the CSO staff. They screen pregnant women on Monday and Thursday and so will not screen other people on those days, said they limit the number of tests they conduct each day ‘for quality assurance reasons’. They have four Lab technicians. Once a month they now meet with the District health office to share data with them and discuss how they are going. Together with the project mentors, they look at their testing results among antenatal patients, TB patients, and others, but none of the staff interviewed were able to articulate any particular expected level of performance, or any discussions about how to ensure the limited testing resources are used most effectively. They have days set aside for testing different types of patients, and that was to manage the workload, not to promote active strategic efforts for best case finding results. The staff report that there are no consequences for losing a patient after they have been diagnosed as having HIV through either Provider Initiated Testing and Counselling (PITC) or through voluntary counselling and testing. If a PLHIV is not enrolled in care after getting a positive test result someone from the clinics will try to ring them once, but there is no rigorous system for follow up. All of these issues require TA to resolve. Links between interventions for KP are frequently inadequate at every stage of the HIV continuum of prevention, care, and treatment. Further understanding of the gaps and experiences of KP and PP is needed, and interventions that address the low levels of knowledge of HIV, prevention options and myths that result in PLHIV being ostracized by their families and communities. The project could benefit the national program and Papua based services by generating greater understanding of what drives demand, influences health seeking behaviour and supports decision making. It may be possible to do more, and more effective demand creation work by developing a deeper and more nuanced understanding of the KP and PP, and how they want to engage with the CoPCT. This is especially needed in Papua, where there are many assumptions or anecdotes about why demand is so low and loss to follow up rates are so high. The evaluation team was told repeatedly by CSO and puskesmas and GOI staff in Papua that the sex workers and PLHIV are highly mobile and change their phone numbers often and walk off into the highlands and are not seen for months. However, these assumptions and anecdotes have not been tested, and there is no evidence drawn from the KP and PP to inform understanding of the barriers and health decision making from their perspective. When asked about how other chronic disease programs reach and retain patients, for example diabetics, the staff working on HIV had no idea, and again resorted to assumptions. It was clear there was no communication between programs, or application of lessons learned by other health program areas. To date, there have been no systematic studies of KP and PP movements, and greater understanding s needed of the patients’ journey through the system, the decision points and barriers faced by PLHIV, and 31 | LINKAGES MIDTERM EVALUATION REPORT USAID.GOV consequently there is limited informed understanding of what the patient faces at each stage across the CoPCT. Service providers can only guess at the barriers and considerations KP are faced with at each stage that may affect access. One of the highlights of field site visits in Papua was seeing the CSO displaying their Capacity Assessment scores and leader boards, showing how many PP were being reached, tested and linked to HIV treatment by each of the community-based staff. The project’s efforts to increase Technical, Programmatic and Financial Performance across the CSO partners has resulted in a paradigm shift, from discussing activities to discussing capacity and results. The project conducted intensive, week long training in 2017 with all CSO partners in Papua, and the partners were clearly well oriented to the project, and actively discussing the implications of targets, results-based monitoring for payments and technical threshold tracking. The project has measurably improved capacity and embedded a sense of responsibility for results among the CSO partners. As a result of the project’s TA, CSO have a clear understanding of their current capacity and plans for strengthening organisational, financial and technical capacity. They have worked alongside LINKAGES to establish formal referral systems and agreements are now in place in the three Districts. Unfortunately, the coverage of small CSO remains limited, and it is still very difficult to increase reach and coverage to the small highland villages. Mechanisms for increasing access to community-based testing require further exploration. The project needs to work with local stakeholders to identify opportunities to integrate HIV and TB testing into other regular health screening or outreach. The HIV program did not appear to be linked to or learning from other areas of the health system including non-communicable disease or maternal and child health programs. There may be benefits to greater collaboration, and these need to be explored. STRENGTHENING M&E AND SI SYSTEMS Coordination and planning of the National Program has been hampered by incomplete and unreliable data, the absence of evidence to support advocacy and decision making and insufficient coordination and planning forums where program impact and progress is monitored. LINKAGES has conducted activities and developed mechanisms to address each of these challenges and strengthen the systems underpinning the national program. LINKAGES SI support activities include: • Providing support for the analysis of the 2015 IBBS data. • Placing time-bound personnel in key locations where they reduce data gaps and re-invigorate data collection systems. • Providing formal training and embedding utilization of LINKAGES-developed guidance, for example the SIHA verification tool. • Introducing data visualization tools, including the cascade generator, to facilitate data use and promote an understanding of how key populations move – or don’t move – through a health service continuum of care by analyzing four SIHA indicators to construct cross-sectional cascades for new clients and two SIHA indicators to construct cross-sectional cascades for current clients. • Developing district- and site-level CoPCT baseline cascades. • Rolling out of new M&E forms and systems to document reach and test activities. • Establishing 16 site performance thresholds to track quality improvement measures against 90- 90-90 goals. USAID.GOV LINKAGES MIDTERM EVALUATION REPORT | 32 During this programmatic period, the project drafted Terms of Reference for M&E personnel and is currently working with district and provincial GOI bodies to place individuals in these positions, including the placement of a dedicated IT consultant to support SIHA at the national level. The program further supported four Papua-based M&E personnel to participate in a SIHA data validation consultation in November 2015 and worked with the PAC to help the institution assume full responsibility of the Jakarta AIDS Information System (JAIS) in December 2015. To strengthen size estimation, surveillance and epidemiological modeling, LINKAGES partnered with the USAID/DERAP program to support the facilitation of a city-wide key population mapping exercise in March 2016. While the exercise yielded critical information about the size and extent of outreach coverage, questions remained about key population size estimates and the denominators that feed into Jakarta 90-90-90 target setting. In Year 2, the project worked closely with UNAIDS and the MOH to gain consensus around key population size estimation figures for Jakarta. This provided the foundation information for the Jakarta Fast Track Initiative targets. Key informants from the Office of the Sub-directorate of HIV and AIDS and STIs of the Ministry of Health described how they send a quarterly performance report to each province, using data from SIHA, and expect the managers to address any problems. Program staff report that there is no systematic monitoring of what action, if any, is taken at the province level. There are no consequences if the reports show significant breakdowns in the cascade. The national program sends out a mentoring team, twice yearly to most places. In discussion with the PHO and PAC, LINKAGES placed six time-bound Fast Track contractors in five DHOs and one PAC. The project developed job descriptions and scopes of work to guide all the time bound and short-term consultants. Interviews with Health Office staff and indicate that they are appreciative of the consultants and the role they play, especially as staffing shortages are critical and without the time bound consultants a lot of data would not be entered and uploaded into SIHA online, especially in Papua. Without the data, decision making is hampered. At several sites in Papua government staff reported they ‘get overwhelmed at times’, and that ‘services close to town get swamped’. The regular capacity building and coordination meetings with LINKAGES provides them with an opportunity to reflect on their capacity, think about the most recent challenges and how they are currently dealing with them. District Health Office and puskesmas staff report that discussions about planning rarely take place in the absence of LINKAGES staff. CONCLUSIONS SIHA data quality and completeness is improving as a result of the project. The data verification processes and regular data review sessions facilitated by the project are contributing to this improvement However, some is due to LINKAGES time-bound personnel entering and cleaning the data. Although this is useful, especially in Papua, it is not sustainable. Cascade analyses highlight the need for time-bound M&E staff at district and provincial levels to spend less time entering data and more time cultivating a culture of continual data use. Data collection, entry and cleaning tasks are onerous, and much of the data does not seem to be used. It’s easy to see why it is a low priority for health care providers. There are on-line and off-line data collection systems, and double data entry in the paper and on-line as services keep their own patient records and spread sheets and then upload data monthly to SIHA for reporting purposes only. SIHA has 33 | LINKAGES MIDTERM EVALUATION REPORT USAID.GOV eleven standardized and mandatory indicators which translate to over a hundred data entry points due to the subsets. Instead of rationalizing and using what is already in the system, new indicators are still being added. The Cascade generator has curated the data and presents it in a highly visual, easy to read dashboard. This is demonstrating what can be done with a much less burdensome set of indicators. The project could help to further promote data functionality by consulting clinical and program management staff at site level to see which indicators, they would find useful for program improvement if the data could be presented visually. The DOKLING system is providing the program with useful testing cascade data. The recent results show that yield is higher at sites using the Enhanced Outreach Approach, for example testing yield was ~2% among MSM in sites not supported by LINKAGES, compared to 17% amongst project sites. The project should establish mechanisms to assess the performance and effectiveness of LINKAGES’ technical assistance that could highlight any differences in sites receiving TA when compared to other sites (e.g. % tested for HIV among reached, ART linkage rate among newly diagnosed PLHIV, rate of ART initiation). This could serve as a “proxy” in measuring how “effective” LINKAGES is in facilitating 90-90-90 in comparison to the national program. This would help to shift the project’s focus over the next two years from reporting data (within SIHA) and generating dashboard reports to regular analysis and use of data at the site level. RECOMMENDATIONS SI recommendations: 1. Assist in improving and streamlining of current patient data flow at LINKAGES supported facilities. At minimum, the following should be considered: a. A detailed assessment of patient data flow and the associated health information systems used at selected high-volume ART health facilities. This includes a thorough mapping of the individual patient and aggregated data flow, identification of duplications in data collection and entry, and inefficiencies/challenges in the various data management systems used to generate SIHA offline (individual-level) and SIHA aggregated data b. Modification of the current data management tool (an MS Excel-based “aggregated data file”) used by Ruang Carlos clinic to include advanced functions and formulas for data aggregation and analysis 2. Provide technical direction and support on the improvement of SIHA Offline. At minimum, the following should be considered: a. Documentation and analysis of the most common SIHA Offline issues found across ART facilities (i.e. tracking of frequently asked questions via SIHA telephone hotline) in order to assess system vulnerabilities and weaknesses in the current SIHA Offline version b. Provide structural modifications/enhancements to the existing version of SIHA, including the following: USAID.GOV LINKAGES MIDTERM EVALUATION REPORT | 34 i. Auto-calculation and auto-population of aggregated SIHA Offline results of ART report [into site-level SIHA reports that are submitted to the MoH] once individual￾level data is entered into SIHA Offline ii. Create automatic data quality/logic checks into SIHA (e.g. flash box to alert data entry clerk when numbers entered for newly initiated on ART is greater than then total on ART in the reporting period) c. Assess reporting completeness and frequency of data use for the existing “mandatory” MoH HIV indicators in SIHA across all ART facilities in Jakarta as an advocacy tool/evidence for streamlining the number of “core” indicators for the national HIV program 3. The project should establish mechanisms to assess the performance and effectiveness of LINKAGES’ technical assistance that could highlight any differences in sites receiving TA when compared to other sites. 4. Greater engagement of LINKAGES SI team in strengthening the capacity of PHO, DHO, and ART facility SI staff on data use and analysis is needed. In addition to generating monthly cascade dashboards, LINKAGES should facilitate the following: a. In collaboration with PHO, DHO, and facility staff in Jakarta and Papua, monthly analysis of site-level and aggregated level clinical data should be conducted to assess performance and investigate data quality issues reported in SIHA b. Quarterly meetings with DHO and PHO should be better structured, with the primary intent of discussing SIHA results and trend data for the reported period. Technical presentations on strategic information can also be conducted periodically and should be considered a core component of these meetings c. Intentional knowledge transfer should take place for the LINKAGES M&E secondment to the PHO/DHO through a mentorship pairing system where the LINKAGES M&E secondment is assigned as a “mentor” to one or two public servant (SI staff) at their seconded office, where concrete capacity building plans should be developed and evaluated annually. 5. Modification of existing data collection system to demonstrate “different models/configurations” of services piloted in the Enhanced Outreach Approach (EOA) in order to assess the effectiveness and uptake of each “model”. As such, a minimum of the following modalities of services should be considered for data collection and analysis: a. Proportion of online outreach vs. venue-based outreach b. Proportion of completed testing referral among those reached online vs. venues c. Segmentation of assessed risk behaviors (via risk assessment form) among KP reached by age and by outreach modalities (i.e. online vs. venue) d. Percent HIV positivity yield among those tested as a result of online outreach vs. venue-based outreach 35 | LINKAGES MIDTERM EVALUATION REPORT USAID.GOV e. Mapping of peer recruitment network for HIV testing (“referral chain network”) and HIV positivity rates among these social networks 6. Strengthen the capacity of HIV advocacy networks (i.e. Indonesia AIDS coalition) in knowledge management, including a. The provision of technical assistance in the systematic documentation of advocacy and technical briefs; and b. The use and analysis of existing epidemiological, costing, and program data for advocacy with the government 7. Operational research to assess the mobility of ART patient should be considered. Study can be conducted using two different methods in order to triangulate results. a. Random sampling of newly initiated ART patients at selected facilities within a 3 month recruitment period and follow this cohort for 6 months to determine where ART clients are accessing ART b. Analysis of official registered place of residence (using information on NIK or JKN card) vs. location of ART initiation and continuation using existing patient-level SIHA data (offline site￾level SIHA) at selected sites. STRENGTHENING SYSTEMS FOR ASSURING QUALITY OF SERVICE DELIVERY PROGRAMS The national program is negatively impacted by a myriad of conflicting regulations, decrees and policies, many of which the national program and the MOH are powerless to change. The MOH provides no continuing medical education, and robust accountability systems are also missing, which can leave staff unsure of their scope of practice. HIV program coordination meetings are rarely attended by management level staff. Therefore, the project needs to focus on influencing policy and strengthening systems and building the capacity of health care workers at all levels. The monthly meetings to coordinate the interface between health facilities and CSO are addressing logistics problems and improving the links between the community organizations and primary health care centers, but they inevitably focus on mobile testing and outreach discussions. The meetings are attended by the SIHA data entry clerk, and/or a doctor or lab staff, who do not have the skills to review overall program success or have the authority to make changes to better distribute workloads. These coordination meetings are not providing high-level direction and feedback to the HIV program manager or serving as an accountability system for the puskesmas manager. When visiting the health facilities, it was clear that although staff can talk about what they are doing and are starting to use the Dashboards to show how much they are doing, they are unable to assess how effective their efforts are, or review how well the program is being implemented at the site and District level. During discussions with the evaluation team, informants could identify problems, and overwhelming workload demands, but were not in a position to offer solutions. This was consistent across the GOI services in Papua and Jakarta. The motivated staff are addressing problems by working additional, unpaid hours, prioritizing service delivery over data entry, filing and reporting. The burden of inadequate systems, supports and leadership is being carried by less senior staff, and detracting from their ability to provide good health care. Most were USAID.GOV LINKAGES MIDTERM EVALUATION REPORT | 36 unable to decide what they could do differently or what else needs to be done to improve program performance. A slightly different problem was evident at the private clinics, in that they are highly skilled at attracting and enrolling KP, but not thinking about how to work as part of the national program. For example, at Carlo Clinic, they have a large patient load, growing by 80-100 new patients on treatment per month. However, many new GOI treatment sites have opened around them, but they are not transferring any patients to these sites. Currently the TA seems to be too fragmented and does not capitalize on all the expertise the project has to offer. Although TA provided by the project is increasing organizational and technical capacity, it is not coming together to significantly and quickly improve program performance. Now that the project has introduced a range of systems to improve reach, testing, and patient care, data collection and reporting, the next phase of the project needs to bring it all together. TA needs to result in the appropriate senior staff being able to competently assess and improve program functioning and results. STRENGTHENING CAPACITY OF CSO AND GOI PARTNERS LINKAGES has addressed the organizational and technical capacity of CSO partners and monitored changes in capacity. CSO is Indonesia are entirely reliant on donor funds and, in Papua, support from faith￾based organizations. The GOI does not provide any support, either financially or for organizational capacity development. Therefore, there are few resources available in Indonesian to support good governance. While the GOI objection to CSO seems to be primarily that they do not want to foster any avenue for advocacy and are not open to critique, CSO have tried to stress their desire to be partners in the national response to the HIV epidemics, rather than detractors. The project has introduced a structured Organisational Capacity building program, including a scoring system, modules and site visits/checks and Leadership development modules. The project is measuring capacity development results using an Organizational Performance Index (OPI), and data that captures the community based activities and their impact including the number of new PLHIV identified, by whom, and their treatment status). The CSO partners are deeply engaged in the process and monitoring their increased capacity. A key informant in Papua told the evaluators “I thought I was a good manager, until our organization was assessed, and I saw the OPI scores; it really motivated me and gave me a way of actively improving.” Overall the average score increased from 1.6 to 2.3 in the space of less than a year. CONCLUSIONS The evaluation finds that the project organizational capacity development activities are being delivered as intended and are reaching the proposed target organizations. Furthermore, the organizational capacity development of the CSO partners was well documented at baseline and at 12 months follow-up, using the ITOCA system and Organizational Performance Index. Capacity building is addressing organizational performance by skilled staff, and the support is well accepted by the CSO. UPTAKE AND SUSTAINABILITY: The decentralized, district-based health and HIV service delivery model relies on close and meaningful collaboration between the service providers in the puskesmas and hospitals and the CSO who refer patients to them, accompany them for testing and treatment imitation and support them once they are on treatment. However, with CSO activities largely funded from external sources, sustaining outreach and support at the levels needed to reach the targets is unlikely. LINKAGES 37 | LINKAGES MIDTERM EVALUATION REPORT USAID.GOV has been working to improve the organizational, financial management and planning and reporting capacity of CSO, especially in Papua. Increased capacity will only contribute to sustainability if the CSO can access the funds they need to continue operating. LINAKGES has taken a three-pronged approach to sustainability: • Focusing on strengthening systems and approaches that are already embedded in or under pin the national program (such as improving SIHA data collection and use), • Developing new systems that clearly enhance the existing system (online mobile testing DOKLING system) • Developing the evidence to underpin new interventions and systems (PrEP readiness, self – testing); The project is supporting transition to GOI funding in 2020, through the development of information and evidence of unit costing (with other USG partners) for consideration by the Multi-donor Transition Fund. The Fund group is looking at: • How to get prevention and reach activities funded • How to secure additional funding to scale up treatment availability • How to establish mechanism that will secure CSO funding from the GOI • The role of the Universal Health Scheme in the HIV program implementation, including for KP who have no ID cards. A high-level GOI employee stated that the arguments around making services available to people without ID cards is unlikely to get traction, because, as he explained: “There are rights, and there are responsibilities: if you want the right to get free or subsidized health care services, you have a responsibility to demonstrate you are a citizen and hold the correct documentation” (Jakarta, manager GOI) Recommendation: In the final year of the project, TA support at above site level should focus on the networks and linkages to other funding and TA to support CSO financial sustainability and social entrepreneurship. This should include a strategy to position CSO as organizations capable of undertaking niche services on behalf of the government on a contracting-out basis. However, the current TA approach is fragmented. TA teams that include organizational capacity development, clinical mentoring and strategic Information need to come together under a LINKAGES TA leader who has the capacity to link all the information together and help sites to identify the overall effectiveness of their program, and which aspects need further strengthening to improve results. The TA team leader should meet not only with the HIV Doctor but also with Head of the hospital program or the puskesmas. These meetings could eventually be conducted by the District health officer managing the HIV program, but for now they will need to develop the capacity to lead discussions and ask questions to elicit the story from the data. There is no evidence that DHO staff have the expertise to lead these discussions yet. The project is facilitating monthly meetings between the puskesmas and the CSO, which resolves some problems. However, the head of the puskesmas does not attend these meetings and is not asked to account for how well the program is working or what else is needed to improve the impact of USAID.GOV LINKAGES MIDTERM EVALUATION REPORT | 38 the program and contributing to the achievements of the District. Where there are sites that are doing well on any aspect of the program they could be helping other sites. RECOMMENDATIONS 1. Review and revise the TA approach to bring together a TA team that understands the data for each location they support, and can help to increase the effectiveness of the system, not just the individual doctor or lab staff. A TA Team Leader from LINKAGES should meet with the Program manager as well as the doctor, data entry and lab staff at each puskesmas. The manager should be held accountable for achieving the results, not just reaching targets. The District health officers need support to facilitate conversations within and across puskesmas and private providers to identify bottlenecks, challenges and solutions. 2. Strengthen capacity through leadership development programs and incentives. Participation should be tiered to reflect varying levels of commitment, so it’s clear there are no ‘free rides’: investment in capacity development must be reflected in increased results. 3. Foster a culture of accountability for results and shared responsibility to identify and address program short comings, bottle necks and gaps. Focus planning, coordination and monitoring capacity building on overcoming the pervasive focus on completing tasks at the expense of achieving results. This will require a consistent approach across the LINKAGES teams, and explicit efforts to help stakeholders at every level to identify opportunities to initiate system improvements. 4. Explore opportunities to work with others with a remit for health system strengthening to promote leadership that produces strategic policy frameworks, effective monitoring and oversight, coalition building and accountability. CONSOLIDATED RECOMMENDATIONS 1. Continue to strengthen the capacity of service delivery sites to analyze their patient loads, testing strategies and collaboration efforts, and plan to use existing resources more strategically, in discussion with the DHO and Puskesmas Managers, to increase retention across the cascade. 2. Assist CSO to identify how changes in their overall intervention-mix and skill mix could create efficiencies and increase impact. 3. Collaborate with Challenge TB and other stakeholders to systematically increase HIV testing among TB patients by addressing the capacity, confidence and accountability of TB staff, adding functionality to DOKLING to support coordination and specimen tracking. 4. Continue to support puskesmas and CSO staff to work together to optimize community based testing, increase HIV yield and support KP across the CoPCT. This will require capacity building to ensure health and CSO staff can analyze the dashboard data and refine their strategies based on results. 39 | LINKAGES MIDTERM EVALUATION REPORT USAID.GOV 5. Bring together key stakeholders to remove the barriers to initiating newly diagnosed PLHIV onto anti-retro viral therapy and retaining them on treatment when they move locations to reduce the delay between diagnosis and treatment and high rates of lost to follow up. 6. System changes are required to free up places at private clinics for PLHIV without documentation, and move stable, treatment experienced patients to new clinics. This may require a staged approach, as the clinics are likely to be reluctant to refer patients on. Change in the distribution of patients has the potential to realize widespread benefits, as working with stable patients will increase the confidence and expertise of doctors in new treatment sites, as well as reducing travel time for patients. 7. Demonstrate interventions that assist the GOI to understand the benefits of and challenges to expanding access to and reporting on viral load testing. Contribute to the development of implementation plans designed to overcome infrastructure and program complexities, and address the affordability constraints. 8. To support decentralization and increase demand for services, investigate the mobility of ART patients and see how much demand in Jakarta and Jayapura is generated from people living outside the catchment area. Use this information to support Districts to rationalize and optimize patient loads, and reduce travel for KP. a. Map home versus place of treatment and identify opportunities for high load clinics to transfer established patients out to new sites and mechanisms for doing this safely b. Study newly diagnosed PLHIV at selected facilities and follow this cohort to determine specifically how they can be retained in or lost to treatment systems c. Analysis of official registered place of residence (using information on NIK or JKN card) vs. location of ART initiation and continuation using existing patient-level SIHA data (offline site￾level SIHA) at selected sites 9. Demand creation in Papua is hampered by poor coordination, and bottlenecks at key decision￾making points. Leadership is concentrated among the CSO and faith based organizations. Continue to explore above site-level opportunities to ensure changes in TA provision do not result in the active leaders in the response no longer being linked to GOI services, and not sufficiently resourced to continue to provide outreach and support services. 10. Demand creation is also hampered by limited knowledge of prevention and other programming options among the PP and other key stakeholders. Conduct studies that generate greater understanding of the PP and their movements, health-seeking behavior and patient journeys in Papua, to inform program planning and campaign design. 11. Continue to develop the links between KP networks, CSO and legal and Human Rights organizations, and build capacity to ensure case files and other data can be effectively used for advocacy and to gain local and international support for law reform and further protection of the rights of KP. USAID.GOV LINKAGES MIDTERM EVALUATION REPORT | 40 12. Build on the training to reduce stigma and discrimination in health care settings to reduce S&D among TB service providers, and other health care professional engaged in HIV testing and treatment. 13. Due to the resistance to introducing differentiated outreach and support services, they may need to be packaged and heavily promoted; impact monitored and results widely shared to get buy in. Regardless of levels of support for new ideas, LINKAGES must continue to undertake this important work with TA partners. Wherever possible, timely cross-sectional cascades should be generated to assesses overall performance for each specified KP or subpopulation within each District to demonstrate successful interventions. 14. Long-term leadership and pressure for improvement in the national HIV program will come from the KP networks and CSO; therefore the project should invest in leadership development and capacity assessment and building for CSO. KP Network leaders must be supported to engage in coordination and planning forums, and to continue to develop the capacity to drive the national response. 15. Use DOKLING and other methods to strengthen facility-facility referral linkages and/or to organize VL specimen transport 16. Advocate for policy level approval to allow self –testing, online access to risk assessment and self￾referral for testing, and monitor the impact this has on demand for testing and treatment. 17. Prioritize TA for leadership and capacity development among KP Networks, CSO and allies in the research, public health and human rights sectors with a long-term commitment to the HIV response, to sustain the focus on KP beyond the life of the project and secure KP participation in decision making systems. 18. Extend the stakeholder mapping exercise to document what drives and motivates people in positions of influence to act (or not act), understand the alliances and networks that exist and what shapes them, and opportunities for expanding the projects sphere of influence and leveraging existing resources. 19. Increasing prevention interventions, reach and testing among female sex workers is already challenging, and is likely to become more difficult in the coming year, despite little evidence that a punitive approach improves the health of sex workers and their clients. The project and the JFTI find it challenging to achieve the SW targets, largely due to the environment that stigmatizes and marginalizes sex workers, although many of the women identified as ‘other’ in key data sets are likely to be current or former sex workers. The project should accelerate efforts to understand changes in the sex industry, mobility patterns, online networks and provide differentiated interventions that reach sex workers and ensure access to services. 20. Reduce S&D among family members of PLHIV in Papua, where knowledge of HIV is limited, and myths persist. Significant efforts are needing to overcome the ignorance and stigma that reduces demand for quality services in Papua. Investment in raising awareness of HIV in Papua should be explored as a foundational activity to improve demand across the CoPCT and increase the quality of life for PLHIV. 41 | LINKAGES MIDTERM EVALUATION REPORT USAID.GOV 21. Explore the option of promoting economies of scale in Papua via CSO alliances, shared ‘back office’ facilities and networks of CSO that plan and coordinate activities together. Extensive demand creation is needed to overcome existing suspicion of HIV treatment with ARV. 22. Address the lack of HIV program coordination and accountability for results in Papua, in collaboration with the GOI and local stakeholders. This is critical to improve the response and maintain the decentralization momentum. 23. Focus on continuing to support de-centralizing ARV service delivery and supporting facilities to initiate ART (particularly in Jayawijaya), through collaboration with other USAID funded programs, such as health system strengthening, supply chain management and Challenge TB. 24. Compile a suite of resources addressing questions and concerns regularly covered in Clinical Mentoring and lift the level of mentoring to focus on establishing sustainable mentoring systems between hospitals and puskesmas staff. Without this, clinical mentoring will address only the concerns of a never-ending stream of short term medical staff and will not result in sustainable improvements to service quality. 25. Collaborate with the USAID Procurement and Supply Chain Management Project to secure the supplies needed to scale up community based mobile testing in remote, highland and coastal areas, and support multi-month scripting. 26. Assist in improving and streamlining of current patient data flow at LINKAGES supported facilities. At minimum, the following should be considered: a. A detailed assessment of patient data flow and the associated health information systems used at selected high-volume ART health facilities. This includes a thorough mapping of the individual patient and aggregated data flow, identification of duplications in data collection and entry, and inefficiencies/challenges in the various data management systems used to generate SIHA offline (individual-level) and SIHA aggregated data b. Modification of the current data management tool (an MS Excel-based “aggregated data file”) used by Ruang Carlos clinic to include advanced functions and formulas for data aggregation and analysis 27. Provide technical direction and support on the improvement of SIHA. At minimum, the following should be considered: a. Documentation and analysis of the most common SIHA issues found across ART facilities (i.e. tracking of frequently asked questions via SIHA telephone hotline) in order to assess system vulnerabilities and weaknesses in the current SIHA version b. Provide structural modifications/enhancements to the existing version of SIHA, including the following: i. Auto-calculation and auto-population of aggregated SIHA results [into site-level SIHA reports that are submitted to the MoH] once individual-level data is entered into offline SIHA USAID.GOV LINKAGES MIDTERM EVALUATION REPORT | 42 ii. Create automatic data quality/logic checks into SIHA (e.g. flash box to alert data entry clerk when numbers entered for newly initiated on ART is greater than then total on ART in the reporting period) c. Assess reporting completeness and frequency of data use for the existing 11 “mandatory” MoH HIV indicators in SIHA across all ART facilities in Jakarta as an advocacy tool/evidence for streamlining the number of “core” indicators for the national HIV program 28. The project should establish mechanisms to assess the performance and effectiveness of LINKAGES’ technical assistance that could highlight any differences in sites receiving TA when compared to other sites. 29. Greater engagement of LINKAGES SI team in strengthening the capacity of PHO, DHO, and ART facility SI staff on data use and analysis is needed. In addition to generating monthly cascade dashboards, LINKAGES should facilitate the following: a. In collaboration with PHO, DHO, and facility staff in Jakarta and Papua, monthly analysis of site-level and aggregated level clinical data should be conducted to assess performance and investigate data quality issues reported in SIHA b. Quarterly meetings with DHO and PHO should be better structured, with the primary intent of discussing SIHA results and trend data for the reported period. Technical presentations on strategic information can also be conducted periodically and should be considered a core component of these meetings c. Intentional knowledge transfer should take place for the LINKAGES M&E secondment to the PHO/DHO through a mentorship pairing system where the LINKAGES M&E secondment is assigned as a “mentor” to one or two public servants (SI staff) at their seconded office, where concrete capacity building plans should be developed and evaluated annually. 30. Modification of existing data collection system to demonstrate “different models/configurations” of services piloted in the Enhanced Outreach Approach (EOA) in order to assess the effectiveness and uptake of each “model”. As such, a minimum of the following modalities of services should be considered for data collection and analysis: a. Proportion of online outreach vs. venue-based outreach b. Proportion of completed testing referral among those reached online vs. venues c. Segmentation of assessed risk behaviors (via risk assessment form) among KP reached by age and by outreach modalities (i.e. online vs. venue) d. Percent HIV positivity yield among those tested as a result of online outreach vs. venue-based outreach e. Mapping of peer recruitment network for HIV testing (“referral chain network”) and HIV positivity rates among these social networks 31. Strengthen the capacity of HIV advocacy networks (i.e. Indonesia AIDS coalition) in knowledge management, including 43 | LINKAGES MIDTERM EVALUATION REPORT USAID.GOV a. Provision of technical assistance in the systematic documentation of advocacy and technical briefs; and b. Use and analysis of existing epidemiological, costing, and program data for advocacy with the government 32. Operational research to assess the mobility of ART patient should be considered. Study can be conducted using two different methods in order to triangulate results. a. Random sampling of newly initiated ART patients at selected facilities within a 3-month recruitment period and follow this cohort for 6 months to determine where ART clients are accessing ART b. Analysis of official registered place of residence (using information on NIK or JKN card) vs. location of ART initiation and continuation using existing patient-level SIHA data (offline site￾level SIHA) at selected sites. 33. Review and revise the TA approach to bring together a TA team that understands the data for each location they support and can help to increase the effectiveness of the system, not just the individual doctor or lab staff. A TA Team Leader from LINKAGES should meet with the Program manager as well as the doctor, data entry and lab staff at each puskesmas. The manager should be held accountable for achieving the results, not just reaching targets. The District health officers need support to facilitate conversations within and across puskesmas and private providers to identify bottlenecks, challenges and solutions. 34. Strengthen capacity through leadership development programs and incentives. Participation should be tiered to reflect varying levels of commitment, so it’s clear there are no ‘free rides’: investment in capacity development must be reflected in increased results. 35. Foster a culture of accountability for results and shared responsibility to identify and address program shortcomings, bottle necks and gaps. Focus planning, coordination and monitoring capacity building on overcoming the pervasive focus on completing tasks at the expense of achieving results. This will require a consistent approach across the LINKAGES teams, and explicit efforts to help stakeholders at every level to identify opportunities to initiate system improvements. 36. Explore opportunities to work with others with a remit for health system strengthening to promote leadership that produces strategic policy frameworks, effective monitoring and oversight, coalition building and accountability. USAID.GOV LINKAGES MIDTERM EVALUATION REPORT | 44 ANNEX I: EVALUATION METHODOLOGY AND TIMELINE METHODOLOGY AND APPROACH The evaluation team will conduct a midterm process evaluation of LINKAGES in Indonesia. The evaluation is expected to generate findings related to performance to date, as well as recommendations for future implementation stages. The mixed-methods evaluation will utilize a framework co-designed by the evaluators, the commissioning clients and those intended to benefit from the program. This approach provides an opportunity to support innovation and ongoing adaptation of the program, as it is implemented, in a complex and changing context and generate practical recommendations, to inform program management and improve impact in the remaining years of implementation. The evaluation team will take into account the two very different contexts (Jakarta and the national level, and in two districts in Papua Province) in which LINKAGES is being implemented. It cannot be expected that the approach to the program adopted in Jakarta will simply be replicated in the Papuan context and achieve the same outcomes. Contextual differences to be considered include the differing social and political drivers of the epidemic, the unique geographic, demographic and organizational contexts of each implementation location, the range of intended beneficiaries and target groups, variations in program staffing, and the different epidemics being addressed (both concentrated and generalized). In evaluating progress in enhancing and sustaining demand, the evaluation will consider the extent to which the program address the multiple determinants of access, including the many difficulties in gaining access to the resources necessary for HIV prevention and treatment, and the ability to mobilize household and community assets, including human capital (local knowledge, education, skills), social capital (social networks and affiliations), physical capital (infrastructure, equipment, and means of transport) and financial capital (cash and credit). The evaluation will look for evidence of increasing equity (defined as equal access to health care by those in equal need)8 as well as overall increase in demand. For a detailed evaluation timeline, see Annex IV TEAM PLANNING The first stage of the evaluation will take place during the Team Planning period, which will occur from 2nd -10th August. At this stage, only the Team Leader and the Research Assistant are actively engaged. The USAID members of the team will participate in the evaluation during the data collection activities. Their involvement in analysing the data and drafting the final report are yet to be negotiated. During the Team Planning period the team will consultant with USAID/Indonesia, the LINKAGES Program Senior Management Team and Key Affected Populations (KAP) to develop the evaluation methodology, work plan and schedule and selection of key informants and site visits. Incoming briefing meetings will be held with USAID and LINKAGES Indonesia and MESP key personnel. A list of LINKAGES partners and collaborators will be obtained and overview of the key program activities and technical approach in Jakarta 8 Oliver A, Mossialos E (2004) Equity of access to health care. Outlining the foundations for action. J Epidemiol Community Health 58: 655–658. 45 | LINKAGES MIDTERM EVALUATION REPORT USAID.GOV and Papua will be provided. During Team Planning period, the data collection instruments and the sampling and fieldwork plan will also be developed. Key Informants and sites for the field work will be selected, to ensure the evaluation team gains an understanding of each aspect of the program implementation. The team will organize the site visits and schedule for meeting with Key Informants, seek travel approval and obtain permits for the Papua site visits. Consistent with the principle of ensuring there is meaningful involvement in all aspects of the response to HIV by the KAP, and the USAID Evaluation Policy, a co-design workshop was held on 8 August 2017 with a small group of people from Civil Society Organisations (CSO) working closely with or led by KAP. Prior to the workshop a draft evaluation design document and draft interview guides will be prepared, as a starting point for discussion. Prior to submitting this deliverable, the workshop was held at Mercure Hotel Sabang Jakarta. The purpose of the workshop is to get input from participants for the refinement of data collection instruments taking into account KAP perspectives. Participants of the workshop consisted of CSO and KAP representatives. Thirteen out of 15 invitees were able to attend the workshop. These participants came from Perkumpulan Keluarga Berencana Indonesia (PKBI), AIDS Research Center (ARC) of Atma Jaya University, Yayasan Intermedika (YIM), Aura Buana, Organisasi Perubahan Sosial Indonesia (OPSI), Fokus Muda, Indonesia AIDS Coalition (IAC), and Yayasan Kusuma Buana. During the workshop participants were invited to have input into developing the methods, data collection instruments, selection of key informants and the evaluative criteria. For example, the evaluative criteria for measuring enhanced demand will be influenced by what availability, accessibility, affordability, adequacy, and acceptability mean, in practical terms, to KAP. Figure 1, below, illustrates how demand variables and evaluative criteria were generated with the KAP during this workshop. FIGURE 2. DEMAND VARIABLE AND EVALUATION CRITERIA During the Team Planning period, Key informants (KI) will be selected, based on the input from USAID, LINKAGES and the Co-design workshop. Key informants and sites will be contacted to confirm the field work schedule, prior to departure for Papua on 14th August 2017. USAID.GOV LINKAGES MIDTERM EVALUATION REPORT | 46 THE DATA COLLECTION METHODS DOCUMENT REVIEW The evaluation will collect quantitative and qualitative data from a variety of secondary sources. USAID will provide key background and contextual documents, program design documents and monitoring reports. LINKAGES will provide access to program performance and monitoring data, and internal reports and examples of assessments and other key documents. (See Annex IV for a list of key documents reviewed.) A scan of the literature related to the national program, capacity building and technical assistance provided background information, to inform data collections instruments. Evaluation team members will review the following categories of key background documents: • Government of Indonesia documents: including the National Strategic Action Plan, fast track and other strategy documents, epidemiology and surveillance reports • USAID documents: including USAID’s Evaluation Policy, the Cooperative Agreement for the LINKAGES or design documents for the Project; and USAID/Indonesia’s comments on work plans and performance monitoring plans. • LINKAGES documents: including annual work plans, progress reports, the M&E plan and performance monitoring data, any baseline documentation, relevant assessment and research reports; and the Program Logic Document/strategic plan. • CSO documents: including Annual Reports, communication with LINKAGES, Advocacy documents and reports on activities and programs REVIEW OF PERFORMANCE RELATED DATA The performance monitoring data for LINKAGES, including activity data, will be reviewed to identify key outputs and where possible outcomes related to the three result areas. Data review will include the Direct Service Delivery (DSD) and Technical Assistance - Service Delivery Improvements (TA-SDI) data, as outlined in the LINKAGES Activity Monitoring and Evaluation Plan (AMEP). This will include the Tracked CSO Outreach performance data, the results of high yield testing strategies, HIV Testing technical Performance Scores, Sub-awards data and the PEPFAR Indicators data included in quarterly reports. KEY INFORMANT INTERVIEWS, SITE VISITS AND DISCUSSIONS USAID/Indonesia and LINKAGES will provide the Evaluation Team with stakeholder information and contact details for partners and implementers, stakeholders and collaborators, from which the Evaluation team can work to create a comprehensive list of key informants. Prior to starting data collection, the Evaluation team will provide USAID with a list and schedule for KI interviews and site visits in Jakarta and Papua. The Evaluation team will continue to share updated lists of interviewees and schedules as meetings and interviews take place, and as informants are added to or deleted from the schedule. The following categories of key informants will be interviewed: • USAID/Indonesia Health Office staff and other USG HIV Program staff • LINKAGES Consortium Members in Indonesia (senior management team and key technical staff) • LINKAGES implementing partners, strategic partners, and collaborative partners Key Government agencies at national, provincial, and district levels 47 | LINKAGES MIDTERM EVALUATION REPORT USAID.GOV • Program beneficiaries, volunteers, and KAP community members • Selected multilateral organizations and bilateral development partners (UNAIDS, WHO, UNFPA, UNDP) • Civil Society Organizations • University based collaborators and partners • District government provided Primary Health Care Facilities (Puskesmas) All interviews with implementing partners, partners and collaborators, KAP and CSO, and program beneficiaries, will be conducted without the presence of LINKAGES or USAID /Indonesia staff and their implementing agencies staff to minimize the risk of bias in responses. Interview guides have been drafted/developed for each category of key informant, based on the evaluation questions, to ensure a consistency in approach by team members conducting interviews. Given that information relating to some of the evaluation questions is contained in LINKAGES progress reports and performance monitoring data, the interview guides are focused on seeking information not contained in those sources or verifying claims made in progress reports and better understanding what is (or is not) working well and why. The interview guides are semi-structured, not intended to be used as a questionnaire, but more as a guide to ensure that all key areas will be covered, across a range of KI. The interview guides provide a range of questions that the team can choose from, depending on the situation, the knowledge of the people available at the site visits and the local circumstances. Interviews will be held in surroundings familiar to the interviewee and provide an opportunity for intended beneficiaries and partners to meaningfully engage in a dialogue with the evaluation team. Following the completion of stakeholder interviews, as per the developmental Evaluation approach, the evaluation team will meet with USAID/Indonesia Health Office staff, LINKAGES Senior Management Team and technical staff and other key stakeholders to ask to follow up questions and discuss issues arising during the initial data collection, review and explore possible interpretation of preliminary findings and generate recommendations. As per the Developmental Evaluation approach, collaboration and dialogue with stakeholders will be further promoted by their participation in a debriefing meeting conducted by the evaluation team at the end of the field work. The Evaluation team will meet with USAID/Indonesia Health Office staff, LINKAGES senior management and technical staff and other key stakeholders to review and explore possible interpretation of preliminary findings, and feedback on conclusions and proposed future directions will be sought. ANALYSIS The evaluation team will undertake ongoing analysis of all qualitative data using grounded analysis and, in relation to Result Area 3, pre-determined evaluative criteria that defines the threshold for ‘strengthened’. Team members will broadly identify themes emerging from the interviews, discussions and conversations during site visits, and identify the range of positions and perceptions expressed across each theme. The content of the qualitative data will be analyzed to both identify what was said, as well as what may have been inferred or implied. The analysis will be further informed by a discourse analysis. The discourse analysis will allow the team to take into account the context in which opinions are developed by each stakeholder and in which data is generated. This will assist the team to weight the data and take into consideration the attitudes partners USAID.GOV LINKAGES MIDTERM EVALUATION REPORT | 48 may display in relation to receiving Technical Assistance, the power relationships and the possible prejudices or positions of KI. The discourse analysis may also include analysis of body language to give a rich source of data surrounding the actual words used. Context analysis will also be undertaken to make sense of the data collected and to highlight the important messages, features or findings for the two distinct program implementation sites (Jakarta and Papua). This iterative process will allow for emerging issues to be explored and potential findings to be tested as the evaluation progresses. Following the completion of key stakeholder interviews and site visits, the evaluation team will conduct a thorough analysis of all data, both qualitative and quantitative, and develop preliminary findings and conclusions in relation to the evaluation questions in the SOW. This analysis will inform the preliminary findings and provide the basis upon which the evaluation report written. MIDPOINT DEBRIEF WITH USAID/INDONESIA The evaluation team’s preliminary findings and conclusions will be presented to USAID/Indonesia and LINKAGES for the purposes of feedback, validation and further input. This debriefing will take place on 25 August 2017, so the Mission’s inputs can be considered in drafting the evaluation report. REPORT WRITING The evaluation report will be prepared in accordance with the USAID Evaluation report template provided MESP, to ensure the quality of evaluation reports, as set out in Appendix 1 of USAID’s Evaluation Policy. In particular, emphasis will be placed on demonstrating the quantitative and qualitative evidence on which findings are based. A draft table of contents for the evaluation report and writing allocations will be agreed upon at the evaluation team’s initial planning meeting at the commencement of in-country work so that team members have a clear understanding of the key deliverable and their inputs. END OF EVALUATION MISSION DEBRIEFINGS On the second last day of the in-country work, the evaluation team leader will present a summary of key findings, conclusions and proposed future directions and recommendations to USAID/Indonesia Health Office. On the last day, the Team Leader and USAID will hold a discussion with LINKAGES Senior Management Team to finalise recommendations. Pending approval from USAID, the stakeholders and KAP who participated in the evaluation planning workshop will receive a 1-page summary of the evaluation findings. REVIEW OF DRAFT EVALUATION REPORT AND FINALIZATION USAID/Indonesia will review and revise the draft evaluation report prior to the final report being submitted by the Team Leader. Feedback will be sought at all debriefing meetings and via the circulation of the near-final draft report to inform revisions, however the Evaluation Team Leader retains the right to make the final decisions related to findings, conclusions and suggested ways forward. 49 | LINKAGES MIDTERM EVALUATION REPORT USAID.GOV LIMITATIONS • Due to the large number and type of stakeholders, not possible to interview all. The number of stakeholders interviewed was maximized by the evaluation team splitting into two sub-teams during site visits, • Language barriers and the need to use interpreters for interviews, particularly for CSO implementing partners and project beneficiaries. TEAM COMPOSITION Jenne Roberts, Team Leader: Jenne is an International Public Health specialist, with more than 20 years’ experience designing, evaluating and managing health programs, including HIV programs in South East Asia. She advises clients on the effectiveness of their programs and identifies opportunities for increasing the program impact. Jenne is responsible for designing the evaluation methodology, selecting sites and key informants, analysing the data, conducting the evaluation and preparing the evaluation report. Ratna Soehoed, Research Assistant. Ratna is local consultant with 10 years’ experience working with on HIV projects in Indonesia. Ratna assisted the evaluation team in conducting primary as well as secondary data collection, developing the data collection schedule in Papua and DKI Jakarta and liaising between the evaluator and the informants. She also helped to prepare the evaluation report and kept track of program and performance documents. Maria Au, USAID Washington expert. Maria Au is Senior Advisor for Research, Monitoring and Evaluation in the SIEI/Strategic Information Team, Office of HIV/AIDS at USAID. Maria participated in data collection in DKI Jakarta, document review and provided substantial input in identifying findings and recommendations related to program performance data and the use of strategic information. Ngoc Thi Minh Nguyen. Ngoc is the the HIV/AIDS Care and Treatment Team Leader for USAID /Vietnam. Ngoc participated in data collection in Papua and DKI Jakarta, document review and preparing overall findings in relation to HIV/AIDS care and treatment conducted by health facilities, District Health Offices (DHO), and Provincial Health Office. Ricky Andriansyah. Ricky has worked on a number of health and HIV projects in Indonesia and is currently the M and E Adviser for USAID seconded to work with the Ministry of Health’s HIV Program. As a member of evaluation team, Ricky was involved in data collection in Papua and DKI Jakarta, document review and preparing overall findings in relation to data collection and utilization. MESP SUPPORT Technical support and direction for the evaluation will be provided by MESP Chief of Party Jonathan Simon. Djoko Hartono of the MESP team will provide general oversight and research in support of the evaluation. Scheduling, coordination, and logistics will be managed by MESP staff (Rina) and research assistant (Ratna), in coordination with LINKAGES implementing partners and USAID. MSI headquarters in Washington will provide additional technical support, as required. USAID.GOV LINKAGES MIDTERM EVALUATION REPORT | 50 LEVEL OF EFFORT The level of effort (LOE) for the LINKAGES midterm evaluation team ranges from 46 days for team leader and 30 days for research assistant. An interpreter will also be provided, especially during fieldwork And the workdays allocated for the interpreter is 12 days. The following table summarizes the estimated LOE for the evaluation, by position and task. Management/ Planning Position Doc Review Int’l Travel Initial Briefing & Team Planning Methodology Development Data Collection (Jakarta & Papua) Analysis Drafting Final Report Total Days Team Leader (international) 1 3 2 3 6 9 6 10 6 46 Research Assistant (Indonesian) 1 3 0 3 6 11 6 0 0 30 Interpreter (Indonesian) 0 0 0 0 2 10 0 0 0 12 DELIVERABLES 1. Evaluation design, including detailed research methodology, drafts of data collection instruments, sampling plan, implementation and fieldwork schedule. 2. Draft Midterm Evaluation report. USAID will provide written comments on the draft report within 10 working days of receiving the document. 3. Final Midterm Evaluation Report, incorporating USAID comments on the draft. If requested by USAID request, MESP will provide a comprehensive version for USAID and a redacted version for external partners. In addition, MESP will translate portions of the Midterm Evaluation Report into Bahasa Indonesia, at the request of USAID. 4. Oral presentations of key findings, conclusions and recommendations from Final Study Report to USAID and external partners, including separate presentation slide decks. Date Task 4 May 2017 – 25 July 2017 Linkages Evaluation team recruited and employment agreements finalized (MESP) 8 – 19 May 2017 Linkages-related project documents collected and stored in MSI’s Egnyte drive (MESP) 9 June 2017 Linkages midterm evaluation SOW approved 18 –27 July 2017 Preliminary meetings MESP w/ USAID & IP. Key stakeholders and informants identified (MESP) 26 July 2017 Literature review begins. Desk study begins (consultants - home based). 51 | LINKAGES MIDTERM EVALUATION REPORT USAID.GOV Date Task 2 August 2017 Team mobilizes to Jakarta. Team leader and research assistant start in-country work. Meeting with MESP (morning) and USAID Linkages (afternoon) USAID provides LINKAGES with a copy of the SOW 4 August, 9 am @ LINKAGES office, Rasuna Epicentrum Kuningan, Level 18 Evaluation team meets with LINKAGES team. Jenne provides an overview of the Evaluation process and SOW, Caroline and team provide a program overview and introduce key team members. timeline negotiated 3, 4 Aug 2017 Work on the Evaluation design, including data collection instruments and getting the schedule locked in 7 Aug COB LINKAGES will provide a briefing package, including the details of who to meet with in Papua 3 – 14 August 2017 TPMs and in-brief w/ USAID, Linkages IP and GOI. Evaluation design, including data collection instruments finalized. 8 August 2017 One day workshop (in Jakarta) for instrument designs w/ invited KAP 10 August 2017 Draft evaluation design, including detailed research methodology, data collection instruments, sampling plan, implementation and fieldwork schedule (Deliverable 1) completed and shared with USAID Washington team. 10 August Tetty from USAID will have the regular meeting with Ibu Endang from the MoH and advise her the evaluation has commenced 11 Aug 9 am at LINKAGES office Start getting briefings from LINKAGES, Papua briefing at LINKAGES for Jenne, Ratna, Ricky 12 August 2017 Arrival of USAID Washington expert in Jakarta (Note: Maria from USAID/W will join the team starting from August 21 up to September 1, 2017. 13 – 14 August 2017 Team Planning Meeting (TPM) and in-brief w/ USAID (cont’) Sylvia will send letters of invitation to key informants asking them to meet with the evaluation team 14 August 2017 (night) Team depart to Papua (Jayapura) Note: the team members who will be going to Papua are (a) MESP: Jenne, research assistant, and translator; and (b) USAID: Ngok and Ricky. 15 – 16 August 2017 Fieldwork in Jayapura, this will include meeting with the USAID funded GBV project, WINROC 17 August 2017 Team depart to Jayawijaya (Wamena) NOTE, this is a national holiday 17 – 19 August 2017 Fieldwork in Jayawijaya (Wamena) 20 August 2017 Team depart from Wamena to Jayapura - Jakarta 21 Aug Maria Au arrives 21 August 2017 Evaluation team will meet with the UN agencies, UNAIDS, WHO, UNFPA and UNDP 21 – 24 August 2017 Continue Evaluation work in Jakarta, including field data collection in Jakarta, consolidated data analysis, preparing plan for out briefs w/ USAID, GOI and IP, and for report writing. 22 August 9am until noon at LINKAGES Evaluation team to meet with LINKAGES for data collection, by team and result area, Ngoc to meet with Clinical Team 23- 24, 25 August Data collection in Jakarta, via site visits and interviews 25 August 2017 Meet with USAID front office, meet and greet USAID.GOV LINKAGES MIDTERM EVALUATION REPORT | 52 Date Task Outbrief with USAID (in the afternoon: 13:00 to 14:30) 26 August Ngoc leaves 28 August – 6 September 2017 Data analysis, draft findings, and preparation of out-briefs to be continued 28-31 Aug Maria is in Jakarta, but working on a cohort study 1 September 2017 USAID team depart to home base: Ngok and Maria 7 September 2017 Out-briefs on progress to date and preliminary findings held for GOI and USAID, prior to the USAID team going to Africa 8 September 2017 9 am at LINKAGES At LINKAGES to do shared interpretation of the preliminary findings and crafting useful recommendations. Out-briefs on progress to date and preliminary findings held for Linkages IP and USAID 21 September 2017 Team leader leaving Jakarta 11 – 14 September 2017 Report writing continues. Findings, conclusions and recommendations developed (home-based). 15 - 18 September 2017 Submission of draft report to MSI for technical review. 18 September Ibu Ratana last day of contract 19 – 20 September Final edits by the team leader before submission to USAID 22 September 2017 Submission of final draft reports to USAID (by MESP). 26 September 2017 Comments/feedback from USAID received 27 – 28 September 2017 Evaluation team leader incorporates feedback from USAID 29 September 2017 Final draft report submitted to MSI. 02 October 2017 MSI internal technical review completed. 03 October 2017 Final report submitted to USAID and shared with implementing partners. 53 | LINKAGES MIDTERM EVALUATION REPORT USAID.GOV ANNEX II: DATA COLLECTION SCHEDULE Fieldwork for data collection will be carried out in Jakarta and Papua. Data collection in Jakarta will be carried out after fieldwork in Papua completed. Detailed field schedule for data collection in Papua is as follows: SCHEDULE DATA COLLECTION IN PAPUA – MIDTERM EVALUATION LINKAGES INDONESIA PAPUA, 15-20 AUGUST 2017 JAYAPURA Time Venue Name Position Contact Details Possible Areas to discuss TUESDAY, 15 AUGUST 2017 12.00 – 14.00 Provincial Health Office (PHO) Dr Silwanus Sumule Secretary of Provincial Health Office (PHO) 08114804871 - Papua response on HIV and possible LINKAGES support Dr Beeri Wopari Head of ATM Unit, (PHO) 085244076999 - Coordination effort on TB-HIV - Feedback on overall LINKAGES results Dr Rindang Pribadi Head of AIDS section, PHO 081248045006 - Papua target setting - ART decentralization efforts Lusia Ang Head of Pharmacy Warehouse, PHO 085344038855 - ARV decentralization in relation to ARV logistics, reagent, HIV, viral load test kit, etc. Michael Rian PHO M&E 081344116814 - Application of LINKAGES tools i.e. cascade generator SIHA - system strengthening and data use 14.30 – 16.00 Puskesmas PKR (Pusat Klinik Reproduksi) Jayapura Dr Hesti Purikasari PKR Jayapura (this is a central support TA facility) 081334748137 - ART initiation 16.00 – 17.00 ICDP office John Rahail Director – ICDP Jayapura 08124209941 081354040142 - Application of EOA - Organizational development and transitioning ICDP Team (outreac workers, M&E staff, PE) - 19.00- 21.00 Hotspot FSW/MSM - WEDNESDAY, 16 AUGUST 2017 08.00 – 9.30 Jayapura District Health Office (DHO) Arif Dwi Darmanto Head of District Health Office (DHO) Kota Jayapura 085244049766 - Papua district level coordination and implementation - Application of LINKAGES tools and or collaboration with LINKAGES on CoPTC priorities USAID.GOV LINKAGES MIDTERM EVALUATION REPORT | 54 JAYAPURA Time Venue Name Position Contact Details Possible Areas to discuss Drg Edy Simangungsong Head of P2M, DHO Kota Jayapura 0811483863 - DHO M&E officer - 10.30 – 12.00 Puskesmas Waena Dr Triska Ferianti PKM Waena in Jayapura 081248054909 - CoPTC service delivery areas - LINKAGES clinical mentoring - Application of LINKAGES tools including cascade generator ART Patients - 14.00 – 15.30 Provincial AIDS Commission (PAC) Dr Constant Karma (PAC Papua) Secretary of PAC 08124810600 - Technical Assistance on VMMC - Papua target setting - Papua central support strategy and possible LINKAGES contributions Yusuf (PAC Papua) PAC M&E officer 085254238344 - Outreach Databank development and improvement Asikin (DAC Papua) Program Coordinator DAC Jayapura 085244619633 - Databank development and implementation - Overall on LINKAGES potential contribution including facilitating CSO and health facility M&E officer - 16.00 – 17.00 ICDP office/ Hotspot John Rahail & ICDP Team (outreac workers, M&E staff, PE) Director – ICDP Jayapura 08124209941 081354040142 - Application of EOA - Organizational development and transitioning Priority Population - DISTRICT WAMENA Time Venue Name Position Contact Details Possible Areas to discuss THURSDAY, 17 AUGUST 2017 Depart to Wamena 14.00 – 16.00 TALI Office (CSO)* John Nap Director – TALI Wamena 08114825556 - Application of EOA particularly work with key populations - Organizational development and transitioning Outreach workers, PEs - 19.00 – 21.00 FSW Hotspot FSWs Discuss with FSW FRIDAY, 18 AUGUST 2017 09.00 – 10.30 DHO of Jayawijaya Dr Yuristianti Ka Sie HIV District Health 08112956620 - Papua district level coordination and implementation 55 | LINKAGES MIDTERM EVALUATION REPORT USAID.GOV DISTRICT WAMENA Time Venue Name Position Contact Details Possible Areas to discuss Andayani (Anti) Office Jayawijaya - Application of LINKAGES tools and or collaboration with LINKAGES on CoPTC priorities M&E officer - 11.00 – 12.30 Clinic of Kalvari Wamena Man Logo Director – Klinik Kalvari Wamena 081247337708 - Improving up-take and quality of clinical services - Organizational development and transitioning ART Patients - 13.00 – 14.30 DAC of Jayawijaya Daulat Siregar Secretary of DAC Jayawijaya 081248832318 082238972059 - Databank development and challenges - Overall on LINKAGES potential contribution including facilitating CSO and health facility M&E officer 19.00 – 21.00 FSW Hotspot TALI - Priority population Discuss with FSW SATURDAY, 19 AUGUST 2017 10.00 – 12.00 PKM Wamena Erni Utari PKM Wamena Kota in Jayawijaya 081248478210 082248035859 - CoPTC service delivery areas - LINKAGES clinical mentoring - Application of LINKAGES tools including cascade generator Patients - 13.00 – 21.00 TALI/Hotspot* TALI – Key/priority population - SUNDAY, 20 AUGUST 2017 - Depart to Airport, fly back to Jakarta ‘* With appointment regarding independence day holiday. USAID.GOV LINKAGES MIDTERM EVALUATION REPORT | 56 Schedule Data Collection in Jakarta – Mid Term Evaluation LINKAGES Indonesia Time Venue Name Position Contact Details Possible Areas to Discuss MONDAY, 21 Augt 2017 / Jakarta 9.00- 10.00 Yayasan Pesona Jakarta (Tebet) Juna Rislon Damanik (Erick) Putra Lubis Program Manager and M&E and ICT Officer 081311379909 ypj.pesonajakarta @gmail.com - Application of EOA - Overall program management - Effort to reach MSM in virtual spaces - Use of outreach dashboard for quality improvement *Can set up discussion with MSM clients 11.00 – 12.00 IAC office (Tebet) Aditya Wardhana Project Director – IAC 08119939399 awardhana@iac.or.id - Strengthening coordination of key population association - Organizational development for expanded global funds role and potential LINKAGES contribution 14.00 – 16.00 Bandung Wangi (Jatinegara) Endang Supriyati Program Director - Bandungwangi endangfres@gmail.com; +62 812-1340-6384 TUESDAY, 22 Augt 2017 / Ministry of Health 09.00 - 15.00 Linkages Meeting in Linkages - WEDNESDAY, 23 Augt 2017 10.00 – 12.00 CDC MOH Endang Budihastuti Head of Subdit HIV and IMS 081287634185 - National response on HIV and LINKAGES potential contributions Victoria Indrawati SIHA and M&E Coordinator 08129672549 - SIHA system strengthening - Improving data use through cascade generator and other tools Triya Dinihari Head of HIV and STI Prev Unit tndinihari@yahoo.com 087880871040 - Jakarta Fast Track - Overall National Fast Track strategy Vini Tobing IBBS Coordinator 0811178870 - IBBS and size estimation Helen Dewi Staff Subdit HIV (HP: 081905207777 & email: helendewi@yahoo.com - Stigma and discrimination reduction (enabling environment) - Coordination for new strategy for new 57 | LINKAGES MIDTERM EVALUATION REPORT USAID.GOV Time Venue Name Position Contact Details Possible Areas to Discuss technology mediated strategy 13.00 – 15.00 NAC Halik Sidik NAC staff (halik.sidik @aidsindonesia.or.id; 08111494628) Improving outreach approaches (coordinated UIC and outreach information system) Ichwan Mochammad PAC DKI Jakarta Head of M&E PAC 08119690574 - Outreach Databank development and improvement - Jakarta Fast Track Soekarno Secretary DAC West Jakarta 0816941690 - Jakarta response on HIV and possible LINKAGES contributions - Overall LINKAGES coordination in district level 15.30 – 16.30 Ruang Carlo Fajar Prabowo Program Manager YKS (Yayasan Kasih Suwitno) 082196754400 - Improving key population access and retention in Ruang Carlo THURSDAY, 24 Augt 2017 10.00- 12.00 MOH DKI Jakarta Province Dr Inda Mutiara Ka. Sie PMTVZ Dinkes Provinsi DKI Jakarta indadinkesdki @gmail.com; 087778287606 - Jakarta Fast Track and LINKAGES potential contributions Dr Finan Pengelola Program HIV Dinkes Provinsi DKI Jakarta finan54santana @yahoo.com; 08129999892 - Application of LINKAGES tools i.e DOKLING - Coordination with other implementing partners - Quality improvement through clinical mentoring M Gupron (re to M&E) M&E Officer PHO 082213061950 - Application of LINKAGES tools i.e cascade generator - SIHA system strengthening and data use 14.00 – 15.30 Puskesmas Cengkareng Dr Nana Etty Supriati DHO Jakarta Barat 0821-3751-4100 0815-9677-958 -Jakarta district level coordination and fast track implementation -Application of LINKAGES tools and or collaboration with LINKAGES on CoPTC priorities Dr Devi Koord HIV Puskesmas Cengkareng 089651327954 085711199934 -CoPTC service delivery areas -LINKAGES clinical mentoring USAID.GOV LINKAGES MIDTERM EVALUATION REPORT | 58 Time Venue Name Position Contact Details Possible Areas to Discuss -Application of LINKAGES tools including cascade FRIDAY, 25 Augt 2017 09.00- 10.30 Kemenkes Samhari Baswedan Secretary CCM HP: 0811807030 & email: samharib @yahoo.com or samhari.ccm @gmail.com - Improving technical leadership and coordination and potential LINKAGES contribution 11.00 USAID Debrief USAID - Need to send our brief CV to Tetty 14.00- 17.00 MESP Muhammad Ahmad Project Director - Karisma 081187770636 yayasan@karisma.or.id Organizational capacity development using LINKAGES tools MESP Raditya Program Director -GEMPITA 081210404221 - LINKAGES tools application (DOKLING and databank) MESP Edo Nasution Coordinator PKNI (PWID Association) 087873460077 - INPUID tool kit MESP Adi Nugroho GWL-INA 081228301918 Adi.nugroho @gwl-ina.or.id - Intervention supported by LINKAGES in year 1 - Coordination on LGBT service delivery and enabling environment intervention MESP Evi Sukmaningrum Head of Research Center University of Atmajaya 0856140045456 Evi.sukmaningrum @gmail.com - PrEP acceptability study MONDAY, 28 Augt 2017 09.00 – 10.30 UNAIDS (Menara Thamrin) Tina Boonto (newly arrived) Country Director UNAIDS boontok@unaids.org +628119444716 - Jakarta Fast Track activation - Contribution of LINKAGES to national response Elis Widen Community Adviser UNAIDS widene@unaids.org; 08121970449 - Fast Track Roadmap and activation Tiara Mahatmi Nisa NPO WHO nisat@who.int 08111581978 085959130008 - Fast Track Roadmap and activation 11.00- 12.00 Spiritia Daniel Marguari CEO Spiritia HP: 0811247706825 or 0818780455 & email: danielmarguari @gmail.com - Global Funds/PEPFAR coordination on community service system strengthening and LINKAGES potential contributions 14.00 – 15.00 Challenge-TB Agnes Gebhard KNCV- Menara Kuningan Country Director Challenge TB agnes.gebhard @kncvtbc.org 62-21 83793350/51/52 - SUFA implementation and acceleration, TB￾HIV integration, including in Papua 59 | LINKAGES MIDTERM EVALUATION REPORT USAID.GOV Time Venue Name Position Contact Details Possible Areas to Discuss TUESDAY, 29 Augt 2017 09.00 – 10.30 Other USG HIV Program (USD HIV) (Mega Kuningan) David Papworth Arief Sudrajat Chief of Party GHSC PSM (Global Health Supply Chain - Procurement Supply Management) Supply Chain Technical Director DPapworth @ghsc-psm.org ASudrajat @ghsc-psm.org 087882405624 - Collaboration on SUFA implementation and acceleration 11.30 – 13.00 Angsa Merah (Panglima Polim) Felix Neuenschwander Program Manager - Angsamerah Clinic 082114411421 - Use of data to improve quality and up take of clinical services 14.00 – 15.30 PKM Pasar Rebo or DHO Jakarta Timur (Cijantung) Dr Ovy DHO Jakarta Timur 0816-4275-567 - Jakarta district level coordination and fast track implementation Application of LINKAGES tools and or collaboration with LINKAGES on CoPTC priorities Dr Sahiyatun Nawiyah Koord HIV Puskesmas Pasar Reb 0813-1675-8339 - CoPTC service delivery areas - LINKAGES clinical mentoring - Application of LINKAGES tools including cascade generator and DOKLING WEDNESDAY, 30 Augt 2017 09.00 – 10.30 Kala Fin Chief of Party MAJU – The Asia Foundation (BARITO Blok M) kala.finn @asiafoundation.org - Enabling environment particularly on LGBT (this partnership has not yet been activated) USAID.GOV LINKAGES MIDTERM EVALUATION REPORT | 60 ANNEX III: DATA COLLECTION INSTRUMENTS INTERVIEW GUIDE FOR THE PROVINCIAL AIDS OFFICE 1. What are your own aspirations for halting the epidemic in Papua, what are you hoping to do, or do more effectively, as you move forward? 2. What have you been able to achieve in the last 2 years that you are most proud of? 3. Are there any improvements or new approaches you are thinking of introducing? What do you need more evidence for? In order to change things at the system level? What should any operational research be looking at? 4. What are the 3 key things they need additional TA and capacity building on to really improve the response in Papua 5. How are they learning from other systems, like retaining people with diabetes in the chronic disease system? How do they talk across the health system? Have you been able to leverage any other programs to benefit the HIV program? (prompt for distribution of treatment, support for transport or multi-month scripting) 6. What programmatic areas do you receive technical assistance from LINKAGES to address? Note to Interviewer – in which of these areas • Prevention, outreach • HIV testing and counselling • Linking PLHIV to care and treatment services • Initiating PLHIV on Care and treatment • Retaining PLHIV in the treatment system • Combating stigma and discrimination among Health Care Workers • Promoting an enabling environment • Monitoring progress and effectiveness (data collection, data management and quality, turning data into information, using data to inform decision making) 7. What has LINKAGES done to help you respond to the challenges you are experiencing in the HIV program? Note, LINKAGES approach includes TA to enhance: • Reach and re-engage • High Yield • Test and Start • Viral suppression • Cross cutting areas for TA: gender based violence, stigma, human rights; data use and monitoring systems; capacity building 8. Have you experienced any benefit from the recent data review and consensus workshops? 9. How do you break the targets down to the District and facility level to make sure you meet Provincial targets? Can you track it and give feedback and support. What else do you use besides the targets to motivate people to work together and achieve good results? 61 | LINKAGES MIDTERM EVALUATION REPORT USAID.GOV Interview guide for the Provincial Health Office and the District Health Offices (Technical Assistance – Service Delivery Improvement) PART A: GENERAL QUESTIONS 1. What programmatic areas do you receive technical assistance from LINKAGES to address? Note to Interviewer – in which of these areas • Prevention, outreach • HIV testing and counselling • Linking PLHIV to care and treatment services • Initiating PLHIV on Care and treatment • Retaining PLHIV in the treatment system • Combating stigma and discrimination among Health Care Workers • Promoting an enabling environment • Monitoring progress and effectiveness (data collection, data management and quality, turning data into information, using data to inform decision making) 2. What has LINKAGES done to help you respond to the challenges you are experiencing in the HIV program? Note to interviewer – identify the nature of the TA: • Clinical support • Feasibility Assessments • Research • Program Design • Program Monitoring • Impact or other evidence generating Assessments • Legal and Policy Review and Negotiation • Advocacy or advocacy capacity building • Partner Coordination • Data Analytics, cascade generator, Note, LINKAGES approach includes TA to enhance: • Reach and re-engage • High Yield • Test and Start • Viral suppression • Cross cutting areas for TA: gender-based violence, stigma, human rights; data use and monitoring systems; capacity building 3. To what extent have LINKAGES activities strengthened the capacity of CSO and GOI partners in achieving the goals and targets of Indonesia’s National HIV Program? What has been done, how are the activities planned and selected, are the capacity building efforts relevant, acceptable, likely to result in better outcomes for KAP USAID.GOV LINKAGES MIDTERM EVALUATION REPORT | 62 4. To what extent have project activities strengthened the capacity of national and sub-national (province and district) governments and other stakeholders in managing Indonesia’s National HIV Program? 5. How were you able to identify your technical assistance needs? Prompts: • Was a TA needs assessment conducted? Who participated? Has this process changed over time? • Who makes the decisions about the TA you receive? Are you able to hold that decision maker accountable? • How do you decide if TA is an appropriate solution for the development challenge, opportunity or need your program is experiencing? How do you know if TA will help to achieve the desired program outcome(s)? 6. How does the TA system work? Was a TA request process developed? Were TA feedback report templates developed? Do the TA provider and recipient determine TA needs together, review the success of the TA process together, plan for changes in practice as a result of TA 7. Would you describe the TA you receive as: • Comprehensive (why/why not?) • Well integrated into your workplace/ other activities (why/why not?) • Enhancing your understanding of innovative approaches that yield results(why/why not?) 8. What are the strengths/weaknesses of the way LINKAGES provides you with Technical Assistance? Is it dynamic, continuous, inspiring? 9. Are there any barriers to implementing change that affect the potential effectiveness of the Technical Assistance you receive? 10. How do you assess whether the Technical Assistance is benefitting you/your program? Are you negotiate with LINKAGES if the TA needs to be modified? 11. To what extent does the Technical Assistance contribute to longer term capacity building for your team? 12. Would you suggest any changes in LINKAGES’s approach? 13. What do you consider to be LINKAGES’s biggest contribution to your own efforts to [address HIV]? 14. Note to interviewer: Specify if appropriate for e.g. …your own efforts to improve outreach, your own efforts to increase participation in HCT? Increase the yield from HCT? Enrol those who test positive in treatment and care? Increase retention in ART treatment? 15. In the remaining project years, what are the highest priorities for LINKAGES to focus on? 16. How will you sustain the HIV and the TB integration programs after the international donor funds program ends in 2019? 63 | LINKAGES MIDTERM EVALUATION REPORT USAID.GOV PART B: QUESTIONS SPECIFICALLY FOR PAPUA PROVINCIAL HEALTH OFFICE AND THE PROVINCIAL AIDS OFFICE 1. Can you tell me about the Papua central support strategy and LINKAGES contributions? 2. Can you tell us about the HIV Program in Papua? What are your key challenges? What are the strengths of the program? What are your priorities? How successful do you think your program is? Why? 3. How has LINKAGES been involved in TB-HIV program integration and coordination? Can you tell me about the findings of the TB/HIV Preliminary Assessment in Papua, and the actions you decided to take as a result of the assessment? 4. Can you tell me the proportion of ART patients who were screened who are receiving TB treatment (TX_TB)? 5. How well are the TB and HIV programs working together? 6. How have you worked with LINKAGES on VMMC? What are the results so far? 7. How useful is the current work bringing together people with a range of expertise to look at the epidemiology of HIV in Papua and set targets? 8. What are the issues surrounding mobile testing, and how are they being addressed? 9. What is happening to decentralize the supply of ART? How has TA assisted you in relation to ARV logistics, distributing reagents and viral load test kits? 10. There seem to be some impediments to test and start, can you explain how you are reducing the barriers for PLHIV? 11. Can you tell me about how you are using the cascade generator? 12. Where are you up to with the Outreach Databank development and improvement? How is LINKAGES assisting with this project? 13. Have any new systems or solutions developed in one District been adopted by or rolled out to other Districts? 14. How do you enable the CSO working with the different KAP to work together? How about for PLHIV who use drugs, or MSM that sell sex, or sex workers who use drugs? How do you reach people and support them if they have a range of risk factors? 15. How well developed is the Draft SOP for HIV screening of TB patients at puskesmas? Are you happy with the way this has been developed? PART C: QUESTIONS SPECIFICALLY FOR THE DISTRICT HEATH OFFICES 1. How long have you been working on HIV at this DHO? Are you familiar with the LINKAGES program? 2. Can you tell us about the HIV Program in this District? What are your key challenges? What are the strengths of the program? What are your priorities? How successful do you think your program is? Why? USAID.GOV LINKAGES MIDTERM EVALUATION REPORT | 64 3. How many ART initiation sites are there in this District? What is needed to decrease the delays in Test and Start, and increase the quality of the treatment and its effectiveness? What plans do you have to increase access to viral load testing, or any other mechanism for monitoring treatment effectiveness? 4. What assistance and support do you receive from LINKAGES? 5. Can you tell me about how you are using the cascade generator? What difference does it make to the work you do? Can you give me an example of any changes you have made, based on what you can tell from the cascade data? 6. How do you link and coordinate with the CSO in your district to provide services for HIV and TB? 7. What are the challenges to providing mobile testing? How do you link and coordinate with the CSO in your district to coordinate mobile testing services? What are the testing quality concerns? How can they be addressed? 8. Where are you up to with the Outreach Databank development and improvement? How is LINKAGES assisting with this project? 9. Have any new systems or solutions developed in one District been adopted by or rolled out to other Districts? 10. Can you explain how your cascade metrics and EOA performance thresholds are used to measure and track programmatic outcomes? 11. What are the key elements of the MOU you have with CSO in your district? Have the new MOU’s helped you to enroll PLHIV in treatment and retain them? PART C: QUESTIONS SPECIFICALLY FOR THE PUSKSESMAS 1. How long have you been working on HIV at this puskesmas? Are you familiar with the LINKAGES program? 2. How do you work with the CSO in your district to coordinate testing services? Including any mobile testing services, and follow up? What about the quality of the testing? Are there any quality assurance mechanisms? 3. Can you tell me about the use of the Client Management Data base, and if it has made any difference to the testing yield here over the last 6 months? 4. What assistance and support do you receive from LINKAGES? 5. Can you describe the clinical mentoring support you receive? Would you describe the assistance you receive as: • Comprehensive (why/why not?) • Well integrated into your workplace/ other activities (why/why not?) • Enhancing your understanding of innovative approaches that yield results (why/why not?) 6. What are the key elements of the MOU you have with CSO in your district? How are you working together to reduce the high rates of loss to follow up of PLHIV? What implications do these rates 65 | LINKAGES MIDTERM EVALUATION REPORT USAID.GOV have for your programming ? (prompt re multi-drug resistance and need for second line, forecasting, death rates, impact on willingness to test) 7. Has the Enhanced Outreach approach helped you to test more people for HV? Or to enrol PLHIV in treatment and retain them? How have things changed since the Community Support workers have been case managing people living with HIV, not just doing prevention outreach? 8. Is there a quality assurance system in place? How do you use it? Have you had any assistance from LINKAGES to set up or operate the QA system? 9. What are you doing to support good adherence among the PLHIV and retain them in the care and treatment system? Where do these efforts fall down? What are the results of ineffective support for the PLHIV? 10. When will you start implementing the SOP for HIV screening of TB patients at puskesmas? Are you ready to implement the SOP for HIV screening of TB patients at puskesmas? In some places there has been reluctance among health professionals to screen TB patients for HIV, has this been an issue here? What have you done to overcome this reluctance and prepare the health professionals? 11. How do you make sure all sorts of people, regardless of their risk factors, feel respected and welcome at this service? 12. Why is there such a delay in initiating treatment for people testing HIV positive? How many tests must they have before initiating treatment? What are the processes you go through when /before initiating treatment? What are the challenges for puskesmas that have become Treatment Initiation sites? Do you require additional TA to be able to confidently initiate treatment? Are there other things you need? (Prompt re better support from the more experienced providers at the main hospitals, clearer treatment guidelines.) INTERVIEW GUIDE FOR CSO PARTNERS IN PAPUA 1. What services do you provide for priority populations? Do you have ownership of the activities, would you provide these services /use these services if LINKAGES was not paying for it? Is it really valued by you? 2. What has working with LINKAGES been like for your CSO? 3. In the past the result of the HIV test was given by the counsellor, and their role stopped there, no one would make sure the person receives follow up treatment. Has this changed with the introduction of the Community Based Supporters and the Enhanced Outreach Approach? How? How much? 4. Can you describe the Technical Assistance/support you get from LINKAGES? 5. Would you describe the demand creation assistance you receive as: • Comprehensive (why/why not?) • Well integrated into your workplace/ other activities (why/why not?) • Enhancing your understanding of innovative approaches that yield results (why/why not?) USAID.GOV LINKAGES MIDTERM EVALUATION REPORT | 66 6. Is there any evidence of impacts, in the form of increased demand for existing services? Or demand for new and emerging technologies or services? What do you think TA has helped you to achieve? 7. What are the different factors affecting the demand for services by different kinds of people in your district? 8. What are you doing to support good adherence among the PLHIV and retain them in the care and treatment system? Where do these efforts fall down? What are the results of ineffective support for the PLHIV? 9. Besides LINKAGES activities, what else may be affecting demand for services? 10. How would you describe the factors that contribute to the success of the demand creation activities? And the challenges to enhancing and sustaining demand? 11. Are you using an Enhanced Outreach Approach in this District? How is this different to the way you were doing outreach in the past? 12. What do you offer in your service package to people living with HIV? 13. What else, besides HIV services, do you offer to people living with HIV? How do you make sure people get all their health care needs met? 14. Are there systems collecting the same data? Which systems and services have a specific data base? Are they duplicating the information collected? Are the online systems helpful? To what extent does the mobile data base help the CSO to do outreach? 15. How do you use the EOA tools to identify new, high-risk (“focus”) clients who either (a) need an HIV test; (b) need HIV treatment; or (c) need help reducing their HIV risk or need support to be retained in the HIV service system. 16. How does the “peer mobilizer” (peer volunteers) system work to help your community-based supporters expand outreach coverage, and recruit additional clients from among networks of friends and sexual/injecting partners? 17. How do you know if you are effectively assisting clients to access and navigate clinical services like HIV testing, treatment, and viral load testing, and other relevant services like TB and methadone maintenance treatment? Is there a quality assurance system? Or do you get regular feedback from the clients somehow? 18. What systems do you have to ensure the Community based supporters follow-up “focus” individuals to support them to register in care, initiate ART, keep appointments and/or be retained in the HIV service system? What are your loss to follow up rates? What do you do to keep these rates as low as possible? 19. What support do you need to address things like working with clients that don’t want to use condoms? 20. How do we enable the KAP to share information with their peers, when the service provider is no longer there? 21. How do you understand the extent of the networks, and how do you work through peer networks? 67 | LINKAGES MIDTERM EVALUATION REPORT USAID.GOV 22. How do you support KAP to serve as role models? What capacity building have you provided to enable KAP to promote behavior change among their peers? 23. Do you think the KAP would be more likely to come to the service if there were other KAP working there/providing the services? 24. To what extent has the CSO enabled the development of KAP, so they can take on leadership roles, or address barrier issues such as violence 25. To what extent do the data systems help CSO develop their systems, and do better, more targeted outreach? Note to interviewers: Following the ‘5Ps’ from the Enhanced Outreach Approach means that community￾based supporters will do the following, how often do these things get mentioned? 1. Prioritize contacts with key or priority populations, with a focus on identifying undiagnosed, unstable, or loss-to-follow up PLHIV. 2. Protect sensitive information on client identities and behavioral risk information through strict adherence to the use of UICs. 3. Perform a personalized risk assessment that segments client needs for differentiated – or customized - support. 4. Promote the benefits of HIV testing, ART, TB screening, retention in the HIV service system, viral testing, HIV prevention products and other related health services through guided social and behavior change communications prompts. 5. Provide targeted communications (messaging and/or materials), and prevention products, such as condoms, needles, or in some cases, transportation assistance. 6. Pass it on, by sharing materials and optimizing referrals through a client’s KP or PLHIV networks. INTERVIEW GUIDE FOR KAP/PRIORITY POPULATIONS Please choose from the questions below: 1. How do you know what services are available? Do you know where they are and who can use them? How they work? 2. If you are visited by the NGO, what do you get, has this changed over the last few years, or are the services you get the same? 3. Do you think the range and type of services that the community-based supporter and the CSO provide are right for you? Are these services really needed by you? Would you use these services if LINKAGES was not paying for it? 4. What HIV services have you used? Including receiving information or other things from the Community based supporter? Has the service met your basic needs? Is it really valued by you? Do you ever use the testing/treatment services? If no, what are the reasons you don’t use these services (convenience, remoteness, privacy, fear of what they find out, fear of unfriendly services, what is your definition of a friendly service) USAID.GOV LINKAGES MIDTERM EVALUATION REPORT | 68 5. What benefit do you think you get from all these services? How do you feel about being able to get this support and these services? Be specific (free condoms, HIV testing, transport support to get to the puskesmas/hospital) 6. Why do you use this service? Does the information you get from the community-based supporter influence what you do, and what services you use? 7. What do you think constitutes a ‘friendly’ service? Prompts: Attitude, availability, infrastructure, the tools and equipment and perishables 8. In the past the result of the HIV test was given by the counsellor, and they would not make sure the person receives follow up treatment. But now, the whole process in the hands of one person, the community-based supporter, who supports you to get HIV testing, access to condoms, and then if people need HIV treatment they take them to get enrolled in care. Have you noticed any changes since the community-based supporters started? Have you got a community support person? What do they do with you? 9. When the community-based supporter or outreach worker asks you for your details, like age, where you live, do you know your HIV status, do you think it is of benefit to give them this information? Do you know where there data and details go? Do you know who has access to your information? 10. If the condoms are not given freely, do you ever buy them? Where from? 11. Who do you want to provide the services you need? Who do you want to get prevention education, testing, linked to treatment by? 12. Have you been tested at a mobile HIV testing service? Is this a service you want to use? Why? What do you think of the mobile testing service? How well does this type of service protect your privacy? Is protecting the information about your status important to you? 13. Is there any difference when you go to the facility by yourself, compared to if you go there with an outreach workers or a CSO worker? 14. What do you know about the availability of HIV treatment? Where do you get your information? 15. What impediments have you encountered in accessing prevention, testing, and treatment and care services? 16. Does the program /service have a system for allowing you to say what you want? Do they respond to your requested changes? Or do you have to ask for improvements over and over? 17. Can you get the services you need without having to go to a government health Centre or service and sharing private information? Do you have to reveal a lot of personal information before you can get services? 18. Can you get what you need (injecting equipment, condoms, testing) without having to show your ID card, or without exposing yourself to stigma and discrimination? 69 | LINKAGES MIDTERM EVALUATION REPORT USAID.GOV ANNEX IV: SOURCES OF INFORMATION DOCUMENTS REVIEWED TA AREA DEFINE DESIGN DEVELOP DO DETERMINE DEMONSTRATE DUPLICATE Results Area 1 Reach & Re-Engage Site-Level 1. Outreach Flow & Standards Assessment (Jan. 2016, JKT/Sep. 2016, Papua) 2. ICT assessment report (Apr 2016, JKT) 3. Facilitator Guide, MSM/JKT (Sep 2016) Site-Level 1. EOA Service Package, April 2017 2. MSM FGD Discussion Notes, Oct 2016 Site-Level 1. CSO sub-awards, April 2016 – September 2017 (JKT/Papua) [see Sharepoint site] 2. EOA M&E tools package 3. Client Management Database (CMD) [demo only] Site-Level 1. EOA Phase 1 Training and Consultation Curricula & Consolidated Draft Report, Jun 2017 2. CSO outreach performance dashboards (up to June 2017) 3. Client Management Database (CMD) [demo] Site-Level 1. CSO outreach performance dashboards (up to June 2017) 2. SIMS schedule & scoring, 2016 – 2017 (see RSA 3) 3. CMD dashboard templates, Draft Jun 2017 4. Differentiated outreach conceptual framework, draft Jul 2017 Site-Level 1. Outreach Technical Performance Threshold Tracking (Drafts, Q1- 3/FY17) Site-Level 1. Outreach TA package, Draft April 2017 2. LINKAG ES – Spiritia Outreach Partnersh ip Discussio n PPT, Draft, Apr 2017 Central Support 1. GFATM UIC Report, June 2016 (contributor) 2. ICT Approach, Apr 2016 3. Sex Worker core package concept note development (contributor), Oct 2015 Central Support 1. LINKAGES ICT Strategy, Jan 2016 2. TA Strategy for CSOs, Draft, Aug 2016 3. COMS Overview, Draft June 2016 4. 6-pack MSM ICT Intervention Package, Draft Jun 2017 5. HAVAS SOW & Deliverables, Draft Sept – Dec 2017 Central Support 1. PAC BCC Analysis & Recommendations PAC, 2015 2. COMS Overview, Draft June 2016 Central Support 1. KP (MSM) Hotspot Mapping, PAC, Dec 2015 2. Databank Manual (JKT), Draft Mar 2017 Central Support 1. Partnership agreement (SOW), Big Data Indonesia, Draft Jun 2017 2. Partnership agreement (SOW), PLUS, Draft Jun 2017 Central Support 1. Jakarta Databank Tracking Report, Q1-3, July 2017 Central Support 1. Databank Manual (Jayapura), Draft August 2017 2. LINKAGES – Spiritia Outreach Partnership Discussion PPT, Draft, Apr 2017 3. Databank Brief, August 2017 [demo] USAID.GOV LINKAGES MIDTERM EVALUATION REPORT | 70 TA AREA DEFINE DESIGN DEVELOP DO DETERMINE DEMONSTRATE DUPLICATE High Yield HIV Testing Site-Level 1. CSO Mobile HIV Testing Assessment, Nov 2016 Site-Level 1. Facility TA Package, Preliminary Draft, Jun 2017 Site-Level 1. Clinic sub-awards, April 2016 – September 2017 (JKT/Papua) [see SharePoint site] 2. UNITAID self-testing scale-up proposal (available upon request) Site-Level 1. CoPTC Clinical Mentoring TA Snapshot, Q1/FY17 (JKT) Site-Level 1. Clinical Cascade Performance Dashboards (up to June 2017) 2. HTC Performance, Extended Hours Facility, JKT (e.g. Taman Sari), Jan 2017 Site-Level 1. HTC Technical Performance Threshold Tracking_JKT (Drafts, Q1-3/FY17) 2. HTC Technical Performance Threshold Tracking Papua (Drafts, Q2-3/FY17) Site-Level 1. LINKAGES – Spiritia Outreach Partnership Discussion PPT, Draft, Apr 2017 (in Reach & Re￾Engage; see also Central Support) Central Support Central Support 1. DOKLING Overview, Jun 2017 Central Support 1. DOKLING Manual, Draft Apr 2017 Central Support 1. DOKLING database [demo] Central Support 1. CONTROL (Addition to DOKLING), Draft Jul 2017 Central Support 1. DOKLING, 2-Month Implementation, Jun 2017 Central Support Test & Start & Retain (SUFA Implementation & Acceleration) Site-Level 1. HRH Analysis, JKT, Draft Jun 2016 2. CSO-Health Facility Networks/JK T, Dec 2016 3. CSO-Health Facility Networks/Pa pua, Dec 2016 4. SUFA Target Facilities, Jakarta, Jul 2017 Site-Level 1. SUFA Acceleration TA Strategy, Preliminary Draft Jun 2017 Site-Level 1. Site-Level TA Checklist, 2016 2. Clinic sub-awards, April 2016 – September 2017 (JKT/Papua) [see SharePoint site] 3. EMMP Checklist, 2016 Site-Level 1. SIMS Follow Up Plan, e.g. Angsamerah/JKT, Jan 2017 2. Clinical Mentoring Visit, e.g. PKR Wamena Kota/Jayawijaya, Aug 2016 3. Clinical Mentoring Visit, e.g. PKR Asologaima/ Jayawijaya, Mar 2017 Site-Level 1. Clinical Cascade Performance Dashboards (up to June 2017; see file in HIV testing) 2. KP-Friendly Clinic (DSD) Cascade Performance, Q1/FY17, Jakarta 3. Doctor2Doctor Hotline Review Draft, Jun 2017 Site-Level 1. SUFA Technical Performance Threshold Tracking Tool (see RSA 3, Performance Parameters) Site-Level Central Support Central Support Central Support 1. ARV De￾Centralization Process, Jayawijaya, Draft May 2017 Central Support 1. National Clinical Mentoring Workshop(s) (materials available upon request) Central Support Central Support Central Support 71 | LINKAGES MIDTERM EVALUATION REPORT USAID.GOV TA AREA DEFINE DESIGN DEVELOP DO DETERMINE DEMONSTRATE DUPLICATE 1. Summary, Supply Chain Baseline Data (Supply Chain Program), May 2017 1. SUFA Acceleration TA Strategy, Preliminary Draft Jun 2017 (see Site-Level) 2. National Clinical Guidelines & Mentorship Materials (available upon request) [contributor] TB/HIV Integration Site-Level 1. TB/HIV Preliminary Assessment , Papua, Draft Nov 2016 (CTB co-funding) 2. Priority TB/HIV TA Facilities, Jakarta, Aug 2017 Site-Level 1. TB/HIV TA Approach with CTB, Draft Nov 2016 2. TB/HIV Integration Model, Papua, Mar 2017 Site-Level 1. EOA M&E Package (see Reach & Re￾Engage) 2. TB/HIV Acceleration Strategy, Preliminary Draft Mar 2017 (with CTB) Site-Level 1. TB/HIV TA Activity Highlights, Jul 2017 Site-Level 1. CMD dashboard templates, Draft Jun 2017 (see Reach & Re-Engage section) Site-Level 1. TB/HIV Technical Performance Threshold Tracking Tool (see RSA 3, Performance Parameters) Site-Level Central Support 1. JEMM TB Review (available upon request) [contributor] Central Support 1. TB/HIV Recommendations (JEMM), Mar 2017 [contributor through support to CTB] Central Support 1. SOP, IPT, Draft Apr 2017 2. SOP, TB Screening for PLHIV, Draft, Apr 2017 3. SOP, TB Screening for PLHIV, Draft, Apr 2017 Central Support Central Support Central Support 1. District-Level TB/HIV Performance Dashboards (CTB) [available upon request] Central Support Viral Suppression Site-Level Site-Level 1. VL Suppression TA Approach, Draft Mar 2017 Site-Level 1. Clinic sub-awards, April 2016 – September 2017 (JKT/Papua) [see Sharepoint site] Site-Level Site-Level 1. VL Testing Coverage (Ruang Carlo, Jakarta), 2016 2. VL Testing Coverage (Ruang Carlo & Angsamerah, Jakarta), 2017 Site-Level Site-Level USAID.GOV LINKAGES MIDTERM EVALUATION REPORT | 72 TA AREA DEFINE DESIGN DEVELOP DO DETERMINE DEMONSTRATE DUPLICATE Central Support Central Support 1. VL Suppression TA Approach, Draft Mar 2017 (see Site Level) Central Support 1. GeneXpert Testing Estimator Tool, Draft May 2017 2. VL Testing Pilot, 2017 (contributor) Central Support Central Support Central Support Central Support Results Area 2 PrEP Feasibility in Jakarta Central Support Central Support 1. PrEP Acceptability Protocol, Mar 2017 2. PrEP Implementation Study Synopsis, May 2017 3. PrEP’d Research Project Protocol, Version 2, Aug 2017 Central Support 1. Regional PrEP Consultation Agenda, Jan 2017 2. PrEP Acceptability Study Narrative Report, June 2017 3. PrEP Acceptability Study Narrative Report, July 2017 Central Support 1. Draft PrEP Workplan, 2016-2017 Central Support Central Support Central Support VMMC TA in Papua Central Support 1. Avenir Health Modeling Synopsis and PPT, Feb 2017 2. VMMC in Papua Summary, Feb 2017 Central Support 1. VMMC TA Rationale, Feb 2017 2. VMMC Expert Recommendations , Mar 2017 Central Support 1. VMMC Expert Review Report, Mar 2017 Central Support Central Support Central Support Central Support 73 | LINKAGES MIDTERM EVALUATION REPORT USAID.GOV TA AREA DEFINE DESIGN DEVELOP DO DETERMINE DEMONSTRATE DUPLICATE KP Engagement in HIV Service System Central Support 1. KP Programming Priorities in Jayapura, Dec 2015 2. GBV Assessment 1, Papua, Jun 2016 3. Papua Preliminary GBV Assessment Debrief_Sep 2016 4. GFATM Pre￾Assessment of Human Rights Barriers for KP and TB Clients, Overview, July 2017 (replication/as sessment by USAID) Central Support 1. GBV TA Approach in Papua_ Draft, Sep 2016 2. Training Proposal for Health Worker Stigma and Discrimination Reduction, Draft 1, 2017 3. SMS2 Conceptual Framework (HQ), Draft Jan 2017 4. VPR Core Training Package (HQ), Jun 2017 5. RFP_new KP Clinic (JKT), Jul 2017 Central Support 1. IAC Sub-award_FY17 (see Sharepoint site, discuss IAC deliverables) 2. Summary, IDUIT Guidance, 2017 3. IDUIT Guidance, Indonesian, 2017 4. Health4All Training Guide, Draft Indonesian, Jan 2017 5. Country Action Plan for Stigma & Discrimination Reduction, May 2017 Central Support 1. IAC Sub-Award Deliverables (discussion) Central Support 3. Developing & Implementing KP VPR Program Elements, Draft Jun 2017 Central Support Central Support USAID.GOV LINKAGES MIDTERM EVALUATION REPORT | 74 TA AREA DEFINE DESIGN DEVELOP DO DETERMINE DEMONSTRATE DUPLICATE Results Area 3 Stakeholder Coordination, Planning & Implementation Central Support 1. GFATM PR TA Needs, Jul 2016 2. e-Health Start Ups in Indonesia, November 2016 3. La Pago epidemic control consultation report, Apr 2016 4. HIV National Review, 2016 (contributor) 5. Stakeholder Coordination Consultation, Jayawijaya, Mar 2017 Central Support 1. Jakarta Fast Track Roadmap, Draft Aug 2016 2. TOR, Stakeholder Analysis, Jan 2017 3. Jakarta Fast Track Action Plan template, draft May 2017 Central Support 1. Jakarta Fast Track Target Setting Exercise, Jul 2016 2. CCM Sub-Award, FY17 (see SharePoint Site) 3. PHO Papua – LINKAGES Work Plan Presentation (meeting minutes, Jan 2017) 4. Mimika Strategic Plan Draft, Feb 2017 (contributor) 5. GFATM Concept Note Development, e.g. PWID TWG, Mar 2017 6. SOW, Time-Bound Personnel (JKT), Draft Jan 2017 7. Papua Target Setting Exercise, May-June 2017 8. GFATM Concept Note (see SharePoint site, Enabling Environment) Central Support 1. Report on HIV situation, response & gaps in Jayawijaya, 2016 (SharePoint/Papua) 2. Workflow, Data Flow & Client Flow Workshop Report, PHO, August 2016 3. Stakeholder Analysis Report, Draft July 2017 Central Support Central Support 1. Jakarta Cascade Performance, 2016 2. Performance Against Fast Track Targets, 2016-2017 3. Jakarta Cascade Performance, Jan – Mar 2017 4. Papua Cascade Performance, Jan – Mar 2017 Central Support 75 | LINKAGES MIDTERM EVALUATION REPORT USAID.GOV TA AREA DEFINE DESIGN DEVELOP DO DETERMINE DEMONSTRATE DUPLICATE M&E/SI System Strengthening (deliverables are also found in all other RSA) Site-Level & Central Support 1. CSO M&E Assessment, e.g. REMPAH, Jan 2016 2. LINKAGES Master Site List, FY17 Site-Level & Central Support 1. IBBS Questionnaire (MSM), Jul 2017 (contributor) Site-Level & Central Support 1. Epidemiology Update, Nov 2016 (contributor) 2. KP Population Size Estimates, Nov 2016 (contributor) 3. SIHA verification tool, 2017 4. SOW_ Time-Bound M&E Personnel (Papua), Jan 2017 5. Cascade generator user guide, Mar 2017 Site-Level & Central Support 1. Workflow, Data Flow & Client Flow Workshop Report, PHO, August 2016 (in Stakeholder file) 2. SIHA System Maintenance, Tri SoW, 2016-2017 3. IBBS RDS Field Guide, May 2017 4. Cascade Generator Overview, Jun 2017 Site-Level & Central Support 1. SIMS Schedule & Action Panning, 2016-2017 Site-Level & Central Support 1. High Performing Sites, 2015 2. DQA Results, 2016 Site-Level & Central Support 1. Cascade generator [demo] Technical, Programmatic & Financial Performance Site-Level 1. ITOCA Instrument, Apr 2016 2. OPI Domains & Scores, Apr 2016 3. CoPTC Technical Performance Threshold Operational Guidance, Update Jul 2017 4. Pull TA Decision Path, Draft 2017 Site-Level 1. LINKAGES TA Approach, 2016 2. Leadership Module TA Strategy, Draft Mar 2017 3. ToR Financial and HR Management TA, 2017 Site-Level 1. Institutional Strengthening Plan (ISP), e.g. Bandungwangi Apr-Sep 2016 2. CSO HR Systems Strengthening Module, 2017 3. CSO Leadership Strengthening Plan, Draft Mar 2017 4. CoPTC Technical Performance Snapshot, Draft Aug 2017 (forthcoming) Site-Level 1. ITOCA Facilitation Guide, 2016 2. DSD Partner Feedback Consultation Agenda (Papua), Aug 2016 3. LINKAGES TA Highlights, Apr 2017 4. DSD Partner Feedback Consultation Agenda (JKT), May 2017 5. Finance & HR Mgmt Workshop Report, Mar 2017 6. CoPTC Technical Threshold QI Decision Path, Draft Aug 2017 Site-Level 1. OPI Round 1 Result, e.g. Angsamerah, 2016 2. OPI Round 2 Result, e.g. Angsamerah, 2017 3. Technical Threshold Performance Tracking (see Results Area 1) 4. Sub-award (Financial) Performance Tracking, Jun 2017 Site-Level 1. Grants Management Dashboard (last updated June 2017) 2. Community Solutions Platform (demo) Site-Level 1. LINKAGES – Spiritia Outreach Partnership Discussion PPT, Draft, Apr 2017 (see in Reach & Re￾Engage, RSA 1) 2. IAC, GFATM PR, Collaboration Meeting Notes, Jun 2017 USAID.GOV LINKAGES MIDTERM EVALUATION REPORT | 76 BIBLIOGRAPHY Bell, Stewart. The Martyr’s Oath: The Apprenticeship of a Homegrown Terrorist. Mississauga, ON: Wiley, 2005. Biale, David, ed. Cultures of the Jews: A New History. New York: Schocken, 2002. Bowker, Michael. Fatal Deception: The Untold Story of Asbestos: Why It Is Still Legal and Still Killing Us. N.p.: Rodale, 2003.