November 2017 This publication was produced for review by the United States Agency for International Development. It was prepared independently by Soliman F. Guirgis, Pandu Riono, Aang Sutrisna, and Fitriyani Yasir on behalf of Management Systems International, a Tetra Tech Company. MIDTERM EVALUATION USAID/INDONESIA CHALLENGE TB PROJECT USAID/Indonesia Challenge TB Project Midterm Evaluation i MIDTERM EVALUATION USAID/INDONESIA CHALLENGE TB PROJECT USAID Contract No. AID-497-C-16-00006 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. USAID/Indonesia Challenge TB Project Midterm Evaluation ii ACKNOWLEDGEMENTS The Challenge TB (CTB) Midterm Evaluation is the consolidated work of the evaluation team commissioned by USAID/Indonesia. However, it would not have been possible without the kind support and help of many individuals and organizations. The evaluation team would like to express its sincere thanks and appreciation to the Indonesian officials from the Ministry of Health, Ministry of Home Affairs, and Ministry of Law and Human Rights at the central, provincial, and district levels who arranged their schedules to meet with us and share their insights on the CTB project activities. Also, the evaluation team would like to thank the CTB project management and staff in the central and regional offices for coordinating and facilitating the visits and sharing their comments and suggestions on the project’s activities. Special thanks should go to KNCV and the CTB Chief of Party, and to the project’s central and regional staff and implementing partners—FHI 360, the World Health Organization (WHO), and Interactive Research and Development (IRD)—for their cooperation in providing documentation and information about their activities, as well as their assistance in arranging contacts and meetings with relevant stakeholders. The evaluation team is also grateful for the guidance, support and inputs provided by Mr. Jonathan Ross, Ms. Zohra Balsara, and Dr. Alia Hartanti from the USAID/Indonesia Health Office; Ms. Amita Kulkarni from the USAID/Indonesia Program Office; and Dr. William Wells, Mr. Nicholas Enrich and Mr. Kaiser Shen from USAID/Washington. Finally, special thanks go to the USAID Monitoring and Evaluation Support Project (MESP) for their technical and logistical support to the design and implementation of this evaluation, particularly Mr. David Callihan, Ms. Retno Handini, and Mr. Jonathan Simon. USAID/Indonesia Challenge TB Project Midterm Evaluation iii CONTENTS Acknowledgements............................................................................................................................................ ii Contents.............................................................................................................................................................. iii List of Figures...................................................................................................................................................... iv List of Tables....................................................................................................................................................... iv Acronyms and Abbreviations........................................................................................................................... v Executive Summary ............................................................................................................................................1 Project Background............................................................................................................................................9 Evaluation Purpose & Evaluation Questions...............................................................................................11 Evaluation Purpose ............................................................................................................................11 Evaluation Questions........................................................................................................................11 Evaluation Methods & Limitations................................................................................................................12 Data Collection Tools and Instruments......................................................................................................13 Data Analysis......................................................................................................................................................15 Findings, Conclusions, And Recommendations.........................................................................................16 Q1. Project Performance.................................................................................................................16 Q2. Program Management ..............................................................................................................26 Q3. Information Generation And Use .........................................................................................30 Q4. Adaptation To Indonesia’s Health System...........................................................................34 Q5. Lessons And Best Practices ....................................................................................................37 Q6. Sustainability ...............................................................................................................................40 Findings from CTB Papua ................................................................................................................42 Long-Term Recommendations for USAID.................................................................................................45 Annex I: Evaluation Statement of Work .....................................................................................................46 Annex II: District Selection Criteria ...........................................................................................................59 Annex III: Data Collection Instruments ......................................................................................................60 Annex V: Disclosure of Any Conflicts of Interest................................................................................. 102 Annex VI: Response of CTB to the Mid-Term Evaluation................................................................... 104 USAID/Indonesia Challenge TB Project Midterm Evaluation iv LIST OF FIGURES Figure 1: CTB Progress toward Quantifiable Targets against Expected Results, 2015-2017....................................17 Figure 2: Number of CTB Key Activities in APA-3 (Quarters 1-3) and Progress against Targets ........................18 Figure 3: Percentage of TB Cases Notified out of Estimated Cases by District Category, 2015–2017.................20 Figure 4: Number of CTB Key Activities in APA-3 (Quarters 1-3) and Progress towards Targets by Intervention Areas................................................................................................................................................................................22 Figure 5: Percentage of Respondents Who Agree that CTB has Strengthened Local Capacity ..............................24 Figure 6: Short-Term Technical Assistance, Years 1 & 3 ...................................................................................................28 Figure 7: Percentage of Respondents Who Agree with the Statement “CTB Management Arrangements and Staffing Structure and Capacity are Optimal for Achieving Project Objectives Efficiently and Effectively” ...........................................................................................................................................................................................29 Figure 8: Percentage of Respondents who Agree with the Statement “Information Generated by the Project has been Used to Support Achievement of Objectives and Outcomes”..............................................................31 Figure 9: Percentage of Respondents who Agree with the Statement “CTB Work Plans and Strategies are Responding Effectively to the Changes in the Government Structures and Policies”................................36 Figure 10: Percentage of Respondents who Agree with the Statement “Lessons Learned and Best Practices from CTB Implementation Could be Replicated in Non-CTB Districts” ................................................................38 Figure 11: Percentage of Respondents who Agree with the Statement “CTB Methodologies, Interventions, and Management are Setting the Stage for Future Sustainability of Project Outputs and Outcomes” .........41 Figure 12: Percentage of Respondents’ Agreement with Six CTB Positive Statements, Papua Province ..............43 LIST OF TABLES Table 1: Number of KNCV and Partner Staff in APA-1, APA-2, and APA-3 ................................................................28 Table 2: Summary of District Dashboard Indicators, Targets, Baselines, and Latest Results (APA-3) for the 16 CTB districts............................................................................................................................................................................32 Table 3: Average Dashboard Score by District.....................................................................................................................44 USAID/Indonesia Challenge TB Project Midterm Evaluation v ACRONYMS AND ABBREVIATIONS AIDS AMEP Acquired Immune Deficiency Syndrome Activity Monitoring and Evaluation Plan AOR/COR Agreement Officer's Representative/Contracting Officer’s Representative APA Annual Plan of Activities ARV Antiretroviral Bappeda Badan Perencanaan Pembangunan Daerah (Local Development Planning Bureau) BCC Behavior Change Communication BPJS Badan Penyelenggara Jaminan Sosial (Social Insurance Administration Organization) BBLK Balai Besar Laboraturium Kesehatan (Health Laboratory Great Hall) CCM Country Coordination Mechanism CDR Case Detection Rate C/DST Culture and Drug Susceptibility Testing CEPAT Community Empowerment of People Against Tuberculosis CN Case Notofication COP Chief of Party CSO Civil Society Organization CTB Challenge Tuberculosis (project) DAP District Action Plan DHO District Health Office DKI Jakarta Daerah Khusus Ibukota Jakarta (Special Capital Region of Jakarta) DM Diabetes Mellitus DOTS Directly Observed Treatment Short-course DR-TB Drug-Resistant TB DSM Diagnostic and Statistical Manual of Mental Disorder ECR Enhanced Cohort Review EQA External Quality Assessment ERR Electronic Recording and Reporting e-TB Manager Electronic Tuberculosis Manager FGD Focus Group Discussion FHI 360 Family Health International 360 GFATM Global Fund to Fight AIDS, TB, and Malaria GLI Global Laboratory Initiative GOI Government of Indonesia HDL Hospital DOTS Linkage HIV Human Immunodeficiency Virus HIV/AIDS Human Immunodeficiency Virus/Acquired Immune Deficiency Syndrome HQ Headquarters ICF Intensified Case Finding IDR Indonesian Rupiah IRD International Relief and Development IUATLD International Union Against Tuberculosis and Lung Disease JEMM Joint External Monitoring Mission JKN Jaminan Kesehatan Nasional (National Health Insurance System) KII Key Informant Interview KNCV Koninklijke Nederlandse Chemische Vereniging (Royal Dutch Chemical Association) USAID/Indonesia Challenge TB Project Midterm Evaluation vi KPMA-UGM Pusat Kebijakan dan Pembiayaan Manajemen Asuransi Kesehatan Universitas Gajah Mada (Centre for Strengthening the Evidence Based Health Financing Policy University of Gajah Mada) LKNU Lembaga Kesehatan Nadhatul Ulama (Health Organization of Nadhatul Ulama) LOE Level of Effort LPA Line Probe Assay LTBI Latent Tuberculosis Infection LQAS Lot Quality Assurance Sampling System LQMS Laboratory Quality Management System M&E Monitoring and Evaluation MDR-TB Multidrug Resistant TB MESP Monitoring and Evaluation Support Project MICA Monthly Interim Cohort Analysis MOF Ministry of Finance MOH Ministry of Health MOHA Ministry of Home Affairs MOLHR Ministry of Law and Human Rights MOU Memorandum of Understanding MSI Management Systems International MSS Minimum Standards of Services NAP National Action Plan ND New Drug NGO Nongovernmental Organization NTP National TB Program NSP National Strategic Plan PCO Provincial Coordinator Officer PHC Primary Health Care PEPFAR U.S. President’s Emergency Plan for AIDS Relief PHC Primary Health Center PHO Provincial Health Office PMDT Programmatic Management of Drug-resistant TB PPM Public-Private Mix PSE Psychosocial Economic Support Puskesmas Pusat Kesehatan Masyarakat (Public Primary Care Clinic) Q1,2,3 Quarter 1, 2, 3 RSUD Rumah Sakit Umum Daerah (District Hospital) SDoH Social Determinants of Health SIKM Strategic Information and Knowledge Management SITT Sistem Informasi TB Terpadu (National Integrated TB Information System) SHP Strategic Health Purchasing SOW Statement of Work SS Sputum Smear STTA Short Term Technical Assistance STR Shortened Treatment Regimens TB Tuberculosis TB-DM Tuberculosis-Diabetes Mellitus TB/HIV Tuberculosis/Human Immunodeficiency Virus TB JEMM Tuberculosis Joint External TB Monitoring Mission TOR Terms of Reference TPM Team Planning Meeting USAID/Indonesia Challenge TB Project Midterm Evaluation vii TSR TB Success Rate TWG Technical Working Group UNAIDS United Nations Programme on HIV and AIDS USAID United States Agency for International Development USD US Dollar USG United States Government WHO World Health Organization USAID/Indonesia Challenge TB Project Midterm Evaluation 1 EXECUTIVE SUMMARY Project Background Challenge TB (CTB) Indonesia is a five-year project funded by the United States Agency for International Development (USAID) that supports the Government of Indonesia (GOI) to implement the National Strategic Plan for TB (NSP) under the National TB Program (NTP). The activity is implemented by the KNCV Tuberculosis Foundation (KNCV) and its coalition of in-country partners, including the World Health Organization (WHO), FHI 360, and Interactive Research and Development (IRD). Evaluation Purpose and Evaluation Questions The purpose of this midterm evaluation is to assess CTB Indonesia’s performance, outcomes, and progress towards intended results, and to recommend actions to improve and enhance project implementation and impacts. The six evaluation questions are: Q1. Has CTB strengthened the local capacity of the TB system to deliver strategic, effective TB programming in the five intervention areas? (Project Performance) Q2. Are CTB management arrangements and staffing structure and capacity optimal for achieving project objectives efficiently and effectively? (Program Management) Q3. Has the information generated by the project been used to support achievement of objectives and outcomes? (Information Generation and Use) Q4. To what extent has CTB been adapting its approach and activities to recent systemic changes in Indonesia’s health system, i.e., the revised Law on Decentralization (Law 23, minimum standards of Services [MSS], etc.) and the National Health Insurance (JKN) scheme, and their impact on the provision of TB services? Q5. Are there lessons and best practices from CTB implementation that could be replicated in non￾CTB districts? Q6. Are CTB methodologies, interventions, and management structures/arrangements setting the stage for future sustainability of project outputs and outcomes? (Sustainability) Evaluation Questions, Design, Methods and Limitations The evaluation methodology involved a review and analysis of quantitative data in combination with the application of qualitative techniques to obtain information from various CTB counterparts, partners and stakeholders. The evaluation team visited 12 of 16 CTB intervention districts across the six provinces where the project works. There are some limitations to the evaluation design. First, because we used purposive sampling methods in selecting districts, individuals for key informant interviews (KIIs), and focus group discussion (FGD) participants, the evaluation was not able to generate findings that statistically represent the larger population from which the respondents were drawn. Second, because the key informants constituted one of the primary sources of data, that data could be subject to personal biases and experience with the program. USAID/Indonesia Challenge TB Project Midterm Evaluation 2 Findings and Conclusions Q1. Project Performance The evaluation team used the expected achievements of CTB Indonesia over the life of the project (five years) to quantitatively assess the midterm performance of the project. The evaluation team reviewed annual and quarterly reports in the Annual Plan of Activities (APAs) for Years 1-3, and also analyzed accessible secondary data. This revealed that data for three out of nine quantifiable indicators have not been reported or are not reliable enough for analysis. Approximately 71% and 80% of the activities planned in Quarters 2 and 3, respectively, in the APA Year 3 (APA-3), completely or partially met their targets. CTB Indonesia has contributed in the five technical intervention areas with different degrees of impact, as summarized below. A. Ensuring universal access by integrating TB in the National Health Insurance System, and securing increased local government funding for TB: CTB has supported the NTP to strengthen political commitment for TB and has provided technical assistance and guidance to finalize the TB-JKN technical guideline. Also, CTB has assisted the Ministry of Home Affairs (MOHA) to develop a decree on District Action Plans (DAPs) and is now supporting the development of joint legislation between MOHA, Ministry of Health (MOH), the National Planning Bureau and the Ministry of Villages to finalize the legal foundation for inter-sectoral funding of TB control at the district level. This acknowledgment did not prevent most of the CTB beneficiaries from expressing their concerns about the readiness of the project to share the knowledge and transfer the skills required to complete this process and to develop a feasible, replicable model that can be applied in other non-CTB districts. B. Increasing case detection: Intensified case finding (ICF) to address the current gap in notification: The number of all forms of TB cases notified in CTB districts increased from 44,253 in 2014 to 68,914 (56% higher) in 2016 compared to non-CTB districts where the number of TB cases notified slightly increased from 280,286 to 291,644 (4% higher) over the same time period. CTB has contributed to strengthening this area through the development and dissemination of the national technical guidelines for ICF among key populations. CTB developed a district-based Public-Private Mix (PPM) and successfully piloted WiFi TB, which is a mobile phone app for notification of TB patients. CTB beneficiaries at different levels believe that the significant ICF efforts by CTB should be strengthened and sustained through advocacy with higher authorities, work on regulations, and creating feasible models that can be replicated and integrated into the NTP operational system. C. Ensuring the quality of treatment and care for TB, drug-resistant TB, and TB/HIV co-infection: CTB has contributed significantly to this intervention through technical training and mentoring of providers; improving the diagnosis, referral and management of drug-resistant TB (DR-TB) by service providers; establishing and empowering cured patient groups to support their peers; introducing and implementing a shorter regimen for treatment of MDR-TB with active TB drug￾safety monitoring and management (abbreviated as aDSM) for effective monitoring of adverse events; and providing technical assistance to NTP staff to certify and institutionalize the enhanced cohort review (ECR) as part of programmatic management of drug-resistant TB (PMDT). There is a common concern among all CTB beneficiaries about the replicability and scaling-up strategies/plans of CTB to integrate and sustain such activities within the current NTP structure and systems. USAID/Indonesia Challenge TB Project Midterm Evaluation 3 D. Expanding the network of diagnostic services: CTB is complementing procurements funded by the Global Fund to Fight AIDS, Tuberculosis and Malaria (GFATM) through provision of technical assistance for GeneXpert placement, installation and use. Installation of GeneXpert in health facilities creates hubs for the diagnostic service network. CTB technical assistance for training on the Lot Quality Assurance Sampling System (LQAS) and TB sample transfer is acknowledged and appreciated. The number of laboratories performing culture and drug susceptibility testing (C/DST) increased from 0 in 2014 to 14 in 2017, and number of C/DST labs implementing the laboratory quality management system (LQMS) increased from 8 in 2014 to 14 in 2017. E. Strengthening monitoring and evaluation (M&E), surveillance and operations research: CTB piloted several initiatives (tools and systems) to strengthen M&E in the national system (such as WiFi TB), continued improvements in the Electronic TB (e-TB) Manager for MDR, and provided technical assistance for the electronic data entry system for DS-TB of the National Integrated TB Information System (Sistem Informasi TB Terpadu [SITT]). Despite the initiatives and innovations that CTB is developing to support the TB system, more follow up on the use of these tools and systems is required to ensure their effective use by the assigned personnel and to provide any troubleshooting required. Programmatic staff who use the data formats produced by CTB for planning and monitoring purposes say they are difficult to understand and use. Progress on the six quantifiable indicators (numerical indicators) in the draft annual report of APA-3 vary from 39% (TB patients who are tested for HIV) to 92% (Xpert testing for presumptive DR-TB). Q2. Program Management The current CTB Indonesia management structure (established in Sept 2016) is a hierarchical organizational structure with the project’s staff grouped based on technical specialties, provision of services to key populations, functions, or location. Although the current project management structure does not reflect the five key intervention areas, nor the three main objectives of the project, the structure, components, hierarchy, and reporting/coordination lines are supporting different technical areas the project is working on and different key populations the project is targeting. To strengthen the technical component of the project in guiding and following up on the different technical programmatic interventions, the project’s Chief of Party (COP) has separated technical and operation teams to relieve technical staff from administrative assignments and to boost administrative/operational support to the field. Conclusion CTB Indonesia is making substantial efforts to strengthen the technical skills of local personnel working in the fields of TB management and care, mainly at the district level in each of the five intervention areas. In its APA-4, CTB plans to roll out the district-based PPM, and scale up the GeneXpert testing and PMDT. However, CTB has limited initiatives that go beyond enhancing skills into building institutional systems and platforms to ensure replicability and sustainability of these efforts by the NTP and Provincial Health Offices (PHOs) in the current system. Documentation and structured dissemination of different CTB processes for planning, managing and evaluating different innovations would assist in knowledge and skills transfer to non-CTB districts. USAID/Indonesia Challenge TB Project Midterm Evaluation 4 The management structure tends to have more operational staff than technical staff, which may be an indication of the “doer” nature of the project versus the “technical assistance provider” role, though this is different from the original mandate of the project. Over the remaining two years, CTB should explore different options to maximize the utilization of its staff in working with their respective counterparts within the national system, on a mentoring basis, to strengthen the programmatic strategic planning, management and advocacy aspects. NTP staff at the national and provincial levels indicated that they have no or limited say in the number and/or the nature of short-term technical assistance (STTA) provided by CTB, who (according to NTP staff) mainly focus on technical issues and do not address NTP management, leadership and planning needs. Q3. Information Generation and Use There are seven abstracts/posters, one technical document and one scientific publication listed as project products in the April – June 2017 quarterly report. Most of these information products are in the areas of TB treatment management and case reporting. Interviews with NTP staff at the district level revealed that there is appreciation for the information generated and utilized by the project to improve the quality of TB programming and services. However, the levels of agreement with this positive statement among provincial (73%) and national (67%) NTP staff are significantly lower than the levels of agreement found among district-level NTP staff. Respondents from health facilities also said they appreciate the project for its role in empowering district/provincial laboratories and revitalizing the mechanism for quality control of sputum smear reading (through implementation of the e-TB 12 application). Many of the interviewed CTB and NTP staff at provincial and district levels emphasized the complexity of the dashboard indicators, saying the list is long, confusing and some of the indicators need better definitions. The evaluation team found that district dashboard indicators are measured but not used effectively for planning and strategy development purposes. Conclusion The CTB Indonesia COP made significant changes in the management structure by separating the technical and operational teams to strengthen the technical component of the program and increase the effectiveness of the operational component. The COP also streamlined communication between CTB regional offices and the home office to improve their monitoring and evaluation functions, and respond quickly and more effectively to any programmatic and/or administrative issues in the field. However, certain factors hinder the optimal efficiency and effectiveness of the current management system and staffing structure of the project in fulfilling its role as technical assistance provider to the NTP at the national and local levels so that the NTP can strengthen its capacities to sustain and scale up successful CTB interventions. These factors include the centralized management approach with limited delegation; the demanding reporting system between sub￾national and national levels; the convoluted communication and decision-making process between CTB management and CTB partner representatives; and lack of clarity regarding the systems for knowledge sharing and skills transfer between international STTA consultants and local partners and experts (to strengthen the technical assistance mandate of the project). USAID/Indonesia Challenge TB Project Midterm Evaluation 5 Q4. Adaptation to Indonesia’s Health System There are four CTB key activities under Objective 1 on “Improved access” and Objective 3 on “Strengthened TB platform” in APA-3 that directly address decentralization and the JKN. Decentralization. From the document review and KIIs, the evaluation team found that the District Action Plan (DAP) is one of the main approaches of the CTB project in responding to health system decentralization, in addition to strengthening the capacity of districts and health facilities staff in various technical and programmatic areas. National Health Insurance (JKN). Ensuring universal access to health insurance by integrating TB into the JKN is part of CTB’s five technical intervention areas, and the project finalized the National Health Insurance Guideline for TB services in APA-1. However, there are multiple issues with the practicality of this guideline and it has not been implemented as such. CTB activities in this intervention area are limited compared to the key activities that are related to improving technical capacity in TB diagnosis and treatment, and community participation. Although that there are some opportunities that CTB should take advantage of to improve the TB program by using JKN funding, CTB has been instructed by USAID not to engage in activities with JKN, given the relatively unsuccessful experience of the project with the technical guideline development and the project’s lack of experience in advising the NTP in this area. However, other USAID-funded projects such as the Strategic Health Purchasing (SHP) project will continue providing support by organizing consultations as needed. Q5. Lessons and Best Practices In APA-2, CTB Indonesia added two specific indicators to monitor budget allocation, progress and achievements in generating, documenting, disseminating and facilitating adoption and replication of lessons and best practices from implementation. Conclusion CTB Indonesia has generated important information related to progress and achievements from its activities. However, dissemination and utilization of this information is still limited and has not motivated key stakeholders to take necessary actions to improve their performance. While the mandate of the project requires improvement in the performance of district-level key stakeholders in order to achieve the project objectives and outcomes, most of the information generated by the project is not yet produced in appropriate forms for the intended audience/s and purposes (e.g., no policy briefs have been produced for policy makers). Conclusion CTB has been adapting its approaches and activities to Indonesia’s new health system, mainly by facilitating the DAP process. To ensure more alignment with and support for the current changes, CTB should work closely with core stakeholders, in a participatory and collaborative manner, to develop a replicable model and technical guidelines for DAP development that can be applied in non-CTB districts. USAID/Indonesia Challenge TB Project Midterm Evaluation 6 Eight1 best practices and success stories have been identified by CTB in its annual and quarterly reports. According to the interviews with project staff at national, provincial and district levels, these best practices are related to improved care for patients with drug-resistant TB (DR-TB) (ex-patient empowerment to support the treatment program and diagnosis), collaboration on TB-DM, TB screening in prison settings, the mandatory notification app, and costed DAPs. These are successful interventions and worth considering for replication. In addition, most key respondents in district and health services facilities said they believe that the CTB approaches to revitalizing the district-level quality control system of sputum smear, and building a diagnostic networking and referral system, are good and needed initiatives that should be replicated in other districts. Q6. Sustainability The CTB Indonesia project is designed to provide sustainable solutions and replicable approaches by supporting the NTP and other partners in designing and implementing a set of interventions at the district level. The project does this by providing a considerable number of staff and by developing ‘’intensified’’ packages of technical assistance. However, many of the CTB project’s strategies and approaches are bound by the requirement to provide certain technical deliverables, with no clear sustainability/institutionalization initiatives beyond fulfilling the requirement for these deliverables. More than 80% of respondents from the CTB project team agreed that the project’s strategy and approaches will sustain CTB outputs and outcomes. However, half of NTP staff at the national level and one-fifth of staff at the provincial level did not agree with the statements that CTB implementation outputs and outcomes will be sustained without the project’s support. Several factors may contribute to 1 These are: TB District Action Plans as a way to eliminate TB through local commitment and ownership; Mandatory notification apps that increase the case notification by simplifying reporting; the new TB drug Bedaquiline made available in Indonesia; psychosocial support from ex-patients that motivates other patients to return to treatment; faster test results that enable faster treatment initiation; DR-TB patient empowerment; PMDT microtraining for district health officers to build self￾reliance for DR-TB treatment; and prison cadres training on cough surveillance. Conclusion Several of CTB’s best practices and lessons learned have been adopted and replicated by other organizations and the NTP. For example, Aisyiyah (PR GF-TB) adopted the DAP development approaches and processes and replicated them in more than 50 non-CTB districts where it is working; the organization LKNU (Lembaga Kesehatan Nadhatul Ulama; Health Organization of Nadhatul Ulama) adopted the CTB approach in supporting TB patients’ education sessions in hospitals and replicated it on the primary health center (PHC) level with some modification; and DR-TB benchmarking for treatment assessment has become a national tool for hospital accreditation and has been scaled up to the national level by the NTP. There are several other best practices and lessons that could potentially be replicated, such as: approaches to revitalize the district-level quality control system of sputum smear, and the referral network and quality control systems for microscopic examination, recording and reporting using e￾TB 12 application for all districts/cities with high TB epidemics in West Java and neighboring CTB districts in East Java. However, more efforts are still required by the project to document and institutionalize these processes and approaches to make them feasible and easily replicable in other non-CTB districts. USAID/Indonesia Challenge TB Project Midterm Evaluation 7 shaping this opinion on the sustainability of different CTB activities: a. Dependency on the financial and technical resources provided by the project which cannot be secured by local authorities; b. The large number of CTB staff at the district and provincial levels who take over the functions of the NTP/MOH staff and do the work by themselves rather than building capacity and transferring knowledge and skills to local staff to ensure sustainability; and c. Lack of the documentation of processes and of feasible models that can be replicated in non￾CTB districts; and d. A greater focus on addressing mainly technical challenges compared to building TB management and institutional capacity (as indicated by a number of NTP staff at all levels and in different locations). CTB initiates community peer support programs through empowering cured TB patients. This kind of community approach produces good outcomes that support patients in some areas. Papua Province The evaluation team has applied the same evaluation questions on Papua. In Year 3 of the project; CTB added three districts in Papua as priority sites. The project is playing an important role in strengthening the capacity of local key stakeholders and health service providers in managing and monitoring the TB program’s activities. The evaluation team found that the CTB project in Papua has some technical and administrative challenges, which are attributed to the remoteness of Papua and management changes. These challenges include: a. Difficulty in identifying and recruiting high-quality technical staff to deliver technical assistance in Papua. b. The change in project management in Papua from KNCV to FHI 360 in APA-3. This impacted some of the project’s deliverables because FHI 360 staff are more experienced in managing technical assistance for TB-HIV and TB in prison settings, but do not have the same level of experience in other components of TB management and care. Also, several technical positions such as diagnostic services specialists and data management improvement consultants were either newly introduced or are being filled by junior technical staff who are less experienced and thus not accepted and appreciated by the CTB partners’ more senior technical staff. CTB should give special consideration to its plans and approaches for the project districts in Papua given the province’s remote location. This can be done by: supporting FHI 360 in its management of the project and in its leading implementation role in Papua; creating the financial and administrative arrangements necessary to attract competent technical staff in these remote areas; addressing basic Conclusion CTB does not have a clear sustainability strategy and/or plan to go beyond enhancing technical capabilities of personnel to building institutionalized, sustainable systems. However, CTB’s tested methodologies, approaches and technical capabilities could provide a solid foundation to develop sustainable and institutionalized approaches and systems within the NTP at the national and provincial levels. USAID/Indonesia Challenge TB Project Midterm Evaluation 8 health services/system challenges related to the geography of Papua such as task shifting due to the limited number of health workers at remote Puskesmas; and advocating for more resource mobilization from GOI to support the health system infrastructure. Long-term Recommendations for USAID Below are the evaluation team's recommendations for USAID's future programming in TB management and care in Indonesia: • Consider engaging in high-level policy dialogue with the government (along with other leading donors) to advocate for a national focus on TB prevention, management and care with emphasis on key populations and emerging MDR-TB. • Support the design and implementation of a strategic and comprehensive cross-cutting BCC strategy to address TB-related information misconceptions, stigma, health-seeking practices, and support marketing of quality services. • Support high-level inter-departmental and inter-sectoral platforms to address the relevant social determinants of health (SDoH) in the planning and implementation processes of the national TB program and its activities at the national and regional levels. • Given the country's large population and regional variations, USAID should continue targeting its resources strategically to specific geographic areas, activities, key populations and high-level institutional/regulatory changes to achieve maximum lasting impacts. • Continue focusing on and maximizing private sector and NGO active involvement in TB early detection and management, as their work is attracting growing numbers of persons with TB, particularly in urban areas. • Continue supporting innovative and advanced technological systems and tools to streamline TB early diagnosis, notification and management, and support institutionalization of these systems and tools. USAID/Indonesia Challenge TB Project Midterm Evaluation 9 PROJECT BACKGROUND Challenge TB (CTB) is USAID’s flagship global mechanism for implementing the tuberculosis (TB) strategy of the United States Government (USG), as well as contributing to TB/HIV activities under the U.S. President’s Emergency Plan for AIDS Relief (PEPFAR). Launched on October 1, 2014, CTB works through five key technical intervention areas and is aligned with the USG strategy to prevent and control TB. CTB has three objectives and 11 sub-objectives, each with several focus areas for interventions: Objective 1: Improved access to high-quality, patient-centered TB, drug-resistant TB (DR-TB), and TB/HIV services by: 1. Improving the enabling environment; 2. Ensuring a comprehensive, high quality diagnostic network; and 3. Strengthening patient-centered care and treatment. Objective 2: Prevent transmission and disease progression by: 4. Targeted screening for active TB; 5. Implementing infection control measures; and 6. Managing latent TB infection. Objective 3: Strengthen TB service delivery platforms by: 7. Enhancing political commitment and leadership; 8. Building comprehensive partnerships and informed community engagement; 9. Strengthening drug and commodity management systems; 10. Ensuring quality data, surveillance and monitoring and evaluation; and 11. Supporting human resource development. CTB Indonesia is a five-year project funded by USAID that supports the Government of Indonesia’s (GOI’s) National TB Program (NTP) through implementation of the National Strategic Plan (NSP) for TB. The activity is implemented by KNCV Tuberculosis Foundation (KNCV) and its coalition of in￾country partners, the World Health Organization (WHO), Interactive Research and Development (IRD), and FHI 360. CTB aims to develop comprehensive, effective, and cost-efficient interventions that can be rolled out and scaled up across Indonesia using primarily domestic resources plus, in the shorter term, other development assistance resources (notably from the Global Fund to Fight AIDS, TB, and Malaria [GFATM]). The CTB project in Indonesia is part of the global CTB project, and is one of the few CTB country projects led by KNCV that works on all of the project’s three main objectives and 11 sub￾objectives, which significantly increases the magnitude, scope and complexity of the project. Funding for CTB Indonesia is approximately US$40 million over the life of the project. In October 2017, CTB Indonesia started the fourth year of implementation, and the project aims to continue its activities to work on: mobilization of local resources, including financing; increasing case detection and treatment success rates; and improving quality of care and reporting. CTB Indonesia will also work on providing guidance for relevant policies and regulations to support implementation of the NTP. At the national level, CTB Indonesia provides technical assistance to the Directorate General of Disease Prevention and Control under the Ministry of Health (MOH) as the main partner. Other governmental and non-governmental institutions also receive technical assistance including provincial and district health offices, professional associations, patients’ groups, community organizations, and local partners in supported provinces. In Indonesia, the predecessor projects to Challenge TB Indonesia, TB CAP (2005-2010) and TB CARE I (2010-2015), focused primarily on central-level technical guidance, with additional significant activities implemented at provincial level. However, soon after the start of Challenge TB Indonesia, USAID and USAID/Indonesia Challenge TB Project Midterm Evaluation 10 Challenge TB jointly concluded that a greater focus was needed at the district level to reflect the importance of this organizational unit in the decentralized Indonesian health system and to establish model areas of success that could be replicated. Therefore, while Challenge TB Indonesia continued critical work at national and provincial levels, 10 initial districts in five provinces were selected for greater focus based on the following criteria: (i) large provinces prioritized by the NTP with a high burden of TB and HIV; (ii) USAID/Indonesia priority provinces; (iii) ability to complement previous investments; and (iv) presence of other USAID partners (to increase synergy). The selected provinces and districts were: • DKI Jakarta2 – North Jakarta and East Jakarta • West Java – Bandung City and Bogor District • Central Java – Semarang City and Surakarta (Solo) City • East Java – Jember District and Tulung Agung District • North Sumatra – Medan City and Deli Serdang District These 10 districts received an intensified package of assistance, including all components of TB management and care. The purpose of these intensified packages was to design and test small-scale models at the district level to better define a sustainable and scalable model TB program. In addition, CTB supported local health service providers and partners in four other provinces—West Sumatra, South Sulawesi, West Papua, and Papua—to provide a more specified package of technical assistance based on local needs for expansion and quality assurance of essential components of TB control, including laboratories, TB/HIV services, programmatic management of drug-resistant TB (PMDT) expansion, and intensified TB case finding. CTB Indonesia also provides support to the principal recipients of funding from GFATM; KNCV assists recipients of GFATM funding to address technical and managerial issues, and to support planning, implementation, and troubleshooting. CTB support is complementary to GFATM assistance. In Year 3 of CTB, the project and USAID/Indonesia agreed to add three districts in Papua Province (Jayapura City, Jayawijaya District and Mimika District) and three districts in DKI Jakarta (Jakarta Selatan, Jakarta Pusat, and Jakarta Barat) as priority sites. At the same time, three of the provinces receiving specified packages were graduated from the project, and thus in Year 3, CTB worked in six provinces and 16 districts, as follows: • DKI Jakarta – Central, West, North, East, and South Jakarta • West Java – Bandung City and Bogor District • Central Java – Semarang City and Surakarta (Solo) City • East Java – Jember District and Tulung Agung District • North Sumatra – Medan City and Deli Serdang District • Papua – Jayapura City, Mimika District, and Jayawijaya District The estimated total population in the 6 CTB supported provinces is 146 million people (57% of Indonesia total population in 2015). The 16 CTB districts have a total population of 28.7 million people. 2 Daerah Khusus Ibukota Jakarta (Special Capital Region of Jakarta). USAID/Indonesia Challenge TB Project Midterm Evaluation 11 EVALUATION PURPOSE & EVALUATION QUESTIONS EVALUATION PURPOSE The purpose of this midterm evaluation is to assess CTB Indonesia’s performance, outcomes, and progress towards intended results, and to recommend actions to improve and enhance project implementation and impacts. To this end, the evaluation team reviewed project reports, strategies and plans, and conducted key informant interviews (KIIs) and focus group discussions (FGDs) with a variety of stakeholders. In addition, the evaluation team analyzed CTB’s role within the GOI’s overall response to TB. The primary audiences for this midterm evaluation are the USAID/Indonesia Health Office, USAID/Washington, and the CTB implementing partners. These stakeholders will use the evaluation findings, conclusions, and recommendations to strengthen project implementation. This midterm performance evaluation is being undertaken to analyze the CTB project performance to date and to develop recommendations for improvements needed in order for the project to meet its intended purpose. In addition, the evaluation analyzes the value added by the project to knowledge and skill enhancement and organizational capacity building of local partner institutions. Looking beyond the primary audiences, findings on project performance, program management, information generation and use, and sustainability will enable all project stakeholders, including the GOI, to understand CTB’s strengths as well as areas where technical, administrative and management efforts could be improved and strategic approaches refocused. EVALUATION QUESTIONS To guide this evaluation, USAID identified the key questions listed below, as well as lines of inquiry for each question (see Annex III). The evaluation team ensured that questions were phrased (either in English or Bahasa) in neutral, non-leading ways. In addition, the interviewers probed to understand the reasons and rationales for the answers received in response to each question. Lines of inquiry were used as illustrative and probing items/questions to stimulate discussion and obtain more information to assist interviewees in answering each question. The evaluation team identified the appropriate questions and lines of inquiry based on the role of the interviewee in relation to the project, and the purpose of the interview. The six evaluation questions are: Q1. Has CTB strengthened the local capacity of the TB system to deliver strategic, effective TB programming in the five intervention areas? (Project Performance) Q2. Are CTB management arrangements and staffing structure and capacity optimal for achieving project objectives efficiently and effectively? (Program Management) Q3. Has the information generated by the project been used to support achievement of objectives and outcomes? (Information Generation and Use) Q4.To what extent has CTB been adapting its approach and activities to recent systemic changes in Indonesia’s health system, i.e., the revised Law on Decentralization (Law 23, minimum standards of Services [MSS], etc.) and the National Health Insurance (JKN) scheme, and their impact on the provision of TB services? Q5. Are there lessons and best practices from CTB implementation that could be replicated in non￾CTB districts? Q6. Are CTB methodologies, interventions, and management structures/arrangements setting the stage for future sustainability of project outputs and outcomes? (Sustainability) USAID/Indonesia Challenge TB Project Midterm Evaluation 12 The evaluation team applied the same questions during its field visits to two districts in Papua Province; the special section on the findings in Papua begins on page 46 of this report. Given the short time remaining for the CTB Indonesia project and the wide range of activities currently being implementing under the project, the evaluation team tried to propose recommendations that fulfill the following criteria: strategic and well defined; doable; can result in impacts during the remaining life of the project; and set a solid foundation for the project’s successor. In addition, the team proposed longer-term recommendations to USAID to be considered in the process of designing future TB projects. EVALUATION METHODS & LIMITATIONS The midterm evaluation Team Planning Meeting (TPM) process took place in Jakarta during the week of September 25-28, 2017, to develop the evaluation design document. Evaluation team members, representatives of USAID’s program and technical offices, the Chief of Party (COP) and Senior Monitoring and Evaluation (M&E) Advisor of the Monitoring and Evaluation Support Project (MESP), and the CTB COP and Technical Coordinator participated in the TPM. The TPM used the Evaluation Analytical Framework – Getting to Answers Matrix (see Annex IIIA) developed by Management Systems International (MSI) as a foundational tool to facilitate and structure the design and development processes of the evaluation methodology and fieldwork plan. Participation of USAID representatives at the TPM on the first day, and their presentation and involvement in discussions with the evaluation team, were valuable in focusing the evaluation task and design, including the selection of field sites to visit and key informants to interview. The evaluation methodology involved a review and analysis of quantitative data combined with the application of qualitative techniques to obtain information from various CTB counterparts, partners and other stakeholders. By using a mixed evaluation method approach to analyze variables corresponding with project objectives and outcomes, the evaluation team gleaned objective insights related to the six evaluation questions. SELECTION OF DISTRICTS The evaluation team visited 12 districts out of 16 CTB intervention districts across the six provinces where the project works. The four selection criteria for these districts were: 1) have received key interventions; 2) are representative of either high-performing or low-performing districts; 3) reflect USAID priority issues (e.g., TB private sector study locations, management issues); and 4) logistical considerations. These criteria helped identify factors that contributed most significantly to improved outcomes as well as inhibiting factors to positive performance. Per these criteria, the evaluation team selected the districts listed below in consultation with USAID/Indonesia. Annex II presents the detailed criteria scores for all CTB districts reviewed. • Deli Serdang District • Medan City • Central Jakarta City • East Jakarta City • North Jakarta City • Bogor District • Bandung City • Surakarta City • Semarang City • Jember District • Jayapura City • Mimika District USAID/Indonesia Challenge TB Project Midterm Evaluation 13 DATA COLLECTION TOOLS AND INSTRUMENTS The evaluation team used purposive sampling and both quantitative and qualitative methodologies to address the six evaluation questions, including (but not limited to) document and literature review, analysis of secondary data, KIIs, FGDs, and a stakeholder questionnaire. The exact number of interviews and site visits locations was finalized in consultation with USAID prior to the beginning of fieldwork. Annex IVA presents the schedule for the fieldwork conducted. Following fieldwork, the team synthesized and analyzed the collected data, guided by the analytical framework for the evaluation, the Getting to Answers Matrix. The major elements of the evaluation methodology are listed below. 1. Document and Literature Review The evaluation team conducted a detailed desk review of available project documents listed in the Statement of Work (SOW), as well as data from sources provided by USAID. The team also reviewed documents recommended by the Mission including annual reports, work plans, strategies, studies, M&E plans and data, relevant GOI national strategies, plans related to TB, and other relevant USAID program documents. The documents list includes, but is not limited to the following: • TB National Strategic Plan for Indonesia • CTB project design document • CTB M&E Plan (with annual indicators: achieved vs projected) • CTB Cooperative Agreement, and program description from the Project Agreement • CTB STEP Table and Activity Monitoring and Evaluation Plan (AMEP) • CTB annual progress reports • CTB work plans • CTB external management review • Other CTB technical materials, e.g., case studies, factsheets, infographics, profiles, program briefs, technical briefs • TB CARE Indonesia Final Report • TB Joint External Monitoring Mission (JEMM) report, 2017 The literature review also included: a study of the epidemiological context of TB in Indonesia, key approaches and activities, institutions involved, as well as other policies that may affect the implementation of CTB. 2. Collection and Analysis of Secondary Data The objective of secondary data analysis was to determine the progress achieved and any trends across the various programmatic areas of the project. The evaluation team made extensive use of all data available that was pertinent to the evaluation questions. These included: • Annual district monitoring data, including data on the estimated number of people with TB, TB cases notified, number of TB patients with HIV status, and multidrug resistant TB (MDR-TB) cases. • Epidemiological data to compare effectiveness of TB programs in CTB districts and non-CTB districts. This data were triangulated with other data to demonstrate potential contributions of CTB programs at the district level. USAID/Indonesia Challenge TB Project Midterm Evaluation 14 3. Key Informant Interviews KIIs provided insights on the effectiveness of CTB technical approaches, contributions, and gaps. The team conducted semi-structured interviews (see Annex IIIB for KII guides) with informants at central, provincial, and district levels. Each of the six evaluation questions includes a list of illustrative ideas/backgrounds (lines of inquiry) to be considered during the interview as appropriate. Lines of inquiries are not sub-questions to be answered, but are rather used to inform and frame questions in the appropriate way according to the interviewee’s background and responsibilities in relation to CTB. In addition, the evaluation team tried to quantify the qualitative perceptions of partners and stakeholders regarding CTB’s work and performance through the use of a short, self-administered questionnaire with Likert scales (see Annex IIIC). This questionnaire provided useful comparative quantitative data that supplemented other data/information generated from other sources. The following are the key stakeholder and important informants interviewed for this evaluation: • USAID/Indonesia (AOR/CORs and Health Office M&E Specialist) • USAID/Washington • KNCV (the Implementing Partner) • WHO • FHI 360 • NTP • The Global Fund – Country Coordination Mechanism (CCM), TB Principal Recipients (PRs), Technical Working Group (TWG) • World Bank • United Nations Programme on HIV and AIDS (UNAIDS) • GOI Ministry of Health – NTP, etc. • Project health officers in districts • Beneficiaries in districts (public and private hospitals, nongovernmental organizations [NGOs] and patient groups) • Local governments (provincial and district health offices) • Service providers For the final list of the key informants interviewed, see Annex IVB. 4. Focus Group Discussions The evaluation team originally prepared to conduct FGDs with representatives of community organizations (Aisiyah and Lembaga Kesehatan Nadhatul Ulama [LKNU]) and partners working with CTB Indonesia (FHI 360 CTB staff). The FGDs were replaced by in-depth interviews with these representatives due to logistical reasons. The purpose of these interviews was to gain an in-depth understanding of interviewee experiences with project interventions, and to provide them with the opportunity to share their ideas for how to enhance those interventions in the future. The results of the in-depth interviews complemented the information gained from KIIs. A guide for facilitating the evaluation FGDs is provided in Annex IIID. LIMITATIONS OF EVALUATION METHODS The evaluation team acknowledges some limitations to the evaluation design outlined here. First, due to the use of purposive sampling methods in selecting districts, individuals for KIIs, and FGD participants, USAID/Indonesia Challenge TB Project Midterm Evaluation 15 the evaluation was not able to generate findings that statistically represent the larger population from which they are drawn. Second, because the key informants constituted one of the primary sources of data, that data could be subject to personal biases and experience with the program. The possible causes that might led to making the quantitative results more visible and positive, in some areas: the self-administered questionnaire was applied on larger number of interviewees, and the reluctance of some respondents to document negative comments in writing and preferred to mention them verbally during the interviews. GENDER CONSIDERATIONS The evaluation approach incorporated a gender analysis across all levels and also analyzed variables that might place certain individuals or populations at a disadvantage. Specific attention was given to gender considerations while collecting and analyzing data. Specifically, the team ensured that gender was incorporated into: 1) evaluation design—both women and men were interviewed as KIIs and in FGDs; 2) data collection tools, which included questions related to equal access for both genders; and 3) data tabulation, analysis of results, conclusions and recommendations. ETHICAL CONSIDERATIONS AND CONFIDENTIALITY The team obtained verbal and written consent (if required) from all participants of KIIs and FGDs according to USAID Evaluation Policy guidelines. Interviewees were given the option to opt out of particular questions or the whole interview, and the information provided as part of interviews or discussions was not linked to any specific person in the midterm evaluation reports. DATA ANALYSIS Since this is a mixed-method evaluation integrating both quantitative and qualitative data, the team triangulated results from the different data sets. The results from different methods were compared, contrasted, and validated. During the data analysis stage, findings from different methods that were found to be similar or that reinforced one another indicated greater confidence in these findings. If the findings from different methods varied significantly, the team considered and addressed potential causes of the divergent findings. For the process of triangulation, the team used the approach known as parallel combination. In this approach, each data collection method is carried out in its entirety and all are analyzed separately over the same period of time. Then the results are triangulated or synthesized as appropriate. Several types of data analysis were conducted based on the evaluation design, including: Document review. Comparisons were made between the original program design, implementing partner agreements, and implementation work plans to identify potential program strengths, weaknesses, and future modifications. Qualitative data analysis. A framework for data analysis was designed before data collection began in order to cover all pertinent points. This framework specified the main topics to be analyzed, organized by evaluation question. Quantitative data analysis. The evaluation team analyzed and compared relevant available data from epidemiological reports, the self-administrated questionnaires and coded answers of the KIIs and FGDs that is pertinent to the evaluation questions, to determine progress and trends across CTB’s various programmatic fronts. USAID/Indonesia Challenge TB Project Midterm Evaluation 16 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS Q1. PROJECT PERFORMANCE Has CTB strengthened the local capacity of the TB system to deliver strategic, effective TB programming in the five intervention areas? (Project Performance) Based on the systematic gap analysis process undertaken during the development of Indonesia’s NSP and a review of the GFATM investments, it was agreed by USAID and GOI to prioritize the following five key strategic interventions to address the major gaps of the NTP: 1. Ensuring quality of treatment and care for TB, DR-TB, and TB/HIV co-infection. 2. Increasing case detection—intensified case finding to address the gap in notification. 3. Ensuring universal access by integrating TB into the National Health Insurance System, and securing increased local government funding for TB. 4. Expanding the network of diagnostic services. 5. Strengthening M&E, surveillance, and operations research. There is a lack of knowledge among the CTB Indonesia management staff about these five key strategic intervention areas, and as a result the project’s annual action plans (APAs) are driven by CTB objectives and sub-objectives with no linkages nor reference to these key areas. However, there are several commonalities between the CTB key strategic intervention areas and the project’s objectives and sub￾objectives. The evaluation team used the expected achievements of CTB over the life of the project (five years) to quantitatively assess the mid-term performance of the project. The expected achievements are: • Increase the number of TB case notifications in 16 CTB districts by public and private sector providers from 58,847 (2015) to 114,750, with the proportion of case notifications contributed by private providers increasing from 18% (2014) to 30% (2019). • Achieve 100% of Xpert testing among presumptive DR-TB patients in six CTB provinces (this target was not present in the Year 3 work plan but added in the Year 4 workplan). • Increase the proportion of MDR-TB patients enrolled in care from 77% (2015) to >95% in 6 CTB provinces, with more than 90% still on treatment after 6 months. • Increase the proportion of TB patients who are tested for HIV from 18% (2015) to 75% in 16 CTB districts, with 100% of HIV-positive TB patients started on antiretrovirals (ARVs). (The target for HIV testing was 90% in the APA-3 work plan, but was brought down to 60% and finally to 75% in the final version of the APA-4 work plan.) • Achieve all 11 Global Laboratory Initiative (GLI) standards for TB microscopy across the laboratory network (countrywide) with good quality performance in 80% of facilities. • Adoption of a patient/case-based, real-time Electronic Recording and Reporting (ERR) system that is functional at national and subnational levels for both TB and MDR-TB which meets WHO standards for quality TB surveillance data. There are nine quantifiable indicators that can be assessed based on CTB’s five years expected achievements statement. The evaluation team reviewed annual and quarterly reports in the APAs for USAID/Indonesia Challenge TB Project Midterm Evaluation 17 years 1–3, and analyzed accessible secondary data. This revealed that data for three out of nine3 quantifiable indicators are not reported or reliable enough for analysis. Figure 1: CTB Progress toward Quantifiable Targets against Expected Results, 2015-2017 Achievements' progress of the six quantifiable indicators (with numerical outcomes) in the draft Progress for the six quantifiable indicators (those with numerical outcomes) in the draft annual report of APA-3 vary from 39% (TB patients who are tested for HIV) to 92% (Xpert testing for presumptive DR￾TB). The evaluation team used linear regression of previous results to predict the progress towards the six targets at the end of the five-year of the CTB project.: in only one area (Xpert testing for presumptive DR-TB) is the project expected to achieve the intended result. The project is only expected to reach 40–80% of the intended results by the end of the five-year project based on progress to date as measured by the other five indicators. To assess if CTB has strengthened the local capacity of the TB system to deliver strategic and effective TB programming in the five intervention areas, the evaluation team reviewed work plans and reports from the latest available and approved APA (APA-3 Quarters 1, 2, and 3) to identify and cluster the relevant activities by key intervention area, assess the progress of their implementation, and examine how they were utilized and integrated into the TB system by CTB beneficiaries at different levels. 3 MDR-TB patients’ enrollment to care, MDR-TB patients still on treatment after 6 months, and HIV-positive TB patients started on ARVs. USAID/Indonesia Challenge TB Project Midterm Evaluation 18 Figure 2: Number of CTB Key Activities in APA-3 (Quarters 1-3) and Progress against Targets Source: Summarized by Evaluation Team based on CTB quarterly report Figure 2 shows that around 71% and 80% of the activities planned in APA-3 Quarters 2 and 3, respectively, completely or partially met their targets (as indicated in quarterly reports). Several factors may explain this slowdown in achieving the targets of the planned activities during APA￾1, APA-2, and the first quarter of APA-3. First, a strategic decision was made to move to a district-based approach to plan and fund TB activities towards the end of APA-1, and immediately afterwards the COP left and the position stayed vacant for a year with minimal progress on the ground. In addition, other factors contributed to the slowdown: the delay in finalization and approval of the USAID-MOH Memorandum of Understanding (MOU) (approved in December 2016), changes in the NTP leadership and shifts in its priorities, and the cumbersome approval process of the CTB annual work plans which takes a relatively long time and requires that staff implement the annual work plan activities in around eight months. To overcome this delay, the project developed a focused and ambitious work plan in Year 3 to catch up on the delayed activities. However, the plan was beyond the capacity of the project staff and project partners, as reflected in the comments by CTB staff in regional offices who said the staff “work as fire fighters” to catch up on activities’ deadlines. Despite these hindering factors, CTB has contributed in the five technical intervention areas with different degrees of impact, as summarized below. A. Ensuring universal access by integrating TB into the National Health Insurance System, and securing increased local government funding for TB CTB supported the NTP to strengthen political commitment for TB and provided technical assistance and guidance to KPMA-UGM (Pusat Kebijakan dan Pembiayaan Manajemen Asuransi Kesehatan Universitas Gajah Mada [Centre for Strengthening the Evidence-Based Health Financing Policy University of Gajah Mada]) to finalize the TB-JKN technical guideline. Also, CTB has assisted the Ministry of Home Affairs (MOHA) to develop a decree on District Action Plans (DAPs) and is now supporting the development of joint legislation between MOHA, MOH, the National Planning Bureau and the Ministry of Villages to finalize the legal foundation for intersectoral funding of TB control at the district level. 8 20 27 19 29 30 25 20 14 0% 20% 40% 60% 80% 100% Q1 Q2 Q3 Met Partially Not meeting the target USAID/Indonesia Challenge TB Project Midterm Evaluation 19 Findings from the field: The CTB efforts and skills in facilitating the process of DAP development are acknowledged by different CTB beneficiaries, particularly the DHOs and service providers. This acknowledgment did not prevent most of the CTB beneficiaries from expressing their concerns about the readiness of the project to share the knowledge and transfer the skills required to complete this process and to develop a feasible, replicable model that can be applied in other non-CTB districts. B. Increasing case detection: Intensified case finding (ICF) to address the current gap in notification CTB has contributed to strengthening this area through the development and dissemination of the national technical guidelines for ICF among key populations. CTB also developed a district￾based Public-Private Mix (PPM) and successfully piloted WiFi TB, which is a mobile phone app for notification of TB patients in private sector. In addition to NTP, CTB is collaborating with agencies responsible for the National AIDS Program (NAP), the Ministry of Law and Human Rights (MOLHR), and the staff of the National Diabetes Mellitus & Metabolic Disease Program. CTB developed and successfully piloted WiFi TB,. This app helps private service providers to easily report TB patients to a puskesmas,4 which in turn reports through the NTP reporting system SITT (Sistem Informasi TB Terpadu). However, in Bandung city (West Java), the pilot failed due to poor design and execution, no clear technical guidelines, and weakness of the medical association that was tasked with pressuring physicians to report. In addition, CTB has provided data management training to provincial health offices (PHOs), DHOs and staff of other health facilitates. Findings from the field: CTB beneficiaries at different levels believe that the significant ICF efforts by CTB Indonesia should be strengthened and sustained through advocacy with higher authorities, work on regulations, and creating feasible models that can be replicated and integrated into the NTP operational system. In addition, the ICF effort should be expanded to cover vulnerable populations such as households in urban slums and workers in textile factories. Findings from secondary data analysis: - The number of all forms of TB cases notified in CTB districts increased from 44,253 in 2014 to 68,914 (56% increase) in 2016 compared to non-CTB districts where the number of TB cases notified slightly increased from 280,286 to 291,644 (4% increase) over the same time period. The number of all forms of TB cases notified in CTB districts in 2016 (68,914) is 60% of the CTB expected achievements in 2019 (114,750) - The percentage of TB cases notification from private providers in the 16 CTB districts has increased from 18% of all forms of TB cases notified in 2014 to 23% in 2016. - The evaluation team compared the progress of TB case notification from 16 CTB districts in 2015–2017 with other districts within and beyond the six CTB provinces using SITT data, with estimated TB cases by district in 2017 as the denominator. This analysis revealed that notification of TB cases in 16 CTB districts increased significantly, from 52% of estimated cases in 2015 to 66% in 2017, 5 while the percentage of the TB case notification in non-CTB districts was relatively stagnant (see Figure 2). Even in 2015, the rates of notifications in CTB districts were higher than elsewhere in CTB provinces, which were, in turn, higher than in non-CTB provinces. 4 Puskesmas are Indonesian public primary care clinics (Pusat Kesehatan Masyarakat). 5 These figures are based on calendar years (Jan-Dec) achievements. USAID/Indonesia Challenge TB Project Midterm Evaluation 20 Figure 3: Percentage of TB Cases Notified out of Estimated Cases by District Category, 2015–2017 Note: * the calculation is using data until Q3 2017, for Q4 2017, the calculation is using projected data (using highest number of Q1-Q3 number for estimating Q4) C. Ensuring the quality of treatment and care for TB, MDR-TB, and TB/HIV co-infection CTB has contributed significantly to this intervention area through technical training and mentoring of providers; improving the diagnosis, referral and management of MDR-TB by service providers; establishing and empowering cured patient groups to support their peers; introducing and implementing a shorter regimen for treatment of MDR-TB; and providing technical assistance to NTP staff to certify and institutionalize the enhanced cohort review (ECR) as part of PMDT. Findings from the field: Although the efforts and contributions of CTB in this area are recognized, valued and appreciated by District Health Offices (DHO) and service providers in different types of facilities in the CTB districts, there is a common concern among most of CTB beneficiaries about the replicability and scaling-up strategies/plans of CTB to integrate and sustain such activities within the current NTP structure and systems. Findings from secondary data analysis: - The percentage of MDR-TB patients still on treatment and culture negative six months after starting MDR-TB treatment in six CTB provinces significantly increased from 47% in 2016 to 82% in 2017 (as of June 2017). The CTB expected target over the life of the project for this indicator is 90%. However, the number of persons with MDR-TB who received treatment is 30% lower than the number of identified persons with MDR-TB; this remains a programmatic concern. - The percentage of registered new and relapse TB cases with documented HIV status toward the CTB expected target (68,850) in 16 CTB districts increased from 30% in 2015 to 39% in 2017 (as per June 2017). D. Expanding the network of diagnostic services USAID/Indonesia Challenge TB Project Midterm Evaluation 21 CTB is filling the gap in GFATM activities by supporting the technical aspects of GeneXpert placement, installation and use. Installation of GeneXpert in health facilities creates hubs for the diagnostic service network. CTB technical assistance is acknowledged and appreciated in training on the Lot Quality Assurance Sampling System (LQAS), and TB sample transfer. Findings from the field: The percentage of Global Laboratory Initiative (GLI) standards for TB microscopy toward the CTB expected target for the life of the project (meet 11 GLI standards) increased from 64% in 2015 to 73% in 2016. E. Strengthening M&E, surveillance and operations research CTB piloted several initiatives to strengthen M&E, such as WiFi TB, continued improvements of the Electronic TB (e-TB) Manager system for MDR-TB, and technical assistance related to SITT. Also, the Monthly Interim Cohort Analysis (MICA) and ECR provide a promising pathway to improve the quality of MDR-TB data (correctness, completeness, and timeliness), although measurement of this possible effect remains a work in progress. In addition, CTB has assisted in the identification of minimum notification/reporting requirements for private sector notification and collaboration in the NTP, laying the basis for the development of WiFi TB. Findings from the field: Despite the initiatives and innovations that CTB is developing to support the TB system, delayed and incomplete electronic reporting remains a significant challenge. More follow up and troubleshooting are needed, as well as development of plans to integrate these innovations into the current system. Data produced by CTB is difficult to understand and use, and is not tailored to serve the needs of different audiences. Findings from secondary data analysis: Progress toward ten of the project indicators (from the CTB list of expected achievements over the life of the project) can be measured. Of these, reliable numerical data exists for measuring six: 1) TB case notification; 2) case notifications from private providers; 3) Xpert testing for presumptive MDR-TB patients; 4) TB patients who are tested for HIV; 5) GLI standards for TB microscopy; and 6) microscopy laboratory performance. For the other four indicators there is no reliable data and/or they require qualitative analysis. They are: 7) MDR-TB patients’ enrollment in care; 8) MDR-TB patients still on treatment after six months; 9) HIV-positive TB patients started on ARVs; and 10) the real￾time case-based ERR system is functional for both TB and MDR-TB. Figure 4 shows the number of key activities and progress towards set targets by intervention areas in APA-3 by quarter. APA-3 Quarter 3 had the highest number of activities that either completely or partially met their targets, especially those under “Expanding diagnostic services.” This is consistent with the quantitative analysis of the CTB results which showed that the project achieved around 92% of the intended results for this component of APA-3. USAID/Indonesia Challenge TB Project Midterm Evaluation 22 Figure 4: Number of CTB Key Activities in APA-3 (Quarters 1-3) and Progress towards Targets by Intervention Areas CTB Objectives and Work plans The review of the project’s objectives, work plans and reports, as well as the latest national TB strategy (in Bahasa) and findings from the field, indicated that the CTB approaches and activities are aligned and consistent with the NTP Strategic Plan mandates and priorities. Starting from APA-2, the project management team focused on Objective 1: “Improved access to quality patient centered care for TB, TB/HIV & MDR-TB services,” with a secondary focus on Objective 3: “Strengthened TB platforms” to ensure the necessary enabling environment, political commitment and leadership, along with adequate allocation of resources, required to achieve Objective 1. These strategic changes were a programmatic response by the project to the changes in the NTP priorities, which focused on supporting local governments at the district level to facilitate the development process of the DAPs. This change had been identified by NTP as a priority in the National Action Plan for TB elimination. One DHO director interviewed expressed his appreciation for the value and importance of the CTB activities in addressing the pressing TB in Indonesia when he said, “We need every effort, including international organizations, to fight TB in Indonesia.” The evaluation team received similar messages during the interviewing process of several DHO directors and staff in different locations. A Common statement made by service providers in several puskesmas visited was that the CTB training content and methodologies were of high quality and were very useful for their work. Acknowledgement and appreciation of the quality and value of CTB activities and interventions in strengthening the capabilities of staff and supporting the TB program was very common among most DHO staff and service providers (in puskesmas, public and private hospitals, labs, and clinics) trained by CTB. Although all those interviewed in the KIIs expressed clearly and correctly the objectives and benefits of the activities they were involved in, none could identify the CTB mandates and approaches, and how the program is supporting the national TB program in Indonesia. This finding suggests the need for more briefing of the project beneficiaries on the project’s goals, objectives, activities and partners. Also, many of those interviewed did not recognize the name of the project when they heard “Challenge TB”; though they immediately identified the project and its activities when the interviewers used the name of the implementing organization, KNCV. This shows the greater recognition of KNCV (as an organization) at the cost of CTB (as a project) and its mandates. USAID/Indonesia Challenge TB Project Midterm Evaluation 23 In its effort to intensify case findings (ICF) among the key vulnerable population, the CTB COP, FHI 360 Country Director, and Director General of Corrections signed an Action Plan (work plan) between the Directorate General of Corrections, MOLHR, and CTB in April 2016. The purpose of this action plan is to provide the framework for CTB collaboration with MOLHR on implementation of ICF for TB, including supporting and strengthening linkages so that all inmates can have access to TB, DR-TB and TB/HIV services; development of a post-release program; and articulation of an exit strategy. However, the evaluation team noticed that more collaborative work is required between CTB and MOLHR on the strategic planning, policy and operational levels to move the project closer to actual exit from this field of work. By the end of APA-3 Quarter 3, with support from CTB, 11 prisons/detention centers conducted the annual TB mass screening using the new screening methodology (i.e., modified symptom and chest X￾Ray screening with GeneXpert examination) on 10,594 inmates. Interviews with regional MOLHR officials and field visits to prisons in certain districts revealed the magnitude and good quality of the work that CTB has done in training and technical support for the medical officers in selected prisons in the field of TB management and care and in conducting the TB mass screening of the prisons’ staff and inmates for the first time. These activities are highly valued and are supported by MOLHR at the central and regional levels. Referring to a CTB event, a medical officer in one district prison noted the importance of continuing and expanding these activities to include entry, regular, and pre-release screening: “It was a successful, important event, and we hope it will continue and the project can train more paramedics and nurses, and develop a system and regulations with MOLHR so we can continue without the support of CTB.” However, the reactions from the central officials of MOLHR were more conservative; interviews with these officials indicated that although they value the support they are receiving from CTB and FHI 360 at the district and national levels, respectively, they do not think it is a sustainable or scalable model because of the GOI budget limitation. Therefore, more advocacy efforts are needed targeting MOLHR authorities, as well as more work on the relevant regulations and policies, to facilitate adoption of these activities and their integration into the screening programs for inmates and staff. In addition, the central MOLHR officials made several common statements about the project: • CTB is less collaborative in the process of designing the project strategy compared with the preceding TB CARE I program; • The project should work on clarifying KNCV roles and responsibilities in this activity vis-à-vis FHI 360, since activities at the national and prison levels are provided by different organizations (FHI 360 at the national level and KNCV at the prison level); and • CTB does not pay resource person fees to civil servants who work for the prisons and attend events/workshops, but these resource persons are entitled to the fee, as stipulated in the relevant Ministry of Finance decree. This situation means General Directorate Correction staff have to look for money from another source to cover this cost. It has been a burden, especially when activities are not well coordinated. It seems this issue was not limited to MOLHR staff, as it was mentioned by several respondents from PHOs and DHOs. The CTB collaborative approach—working with DHOs and partners in setting the operational priorities, discussing the action plans, and coordinating activities—is recognized by partners. This is clear from the field interviews in different districts and the analysis of results from the self-administered questionnaires. However, staff in a couple of regional CTB offices noted that meetings and sharing action plans with partners and DHOs have not been common practices during the last year of the project. USAID/Indonesia Challenge TB Project Midterm Evaluation 24 DHOs and service providers say that they strongly believe CTB contributes significantly in supporting the local TB programmatic activities, especially in facilitating the development process of the costed DAP. A DHO director noted that, “DAP is a complicated process and not have happened without the systematic process that CTB implemented.” However, this acknowledgment did not prevent these partners from noting that the CTB process for developing the DAPs is complicated, long, and needs skillful persons and intensive coordination and follow up. In their opinion, these factors make the process non-replicable in other districts as the local government personnel lack these skills, and it is not clear if CTB has a plan or strategy to simplify the process and transfer the skills required to local government personnel so the process can be easily replicated in other districts. Higher up in the system, at the levels of NTP management and in the PHOs, the recognition, acknowledgment and appreciation of the contribution of CTB activities in supporting the NTP are lower compared to that from DHOs and service providers. This appears to be because CTB staff do not exert the same effort in involving these stakeholders compared to the DHOs and service providers in puskesmas, private facilities and labs. In interviews, NTP and PHO staff said that: • CTB has a project mind-set, focusing on its assigned areas without a real effort to impart technical know-how or transfer knowledge to develop replicable models that can be applied in other districts. • CTB is creating a parallel system that is not necessarily supportive of the current system. Examples include MICA and the CTB overall approach which operate in parallel with the regular cohort assessment by the MOH and the TB program management guidelines for DHOs. • Most of the short-term technical assistance (STTA) provided by CTB focuses on technical issues. More attention should be given to leadership, project management and planning skills which are required to strengthen the system and increase its efficiency. PHO staff do not feel close to the project, mentioning limited involvement in the planning and implementation processes. Figure 5: Percentage of Respondents Who Agree that CTB has Strengthened Local Capacity 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% CTB NTP Partner Service Provider National Province District National Province District All Respondent CTB Staffs NTP Staffs Strongly disagree disagree Neutral Agree Strongly Agree USAID/Indonesia Challenge TB Project Midterm Evaluation 25 Figure 5 summarizes the analysis of the responses to the self-administered questionnaire at the national, provincial and district levels on the question of agreement with the statement: CTB has strengthened the local capacity of the TB system to deliver strategic, effective TB programming in the five intervention areas. This analysis is consistent with the findings from the KIIs: most of the respondents agreed or strongly agreed with the statement. However, the level of agreement is lower at the national and provincial levels, even among CTB staff compared to DHO staff and service providers. MOHA recognition of CTB activities and the understanding of its roles are limited, as described by one of the MOHA officials: “I heard about CTB but don’t know anything about what they do in the field of TB and they never approached us. Involvement of MOHA in the DAP process is very important to address any barriers and monitor the progress.” Although CTB is working extensively at the district level to develop DAPs, coordination with MOHA at the central level is still required for better follow up and monitoring of this process. There has been a fairly major increase in the project’s awarding of sub-grants to local NGOs in the past year. This is something that should be continued, strategically scaled-up and utilized as it provides appropriate platforms for CTB to transfer its skills, tools and approaches to other districts. In collaboration with NGOs, CTB is supporting and empowering cured MDR-TB patients to guide and support their peers and collaborate with Aisyiyah community outreach activities in certain communities. However, CTB tends to be conservative in utilizing and capitalizing on the successful experiences and resources of other NGOs in the same districts. Representatives of a large NGO described CTB as territorial and felt that CTB considered other players in the same communities as competitors. One community group called PAMALI (assessing the quality of services) tried to access CTB service quality improvement tools in order to integrate them with their own to create a single package; however, CTB was reluctant and resistant to sharing, which created confusion in the community because two different tools are used in the same area. CTB developed and successfully piloted WiFi TB, the mobile app for TB notification by private service providers. The app provides a user-friendly tool for private providers to share data about their TB patients. Private sector providers value the WiFi TB initiative and think it is good for their work (reporting). Interviews with private providers in clinics and hospitals showed that CTB focuses its training with the private sector on the reporting application WiFi TB but not on basic TB management and care information, which is something requested by all private providers interviewed. Although the WiFi app increases notifications from the private sector, it increases the burden on puskesmas, as they have to add new records manually to SITT. Under the new reporting system (SITB), which will replace both SITT and e-TB manager, it should be possible to have integration with WiFi TB to streamline this important initiative. Conclusion CTB Indonesia is making substantial efforts to strengthen the technical skills of local personnel working in the fields of TB management and care, mainly at the district level in each of the five intervention areas. In its APA-4, CTB plans to roll out the district-based PPM, and scale up the GeneXpert testing and PMDT. However, CTB has limited initiatives that go beyond enhancing skills into building institutional systems and platforms to ensure replicability and sustainability of these efforts by the NTP and Provincial Health Offices (PHOs) in the current system. Documentation and structured dissemination of different CTB processes for planning, managing and evaluating different innovations would assist in knowledge and skills transfer to non-CTB districts. USAID/Indonesia Challenge TB Project Midterm Evaluation 26 Recommendations 1. Establish a platform for participatory action planning, monitoring, and sharing of best practices with different levels of government, partners and other stakeholders in order to develop and institutionalize replicable models of the successful interventions, strengthen the current system, and address gaps in TB management and care. Specifically, CTB should: a. Capitalize on the experience the project has accumulated in facilitating the DAPs in the 16 districts to document the process, develop a feasible, replicable model, and equip the PHOs with the resources, tools and skills to expand this effort to non-CTB districts. b. Increase the visibility and recognition of its mandate, objectives, approaches and values. This will increase synergy with activities conducted by other partners on the ground, which eventually will maximise the impacts of the different interventions. c. Strengthen its work on policy analysis, formulation and advocacy for the appropriate regulatory changes that support the advancement of TB management and care services in the country and facilitate the institutionalization and sustainability of successful CTB successful activities. 2. Expand support, institutionalize and scale up strategic and successful interventions. Specifically, CTB should: a. Leverage the private service providers’ contribution to TB management and care services by involving their relevant associations/groups, providing more training of trainers to establish a core team of master trainers on TB-related technical issues, addressing barriers that may hinder the utilization of the WiFi TB application for TB notification, and ensuring that SITB can accommodate notifications from the private sector. b. Address barriers that hinder the current prison mass screening program, and institutionalize appropriate systems to sustain it. c. Capitalize on opportunities to replicate the cured patients’ groups and strategically coordinate with and utilize available NGO resources and networks to scale up the community outreach and awareness raising activities and replicate them in other areas. Q2. PROGRAM MANAGEMENT Are CTB management, staffing structure and capacity optimal for achieving project objectives efficiently and effectively? (Program Management) To answer this question, the evaluation team reviewed and studied the following issues: the project’s management and staffing structure, relationship with GOI, roles of and coordination with CTB partners, and nature and number of STTA consultants. The current CTB management structure (instituted in Sept 2016) is the third evolution of the project’s management structure. It is a hierarchical organizational structure in which the project’s staff are grouped based on technical specialties (e.g., labs, MDR-TB), provision of services to key populations (TB-HIV, TB-DM, TB in children), functions (ICT, policy, M&E), or location (Papua). Each group has a supervisor with clear reporting and coordination lines with the other components of the project. Although the current project management structure does not reflect the five key intervention areas, nor the three main objectives of the project, the management structure, components, hierarchy, and USAID/Indonesia Challenge TB Project Midterm Evaluation 27 reporting/coordination lines are supporting different technical areas the project is working on and different key populations the project is targeting. The current management diagram goes to the level of individual unit for each task, with clear positioning of the CTB partners in the areas where they are well matched: IRD and WHO are part of the policies and systems areas; and FHI 360 is part of the vulnerable group team. All of the partners report to the Technical Assistance Coordinator, and have lines of reporting and coordination with the other technical and operational teams. There are also functional coordination and reporting lines between different components of the project and between the national and subnational offices. Most of the CTB provincial coordinator officers (PCOs) and regional staff expressed their comfort with the current management structure, with the streamlined communication with different components of the program, and with the prompt responsiveness and support they are getting from the home office. However, they noted that they have not been involved in nor been asked to comment on the latest changes in the project’s management structure. The project COP is a technical expert, with a clear vision and strategies with regards to the project’s priorities and activities and how to achieve project objectives. This level of competency is not always evident in the next layer of the project’s leadership and management, which may reflect both weaknesses at this level and a lack of delegation from the COP. This puts a huge burden on the COP and hinders the prompt responsiveness of the management team to strategic and technical issues/requests. To strengthen the technical component of the project in guiding and following up on the different technical programmatic interventions, the COP has separated technical and operation teams to relieve technical staff from administrative assignments and to boost administrative and operational support to the field. However, this is not the case in the regional CTB offices, as some regional technical staff are still doing financial and other administrative tasks, which according to them, consume up to 30% of their time at the cost of their technical assignments. As described by several CTB PCOs and other staff, this situation has been aggravated by i) the demanding CTB reporting system (from the home office); and ii) the micromanagement and interference of the project coordinator in the daily operational (programmatic and administrative) activities of the regional offices, without full knowledge of the realities on the ground. The management structure tends to have more operational staff than technical staff, which may be an indication of the “doer” nature of the project versus the “technical assistance provider” role; this is different from the original mandate of the project. The other change that the COP made to the current management structure that was appreciated by the PCOs was strengthening the roles of the regional M&E and data officers, and strengthening utilization of their work at the regional level. This was done through creating coordination and reporting lines between the regional M&E and the PCOs and Strategic Information and Knowledge Management (SIKM) Coordinators and not limiting the coordination and reporting to SIKM Coordinators only, as was the case in the previous structures. There are no documents and/or criteria that clearly determine the roles and the level of effort requested from each partner. FHI 360 has received different roles and responsibilities every year: in APA-1 it was given full responsibility for TB in prisons and TB/HIV implementation; in APA-2 FHI 360 was informed that all its assigned interventions would be taken over by KNCV, so they handed them over; and in APA-3 FHI 360 was assigned the role of TB in prisons and TB/HIV at the national level only (in addition to Papua). These abrupt changes limit the sustainability and effectiveness of the interventions with local counterparts. The work plan is developed mainly by KNCV and sent to partners to add their activities, rather than developing it as an integrated plan with inputs from different components within one project. USAID/Indonesia Challenge TB Project Midterm Evaluation 28 Another issue that adds to the complexity of the planning process is the long and convoluted lines of management/reporting between partner representatives within the CTB project, their respective organizations’ management staff, and the management staff of the CTB project. The partner representatives are asked to report to their respective organization’s management staff, who (should) subsequently discuss the matter with the CTB management team before taking a decision. The nature of CTB is a technical assistance provider program developed to support the current system in Indonesia and strengthen its capabilities in TB management and care. The significant operational component of the project and the continuous increase in staffing by CTB to support the “doer” role of the project on the ground is not supportive of the technical assistance nature of the project; the total number of CTB staff increased in APA-3 by 21% compared to APA-1, while for KNCV the increase was around 27% over the same period. This is despite a reduction or minimal increase in CTB partners’ staffing levels. By comparison, the CTB project in Nigeria has a similar scope and almost similar budget for APA-3, but is operating with half the number of staff compared to CTB Indonesia. Table 1: Number of KNCV and Partner Staff in APA-1, APA-2, and APA-3 The same argument applies for the factors that determine the number, level of effort, and assignments of the international STTA consultants. Several interviewees from the NTP at the national and provincial levels indicated that they have no/or limited say in the number and/or the nature of STTA provided by CTB, who (according to them) mainly focus on technical issues and do not address NTP management, leadership and planning needs. Despite a slight decrease in the number of the international consultants providing STTA between APA-1 and APA-3, and a significant increase in the use of local consultants, there is no clear strategy for knowledge transfer and skills sharing between international experts on one side and local partners and consultants on the other side in order to build up local capabilities. Figure 6: Short-Term Technical Assistance, Years 1 & 3 Total KNCV FHI 360 IRD WHO Note APA 1 122 101 15 0 6** * 2 part time ** 1 part time *** 1 part time APA 2 110 97 6* 1 6** APA 3 148 128 10** 2*** 8** USAID/Indonesia Challenge TB Project Midterm Evaluation 29 Figure 7: Percentage of Respondents Who Agree with the Statement “CTB Management Arrangements and Staffing Structure and Capacity are Optimal for Achieving Project Objectives Efficiently and Effectively” Findings from the analysis of the self-administered questionnaire show that most of the respondents at different levels agree or strongly agree with the statement: “CTB management arrangements and staffing structure and capacity are optimal for achieving project objectives efficiently and effectively.” However, the findings show fewer agree or strongly agree with this statement at the national and provincial levels compared to the district and service provider levels. This is consistent with the views of NTP/PHO staff who said in the interviews that the project keeps the national and provincial authorities at a distance in the planning and consultation stages (compared to district-level staff and service providers who are actively engaged and are direct beneficiaries of the program). Conclusion The CTB Indonesia COP did make significant changes in the management structure by separating the technical and operational teams to strengthen the technical component of the program and increase the effectiveness of the operational component. The COP also streamlined communication between CTB regional offices and the home office to improve their monitoring and evaluation functions, and respond quickly and more effectively to any programmatic and/or administrative issues in the field. However, certain factors hinder the optimal efficiency and effectiveness of the current management system and staffing structure of the project in fulfilling its role as technical assistance provider to the NTP at the national and local levels so that the NTP can strengthen its capacities to sustain and scale up successful CTB interventions. These factors include the centralized management approach with limited delegation; the demanding reporting system between sub-national and national levels; the convoluted communication and decision-making process between CTB management and CTB partner representatives; and lack of clarity regarding the systems for knowledge sharing and skills transfer between international STTA consultants and local partners and experts (to strengthen the technical assistance mandate of the project). 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% CTB NTP Partner Service Provider National Province District National Province District All Respondent CTB Staffs NTP Staffs Strongly disagree disagree Neutral Agree Strongly Agree USAID/Indonesia Challenge TB Project Midterm Evaluation 30 Recommendations 1. Management style: a. Increase the degree of delegation and autonomy given to the second layer of management staff. b. Streamline management tools and increase autonomy of the regional project staff to enable better performance and results. c. Strategically identify and design, in consultation with NTP and local partners, the activities that best complement the current NTP program components, empower NTP staff, and enhance the integration and sustainability of new interventions. d. Jointly with NTP, critically assess the number and the role of staff and better align them with the technical assistance mandate of the project in supporting the current NTP systems, strategies and approaches. 2. Utilization of the full capacities of CTB partners and counterparts: a. Develop and share with partners a set of criteria that details the roles and level of effort of each partner within the technical assistance mandate of the project. b. Ensure active and strategic participation of the project’s partners and counterparts in a collective development process of the project’s annual action plan and strategies. c. Streamline communication and decision-making processes between the CTB management team and partner management team to improve supervision of partner representatives hosted by the project. 3. STTA: a. Set criteria to determine, in a rational and strategic way, the role of STTA in the project to support the project’s technical assistance mandate (i.e., to fill in technical and project management gaps). b. Critically assess and track the use and impacts of STTA, and maximize the development and utilization of local staff in technical and program management areas, including their leadership capabilities. Q3. INFORMATION GENERATION AND USE Has the information generated by the project been used to support achievement of objectives and outcomes? The CTB project has a specific sub-objective on “Quality data, surveillance and M&E” under the objective “Strengthened TB Platforms”. This sub-objective aims at improving the quality of NTP data, and generating information to support achieving the project outcomes. There are eleven key activities in APA-3 under this sub-objective that cover a wide range of interventions focused on improving NTP data recording and the reporting system/application; improving the capacity of local data officers; translating data into epidemic modeling applications; and operational research to inform programmatic planning and support the project’s advocacy work. These key activities were monitored using four mandatory outcome indicators in APA-1, which were reduced to two in APA-2. In addition, the project introduced a district dashboard in APA-3 to help CTB staff and key stakeholders at the district and provincial level monitor their progress indicators. There are 31 indicators in the dashboard, 13 of them to monitor CTB activities, and the others to assess the outputs of the NTP. USAID/Indonesia Challenge TB Project Midterm Evaluation 31 The evaluation team reviewed the APA-1 and APA-2 reports and the reports for Quarters 1-3 of APA-3 and interviewed different key informants in order to assess the capacity of the project staff to analyze data and provide it to the project staff, partners and NTP on different levels, in a format that can inform the project planning and evaluation processes and support the project’s achievements dissemination and utilization. The project has generated important technical information in several formats related to the progress, achievements and lessons learned from its activities. These include regular project reports, abstracts/posters, presentations, scientific publications and technical documents. There were seven abstracts/posters, one technical document and one scientific publication listed as project products in the April–June 2017 quarterly report. Most of these information products are related to TB treatment management and case reporting, and linked to the annual International Union Against Tuberculosis and Lung Disease (IUATLD) global TB meeting. Interviews with NTP staff at the district level revealed that there is appreciation for the information generated and utilized by the project to improve the quality of TB programming and services. Most of these informants (90%) agreed that the project plays an important role in improving the capacity of data officers at the facility level to provide quality and timely data on TB cases, which in turn leads to progress by districts towards achieving the TB case notification targets and fulfilling the mandatory notification regulation. However, the levels of agreement with this positive statement are significantly lower among NTP staff at the provincial (73%) and national (67%) levels compared to district-level NTP staff. This pattern is consistent with the high volume of CTB project activities on the district levels. Figure 8: Percentage of Respondents who Agree with the Statement “Information Generated by the Project has been Used to Support Achievement of Objectives and Outcomes” Currently, with technical assistance from the CTB project, nine provinces have used the TB burden estimation in their planning process: Maluku, Sulteng, Sulut, Banten, Sumut, West Nusa Tenggara (Nusa Tenggara Barat [NTB]), and Kepri. Respondents from health facilities say they appreciate the project for its role in strengthening district/provincial laboratories and revitalizing the mechanism for quality control of sputum smear reading (through implementation of the e-TB 12 application). The quality control results are used by district-level actors, including CTB technical officers, to monitor and improve the quality of TB USAID/Indonesia Challenge TB Project Midterm Evaluation 32 diagnostics at health facilities. However, the dashboard result for indicator number 7 (% of all sputum smear laboratories that passed external quality assurance) has not improved yet. Only three of 11 key activities in the “Quality data, surveillance and M&E” sub-objective achieved their milestones established in APA-3. At the end of APA-3 Quarter 3, eight (73%) of the key activities either partially reached or did not reach their milestones. The evaluation team found that the results of two mandatory outcome indicators in the reviewed APA￾2 annual report were not reflected in the reporting on the project’s progress: the first indicator (score of electronic recording and reporting system) has the same target as its baseline (score = 3  a patient/case-based, real-time ERR system functions at national and subnational levels for both TB and MDR-TB); and there has been no updated information provided on the second indicator (Standards and benchmarks to certify surveillance systems and vital registration for direct measurement of TB burden have been implemented) since 2013. The evaluation team also could not identify indicators, either the mandatory outcome indicators (two indicators) or in the dashboard (31 indicators for progress and outputs of the district system and its performance), that allow assessment of the technical assistance mandate of the project, such as percentage of districts with improved resources and capacity to develop and operationalize DAPs (which could be composite indicators related to specific types of technical assistance provided by CTB). Many of the interviewed CTB and NTP staff at provincial and district levels emphasized the complexity of the dashboard indicators, saying that the list is long, confusing and some of the indicators need better definitions. In addition, they said that several indicators are not clear, and it is not easy to collect information about them—such as indicator number 22 (% of patients with pulmonary TB who are tested by Xpert) and 25 (% of RR drug resistant patients with 2nd line Probe Assay [LPA] results)—or are not measured during either the baseline or status/results in Quarter 4 of APA-3. (See Table 2) Although the evaluation team found that district dashboard indicators are useful and are used by the CTB district-based staff, these same staff noted that they cannot use the dashboard data easily and effectively for planning and strategy development purposes. Many of the CTB and NTP staff at district level: a. Do not fully understand the purposes of the dashboard (which is to assess the district activities of the NTP and to monitor project performance); b. Believe the dashboard has a long list of indicators, is still in the development stage and requires more work to make it user-friendly; and c. Feel frustrated when trying to fulfill the aspirational targets in the dashboard because they believe the targets cannot be achieved within the time allocated. Table 2: Summary of District Dashboard Indicators, Targets, Baselines, and Latest Results (APA-3) for the 16 CTB districts Indicator Target October 2016 July 2017 1) % of public non-NTP hospitals under Hospital Directly Observed Treatment Short-course (DOTS) Linkage (HDL) 100% 77% 87% 2) % of private hospitals under Hospital DOTS Linkage (HDL) 80% 38% 45% 3) % of general practitioner physicians in Primary Health Care Centers (HCs) engaged in TB management >90% 0% 0% 4) % of PHCs with active public-private mix (PPM) network 100% 0% 7% 5) % of PHCs actively reporting in SITT 100% 95% 97% USAID/Indonesia Challenge TB Project Midterm Evaluation 33 Indicator Target October 2016 July 2017 6) % of detention centers conducting combined routine care and screening for TB 80% 43% 100% 7) % of all sputum smear (SS) labs passed external quality assessment (EQA) >95% 51% 52% 8) Average Xpert utilization rate in the past 3 months 80% 53% 52% 9) % of non Xpert sites (PHC, hospitals and private labs) testing sputum at an Xpert site 100% 22% 40% 10) % of sub-district PHCs managing MDR-TB patients 80% 45% 54% 11) % district with "monthly interim cohort analysis" 100% 0% 67% 12) % of TB treatment facilities with on-site HIV test 95% 29% 40% 13) Domestic allocation per capita for DAP-TB (0.8 USD) 1% 6% 14) % overall case detection rate (CDR) (based on district estimate) >80% 55% 66% 15) % notification of childhood TB 12% 11.7% 14.8% 16) % of case notification (CN) through public-public collaboration 25% 31% 31% 17) % of CN through public-private collaboration 25% 20% 27% 18) % of CN by PHCs 80% 49% 67% 19) TB success rate (TSR) among all TB cases >90% 79% 69% 20) % of TB patients knowing their HIV status 95% 18% 27% 21) % of TB/HIV patients on ARVs 95% 19% 39% 22) % of PT TB patients tested by Xpert 95% NA NA 23) % of new TB patients tested by Xpert 20% 1% 4% 24) % of Rr* patients diagnosed out of total estimated Rr among notified PTB 80% 34% 69% 25) % of Rr patients with 2nd line LPA results >90% NA NA 26) % of diagnosed Rr patients enrolled on MDR treatment 95% 72% 77% 27) % of MDR patients receiving full psychosocial economic support (PSE) package >95% 0% 71% 28) % MDR patients still on treatment at 6 months of treatment >95% 66% 72% 29) % of newly enrolled Rr patients using the basic 20M regimen 95% 100% 100% 30) % of Rr patients enrolled on shortened treatment regimens (STR) (start July 2017) >80% NA 33% 31) % of Rr patients enrolled on ND regimen** >10% 4% 13% Note: *Rr patients: Patient of bacteriologically confirmed, clinically diagnosed or unconfirmed MDR-TB **ND Regimen: new drug regimen (bedaquiline or delamanid) Green: ≥100% Target Achieved Yellow: ≥75% - <100% Target Achieved Orange: ≥50% - <75% Target Achieved Red: < 50% Target Achieved Conclusion CTB Indonesia has generated important information related to progress and achievements from its activities. However, dissemination and utilization of this information is still limited and has not motivated USAID/Indonesia Challenge TB Project Midterm Evaluation 34 key stakeholders to take necessary actions to improve their performance. While the mandate of the project requires improvement in the performance of district-level key stakeholders in order to achieve the project objectives and outcomes, most of the information generated by the project is not yet produced in appropriate forms for the intended audience/s and purposes (e.g., no policy briefs have been produced for policy makers). Recommendations 1. Revise the project’s process, output and outcome indicators matrix to better capture the project’s performance and to reflect the technical assistance nature of the project. a. Differentiate between the process, output, and outcome indicators, ensuring the project’s indicators for NTP are impact indicators. b. Consider less aspirational targets and less complicated data collection indicators to avoid demotivating CTB district-level staff. c. Standardize and clarify the definitions of indicators with the local staff, and ensure they are capable of collecting accurate information. 2. Strengthen the dissemination and utilization of the project’s M&E data, best practices and lessons learned at different levels and among different partners and counterparts. a. Generate information in user-friendly formats appropriate for the intended audiences and their purposes; link all M&E data created by the project to NTP plans and strategies and utilize the data generated for advocacy and regulatory/policy change initiatives. b. Build an appropriate platform for all key stakeholder so that information generated can be shared and utilized by partners. 3. Disaggregate data by the defined key populations and risk factors, regions, types of facilities, and gender, and utilize the results of other relevant studies to better inform the strategies and plans to address more specific areas and key populations. Q4. ADAPTATION TO INDONESIA’S HEALTH SYSTEM To what extent has CTB been adapting its approach and activities to recent systemic changes in Indonesia’s health system, i.e., the revised Law on Decentralization (Law 23, MSS, etc.) and implementation of National Health Insurance (JKN)? There are four CTB key activities under Objective 1 on “Improved access” and Objective 3 on “Strengthened TB platform” in APA-3 that directly address decentralization and JKN. In addition, CTB approaches and activities overall are designed to respond to recent health system changes related to decentralization and minimum service standards for health services at district level. CTB is undertaking a wide range of initiatives to strengthen the TB program at district level. This approach is different from the previous USAID-funded TB project in Indonesia (TB CARE I), which focused on improving the capacity of the NTP at national and provincial levels. However, since CTB Indonesia has primarily limited itself to the indicators devised by the global CTB project, there is no mandatory outcome indicator that can demonstrate the achievement of CTB's approach and activities on the Indonesia-specific topics of decentralization and JKN. Decentralization. From the document review and KIIs, the evaluation team found that the District Action Plan (DAP) is one of the main approaches of the CTB project in responding to health system decentralization, in addition to strengthening the capacity of districts and health facility staff in various USAID/Indonesia Challenge TB Project Midterm Evaluation 35 technical and programmatic areas. There are two key activities related to the decentralization process, and both of them are meeting APA-3 quarterly milestone targets. The project has successfully facilitated key stakeholders in all 16 CTB districts to develop costed DAPs, with four of them enacted through head of district decree. CTB has extensively supported these processes by providing the required technical and financial assistance throughout the planning and approval processes. This effort is recognized and appreciated by the DHOs, especially CTB’s engagement with the relevant non-health sectors and the achievement of gaining their commitment. A DAP is a crucial strategic and planning tool that can serve as an official reference document to strengthen and sustain the TB program at local levels by channeling political will and commitment to attract and allocate more funding and resources to increase the coverage and quality of TB services. The evaluation team found that the signed DAP for Tulungagung district in East Java became a strong basis for the District Planning Bureau to approve a specific budget code for the TB program, and to increase the budget from IDR 47 million in 2017 to IDR 1 billion for 2018 (in the district government’s annual budget proposal). In another CTB district, the evaluation team was informed by DHO officers that they increased the TB program budget from IDR 50 million in 2016 to 450 million in 2017 by using the draft DAP as a reference document. However, the comments of key stakeholders in CTB districts about this process vary; most of them say they still require more support through advocacy and technical assistance and guidance to translate their DAP into better budgets and activity plans (to improve TB programming and implementation at the local level). Based on KIIs with 27 CTB staff at provincial and district levels, it appears that the CTB national office did not coordinate with the project field officers when the project selected a national-level NGO to advocate with the House of Regional Representatives in some CTB districts to increase political commitment and budget allocation for TB programs. The NGO activities and approaches are not aligned with the DAP, and it has not been utilized for the advocacy purpose. The analysis of responses to the self-administered questionnaire shows that around 80% of the interviewed key respondents agree that the CTB work plans and strategies are responding effectively to changes in the government structures and policies. However, the level of agreement with this statement among national NTP staff is lower than with other groups of interviewees. The reasons could be (as described by one of the NTP officials at the national level) that the nature of CTB approaches/activities do not give enough attention to strengthening NTP’s institutional capabilities compared to the TB CARE I project, which provided intensive technical assistance to the NTP and PHOs (it covered more districts in the provinces). Once the NTP has completed their DAPs with CTB support, it is expected that this will generate additional burden on CTB to provide more technical assistance to support effective implementation of the DAP. This could create a dangerous feedback loop of assistance that cannot be sufficiently provided without involving and empowering the national and provincial levels to accommodate these increasing demands at the district levels. The percentage of NTP staff at district level who agree that CTB has been adapting its approach and activities to recent systemic changes in Indonesia’s health system (i.e., decentralization and JKN) is 80%, while the level of agreement among the national NTP staff is 75%. This lower level of acknowledgment and support to CTB activities at the national and provincial level compared to district level is consistent across questions. This might suggest that district-level stakeholders are happy because CTB districts are doing well, while the national and provincial stakeholders are worried about how all of the work will get translated to non-CTB districts—hence their lower level of enthusiasm across multiple questions. The same pattern also is evident among CTB staff respondents, with 90% of staff at the district level agreeing with the statement, compared to 80% and 67% of the provincial and national staff, respectively. More national NTP staff (75%) are in agreement with this statement compared to national CTB staff (67%). This could be due to greater knowledge among CTB staff of the implementation challenges compared to NTP staff, which makes the former less supportive of the statement. USAID/Indonesia Challenge TB Project Midterm Evaluation 36 Figure 9: Percentage of Respondents who Agree with the Statement “CTB Work Plans and Strategies are Responding Effectively to the Changes in the Government Structures and Policies” National Health Insurance (JKN). Ensuring universal access to health insurance by integrating TB in the JKN is part of CTB’s five technical intervention areas, and the program finalized the National Health Insurance Guideline for TB services in APA-1. Although that there are some opportunities that CTB should take advantage of to improve the TB program by using JKN funding, CTB has been instructed by USAID not to engage in activities with JKN given the relatively unsuccessful experience of the project with the development of technical guideline and lack of experience within the project staff to advise the NTP in this area. However, other USAID-funded projects such as the Strategic Health Purchasing (SHP) project will continue providing support to JKN as needed, with TB inputs from CTB as requested. There is limited information from the KIIs about CTB efforts to integrate TB services into the JKN funding mechanism. However, there is a lesson from the KIIs on how CTB officers at the district level provide practical solutions to access the JKN funding mechanism for the costs of supporting laboratory tests using GeneXpert. Splitting laboratory test costs between primary health centers (PHCs) and hospitals proved to be an acceptable unwritten mechanism used by health facilities for GeneXpert testing in the districts to reduce patient costs during the process. According to KIIs with DHO staff, there are several unutilized opportunities to access the JKN funding mechanism for the TB program, but these need CTB technical assistance. For example, the JKN capitation fund for PHCs can be allocated for establishing and supporting patient support groups to improve treatment adherence and also to conduct investigation activities. CTB is supporting patient support groups in some districts but the funding is not sustainable. It would be more sustainable to assist health facilities to access the JKN capitation fund to support the patient support groups or any other related community action. JKN systems can also be utilized to contribute to increasing the TB case notification for the PPM scheme. DHO staff suggest that the CTB program support MOH and the Social Insurance Administration Organization (Badan Penyelenggara Jaminan Sosial [BPJS]) to develop an integrated scheme for private hospitals (and the primary health care level, if a specific TB payment is introduced on top of capitation) that would integrate the JKN fund claim for TB services and the case notification report. Under such a scheme, BPJS would pay the reimbursement after receiving notified cases. This idea can encourage private sector actors in finding new cases. A health financing project supported by USAID Indonesia is currently initiating conversations with GOI stakeholders around these types of ideas. USAID/Indonesia Challenge TB Project Midterm Evaluation 37 In the correctional facilities, CTB is providing technical guidance on the process of conducting mass screening for 24 prisons and gradual mass screening for 11 prisons (since the beginning of Year 2). Conclusion CTB has been adapting its approaches and activities to Indonesia’s new health system, mainly by facilitating the DAP process. To ensure more alignment with and support for the current changes, CTB should work closely with core stakeholders, in a participatory and collaborative manner, to develop a replicable model and technical guidelines for DAP development that can be applied in non-CTB districts. Recommendations 1. Support the development of a National TB Action Plan to guide and support the rollout of the DAP process to the hundreds of non-CTB districts, and actively engage all relevant departments (MOHA and Bappeda6) in the DAP development and approval process. 2. Expedite the approval and dissemination process of the technical guideline for replicating the DAP process in non-CTB districts, and provide clear tools, flow charts, and documented, doable steps for the process, with the involvement of MOHA and Bappeda as a catalyst for the process. 3. Collaborate with individuals providing USAID-funded technical assistance on health financing, especially as it relates to TB services. Utilize the opportunity of JKN capitation funding at the puskesmas level for activities such as supporting patient support groups, community case finding, outreach to private providers, and conducting contact investigations. 4. Emphasize customized advocacy efforts targeting the right stakeholders in different relevant sectors such as the MOHA, Ministry of Social Affairs, and Ministry of Village Affairs to initiate policy and regulatory reviews and changes to facilitate and support TB management and care activities. Q5. LESSONS AND BEST PRACTICES Are there lessons and best practices from CTB implementation that could be replicated in non￾CTB districts? In APA-2, the project added two specific indicators to monitor budget allocation, progress and achievements in generating, documenting, disseminating and facilitating adoption and replication of lessons and best practices from CTB implementation. Those indicators are under Objective 3 “Strengthened TB Platforms” and Sub-objective 10 “Strengthening Quality data, surveillance and M&E”, with two key activities for operational research. In addition to these specific key activities and indicators for generating and replicating lessons and best practices, all components of the project are designed to improve national TB program performance through innovative approaches and intervention models in a small number of districts that can be adopted, replicated and sustained. 6 Badan Perencanaan Pembangunan Daerah (Local Development Planning Bureau). USAID/Indonesia Challenge TB Project Midterm Evaluation 38 The evaluation team reviewed the reports of APA-1 and APA-2 and Quarters 1 to 3 of APA-3, and asked all key informants interviewed to fill in one Likert question to obtain their opinions on lessons and best practices from implementation of the project that could be replicated outside the project locations. The outcomes of the KIIs and the review of the APA reports revealed eight7 best practices identified by the project in their annual and quarterly reports. According to the interviews with the project staff at national, provincial and district levels, these best practices are related to support of patients with MDR￾TB (ex-patient empowerment to support the treatment program and diagnosis), collaboration on TB￾DM, TB screening in prison settings, the mandatory notification app, and costed DAPs. This is an illustrative list of the CTB successful interventions that are worth considering for replication. In addition, most key respondents in district and health services facilities said they believe that the other best practices related to CTB approaches are revitalizing the district-level quality control system of sputum smear, and establishing a referral network and quality control system for microscopic examination including training, recording and reporting components using the e-tb 12 application for all districts/cities with high TB epidemics (as currently being done in West Java and neighboring CTB districts in East Java). The analysis of the self-administered questionnaire responses shows that more than 90% of the project and NTP staff at provincial and district levels agree that the project has generated replicable and adaptable intervention models. However, around 78% of the CTB project staff and only a quarter of NTP staff at national level agree that best practices from CTB implementation can be replicated in other districts without the support of the project. Figure 10: Percentage of Respondents who Agree with the Statement “Lessons Learned and Best Practices from CTB Implementation Could be Replicated in Non-CTB Districts” There are several main reasons given for this disagreement: significant resources are required (financial resources, number and skills of human resources, and time) that cannot be obtained by any district; lack of an adequate regulatory framework; CTB focuses on technical issues without considering the limited 7 These are: TB District Action Plans as a way to eliminate TB through local commitment and ownership; Mandatory notification apps that increase the case notification by simplifying reporting; the new TB drug Bedaquiline made available in Indonesia; psychosocial support from ex-patients that motivates other patients to return to treatment; faster test results that enable faster treatment initiation; DR-TB patient empowerment; PMDT microtraining for district health officers to build self￾reliance for DR-TB treatment; and prison cadres training on cough surveillance. USAID/Indonesia Challenge TB Project Midterm Evaluation 39 capacity of the health system; lack of active involvement of the NTP at the national and provincial levels in planning activities based on their needs; and finally, the lack of knowledge and skills transfer to the staff of the NTP to carry on these activities without support of the project. One of the NTP officials at the national level stated that, “Although many of the CTB project’s activities are good and needed, they are designed with [a] project mindset to achieve the project’s targets with no vision of scalability nor replicability.” According to KIIs with members of two large faith-based organizations that are actively involved in the TB program in Indonesia, there are several CTB project approaches and interventions that are being replicated/utilized by their organizations, including the tools and process used for DAPs, and empowering communities to include ex-patients of MDR-TB in providing guidance and moral support to their peers who have TB. There are several practical and non-systemic approaches that have been undertaken by the project field staff to address challenges during the implementation of various project activities. These approaches are well recognized and appreciated by key stakeholders but have not yet been documented or shared with other project locations. These include informal advocacy to DHO leadership that resulted in increasing the number of TB program managers at district level, and splitting the cost of GeneXpert tests between hospitals and primary health care centers (to avoid patients using JKN from going back and forth to different hospitals due to a daily ceiling of laboratory testing costs). According to interviews with the project team at the national level, the knowledge management platform for sharing best practices and lessons within the project consortium and its partners is still under development, and external sharing of best practices through technical briefs is currently in the planning stage. Also, the project has not shared best practices and lessons from CTB implementation in appropriate formats for different audiences. Conclusion Several of the CTB’s best practices and lessons learned have been adopted and replicated by other organizations and the NTP. For example, Aisyiyah (PR GF-TB) adopted the DAP development approaches and processes and replicated them in more than 50 non-CTB districts where it is working; the organization LKNU (Lembaga Kesehatan Nadhatul Ulama; Health Organization of Nadhatul Ulama) adopted the CTB approach in supporting TB patients’ education sessions in hospitals and replicated it on the primary health centers (PHCs) level with some modification; and DR-TB benchmarking for treatment assessment has become a national tool for hospital accreditation and has been scaled up to the national level by the NTP. There are several other best practices and lessons that could potentially be replicated, such as: approaches to revitalize the district-level quality control system of sputum smear, and the referral network and quality control systems for microscopic examination, recording and reporting using e-TB 12 application for all districts/cities with high TB epidemics in West Java and neighboring CTB districts in East Java. However, more efforts are still required by the project to document and institutionalize these processes and approaches to make them feasible and easily replicable in other non-CTB districts. Recommendations 1. Provide technical assistance to the MOH at local and central levels to institutionalize and scale￾up best practices and lessons from CTB implementation. The project should: a. Document and share best practices in an appropriate form for various stakeholders; b. Design any remaining CTB activities with the NTP staffing and capacity levels in mind; USAID/Indonesia Challenge TB Project Midterm Evaluation 40 c. Outline, with NTP, a process by which improvements can be deliberately spread in the Indonesian governance context, given that the real implementation and mandate for activities is at the district not the national level; d. Develop and implement an advocacy plan for policies and regulatory changes required to adopt and replicate best practices and lessons; and e. Continue to finalize development of the platform for sharing and collecting best practices from and with field staff and other stakeholders. 2. Consider innovations in programmatic planning, management and leadership and not limit them only to technical issues. The success of implementations depends mainly on good planning, strong leadership, and a solid management system. Q6. SUSTAINABILITY To what extent are CTB methodologies, interventions, and management setting the stage for future sustainability of project outputs and outcomes? The CTB project is designed to provide sustainable solutions and replicable approaches through providing considerable staff and developing ‘’intensified’’ packages of technical assistance to support the NTP and other partners in designing and implementing a set of interventions at the district level. The project’s work plans are full of activities that are described as technical assistance and are aimed at sustaining different project activities. However, the project is not clear about how such activities would achieve the sustainability goal. In addition, the evaluation team noted that sustainability is not explicitly mentioned as one of the project strategies or approaches, nor is it measured via the project’s mandatory outcomes indicator list. Many of the CTB project’s strategies and approaches are bound by certain technical deliverables with no clear sustainability/institutionalization initiatives beyond fulfilling these deliverables requirements. These include: • Strengthening the technical capabilities of the staff at DHOs and puskesmas levels, but with minimal efforts to establish a knowledge sharing and skills transfer system to maintain the project’s training, coaching and supervision work beyond the life of the project. • The development process of DAPs, which are linked to the presidential decree on minimum service standards. This is a core part of Indonesia’s strategy to sustain effective TB management and care programs. The CTB project had made significant and successful efforts to facilitate this process in certain districts, but with no feasible and clear guidelines on how to replicate this process and expand it to other districts, such as through equipping the PHOs with the required skills and tools to replicate this model in other districts in their provinces. In addition, more work is required by the project on the advocacy and regulatory levels to get the plans approved and budgets allocated, and integration of TB services into the JKN funding mechanism. There is one key activity in APA-3—sustainable funding linked to quality assurance of TB diagnosis and care—which partially met its milestones target up until the Quarter 3 of APA-3. In addition, CTB monitors domestic allocation per capita through one indicator in the district dashboard, and there is only one (Mimika district) out of 16 CTB districts that has met the minimum budget allocation standard. Two positive statements were asked in the self-administered questionnaire related to a feasible sustainability strategy and the focus on increasing domestic resources to support NTP activities. The aim USAID/Indonesia Challenge TB Project Midterm Evaluation 41 was to obtain respondents’ opinions on CTB methodologies, interventions, and management structure for future sustainability of the project outputs and outcomes. Figure 11: Percentage of Respondents who Agree with the Statement “CTB Methodologies, Interventions, and Management are Setting the Stage for Future Sustainability of Project Outputs and Outcomes” More than 80% of respondents from the CTB project team agreed that the project’s strategy and approaches will sustain CTB outputs and outcomes. However, half of NTP staff at the national level and one-fifth at the provincial level did not agree with the statements that CTB implementation outputs and outcomes will be sustained without the project’s support. KIIs with six NTP staff at national level and 15 at provincial level indicated that most of CTB’s outputs and outcomes tend to be limited to enhancing the technical capacity of local partners’ staff. As an example, CTB is enhancing the capacity of health facility staff in recording and reporting TB and DR-TB cases (through training workshops, on-the-job training, and mentoring by dedicated data officers at district levels hired by CTB), rather than building the institutional capacities of the DHOs and PHOs to address the high turnover of the facilities’ data officers (i.e., CTB is not establishing a team of capable trainers to sustain the knowledge transfer, nor is it working on securing the funding for regular training and mentoring). CTB initiates community peer support programs through empowering cured TB patients. This kind of community approach produces good outcomes that support patients in some areas, such as Jember District (Sekawan) and Surakarta City (Semar). The activities aim at supporting persons with TB through education and guidance, and accompanying them during their visits to health services. LKNU (the health unit of the biggest faith-based organization in Indonesia) adopted this approach and is scaling it up with some modification. According to KIIs at the national level, CTB is piloting several innovative interventions, but most of them are designed with limited consideration and assessment of the health system’s structure, potentials and limitations. These could be major enabling or hindering factors to the adoption and sustainability of these new interventions. For example, CTB piloted a mobile application for compulsory TB notifications by private healthcare providers without a proper study on why some private providers are reluctant to report TB cases despite the availability of the WiFi TB system. In this activity CTB also did not address the rules that govern how private providers report nor how the data recipients process the data (the data is sent to the puskesmas where staff should re-enter it into the SITT in a parallel reporting system, which increases the workload and is not convenient for staff at the puskesmas level). USAID/Indonesia Challenge TB Project Midterm Evaluation 42 Conclusion CTB does not have a clear sustainability strategy and/or plan to go beyond enhancing technical capabilities of personnel to building institutionalized, sustainable systems. However, CTB’s tested methodologies, approaches and technical capabilities could provide a solid foundation to develop sustainable and institutionalized approaches and systems within the NTP at the national and provincial levels. Recommendations Opportunities still exist to accelerate the project’s capacity strengthening efforts towards building sustainable organizational systems in institutions. This could be done by: 1. Working with NTP at the national and regional levels to develop simple, feasible guidelines and a standard template to develop a replicable model of the Costed District Action Planning Process. 2. Contributing to strengthening and institutionalizing the diagnostic service network, MDR-TB management, and TB prevention (latent tuberculosis infection [LTBI] identification and management). 3. Addressing the technical and structural barriers to streamlining and institutionalizing the private sector notification system. FINDINGS FROM CTB PAPUA In Year 3 of the project, CTB added three districts in Papua Province (Jayapura City, Jayawijaya District and Mimika District) as priority sites, in addition to the provincial-level support to Papua that existed during TB CARE I and the earlier years of CTB. These three districts out of the 30 districts in Papua are considered relatively easier to access, having better quality of basic health services and are contributing to 17% of estimated TB cases in Papua Province. The project is playing an important role in strengthening and improving the capacity of local key stakeholders and health service providers in managing and monitoring the TB program’s activities through: providing a specified package of technical assistance based on local needs for expansion and quality assurance of essential components of TB control, including laboratories, TB/HIV, PMDT expansion, intensified TB case finding, increasing political and funding commitment to the costed DAP for TB program, strengthening the local services and diagnostic networks, and empowering health services’ personnel through training, mentoring and technical assistance on TB reporting, diagnosis, management and quality control. CTB strategies, approaches and activities in Papua are similar to the project’s approach and work in other districts, despite the greater magnitude of the public health challenges in Papua compared to other regions of Indonesia. The evaluation team visited four health facilities and interviewed 14 key informants from CTB, NTP, health facilities and partners in two out of the three CTB districts in Papua. In addition, the team also asked the interviewed key informants to fill in a self-administered questionnaire to obtain their feedback on the six evaluation questions. USAID/Indonesia Challenge TB Project Midterm Evaluation 43 Figure 12: Percentage of Respondents’ Agreement with Six CTB Positive Statements8, Papua Province The analysis of the self-administered questionnaires shows that respondents from the CTB project in Papua have a higher level of agreement with all six of the positive statements from the evaluation questions compared to NTP staff at the provincial and district levels in Papua. The greatest difference is on sustainability of the project outputs and outcomes (Question 6); all CTB staff agree that the project deliverables will be sustained while only two-thirds of respondents from NTP (province and district level) agree with the statement. Other key informants made similar statements as the NTP staff: most of the CTB activities in Papua are implemented like “kejar tayang” (“running on deadline”) due to the late annual work plan approval process, and there is too much focus on achieving the project’s outputs rather than giving equal attention to building the local institutional capabilities of the national project’s partners to adopt and sustain these activities. “Providing a consultant for developing and facilitating a District Action Plan will not make the DHO able to revisit and adjust it in the future or even properly plan better activities after having the local budget increased.” – KII respondent from NTP-district level 8 The statements are: (1) “CTB has strengthened the local capacity of the TB system to deliver strategic, effective TB programming in the five intervention areas”; (2) “CTB management arrangements and staffing structure and capacity are optimal for achieving project objectives efficiently and effectively”; (3) “Information Generated by the Project has been Used to Support Achievement of Objectives and Outcomes”; (4) “CTB Work Plans and Strategies are Responding Effectively to the Changes in the Government Structures and Policies”; (5) “Lessons Learned and Best Practices from CTB Implementation Could be Replicated in Non-CTB Districts”; (6) “CTB Methodologies, Interventions, and Management are Setting the Stage for Future Sustainability of Project Outputs and Outcomes.” USAID/Indonesia Challenge TB Project Midterm Evaluation 44 The review of the district dashboard, and of the district system and performance, revealed that the three CTB districts in Papua have higher progress scores than the average for the 16 CTB districts, in part due to lower baselines. Two of the CTB-districts in Papua (Mimika and Jayawijaya) are the best among all CTB districts in improved performance, again because both districts started with low baseline scores due to the weak management role of the DHO, with almost no TB program activities in existence except for provision of basic diagnostics and treatment. The significant CTB role in supporting the DHO through providing the required technical assistance, training of staff, and resources has resulted in significant impacts on both districts’ scores. Table 3: Average Dashboard Score by District District Dashboard Score Difference Baseline APA-3 Q4 Medan 2,32 2,65 14% Deli Serdang 2,14 2,80 31% North Jakarta 1,96 2,38 21% East Jakarta 2,43 2,63 8% Central Jakarta 2,17 2,54 17% West Jakarta 2,19 2,46 12% South Jakarta 2,38 2,58 8% Bandung 2,39 2,25 -6% Bogor 2,09 2,25 8% Surakarta 2,45 3,15 29% Semarang 2,32 3,00 29% Tulungagung 1,96 2,22 13% Jember 2,09 2,57 23% Jayapura 2,40 3,00 25% Jayawijaya 1,87 2,60 39% Mimika 1,88 2,83 51% All 2,19 2,62 20% The evaluation team found that the CTB project in Papua has some technical and administrative challenges, which are attributed to the remoteness of Papua and management changes. These include: 1. Difficulty in identifying and recruiting high quality technical staff to deliver technical assistance in Papua. 2. The change the project implementation management in Papua from KNCV to FHI in APA-3. This has impacted the achievement of the project’s deliverables because: - FHI staff are more experienced in managing technical assistance for TB-HIV and TB in prison settings, and do not have the same level of experience in other TB management and care components; and - Several technical roles, such as support for diagnostic services and data management improvement, are either not well planned in APA-3 or are filled by junior technical staff who USAID/Indonesia Challenge TB Project Midterm Evaluation 45 do not have sufficient experience and clout to be accepted and appreciated by the more senior technical staff who are receiving the technical assistance. CTB should give special consideration in their plans and approaches in Papua districts given its remoteness through Sustaining and supporting the CTB-FHI management team. This would include: - Establishing special financial and administrative conditions to attract competent technical staff in these remote areas. - Addressing solutions to geographically-specific, basic health services/system challenges such as greater use of task shifting (due to limited number of health workers at remote Puskesmas). - Advocating for more resource mobilization to support the improvement of the health infrastructure. LONG-TERM RECOMMENDATIONS FOR USAID Several findings have been identified and discussed in this report concerning program design, management partnerships, and approaches for sustainability. Informed by these findings, below are the evaluation team's recommendations for future USAID's programming in TB management and care in Indonesia: • Consider engaging in high-level policy dialogue with the government (along with other leading donors) to advocate for a national focus on TB prevention, management and care with emphasis on key populations and emerging MDR-TB. • Support the design and implementation of a strategic and comprehensive cross-cutting BCC strategy to address TB-related information misconceptions, stigma, health-seeking practices, and support marketing of quality services. • Support high-level inter-departmental and inter-sectoral platforms to address the relevant social determinants of health (SDoH) in the planning and implementation processes of the national TB program and its activities at the national and regional levels. • Given the country's large population and regional variations, USAID should continue targeting its resources strategically to specific geographic areas, activities, key populations and high-level institutional/regulatory changes to achieve maximum lasting impacts. • Continue focusing on and maximizing private sector and NGO active involvement in TB early detection and management, as their work is attracting growing numbers of persons with TB, particularly in urban areas. • Continue supporting innovative and advanced technological systems and tools to streamline TB early diagnosis, notification and management, and support institutionalization of these systems and tools. USAID/Indonesia Challenge TB Project Midterm Evaluation 46 ANNEX I: EVALUATION STATEMENT OF WORK STATEMENT OF WORK Midterm Evaluation of the Challenge TB Indonesia Activity 1. INTRODUCTION Indonesia, with a population of 250 million, is both a high TB and a high HIV burden country, with the fastest￾growing (mainly concentrated) HIV epidemic in the region. The 2014 TB prevalence survey showed that prevalence is 2.5 times higher than previously estimated (average 660/100,000 population) and that estimated incidence is more than one million new cases per year. Challenges in addressing this prevalence include low case notification rates (only 327,103 TB cases were notified in 2013), and drug resistance (2% of all new TB cases and 12% of retreatment cases are estimated to be drug resistant). Tuberculosis remains a prominent contributor to the overall burden of disease in the country. TB is the fourth highest source of morbidity and mortality, causing around 100,000 deaths each year in Indonesia. 9 Although external funding sources amount to just 1 percent of total health expenditures in Indonesia, they remain an important source of financing and technical assistance for programs addressing immunization, HIV, TB, and malaria. In 2015, Indonesia’s Ministry of Health (MOH) estimated that external assistance was as high as 60% for TB, which was reduced from around 70 percent in 2014. The share of domestic funding for the National TB Program (NTP) has increased since 2009, and the ratio of external to domestic spending for TB was close to 50:50 in 2014 because the Government of Indonesia (GOI) started to fully finance the provision of first-line TB medicines, reagents, laboratory supplies and consumables. Today, the Global Fund continues to be the main source of external financing, followed by USAID.10 In the context of declining external assistance for TB and the chance of disruption of TB services, it is important for the GOI to ensure the control of multi-drug resistant (MDR) TB. As Indonesia continues to ensure domestic financing for TB, the country also needs to continue to strengthen the supporting health system functions. The key to NTPs’ sustainability is to better integrate USAID’s investments within the context of universal health care (UHC) and look to the National Health Insurance System (JKN) as it expands coverage. II. ACTIVITY DESCRIPTION Challenge TB (CTB) is a five-year cooperative agreement and one of the main global mechanisms to support USAID’s current strategy for TB, which started in October 2014. The purpose of CTB is to support countries with a high burden of TB, HIV-related TB, and MDR TB, in order to achieve their National Strategic Plans (NSPs) for TB and to contribute to achievement of Millennium Development Goals and Sustainable Development Goals. CTB’s goal is to strengthen TB services to improve prevention, diagnosis, treatment, and care, especially for vulnerable and high-risk populations, by utilizing locally owned resources, innovation, and research, and by coordinating and collaborating with the Global Fund to maximize limited resources. Challenge TB for Indonesia was launched in January 2015. The activity is implemented by KNCVand its coalition of in-country partners, including WHO and FHI 360. CTB Indonesia is a five-year USAID-funded project supporting the GOI’s National TB Program with implementation of the NSP. CTB aims to develop comprehensive, effective, and cost-efficient interventions that can be rolled out across Indonesia using domestic resources and other development resources. The five-year framework is in line with Indonesia’s NSP for TB control. Based on the systematic gap analysis process during the development of the NSP and a review of the Global Fund investments, it 9 CTB website (https://www.challengetb.org/where#asia) 10 Indonesia Health Financing System Assessment, 2016 USAID/Indonesia Challenge TB Project Midterm Evaluation 47 was agreed by USAID and GOI to prioritize the following five key strategic interventions to address the major gaps of the NTP: 1. Ensuring universal access by integrating TB in the National Health Insurance System, and securing increased local government funding for TB. 2. Increasing case detection: Intensified Case Finding to address the current gap in notification. 3. Ensuring the quality of treatment and care for TB, drug- resistant TB, and TB/HIV co-infection. 4. Expanding the network of diagnostic services. 5. Strengthening M&E, surveillance and operations research. At the national level, CTB provides technical assistance to the MOH Directorate of CDC as the main partner. Additionally, other government institutions, Provincial and District Health Offices, professional associations, community organizations, and local partners in supported provinces receive technical assistance. At the sub-national level, CTB works in nine provinces overall. However, to align NTP and USAID focus priority areas and issues, 5 provinces with 10 initial districts were selected based on following criteria: (i) large provinces prioritized by NTP with high burden of TB and HIV; (ii) USAID Indonesia priority provinces; (iii) ability to complement previous investments; (iv) presence of other USAID partners to increase synergy. These provinces and districts are: • DKI Jakarta –North Jakarta and East Jakarta • West Java –Kota Bandung and Kab. Bogor • Central Java –Kota Semarang and Kota Surakarta (Solo) • East Java – Kab. Jember and Kab. Tulung Agung • North Sumatra – Kota Medan and Kab. Deli Serdang These 10 districts receive an intensified package of assistance, including all components of TB control (i.e., laboratory network strengthening, PPM, PMDT, TB/HIV, and surveillance). The purpose of these intensified packages is to design and test small-scale models at the district level to better define a sustainable and scalable model TB program. CTB initiated work in the 10 districts in the first two years and will expand to 20 districts in these same provinces in Year 3. Additionally, CTB works to support local health services and partners in fourother provinces – West Sumatra, South Sulawesi, West Papua, and Papua – to provide a more specified package of technical assistance based on local needs regarding expansion and quality assurance of essential components of TB control, including laboratories, TB/HIV, PMDT expansion, and intensified TB case finding. CTB Indonesia will also provide support to the principal recipients of Global Fund TB. As a sub-recipient, KNCV assists recipients to address specific technical and managerial issues, and to support in planning, implementation, and troubleshooting. CTB support is complementary to Global Fund assistance. In Year 3 of CTB, the project and USAID Indonesia agreed to add three districts in Papua Province (Kota Jayapura and Kabs. Jayawijaya and Mimika) and three districts in DKI Jakarta (Jakarta Selatan, Jakarta Pusat, and Jakarta Barat) to be included as priority sites. At the same time, three of the provinces receiving specified packages were graduated, and thus in Year 3, CTB works in 6 provinces and 16 districts, as follows: • DKI Jakarta –Central, West, North, East, and South Jakarta • West Java –Kota Bandung and Kab. Bogor • Central Java – Kota Semarang and Kota Surakarta (Solo) • East Java – Kab. Jember and Kab. Tulung Agung • North Sumatra – Kota Medan and Kab. Deli Serdang • Papua – Kota Jayapura, Kab. Mimika, Kab. Jayawijaya Total funding of CTB Indonesia approximately $40 million over the life of the project. In 2017, CTB is entering the third year of implementation, and the project aims to increase case detection and treatment success rates; improve quality of care and diagnostics, reporting, recording; and improve mobilization of local resources, including financing. CTB will also work on policy guidance and regulation to provide support to NTP implementation. USAID/Indonesia Challenge TB Project Midterm Evaluation 48 III. BACKGROUND AND DEVELOPMENT HYPOTHESIS There is no officially documented Results Framework for CTB in Indonesia, however the country achievements by objective in Annual Work Plans and the monitoring and evaluation (M&E) framework that are prepared for USAID each year serve as the operational Results Framework. CTB Indonesia’s objectives, sub-objectives, and intended outputs are detailed in the table below. Objectives Sub-Objectives Intervention Areas and Intended Outputs 1. Improved access to quality patient centered care for TB, TB/HIV and MDR-TB services 1. Enabling environment 1.1. Provision of services according to national guidelines for all care providers and risk groups 1.2. Demand side: Community empowered, especially among risk groups 1.3. Demand side: Health seeking behavior improved for types of services 1.4. Provider side: Patient centered approach integrated into routine TB services for all care providers for a supportive environment 2. Comprehensive, high quality diagnostics 2.1. Access to quality TB diagnosis ensured 2.2. EQA network for lab diagnostics & services functioning 2.3. Access to quality culture/DST ensured 2.4. Access, operation and utilization of rapid diagnostics (i.e. Xpert) ensured for priority populations 2.5. Laboratory information management system operational and utilized 2.6. Expedient laboratory specimen transport and results feedback system operational 2.7. Bio-safety measures in laboratories ensured 3. Patient-centered care and treatment 3.1. Ensured intensified case finding for all risk groups by all care providers 3.2. Access to quality treatment and care ensured for TB, DR TB and TB/HIV for all risk groups from all care providers 2. Prevention of transmission and disease progression 4. Targeted screening for active TB 4.1. Contact investigation implemented and monitored 4.2. TB social determinants identified, appropriate interventions designed, implemented and monitored 5. Infection control 5.1. Compliance with quality TB-IC measures in health care, community and congregate settings ensured 5.2. TB surveillance among HCW ensured 6. Management of latent TB infection 6.1. LTBI diagnosis and treatment among high risk groups ensured 3. Strengthened TB platforms 7. Political commitment and leadership 7.1. Endorsed, responsive, prioritized and costed strategic plan available 7.2. In-country political commitment strengthened 7.3. Leadership and management competencies and capacities of NTPs ensured 8. Comprehensive partnerships and informed community involvement 8.1. National partnership and coordinating bodies functioning with appropriate representation and capacity 8.2. Global Fund grant ratings improved 9. Drug and commodity management systems 9.1. Well-functioning procurement and supply chain management system in place 9.2. New and ancillary drug regimens for TB/MDR/LTBI patients available, as appropriate 10. Quality data, surveillance and M&E 10.1. Well-functioning case or patient-based electronic recording and reporting system is in place USAID/Indonesia Challenge TB Project Midterm Evaluation 49 Objectives Sub-Objectives Intervention Areas and Intended Outputs 10.2. Epidemiologic assessments conducted and results incorporated into national strategic plans 11. Human resource development 11.1. Qualified staff available and supportive supervisory systems in place 12. Technical supervision 12.1. Technical supervision IV. INFORMATION SOURCES USAID suggests the following materials for the CTB Midterm Evaluation desk review: 1. CTB Cooperative Agreement and technical program description from the Project Agreement 2. CTBSTEPS Table and AMEP 3. CTB Quarterly and Annual Progress Reports 4. CTB Work Plans 5. CTB external management review 6. Other relevant project document and reports 7. Other CTB technical materials, e.g., case studies, factsheets, infographics, profiles, program briefs, technical briefs, et al. 8. TB CARE Indonesia final report 9. TB National Strategic Plan for Indonesia, 2014 10. TB National Strategic Plan for Indonesia, 2015 (revised) 11. TB JEMM report, 2017 V. EVALUATION PURPOSE, AUDIENCE, AND INTENDED USES Purpose The purpose of this midterm evaluation is to improve the implementation and enhance the impacts of CTB Indonesia. The evaluation will achieve this purpose by assessing the project’s performance, its outcomes, its progress towards intended results, and CTB’s qualifications and position in the GOI’s response to TB, and then recommending ways in which implementation can be enhanced. Audiences and Intended Uses This midterm evaluation is being undertaken to analyze the CTB project performance to date and obtain recommendations on improvements needed for the project to meet its intended purpose. In addition, the evaluation will analyze the value-added to knowledge and skill enhancement and organizational capacity building of local partner institutions. The evaluation conclusions and recommendations are to be used by USAID/Indonesia’s the USAID/Indonesia Health Office and the implementing partner to strengthen project implementation. Findings will enable all project stakeholders, including GOI, to understand CTB’s strengths as well as areas where technical, administrative and management efforts could be improved and strategic approaches refocused. This evaluation will therefore serve to provide information on project performance, program management, information generation and use, and sustainability. More specifically, it aims to: • Provide information on the impacts made by CTB on strengthening the capacity of the local TB system to deliver on the five intervention areas (1) Ensuring universal access by integrating TB in JKN and ensuring secured local budget for TB; (2) increasing case detection; (3) improving quality of treatment and care; (4) expanding diagnostic services; and (5) strengthening surveillance and M&E. • Provide recommendations on where USAID can better invest its efforts to achieve greater impact on TB, particularly in an environment with declining external assistance and increases in domestic financing for health. USAID/Indonesia Challenge TB Project Midterm Evaluation 50 This midterm evaluation will be a performance evaluation as defined in the USAID Evaluation Policy (Annex A). All evaluation materials will be posted to the USAID Development Exchange Clearinghouse (DEC). VI. EVALUATION QUESTIONS To guide this evaluation, USAID has identified key questions (below) and lines of inquiry for each question (Annex B). 1. To what extent has CTB strengthened the local capacity of the TB system to deliver strategic, effective TB programming in the five intervention areas? (Project Performance) 2. Are CTB management arrangements and staffing structure and capacity optimal for achieving project objectives efficiently and effectively? (Program Management) 3. How has the information generated by the project been used to support achievement of objectives and outcomes? (Information Generation and Use) 4. To what extent has CTB been adapting its approach and activities to recent systemic changes in Indonesia’s health system, i.e., the revised Law on Decentralization (Law 23, MSS, etc.) and implementation of National Health Insurance (JKN)? 5. What lessons and best practices from CTB implementation could be replicated in non-CTB districts? 6. To what extent are CTB methodologies, interventions, and management setting the stage for future sustainability of project outputs and outcomes? (Sustainability) VII. GENDER CONSIDERATIONS In accordance with USAID’s Automated Directive System (ADS) 201 point 7, the research design for this evaluation will consider gender-specific and differential effects of CTB Indonesia. The evaluation team will explore gender aspects of the activity per the questions and data sources in Annex C. VIII. DATA COLLECTION AND ANALYSIS METHODS The evaluation team will propose a comprehensive Evaluation Design and Work Plan to address the evaluation questions, including setting criteria for site visits and field data collection. The objective will be to maximize the evaluation team’s ability to develop evidence-findings, conclusions, and recommendations that address the purpose and objectives of this evaluation. Within this Design, the evaluation team will propose the best and most rigorous methods for data collection appropriate to address the evaluation questions. The evaluation team will conduct a Team Preparation Meeting during the first week in country to meet with USAID, discuss the evaluation SOW, and prepare detailed data collection and analysis methods. Annex D provides an illustrative “Illustrative Analytical Framework for the Evaluation” matrix, including a range of potential data collection methods that may be suitable for each evaluation question. This matrix will be further developed by the evaluation team in the Evaluation Design. Data Analysis Methods Data analysis methods to be proposed by the evaluation team will respond to the nature and breadth of each evaluation question and related line of inquiry, as well as the fieldwork schedule and site visit itinerary. Whatever data analysis methods are chosen for this evaluation, they must be justified in terms of their fit with the data collected for each question and the types of answers that USAID seeks. Time and cost considerations will be important in this process. IX. DELIVERABLES The evaluation team will be responsible for the following deliverables. Specific due dates will be proposed in the Evaluation Design, following the TPM. USAID/Indonesia Challenge TB Project Midterm Evaluation 51 Deliverable Estimated Due Date 1. Evaluation Design and Work Plan draft, including detailed research methodology, drafts of data collection instruments, sampling plan, and implementation plan & schedule 1. One week after completing the TPM in Jakarta and associated meetings with USAID and the CTB implementing partner 2. Final Evaluation Design and Work Plan 2. One week after receiving USAID comments 3. Draft Midterm Evaluation Report with draft set of infographics highlighting CTB results 3. Within three weeks after completion of fieldwork 4. Oral presentation(s) to USAID of key findingsand any preliminary conclusions and recommendations. PowerPoint presentation on how USAID programmatic and administrative processes could be improved for future TB activities (e.g., start-up, management, leadership support, and technical intervention design) 4. Within four weeks after completion of fieldwork 5. Final Midterm Evaluation Report and infographics 5. Two weeks after receiving last USAID comments on the Draft Evaluation Report and associated materials X. REPORTING AND DISSEMINATION The format of the evaluation report should follow USAID guidelines set forth in the USAID Evaluation Report Template (http://usaidlearninglab.org/library/evaluation-report-template) and the How-To Note on Preparing Evaluation Reports (http://usaidlearninglab.org/library/how-note-preparing-evaluation-reports). Evaluation team members will be provided with the USAID’s mandatory statement of the evaluation standards they are expected to meet (see Annex A) XI. TEAM COMPOSITION The suggested composition of the CTB Indonesia midterm evaluation team would be four people: • Team Leader/Evaluation Specialist (with TB background), international • TB Expert, Indonesian • Evaluation Specialist, Indonesian • Research Assistant, Indonesia Each Team member will have writing responsibilities for the Evaluation Report draft and final version, per assignments by the Team Leader. All team members should have the following qualifications: • Strong knowledge of Indonesia and the Indonesian health sector, including TB/NTP. • Expertise in program evaluations, including qualitative &quantitative evaluation practices. • Knowledge of USAID programming practices. • For the Team Leader, prior successful experience in leading evaluation or research teams. • Excellent writing and inter-personal communication skills. XII. USAID PARTICIPATION Regular communication between the evaluation team and the designated USAID Activity Manager will be essential to the successful execution of theCTB midterm evaluation. The evaluation team will keep USAID apprised of changes and developments that necessitate any significant decision-making or modification of the approved USAID/Indonesia Challenge TB Project Midterm Evaluation 52 evaluation design. Possible USAID participation in the data collection phase of the evaluation will be discussed in the TPM, prior to the start of fieldwork. XIII. SCHEDULING AND LOGISTICS USAID Indonesia has requested MESP to finalize the SOW and implement this midterm evaluation of CTB, including handling all logistics. The chart below presents an estimated timetable for this task. Estimated CTB Midterm Evaluation Timeline Task/Deliverable May 2017 June July August Sept Evaluation SOW Evaluation Preparation and TPM X Evaluation Design & Work Plan X Desk Review Fieldwork In-Country USAID Debrief(s) Analysis & Report Writing Draft Eval Report &Other Deliverables Final EvalReport& Deliverables XIV. BUDGET MESP will submit to USAID an estimated budget for this midterm evaluation, once the SOW is agreed. USAID/Indonesia Challenge TB Project Midterm Evaluation 53 ANNEX A OF SOW: USAID EVALUATION POLICY APPENDIX I USAID EVALUATION POLICY, APPENDIX 1 CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT • The evaluation report should represent a thoughtful, well-researched and well organized effort to objectively evaluate what worked in the project, what did not and why. • Evaluation reports shall address all evaluation questions included in the scope of work. • The evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline need to be agreed upon in writing by the technical officer. • Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists, and discussion guides will be included in an Annex in the final report. • Evaluation findings will assess outcomes and impact on males and females. • Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). • Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings should be specific, concise and supported by strong quantitative or qualitative evidence. • Sources of information need to be properly identified and listed in an annex. • Recommendations need to be supported by a specific set of findings. • Recommendations should be action-oriented, practical, and specific, with defined responsibility for the action. USAID/Indonesia Challenge TB Project Midterm Evaluation 54 ANNEX B OF SOW: EVALUATION QUESTIONS AND LINES OF INQUIRY Evaluation Question Lines of Inquiry 1. To what extent has CTB strengthened the local capacity of the TB system to deliver strategic, effective TB programming in the five intervention areas? • To what extent have CTB activities strengthened the NTP, and how well have activities coordinated with sub-national level actors and activities? • To what extent has CTB strengthened health services and partners at the provincial and district levels? • To what extent haveCTB activities strengthened capacity of the community approach? • To what extent have CTB activities improved the private sector’s capacity to deliver TB services? • How have CTB activities impacted or changed the relationship between the NTP (central and local government) actors and community actors, and between NTP actors and private sector actors (e.g., creation of PPM organizations)? 2. Are CTB management arrangements and staffing structure and capacity optimal for achieving project objectives efficiently and effectively? • Is CTB’s organizational capacity (structure, systems, strategic capacity) sufficient and effective to deliver program results? • Are internal CTB human resources, strategic interests, monitoring systems, finance systems, and overall structure contributing to the organizational capacity to deliver desired program results (distinguish between CTB and CTB Indonesia)? • To what extent has CTB effectively partnered or utilized its in-country sub￾partners (WHO and FHI360), and how has the prime’s relationship with the sub-partners evolved over the project period in terms of the division of work? • How can CTB’s design, management, and implementation become more efficient, effective, and relevant to achieving program goals? How is the effectiveness and efficiency of the current level of staffing? What is the balance between in-country staff capacity and regional/HQ support to ensure effectiveness and efficiency of project? 3. How has the information generated by the project been used to support achievement of objectives and outcomes? • To what extent is new information being generated by the project? • How are the data and information generated shared between the prime and sub-partners, and with national and local stakeholders, and beyond? • To what extent are the data, information and implementing models generated by CTB and used by the implementing partners and national and local stakeholders being fed back into the system to inform current and future programming by the NTP and provincial and district governments? • What are the mechanisms for disseminating these models and are they sufficient? Have they yielded significant impact in the decentralized Indonesian context? • What knowledge management mechanisms have been put in place to ensure the project is analyzing and addressing gaps in knowledge? USAID/Indonesia Challenge TB Project Midterm Evaluation 55 Evaluation Question Lines of Inquiry 4. To what extent has CTB been adapting its approach and activities to recent systemic changes in Indonesia’s health system, i.e., the revised Law on Decentralization (Law 23, MSS, etc.) and implementation of National Health Insurance (JKN)? 5. What lessons and best practices from CTB implementation could be replicated in non￾CTB districts? 6. To what extent are CTB methodologies, interventions, and management setting the stage for future sustainability of project outputs and outcomes? • What measures or mechanisms have been put in place by CTB to achieve sustainability, and which remaining issues still need to be addressed? How is the sustainability measured by the project? • To what extent did CTB effectively transition activities from TB Care – i.e., which activities were maintained, terminated, or changed? • Are CTB programming and systems congruent with USAID strategic interests? • To what extent is CTB’s portfolio coordinated with USAID/Indonesia’s TB activities implemented by other projects (i.e., Promoting the Quality of Medicines, CEPAT, Linkages TB HIV, and BANTU) to maximize effort and achieve greater results? • How effective and efficient are CTB’s collaboration and coordination with GOI to achieve greater results in implementation of NTP? • How does CTB ensure that Global Fund TB programming under KNCV is complementary to USAID’s TB programming? • How replicable, adoptable, and sustainable are CTB components by Indonesia (i.e., non-external partners)? • How well is CTB aligning its activities with the goals of the GOI and NTP policies and programs? • To what extent is the project evaluating the rationale for USAID to continue or expand investments in the selected geographic locations? USAID/Indonesia Challenge TB Project Midterm Evaluation 56 ANNEX C OF SOW: ILLUSTRATIVE GENDER ASPECTS OF EVALUATION QUESTIONS Evaluation Question Possible Gender Disaggregation Required 1. To what extent has CTB strengthened the local capacity of the TB system to deliver strategic, effective TB programming in the five intervention areas? • Gender disaggregated perception data from different stakeholder groups related to the local capacity of TB system 2. Are CTB management arrangements and staffing structure and capacity optimal for achieving project objectives efficiently and effectively? • Gender disaggregated perception data from different stakeholder groups related CTB management arrangements. 3. How has the information generated by the project been used to support achievement of objectives and outcomes? • Gender disaggregated perception data from different stakeholder groups related to the use of CTB data and information 4. To what extent has CTB been adapting its approach and activities to recent systemic changes in Indonesia’s health system, i.e., the revised Law on Decentralization (Law 23, MSS, etc.) and implementation of National Health Insurance (JKN)? 5. What lessons and best practices from CTB implementation could be replicated in non-CTB districts? • Gender disaggregated perception data from different stakeholder groups related to the use of CTB data and information 6. To what extent are CTB methodologies, interventions, and management setting the stage for future sustainability of project outputs and outcomes? • Gender disaggregated perception data from different stakeholder groups related to suggested program strategy for future CTB activity in Indonesia USAID/Indonesia Challenge TB Project Midterm Evaluation 57 ANNEX D OF SOW: ILLUSTRATIVE ANALYTICAL FRAMEWORK FOR THE EVALUATION: GETTING TO ANSWERS Evaluation Question Data Source Data Collection Methods Sampling or Selection Plan Data Analysis Methods 1. To what extent has CTB strengthened the local capacity of the TB system to deliver strategic, effective TB programming in the five intervention areas? • • STEPS table/M&E plan • Quantitative and Qualitative data from annual report • Primary data (focus group, in depth interview with key stakeholders) • Desk review • Key informant interview (structured interview/semi structured interview/FGD) • Purposive sampling • Quantitative and qualitative analysis 2. Are CTB management arrangements and staffing structure and capacity optimal for achieving project objectives efficiently and effectively? • STEPS table/M&E plan • Quantitative and Qualitative data from annual report • Primary data (focus group, in depth interview with key stakeholders) • Desk review • Key informant interview (structured interview/semi structured interview/FGD) • Purposive sampling • Quantitative and qualitative analysis 3. How has the information generated by the project been used to support achievement of objectives and outcomes? • STEPS table/M&E plan • Quantitative and Qualitative data from annual report • Primary data (focus group, in depth interview with key stakeholders) • Desk review • Key informant interview (structured interview/semi structured interview/FGD) • Purposive sampling • Quantitative and qualitative analysis 4. To what extent has CTB been adapting its approach and activities to recent systemic changes in Indonesia’s health system, i.e., the revised Law on Decentralization (Law 23, MSS, etc.) and implementation of National Health Insurance (JKN)? • STEPS table/M&E plan • Quantitative and Qualitative data from annual report • Primary data (focus group, in depth interview with key stakeholders) • Desk review • Key informant interview (structured interview/semi structured interview/FGD) • Purposive sampling • Quantitative and qualitative analysis USAID/Indonesia Challenge TB Project Midterm Evaluation 58 Evaluation Question Data Source Data Collection Methods Sampling or Selection Plan Data Analysis Methods 5. What lessons and best practices from CTB implementation could be replicated in non-CTB districts? • STEPS table/M&E plan • Quantitative and Qualitative data from annual report • Primary data (focus group, in depth interview with key stakeholders) • Desk review • Key informant interview (structured interview/semi structured interview/FGD) • Purposive sampling • Quantitative and qualitative analysis • 6. To what extent are CTB methodologies, interventions, and management setting the stage for future sustainability of project outputs and outcomes? • STEPS table/M&E plan • Quantitative and Qualitative data from annual report • Primary data (focus group, in depth interview with key stakeholders) • Desk review • Key informant interview (structured interview/semi structured interview/FGD) • Purposive sampling • Quantitative and Qualitative analysis USAID/Indonesia Challenge TB Project Midterm Evaluation 59 ANNEX II: DISTRICT SELECTION CRITERIA District Number of TB service providers Estimation TB cases 2017 # of TB cases notified # of TB patients with HIV status % of TB cases notified % of TB patients with HIV status TB private sector location Selected district 2015 2016 2015 2016 2015 2016 2015 2016 DELI SERDANG DISTRICT 39 12,106 2,991 2,813 268 164 25% 23% 9% 6% ✓ MEDAN CITY 67 14,141 6,518 7,327 849 1,646 46% 52% 13% 22% ✓ ✓ WEST JAKARTA CITY 88 8,564 5,007 5,646 1,230 2,113 58% 66% 25% 37% CENTRAL JAKARTA CITY 35 9,075 4,129 4,183 607 456 45% 46% 15% 11% ✓ SOUTH JAKARTA CITY 92 3,271 3,540 5,297 972 1,197 108% 162% 27% 23% EAST JAKARTA CITY 103 7,553 7,336 8,896 1,262 1,156 97% 118% 17% 13% ✓ ✓ NORTH JAKARTA CITY 55 9,793 2,512 3,747 269 273 26% 38% 11% 7% ✓ ✓ BOGOR DISTRICT 111 19,521 7,451 9,271 296 280 38% 47% 4% 3% ✓ BANDUNG CITY 88 6,471 7,248 8,927 671 780 112% 138% 9% 9% ✓ SURAKARTA CITY 26 1,731 1,755 1,645 654 242 101% 95% 37% 15% ✓ SEMARANG CITY 50 5,806 2,969 3,183 722 814 51% 55% 24% 26% ✓ JEMBER DISTRICT 56 7,397 3,126 3,331 1,267 1,543 42% 45% 41% 46% ✓ ✓ TULUNGAGUNG DISTRICT 36 3,095 817 991 254 434 26% 32% 31% 44% JAYAWIJAYA DISTRICT 11 947 632 511 376 221 67% 54% 59% 43% JAYAPURA CITY 15 1,386 1,010 1,703 487 738 73% 123% 48% 43% ✓ MIMIKA DISTRICT 13 999 988 1,500 452 909 99% 150% 46% 61% ✓ Note: Selected due to program performance Selected due to location of TB Private Sector Study Selected due to different office management USAID/Indonesia Challenge TB Project Midterm Evaluation 60 ANNEX III: DATA COLLECTION INSTRUMENTS Evaluation Question Lines of Inquiry 1. Has CTB strengthened the local capacity of the TB system to deliver strategic, effective TB programming in the five intervention areas? If so, how? If not, why not? What would you recommend to ensure it does during the rest of the project period? • Has the CTB strengthened the NTP? If yes, in what way? If not, how the CTB could improve their contribution in strengthening the NTP? • Has the CTB strengthened the capacity of DHO, PHO and health facilities? If yes, in what aspects? What could be done to improve their contribution? • Has the CTB improved the capacity of the community approach? If not, how the CTB could improve their contribution in improving the capacity of community TB control? • Has CTB activities improved the private sector’s capacity to deliver TB services? If yes, in what aspects? What could be done to improve their contribution? • Are they any impact as result of CTB activities on the type of collaboration between actors in national TB control, eg relationship between the NTP (central, provincial and district), civil society and private sectors? • Are there CTB strategies that target key populations, and capacity building strategies at subnational (provincial) levels? • What is the process of strategies development processes with NTP and other counterparts? 2. Are CTB management arrangements and staffing structure and capacity optimal for achieving project objectives efficiently and effectively? If so, how? Is this optimal? If not, what would you recommend to ensure they do during the rest of the project period? • Is CTB’s organizational capacity (structure, systems, and strategic capacity) sufficient and effective to deliver program results? • Are internal CTB human resources, strategic interests, monitoring systems, finance systems, and overall structure contributing to the organizational capacity to deliver desired program results (distinguish between CTB and CTB Indonesia)? • Has the fund allocation over the CTB objectives and priorities reflects the CTB efficiency in TB control program. • Has the CTB effectively partnered or utilised its country sub partners? How has the prime’s relationship with the sub partners evolved over the project period in terms of the division of work? Are they any measures should be taken to improve the relationships? • Availability of MOUs. Communication plans, agreements? • How can CTB’s design, management, and implementation become more efficient, effective, and relevant to achieving program goals? How is the effectiveness and efficiency of the current level of staffing? What is the balance between in-country staff capacity and regional/HQ support to ensure effectiveness and efficiency of project? USAID/Indonesia Challenge TB Project Midterm Evaluation 61 Evaluation Question Lines of Inquiry • Role of short term technical assistance in supporting the program. Has the TA strategically addressed the most needs for TB control in the catchment areas? • To what extent did CTB effectively transition activities from TB Care – i.e., which activities were maintained, terminated, or changed? 3. Has the information generated by the project been used to support achievement of objectives and outcomes? If so, how? If not, why not? What would you recommend to ensure it is during the rest of the project period? • Has the CTB generated information that benefit to planning, implementing and evaluating TB control (at all level)? If yes, what kind of data they have generated? • How are the data and information generated shared between the prime and sub-partners, and with national and local stakeholders, and beyond? • Are the data, information and implementing models generated by CTB used by the implementing partners and national and local stakeholders being fed back into the system to inform current and future programming by the NTP and provincial and district governments? • What are the mechanisms for disseminating these models and are they sufficient? Are they any impacts in the decentralized Indonesian context? • What knowledge management mechanisms have been put in place to ensure the project is analysing and addressing gaps in knowledge? • Are M&E data fully utilized to inform project planning and support the NTP planning and assessment? • Do M&E staff have the analytical skills needed for data analysis and dissemination? 4. To what extent has CTB been adapting its approach and activities to recent systemic changes in Indonesia’s health system, i.e., the revised Law on Decentralization (Law 23, MSS, etc.) and implementation of National Health Insurance (JKN)? • Has CTB strengthened the DHO capacities to respond to TB prevention and care? If yes, in what way? If not, what could they improve to strengthen the DHO capacities? • Are there CTB strategies that address JKN policies and implementation to cover all TB services? If so, what are they? It not, what you recommend that they do? 5. Are there lessons and best practices from CTB implementation could be replicated in non-CTB districts? If so, what are they? Can they be implemented with no outside technical support? Limited technical support? Or is the same level of support needed to • Are there any impacts of CTB approaches/strategies on the national/subnational programmatic interventions and indicators? • Of the CTB’s strategies, what have been worked and not? What could they do differently for a better result? • How Could CTB the best practices (if any) be sustainable by the NTP resources and management structure? • How CTB is advocating for their best practices and supporting NTP to integrate them in its plans? USAID/Indonesia Challenge TB Project Midterm Evaluation 62 Evaluation Question Lines of Inquiry ensure lesson/best practices are successful? 6. Are CTB methodologies, interventions, and management setting the stage for future sustainability of project outputs and outcomes? If so, how? If not, how would you recommend CTB adjust their methodologies, interventions, and management structures to ensure future sustainability? • What measures or mechanisms have been put in place by CTB to achieve sustainability, and which remaining issues still need to be addressed? How is the sustainability measured by the project? • The level of progress on increasing contributions from domestic resources. • Have CTB’s portfolio coordinated with USAID/Indonesia’s TB activities implemented by other projects (i.e., Promoting the Quality of Medicines, CEPAT, Linkages TB HIV, and BANTU) to maximize effort and achieve greater results? • Is CTB’s collaboration and coordination with GOI effective and efficient? to achieve greater results in implementation of NTP? • Does CTB try to ensure that Global Fund TB programming under KNCV complement USAID’s TB programming? If so, how? If not, what should/could they be doing differently? • Are CTB components replicable, adoptable, and sustainable? If so, how they could they be by Indonesia (i.e., non-external partners)? If not, what should they do differently to ensure they are? • How well is CTB aligning its activities with the goals of the GOI and NTP policies and programs? • To what extent is the project evaluating the rationale for USAID to continue or expand investments in the selected geographic locations? USAID/Indonesia Challenge TB Project Midterm Evaluation 63 ANNEX III A: ANALYTICAL FRAMEWORK – GETTING TO ANSWERS Evaluation Question Illustrative information/variables required to answer the questions Data Source Data Collection Methods Sampling or Selection Plan Data Analysis Methods 1. Has CTB strengthened the local capacity of the TB system to deliver strategic, effective TB programming in the five intervention areas? • • Quality of district action plans developed by CTB TA. • Comparative outcomes between districts with CTB interventions Vs non-CTB districts. • KII respondents who agree that CTB workplans/strategies have been developed with their active participation. • KII respondents who agree that CTB technical assistance is helping them to building their capacities (programmatic, technical and financial) • KII respondents (NGOS and private sector) who agree CTB activities have strengthened their capacities in TB prevention and care. • STEPS table/M&E plan • Quantitative and Qualitative data from annual programmatic and M&E reports, and workplans, • KIIs with TB program managers at national, provincial and districts levels. • KIIs with NGOs, private sector and other USAID partners. • Documents review and analysis • Key informant interviews • Stakeholders questionnaire • Secondary data analysis • Purposive sampling considering: gender, programmatic, geographical, and strategic priorities shared by CTB and USAID • Quantitative and qualitative analysis • Triangulation 2. Are CTB management arrangements and staffing structure and capacity optimal for achieving project objectives efficiently and effectively? • Management structure • Staffing job description • Performance appraisals • Agreements with partners • Lines of reporting and communication among CTB and between CTB and partners • Numbers of STTA, TOR and reports • KIIs with USAID and implementing partners on partnership management • Minutes of meetings with partners. • STEPS table/M&E plan • STTA TORs, reports. • Agreements with partners • KIIs with partners and USAID. • CTB workplans and budget • Documents review and analysis • Key informant interviews • Stakeholders questionnaire • Secondary data analysis • Purposive sampling considering: gender, programmatic, geographical, and strategic priorities shared by CTB and USAID • Quantitative and qualitative analysis • Triangulation USAID/Indonesia Challenge TB Project Midterm Evaluation 64 Evaluation Question Illustrative information/variables required to answer the questions Data Source Data Collection Methods Sampling or Selection Plan Data Analysis Methods • KIIs with partners who agree that they are involved in CTB annual plans, strategies and reports. 3. Has the information generated by the project been used to support achievement of objectives and outcomes? • KIIs with NTP, USAID, local partners who agree that CTB has generated useful information to support achievements of objectives and outcomes. • Sample of information generated and how it is utilized in informing the CTB plans and strategies. • CTB plans and budget line items to generate, disseminate and utilize the information. • KIIs with respondents from NTP, national and district partners (from CTB and non-CTB areas of implementation) on the impact of the information generated by CTB on the outcomes of the TB prevention and care on district levels. • STEPS table/M&E plan • Quantitative and Qualitative data from annual programmatic and M&E reports, and workplans • Primary data from Key Informants (NTP, USAID, Global Fund, partners) • Documents review and analysis • Key informant interviews • Stakeholders questionnaire • Secondary data analysis • Purposive sampling considering: gender, programmatic, geographical, and strategic priorities shared by CTB and USAID • Quantitative and qualitative analysis • Triangulation Q4. To what extent has CTB been adapting its approach and activities to recent systemic changes in Indonesia’s health system, i.e., the revised Law on Decentralization (Law 23, MSS, etc.) • KIIs with NTP (national and district), MOHA, USAID (health office) and partners on impacts of CTB activities to enhance the capacities of the local government. • Number of districts action plans developed with the TA from CTB. • CTB action plans and budgets allocations for these changes to support district capacities. • STEPS table/M&E plan • Quantitative and Qualitative data from annual programmatic and workplans, • Key Informants with USAID (health office), NTP districts, • Documents review and analysis • Key informant interviews • Stakeholders questionnaire • Secondary data analysis • Purposive sampling considering: gender, programmatic, geographical, and strategic priorities shared by CTB and USAID • Quantitative and qualitative analysis • Triangulation USAID/Indonesia Challenge TB Project Midterm Evaluation 65 Evaluation Question Illustrative information/variables required to answer the questions Data Source Data Collection Methods Sampling or Selection Plan Data Analysis Methods and implementation of National Health Insurance (JKN)? • Decentralization and JKN legislations and policies partners and MOHA • Evaluation report on Universal Health Care Coverage. • District action plans 5. Are there lessons and best practices from CTB implementation could be replicated in non￾CTB districts • CTB innovation description and report • KIIs with NTP, provincial health offices and MOHA regarding best practices shared by CTB and how they are integrated in their action plans. • District action plans from CTB and non-CTB assisted districts. • STEPS table/M&E plan • Quantitative and Qualitative data from annual programmatic and workplans, • Innovation reports • District action plans • Primary data from Key Informants (NTP, provincial health offices, and MOHA) • Documents review and analysis • Key informant interviews • Stakeholders questionnaire • Secondary data analysis • Purposive sampling considering: gender, programmatic, geographical, and strategic priorities shared by CTB and USAID • Quantitative and qualitative analysis • Triangulation • 6. Are CTB methodologies, interventions, and management setting the stage for future sustainability of project outputs • KIIs with respondents from NTP on adoptable sustainable CTB components. • KIIs with NTP and implementing partners on the methodologies used by CTB to build and strengthening their capacities (programmatic, technical and financial). • STEPS table/M&E plan • Quantitative and Qualitative data from annual programmatic and workplans, • Primary data from Key Informants • Documents review and analysis • Key informant interviews • Stakeholders questionnaire • Secondary data analysis • Purposive sampling considering: gender, programmatic, geographical, and strategic priorities • Quantitative and Qualitative analysis • Triangulation USAID/Indonesia Challenge TB Project Midterm Evaluation 66 Evaluation Question Illustrative information/variables required to answer the questions Data Source Data Collection Methods Sampling or Selection Plan Data Analysis Methods and outcomes? If so, how? • KIIs with NTP on the role of CTB in assisting the national program in developing the annual workplans and strategies. • NTP technical working group activities (NTP, TB technical group, and IPs) shared by CTB and USAID USAID/Indonesia Challenge TB Project Midterm Evaluation 67 ANNEX III B: KEY INFORMANT INTERVIEW GUIDES KII Guide for Respondents from Implementing Partner To the interviewer: All the questions should be asked in neutral non-leading way. Select the lines of inquiry that are appropriate to the role of the interviewee in relation to the project and the purpose of the interview Date of Interview: Interviewer/s: Interviewee/s: Title/Position(s): Thank you for making the time to meet with us. We are members of the evaluation team who was assigned by USAID to collect information for a mid￾term evaluation of the Challenge TB Project in Indonesia. The purpose of this evaluation is to help the project in enhancing and strengthening its activities to support the National TB Program (NTP) and other local partners and counterparts in the TB prevention and care programs and activities. We are here today to talk about your work/role in relation to this project over the past three years and the type of support you received (are receiving) from the project. You have been identified as a key person to inform this evaluation and we appreciate your views and experiences as one of the key counterparts of the CTB project. This interview shouldn’t take more than 60 minutes and there are no right or wrong answers, so please feel free to say and discuss your opinion freely. All answers will remain confidential; however, you may choose not to answer any questions if you are not comfortable or stop the interview at any time. Any information or examples we gather during this interview will not be attributed to any specific person, unless you tell us that you would be willing to have your responses quoted in the report. Shall I/we begin the interview! Before we begin, do you have any questions about the interview? { } consent provided ________________ Interview initials. For the interviewer (s): The actual questions will be tailored based on the individual interviewed, purpose of the meeting and time available. Each of the six evaluation questions has a list of illustrative ideas/backgrounds (lines of inquiry) to be considered during the interview whenever is appropriate. Lines of inquiries are not sub-questions to be answered, rather used to inform and frame your questions in the appropriate way according to the interviewee background and responsibilities in relation to the CTB. USAID/Indonesia Challenge TB Project Midterm Evaluation 68 Evaluation Question Lines of Inquiry 1. Has CTB strengthened the local capacity of the TB system to deliver strategic, effective TB programming in the five intervention areas? • • To what extent have CTB activities strengthened the NTP, and how well have activities coordinated with sub-national level actors and activities? • To what extent has CTB strengthened health services and partners at the provincial and district levels? • To what extent have CTB activities strengthened capacity of the community approach? • To what extent have CTB activities improved the private sector’s capacity to deliver TB services? • How have CTB activities impacted or changed the relationship between the NTP (central and local government) actors and community actors, and between NTP actors and private sector actors (e.g., creation of PPM organizations)? • Are there CTB strategies that target key populations, and capacity building strategies at subnational (provincial) levels? • What is the process of strategies development processes with NTP and other counterparts? 2. Are CTB management arrangements and staffing structure and capacity optimal for achieving project objectives efficiently and effectively? • Is CTB’s organizational capacity (structure, systems, and strategic capacity) sufficient and effective to deliver program results? • Are internal CTB human resources, strategic interests, monitoring systems, finance systems, and overall structure contributing to the organizational capacity to deliver desired program results (distinguish between CTB and CTB Indonesia)? • To what extent the fund allocation over the CTB objectives and priorities reflects the CTB efficiency in TB control program. • To what extent has CTB effectively partnered or utilized its in-country sub-partners (WHO and FHI 360), and how has the prime’s relationship with the sub-partners evolved over the project period in terms of the division of work? Availability of MOUs. Communication plans, agreements? • How can CTB’s design, management, and implementation become more efficient, effective, and relevant to achieving program goals? How is the effectiveness and efficiency of the current level of staffing? What is the balance between in-country staff capacity and regional/HQ support to ensure effectiveness and efficiency of project? • Role of short term technical assistance in supporting the program. • To what extent did CTB effectively transition activities from TB Care – i.e., which activities were maintained, terminated, or changed? 3. Has the information generated by the project been used to support achievement of objectives and outcomes? • To what extent is new information being generated by the project? • How are the data and information generated shared between the prime and sub-partners, and with national and local stakeholders, and beyond? • To what extent are the data, information and implementing models generated by CTB and used by the implementing partners and national and local stakeholders being fed back into the system to inform current and future programming by the NTP and provincial and district governments? USAID/Indonesia Challenge TB Project Midterm Evaluation 69 Evaluation Question Lines of Inquiry • What are the mechanisms for disseminating these models and are they sufficient? Have they yielded significant impact in the decentralized Indonesian context? • What knowledge management mechanisms have been put in place to ensure the project is analysing and addressing gaps in knowledge? • Are M&E data fully utilized to inform project planning and support the NTP planning and assessment? • Do M&E staff have the analytical skills needed for data analysis and dissemination? 4. To what extent has CTB been adapting its approach and activities to recent systemic changes in Indonesia’s health system, i.e., the revised Law on Decentralization (Law 23, MSS, etc.) and implementation of National Health Insurance (JKN)? • To what extent have CTB activities strengthened the district government capacities to respond to TB prevention and care? • Are there CTB strategies that address JKN policies and implementation to cover all TB services? 5. Are there lessons and best practices from CTB implementation could be replicated in non-CTB districts? • What are the impacts of these best practices and lesson learned on the national/subnational programmatic interventions and indicators? • Could these best practices be sustainable by the NTP resources and management structure? • How CTB is advocating for these practices and supporting NTP to integrate them in its plans? • 6. Are CTB methodologies, interventions, and management setting the stage for future sustainability of project outputs and outcomes? • What measures or mechanisms have been put in place by CTB to achieve sustainability, and which remaining issues still need to be addressed? How is the sustainability measured by the project? • The level of progress on increasing contributions from domestic resources. • To what extent is CTB’s portfolio coordinated with USAID/Indonesia’s TB activities implemented by other projects (i.e., Promoting the Quality of Medicines, CEPAT, Linkages TB HIV, and BANTU) to maximize effort and achieve greater results? • How effective and efficient are CTB’s collaboration and coordination with GOI to achieve greater results in implementation of NTP? • How does CTB ensure that Global Fund TB programming under KNCV is complementary to USAID’s TB programming? • How replicable, adoptable, and sustainable are CTB components by Indonesia (i.e., non-external partners)? • How well is CTB aligning its activities with the goals of the GOI and NTP policies and programs? • To what extent is the project evaluating the rationale for USAID to continue or expand investments in the selected geographic locations? USAID/Indonesia Challenge TB Project Midterm Evaluation 70 KII Guide for Respondents from NTP Program Managers at National, Provincial, and District levels To the interviewer: All the questions should be asked in neutral non-leading way. Select the lines of inquiry that are appropriate to the role of the interviewee in relation to the project and the purpose of the interview Date of Interview: Interviewer/s: Interviewee/s: Title/Position(s): Thank you for making the time to meet with us. We are members of the evaluation team who was assigned by USAID to collect information for a mid￾term evaluation of the Challenge TB Project in Indonesia. The purpose of this evaluation is to help the project in enhancing and strengthening its activities to support the National TB Program (NTP) and other local partners and counterparts in the TB prevention and care programs and activities. We are here today to talk about your work/role in relation to this project over the past three years and the type of support you received (are receiving) from the project. You have been identified as a key person to inform this evaluation and we appreciate your views and experiences as one of the key counterparts of the CTB project. This interview shouldn’t take more than 60 minutes and there are no right or wrong answers, so please feel free to say and discuss your opinion freely. All answers will remain confidential; however, you may choose not to answer any questions if you are not comfortable or stop the interview at any time. Any information or examples we gather during this interview will not be attributed to any specific person, unless you tell us that you would be willing to have your responses quoted in the report. Shall I/we begin the interview! Before we begin, do you have any questions about the interview? { } consent provided ________________ Interview initials. For the interviewer (s): The actual questions will be tailored based on the individual interviewed, purpose of the meeting and time available. Each of the six evaluation questions has a list of illustrative ideas/backgrounds (lines of inquiry) to be considered during the interview whenever is appropriate. Lines of inquiries are not sub-questions to be answered, rather used to inform and frame your questions in the appropriate way according to the interviewee background and responsibilities in relation to the CTB. Evaluation Question Lines of Inquiry 1. Has CTB strengthened the local capacity of the TB system to deliver strategic, effective TB • To what extent have CTB activities strengthened the NTP, and how well have activities coordinated with sub-national level actors and activities? USAID/Indonesia Challenge TB Project Midterm Evaluation 71 Evaluation Question Lines of Inquiry programming in the five intervention areas? • • To what extent has CTB strengthened health services and partners at the provincial and district levels? • How have CTB activities impacted or changed the relationship between the NTP (central and local government) actors and community actors, and between NTP actors and private sector actors (e.g., creation of PPM organizations)? • Are there CTB strategies that target key populations, and capacity building strategies at subnational (provincial) levels? • What is the process of strategies development processes with NTP and other counterparts? 3. Has the information generated by the project been used to support achievement of objectives and outcomes? • To what extent is new information being generated by the project? • To what extent are the data, information and implementing models generated by CTB and used by the implementing partners and national and local stakeholders being fed back into the system to inform current and future programming by the NTP and provincial and district governments? • Are M&E data fully utilized to inform project planning and support the NTP planning and assessment? 4. To what extent has CTB been adapting its approach and activities to recent systemic changes in Indonesia’s health system, i.e., the revised Law on Decentralization (Law 23, MSS, etc.) and implementation of National Health Insurance (JKN)? • To what extent have CTB activities strengthened the district government capacities to respond to TB prevention and care? • Are there CTB strategies that address JKN policies and implementation to cover all TB services? 5. Are there lessons and best practices from CTB implementation could be replicated in non-CTB districts? • What are the impacts of these best practices and lesson learned on the national/subnational programmatic interventions and indicators? • Could these best practices be sustainable by the NTP resources and management structure? • 6. Are CTB methodologies, interventions, and management setting the stage for future sustainability of project outputs and outcomes? • The level of progress on increasing contributions from domestic resources. • How effective and efficient are CTB’s collaboration and coordination with GOI to achieve greater results in implementation of NTP? • How replicable, adoptable, and sustainable are CTB components by Indonesia (i.e., non-external partners)? • How well is CTB aligning its activities with the goals of the GOI and NTP policies and programs? USAID/Indonesia Challenge TB Project Midterm Evaluation 72 KII Guide for Respondents from Community or Private Organizations To the interviewer: All the questions should be asked in neutral non-leading way. Select the lines of inquiry that are appropriate to the role of the interviewee in relation to the project and the purpose of the interview Date of Interview: Interviewer/s: Interviewee/s: Title/Position(s): Thank you for making the time to meet with us. We are members of the evaluation team who was assigned by USAID to collect information for a mid￾term evaluation of the Challenge TB Project in Indonesia. The purpose of this evaluation is to help the project in enhancing and strengthening its activities to support the National TB Program (NTP) and other local partners and counterparts in the TB prevention and care programs and activities. We are here today to talk about your work/role in relation to this project over the past three years and the type of support you received (are receiving) from the project. You have been identified as a key person to inform this evaluation and we appreciate your views and experiences as one of the key counterparts of the CTB project. This interview shouldn’t take more than 60 minutes and there are no right or wrong answers, so please feel free to say and discuss your opinion freely. All answers will remain confidential; however, you may choose not to answer any questions if you are not comfortable or stop the interview at any time. Any information or examples we gather during this interview will not be attributed to any specific person, unless you tell us that you would be willing to have your responses quoted in the report. Shall I/we begin the interview! Before we begin, do you have any questions about the interview? { } consent provided ________________ Interview initials. For the interviewer (s): The actual questions will be tailored based on the individual interviewed, purpose of the meeting and time available. Each of the six evaluation questions has a list of illustrative ideas/backgrounds (lines of inquiry) to be considered during the interview whenever is appropriate. Lines of inquiries are not sub-questions to be answered, rather used to inform and frame your questions in the appropriate way according to the interviewee background and responsibilities in relation to the CTB. Evaluation Question Lines of Inquiry 1. Has the CTB strengthened the local capacity of the TB system to deliver strategic, effective TB • To what extent has CTB strengthened health services and partners at the provincial and district levels? • How have CTB activities impacted or changed the relationship between the NTP (central and local government) actors and community actors, USAID/Indonesia Challenge TB Project Midterm Evaluation 73 Evaluation Question Lines of Inquiry programming in the five intervention areas? • and between NTP actors and private sector actors (e.g., creation of PPM organizations)? 3. Has the information generated by the project been used to support achievement of objectives and outcomes? • To what extent is new information being generated by the project? • How are the data and information generated shared between the prime and sub-partners, and with national and local stakeholders, and beyond? • To what extent are the data, information and implementing models generated by CTB and used by the implementing partners and national and local stakeholders being fed back into the system to inform current and future programming by the NTP and provincial and district governments? • What are the mechanisms for disseminating these models and are they sufficient? Have they yielded significant impact in the decentralized Indonesian context? 4. To what extent has CTB been adapting its approach and activities to implementation of National Health Insurance (JKN)? • Are there CTB strategies that address JKN policies and implementation to cover all TB services? • 6. Are CTB methodologies, interventions, and management setting the stage for future sustainability of project outputs and outcomes? • How replicable, adoptable, and sustainable are CTB components by Indonesia (i.e., non-external partners)? • How well is CTB aligning its activities with the goals of the GOI and NTP policies and programs? USAID/Indonesia Challenge TB Project Midterm Evaluation 74 KII Guide for Respondents from USAID Partners for Related TB Projects To the interviewer: All the questions should be asked in neutral non-leading way. Select the lines of inquiry that are appropriate to the role of the interviewee in relation to the project and the purpose of the interview Date of Interview: Interviewer/s: Interviewee/s: Title/Position(s): Thank you for making the time to meet with us. We are members of the evaluation team who was assigned by USAID to collect information for a mid￾term evaluation of the Challenge TB Project in Indonesia. The purpose of this evaluation is to help the project in enhancing and strengthening its activities to support the National TB Program (NTP) and other local partners and counterparts in the TB prevention and care programs and activities. We are here today to talk about your work/role in relation to this project over the past three years and the type of support you received (are receiving) from the project. You have been identified as a key person to inform this evaluation and we appreciate your views and experiences as one of the key counterparts of the CTB project. This interview shouldn’t take more than 60 minutes and there are no right or wrong answers, so please feel free to say and discuss your opinion freely. All answers will remain confidential; however, you may choose not to answer any questions if you are not comfortable or stop the interview at any time. Any information or examples we gather during this interview will not be attributed to any specific person, unless you tell us that you would be willing ot have your responses quoted in the report. Shall I/we begin the interview! Before we begin, do you have any questions about the interview? { } consent provided ________________ Interview initials. For the interviewer (s): The actual questions will be tailored based on the individual interviewed, purpose of the meeting and time available. Each of the six evaluation questions has a list of illustrative ideas/backgrounds (lines of inquiry) to be considered during the interview whenever is appropriate. Lines of inquiries are not sub-questions to be answered, rather used to inform and frame your questions in the appropriate way according to the interviewee background and responsibilities in relation to the CTB. Evaluation Question Lines of Inquiry 1. Has CTB strengthened the local capacity of the TB system to deliver strategic, effective TB programming in the five intervention areas? • • What is the process of strategies development processes with NTP and other counterparts? USAID/Indonesia Challenge TB Project Midterm Evaluation 75 Evaluation Question Lines of Inquiry 3. Has the information generated by the project been used to support achievement of objectives and outcomes? • To what extent is new information being generated by the project? • How are the data and information generated shared between the prime and sub-partners, and with national and local stakeholders, and beyond? 4. To what extent has CTB been adapting its approach and activities to recent systemic changes in Indonesia’s health system, i.e., the revised Law on Decentralization (Law 23, MSS, etc.) and implementation of National Health Insurance (JKN)? • To what extent have CTB activities strengthened the district government capacities to respond to TB prevention and care? • Are there CTB strategies that address JKN policies and implementation to cover all TB services? • 6. Are CTB methodologies, interventions, and management setting the stage for future sustainability of project outputs and outcomes? • To what extent is CTB’s portfolio coordinated with USAID/Indonesia’s TB activities implemented by other projects (i.e., Promoting the Quality of Medicines, CEPAT, Linkages TB HIV, and BANTU) to maximize effort and achieve greater results? • How replicable, adoptable, and sustainable are CTB components by Indonesia (i.e., non-external partners)? • How well is CTB aligning its activities with the goals of the GOI and NTP policies and programs? USAID/Indonesia Challenge TB Project Midterm Evaluation 76 KII Guide for Respondents from GOI Agencies (MOHA, MOLHR) To the interviewer: All the questions should be asked in neutral non-leading way. Select the lines of inquiry that are appropriate to the role of the interviewee in relation to the project and the purpose of the interview Date of Interview: Interviewer/s: Interviewee/s: Title/Position(s): Thank you for making the time to meet with us. We are members of the evaluation team who was assigned by USAID to collect information for a mid￾term evaluation of the Challenge TB Project in Indonesia. The purpose of this evaluation is to help the project in enhancing and strengthening its activities to support the National TB Program (NTP) and other local partners and counterparts in the TB prevention and care programs and activities. We are here today to talk about your work/role in relation to this project over the past three years and the type of support you received (are receiving) from the project. You have been identified as a key person to inform this evaluation and we appreciate your views and experiences as one of the key counterparts of the CTB project. This interview shouldn’t take more than 60 minutes and there are no right or wrong answers, so please feel free to say and discuss your opinion freely. All answers will remain confidential; however, you may choose not to answer any questions if you are not comfortable or stop the interview at any time. Any information or examples we gather during this interview will not be attributed to any specific person, unless you tell us that you would be willing or have your responses quoted in the report. Shall I/we begin the interview! Before we begin, do you have any questions about the interview? { } consent provided ________________ Interview initials. For the interviewer (s): The actual questions will be tailored based on the individual interviewed, purpose of the meeting and time available. Each of the six evaluation questions has a list of illustrative ideas/backgrounds (lines of inquiry) to be considered during the interview whenever is appropriate. Lines of inquiries are not sub-questions to be answered, rather used to inform and frame your questions in the appropriate way according to the interviewee background and responsibilities in relation to the CTB. Evaluation Question Lines of Inquiry 1. Has the information generated by the project been used to support achievement of objectives and outcomes? • To what extent is new information being generated by the project? • How are the data and information generated shared between the prime and sub-partners, and with national and local stakeholders, and beyond? USAID/Indonesia Challenge TB Project Midterm Evaluation 77 Evaluation Question Lines of Inquiry • Are the information generated in usable format that can be used for its intended purpose? • To what extent are the data, information and implementing models generated by CTB and used by the implementing partners and national and local stakeholders being fed back into the system to inform current and future programming by the NTP and provincial and district governments? • What are the mechanisms for disseminating these models and are they sufficient? Have they yielded significant impact in the decentralized Indonesian context? 2. To what extent has CTB been adapting its approach and activities to recent systemic changes in Indonesia’s health system, i.e., the revised Law on Decentralization (Law 23, MSS, etc.) and implementation of National Health Insurance (JKN)? • Have the CTB activities strengthened the district government capacities to respond to TB prevention and care? • Are there CTB strategies that address JKN policies and implementation to cover all TB services? • 3. Are CTB methodologies, interventions, and management setting the stage for future sustainability of project outputs and outcomes? • How replicable, adoptable, and sustainable are CTB components by Indonesia (i.e., non-external partners)? • How well is CTB aligning its activities with the goals of the GOI and NTP policies and programs? USAID/Indonesia Challenge TB Project Midterm Evaluation 78 ANNEX III C: QUESTIONNAIRE TO STAKEHOLDERS Thank you for making the time to meet with us. This questionnaire is part of a mid-term evaluation that USAID Indonesia is conducting to the Challenge TB Project in the country to assess the efficiency and effectiveness of the project in supporting the National TB Program and its partners in the area of TB prevention and care to achieve the country strategic goals in this field. Your participation in completing this questionnaire is completely voluntary. Your participation will contribute to the evaluation team ability to provide useful input to USAID and CTB project in Indonesia. We expect that completing the questionnaire shouldn’t take more than 15-20 minutes. Responses will be kept confidential; unless you tell us that you would be willing or have your responses quoted in the report. Please complete the questionnaire and email it back to fitriyani.yasir@gmail.com or retno.handini@msiworldwide.com. If you have any questions or having trouble in access the survey, please contact Fitri or Dini through email. Q1 What is the organization/program you are working for? Q2 What is the working relationship of your organization with CTB? ( ) International Partner ( ) Local Partner ( ) Collaborator/International Organization ( ) Implementing Organization ( ) Donor (provide fund to CTB) ( ) Other (please specify) Q3 To what extent do you agree with the following statements – we appreciate your comments on any of your selections. No Statement Strongly Disagree Do Not Agree Neutral Agree Strongly Agree N/A or Don’t Know 1 CTB’s organizational capacity (structure, systems, management, strategic capacity) is sufficient and effective to fulfill the CTB objectives 2 CTB activities are implemented as per the workplans without delays USAID/Indonesia Challenge TB Project Midterm Evaluation 79 No Statement Strongly Disagree Do Not Agree Neutral Agree Strongly Agree N/A or Don’t Know 3 CTB workplan activities strengthened the NTP capacities 4 CTB project components are replicable and adaptable by Indonesian organizations working in the fields of TB prevention and care. 5 CTB has a doable sustainability strategy and indicators which are regularly measured by the project. 6 CTB workplans and strategies are responding effectively to the changes in the government structures and policies. 7 CTB activities are well coordinated with sub￾national level actors and activities. 8 CTB activities are focused on increase domestic resources (human resources, systems and funds) and sustainability 9 CTB workplan activities are aligned with national strategic goals and objectives for TB prevention and control 10 CTB is relying on short term international technical assistance to deliver the workplan activities 11 The private sector has been supportive and involved as planned 12 CTB activities improved the private sector’s capacity to deliver TB services 13 New information and implementing models to improve TB program being generated by CTB and adopted by NTP USAID/Indonesia Challenge TB Project Midterm Evaluation 80 No Statement Strongly Disagree Do Not Agree Neutral Agree Strongly Agree N/A or Don’t Know 14 Fund allocation over the CTB objectives and activities have reflected the CTB efficiency in TB control program. 15 CTB has effectively partnered or utilized its in￾country sub-partners (WHO and FHI 360) 16 Short term technical assistance provides significant contribution in supporting the program USAID/Indonesia Challenge TB Project Midterm Evaluation 81 ANNEX III D: FOCUS GROUP DISCUSSION GUIDE We are here to conduct a midterm evaluation of Challenge TB Project to help it in enhancing and strengthening the National TB Program (NTP) and other local partners and counterparts in the TB prevention and care programs and activities. We are here today to talk about your work/role in relation to this project over the past three years and the type of support you received (are receiving) from the project. This group discussion should not take more than 45-60 minutes and there is no right or wrong answers, so please feel free to say and discuss your opinion freely. All l answers will remain confidential; however, you may choose not to answer any questions if you are not comfortable. We hope you will, as your responses will assist the project in improving its activities to support the National TB Program. May be begin? Start with introducing yourself and the group. Interviewer/s: ________________ Date: ___________________ Location: ____________________ Participants involved in the discussion: 1. To what extent has CTB strengthened the local capacity of the TB system to deliver strategic, effective TB programming in the five intervention areas? Prompt to discuss to what extent have CTB activities strengthened the NTP, the health services and partners at the provincial and district levels, and the capacity of the community approach? 2. Are CTB management arrangements and staffing structure and capacity optimal for achieving project objectives efficiently and effectively? Prompt to discuss the efficiency of the CTB’s organizational capacity, human resources, partnerships and strategic interests in delivering the program results? 3. How has the information generated by the project been used to support achievement of objectives and outcomes? Elaborate to discuss how are the data and information generated shared between the prime and sub-partners, and with national and local stakeholders, and beyond? 4. To what extent has CTB been adapting its approach and activities to recent systemic changes in Indonesia’s health system, i.e., the revised Law on Decentralization (Law 23, MSS, etc.) and implementation of National Health Insurance (JKN)? Elaborate to discuss to what extent have CTB activities strengthened the district government capacities to respond to TB prevention and care. 5. What lessons and best practices from CTB implementation could be replicated in non-CTB districts? Elaborate to discuss what are the impacts of these best practices and lesson learned on the national/subnational programmatic interventions and indicators? 6. To what extent are CTB methodologies, interventions, and management setting the stage for future sustainability of project outputs and outcomes. Elaborate to discuss What measures or mechanisms have been put in place by CTB to achieve sustainability, and which remaining issues still need to be addressed? How is the sustainability measured by the project? USAID/Indonesia Challenge TB Project Midterm Evaluation 82 ANNEX IV: SOURCES OF INFORMATION USAID/Indonesia Challenge TB Project Midterm Evaluation 83 ANNEX IV A: DETAILED FIELD SCHEDULE Date and Time Institution Stakeholders/Informants Monday, 2 October 2017 10.00-11.30 USAID Indonesia - Alia Hartanti - Adi S 13.00-16.00 KNCV - Chief of Party - Technical Assistance Coordinator - Monitoring and Evaluation Coordinator Tuesday, 3 October 2017 07.30-09.00 National TB Program (NTP), Ministry of Health Manager 09.00-10.00 Global Fund TB - Project Management Coordinator - M&E Staff 13.30-14.30 WHO TB Advisor 20.00-21.00 USAID Washington Senior TB Technical Officer Wednesday, 4 October 2017 09.30-10.30 Ministry of Law and Human Right Head of Section of Mental and Palliative Directorate General of Correction Affair 13.00-14.00 Ministry of Home Affairs Head of Section of Area II, Sub Directorate of Health 13.00-15.00 NGO: 1. LKNU (TB Cepat) 2. Aisiyah 1. Chief of Party 2. Advocacy, Communication, and Social Mobilization (ACSM) Specialist USAID/Indonesia Challenge TB Project Midterm Evaluation 84 Date and Time Institution Stakeholders/Informants Week 1 - Team 1 Jakarta: North Jakarta City & Central Jakarta City Thrusday, 5 October 2017 - North Jakarta City 08.00-09.00 North Jakarta City Health Office (DHO) - Head of Section of Communicable Disease - TB Supervisor 10.00-11.00 Kepala Gading Subdistrict Primary Health Care (Puskesmas) TB Coordinator 11.30-12.30 Sulianti Saroso Hospital TB Technical Coordinator Friday, 6 October 2017 - Central Jakarta City 08.00-09.00 Jakarta Province Health Office (PHO) Head of Section of Communicable Disease 09.00-10.00 Jakarta Province CTB Team Jakarta Provincial Coordinator Officer 10.30-11.30 Central Jakarta City Health Office(DHO) - Head of Section of Communicable Disease - TB Supervisor 13.00-14.00 Senen Primary Health Care (Puskesmas) TB Coordinator USAID/Indonesia Challenge TB Project Midterm Evaluation 85 Date and Time Institution Stakeholders/Informants Week 1 - Team 2 East Jakarta (Jakarta) and Bogor District (West Java) Thrusday, 5 October 2017 - East Jakarta City 08.00-09.00 East Jakarta City Health Office (DHO) - Head of Section of Communicable Disease - TB Supervisor 10.00-11.00 Keramat Jati Primary Health Care (Puskesmas) TB Coordinator 12.00-13.00 Persahabatan Hospital TB Coordinator Friday, 6 October 2017 – Bogor District 08.00-09.00 Bogor District Health Office - Head of Section of Communicable Disease - TB Supervisor 09.00-09.50 CTB Bogor Team - Technical Officer - Data Officer/M&E Officer - Project Assistant 10.00-11.00 Cibinong Local Government Hospital (RSUD) Head of Medical Unit 11.00-12.00 Bogor Health Laboraturium (BLK) Head of Laboraturium USAID/Indonesia Challenge TB Project Midterm Evaluation 86 Date and Time Institution Stakeholders/Informants Week 2 - Team 1 South Sumatera: Medan City and Deli Serdang District Monday, 9 October 2017 08.00-09.00 North Sumatera Provincial Health Office (PHO) - Head of Communicable Disease - Head of Section of Communicable Disease - Staff of TB Program 09.15-10.15 South Sumatera CTB Provincial Team - Provincial Coordinator Officer - Senior Technical Officer - Data Officer/M&E Officer 10.30-11.30 Medan City Health Office (DHO) TB Supervisor 13.00-14.00 Medan Correctional Facility Class II A for Woman (Prison) Doctor of Facility 14.30-15.30 Deli Serdang District CTB Team Technical Officer Tuesday, 10 October 2017 09.00-10.00 Deli Serdang District Health Office (DHO) - Head of Communicable Disease - Head of Section of Communicable Disease - TB Supervisor 10.15-11.15 Lubuk Pakam Primary Health Care (Puskesmas) - TB Coordinator - Staff of TB Program 14.00-15.00 JKM Cepat (NGO) Director 15.30-16.30 Aviati Private Clinic - Doctor of Clinic - Nurse USAID/Indonesia Challenge TB Project Midterm Evaluation 87 Date and Time Institution Stakeholders/Informants Week 2 - Team 2 Jember District (East Java) and Bandung City (West Java) Monday, 9 October 2017 – Surabaya City 08.00-09.00 East Java Provincial Health Office (PHO) - Head of Communicable Disease - TB Supervisor 09.00-10.00 East Java CTB Provincial Team - Provincial Coordinator Officer - Technical Officer - Monitoring and Evaluation Officer - Office Staff 10.30-11.30 BBLK Surabaya (Laboraturium) Head of Section of Technical Guidance Tuesday, 10 October 2017 – Jember District 08.00-09.00 Jember District Health Office (DHO) Head of Communicable Disease 09.00-10.00 CTB District Team - Technical Officer Jember Distric - Data Officer/M&E Officer - Project Assistant 11.00-12.00 Jember Lung Hospital - Head of TB Program - Head of Section of Reasearch and Development - Promotion Coordinator 13.00-14.00 Sekawan (Patient Organization) Secretary Wednesday, 11 October 2017 – Surabaya City 07.30-08.00 Additional interview with PHO and CTB Team - PHO TB Coordinator - Senior Tehnical Officer of East Java CTB Team - Lumajang District Technical Officer Thursday, 12 October 2017 – Bandung City 08.00-09.00 Bandung City Health Office (DHO) Head of Section of Communicable Diasease 09.15-10.15 West Java Provincial CTB Team Provincial Coordinator Officer 10.15-11.15 West Java Provincial Health Office (PHO) TB Supervisor 11.30-13.00 Garuda Primary Health Care (Puskesmas) - Head of Puskesmas - Nurse 13.30-14.30 Yayasan Pelita (NGO) - Manager of Health Division - Staff USAID/Indonesia Challenge TB Project Midterm Evaluation 88 Date and Time Institution Stakeholders/Informants Week 3 - Team 1 Central Java: Semarang City and Surakarta City Monday, 16 October 2017 – Semarang City 08.00-09.00 Central Java Provincial Health Office (PHO) - Head of Communicable Diasease - Staff 09.15-10.15 Central Java CTB Team - Provincial Coordinator Officer - Senior Technical Officer 10.30-11.30 Semarang City Health Office (DHO) - Head of Communicable Disease - Head of Section of Communicable Diasease - TB Supervisor 13.00-14.00 Semarang Correctional Facility Class I Kedungpane (Prison) Doctor of facility Tuesday, 17 October 2017 – Surakarta City 09.30-10.30 Technical Officer Solo City 10.30-11.30 Semar (Patient Organization) - Head of Organization - Deputy 13.00-14.00 Surakarta City Health Office (DHO) - Head of Communicable Diasease - Head of Section of Communicable Diasease - TB Supervisor 14.30-15.30 Kasih Ibu Private Hospital - Deputi of Medical Unit - Clinical Pathologyst - Head of Micro Laboraturium - Pulmonologyst - DOTS Nurse Wednesday, 18 October 2017 – Surakarta City 08.00-09.00 Purwosari Primary Health Care (Puskesmas) - Head of Puskesmas - TB Coordinator USAID/Indonesia Challenge TB Project Midterm Evaluation 89 Date and Time Institution Stakeholders/Informants Week 3 - Team 2 Papua: Jayapura City and Mimika District Monday, 16 October 2017 – Jayapura City 08.00-09.00 Papua Provincial Health Office (PHO) Head of Section of Communicable Disease 09.30-10.30 Jayapura District Hospital (RSUD) - Head of TB-HIV Workgroup - Head of TB and VCT 11.00-12.00 Jayapura Health Laboraturium Head of Laboraturium 14.00-15.00 Jayapura District Health Office (DHO) Head of Communicable Disease 15.30-16.30 Linkages FHI360 and CTB Team - Senior Technical Officer Linkages - Data Officer/M&E Officer Wednesday, 18 October 2017 – Mimika District 08.00-09.00 Mimika District Health Office (DHO) - Head of Communicable Diasease - Head of Section of Communicable Diasease - TB Supervisor 09.00-10.00 Mimika CTB Team - Technical Officer - Data Officer/M&E Officer 10.30-11.30 Timika Jaya Primary Health Care (Puskesmas) Staff of Laboraturium 13.00-14.00 RSUD Mimika (Hospital) TB Coordinator USAID/Indonesia Challenge TB Project Midterm Evaluation 90 Date and Time Institution Stakeholders/Informants Jakarta Friday, 23 October 2017 14.00-16.30 FHI 360 - Project Coordinator - Senior Technical Officer - Technical Officer Monday, 23 October 2017 10.00-12.00 KNCV - COP - SIKM coordinator - TA coordinator - Project directors - Project coordinators USAID/Indonesia Challenge TB Project Midterm Evaluation 91 ANNEX IV B: PERSONS INTERVIEWED This annex lists the names, positions and organizations of all key informants who participated in individual and small group interviews conducted as part of the Challenge TB Midterm Evaluation. Name and Position Organization Email CTB Midterm Evaluation Team Soliman Guirgis Team leader sfarah2008@gmail.com Pandu Riono TB Expert pandu.riono@gmail.com Aang Sutrisna Evaluation Specialist aang.sutrisna@gmail.com Fitriyani Yasir Research Assistant fitriyani.yasir@gmail.com USAID Nicholas Z Enrich USAID Washington nenrich@usaid.gov Willian Wells, Senior TB Technical Officer USAID Washington wwells@usaid.gov Kaiser Shen USAID Washington kshen@usaid.gov Ellen Halbach, Global Health USAID Washington khalbach@usaid.gov Jonathan Ross, USAID Indonesia jross@usaid.gov Zohra Balsara, Deputy Health Officer USAID Indonesia zbalsara@usaid.gov Aliya Hartanti USAID Indonesia ahartanti@usaid.gov Adi Sarininggar USAID Indonesia Adi.sarininggar@usaid.gov Sutan Royansyah, Monitoring and Evaluation USAID Indonesia sroyansyah@usaid.gov Monitoring and Support Program (MESP) USAID Jonathan Simon, Chief of Party MESP USAID jonathan.simon@msiworldwide.com Retno Handini, Senior M&E Advisor MESP USAID retno.Handini@msiworldwide.com USAID/Indonesia Challenge TB Project Midterm Evaluation 92 Name and Position Organization Email NATIONAL LEVEL INTERVIEWS Government of Indonesia – National Level Asik Surya, Manager National TB Program (NTP), Ministry of Health asiksurya@yahoo.com Regina Tambunan, Staff of Monitoring and Evaluation Global Fund TB Regina_gina@yahoo.com Dina Frasasti, Technical Officer of PMDT Global Fund TB dinafrasasti@gmail.com Budiarti Setiyaningsih Global Fund TB Budiarti_setiyaningsih@yahoo.co.id Triya N Dinihari Sub Directorate of HIV/AIDS, Ministry of Health tndinihari@yahoo.com Zamhir Islamie, S.Sos, MPA, Head of Section of Area II, Sub Directorate of Health Ministry of Home Affairs Zamhir.islamie@gmail.com Ummu Salamah, Head of Section of Mental and Palliative Directorate General of Correction Affair Ministry of Law and Regulation arsanirosnani@gmail.com Implementing Partners Dr. Agnes Gebhard, Country Director/Chief of Party KNCV Agnes.gebhard@kncvtbc.org Hanneke Oudleh, Director Operational KNCV hannekeoudleh@kncvtbc.org Tija Candyana Yohan, Project Coordinator KNCV Candyana.yohan@kncvtbc.org dr. M Bey A Sonata,Technical Assistance Coordinator KNCV Bey.sonata@kncvtbc.org Daniel Sahanggamu, SIKM Coordinator KNCV Paulus.sahanggamu@kncvtbc.org Dr. M Akhtar, TB Specialist WHO akhtarm@who.int Sri Retno Sary, Software Engineer IRD Sri.retosary@irdinformatics.org Merry Samsuri, Project Coordinator FHI 360 msamsuri@fhi360.org Rini Palupy, Senior Technical Officer FHI 360 rpalupy@fhi360.org Betty Nabawi, Senior Technical Officer FHI 360 bnababan@fhi360.org Miladi Kurniasari, Technical Officer FHI 360 mkurniasari@fhi360.org Related Partners USAID/Indonesia Challenge TB Project Midterm Evaluation 93 Name and Position Organization Email Bowo W, Project Management Coordinator Global Fund TB emailbowo@gmail.com Helmi Suryani N, Staff of Monitoring and Evaluation Global Fund TB Helmi_suryani@yahoo.com NGO Drs. Zaenal Abidin, MSc, Advocacy, Communication, and Social Mobilization (ACSM) Specialist Aisyiyah Zaenal.abidin@pr-tbaisyiyah.or.id Esty Febriani, Chief of Party Cepat LKNU LKNU Esty.febriani@cepat-lknu.org USAID/Indonesia Challenge TB Project Midterm Evaluation 94 Name and Position Organization Email PROVINCIAL LEVEL INTERVIEWS 1-Jakarta Province Dr. Widayastuti, Head of Communicable Disease Jakarta Provincial Health Office (PHO) widyendro@gmail.com Riyanto Santoso W, Provincial Coordinator Officer Jakarta Province CTB Team Riyanto.windesi@kncvtbc.org 1A-Central Jakarta City Dr. Myrna, Head of Section of Communicable Disease Central Jakarta City Health Office (DHO) Myrna3005@yahoo.com Dr. Sylviana S W, TB Supervisor Central Jakarta City Health Office (DHO) Sylviana.sw@gmail.com Murniaty, TB Coordinator Senen Primary Health Care (Puskesmas) Atym23@yahoo.com 1B-East Jakarta City Fiena Fithriah, Head of Section of Communicable Disease East Jakarta City Health Office (DHO) Fiena.p2psjt@gmail.com dr. Ngabila Salama, TB Supervisor East Jakarta City Health Office (DHO) ngabilas@gmail.com Aturut Yansen, TB Supervisor East Jakarta City Health Office (DHO) Athurut26@gmail.com dr. Wiworoningsih, TB Coordinator Keramat Jati Primary Health Care (Puskesmas) Woro0407@gmail.com dr. Erlina Burhan, TB Coordinator Persahabatan Hospital Erlina_burhan@yahoo.com IC-North Jakarta City Dr. Hestia Sushartanti, Head of Section of Communicable Disease North Jakarta City Health Office (DHO) hestiasushartanti@yahoo.com Ajeng Sukmawati, TB Supervisor North Jakarta City Health Office (DHO) Dr.ajeng.s@gmail.com Dian Aristyana, TB Supervisor North Jakarta City Health Office (DHO) Deeann_cubi@yahoo.com Husniati Luthfiyah, TB Coordinator Kelapa Gading Primary Health Care (Puskesmas) Niya_cutee@yahoo.com Dr. Rosamarliana, Sp.P, TB Technical Coordinator Sulianti Saroso Hospital Rosa_pulmo@yahoo.co.id 2-South Sumatera Province Dr. NG Hikmat, Head of Communicable Disease South Sumatera Provincial Health Office (PHO) Dr.hikmet63@gmail.com USAID/Indonesia Challenge TB Project Midterm Evaluation 95 Name and Position Organization Email Dr. Yulia Maryani, MKes, Head of Section of Communicable Disease South Sumatera Provincial Health Office (PHO) yuliamaryani.dr@gmail.com Dr. Henry Pane, MKes, Staff of TB Program South Sumatera Provincial Health Office (PHO) Hendry.iskandar96@yahoo.com Khairima Ulfa, Staff of TB Program South Sumatera Provincial Health Office (PHO) kh41121n4@yahoo.co.id Jafirman Purba, S.Sos, Staff of TB Program South Sumatera Provincial Health Office (PHO) Firmanpurba74@yahoo.com Dr. Junida Sinulingga, MKes, Provincial Coordinator Officer South Sumatera CTB Team Junida.sinulingga@kncvtbc.org Dr. Chandra Wijaya, MKes, Senior Technical Officer South Sumatera CTB Team Chandra.widjaja@kncvtbc.org Surya Lilindari, Data Officer/M&E Officer South Sumatera CTB Team Surya.lilindari@kncvtbc.org 2A-Medan City Diana Harahap, SKM, TB Supervisor Medan City Health Office (DHO) diana.wasor75@gmail.com Dewi Sophia A, SKM, TB Supervisor Medan City Health Office (DHO) dwisophia.anggi@gmail.com Sumartini SKM, MKes, TB Supervisor Medan City Health Office (DHO) amandacayara@gmail.com Dr. Sabar Sumurung, Doctor of Facility Medan Correctional Facility Class II A for Woman (Prison) Sabarnaibaho26@gmail.com Dr. Herawati Lubis, Doctor of Clinic Aviati Private Clinic - Freedy, Nurse Aviati Private Clinic - Dr. Delyuzar, Mked, Director JKM Cepat (NGO) dr_delyuzar@yahoo.com Dr. Eva O K Simatupang, Technical Officer Medan City CTB Team Eva.simatupang@kncvtbc.org 2B-Deli Serdang District Dr. Herri Kurnia, MARS, Head of Communicable Diasease Deli Serdang District Health Office (DHO) herrikurnia1226@gmail.com Hatorangan Sinaga, SKM, Head of Section of Communicable Diasease Deli Serdang District Health Office (DHO) - Ahsanul Husna, S.Si, TB Supervisor Deli Serdang District Health Office (DHO) utacaniago@yahoo.com dr. Juilda R Pasaribu, TB Coordinator Lubuk Pakam Primary Health Care (Puskesmas) Juilda1982pasaribu@gmail.com Purnama Delima, Staff of TB Program Lubuk Pakam Primary Health Care (Puskesmas) Purnama.hutagalung02@gmail.com Sri Wirianty, Staff of Laboraturium Lubuk Pakam Primary Health Care (Puskesmas) Sriwirianty16@gmail.com USAID/Indonesia Challenge TB Project Midterm Evaluation 96 Name and Position Organization Email Sri Wahyuni, Head of Administration Lubuk Pakam Primary Health Care (Puskesmas) Sriwahyuni0707 dr. Agnes Caroline, Technical Officer Deli Serdang District CTB agnes.tarigan@kncvtbc.org 3-West Java Province Aan Sri Andrianti, TB Supervisor West Java Provincial Health Office (PHO) Aan_sriandriyanti2000@yahoo.com Joko Siswanto, MKes, Provincial Coordinator Officer West Java Provincial CTB Team Joko.siswanto@kncvtbc.org 3A-Bandung City Dr. Ira Dewi Sari, MT, Head of Section of Communicable Diasease Bandung City Health Office (DHO) iradije@gmail.com Dr. Nitta K, Head of Puskesmas Garuda Primary Health Care (Puskesmas) Nitta.bartok@yahoo.com Dewi Nur PS, Nurse Garuda Primary Health Care (Puskesmas) dewinurse@yahoo.co.id Utama Turnip, Manager of Health Division Pelita Indonesia (NGO) timkes@pelitaindonesia.org Titus S Maruhawa, Staff Pelita Indonesia (NGO) sohahautitus@gmail.com 3B-Bogor District Dr. Intan Widayati, Head of Section of Communicable Disease Bogor District Health Office (DHO) Intanwidiyati73@gmail.com Aan Setiawan, SKM, TB Supervisor Bogor District Health Office (DHO) tbckabbogor@yahoo.co.id Dr. Sri Irianti H, Head of Laboraturium Bogor District Health Laboraturium iriantidedy@gmail.com dr. Fusia M, Stt, MH.Kes, Head of Medical Unit Cibinong Hospital (RSUD) Uchiefagmail.com Mien Hasanah, Technical Officer Bogor District CTB Team Mien.hasanah@kncvtbc.org Evi Jayanti, Data Officer/M&E Officer Bogor District CTB Team Evi.jayanti@kncvtbc.org Dadang Sukmana, Project Assistant Bogor District CTB Team Dadang.sukmana@kncvtbc.org Reyna Rachmanniar, Data Officer Bogor District Global Fund TB Reyna.niar@gmail.com 4-Central Java Province Dr. Tatik Murhayati, MKes, Head of Communicable Diasease Central Java Provincial Health Office (PHO) Murha823@gmail.com Tri Dewi K, Staff of Communicable Diasease Central Java Provincial Health Office (PHO) Dewi_kristini@yahoo.com USAID/Indonesia Challenge TB Project Midterm Evaluation 97 Name and Position Organization Email Siti Juariah, Staff of Communicable Diasease Central Java Provincial Health Office (PHO) Juariahsiti18@yahoo.com Drg. Endang Nuraini, Provincial Coordinator Officer Central Java CTB Team Endang.nuraini@kncvtbc.org Dr. Ratna Ekasari, MKes, Senior Technical Officer Central Java CTB Team Ratna.ekasari@kncvtbc.org 4A-Semarang City Dr. Mada G, MKes, Head of Communicable Diasease Semarang City Health Office (DHO) madasoebowo@yahoo.com Maryani, SKM, MKes, Head of Section of Communicable Diasease Semarang City Health Office (DHO) maryatiskm@yahoo.co.id Haryati, MKes, TB Supervisor Semarang City Health Office (DHO) nangetalah@yahoo.co.id Mamat, SKM, MKes, TB Supervisor Semarang City Health Office (DHO) Ardisa_ma2t@yahoo.com Dr. Adhi Setiawan, SH, Doctor of facility Semarang Correctional Facility Class I Kedungpane (Prison) ayustia@gmail.com 4B-Surakarta (Solo) City Dr. Dwi Martyaastuti, Head of Communicable Diasease Surakarta City Health Office (DHO) dwimartyastuti@yahoo.co.id Agus Hufron, Skep, Head of Section of Communicable Diasease Surakarta City Health Office (DHO) agushufronslo@gmail.com Hartati, TB Supervisor Surakarta City Health Office (DHO) Tati_solo65@yahoo.com Dr. Liliana S, Head of Puskesmas Purwosari Primary Health Care (Puskesmas) Liliana.subagyo@gmail.com Rustini, Skep, TB Coordinator Purwosari Primary Health Care (Puskesmas) bundaius@yahoo.com Dr. Anita W Purwosari Primary Health Care (Puskesmas) Windriyani.anita@gmail.com Dr. Pricilla Meitasary Purwosari Primary Health Care (Puskesmas) Priccila.m@gmail.com Kusmini Susilawati Purwosari Primary Health Care (Puskesmas) Kusmini.susilawati68@gmail.com Dr. Umar Bahaswan, Deputi I of Medical Unit Kasih Ibu Private Hospital umarbhs@gmail.com Dr. Yulius W, Deputy II of Medical Unit Kasih Ibu Private Hospital yuliuswidy@yahoo.com Dr. Sari Kasih Ibu Private Hospital Sarierdeu2002@gmail.com Dr. Kurniawan, MSc, SpPk, Clinical Pathologyst Kasih Ibu Private Hospital Kurniawan.suatanto@yahoo.co.id Sri Ningsih, Head of Micro Laboraturium Kasih Ibu Private Hospital Sriningsihpandu65@gmail.com USAID/Indonesia Challenge TB Project Midterm Evaluation 98 Name and Position Organization Email Dr. Novita, SpP, Pulmonologyst Kasih Ibu Private Hospital Novi23pramana@yahoo.com Nunik Tri S, AMK, DOTS Nurse Kasih Ibu Private Hospital Trinunik538@gmail.com Apriudin, Head of organization SEMAR (Patient Organization) - Tugiyo, Deputy SEMAR (Patient Organization) Tugiotugio1@gmail.com Dr. Devi Malahayati, Technical Officer Surakarta CTB Team Devimala82@gmail.com Monita Primastiwi, Project Assistant Surakarta CTB Team Monita.primastiwi@kncvtbc.org Acin Tia Sakti, SKM, Enumerator Surakarta CTB Team Acintya97@gmail.com 5-East Java Province Drg. Ansarul Fahrudda, MKes, Head of Communicable Disease East Java Provincial Health Office (PHO) Ansarul_fahrudda@yahoo.com Ayu Kusumayanti, TB Supervisor East Java Provincial Health Office (PHO) Ayu_kusumayanti@yahoo.com Koesprijani, Head of Section of Technical Guidance Surabaya Health Laboraturium (BLK) koesprijani@yahoo.co.id Dr. Dwi Rahayu P, Provincial Coordinator Officer East Java CTB Team Dwi.puspaningsih@kncvtbc.org Ayu Hartini, Senior Technical Officer East Java CTB Team Ayu.pramadiyani@kncvtbc.org Singgih Pribadi, Technical Officer East Java CTB Team Singgih.pribadi@kncvtbc.org Acsandi, Monitoring and Evaluation Officer East Java CTB Team Ascandi.lutfianto@kncvtbc.org Sumaryadi, Office Staff East Java CTB Team sumaryadi@kncvtbc.org Retno Indrawati, Technical Officer Tulungagung CTB Team Retno.indrawati@kncvtbc.org 5A-Jember District Dyah Kusworini, Head of Communicable Diasease Jember District Health Office (DHO) Dyah_ki@yahoo.co.id Suhartini, Technical Officer Jember CTB Team suhartini@kncvtbc.org Slamet Santoso, Data Officer/M&E Officer Jember CTB Team Slamet.santoso@kncvtbc.org Moch. Nuroini, Project Assistant Jember CTB Team Moch.nuroini@kncvtbc.org Roro Antasari, Short Term Enumerator Jember CTB Team rrdrmksm@yahoo.com Dr. Anik Sukmawati, Sp.P, Head of TB Program Jember Lung Hospital Anik.sukma@yahoo.co.id Dr. Sigit Kusumajati, MM, Head of Section of Reasearch and Development Jember Lung Hospital kusumajatisigit@gmail.com Zetiawan Trisno, SKM, Promotion Coordinator Jember Lung Hospital zetiawantrisno@gmail.com Sahid Abdullah, Secretary Sekawan (Patient Organization) - USAID/Indonesia Challenge TB Project Midterm Evaluation 99 Name and Position Organization Email 6-Papua Province Marthen Robaha, SKM, Head of Section of Communicable Disease Papua Provincial Health Office (PHO) Marthen.robaha@yahoo.co.id Selly, Ajawailla, Head of Laboraturium Jayapura Health Laboraturium Cy_labuis@yahoo.co.id Helena Picarima, Senior Technical Officer Linkages Papua, FHI 360 hpicarima@fhi360.org Musa, Data Officer/M&E Officer Papua CTB Team mrapang@fhi360.org 6A-Jayapura City Drg. Eddy HS, Mkes, Head of Section of Communicable Disease Jayapura District Health Office (DHO) Eddyharapan18@gmail.com Indra, Head of Section of Communicable Disease Jayapura District Health Office (DHO) Boedisusanto_74@yahoo.co.id Mahsusriyanti, Staff of TB Jayapura District Health Office (DHO) tbkotajayapura@yahoo.com Oktavina T Tondoh, Staff of HIV Jayapura District Health Office (DHO) Tinarondok16@gmail.com Samuel M Baso, Head of TB-HIV Workgroup Jayapura District Hospital (RSUD) samuelbaso@yahoo.co.id Sitti N Soltief, Head of TB and VCT Jayapura District Hospital (RSUD) Sitti.jsg@gmail.com 6B-Mimika District Fransisca Kilangin, Head of Communicable Disease Mimika District Health Office (DHO) - Usman Alimuda, Head of Section of Communicable Disease Mimika District Health Office (DHO) Monali_tmk@yahoo.co.id Kamaludin, TB Supervisor Mimika District Health Office (DHO) Arifkamaludin1978@gmail.com Dr. Sieltiel, TB Program Coordinator Mimika District Hospital (RSUD) misaelsiki@gmail.com Maria B Lefaan, Staff of Laboraturium Timika Jaya Primary Health Care (Puskesmas) Rennysamosir0404@gmail.com Suprapto, Technical Officer Mimika CTB Team Suprapto_totok@yahoo.co.id Adrianus R Nitit, Data Officer/M&E Officer Mimika CTB Team (FHI 360) adrianfortescuego@gmail.com USAID/Indonesia Challenge TB Project Midterm Evaluation 100 ANNEX IV C: SOURCES OF INFORMATION: BIBLIOGRAPHY OF DOCUMENTS REVIEWED Aditama et al, The Cost of Scaling Up the TB Control Program in Indonesia. Journal Executive Summary. Undated. Alisjahbana, Bachti, Tuberculosis in Indonesia: Tipping the Balance Towards Better Control, TB Operational Research Group, National TB Program. Presentation. Undated. Challenge TB Program, Challenge TB Annual Workplan/Budget APA 1A January-March 2015. Excel Document. Undated. Challenge TB Program, Challenge TB Annual Workplan/Budget APA 1B April-September 2015. Excel Document. Undated. Challenge TB Program, Challenge TB Annual Workplan/Budget APA 2A October-November 2015. Excel Document. Undated. Challenge TB Program, Challenge TB Annual Workplan/Budget APA 2B December 2015-September 2016. Excel Document. Undated. Challenge TB Program, Challenge TB Annual Workplan/Budget APA 3. Excel Document. Undated. Challenge TB Program, Challenge TB Annual Workplan/Budget APA 4. Excel Document. Undated. Challenge TB Program, Challenge TB-Indonesia Year 1 Annual Report October 1, 2014-September 30, 2015. Annual Report. October 30, 2015. Challenge TB Program, Challenge TB-Indonesia Year I1 Annual Report October 1, 2015-September 30, 2016. Annual Report. October 30, 2016. Challenge TB Program, Challenge TB-Indonesia Year 3 Quartely Monitoring Report October-December 2016. Quartely Report. January 30, 2017. Challenge TB Program, Challenge TB-Indonesia Year 3 Quartely Monitoring Report January-March 2017. Quartely Report. April 28, 2017. Challenge TB Program, Challenge TB-Indonesia Year 3 Quartely Monitoring Report April-June 2017. Quartely Report. July 31, 2017. Challenge TB Program, Challenge TB Production Tracker Indonesia. Excel Document. September 20, 2017. Challenge TB Program, Challenge TB Year 4 STEP Table. Excel Document Draft. August 27, 2017. Challenge TB Program, Dashboard Indicator Definition. 2017. Challenge TB Program, District Dashboard. October 25, 2017. Challenge TB Program, Global Drug Facility (GDF) Diagnostic Portfolio Ordering List for All Laboratoriums 2016. Excel Document. June 2017. Challenge TB Program, Job Description of Senior Technical Officer-Laboratory. Job Description. Undated. Challenge TB Program, Job Description of Strategic Information and Knowledge Management (SIKM) Coordinator. Job Description. Undated. Challenge TB Program, Job Description of Technical Officer for Laboraturium. Job Description. Undated. USAID/Indonesia Challenge TB Project Midterm Evaluation 101 Challenge TB Program, Monitoring and Evaluation Plan APA 2. Excel Document. Undated. Challenge TB Program, Monitoring and Evaluation Plan APA 3B. Excel Document. Undated. Challenge TB Program, Monitoring and Evaluation Plan APA 4 (not approved). Excel Document. Undated. Challenge TB Program, KNCV Challenge TB Indonesia Structure APA 2. Organogram. October 27, 2015. Chellenge TB Program, KNCV Challenge TB Indonesia Structure APA 3. Organogram. September 16, 2016. Challenge TB Program, KNCV Challenge TB Indonesia Visio Structure. Organogram. May 16, 2015. Challenge TB Program, Number of CTB Staff. Word Document. Undated. Challenge TB Program, Summary of STTA (days-planned-actual). Excel Document. 2017. Challenge TB Program, USAID’s Flagship TB Control Program 2014-2019. Presentation. Undated. Challenge TB Project, Midterm Management Review: The Challenge TB Program. Management Review. April 2017. Challenge TB Project, Challenge TB Project RFA No. RFA-OAA 14-000006 Cooperative Agreement No. AID-OAA-A-14-00029 Cooperative Agreement. 2014. Law No. 23/2014 about Local Government. 2014. Law No. 40/2004 about National Social Insurance System. 2004. Ministry of Health, Breakthrough Toward Universal Access TB Control National Strategy in Indonesia: 2010-2014. 2010. Ministry of Health, Guidance for TB-HIV Program Collaboration. Guidance. 2015. Ministry of Health, Ministry of Health Decree No. HK.02.02/MENKES/52/2015 about Ministry of Health Strategic Plan 2015-2019. Ministerial Decree. 2015. Ministry of Health, National Action Plan for TB Control in Indonesia 2016-2020. Action Plan Draft. 2016. Ministry of Health, National Action Plan for TB-HIV Collaboration 2015-2019. 2015. Ministry of Health, Technical Guidelines Book for Tuberculosis Services for JKN patients. Guidelines Book. 2015. Ministry of Home Affairs, Decree No.2/2017 about Minimum Standards of Services. Ministerial Decree. 2017. TB Care Program, TB Care I Indonesia Final Report October 1, 2010 – December 31, 2014. Report. December 2014 The Joint External Monitoring Mission for Tuberculosis Control (JEMM TB), The Republic of Indonesia Joint External Monitoring Mission-Tuberculosis. Report. January 2017. USAID Indonesia Tuberculosis Program, Building National Capacity to Control the Spread of Tuberculosis. Press Release. 2014. WHO, Promoting the Law of NGOs and CSOs in Community-Based TB Care and Control, Report of Regional Workshop, Jakarta, Indonesia, 18-20 November 2013. Workshop Report. 2014. USAID/Indonesia Challenge TB Project Midterm Evaluation 102 ANNEX V: DISCLOSURE OF ANY CONFLICTS OF INTEREST USAID/Indonesia Challenge TB Project Midterm Evaluation 103 I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date USAID/Indonesia Challenge TB Project Midterm Evaluation 104 ANNEX VI: RESPONSE OF CTB TO THE MID-TERM EVALUATION GENERAL COMMENT The CTB team appreciates the thoroughness of the Mid Term Evaluation, the completeness of the report and the thoughtful recommendations. GOI ownership and inclusion of innovations in Indonesian legislation, regulations, and systems are conditions for nationwide implementation and sustainable domestic funding. In many cases, policy implementers are not aware that CTB (or the earlier USAID funded flagship projects TBCARE, TBCAP or TBCTA) laid the foundation for current practices. Examples of this are the nationwide uptake of External Quality Assurance for sputum smear examination; the uptake of Xpert as primary test for presumptive TB; the uptake of Xpert for testing for TB in presumptive MDR patients, children, PLHIV and Diabetics; the legislation in support of District Action Planning; the uptake of the DPPM approach as a national strategy (introduction now co-funded by CTB, GF and domestic resources); inclusion of WIFI TB as part of the DPPM approach; uptake of the PMDT benchmarking tool in hospital accreditation; treatment of (pre-)XDR and other complicated MDR patients with Bedaquiline, treatment of uncomplicated MDR TB with the shorter MDR treatment regimen. While recognizing more work needs to be done to document CTB innovations and drive increased coverage and quality of implementation of many of the CTB mediated interventions and good practices, we think the evaluation team has underestimated the extent to which CTB mediated innovations already have become mainstream policy and - in some cases – practice. With most of the CTB mediated innovations reaching maturity, in APA4 the 4th and 5th project years will see a series of publications and an acceleration of uptake. With district and provincial level partners being appreciative of CTB work and methods and especially their agreement with the statement that “lessons learned and best practices from CTB implementation could be replicated in Non-CTB districts” (p.41), we are encouraged to continue our strategy of increasing engagement with provincial and district level NTP and partners, especially in joint planning of activities and developing provincial capacity for dissemination of good practices. The lower score on this at national level is concerning. We will discuss with the national level NTP their concerns and more frequently invite them to CTB supported districts for exchange of ideas. Q1. PROGRAM PERFORMANCE Concerning replication and sustainability of the project, in APA4 CTB has planned to share CTB innovations through a capacity building approach, developing the enabling environment at national (NTP, MOH, MOHA, MOLHR, BAPPENAS, MO Village etc.) and provincial levels, and engaging national and provincial level teams. In APA4, CTB actively facilitates exchange of experiences between neighboring provinces and districts, to ensure cross-fertilization and wider impact of CTB mediated innovations and processes. For instance, to encourage uptake of District Action Planning, local key persons from CTB supported districts will be engaged as resource persons for other districts. Based on knowledge of the national efforts and the planned activities under APA4, the CTB team has a different expectation than the MTE team regarding the expected end-of-project achievements. We expect to fully achieve the targets for case notification (overall and from the private sector), as well as Xpert testing in previously treated patients; the target for HIV testing among TB patients is likely to be achieved for at least 90%. We also expect to achieve the targets on GLI standards and microscopy laboratory performance. Q2. PROGRAM MANAGEMENT CTB will discuss with NTP which steps to take to reach a better balance between technical assistance for TB technical issues and for management/leadership needs. In APA 4, within the CTB project management, Provincial teams have been given more authority in the management of activities. USAID/Indonesia Challenge TB Project Midterm Evaluation 105 Q3. INFORMATION GENERATION AND USE The district dashboard was developed in APA3 to show progress in CTB districts for USAID. In APA4, we will explore the introduction of a simplified version of this dashboard to health management staff at district levels to guide health facility assessment and subsequent action. Such adaptation will be done in a participatory manner involving district level stakeholders, thus optimizing chances for a “good fit” with the needs of district health managers. Concerning dissemination of good practices and lessons learned, we already have planned a documentation and communication strategy to generate user-friendly information to relevant audiences. In APA4 (and APA5), we anticipate a scale-up of CTB approaches, first in the provinces where CTB is active, and in other provinces through the NTP and local partners. In addition to findings of the MTE, CTB contributed to capacity building in Indonesia on the use of the TIME model (in collaboration with LSHTM) to estimate the TB burden at district levels, which was used in the successful GF application for 2018-2020. These estimates are now used for district level monitoring of NTP performance. Q4. ADAPTATION TO INDONESIA’S HEALTH SYSTEM District Action Plan (DAP) is an important approach to strengthen the capacity of districts in response to ongoing process of health system decentralization. The approach was piloted jointly by the NTP and CTB in CTB supported districts. In collaboration with the Ministry of Home Affairs, CTB and the NTP developed this into guidelines for provincial and local governments, in order to achieve sustainable financing of the TB program in Indonesia. In 2017, it was introduced by the Ministry of Health to a large audience of representatives of ministries and provincial and local governments, during the event “Acceleration towards Indonesia Free of TB: Multi-sectoral Contribution”, which mobilized the commitment of Indonesian districts heads to TB elimination. Q5. LESSONS AND BEST PRACTICES In addition to the strategies for documentation and dissemination mentioned by the MTE, since APA3, CTB applies a district “graduation approach”, using the district dashboard. As described under Q3, we will simplify the dashboard to facilitate focused action by local health authorities. Q6. SUSTAINABILITY Since APA3, and intensified in APA 4, in addition to approaches described above, CTB aims to increase sustainability by intensifying the sub-award mechanism to local partners with a perspective of funding beyond the CTB project period. Papua will be given special consideration by re-assessing labor conditions and salary scales at the local Papua level, and sub-sequent discussion with USAID. As an interim measure, CTB has planned an extensive mentoring process using internal CTB technical resource persons and consultants, depending on the needs. Correction on numbers and figures: • Q2 Program Management (page 29 and page 4): We have more technical than operational staff as of October 2017. There were 62 technical staff (19 at RO, 43 in the provinces) and 52 operational staff (27 at RO, 25 in the provinces). • On page 3 section D. Expanding the network of diagnostic services: The number of laboratories performing culture and drug susceptibility testing (C/DST) increased from 8 (not zero) in 2014 to 14 in 2017. USAID/Indonesia Challenge TB Project Midterm Evaluation 106 U.S. AGENCY FOR INTERNATIONAL DEVELOPMENT 1300 Pennsylvania Avenue, NW Washington, DC 20523