USAID INDONESIA 1 MID-TERM PROGRAM PERFORMANCE EVALUATION Community Empowerment of People Against Tuberculosis MAY 2016 This report is made possible by the support of the American People through the United States Agency for International Development (USAID). The contents of this report is the sole responsibility of the authors, Angus CR Walker and Isma Novitasari, and do not necessarily reflect the views of USAID or the United States Government. USAID INDONESIA 3 MID-TERM PROGRAM PERFORMANCE EVALUATION MAY 20, 2016 #AID-497-A-13-00003 #AID-497-A-13-00007 #AID-497-A-13-00011 DISCLAIMER This report is made possible by the support of the American People through the United States Agency for International Development (USAID). The contents of this report is the sole responsibility of the authors, Angus CR Walker and Isma Novitasari, and do not necessarily reflect the views of USAID or the United States Government. MID-TERM PROGRAM PERFORMANCE EVALUATION Community Empowerment of People Against Tuberculosis USAID INDONESIA 5 MID-TERM PROGRAM PERFORMANCE EVALUATION CONTENTS FIGURES .........................................................................................................................................................8 TABLES .........................................................................................................................................................8 ACKNOWLEDGMENTS ..............................................................................................................................9 ACRONYMS ................................................................................................................................................ 10 EXECUTIVE SUMMARY............................................................................................................................ 13 CHAPTER ONE: PROJECT BACKGROUND ..............................................................................................................19 1.1 TB CONTROL PROGRAM IN INDONESIA.............................................................................. 19 1.2 CEPAT PROGRAM.......................................................................................................................... 20 CHAPTER TWO: PROJECT EVALUATION.................................................................................................................25 2.1 EVALUATION PURPOSE .............................................................................................................. 25 2.2 EVALUATION QUESTIONS ......................................................................................................... 25 2.3 METHODOLOGY........................................................................................................................... 26 2.31 Phase One: Start-up Coordination and Desk Review .................................................... 26 2.32 Phase Two: Data Collection ............................................................................................... 26 2.33 Phase Three: Data Analysis................................................................................................. 29 2.4 BIASES AND LIMITATIONS.......................................................................................................... 30 2.5 OUTCOMES..................................................................................................................................... 31 CHAPTER THREE: FINDINGS...................................................................................................................................................33 3.1 STRENGTHS AND ACHIEVEMENTS.......................................................................................... 33 3.11 Strengthening the NTP Response to TB at the Community Level .............................. 33 3.12 Implementation of Health Promotion Strategies............................................................ 35 3.13 Capacity Development of Local Stakeholders ................................................................ 35 3.14 Gains Achieved with Community Organizations and Government Engagement...... 39 3.2 PROJECT IMPLEMENTATION CHALLENGES.......................................................................... 40 3.21 Supply Side of Health Services – Patient Treatment and Laboratory Supply............. 40 3.22 Health Promotion ................................................................................................................ 41 6 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION 3.23 Management.......................................................................................................................... 42 3.24 Sustainability ......................................................................................................................... 44 CHAPTER FOUR: CONCLUSIONS........................................................................................................................................47 4.1 STRENGTHENING THE IMPLEMENTATION OF THE NTP STRATEGY AT THE DISTRICT LEVEL............................................................................................................. 47 4.11 Strengthening the Community-level Response to TB Control ..................................... 47 4.12 The Supply Side of Government-provided TB Health Services.................................... 47 4.13 Capacity Building.................................................................................................................. 48 4.2 IMPROVING EFFECTIVENESS AND COLLABORATION ...................................................... 48 4.21 Health Promotion ................................................................................................................ 48 4.22 Management.......................................................................................................................... 49 4.3 SUSTAINABILITY ........................................................................................................................... 50 CHAPTER FIVE: LESSONS LEARNT..................................................................................................................................53 5.1 CADRE SUPPORT .......................................................................................................................... 53 5.2 SOME WELL-INTENTIONED INTERVENTIONS NEED ADDITIONAL SUPPORT TO BE IMPLEMENTED .................................................................................................................. 53 5.3 SUPPLY SIDE OF HEALTH SERVICES IMPACTS ON HEALTH-SEEKING BEHAVIOR..... 53 5.4 CEPAT PARTNER PERFORMANCE ............................................................................................ 53 5.5 DISTRICT HEALTH SERVICES..................................................................................................... 54 CHAPTER SIX: RECOMMENDATIONS...........................................................................................................................55 6.1 SUPPLY SIDE OF GOVERNMENT-PROVIDED TB HEALTH SERVICES.............................. 55 6.2 HEALTH PROMOTION ................................................................................................................. 56 6.3 MANAGEMENT............................................................................................................................... 57 6.4 SUSTAINABILITY ........................................................................................................................... 59 6.5 FUTURE PROGRAM DESIGN AND DELIVERY........................................................................ 60 USAID INDONESIA 7 MID-TERM PROGRAM PERFORMANCE EVALUATION ANNEX ONE: DISTRICT PARTICIPANTS IN CEPAT PROGRAM ................................................... 62 ANNEX TWO: DATA COLLECTION SCHEDULE.............................................................................. 63 ANNEX THREE: DOCUMENTS REVIEWED........................................................................................ 64 ANNEX FOUR: EVALUATION SCOPE OF WORK............................................................................. 66 ANNEX FIVE: DATA COLLECTION INSTRUMENTS ........................................................................ 79 ANNEX SIX: RESPONDENTS CONSULTED ....................................................................................... 99 ANNEX SEVEN: DISCLOSURE OF ANY CONFLICTS OF INTEREST.......................................... 105 ANNEX EIGHT: STATEMENT OF DIFFERENCES............................................................................. 107 REFERENCES..............................................................................................................................................................................................108 8 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION FIGURES Figure 1-1: Location of CEPAT Partner Projects .............................................................21 TABLES Table 1-1: Short Description of CEPAT Partners ............................................................22 Table 3‑1: CEPAT Project Achievements ..........................................................................34 Table 3‑2: Specific Partner Health Promotion Activities ..............................................36 Table 3‑3: Summary of Key Capacity-building Activities Provided to CEPAT Partners .................................................................................................38 USAID INDONESIA 9 MID-TERM PROGRAM PERFORMANCE EVALUATION ACKNOWLEDGMENTS Thank you to everyone who the evaluation team met and who provided their thoughts, opinions and time which enabled us to produce this evaluation. A special acknowledgement goes to USAID/Indonesia office staff for their extensive support, BANTU Project staff for their assistance with coordination and communication, the Ministry of Health staff at both national and district levels, and CEPAT Project partner staff. We also extend a special appreciation to CEPAT cadres and patients who contributed to the field research. 10 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION APBD Anggaran Pendapatan dan Belanja Daerah/ District Budget Bappeda Badan Perencanaan Pembangunan Daerah / Regional Development Planning Agency BOK Bantuan Operasional Kesehatan / Health Operational Funding CDR Case Detection Rate CDCS Country Development Cooperation Strategy CEPAT Community Empowerment of People Against Tuberculosis COP Chief of Party COR Contracting Officer Representative CSO Civil Society Organization DHO District Health Office DOTS Direct Observation Treatment Short-Course ET Evaluation Team FGD Focus Group Discussion Gerdunas Gerakan Terpadu Nasional/ National Movement against TB GFATM Global Fund to Fight AIDS, Tuberculosis and Malaria GHI Global Health Initiative HIV/AIDS Human Immunodeficiency Virus / Acquired Immune Deficiency Syndrome IDI Ikatan Dokter Indonesia/Indonesia Medical Association IDIs In-Depth Interviews JKM Jaringan Kesehatan/Kesejahteraan Masyarakat / Community Welfare/Health Network KAP Knowledge, Attitude, Practice KII Key Informant Interview KMP Koalisi Masyarakat Peduli TB / Community Coalition for TB KPI Key Performance Indicators LKNU Lembaga Kesehatan Nahdlatul Ulama / Nahdlatul Ulama Health Institute MDR-TB Multi Drug Resistant-Tuberculosis MoH Ministry of Health NGO Non-Governmental Organization NTP National Tuberculosis Program PBNU Pengurus Besar Nahdlatul Ulama/ Executive Board of Nahdlatul Ulama PCA Patient-centered Approach PCTC Patients’ Charter for Tuberculosis Care ACRONYMS USAID INDONESIA 11 PDPI Persatuan Dokter Paru Indonesia (Indonesia Pulmonologist Association) PETA Pejuang Tangguh (Strong Fighter)/MDR-TB Patient Support Group PKK Pembinaan Kesejahteraan Keluarga / Family Welfare Movement PMDT Programmatic Management of Drug-resistant TB PMP Project Monitoring Plan PPNI Persatuan Perawat Nasional Indonesia/ Indonesian National Nurses’ Association Pokja Kelompok Kerja / Working Group Puskesmas Pusat Kesehatan Masyarakat / Community Health Center RCD Roman Catholic Diocese RFA Request For Application SGD Small Group Discussion SOW Scope Of Work SOW EQ Scope Of Work - Evaluation Question TB Tuberculosis TBCTA Tuberculosis Coalition for Technical Assistance TPM Team Planning Meeting UI University of Indonesia URC University Research Co. LLC. USAID United States Agency for International Development USU Universitas Sumatera Utara/ University of North Sumatra YSSK Yayasan Satu Karsa Karya / Karsa Karya Foundation MID-TERM PROGRAM PERFORMANCE EVALUATION USAID INDONESIA 13 MID-TERM PROGRAM PERFORMANCE EVALUATION EXECUTIVE SUMMARY PROJECT BACKGROUND Over recent years, the Indonesia National Tuberculosis (TB) Program (NTP), has made some outstanding progress with fighting the spread of TB. For example, the improvement in the treatment success rate from less than 50 percent in 1999 to almost 88 percent in 2010 and the rapid increase in the TB Case Detection Rate (CDR)1 from 22 percent in 2000 to 61 percent in 2008. Although these achievements are remarkable, Indonesia is still challenged with the task of successfully implementing its TB control programs. The CDR has stagnated or declined slightly since 2010. This is largely due to issues such as under-notification of cases, barriers to accessing diagnostic services and diagnostic delays. Multidrug-resistant TB (MDR-TB) also presents significant challenges to effective TB control. Stigma and low levels of community knowledge about TB also present major challenges. For example, the latest survey by NTP indicates that only 21 percent of the population are aware of the risks of TB and how to properly treat it. Comprehensive and coordinated approaches that engage the community in supporting TB services, including assisting patients to comply with treatment and identify suspected TB cases, still need to be strengthened and expanded. USAID has fully endorsed the NTP strategies that emphasize community partnership and these are regarded as aligned with the United States Agency for International Development (USAID) strategy in its Indonesia Country Development Cooperation Strategy (CDCS). The community role in TB management is expected to build community ownership of TB programs. The Community Empowerment of People Against Tuberculosis (CEPAT) project was launched in 2012, partnered with the NTP, with the aim of enhancing technical capacity and leading to improved TB detection and treatment services. The principal objective of CEPAT is to support the NTP in “achieving universal access to quality and early TB diagnosis and treatment through community empowerment.” To attain this objective, the project uses two components: (i) mobilizing communities to support activities that will improve TB care. Improved TB care includes: better access to, and improved quality of, TB services, increased community knowledge of TB, improved health-seeking behavior, and increased support to patients receiving treatment; and (ii) advocating for increased TB resources and improved TB services. Advocacy will support the efforts of the NTP and demand resources and improved services for TB patients and families. EVALUATION PURPOSE AND EVALUATION QUESTIONS As the CEPAT Program has entered mid-term for the implementing partners,2 USAID requested an external evaluator–through the BANTU Project–to conduct an independent mid-term evaluation. The focus of this evaluation was to assess the program’s performance and its impact. The evaluation was expected to provide insights and important feedback to each of the partners and 1 The Case Detection Rate is calculated as the number of new and relapse TB cases that were notified by NTP, divided by the estimated number of incident cases of TB that year. 2 Lembaga Kesehatan Nahdlatul Ulama (LKNU) and Jaringan Kesehatan Masyarakat (JKM) are in the fourth year of participation in the program and the Roman Catholic Diocese of Timika (RCD) is in the third year. 14 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION stakeholders that should assist them to understand both the strengths of the program and areas where technical, administrative and management efforts could be improved. It would also provide evidence and learning for improving USAID/Indonesia program designs, strategies and policies. Nine key evaluation questions were specifically used to evaluate CEPAT’s performance: 1. To what extent have the program activities strengthened the National TB Program and the national level response to TB at the community level? 2. To what extent have the program activities increased TB detection, prevention and treatment in Indonesia, and been aligned with CEPAT goals and the TB Civil Society Organization (CSO) strategy for Indonesia? 3. To what extent has the program activity strengthened capacity of the national and subnational (province and district) government, private sector, community, and other stakeholders? 4. What is the contribution of each project to the overall USAID CEPAT program goals? 5. How replicable, adaptable/adoptable, and sustainable are the programs/program components? 6. How can the program design, management, and implementation become more efficient, effective and relevant toward achieving program goals? 7. How effective has the collaboration/coordination among the programs been in maximizing efforts and achieving greater results? 8. Does the CEPAT Program align with the country’s other community TB efforts (for example, Aisyiyah funded by Global Fund) and other community health efforts (that is, government-funded community health cadres and community organizations in other health areas in Indonesia) and, if not, what are the lessons to be learned and possibilities for improvement? 9. How can local and national ownership and future commitment to continued implementation of good practices/lessons learned be enhanced? EVALUATION DESIGN, METHODS AND LIMITATIONS The Evaluation Team (ET) used several techniques that entailed a mix of mutually reinforcing qualitative methods and quantitative data that reflected the program logic and research questions being addressed. During the time span of evaluation (April–May 2016), the team also utilized a document and literature review, qualitative data collection, and data analysis to capture the diversity of opinions and perceptions of stakeholders about the CEPAT Program, including its strengths and weaknesses. The qualitative data collection, which included 35 In-depth Interviews (IDI) and 16 focus group discussions (FGD), was conducted in Jakarta, North Sumatra (Deli Serdang and Medan), West Java (Cirebon District), Papua (Timika) and West Papua (Sorong). The ET faced several limitations during data collection, including a lack of access to contacts, no available baseline data, newly appointed government staff members (new roles, limited knowledge) and significant time constraints for the evaluation data collection. These limitations, however, did not prevent the ET from gathering sufficient information and data needed to produce findings, conclusions, and recommendations for this evaluation. FINDINGS, CONCLUSIONS AND RECOMMENDATIONS The key findings, conclusions and recommendations of this evaluation are summarized below under the five themes, relating to the evaluation questions (EQ), which were identified during the data collection stage. USAID INDONESIA 15 MID-TERM PROGRAM PERFORMANCE EVALUATION 1. Strengthening the NTP response to TB at the community level (EQ 1,2,3,4) The ET found that the CEPAT Program had been implemented effectively by the three CEPAT partners across 25 high TB prevalence districts in seven provinces since the program started in 2012. The program was frequently identified by many respondents as a means to extend Puskesmas services to the community through the recruitment and mobilization of cadres. The three CEPAT projects have strengthened the NTP response at the community level through the various activities implemented by TB cadres. These activities include contact tracing, linking suspected TB cases from the community to Puskesmas for screening, providing health promotion activities and supporting patients to comply with TB treatment. Community infrastructure, in the form of 29 Pos TB Desa (Village TB Posts) facilities, has been established in Papua and West Papua provinces. This provides a base for cadres to administer TB drugs to patients and a location where interpersonal health promotion can be provided to patients and small groups. Challenges have been encountered throughout the delivery of the CEPAT Program, most notably the lack of engagement and commitment from District Health Offices (DHO) to support active case finding. Other challenges included some instances of issues associated with the supply side of health services, including negative treatment of TB patients and laboratory supply issues–the latter issue being limited to the provinces of Papua and West Papua. The ET recommends to CEPAT partners that greater emphasis needs to be placed on addressing issues associated with the supply side of government-provided TB health services, including two key aspects: (1) improving patient treatment; and (2) addressing deficiencies associated with laboratory and logistics by supporting Puskesmas/health care providers to: a. adopt and improve the Patient-centered Approach (PCA); and b. proactively identify existing issues and pre-empt future issues to ensure that service continuity and quality is maintained. 2. Implementation of health promotion strategies (EQ 2,3) Respondents involved in this evaluation perceive the program as having high value, mainly due to the community mobilization and health promotion activities. CEPAT partners have invested significant effort and resources into developing and implementing health promotion activities that aim to increase TB health-seeking behavior, reduce stigma and increase health literacy about TB. A diverse range of communication channels and resources have been utilized and developed to disseminate TB-related information to communities. These include religious networks, interpersonal communication via health cadres, radio, television, print materials and social media. Several areas were identified where health promotion could be more effective and, therefore, it is recommended that CEPAT partners improve health promotion impact by: a. establishing a body of formative research to guide the development of strategies and to act as a base line; b. improving health promotion planning and evaluation; c. developing and implementing broad-reaching health communication campaigns; 16 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION d. reviewing existing Information, Education and Communication (IEC) material for effectiveness and assessing IEC supply issues; e. increasing the utilization of social media (in districts where viable); and f. that stakeholders involved in future project design roles conduct a feasibility study into utilizing M-Health (mobile phone-based technology and communications) for community-based TB programs with a view to implementing. 3. Capacity development of local stakeholders (EQ 3,7) Capacity development of local stakeholders has been comprehensive and systematic, ranging from community members improving their TB knowledge to policy makers gaining greater understanding of TB issues. Capacity building has been achieved through a diverse range of partners that provided various capacity-building activities and these include USAID, CEPAT partners, international subgrantees and cadres. Capacity building of CEPAT partners has been an important component of the CEPAT Program to ensure the achievement of effective project management. USAID has provided a range of capacity building to CEPAT partners throughout the program’s implementation. Capacity-building support has also been provided to CEPAT partners through their collaboration with international partners, such as Westat and URC. These international partners have developed project management and technical competencies of JKM and LKNU. Important steps in developing the capacity of Puskesmas facilities have been progressed by LKNU. This includes conducting an assessment of service quality (drug supply, functioning labs, staffing levels), assessing patient satisfaction through surveys and by developing the clinical capacity of some Puskesmas staff to treat MDR-TB patients. 4. Strengthening CEPAT partners (EQ 7,8) There were no issues that required modification to project designs throughout the implementation of all CEPAT partner projects. There were, however, some challenges experienced by CEPAT partners and these included understanding and dealing with the complexities of project administration, funding, time constraints and high staff turnover in some project locations. Most of these issues were overcome with technical assistance provided by USAID or subgrantees or through trial and error experience gained through CEPAT partner’s various program implementation experiences. A need for strengthening CEPAT partner efforts and capacity was identified by the ET given that all CEPAT partners’ experienced significant challenges with achieving engagement and commitment for community-based TB programs from district and provincial government and policy makers. At the national level, engagement between CEPAT and the NTP is also very limited. Furthermore, it was evident, based on feedback from a senior program manager at the NTP office, that they were unaware of RCD’s existence. This is a significant admission, given that RCD operate in locations with the highest national prevalence of TB and HIV/AIDS. It is reasonable to assume, therefore, that networking and communication between CEPAT partners, particularly RCD and heads of the NTP, are areas that require development. The ET found that interagency collaboration between CEPAT partners and relevant TB stakeholders (including the MoH, HIV/AIDS agencies, Aisyiyah, the WHO and Challenge TB) at both the national level and local level was very limited, however, some progress had been achieved at the district level. Several informants suggested that the lack of collaboration caused USAID INDONESIA 17 MID-TERM PROGRAM PERFORMANCE EVALUATION program inefficiencies and duplication of service delivery. Reasons identified for the limited collaboration between CEPAT partners and with other partner agencies included limited budgets to fund travel expenses, limited reach of CEPAT partner’s reports, difficulties establishing mutually convenient timing suitable for meetings and geographic locations of partner agencies. The ET recommends that USAID provide support to CEPAT partners to improve management practices in the following areas: a. knowledge management practices; b. monitoring and evaluation processes; c. collaboration with partner agencies by: i. supporting CEPAT partners to establish periodic district coordination meetings. ii. supporting TB stakeholders to establish a national level technical working group focused on community-based TB services–community mobilization, health promotion, and cadre support. iii. supporting CEPAT partners, particularly those who have frequent access to Jakarta, to participate in relevant technical forums such as those established by the GFATM; and d. collaboration among CEPAT partners by facilitating a cluster approach as opposed to three discrete NGOs implementing program delivery independently. 5. Sustainability (EQ 5,6,9) Some progress and subsequent outcomes have been achieved by CEPAT partners engaging with community organizations and government. This includes LKNU establishing district TB Working Groups (reactivated Gerdunas3 ) in Cirebon and Pokja (Kelompok Kerja/Working Group) in Kediri, East Java. In Cirebon, this reactivated working group succeeded in securing commitments from private hospitals to improve their DOTS service. Advocacy for improved TB resources and services has been conducted by all CEPAT partners. This has resulted in some successes by way of attracting contributions from local government that are allocated to fund cadre activities. Initiatives to identify health service supply issues through the use of patient satisfaction surveys have progressed in all project locations in Java where LKNU operates, including Central and North Jakarta, Depok, Cirebon, Blitar, Kediri and Tuban. It is recommended that USAID support CEPAT partners to develop and progress strategies aimed at improving sustainability of gains achieved. This should include: a. developing the capacity of CEPAT partners’ advocacy techniques specific to their political contexts; b. utilizing and developing existing village-level capacities to improve health services and achieve community-based TB program funding by providing the opportunity for cadres, and other relevant stakeholders, to engage in training that develops capacity in community-level governance topics; and c. developing a media strategy that raises the profile of TB and positions the issue on the political agenda (national and subnational levels) with the aim of achieving greater resource allocation by policy makers. Recommendations for future program delivery design: If the CEPAT Program is extended beyond its current timeframe, then several program design and delivery changes are recommended. These recommendations are made with the aim of 3 Gerdunas: Gerakan Terpadu Nasional/National Coordination Movement against TB is an NTP program to coordinate and improve inter-sector collaboration to fight against TB. 18 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION scaling up the CEPAT Program, broadening its geographic reach, improving efficiency and sustainability. These recommendations encompass: a. establishing a consortium of NGOs that collaborate rather than having discrete NGOs delivering individual projects; b. increasing program focus on advocacy and capacity development rather than service provision; and c. developing the capacity of district health facilities to enable government health services to better address the TB epidemic. USAID INDONESIA 19 MID-TERM PROGRAM PERFORMANCE EVALUATION CHAPTER ONE: PROJECT BACKGROUND 1.1 TB CONTROL PROGRAM IN INDONESIA In the past few years, Indonesia’s National TB Program (NTP), with external support, has made remarkable progress. The CDR increased rapidly from 22 percent in 2000 to 69 percent in 2008. It has been reported, however, that since 2010 this rate has remained stagnant or slightly declined, due to under-notification, barriers in access to diagnosis and diagnostic delays (WHO 2010). Treatment success rates improved from less than 50 percent in 1999 to 88 percent in 2014 (WHO 2015). Despite all the progress that has been achieved, the burden of TB disease in Indonesia is much higher than previously thought (WHO 2015) and Indonesia is still facing many challenges in TB control programs. It is estimated from the latest TB Survey 2013/2014 that the TB incidence rate in Indonesia is 403 (range 278-550) per 100,000 population, with an estimated prevalence (all forms of TB, and including children as well as adults) of 660 (range 528-813) per 100,000 population. The mortality rate in 2014 reached 41 per 100,000 population (WHO 2015). The spread of multi-drug resistant TB (MDR-TB) is one of the most challenging predicaments. The NTP has responded by implementing programmatic management of drug-resistant TB (PMDT) in several sites, with plans to scale up additional sites over the coming years. According to the WHO Global Report 2015, the total number of estimated cases of MDR-TB was 5,600 in 2014, with an estimation of 1.9 percent of new TB cases and 12 percent of retreatment cases to be of MDR-TB. At the end of 2014, it was estimated that 1,812 cases of MDR-TB were confirmed and 1,284 cases were put into treatment. A 2013 report from Management Sciences of Health (MSH)–an international organization working in TB Control–estimated that 30 percent of cases are still not being detected and many patients are diagnosed late (Collins 2013). The National TB Prevalence Survey 2013/2014 claimed an even higher number of undetected cases. The survey reported that only about one-third of the estimated incident cases that occur each year are being detected and reported to national authorities. Private health providers are still providing substandard or incomplete treatment, compared with treatment provided by public health facilities, or not following the Direct Observed Treatment Short-course (DOTS) standard (Ministry of Health 2014) There are also limitations in the health system including laboratory quality; human resources; logistic and data quality; and weakness in the capacity of the community to increase public knowledge of TB, support improved TB diagnosis and treatment, provide adequate support for treatment, and increase demand for better quality TB services. Nevertheless, a large percentage of the Indonesian population still perceive that the quality of public health services and drugs are lower than the quality of services and drugs provided by private health facilities. Given this, many people choose and pay for private health services over public health services. TB is still stigmatized and the general knowledge of communities regarding TB remains low. According to the TB Prevalence Survey conducted by The National Institute of Health Research and Development [Badan Litbangkes] - Ministry of Health (MoH and WHO 2004),only 11 percent of the population are aware of the risks of TB and how to properly treat it. The most at-risk population for TB still does not have access to proper TB services. Comprehensive and coordinated approaches to engage the community in supporting TB services, including helping patients to comply with treatment and identify possible TB cases still needs to be strengthened. The NTP, with funding support from Government of Indonesia (GoI), the Global Fund to Fight AIDS, Tuberculosis and Malaria (GFATM) and USAID, has accelerated the implementation of the National TB Program Strategic Plan 2010-2014 and updated the 2015-2019 strategic plan. The aim of these plans is to reduce TB deaths and prevalence in order to contribute to achieving 20 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION the health development goal of improving the public health status. The MoH Strategic Plan (2014) encompasses seven strategies which are fully consistent with the Stop TB Strategy (WHO), namely: (i) scaling up and improving quality DOTS services; (ii) addressing TB/HIV, MDR-TB and the needs of poor and other vulnerable groups; (iii) engaging all public and private providers with the implementation of International Standards for TB Care (ISTC); (iv) empowering TB patients and communities; (v) strengthening the health system; (vi) increasing the commitment of central and local government; and (vii) enhancing research, development, and utilization of strategic information. 1.2 CEPAT PROGRAM USAID has fully endorsed the NTP strategies that emphasize community partnership and these are regarded as aligned with the USAID strategy in its Indonesia CDCS. The community role in TB management is expected to build community ownership of the TB program. The CEPAT project was launched in 2012, partnered with the Indonesian NTP, with the aim of enhancing technical capacity and skills and leading to improved TB detection and treatment services. The principal objective of CEPAT is to support the NTP in “achieving universal access to quality and early tuberculosis (TB) diagnosis and treatment through community empowerment.” To attain this objective, the project uses two components: Component 1: Mobilizing communities to support improved TB care. Improved TB care includes: better access to, and quality of, TB services, increased community knowledge of TB, improved health￾seeking behavior, and increased support to patients receiving treatment. Component 2: Advocating for increased TB resources and improved TB services. Advocacy will support the efforts of the NTP and demand resources and improved services for TB patients and families. CEPAT activities are targeted to improve TB care, access and services to the most vulnerable populations. This includes people living in urban slums, remote islands, malnourished people, the poor, and contacts of TB patients. Partnership with communities affected by TB is a critical element of overall national TB control activities in Indonesia. As detailed in the Request for Application (RFA),4 activities under Component One are expected to achieve the following results: a. improved case finding; b. increased access to service delivery; c. improved health-seeking behavior to quality TB services; d. increased knowledge and awareness of key TB information; e. increased contact tracing of TB patients; and f. improved and expanded community social support systems for patients during treatment. Under Component Two, the program is expected to: a. increase the district budgets and allocated resources for TB; b. increase advocacy at the local levels for TB services and resources; c. increase the knowledge of TB among district leaders and other key stakeholders; and d. reduce local policies inconsistency with NTP guidance. 4 TB CEPAT RFA 497-11-0000055. USAID INDONESIA 21 MID-TERM PROGRAM PERFORMANCE EVALUATION In order to support high-impact partnerships and local solutions encouraged by USAID Forward,5 cooperative agreements were awarded to three Indonesian nongovernment organizations, namely: Lembaga Kesehatan Nahdlatul Ulama (LKNU); Jaringan Kesehatan/Kesejahteraan Masyarakat (JKM) and the Roman Catholic Diocese of Timika (RCD Timika).6 These cooperative agreements aimed to ensure that local systems own, resource, and sustain the development result that USAID invested. Through CEPAT, USAID therefore aimed to build the capacity of Indonesian NGOs selected for this project. The total funding of all three awards is nearly US$10 million over the life of the project. During the evaluation period (April 2016), the program has entered its fourth year for LKNU and JKM, and its third year for RCD. The program is implemented in 25 districts (see Annex One) in seven provinces: North Sumatra, West Sumatra, West Java, DKI Jakarta, East Java, Papua and West Papua. Partnering with these NGOs, the program focuses on increasing people’s awareness on TB prevention and early detection, as well as fostering local commitment to support the national program. Figure 1-1: Location of CEPAT Partner Projects 5 USAID Forward is an effort to strengthen the Agency by embracing new partnership, investing in the catalytic role of innovation and demanding a relentless focus on results. For more information, see https://www.usaid.gov/usaidforward 6 Cooperative Agreement # AID-497-A-13-00003 (LKNU); Cooperative Agreement # AID-497-A-13-00007 (JKM); and Cooperative Agreement #AID-497-A-13-00011 (RCD Timika). 22 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION The following table depicts the short description of CEPAT partners. Table 1-1: Short Description of CEPAT Partners ASPECT PARTNER LKNU JKM RCD Timika Established 1998 2001 2004 Cooperative agreement AID-497-A-13-00003 AID-497-A-13-00007 AID-497-A-13-00011 Date of award November 8, 2012 December 2012 June 2013 Type of organization Faith-based Academics/Public health network Faith-based Project Location West Java Province (Depok, Cirebon District and West Bandung); DKI Jakarta (North and Central Jakarta); East Java Province (Blitar, Kediri and Tuban) North Sumatra Province (Medan, Sedang Bedagai, Tanjung Balai City and Deli Serdang); West Sumatra Province (Padang, Solok, Tanah Datar, Pasaman, Padang Pariaman and Kep. Mentawai); DKI Jakarta (West, East, North and South Jakarta) Papua Province (Wamena, Timika, Nabire and Paniai); West Papua Province (Sorong) Subgrantees University Research Co., LLC (URC); Indonesian Pulmon￾ologist Association (PDPI); Indonesia Medical Association (IDI); Nurses’ Association (PPNI); MDR-TB Patient Asso￾ciation (PETA); TB Patient and Community Care association (PAMALI).* University of Indonesia (UI); University of North Sumatra (USU); University of Andalas; Westat;** Prince of Songkla University; and Mahidol University from Thailand. Perdakhi Timika; Dekenat Jay￾awijaya; Dekenat Paniai: Yayasan Satu Karsa Karya(YSKK);*** Roman Catholic Diocese, Sorong. Project Objective To support the Indonesian National Tuberculosis Program (NTP) to mobilize commu￾nities in targeted areas to support improved TB care; and To advocate for increased TB resources and improved TB services. To decrease the burden of TB disease in vulnera￾ble populations, especially people living in urban slums, remote islands, post-disaster areas, poor and malnour￾ished people, contacts of TB and MDR-TB patients, and people living with HIV/AIDS. To strengthen the roles of civil society, the Church, and traditional leaders in TB con￾trol, as well as advocating for improved resources, services and TB policy in Papua. * PAMALI (Perkumpulan Masyarakat Pemerhati TB Indonesia) is an Indonesian NGO, established and managed by former TB patients, that champions for rights and improved services for TB patients. ** Westat is an employee-owned statistical survey research corporation based in Rockville, Maryland, United States. *** YSKK is an NGO that focuses on empowering marginalized communities to become self-sufficient and prosperous. USAID INDONESIA 23 MID-TERM PROGRAM PERFORMANCE EVALUATION ASPECT PARTNER LKNU JKM RCD Timika Main Activities TB case finding; (2) TB treat￾ment support; (3) Community Mobilization; (4) Advocacy. Additional activities: MDR-TB case finding; (2) MDR-TB treatment support; (3) Community Health Center quality assessment. (1) TB case finding; (2) TB treatment support; (3) Community Mobilization; (4) Advocacy. (1) TB case finding; (2) TB treatment support; (3) Community Mobilization; (4) Advocacy Key Performance Indicators Number of people with TB symptoms tested; (2) Number of smear positive; (3) Number of patients provided with sup￾port service; (4) Number of TB patients successfully completed DOTS. (1) Number of people with TB symptoms tested; (2) Number of smear positive; (3) Number of patients pro￾vided with support service; (4) Number of TB patients successfully completed DOTS. (1) Number of people with TB symptoms tested; (2) Number of smear positive; (3) Num￾ber of patients provided with support service; (4) Number of TB patients successfully completed DOTS. 24 USAID INDONESIA 25 MID-TERM PROGRAM PERFORMANCE EVALUATION CHAPTER TWO: PROJECT EVALUATION 2.1 EVALUATION PURPOSE As the CEPAT Program has entered mid-term for the implementing partners, USAID requested an external evaluator through the BANTU Project to conduct an independent mid-term evaluation to assess the project performance and its outcomes. The evaluation was expected to provide insights and important feedback to each of the partners and stakeholders that should assist them to understand both the strengths and areas where technical, administrative and management efforts could be improved. It would also provide evidence and learning for improving USAID/Indonesia program designs, strategies and policies. The specific objectives of the evaluation are: • To provide information on the outcomes made by each award in: (i) mobilizing communities to support improved TB care; and (ii) advocating for increased TB resources and improved TB services; including relevant issues, sustainability, and cost effectiveness; • To determine to what extent the USAID CEPAT Program is meeting the objectives; and what challenges, weaknesses, and lessons learned can be drawn from implementation of this program; • To examine whether implementation of programs like CEPAT contribute to the goals of the Indonesian government’s (national to district levels) policies and programs; and • To provide recommendations as the basis from which USAID can better target efforts, particularly in a decreased budget environment, so as to ensure that future efforts can make a big impact. 2.2 EVALUATION QUESTIONS The evaluation reviewed, analyzed, and evaluated CEPAT’s performance according to the nine questions: 1. To what extent have the program activities strengthened the National TB Program and the national level response to TB at the community level? 2. To what extent have the program activities increased TB detection, prevention and treatment in Indonesia, and been aligned with CEPAT goals and the TB CSO strategy for Indonesia? 3. To what extent has the program activity strengthened capacity of the national and subnational (province and district) government, private sector, community, and other stakeholders? 4. What is the contribution of each project to the overall USAID CEPAT program goals? 5. How replicable, adaptable/adoptable, and sustainable are the programs/program components? 6. How can the program design, management, and implementation become more efficient, effective and relevant toward achieving program goals? 7. How effective has the collaboration/coordination among the programs been in maximizing efforts and achieving greater results? 8. Does the CEPAT Program align with the country’s other community TB efforts (for example, Aisyiyah funded by Global Fund) and other community health efforts (that is, government-funded community health cadres and community organizations in other health areas in Indonesia) and, if not, what are the lessons to be learned and possibilities for improvement? 9. How can local and national ownership and future commitment to continued implementation of good practices/lessons learned be enhanced? 26 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION 2.3 METHODOLOGY To gather data required for this evaluation, the ET used several techniques that entailed a mix of mutually reinforcing qualitative methods and quantitative data that reflected the program logic and research questions being addressed. During the time span of the evaluation (April–May 2016), the team also utilized a document and literature review, qualitative data collection, and data analysis to capture the diversity of opinions and perceptions of stakeholders about CEPAT Program, including its strengths and weaknesses. The qualitative data collection included in-depth interviews (IDI) and focus group discussions (FGD). This captured information that informs the evaluation about the local context and enables the collection of examples that evidence the findings in greater detail. The ET conducted the evaluation in a participatory manner that involved the USAID Mission in Indonesia, as well as the implementing partners, LKNU, JKM and RCD Timika. The evaluation was conducted by team leader, Mr. Angus Walker, an expert on health promotion and community mobilization; and Ms. Isma N. Yusadiredja, a public health and evaluation specialist and supported by Ms. Dian Octorina as interpreter/translator. The team had sufficient expertise in evaluation, TB control, and the policy-making processes in Indonesia to be able to assess CEPAT Program and make recommendations regarding future program direction. The phases of the evaluation process are detailed below. A detailed evaluation schedule is included as Annex Two. 2.31 PHASE ONE: START-UP COORDINATION AND DESK REVIEW After the Scope of Work was finalized (see Annex Four), the ET conducted a thorough desktop review. The team spent almost one week reviewing available documents supplied by USAID. The document review also continued during the data collection period. The review included (but was not limited to) project reports, fact sheets and national documents. See Annex Three for a complete list of documents reviewed. The evaluation relied on secondary data to corroborate findings from IDIs and FGDs. Before the commencement of data collection, facilitated by BANTU, the ET attended a Team Planning Meeting (TPM) on April 14 and 15. The meeting aim was to introduce and familiarize all team members, coordinate logistics for field work, and determine the roles and responsibilities of team members. The meeting was attended by the USAID Indonesia Senior Infectious Disease Officer and the Contracting Officer Representative (COR). The meeting on April 15 was also attended by all Chief of Parties (COPs) of CEPAT’s implementing partners. After the TPM, the team then conducted an in-brief meeting on the afternoon of April 15 that was attended by the USAID Indonesia Head of Health Office, the Senior Infectious Disease Officer and the COR. During the in-brief, the team clarified expectations and discussed future utilization of the evaluation to ensure that the work plan was feasible and achievable within the time frame and to ensure that the evaluation was responsive to the Mission’s needs. The ET then finalized the work plan and all data collection instruments were then submitted to USAID Indonesia on April 18 before the launch of the field work component. 2.32 PHASE TWO: DATA COLLECTION The evaluation relied on two main sources of data: (i) a number of IDIs, FGDs and site visits as a primary source; and (ii) existing program documents and other relevant reports as a secondary source. The initial target sample of respondents included individuals and institutions who were directly involved with CEPAT. USAID then narrowed the list and selected priority individuals and institutions during the TPM. IDIs and FGDs included individuals or institutions that were CEPAT USAID INDONESIA 27 MID-TERM PROGRAM PERFORMANCE EVALUATION partners, subgrantees who are involved with CEPAT, NTP officials, local offices, Puskesmas, community and religious leaders, TB cadres and TB patients, other TB partners such as Challenge TB and Aisyiyah as well as USAID. Several individuals and institutions who were not grantees or partners but were regarded as key sources of information were also chosen to further ground the evaluation in the Indonesian context. The final list of informants, therefore, is a result of convenience and snow ball sampling. In-depth Interviews The ET conducted IDIs to triangulate data collected during the desk review and gain further insights into perceptions of the program’s implementation of CEPAT The ET conducted IDIs with institutions or individuals as follows: • CEPAT partners: LKNU, JKM, RCD. • Subgrantees: URC, Westat, University of Andalas. • NTP, the Ministry of Health. • District and Municipality Health Office. • Local government. • Religious and community leaders. • TB cadres. • TB patients. Most interviews were conducted in Bahasa Indonesia and each interview lasted approximately 45-60 minutes. All informants were informed about the agenda of the meeting and, with their consent, the interviews were audio-recorded. A Skype interview was conducted with a respondent who was unable to attend the scheduled IDI. This also has been included in the full list of IDI respondents in Annex Six. A total of 35 IDIs were conducted by the ET. In order to ensure that interview respondents felt as comfortable as possible, the ET, depending on the context and the population being addressed, strategically chose to have particular facilitators or individuals with similar demographic qualities administer or participate in the interviews. The semi- structured interview protocols and guidelines were finalized by the team after IDIs with USAID personnel. IDI questions and protocols are included in Annex Five. Focus Group Discussions A series of FGDs or Small Group Discussion (SGD) sessions were performed during the data collection phase of the evaluation. FGDs or SGDs were conducted mainly with CEPAT partners and TB cadres, depending on availability of the informants. A total of 16 FGDs were conducted. The discussions were held in Bahasa Indonesia and each session was completed in approximately 45-60 minutes. Each group discussion consisted of 5-12 respondents. Prior to the discussion, the participants were informed about the discussion objectives. A tape recorder was used to record the discussion, based on the informed consent of the participants. All FGDs were facilitated by ET members detailed above. FGD protocols and guidelines are included in Annex Five. Site Visits During the data collection, the ET visited several locations identified by USAID and CEPAT partners. The site visits provided the ET with an opportunity to collect in-depth information regarding the implementation of the CEPAT program in the respective sites. 28 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION On April 19-20, 2016, the ET visited Central Jakarta. The team conducted IDIs and FGDs with the following participants: • Municipality Health Office, Central Jakarta. • Puskesmas Tanah Abang staff. • LKNU Headquarters Management Team. • LKNU Central Jakarta Team. • TB cadres in Tanah Abang Subdistrict. From April 21-23, 2016, the ET visited Medan and Deli Serdang in North Sumatra Province to conduct separate IDIs and FGDs. The following key individuals and institutions were interviewed about the implementation of CEPAT programs in North Sumatra: • JKM Management Team. • Subgrantees: University of North Sumatra (USU), University of Andalas. • Westat (Skype). • District/Municipality Health Office in Medan and Deli Serdang. • Puskesmas staff. • Local government officials (subdistrict and village level). • TB Cadres. The ET then visited Cirebon District in West Java from April 25-26, 2016. During the field work, the ET visited the implementation of Desa Siaga (Alert Village) and Koalisi Masyarakat Peduli TB (Community Coalition for TB). • In Cirebon District the team visited key individuals and local officials: • LKNU Cirebon team. • District Health Office. • Village office. • Puskesmas. • Religious and community leaders. • TB cadres. The last field visits were conducted in Timika, Papua and Sorong, West Papua from April 29-May 4 2016. The team visited Timika from April 29-30, 2016 and Sorong from May 2-3, 2016 to meet the following individuals: • RCD Timika Management Team in Timika and Sorong. • Religious leaders. • TB cadres. • Puskesmas staff. USAID INDONESIA 29 MID-TERM PROGRAM PERFORMANCE EVALUATION In Timika and Sorong, the ET also visited Pos TB Desa (Village TB Post) for observation. All data collection methodologies considered the privacy and confidentiality of respondents; the ET also included gender-appropriate questions in all KII and FGD protocols. Both women and men were included in the sampling of stakeholder groups. Lastly, the ET ensured that KIIs and FGDs were conducted at times and places accessible to both men and women equally. After the field work phase was completed, the ET conducted internal briefings with USAID on May 9, 2016 and briefings with all implementing partners which was conducted on May 12, 2016. During the briefings, the team presented preliminary findings and conclusions. The team considered and collected all input from attendees to be incorporated in additional analysis and the final report. 2.33 PHASE THREE: DATA ANALYSIS The data collected for this evaluation contributed to establishing relevant information and findings related to the evaluation questions. In the evaluation work plan, the ET carefully detailed how data would align with each question. During data collection and analysis, the ET conducted debriefs at three-day intervals to prepare a raw data matrix including all IDI and FGD respondent answers to key themes related to evaluation questions. The debriefs and raw data matrix were conducted as part of critical field work analysis in order to discuss: (i) evidence collected, patterns, and discrepancies that helped answer the key themes emerging from the evaluation questions; and (ii) any adjustments that were needed in the evaluation schedule. In general, the themes identified during data collection revolved around the following to address the nine EQs: (1) Strengthening the NTP response to TB at the community level (EQ 1,2,3,4); (2) Implementation of health promotion strategies (EQ 2,3); (3) Capacity development of local stakeholders (EQ 3,7); (4) Strengthening CEPAT partners (EQ 7,8); and (5) Sustainability (EQ 5,6,9). Parallel analysis was used to analyze the evidence from both data sources: (i) IDIs, FGDs and site visits; and (ii) existing program documents and other relevant reports. In this analytical approach, each type of data for an evaluation question was first analyzed according to its respective source, and then across both data sources. For example, the team developed preliminary findings by first analyzing interviews with key informants; then formulated complementary preliminary findings from the key documents and other secondary materials. The overall approach of this evaluation relied on IDIs with a broad range of stakeholders as a primary data source. The ET wrote a summary for each completed IDI and FGD and then transferred responses to the raw data matrix. All respondent names were kept confidential during data collection and analysis. Pre-determined identification numbers were used on data collection forms (on IDI /FGD guidelines and field notes). The collected, primarily qualitative data was analyzed using appropriate qualitative analysis methods that allow for generating both “horizontal” analysis (across main themes to create the big picture) and “vertical” analysis (in-depth understanding of the most important issues). The ET began by discussing and developing codes or themes together, emphasizing the significance of consistency of analysis across different individuals. Given the large amount of information gathered, the ET utilized qualitative analysis matrices (in MS Excel) to handle all data with speed and accuracy. 30 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION 2.4 BIASES AND LIMITATIONS The following are evaluation biases and limitations that must be considered when reviewing this evaluation report: • Recall bias might have been present, such as informants responding to team questions with answers related to a different TB assistance program. To mitigate this potential bias, the team used the introduction time before each interview/FGD to re-emphasize the purpose of evaluation in contributing to community-based TB programs. • Response bias may also have been a problem. For example, respondents could only provide the interviewer positive remarks about an activity or experience in CEPAT because they would like to participate in additional funded projects. To mitigate this, the ET triangulated responses and considered a desk review and interviews across stakeholders before identifying the answer to an evaluation question. • Selection bias in the form of contacts provided by the implementer and USAID may have meant that the team only heard from people with positive experiences. The team mitigated this challenge, however, by including a meeting with each of the following non-CEPAT respondents: Aisyiyah, Challenge TB, GFATM and WHO. • Conformation bias may also have been present, meaning there may have been a tendency for respondents to readily accept conclusions that agreed with one’s beliefs, and discard conclusions that disagreed with them. This was of particular concern during FGDs, when FGD respondents with a lower social status usually tended to agree with beliefs expressed by individuals of a higher social status. To mitigate this potential bias, the ET did their best to ensure that FGDs in particular were made up of respondents of similar status. To minimize bias, the ET used multiple sources of data to triangulate on evaluation questions. By combining information found in documents or interviews from multiple sources, any single piece of biased data did not skew the analysis. In addition, the ET used direct observation based on nuanced knowledge of the evaluation environment to assist in identifying bias in responses. The team also faced the following limitations during the evaluation: • Availability of contacts. Interviews with GoI officials, mainly at the national level, were difficult to schedule because of existing demands on the individuals’ time or the need to accommodate last minute scheduling changes.7 To mitigate this concern, the team remained flexible to the extent possible in order to accommodate as many key informants and focus group participants as possible. • No baseline information available. In order to be able to measure performance, outcome and impacts or estimate changes in the program indicators, robust baseline information is required to compare with the data in this mid-term evaluation. The availability of appropriate baseline data is always critical for performance evaluation, as it is impossible to measure changes without reliable data on the situation before the intervention began. The ET has mitigated this issue by relying on secondary data from the national survey, however, sometimes the secondary data could not be gathered from the health service unit/subdistrict level, making it difficult to see changes due to the intervention in this level. In addition, the implementing partners did not conduct any Knowledge, Attitude, Practice (KAP) assessment prior to the intervention. The ET mitigated this issue by asking informants their qualitative perception of changes according to the participants. 7 The ET was unable to interview the current Head of TB Sub-Directorate, NTP from the Ministry of Health. To mitigate, the ET finally interviewed the former Head of TB Sub-Directorate, NTP who was involved at the commencement of CEPAT Program. USAID INDONESIA 31 MID-TERM PROGRAM PERFORMANCE EVALUATION • High rotation of government staff. The ET found that there has been high rotation of government staff, both at the national and local level. The new staff who were selected as respondents were not, sometimes, familiar with CEPAT interventions or any achievements resulting from CEPAT. To mitigate this issue, the ET interviewed or conducted group discussions with more than one staff at the government office. • Time allotted for evaluation. The ET spent almost four weeks in Jakarta and identified sites for field work. During this four-week period, the team had to assimilate data from KIIs, FGDs, as well as the desk studies and prepare an out-brief presentation. Given the scale and the scope of the CEPAT Program, this is a short time period to undertake an evaluation, and therefore only a limited number of sites were visited/observed. The above limitations, however, did not prevent the ET from gathering information and data needed to produce findings, conclusions, and recommendations for this evaluation. 2.5 OUTCOMES This evaluation report presents the findings by describing the CEPAT Program’s strengths and achievements attained throughout its implementation by the three CEPAT partners. Strengths and achievements are grouped with the following themes: • Strengthening the NTP response to TB at the community level; • Implementation of health promotion strategies; • Capacity development of local stakeholders; and • Strengthening CEPAT partners. Challenges encountered by the partners are also described in the findings section. Challenges are organised into themes of: • The supply side of health services; • Health promotion; • Management; and • Sustainability. The presentation of the findings through these themes encompasses the nine evaluation questions. The conclusion section represents the ET assessment of the findings and these are organised into four themes: • Strengthening the implementation of the NTP strategy at the district level; • Effectiveness and collaboration; • Sustainability; and • Lessons learnt. 32 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION Finally, the recommendation section of this report presents proposed ways forward to address key weaknesses and challenges that were discussed throughout the conclusions section. These are grouped into themes: • The supply side of government-provided TB health services; • Health promotion; • Management; • Sustainability; and • Future community-based TB programs. USAID INDONESIA 33 MID-TERM PROGRAM PERFORMANCE EVALUATION CHAPTER THREE: FINDINGS Since the CEPAT Program started in 2012, CEPAT partners have successfully established and implemented community-based TB projects across 25 high TB prevalence districts in seven provinces. CEPAT partner efforts and outcomes have contributed to the implementation of the NTP. The main improvement achieved has been increased community access to government-regulated TB services, particularly early TB diagnosis and treatment. The key factors identified for achieving this improvement are: 1. Mobilization of communities to support improved TB care, including: providing better access to TB services, improved service standards, increased community knowledge of TB, improved health￾seeking behavior, increased support provided to patients on treatment; and 2. Advocating for increased TB resources and improved TB services. Evaluation findings in this report are presented by: 1. Describing the CEPAT Program strengths and achievements with the delivery of community-based TB projects and activities; and 2. Detailing challenges that have been encountered during the delivery of CEPAT partners’ projects. 3.1 STRENGTHS AND ACHIEVEMENTS This evaluation found that the strengths and achievements of the CEPAT Program implemented by the three CEPAT partners were: 1. Strengthening the NTP response to TB at the community level; 2. Implementation of health promotion strategies; 3. Capacity development of local stakeholders; and 4. Gains achieved with community organisations and government engagement. 3.11 STRENGTHENING THE NTP RESPONSE TO TB AT THE COMMUNITY LEVEL Government health staff and other respondents (including cadres, patients, community leaders, and policy makers) reported that the program is still highly relevant to, and aligned with, the National TB Program Strategy. Puskesmas staff and cadres frequently described CEPAT partner projects as a means for enabling the Puskesmas to extend its services to reach into the community. This achievement has been attained despite challenges that limit the program success, including limited contributions and budget allocation by District Health Services, a lack of dedicated human resources at the district level, geographic challenges and factors associated with decentralization. “I thank CEPAT because, with its dedicated cadres, it helps my roles as TB programmer by bringing suspected TB cases to Puskesmas, as you know, we are [at Puskesmas] so overwhelmed with any other tasks…” (Puskesmas Staff, Cirebon, Focus Group Discussion) 34 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION Quantitatively, all CEPAT partners contributed to strengthening the NTP and two partners (LKNU and JKM) have already reached their project output targets. At the community level, the main impacts achieved as a result of CEPAT projects are: (i) significant increases in CDR in districts where projects are implemented; and (ii) expanding access to treatment, primarily through identification and subsequent testing of TB suspects, both of which are supported by community involvement (cadres). Accumulative data (that is, from project initiation to the most recent available data) from the three CEPAT partners is presented in Table 3-1 below. Based on reports from CEPAT partners, CDR achieved that are attributed to CEPAT partner’s projects, as a proportion of total districts CDR, are; LKNU 26 percent, JKM 30 percent and RCD 30 percent. Table 3‑1: CEPAT Project Achievements ACTIVITY LKNU JKM RCD TOTAL People with TB symptoms tested 10,979 17,794 7,949 36,722 Confirmed smear positive for TB 1,353 2,463 869 4,685 Patients who were provided with support services 3,036 2,463 484 5,983 Patients successfully completed DOTS 1,621 1,298 192 3,111 The ET found that CEPAT projects have been successful in mobilizing communities, through the recruitment of cadres, as a means of contributing to TB control. Since project initiation, up to the evaluation period, CEPAT has been able to mobilize 2,135 active cadres.8 Throughout the same period, the cadre retention rate among CEPAT partners varied between 50 to 70 percent.9 There are some reasons cited for the cadre turnover, such as locating paid employment (which was commonly reported in DKI Jakarta) or cadres relocating to other places which is common for those living in urban areas. It should also be noted that, with the exception of Papua and West Papua, incentive payments that are too low were never mentioned as the reason for cadre turnover. The strengths and achievements of CEPAT partner projects were consistently reported to the ET by district level respondents including Puskesmas service staff, DHO staff, cadres and patients. These reports included a broad range of achievements including greater awareness of TB (for example, transmission, symptoms, treatment) within the community, reduced stigma, improved CDR and treatment compliance. Other DHO staff were also highly appreciative of contributions and developments achieved by CEPAT because the project support resulted in significantly improved quality of TB care provided to TB patients and was not only focusing on quantity gains. 8 These numbers represent all active cadres listed by all CEPAT partners based on total cadres recruited to total cadres that remain active with their cadre roles. 9 JKM noted a higher retention rate. “ ...before CEPAT, the patient dropout rate was around 50 percent, but after CEPAT [was implemented] in Sorong, the dropout rate, I think reduced to at least 15 percent...” (DHO staff, Sorong In-depth Interview) USAID INDONESIA 35 MID-TERM PROGRAM PERFORMANCE EVALUATION The aforementioned quantitative data, detailed in Table 3-1, does contain some inconsistencies inherited from a lack of standardized indicator calculations across the three CEPAT partners. Furthermore, whilst there have been successes and achievements by all CEPAT partners, these were not consistent throughout all project locations. This was due to a range of reasons associated with differences in operating contexts such as capacity of health services, allocation of resources by DHOs and geographic factors. In response to the increasing number of MDR-TB patients in Indonesia, the CEPAT Program launched a training initiative to contribute to MDR-TB support. This initiative has achieved some success, most notably through the LKNU’s community-based activities in North and Central Jakarta. LKNU partnered with the Indonesian Pulmonologist Association (PDPI) and developed several MDR￾TB training modules for cadres. These modules aim to provide MDR-TB cadres with advanced skills and knowledge for better managing MDR-TB patients. CEPAT, with support from PETA (Pejuang Tangguh/ MDR-TB patient support group), have provided training support to Puskesmas staff and cadres. This training has upgraded their counselling skills for MDR-TB patients. The focus of the counselling skills upgrading is directed towards reducing MDR-TB patient losses. Despite some challenges during its implementation, project data indicates that LKNU provided support to MDR-TB patients in North and Central Jakarta with 87 MDR-TB patients being tracked and assessed. From this total, 61 MDR-TB patients started treatment and have received support. LKNU has been able to train and mobilize 24 MDR-TB supporting cadres. LKNU has also developed clinical capacity of some Puskesmas staff to treat MDR-TB patients. According to Puskesmas staff interviewed, the capacity building received helped improve staff communication skills for MDR￾TB patients, as it has been recognised that specific counselling skills are required to ensure MDR-TB patients comply with treatment. Other notable project achievements are located in Timika where RCD has succeeded in establishing 29 Pos TB Desa (Village TB Posts) facilities in Papua and West Papua. These facilities are essentially a simple shelter, however, they are adequate for drug storage, patient discussion and are prominently located in communities with TB signage. This results in improved health service access and, at the same time, could increase community empowerment. 3.12 IMPLEMENTATION OF HEALTH PROMOTION STRATEGIES Key officials interviewed at the national and local level admitted that health promotion was still one of the weaknesses of the TB control program in Indonesia. Limited staff providing health promotion was cited as the main weakness of health promotion strategies in TB control. According to Health Facility Research 2011 (MoH 2012), the total number of health promotion staff at Puskesmas level in Indonesia was only 4,144 or around 1.2 percent of the total number of Puskesmas staff (339,413). This level of human resource dedicated to health promotion at the Puskesmas level severely limits their ability to conduct effective health promotion campaigns without partner support. The Information, Education and Communication (IEC) materials available from the government were mostly in the form of pamphlets or brochures that are regarded as not particularly effective in Indonesia, especially in rural settings (Statistics Indonesia 2013).10 10 According to Indonesia Demographic and Health Survey (IDHS) 2012, only 13 percent of the population are exposed to print media, compared to 85.9 percent of respondents who are exposed to television. “..we have to admit that CEPAT has helped us in improving the quality of TB care provided to [TB] patients. The cadres supported the patients comprehensively from detection, and during treatment by giving motivations until the patients are completely cured. So it improves our treatment outcomes. Previously we only focused on quantity” (DHO Staff, Cirebon, In-depth Interview) 36 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION The three CEPAT partners have developed and implemented a comprehensive range of innovative approaches to health promotion. Existing communication channels have been utilized which have extended reach throughout the population and provide a cost-effective means to disseminate information, mobilize communities and to advocate for improved service delivery. There are some similarities in approach to health promotion across the three partners (for example, interpersonal approach, religious leaders’ engagement), however, each has tailored their approach to best suit their operational (geographic, cultural, socioeconomic) context. For example, LKNU utilizes social media as a means of disseminating TB health promotion material, while RCD is focused on interpersonal communication through cadres and other methods since social media is unavailable to most of the population in Papua and West Papua. Community members and patients frequently reported that they now know and understand, from being informed by cadres, about how TB is transmitted and how to receive treatment. These developments represent a high-value contribution to progressing the objectives of the NTP. They provide what is essentially the only source of reputable information to the public in the current operating environment of the three partners, where TB-focused health promotion provided by government health services is close to non-existent, (with the exception of interpersonal health worker to patient communication). Examples of specific partner activities are depicted in Table 3-2. Table 3‑2: Specific Partner Health Promotion Activities Partner Activities LKNU l Health education at school, madrasah (Islamic School), Pengajian (Koran Recital). l Utilized patient and cadres’ experiences to communicate with public through video via social media and print materials. l Using Nahdlatul Ulama (NU) patrons/leaders as the channel to improve TB awareness among community members. JKM l Utilized the playing of traditional music and dangdut music at community events, such as during World TB Day and to draw crowds. l Health education at Pengajian or at church masses. l Talk show on radio. RCD l Working together with the Bishop of Mimika to engage priests and other church personnel to promote TB knowledge and health-seeking behavior throughout subdistricts. 3.13 CAPACITY DEVELOPMENT OF LOCAL STAKEHOLDERS Based on documents reviewed and information provided to the ET during field work, it is evident that a broad range of capacity-building activity using a range of approaches has been provided to various stakeholders since the program’s inception. Information describing capacity building achieved is based on qualitative information and process indicators because there is no baseline data from which to measure. The following information details the main capacity-building progress achieved through the CEPAT Program with key stakeholders. “…because of cadres providing information, I know about the transmission of TB and how to be able to get cured. Now I know that TB can be cured” – (Respondent in Pulau Soop, Sorong, In-depth Interview) USAID INDONESIA 37 MID-TERM PROGRAM PERFORMANCE EVALUATION a. CEPAT Partners Capacity building of CEPAT partners has been an important component of the CEPAT Program to ensure the achievement of effective project management and compliance to standards. USAID has invested in capacity building of CEPAT partners throughout the program’s implementation through activities such as provincial visits to CEPAT project offices and district visits. An annual coordination meeting is also hosted by USAID, where training is delivered and CEPAT partners share achievements, innovations and challenges. Capacity building gained through CEPAT partners’ collaboration with their international partners, such as Westat and URC has developed the project management and technical competencies of JKM and LKNU. Westat also provided assistance to JKM to develop a range of best practice abstracts and presentations which JKM presented at several international conferences (Paris, Barcelona, Denver, Sydney). As it is the only partner that does not have a subgrantee that provides technical assistance, RCD reported that USAID provided significant technical assistance. This was highly valued as management initially was challenged with working under the Cooperative Agreement approach. Table 3-3 lists the main capacity-building activities and capacities developed that were provided to CEPAT partners throughout implementation. 38 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION Table 3‑3: Summary of Key Capacity-building Activities Provided to CEPAT Partners CEPAT Partner Capacity Developed By Who and How JKM Monitoring and evaluation. Administrative and general management. Financial management. TB technical. Database design. Technical writing skills. Westat: technical assistance. Communication. Project management including development of staff SOPs. Financial management processes including financial software. Development of exit strategy. USAID: annual coordination meeting and provincial visits. Financial management capacity of staff assessed. Capacity plan developed to address gaps. Financial management: external consultant. LKNU Monitoring and evaluation. HIV/TB co-infection and MDR-TB. Field operations management. URC: technical assistance, staff mentoring. Communication. Project management. Financial management processes including financial software. USAID: annual coordination meeting and provincial visits. RCD Management and administrative processes under the co-operative agreement approach. USAID: technical assistance. Communication. Project management. Financial management processes including financial software. USAID: annual CEPAT coordination meeting and provincial visits. b. Policy Makers CEPAT partners have developed the capacity of policy makers through various activities. These include advocacy efforts where information is disseminated and discussed with, for example, a Bupati or Head of DHO. Policy makers have been included in a range of workshops and events that expose them to TB information. Establishing good rapport and maintaining influential relationships with policy makers is a prerequisite for providing capacity building on an individual level and CEPAT partners have established relationships with many policy makers. LKNU has established a good relationship with the Sub-district Head (Camat) in Cirebon which resulted in the prioritization of TB as a health issue and LKNU has gained funding allocations for cadre activities. Likewise, RCD has established good working USAID INDONESIA 39 MID-TERM PROGRAM PERFORMANCE EVALUATION relationships with the Bupati in Paniai, West Papua which, in turn, has resulted in the allocation of funding for cadre support. Some DHO heads said they gain knowledge from CEPAT because they consider it a case study about how to implement successful community-based TB programs. c. Puskesmas Through its focus on the supply side of service provision and support to MDR-TB patients, LKNU has developed the capacity of Puskesmas to improve TB services. LKNU has supported Puskesmas in all its project areas by conducting an assessment of service quality (drug supply, functioning labs, staffing levels) and with the patient satisfaction survey. This aims to assess patient treatment from health staff. Issues identified through these assessments are flagged with the head of Puskesmas and then form the basis of advocacy by CEPAT for improvement. These are important steps in developing the capacity of Puskesmas facilities. Furthermore, LKNU has also developed the clinical capacity of some Puskesmas staff to treat MDR-TB patients. d. Cadres Capacity building of cadres has been achieved through the initial recruitment process and by training provided to them by CEPAT partners. This includes formal and on-the-job training that focused on basic health knowledge about TB, presentation skills, and contact tracing and administrative skills. In addition to recruiting and developing the capacity of cadres, RCD developed community infrastructure in the form of 29 community TB posts. JKM has provided training and information to cadres focused on a range of livelihood opportunities. The aim is to empower cadres to establish small business activities which will provide some income, some of which may be used to fund transport costs during their cadre work. LKNU has trained and mobilized 24 cadres with specialized counselling skills that enable them to effectively support MDR-TB patients. To support the implementation of the MDR-TB training, LKNU developed training modules. e. Communities Communities where CEPAT projects operated have been the recipients of a range of capacity￾building activities. Responses from many respondents, including community members, patients and cadres suggest that health literacy about TB increased and instances of stigma are less common than they were prior to the CEPAT Program. CEPAT partners have trained religious and informal leaders to enable them to promote TB prevention and encourage health-seeking behavior through their gatherings, communities and networks. 3.14 GAINS ACHIEVED WITH COMMUNITY ORGANIZATIONS AND GOVERNMENT ENGAGEMENT Although government engagement and commitment has been a significant issue reported by all CEPAT partners, some important achievements, albeit very limited, have been gained. For example, District TB Working Groups (reactivated Gerdunas) have been established in Cirebon and Pokja (Kelompok Kerja/Working Group) in Kediri, East Java. These working groups consist of representatives from district government (Health, Education, Community Empowerment, Bappeda),11 the PKK,12 the 11 Badan Perencanaan Pembangunan Daerah: The Regional Development Planning Agency. 12 Pembinaan Kesejahteraan Keluarga: Family Welfare Movement. 40 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION Secretariat Daerah (District Secretary) and hospitals. In Cirebon, for instance, this reactivated working group succeeded in securing commitments from private hospitals to improve their DOTS service. The development of working groups is a positive development, however, to ensure sustainability of such groups requires further work because they are essentially driven and coordinated by LKNU. TB Community Support Groups, (which include representatives from religious groups, youth groups, cadres, Puskesmas) have been established in subdistricts in Cirebon. The Camat has been instrumental in gaining commitment from local government to allocate funding for KMP13 activities. Other successes include accessing BOK (Bantuan Operasional Kesehatan/Health Operational Funding) by the head of the Puskesmas in Cirebon to fund cadre incentives, showing a strong commitment to improving community-based TB control. Advocacy efforts by RCD with the Bupati in Paniai, Papua, have resulted in the Bupati agreeing to allocate funds from the APBD (Anggaran Pendapatan dan Belanja Daerah/District Budget) to support community cadres (including TB cadres). Informal and constant advocacy efforts achieved an allocation of 500,000 Indonesian Rupiah per cadre per month. JKM has achieved several commitments for support with organizations, including alumni groups of USU, the private sector and government officials. Some informal community groups have also been established in Medan. 3.2 PROJECT IMPLEMENTATION CHALLENGES CEPAT partners faced numerous challenges during the implementation of their projects and which impacted on project outcomes and these included: 1. Supply side of health services - (patient treatment and laboratory supply); 2. Health promotion; 3. Management; and 4. Sustainability 3.21 SUPPLY SIDE OF HEALTH SERVICES – PATIENT TREATMENT AND LABORATORY SUPPLY A strong supply side of health services is essential for TB active case finding, however, respondents generally had very limited information to contribute regarding ways to improve it. Improving the demand side of health services is an aspect of service provision that most respondents seem to be focused on, given the large volume of information gleaned on this topic. Many respondents, primarily including health service workers, indicated that they did not consider it a factor that influences TB health-seeking behavior or treatment success rates. In addition to this, several Puskesmas staff stated that there was stigma within the workforce of hospitals, Posyandu (Integrated Health Post) and Puskesmas and that this negatively impacted 13 KMP: Koalisi Masyarakat Peduli TB: Community Coalition for TB. A CEPAT staff respondent in Medan suggested that “since TB patients are usually poor, they would be less likely to complain about any services that they receive, even when the treatment is poor”. USAID INDONESIA 41 MID-TERM PROGRAM PERFORMANCE EVALUATION on service provision. Numerous TB patients stated that they discontinued contact with the health service because of negative attitudes by staff towards them. Several patients interviewed provided information about their experience at health facilities and this included having to wait for in excess of four hours for treatment, as well as unfriendly and unhelpful staff. Many patient respondents cited such poor treatment at health facilities as reasons for them discontinuing treatment. Strategies to improve treatment of TB patients have included the development of the Patients’ Charter for Tuberculosis Care (PCTC) which outlines the Rights and Responsibilities of People with Tuberculosis. Documentation explaining the Charter and a copy of the Charter for display purposes had been circulated to all Puskesmas nationwide, however, when the ET asked about the Charter, Puskesmas staff usually responded that they knew of it but they had never displayed it. No copies of the Charter were visible at any health facilities visited by the ET. Progress has been achieved by LKNU to improve the supply side of health service provision to TB patients by conducting the Puskesmas Quality Assessment in every targeted district where they operate. Partnered with PAMALI, since the first year of the project, LKNU has initiated their assessment of TB patients satisfaction with the quality of services received. This PCA was expected to provide evidence of those needs and barriers encountered by patients receiving TB services. In general, health facilities that were assessed showed appreciation for this initiative. They regarded this approach as a means of providing important feedback from which they can improve their performance. Other partners (JKM and RCD), however, are still in the very early stage of adopting this approach. While this PCA was a good practice to improve health facilities’ performance on TB service, they still regarded it as a sensitive issue, particularly in their respective working areas. Issues of laboratory supply and logistics were reported to the ET whilst in Papua. The Wasor TB (Deputy Supervisor/TB Supervisor) in Sorong responded to a question about clinical issues and stated that forward planning of laboratory needs must be based on accurate data but this is not occurring. Insufficient laboratory supplies are sometimes experienced which prevent sputum samples from being tested. This, in turn, impacts on Puskesmas and community TB efforts–for example, when cadres successfully locate suspected TB patients for testing and the screening cannot be conducted because of insufficient laboratory supplies. 3.22 HEALTH PROMOTION The ET found that while significant progress has been achieved in the development and implementation of health promotion strategies and activities by CEPAT partners, improvements could be made that may have a greater impact and would increase the program evidence base generated through more thorough evaluation. For example, CEPAT staff respondents informed the ET that most health promotion activities implemented by the project had been developed based on a Some health workers suggested that TB patients are the problem or should be doing more for themselves. For example, “we wear masks for protection but the patients don’t”. (Mimika District, Papua) “We need to be very careful to introduce this [patient-centered] approach here. The Puskesmas staff would feel that they are being judged.” (CEPAT Partner, Focus Group Discussion) 42 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION broad understanding of community needs rather than being research based (that is, formative research) and that assessment of attributes such as community attitudes, knowledge and behaviors had not been conducted. This lack of formative research also affected evaluation of health promotion strategies since information generated through formative research provides baseline data. Most evaluation information generated from health promotion activities implemented was based on process indicators (for example, number of activities implemented, number of people who attended an event, quantity of IEC materials distributed). This is useful information, however, it lacks the depth of information that is required for impact and outcome evaluation. Evaluation to assess extent of achievement in terms of objectives such as message retention or behavior change is not being conducted by CEPAT partners. Most health promotion interventions that CEPAT partners had developed did not include fundamental planning components such as: analysis of the risk and contributing risk factors; clear measurable objectives related to behavioral change; or any indication of theoretical underpinning. The ET received information from cadres and Puskesmas staff regarding IEC materials that they thought were valuable for communicating information effectively. Several respondents indicated that flipcharts would be more helpful during patient discussions than brochures because flipcharts helped reinforce information to patients more effectively. Respondents explained that the main benefit with flipcharts is that they pictorially communicate ideas to patients and many of the patients have low literacy levels. Many respondents informed the ET that brochures and posters were not a very effective way to provide information to patients or community members and that interpersonal communication or information provided via FM radio was much more engaging. Unlike the comparatively abundant resources of the HIV/AIDS sector, well planned and implemented social marketing campaigns have not been implemented by CEPAT partners. Many village-level respondents (village leaders, cadres and patients) indicated that most people know about HIV/AIDS because they hear about it on the radio, whereas the source of TB information was not often recalled. When asked where they learned about TB, some respondents said that their knowledge was gained from information provided to them many years ago when they were at school. All CEPAT partners have utilized social media to varying extents and purposes. Objectives of social media use by CEPAT partners include promotion of the CSOs’ TB projects, dissemination of health promotion messages including TB treatment options and to illustrate the role of cadres. The ET found that strategic use of social media–for example, to provide planned and coordinated communication for project activities–is limited. LKNU is the only CEPAT partner that provided the ET with any social media monitoring data. Several cadres informed the ET that they have utilized their hand phones to communicate with patients via text messaging (SMS) and phone calls to check on wellbeing and treatment compliance. This communication was used by some cadres because it provides a simple, fast and economical way to check and remind patients to take medication. Some patients, who cadres provided support to, did not own hand phones or they lived in locations where they did not have telecommunications coverage. This form of communication did, therefore, have some limitations. 3.23 MANAGEMENT There were no issues that required modifications to project design of any CEPAT partner projects. There were, however, several issues that required project managers and other staff to adapt to changes in circumstances. These issues included overall complexity of the project (initial steep USAID INDONESIA 43 MID-TERM PROGRAM PERFORMANCE EVALUATION learning curve), funding and time constraints, and high turnover of staff, as experienced at RCD Timika. Management responses of all partners were very adaptive to challenges and changed circumstances. For example, JKM revised its output targets that were regarded as too ambitious in order to ensure that all outputs are achieved. Some aspects of projects, identified by the ET, could be improved–potentially leading to improved outcomes including: a. Information Management During the course of this evaluation, the ET identified weaknesses with information management that impact on CEPAT partners and that could be improved to support project development, attainment of project objectives and implementation of strategies, including: • Important information was not reaching policy makers about significant achievements gained during the implementation of CEPAT partner projects and activities. • Existing research relevant to TB program development is not applied to strategy development–that is, formative research and evaluation material from other implementing partners. b. Monitoring and Evaluation Processes Issues associated with monitoring and evaluation processes included: • CEPAT base line data was not established. This presents an issue for effective monitoring and evaluation of the program since there is limited data to act as a reference point. • Calculations of key performance indicators used by the CEPAT partners are inconsistent. This presents issues when comparison of performance information is sought. • Definition of terms such as active cadre is not uniform across the three CEPAT partners. • Definitions of key performance indicator calculations and terminology are not unified with MoH indicators. c. Collaboration with Partner Agencies The ET found that interagency collaboration between CEPAT partners and relevant TB stakeholders (including HIV/AIDS partners) at both the national level and local level was very limited, however, some progress had been achieved at the district level. JKM has collaborated with Challenge TB by establishing periodic meetings with the agency in Medan and through participation with various stakeholder meetings hosted in Jakarta by Challenge TB. RCD has collaborated with partners Challenge TB and the DHO with the aim of improving the capacity of cadres and health workers, ensuring treatment compliance for TB patients and improving the quality of TB DOTS services in Puskesmas. This initiative has been implemented in the district of Paniai but has been problematic in other project locations. During a KII with the Director TB Services at Challenge TB, the ET was informed that the respondent recognized that engagement between CEPAT and Challenge TB was weak and needed to be strengthened. The respondent stated that operational benefits would likely result from improving communication and collaboration, particularly at the district level. Reasons identified for the limited collaboration between CEPAT partners and other partner agencies included limited budgets to fund travel expenses, difficulties establishing mutually convenient timing suitable for meetings, limited reach of reporting by CEPAT partners and geographic locations of partner agencies. Mixed information was received by the ET from several 44 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION respondents regarding the level of collaboration between CEPAT partners and Aisyiyah.14 One respondent stated that collaboration was poor and that this resulted in programming inefficiencies, including duplication of services and poorly distributed project resources, particularly what was described as the disproportionate allocation of cadres to Puskesmas. CoPs of both CEPAT and Aisyiyah did not raise these issues when they participated in KII with the ET. The ET was informed by representatives from the WHO and the NTP during KII in Jakarta that they never receive updates or reports from CEPAT partners and hence, were unaware of the program’s achievements. Both respondents speculated that the reason for this was that CEPAT partners did not circulate their report to the respective agencies. A KII respondent who is an advisor to the GFATM informed the ET that national-level technical working groups were established for each GFATM global fund disease focus. These working groups evolved into technical forums which provide an opportunity to share ideas and information between stakeholders. CEPAT partners do not currently attend these technical forums. These forums may provide a valuable opportunity for CEPAT partners to liaise directly with other relevant stakeholders and to disseminate and access information. CEPAT and the NTP are clearly aligned and working towards the common vision of the National TB Strategy which is “toward a TB-free, healthy, just and self-reliant society.” While CEPAT extends the reach of the national-level TB response to the community level, the ET did not establish evidence of clear lines of engagement or communication between CEPAT partners with senior NTP managers at the national level. For example, it was evident that program heads at the NTP had not heard about RCD. This is quite remarkable given that RCD operate in provinces with the highest prevalence of TB and HIV/AIDS. It is reasonable to assume, therefore, that networking between CEPAT partners, particuarly RCD and heads of the NTP is very limited and needs to be improved. d. Collaboration Among CEPAT Partners Respondent information and the document review conducted indicated that that there was limited collaboration among CEPAT partners. An annual meeting with senior staff from the CEPAT partners is facilitated by USAID. At this event various training activities are conducted and partner achievements are shared through presentations. Activities such as sharing of expertise to address common project issues or to develop resources adaptable to all partners was not evident. 3.24 SUSTAINABILITY a. Commitment of Local Government Information gleaned during field visits consistently indicated that implementation of the National TB Strategy at the district level focuses mainly on passive case finding. Interviews with key informants and partners underscored the importance of community-based programs to achieve active case finding, however, in reality the respondents concurred that the capacity of the MoH to implement effective community programs was still extremely limited. It is also evident that donor-funded CSOs represent a major contribution to community-based TB activity that achieves active case finding. It should be noted that, as a consequence of decentralization, the success of government TB programs is highly dependent on individual district-level government capacity and budget allocation. This became evident to the ET during their visits to districts where, in some locations, no budget allocations for TB control programs existed. Another issue related to the decentralization was that many DHO heads have no public health background. It was also not uncommon that DHO heads 14 Aisyiyah is a faith-based organization, an autonomous body under the Muhammadiyah umbrella. Aisyiyah is currently a primary recipient of GFATM funding to implement a community-based TB program in Indonesia. USAID INDONESIA 45 MID-TERM PROGRAM PERFORMANCE EVALUATION were unaware of the existence of TB programs operating in their districts despite reporting from CEPAT project offices to DHOs. b. Local Knowledge About Alternative Funding Interviews and discussions with stakeholders at the village level indicated that knowledge about alternative funding opportunities available at the village level is very limited. For example, typical responses from informants at the local level when asked about sustainability and alternative funding options for cadre activities after CEPAT programs had expired resulted in three responses: (i) the activities will not be continued because there will be no funding; (ii) the government will somehow maintain CEPAT support by providing funding; and (iii) cadres will continue to conduct their work as before. There were no responses from respondents indicating definite identified alternative funding options or a means (such as advocacy) to attract alternative funding. c. Cadre Commitment The team observed and collected information from cadres (and other informants) that indicated a very high commitment to their cadre roles and activities. When asked what may happen when the CEPAT program finishes, most cadres responded that they would continue to do their work as best they can and that they hoped support from undefined sources would eventuate. 46 USAID INDONESIA 47 MID-TERM PROGRAM PERFORMANCE EVALUATION CHAPTER FOUR: CONCLUSIONS The CEPAT Program has supported the MoH to implement the NTP Strategy at the district level through the assistance provided by CEPAT partners. The approach taken by CEPAT partners has provided effective local solutions to increasing passive TB case finding. This has been achieved by the three CEPAT partners developing strategies to address their local challenges and to make use of existing opportunities and resources. This has involved mobilizing significant components of the community, including other CSOs, universities, government and individual citizens who have contributed their efforts to achieve the objectives of CEPAT Project. As the CEPAT Program progresses past mid-term and towards its end point, several important aspects have been identified through this evaluation that define its strengths, challenges and where it could be improved, particularly with a view to sustainability. Based on the findings of the ET’s research, the conclusions are presented under the following key themes: 1. Strengthening the implementation of the NTP strategy at the district level. 2. Improving effectiveness and collaboration. 3. Addressing sustainability. 4.1 STRENGTHENING THE IMPLEMENTATION OF THE NTP STRATEGY AT THE DISTRICT LEVEL 4.11 STRENGTHENING THE COMMUNITY-LEVEL RESPONSE TO TB CONTROL Since its establishment in 2012, the CEPAT Program has strengthened the NTP response at the district level across 25 high TB prevalence districts in seven provinces. Evidence gleaned by this evaluation indicates that the CEPAT Program has attained several high-value achievements. In terms of outcomes for communities and individuals, these achievements include having improved many peoples’ lives by linking them to treatment, enabling the successful treatment of over 2,000 patients with TB DOTS and is likely to have prevented the development of many cases of MDR-TB. LKNU, through its project in North and Central Jakarta, has achieved significant outcomes for comparatively large numbers of MDR-TB patients when compared with other CEPAT partners. LKNU has also developed the clinical capacity of some Puskesmas to treat MDR-TB patients. Other CEPAT partners, JKM and RCD provide support to MDR-TB patients when they are identified and are linked to cadres for support. One of the greatest potential values of the CEPAT Program, given the achievements made and positive statements received from a broad range of respondents, is likely to be the fact that the CEPAT projects can demonstrate good practice to policy makers that is achievable at relatively small economic outlay. 4.12 THE SUPPLY SIDE OF GOVERNMENT-PROVIDED TB HEALTH SERVICES The ET identified issues with the supply side of TB health services that impacted upon patient health-seeking behavior (poor patient treatment) and the efforts of personnel involved with 48 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION community-based TB programs–cadres and Puskesmas staff (laboratory and logistics in Papua). Many health staff interviewed did not consider that such issues affect health-seeking behavior, however, patients interviewed by the ET frequently raised this issue as the deciding factor in discontinuing treatment. In a positive development, LKNU has progressed work to address some aspects of poor patient treatment through the Puskesmas Quality Assessment in every targeted district that they operate in. Given that the supply side of government health services impacts active (and presumably passive) case finding, then it should be an issue that is addressed through CEPAT projects. Furthermore, improving the supply side of government health services links to both the NTP and CEPAT objectives. 4.13 CAPACITY BUILDING Capacity building across a broad range of stakeholders from community members to policy makers has been achieved throughout the implementation of the CEPAT Program since it was initially developed. Some technical competencies (for example, health promotion, monitoring and evaluation, management, and advocacy) of CEPAT partners could be improved through various methods, including mentoring, formal training, and the provision of relevant literature such as case studies. Investment in developing these capacities may help to strengthen project outcomes and provide evaluative research that could be utilized to refine existing activities and for advocacy purposes. The three CEPAT partners possess a broad range of high-level capacity in areas ranging from clinical skills to project management and lobbying. This existing capacity could be better utilized through a more collaborative approach across the partners. Valuable work could be developed using a collaborative approach to address common issues, develop shared resources (for example, advocacy materials) and to share specialized competencies of individuals. 4.2 IMPROVING EFFECTIVENESS AND COLLABORATION Information gleaned by the ET suggests that the CEPAT Program has been effective in achieving its objective of mobilizing communities to support improved TB care. Less successful, however, has been CEPAT partner efforts to increase TB resources and to improve TB services. The following information highlights aspects of the program where improvements in outcomes could be achieved. 4.21 HEALTH PROMOTION Health promotion, including community mobilization, forms a key component of the CEPAT Program. CEPAT partners have invested significant resources (financial and human) into health promotion strategies and activities which have been implemented across districts. These activities are very positive because they are often the only TB health promotion activities being implemented at the district level. There are, however, aspects of CEPAT health promotion activities that could be improved to increase impact. A lack of evidence-based planning of health promotion activities during the development phase was frequently indicated by CEPAT staff respondents. Base line information was also typically absent across the CEPAT partners’ health promotion plans. Determining this information prior to developing any plans or resources enables strategies and activities to be more strategic and tailored specifically to address key issues such as: (i) low levels of health-seeking behavior; (ii) the population segment most at risk; (iii) environmental factors such as overcrowding; or (iv) barriers such as stigma. Conducting USAID INDONESIA 49 MID-TERM PROGRAM PERFORMANCE EVALUATION formative research removes the ‘guess work’ from developing strategies based on assumptions or inaccurate information. For some behaviors such as TB health-seeking behavior, there may be no need to increase peoples’ awareness. People may already be informed or even convinced about the need to visit the Puskesmas when they have a persistent cough but barriers may exist that prevent the behavior from occurring (for example, a lack of confidence that patient information will remain confidential). An intervention may, therefore, need to address the issue of patient confidentiality rather than simply disseminating general information about TB. Properly conducted formative research would identify this barrier. This information would then be used to guide the development of an appropriate intervention. The CEPAT projects did not include effectively planned and implemented large-scale health communication campaigns. This presents a lost opportunity since such campaigns have the potential to reach a large proportion of CEPAT partner populations by utilizing a range of existing communication channels. Efforts to utilize these communication channels have already been established by partners– by liaising and training religious leaders. Planning for such campaigns needs to include assessment of resources available, audience segmentation, partner collaboration (especially HIV/AIDS partners) and possible impact on health services–for example, increased demand for TB testing and treatment resources. The potential population reach across districts that have affordable internet access and broad service coverage is significant. Effective utilization of social media can provide cost-effective, high￾speed communication support for health promotion, lobbying and advocacy purposes. A high-value aspect that social media presents to health stakeholders, such as the CEPAT partners, is the capability to provide two-way communication rather than pure dissemination of information. CEPAT partners implementing projects where there is good internet coverage and access to the population could improve their utilization of social media. There has been some good progress with the use of social media by LKNU and JKM, however, the reach of the platforms used is very limited based on monitoring information provided by LKNU. Social media presents a broad-reach communication channel to CEPAT partners at minimal cost to the CEPAT partners. M-Health–the term used to define mobile phone technology to support health initiatives– has a good ‘fit’ with TB programs, primarily because of the need to follow up with patients to aid treatment compliance. Whilst CEPAT implementing partners have not incorporated such communication technology into their strategies and activities, it is noteworthy that several cadres have utilized the practice, albeit in the most basic form, by communicating with patients via text messaging (SMS) from their mobile phones to check on their wellbeing and treatment compliance. 4.22 MANAGEMENT Effective management by CEPAT projects is demonstrated in this evaluation from a range of sources, including interviews with CEPAT project staff, COPs, MoH staff and through the document review. The ET did, however, identify some management aspects that could be improved which may result in significant gains in project performance. These include the following: i. Monitoring and Evaluation Processes As indicated in the findings, monitoring and evaluation processes across all CEPAT partners are a significant weakness. This impacts CEPAT partners in several ways, including: (i) insufficient data is available to quantify project and activity performance; (ii) challenges created when comparing performance accurately between partners; and (iii) communication with MoH about program performance is difficult given that indicators and terminology are not harmonized with MoH indicators. 50 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION ii. Collaboration with Partners Collaboration at the district level between CEPAT and relevant TB partners has occurred to a limited extent. Various reasons exist that present challenges to collaboration with partners, especially in Papua and West Papua. These include limited travel budgets, difficulties establishing mutually convenient schedules for meetings and geographic locations of partner agencies. It is probable that the lack of collaboration with relevant TB partners, especially HIV/AIDS agencies, represents a lost opportunity to share resources (for example, training opportunities, logistics), capitalize on project synergies and exchange information. At the national level, collaboration with the NTP and other partners is very limited. CEPAT partners who are based away from Jakarta, RCD and JKM, are challenged logistically. iii.Collaboration Among CEPAT Partners Collaboration among CEPAT partners does occur at a limited level, essentially by an annual meeting in Jakarta where CEPAT partners have the opportunity to present their project achievements. In the meantime, CEPAT partners are occupied with their large range of project activities and dealing with issues that arise from time to time. While this approach may seem appropriate to ‘getting on with the job’, it does present a missed opportunity to capitalize on the wealth of knowledge and expertise held by CEPAT partner COPs and other staff. Furthermore, while each CEPAT partner operates in environments that differ significantly from those of other CEPAT partners, there would undoubtedly be a range of issues, project tasks and resource development opportunities that are common to all organizations and could be worked on collectively by the partners. iv.Information Management Information management by CEPAT partners is an area that could be improved with limited resource investment to deliver potentially significant project gains. Key current issues include information dissemination to stakeholders and access to, and utilization of, information relevant to program development–such as existing evaluative research, case studies, and formative research. Centralized access to shared information among partners is also limited and an increase in this area would be of benefit. 4.3 SUSTAINABILITY Although it is too early to confirm the sustainability of this approach, the ET found notable achievements that may contribute to sustainability in the future. Importantly, all CEPAT partners–for example LKNU in Cirebon and RCD in Paniai (Papua)– have made progress in advocating to policy makers that has resulted in a level of commitment of funds, albeit limited. Unfortunately, these examples are not widespread and, given that the CEPAT Program has now been in progress for more than three years, it suggests that greater emphasis may need to be allocated by LKNU and JKM to advocacy and lobbying policy makers with the aim of expediting greater commitment and ownership of community-based TB programs. Broadening advocacy and lobbying strategy approaches to increase awareness of TB issues and community expectations of government commitment may contribute to achieving more widespread engagement from government. Given the high level of commitment and dedication demonstrated by cadres, it may be feasible to provide them with the option and capacity to extend their roles to encompass activities such as: (i) advocating for improved health services; (ii) supporting existing USAID INDONESIA 51 MID-TERM PROGRAM PERFORMANCE EVALUATION community-based organizations; and (iii) mobilizing stakeholders to establish new community-based organizations (possibly in partnership with HIV/AIDS community-based organizations). Whilst the current mechanism for delivering the program, by utilizing three discrete CSOs, has achieved effective service delivery and valuable project outcomes, there are some inherent weaknesses and inefficiencies associated with this approach. Such weaknesses and inefficiencies associated with the current approach include: • Variations and inconsistencies across the three CEPAT partner design parameters, performance indicators and reporting processes; • Reporting burden is more cumbersome compared to one agency or a consortium; • Challenges associated with sharing of resources, communication and knowledge between partners; • Diminished advocacy potential for influencing policy makers, particularly at the national level; • Inefficiencies associated with the division of financial resources across the three NGOs administrative systems compared to a consortium; • Development of advocacy and other program resources tend to be developed and utilized by each CEPAT partner on a solitary basis rather than collectively; and • Greater potential for competitive environment across the partners as opposed to a potentially more constructive collaborative environment. Alternative program delivery options, such as using a consortium of NGOs, may provide a more sustainable approach and one that can more easily be scaled up to encompass a greater number of districts. The consortium approach would also help alleviate some issues identified by the ET that are associated with the three different partners such as differences in capacity, resources, networks, and inconsistencies with monitoring and reporting processes. Networking and communication with key agencies such as the NTP and Aisyiyah would also be simplified and most likely strengthened using this approach. The current program delivery mechanism also inherits inefficiencies through its design and delivery mode in that it establishes systems parallel to those of the DHO. For example, Puskesmas located within CEPAT partner project areas already have cadres (with a generic health focus) recruited and mobilized in communities. The CEPAT partners have largely focused on recruitment and capacity development of additional cadres. While this has achieved significant gains with increasing passive case findings it has established a parallel system. The CEPAT Program may have been more productive and have achieved greater sustainable gains, if the program design focused entirely or in part, on strengthening the capacity of DHOs and Puskesmas so that these health facilities could manage and take greater ownership of the task of passive case finding. This approach also has potential value in that it could be scaled up with relative simplicity and speed across multiple districts, given the nationwide coverage of government health services. This approach would also be likely to facilitate more rapid distribution of resources and innovations (for example, revitalizing village TB posts, IEC materials, cadre training materials) to other districts compared to the current CEPAT Program. The CEPAT Program, thus far, has demonstrated through its partners’ achievements and innovations detailed in this evaluation that it can provide a ‘road map’ of how to implement successful case finding strategies, making it an asset that can be capitalized on for developing effective and sustainable programs. 52 USAID INDONESIA 53 MID-TERM PROGRAM PERFORMANCE EVALUATION CHAPTER FIVE: LESSONS LEARNT 5.1 CADRE SUPPORT Evidence collected during this evaluation indicated that cadre motivation and dedication to their work was consistently very high. Cadres, most of whom are female, consistently informed the research team that their primary reward for the work was the personal satisfaction gained by providing support to those in need. Monetary support, when provided, was considered as a practical necessity to fund transport and that such minor funding sometimes would not cover expenditure, particularly when greater distances of travel were involved. Several cadres informed the ET, sometimes with great emotion, that they did not seek additional money but wanted to be recognized for their effort. The lesson learnt is that, at the local level, there are people who are willing to become highly dedicated volunteers to support people who are less fortunate than themselves. 5.2 SOME WELL-INTENTIONED INTERVENTIONS NEED ADDITIONAL SUPPORT TO BE IMPLEMENTED Based on our interviews and observations at government health service facilities, some well￾intentioned interventions will not be implemented by staff or management without greater persuasion from TB stakeholders. For example, the PCTC that details the rights and responsibilities of TB patients was not observed to be displayed at any of the health facilities that the ET visited. When questioned about the Charter, staff typically stated that they had seen it but that it was never put on display. 5.3 SUPPLY SIDE OF HEALTH SERVICES IMPACTS ON HEALTH-SEEKING BEHAVIOR The ET learned from TB patients interviewed that patient treatment, provided by government health services, does impact on patients’ health-seeking behavior and compliance with TB treatment. Interviews with respondents from the supply side of service delivery, primarily Puskesmas staff, did not identify patient treatment as a barrier to health-seeking behavior or treatment. Patients, on the other hand, frequently provided information to the contrary. 5.4 CEPAT PARTNER PERFORMANCE Based on information sourced from informants from a diverse range of locations, including high￾density urban locations in Jakarta to villages located on islands in West Papua, the ET learnt that CEPAT partner performance is very much dependent on district and subdistrict attributes. For example, LKNU cadre retention rates are much lower in Jakarta compared with retention rates for subdistricts of Cirebon. ` 54 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION 5.5 DISTRICT HEALTH SERVICES Whilst CEPAT partners provide DHO with periodic activity reports and liaised frequently with DHO staff, particularly the Wasor TB, DHO heads were often oblivious to either the existence of CEPAT or to the TB activities that it was conducting in their districts and subdistricts. The lesson learnt is that reporting between organizations may occur horizontally, however, senior public servants and other policy makers require personal contact to ensure that information is received by them. The ET was advised by several CEPAT staff and other informants that DHO heads are often political appointments and that their time in the role is usually less than two years. USAID INDONESIA 55 MID-TERM PROGRAM PERFORMANCE EVALUATION CHAPTER SIX: RECOMMENDATIONS To ensure a comprehensive response to the challenges faced by CEPAT and other stakeholders, recommendations have been categorized into the five sections below. Given the limited time remaining for existing CEPAT projects to conclude, some of these recommendations may not, however, be feasible during the current project period but may be incorporated into future program and project designs. 1. The supply side of government-provided TB health services; 2. Health promotion; 3. Management; 4. Sustainability; and 5. Future program design and delivery. 6.1 SUPPLY SIDE OF GOVERNMENT-PROVIDED TB HEALTH SERVICES The ET recommends to CEPAT partners that greater emphasis needs to be placed on improving the supply side of TB health services. Recommendations for improving government health service supply-side issues at the district and subdistrict levels include two key aspects: (i) improving patient treatment; and (ii) addressing deficiencies associated with laboratory and logistics. a. Support Puskesmas/health care providers to improve PCA Patient-centeredness is one of the underlying components of the Stop TB Strategy Patient￾centered Strategy. It aims to enable partnership between the patient and provider, resulting in the best quality of care. It is, therefore, recommended that Puskesmas/health care providers: • Understand and utilize the PCTC as the first step in PCA. The Charter actively supports a patient’s right to proper and complete information about their disease and treatment. It leads to enhanced compliance of the patients to the treatment strategy. It is, therefore, recommended that Puskesmas health staff should understand that patients are not passive recipients of services but active partners. Patients should also understand their rights and responsibilities in TB care. Cadres could be utilized as the channel to inform TB patients about this Charter. • Encourage CEPAT partners to assess TB patients’ satisfaction with services by implementing the patient satisfaction surveys, as has been demonstrated by LKNU. Conducting such a survey has also been mandated by Ministry of Administrative and Bureaucratic Reform Decree No. 25/2009 (Permenpan 25/2009) on measuring public satisfaction by public services (including Puskesmas). There should also be a common understanding among CEPAT partners that treatment success also depends on the quality of TB care services. • Support Puskesmas staff to address issues identified from the surveys. The ultimate goal of the patient satisfaction survey is that public service should be able to identify and manage issues that arise. b. The ET recommends that CEPAT partners extend support to Puskesmas health care providers to: (i) proactively identify existing issues; and (ii) pre-empt future issues to ensure service continuity and quality is maintained. 56 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION To achieve this, CEPAT partners should consider and learn from the successes attained by CEPAT and other TB partners such as the progress achieved by RCD with their partnership approach to health facility checks. Ensuring that rapid notification of laboratory and logistics issues to the DHO, Provincial Health Office and to relevant project agencies (that is, Challenge TB) is a key action to ensure TB service continuity and quality. 6.2 HEALTH PROMOTION Several areas were identified where health promotion could be more effective and, therefore, the ET recommends the following actions to CEPAT partners. a. Establish a Body of Formative Research Formative research should be obtained that accurately defines attributes of the population (this may include health staff if they are the target of a behavioral change intervention) that CEPAT partners intend to engage with. This should establish information about the target population such as existing level of TB knowledge, beliefs, attitudes, where people receive health information, and barriers to adopting the desired TB health behavior. Existing formative research may already exist which should be utilized if it is relevant. b. Improve Health Promotion (Including Community Mobilization), Planning and Evaluation CEPAT partners’ health promotion activities should be evidence-based. This evidence could include, for example, formative research, evaluative research conducted in similar contexts, evidence of best practice approaches and it should have explicit and measurable behavioral objectives. An evaluation plan that measures strategy impact rather than just process should be developed as part of the planning process. Activities such as training cadres should include a pre-test and post-test to assess knowledge and skills gained. This information should then be managed so that it contributes to project monitoring and evaluation processes. The capacity of communication and community mobilization managers and staff should be assessed for gaps in knowledge and skills that may limit the quality of health promotion planning and evaluation. Capacity-building options that should be considered to address any gaps identified could include supervision from technically competent staff and development of planning templates that guide staff through a sequence of planning stages. c. Develop and Implement Broad-reaching Health Communication Campaigns The ET recommends that CEPAT partners develop and implement strategic health communication campaigns aimed at reaching large, vulnerable and high-risk populations across the districts that CEPAT partners operate in. Campaign planning should encompass analysis of target populations, partner collaboration (for example, HIV/AIDS), stakeholders, and the possible impact on health services (that is, increased demand for TB testing and treatment). Methods developed for campaigns should utilize integrated approaches–such as multiple coordinated activities implemented throughout the campaign period. One example could be a comprehensive campaign conducted over several months that may start or end on World TB Day. d. Review and Assess Supply of Effective IEC Materials It is recommended that CEPAT partners review their IEC materials with participation from intended users (Puskesmas staff, cadres, patients) for the purpose of identifying improvements that could be made USAID INDONESIA 57 MID-TERM PROGRAM PERFORMANCE EVALUATION or the need for additional materials. Information gained from the review can then guide the refinement of existing or development of alternative IEC materials. Formative research should be utilized to guide the development of all future IEC materials. In addition to reviewing IEC materials, supply and distribution systems for IEC materials should also be reviewed to identify any problems that need to be addressed. e. Increase Utilization of Social Media It is recommended that CEPAT partners, located where social media is viable, utilize this medium to greater advantage by: • Increasing community following (including policy makers) of social media sources by promoting the existence of CEPAT partner membership of relevant social media platforms, for example, by including reference to social media forums on IEC materials and email signatures; • Frequently updating social media platforms used by CEPAT partners with captivating content; • Monitoring social media activity by viewers; and • Providing greater integration of social media into project strategies. f. M-health for TB Community Programs CEPAT partners involved with future projects should consider utilizing M-health technology for community-based TB programs by conducting a feasibility study. The approach has been used by a range of agencies implementing health projects in Indonesia such as World Vision which is a partner implementing Posyandu Mobile Health, a child growth monitoring program in Jakarta and NTT. Various donors and agencies–such as WHO and the Stop TB Partnership–have piloted M-health for TB in other countries. Application of M-health for utilization in community-based TB programs should include: • disseminating health promotion messaging through short text messaging (SMS); • monitoring treatment compliance; • rapid dissemination of TB laboratory results; and • monitoring drug supply of cadres; 6.3 MANAGEMENT a. Improve Knowledge Management Practices The ET recommends that USAID provide support to CEPAT partners to improve knowledge management practices with the aim of improving organizational performance. The structure and architecture should ideally be online access provided to all CEPAT partners. Potential outcomes of improved knowledge management practices include: • achieving better project decisions, for example by accessing and using evaluation research from other agencies; • improving CEPAT partner organizational behaviors, for example by providing tools to support effective planning and evaluation development by project officers; • improving relationships, for example by providing access to case studies between CEPAT partners regarding best approaches for achieving effective advocacy to policy makers; 58 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION • sharing innovations and achieving collective learning, for example training documents and presentation notes supplied to a CEPAT partner from their international partner agency; and • improving shared access to resources such as advocacy materials (policy briefs) and health promotion materials. b. Monitoring and Evaluation Processes Improving the current CEPAT partner monitoring and evaluation processes were identified as a priority by the ET and it is, therefore, recommended that USAID support CEPAT partners expedite this project component by: • establishing a range of base line project information with the aim of supporting monitoring and evaluation, particularly for the end of CEPAT project evaluations; • establishing standardized calculations of key performance indicators to be used by all CEPAT partners so that project data is directly comparable; • standardizing definitions of terms such as ‘active cadre’ across the three CEPAT partners; and • aligning key performance indicator calculations and terminology with those of the MoH indicators. c. Improve Collaboration with Partner Agencies At the district level CEPAT partners need to establish greater collaboration with key TB partner agencies, for example Aisyiyah, MoH, Challenge TB and HIV/AIDS agencies. It is, therefore, recommended that periodic district coordination meetings be established to enhance collaboration and identify and address: (i) factors that may be affecting TB services (for example, laboratory supplies or scheduled health communications campaigns that may increase demand for TB treatment and screening services); and (ii) duplication of services. At the national level the ET recommends that CEPAT partners, particularly those who have frequent access to Jakarta, participate in relevant technical forums such as those established by the GFATM. Information such as minutes and presentations should be shared with CEPAT partners unable to attend. Rapport and relationship building needs to be established and maintained between CEPAT partners and heads of the NTP. In addition to this, it is recommended that USAID supports stakeholders to establish a technical working group focused on community-based TB services–community mobilization, health promotion, and cadre support. The purpose of the group would be to increase the implementation of, and strengthen, community-based TB strategies, monitor developments such as new partners and policy changes, and to facilitate the collaboration of partners so as to act collectively on shared interests, for example a media strategy. d. Improve Collaboration Among CEPAT Partners It is recommended that USAID take action to improve collaboration between CEPAT partners with the aim of capitalizing on collective knowledge, experience and resources by adapting a cluster approach rather than individual agency. Desired outcomes of this approach include setting common agendas, conducting studies/research/document best practices on particular issues. Objectives to achieve this aim should include: • Increasing CEPAT partner group communication focused on topics such as project progress, issues, innovative approaches, alternative funding options. This should be achieved by periodic teleconferences which would be facilitated by USAID three times a year. USAID INDONESIA 59 MID-TERM PROGRAM PERFORMANCE EVALUATION Change the format of the annual CEPAT partner meeting to a workshop format that: • supports topics progressed during the teleconferences; • delivers short but focused training content, that is, monitoring and evaluation; and • allocates time for CEPAT partners to jointly address issues and to develop new resources such as research briefs. • Change the frequency of the annual meeting/workshop to biannual. 6.4 SUSTAINABILITY The ET recommends that USAID support CEPAT partners to develop and progress strategies aimed at improving sustainability of gains achieved. Given the limited progress achieved to date and the evidence of poor commitment from government health services, greater support should be provided to CEPAT partners to address capacity gaps and improve advocacy efforts: a. Develop the capacity of CEPAT partners in advocacy techniques specific to their political contexts by: • providing training in advocacy techniques, including how policies are developed, government planning and budget cycles, and strategy development; • providing communications training that includes how to develop evidence and materials for advocacy (for example, policy briefs); • sharing CEPAT partners’ existing knowledge of what has and has not worked; • providing CEPAT partners with access to relevant case studies of successful advocacy strategies; and • utilizing guest speakers with successful advocacy backgrounds to provide training at CEPAT workshops. b. Utilize and develop existing village-level capacities to improve health services and achieve community-based TB program funding by providing the opportunity for cadres, and other relevant stakeholders, to engage in training that develops capacity in community-level governance topics including: • training that creates capacity about how to effectively advocate for improved health service delivery. This includes such methods as Citizen Voice and Action including dissemination of the PCTC, obtaining persuasive newsworthy information effectively and disseminating it through communication channels such as radio, social or mainstream media (cadres and cadre coordinators); • livelihood training for cadres (as implemented by JKM) which enables individual income generation that may be utilized to fund ongoing cadre support; • training that develops understanding of government processes–that is, forward planning regarding government planning/budget cycles (village leaders and cadre coordinators); and • dissemination of information to promote knowledge about available funding options and processes–such as the Musrenbang15 process and village funding (village leaders, cadres, and cadre coordinators). 15 Musrenbang (Musyawarah Perencanaan Pembangunan) is a deliberative annual process during which residents meet 60 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION c. Develop a media strategy that raises the profile of TB and positions the issue on the political agenda (national and subnational levels) with the aim of achieving greater resource allocation by policy makers. This would involve a long-term strategy that encompasses: (i) situation analysis of existing media coverage of TB-related information; (ii) monitoring of TB media content; and (iii) development of, and publishing/broadcasting, persuasive media content that complements existing TB strategies. Audiences and stakeholders would consist of MoH staff and management, partners in health such as religious leaders, NGOs, donors, politicians and public servants, national and provincial media councils and outlets, universities and researchers. Collaboration and coordination with TB￾related partners, such as HIV/AIDS stakeholders would strengthen the approach. 6.5 FUTURE PROGRAM DESIGN AND DELIVERY If the CEPAT Program is extended beyond its current timeframe, then several program design and delivery changes are recommended. These recommendations are made with the aim of scaling up the CEPAT program, broadening its geographic reach, improving efficiency and sustainability. a. Establish a consortium of NGOs that collaborate together rather than having discrete NGOs delivering individual projects. The consortium should: • be structured with a lead or managing agent model that coordinates and supports the implementing NGOs; • consist of member NGOs with established connections to government and a proven track record of performance. In addition to due diligence and capacity analysis, this should form a key component of the selection criteria so as to maximize results from lobbying efforts and influencing policy makers; and • utilize common program design and management elements including logframe, monitoring and evaluation indicators and reporting processes. b. Increase program focus on advocacy and capacity development rather than service provision. The consortium should be mandated to: • provide advocacy to policy makers (at both national and district levels) with the aim of increasing resources and commitment from government and other stakeholders to increasing the response to TB; • increase the capacity of district health services, CSOs and other actors so that they are better able to address the TB epidemic; • conduct capacity mapping and address identified capacity gaps of the consortium NGO members, CSOs, and DHOs with which it engages; and • utilize ‘Intermediate Organizations’ (also known as a ‘Civil Society Resource Organization’) as a mechanism to support local NGOs to develop their financial capabilities and diversify their funding sources (Winder 1998). c. Develop the capacity of district health facilities to enable government health services to better address the TB epidemic. With the support of consortium members, and where necessary together to discuss the issues facing their communities and decide upon priorities for short-term improvements. USAID INDONESIA 61 MID-TERM PROGRAM PERFORMANCE EVALUATION with advisor support, the program should work with DHOs to: • develop the capacity of district-based health facilities to recruit, train and mobilize health cadres with a focus on passive case finding; • improve the quality of government TB health services (that is, supply side issues, patient treatment, health promotion) by providing technical support and mentoring; • support provincial health offices and DHOs to develop TB-related advocacy and health promotion materials; and • raise the profile of TB within communities and policy makers. 62 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION ANNEX ONE: DISTRICT PARTICIPANTS IN CEPAT PROGRAM PROVINCE DISTRICT North Sumatra (JKM) Deli Serdang Medan Sedang Bedagai Tanjung Balai City West Sumatra (JKM) Padang Solok Tanah Datar Pasaman Padang Pariaman Kep. Mentawai West Java (LKNU) Cirebon Depok West Bandung East Java (LKNU) Blitar Kediri Tuban DKI Jakarta (LKNU, JKM) North Jakarta South Jakarta East Jakarta West Jakarta Papua (RCD) Timika City/Mimika District Paniai Nabire Jayawijaya West Papua (RCD) Sorong City USAID INDONESIA 63 MID-TERM PROGRAM PERFORMANCE EVALUATION ANNEX TWO: DATA COLLECTION SCHEDULE EVALUATION PROCESS SCHEDULE: PERFORMANCE EVALUATION OF CEPAT PROGRAM 64 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION ANNEX THREE: DOCUMENTS REVIEWED The following documents were reviewed for this evaluation: • CEPAT Factsheets: • CEPAT-JKM Factsheet • CEPAT-LKNU Factsheet • CEPAT-RCD • Infographic CEPAT-LKNU • JKM Final Work Plan • LKNU Final Work Plan • RCD Final Work Plan • JKM Annual Report Year-1 • JKM Annual Report Year-2 • JKM Annual Report Year-3 • LKNU Annual Report Year-1 • LKNU Annual Report Year-2 • LKNU Annual Report Year-3 • RCD Annual Report Year-1 • RCD Annual Report Year-2 • JKM Quarterly Report 1-Year 1 • JKM Quarterly Report 2-Year 1 • JKM Quarterly Report 3-Year 1 • JKM Quarterly Report 4-Year 1 • JKM Quarterly Report 1-Year 2 • JKM Quarterly Report 2-Year 2 • JKM Quarterly Report 3-Year 2 • JKM Quarterly Report 4-Year 2 • JKM Quarterly Report 1-Year 3 • JKM Quarterly Report 2-Year 3 • JKM Quarterly Report 3-Year 3 • JKM Quarterly Report 4-Year 3 • LKNU Quarterly Report 1-Year 1 USAID INDONESIA 65 MID-TERM PROGRAM PERFORMANCE EVALUATION • LKNU Quarterly Report 2-Year 1 • LKNU Quarterly Report 3-Year 1 • LKNU Quarterly Report 4-Year 1 • LKNU Quarterly Report 1-Year 2 • LKNU Quarterly Report 2-Year 2 • LKNU Quarterly Report 3-Year 2 • LKNU Quarterly Report 4-Year 2 • LKNU Quarterly Report 1-Year 3 • LKNU Quarterly Report 2-Year 3 • LKNU Quarterly Report 3-Year 3 • LKNU Quarterly Report 4-Year 3 • LKNU Quarterly Report 1-Year 4 • RCD Quarterly Report 1- Year 1 • RCD Quarterly Report 2- Year 1 • RCD Quarterly Report 4- Year 1 • RCD Quarterly Report 1- Year 2 • CEPAT-Request For Application Document, 2012 • Cooperative Agreement LKNU • Cooperative Agreement JKM • Cooperative Agreement RCD • CSO Strategy Indonesia • Indonesia National TB Control Strategy 2010-2014 • Indonesia National TB Control Strategy 2015-2019 • World Health Organization (WHO). 2006. The Stop TB Strategy: Building on and enhancing DOTS to meet the TB-related Millennium Development Goals. Geneva: World Health Organization. • United States Agency for International Development (USAID). 2014. Local Systems: A Framework For Supporting Sustained Development. Washington DC: USAID. • DQA Recommendations • World Health Organization (WHO). 2010. Global Tuberculosis Control 2010. Geneva. World Health Organization • World Health Organization (WHO). 2015. Global Tuberculosis Report 2015. Geneva: World Health Organization. • Collins, D.H. 2013. The Economic Case for Investing in TB in Indonesia. Medford, Massachusetts: Management Sciences for Health. • Statistics Indonesia (Badan Pusat Statistik—BPS), National Population and Family Planning Board (BKKBN), and Kementerian Kesehatan (Kemenkes—MoH), and ICF International. 2013. Indonesia Demographic and Health Survey 2012. Jakarta: Statistics Indonesia. 66 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION ANNEX FOUR: EVALUATION SCOPE OF WORK COMMUNITY EMPOWERMENT OF PEOPLE AGAINST TUBERCULOSIS (CEPAT) PROGRAM EVALUATION 1. DESCRIPTION OF PROJECT TO BE EVALUATED In 2012, USAID Indonesia launched the Community Empowerment of People Against Tuberculosis (CEPAT) in partnership with Indonesian National Tuberculosis Program (NTP) aiming to enhance technical capacity and skills and leading to improved TB detection and treatment services. The principal objective of CEPAT is to support the NTP in “achieving universal access to quality and early tuberculosis (TB) diagnosis and treatment through community empowerment.” To do this, the project uses two components: • Mobilizing communities to support improved TB care. Improved TB care includes: better access to and quality of TB services, increased community knowledge of TB, improved health-seeking behavior, and increased support to patients on treatment. • Advocating for increased TB resources and improved TB services. Advocacy will support the efforts of the NTP and demand resources and improved services for TB patients and families. Three cooperative agreements were awarded to three Indonesian non-government organizations, they are: Lembaga Kesehatan Nahdlatul Ulama (LKNU), Jaringan Kesehatan/Kesejahteraan Masyarakat (JKM) and Roman Catholic Diocese of Timika (RCD). The program is implemented in North Sumatra, West Sumatra, West Java, DKI Jakarta, East Java, Papua and West Papua provinces. Partnering with these NGOs, the program focuses on increasing people’s awareness on TB prevention and early detection, as well as fostering local commitment to support the national program. The total funding of all three awards is nearly US$10 million over the life of project. This year (2016), the program has entered its fourth year for LKNU and JKM, and its third year for RCD. Also in support of CEPAT, USAID will build the capacity of the Indonesian NGOs selected to carry out the project. The goal of the capacity building is to strengthen the NGO so that it can better contribute to the NTP’s strategic plan and CEPAT. In addition to this award, capacity building is provided by USAID based on each NGO’s needs and funded by USAID directly outside of the CEPAT agreements. CEPAT activities are targeted to improve TB care, access and services to the most vulnerable populations including people living in urban slums, remote islands, malnourished people, the poor, and contacts of TB patients. Partnership with communities affected by TB is a critical element of the overall national TB control activities in Indonesia. The CEPAT project has two main components, i.e. mobilizing communities to support improved TB care; and advocating for increased TB resources and improved services. As detailed in the RFA,16activities under Component 1 are expected to achieve the following results: a) Improved case finding; b) Increased access to service delivery; c) Improved health-seeking behavior to quality TB services; d) Increased knowledge and awareness of key TB information; 16 TB CEPAT RFA 497-11-0000055 dated December 6, 2011. USAID INDONESIA 67 MID-TERM PROGRAM PERFORMANCE EVALUATION e) Increased contact tracing of TB patients; and f) Improved and expanded community social support systems for patients during treatment. Thus, under the Component 2, the program is expected to: a) Increase the district budgets and allocated resources for TB; b) Increase advocacy at the local levels for TB services and resources; c) Increase the knowledge of TB among district leaders and other key stakeholders; and d) Reduce local policies inconsistent with national NTP guidance. Following is a short description of each of the CEPAT awards: 1. Lembaga Kesehatan Nahdlatul Ulama (LKNU). LKNU is an organization under the PBNU (the mother organization called Pengurus Besar Nahdlatul Ulama), was granted a CEPAT award in November 2012. Currently, LKNU is in its fourth year, and the program will be completed in November 2017. LKNU is working in a total of 10 districts in three provinces: DKI Jakarta, West Java, and East Java. LKNU trains and establishes community cadres to conduct contact-tracing, ensuring that TB suspects are referred to appropriate facilities for testing, and confirmed patients are linked to a Directly Observed Treatment Short-course (DOTS) health facility for treatment. Strong coordination established with the Puskesmas/local health centre diagnostic and treatment units, and LKNU cadres work to ensure follow up and provision of community support for TB patients at the community level. Advocacy is also one of LKNU’s objectives. LKNU is advocating for increased local budgets for health programs that have a specific section on TB, as well as advocating for strengthening local health regulations/policies that support the effectiveness of TB control program, and strengthening TB-related services among government and private health providers. By October 2015, LKNU has contributed 26% of districts achievement by assuring that 7,896 TB suspects were tested and treated, among which 10% were smear positive. The program also provides technical assistance in strengthening community groups and patient support groups to engage with local governments in TB control; and initiated new approaches for community support groups to ensure complete treatment of Multi Drug-Resistant (MDR) TB patients in collaboration with Persahabatan Hospital in Jakarta. 2. Jaringan Kesehatan/Kesejahteraan Masyarakat (JKM). JKM was the second local organization that received CEPAT award. The program was started in December 2012 and is expected to complete by end of 2017. JKM is operating in North Sumatra, West Sumatra and DKI Jakarta provinces. The program seeks to empower community ownership of TB programs as they play a key role in TB management, reducing burden on the health system, and reaching out to those who are not being treated or are marginalized. As per December 2015, JKM has trained more than 2,600 TB cadres in 14 districts in all three focus provinces. The program has detected more than 14,000 TB suspects with 1,767 people who are TB positive and put into treatment. 3. Roman Catholic Diocese (RCD) Timika. CEPAT award to RCD was granted on June 2013 and, similar to other CEPAT awards, the program will be implemented for five years until June 2018. The program is implemented in Sorong City, Timika/Mimika, Paniai, Nabire, and Jayawijaya districts. The program supports strengthening the roles of civil society, the Church and traditional leaders in TB control, as well as advocating for improved resources, services and TB policy in Papua. RCD entered its third year of implementation in October 2015, and since then the program has identified more than 5,000 suspected TB patients and 17 percent are TB confirmed. 68 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION In addition to the following information, USAID/Indonesia will provide the evaluation team with a package of briefing materials for each project including: • Program Documents; • Performance Management Plan (where available); • Quarterly/Annual Reports (where available); and • Work Plans; and other relevant document/reports. • Complete list of reading materials is provided in Annex 1. 2. BACKGROUND Tuberculosis (TB) is a highly contagious disease which is transmitted through the air. If left untreated, a person with active TB will infect about 10 to 15 people a year. Globally more than 2 billion people are infected with the bacteria which causes the TB disease. Ten percent of those infected will develop TB disease during their lifetime; the highest risk groups are those living in poverty, malnourished, or with a weakened immune system (i.e. HIV and other diseases). The vast majority of TB deaths are in the developing world, and more than half of all deaths occur in Asia. Indonesia is ranked fifth in global TB burden. In the past ten years, the National TB Program, with support from donors, has made remarkable progress.17 TB case detection rates increased rapidly from 22% in 2000 to 69% in 2008; but since 2010, it has remained steady or slightly declined due to under-notification, barriers in access to diagnosis and diagnostic delays. Treatment success rates improved from less than 50% in 1999 to almost 91% in 2007. A hospital DOTS (DOTS: Directly Observed Treatment Short-course) linkage program has been established in 168 hospitals and strengthened referral links and networks between hospitals and health centers in ten provinces. A national strategic plan for developing a laboratory network was recently finalized. Five laboratories successfully passed panel testing to receive accreditation for first and second line drug sensitivity testing and new cross-check laboratories were established in seven provinces. The spread of multi-drug resistant TB (MDR-TB) is a challenging problem, and the NTP is currently implementing programmatic management of drug resistant TB (PMDT) in several sites with plans to scale up to additional sites over the coming years. In Indonesia, the total number of estimated cases of MDR-TB in 2012 was 5,800 with the estimation of 1.9% of new TB cases and 12% of retreatment cases were estimated to be MDR-TB. At of the end of 2012, 428 cases of MDR-TB were confirmed and 426 cases were on treatment in Indonesia. There are still many other remaining challenges in the program. Thirty percent of the estimated TB cases are still not being detected and many patients are diagnosed late; the private sector is still providing poor quality or incomplete treatment; only a small portion of the estimated number of MDR-TB cases are on treatment; there are limitations in the health systems including laboratory quality, human resources, logistics and data quality; and community systems have limited capacity to increase public knowledge of TB, support improved TB diagnosis and treatment, provide adequate support for treatment, and increase demand for better quality TB services. This project will mainly focus on the last issue, which also intersects with all of the challenges above. The general Indonesian population continues to prefer to seek care in the private sector as well as hospital-based care. Most people perceive the quality of private health services and drugs to be better 17 Indonesia has an estimated total TB prevalence of 660,000 or 285 per 100,000 population and an estimated annual incidence of 430,000 new cases per year (189 per 100,000 population). Annual TB mortality is approximately 61,000 deaths. WHO Global Tuberculosis Control 2010. USAID INDONESIA 69 MID-TERM PROGRAM PERFORMANCE EVALUATION than the public sector and are willing to pay for the additional costs. TB is still stigmatized and the knowledge about the disease and treatment remains low. According to surveys conducted by the National Tuberculosis Program, only 21% of Indonesians are aware of the risks for TB or how to properly treat it (KAP Survey 2004). The most at-risk groups for TB do not have access to proper TB services. There is no comprehensive and coordinated approach to engage the community in supporting TB services including helping patients stay on treatment, or proactively identifying possible TB cases. NTP receives funding from the GOI, the Global Fund to Fight AIDS, Tuberculosis and Malaria (GFATM), and USAID. Indonesia was successfully awarded TB grants in the GFATM Rounds 1, 5, 8, and most recently Round 10, which will accelerate implementation of the NTP Strategic Plan 2010-2014 (“Breakthrough to Universal Access”). “Breakthrough to Universal Access” aims to reduce TB death and prevalence in order to contribute to achieving the health development goal of improving the public health status. Targets for TB control are set according to strategic plan of the Ministry of Health,18 and will be achieved through seven strategies, as follows: • Scaling up and improving quality DOTS service; • Addressing TB/HIV, MDR-TB, and the needs of poor and other vulnerable groups; • Engaging all public and private providers in implementation of International Standards for TB Care; • Empowering TB patients and communities; • Strengthening health system including Human Resources Development (HRD), and TB control program management; • Increasing commitment of central and local government; and • Enhancing research, development, and utilization of strategic information. These seven strategies are fully consistent with the Stop TB Strategy, which is strongly endorsed by USAID and in line with USAID’s strategy.19 To strengthen USAID’s present approach to TB assistance in Indonesia, and consistent with the Stop TB Strategy which emphasizes community partnership, CEPAT project was developed to be implemented through Indonesian NGOs. The community role in TB management is expected to build community ownership of TB programs, reduce the strain on the health system, and reach those infected with TB who are not being treated. USAID/Indonesia TB Program since 2000 USAID has supported the Indonesia National TB Program (NTP) since 2000. Initially, USAID’s support for Indonesia’s NTP was channelled through the TBCTA (Tuberculosis Coalition for Technical Assistance) Project, working closely with the NTP to implement DOTS in nine provinces at the primary health care level. TB control expanded rapidly in Indonesia between 2006 and 2010, reaching wider communities through expanded health facilities (hospitals, clinics, etc.) and integrated TB-HIV services. In 2000-2006, USAID’s Tuberculosis Control Assistance Project (TBCAP) was implemented in 11 provinces at district and provincial levels to address capacity building, quality of care, health system strengthening, integration, coordination and sustainability. A five-year follow-on program, called TB CARE, continued activities 18 To reduce the prevalence of TB from 235 per 100,000 population to 224 per 100,000 population. 19 More details on the Stop TB Strategy can be found at the WHO TB website: http://www.who.int/tb/en/ 70 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION since 2011. TB CARE provided a broad range of support to the NTP including: TB care and treatment (DOTS expansion and strengthening); expansion of programmatic management of drug resistant TB; TB/ HIV care and treatment; and health systems strengthening. In line with national strategy and to help Indonesia make significant progress in achieving the Millennium Development Goal for Combating AIDS, Malaria and other diseases (which include tuberculosis) by 2015. USAID programs will support the reduction of TB infection rates in populations most at risk and result in universal access of quality of TB care among all care providers. Project CEPAT will focus on mobilizing the community to achieve this objective. Current TB Program and Strategy Tuberculosis in Indonesia is a priority for USAID because of Indonesia’s large population, inequity in health indicators, and high TB burden. The Obama Administration’s Global Health Initiative (GHI) of 2009 reflects its strong commitment to global health. Core priorities within the GHI include increased attention to building health systems, private sector engagement, ensuring country leadership and investing in country-led plans. For TB, the GHI also includes ambitious target for increasing the number of patients on treatment and the number of MDR-TB patients diagnosed and on treatment. Additionally, The US government continues to be one of the largest supporters to the Global Fund to Fight AIDS, Tuberculosis, and Malaria (GFATM). The US contributes one-third of the total funding for GFATM and Indonesia is the recipient of substantial GFATM grants in all three disease areas. Embracing Global End TB Strategy, Indonesia aims to eliminate TB by 2050. As strategy to reach the goal of Indonesia free of Tuberculosis, aiming zero deaths, disease and suffering due to TB, National TB Program developed Five Years National Strategic Plan 2015-2019. The national strategy sets new and ambitious targets to recommit the Indonesian government, NTP, all of its partners and donors to a comprehensive strategy and sustained effort to eliminate TB. It is based on urgent need to reform our TB control approach and to respond to the gaps from current new prevalence data. Indonesia completed National Prevalence Survey in 2014. The survey revealed the increasing number of TB prevalence 2.4 times higher than previous survey. The estimation of TB prevalence 660/100,000 population or 1.6 million prevalent cases (0.65% of general population) with 1 million incidence cases every year. The gap between the actual number of TB cases that occur and those that are notified is significantly larger than previously estimated. At present, the gap between TB incidence and case notification, known as the ‘’missing cases’’, is approximately 680,000 TB cases per year, or more than two-thirds of the actual burden. This requires urgent action to prevent excess morbidity, mortality, and transmission from undiagnosed infectious cases in the community. USAID INDONESIA 71 MID-TERM PROGRAM PERFORMANCE EVALUATION The table below outlines the current USAID TB programs and their expected roles to the NTP Strategic Plan. IMPLEMENTING PARTNERS CEPAT CHALLENGE TB PQM JSI/DELIVER STRATEGIC APPROACH Increase notification rate X Ensure treatment success in hospitals and the private sector, as well as the public sector X X Increase coordination between TB and HIV/AIDS Programs, cross programs and cross sectors, in all levels to reduce the TB and HIV burdens in the community X X Ensure universal access to diagnosis and treatment for drug-resistant TB by 2018 X X Empowering TB patients and communities X Expand and strengthen the infrastructures, human resources and management process to implement the national strategy successfully X X Increasing commitment of the central and local government X X Expand and strengthen the data collection and surveil￾lance system to capture strategic information on TB cases diagnosed and treated in all sectors, critically analyze that information for program improvement X X 3. PURPOSE OF CEPAT PROGRAM EVALUATION The purpose of this evaluation is to assess the project performance and its impact. The evaluation will provide insights and important feedback to each of the partners and stakeholders that should assist them to understand both the strengths and areas where technical, administrative and management efforts could be improved. It will also provide evidence and learning for improving USAID/Indonesia program designs, strategies and policies. This evaluation therefore also will serve: • To provide information on the impact made by each award in: (1) mobilizing communities to support improved TB care; and (2) advocating for increased TB resources and improved TB services; including relevant issues, sustainability, and cost effectiveness; 72 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION • To determine to what extent the USAID CEPAT Program is meeting the objectives; and what challenges, weakness, and lessons learned can be drawn from implementation of this program; To examine whether implementation of these programs contributes to the goals of the Indonesian governmental (National – districts) policies and programs; and • To provide recommendations as the basis from which the USAID can better target efforts, particularly in a decreased budget environment, to ensure that our targeted effort can make a big impact. Audiences and Intended Uses The audience of the evaluation report will be the USAID/Indonesia Mission, specifically the Health Office Team, the USAID/Washington, CDC Indonesia, Government of Indonesia, and other donors. The executive summary, expanded executive summary, and final report will be provided to the GOI ministries and stakeholders. The evaluation will provide important feedback to each of the partners and information on both their strengths and weaknesses on the technical, administrative and management aspects. USAID Indonesia will integrate the evaluation’s recommendations to the future CEPAT Program activities and share lessons learned and best practices with the related stakeholders. Level of Measurements: Program Component The main focus of evaluation is: to assess and analyze the individual program award that comprise the USAID Indonesia CEPAT Program; and to determine the impact and progress towards the intended program goals, and examine synergy between programs and with other USAID supported TB programs particularly with TB CARE and Challenge TB. 4. EVALUATION QUESTIONS: 1. To what extent have the program activities strengthened the National TB Program and the national level response to TB at the community level? 2. To what extent have the program activities increased TB detection, prevention and treatment in Indonesia, and been aligned with CEPAT goals and the TB CSO strategy for Indonesia? 3. To what extent has the program activity strengthened capacity of the national and sub-national (province and district) government, private sector, community, and other stakeholders? 4. What is the contribution of each project to the overall USAID CEPAT program goals? 5. How replicable, adaptable/adoptable, and sustainable are the programs/program components? 6. How can the program design, management, and implementation become more efficient, effective and relevant toward achieving program goals? 7. How effective has the collaboration/coordination among the programs been in maximizing efforts and achieving greater results? 8. Does the CEPAT program align with the country’s other community TB efforts (e.g., Aisyiyah funded by Global Fund) and other community health efforts (i.e., government-funded community health cadres and community organizations in other health areas in Indonesia) and, if not, what are the lessons to be learned and possibilities for improvement? 9. How can local and national ownership and future commitment to continued implementation of good practices/lesson learned be enhanced? USAID INDONESIA 73 MID-TERM PROGRAM PERFORMANCE EVALUATION 5. METHODS 1. Draw on international and national literatures and related experiences; 2. Review background materials, including previous study(ies) conducted on each program component (if available), and program documentation; 3. Review the data; 4. Attend a team planning meeting (TPM); 5. Attend an evaluation launch meeting. This meeting will provide the platform for the evaluation a) to initiate discussions with implementing partners and stakeholders; b) to clarify the purpose and expected outcome of the review/evaluation; c) to ensure that implementing partners and evaluation team members are starting from the same frame of reference on the Indonesia situation; and d) to allow for an open and transparent discussion of USAID needs; 6. Conduct in-depth interviews, focus group discussion, semi-structured discussions, interview selected target(s) of program, meetings. 7. Conduct field visits (see attached Annex 2 for detailed proposed schedule): • North Sumatra • DKI Jakarta • West/East Java • Papua/West Papua In achieving the objectives, the evaluation team will have to apply different methods of data collection and analysis, including secondary data review and primary data collection using a combination of qualitative and quantitative methods. The analysis of the collected data/information will be done in a scientific way in order to provide evidence-based conclusions that are reliable, easily understood, useful and particularly applicable for USAID/Indonesia. Process The evaluation team will have to propose an appropriate evaluation methodology, including sample sizes for both quantitative and qualitative data collection; tools and steps for data collection and analysis, which will be reviewed and agreed by USAID before conducting the evaluation. The evaluation team will follow sound accounting procedures and be prudent in using the resources of the evaluation. The evaluation team will also follow a participatory and consultative approach ensuring close involvement of the Government, relevant program partners, and beneficiaries. The evaluation team will have home-based preparation for reviewing different documents and reports related to the program and developing the evaluation tools. The team will also have field work to collect relevant data/information through: i) meetings and discussions with relevant stakeholders, and the representatives of the program partners and beneficiaries; and ii) visiting program sites. Prior to the start of data collection, the evaluation team will develop and present, for USAID review and approval as part of the work plan, a data analysis plan that details, but not limited to, how focus group interviews (if deemed appropriate for the evaluation) will be transcribed and analyzed; what procedures will be used to analyze qualitative data from key informant and other stakeholder interviews; and how the evaluation will weigh and integrate qualitative data from these sources with project performing monitoring records to reach conclusions about the effectiveness and efficiency of the CEPAT projects and program. 74 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION It is anticipated that the evaluation team would have completed preparation (literature review and development of evaluation tools) prior to the field mission. The team shall us e the time during the field mission to collect and analyze data/information and consolidate main findings before conducting the debriefing meeting and final review workshop with stakeholders to present the preliminary results. The information collected will be analyzed by the Evaluation Team to identify correlations and determine the major issues. Data will be disaggregated, where possible, by gender to identify how program inputs are benefiting disadvantaged and advantaged groups. Interviews and Site Visits The Evaluation Team will conduct in-depth interviews and focus group discussions involving the user and community, with the following organizations/staff: • Government: Ministry of Health, including selected province and district health office, selected provincial and district government in Indonesia, and NTP. • Beneficiary: patient and community (e.g. volunteers/cadres, religious/local leaders). • Project staff: COP, Technical Officers, Provincial/District Coordinators. • Other stakeholders: relevant local CSOs, private sectors, universities, sub-awardees/grantees. • Proposed provinces for the site visits are: North Sumatra, DKI Jakarta, West/East Java, and Papua. The team is expected to visit sites as outlined in the suggested schedule (please see Annex 2). The Evaluation Team may be accompanied by a staff member from USAID/Indonesia, as appropriate, to observe interviews and field visits. A list of interviewees and key stakeholders will be provided by USAID prior to the assignment’s inception. 6. DELIVERABLES AND TIMELINE Evaluation Design and Work Plan: A Work Plan and Evaluation Design for the evaluation shall be completed by the Team Leader within two weeks of the award of the contract and presented to USAID/Indonesia. The evaluation design will include a detailed evaluation design matrix (including the key questions, the methods and data sources used to address each question), draft questionnaires and other data collection instruments, and known limitations to the evaluation design. The final design requires USAID/Indonesia approval. The work plan will include the anticipated schedule and logistical arrangements and delineate the roles and responsibilities of members of the evaluation team. Team Planning Meeting: A two-day Team Planning Meeting (TPM) will be held Jakarta at the outset of the evaluation. This meeting will allow USAID/Indonesia to discuss the purpose, expectations, and agenda of the assignment with the evaluation team. In addition, the team will: • Clarify team members’ roles and responsibilities; • Review and develop final evaluation questions; • Review and finalize the assignment timeline and share with USAID Indonesia; • Present and discuss data collection methods, instruments, tools and guidelines; and • Review and clarify any logistical and administrative procedures for the assignment. Methodology Plan: A written methodology and data analysis plan (evaluation design, data analysis steps and detail, and operational work plan will be prepared during the team planning meeting and discussed with USAID prior to implementation. USAID INDONESIA 75 MID-TERM PROGRAM PERFORMANCE EVALUATION List of Interviewees and Schedule: USAID/Indonesia will provide the Evaluation Team with a stakeholder analysis that includes an initial list of interviewees, from which the Evaluation Team can work to create a more comprehensive list. Prior to starting data collection, the Evaluation Team will provide USAID with a list of interviewees and a schedule for conducting the interviews. The Evaluation Team will continue to share updated lists of interviewees and schedules as meetings/interviews take place and informants are added to/deleted from the schedule. Data Collection Tools: Prior to starting field work, the Evaluation Team will share the data collection tools with the USAID Evaluation Program Manager for review, feedback and/or discussion and approval. In-briefing and Mid-term Brief with USAID: The Evaluation Team is expected to schedule and facilitate an in-briefing and mid-term briefing with USAID. At the in-brief, the Evaluation Team should have the list of interviewees and schedule prepared, along with the detailed chart mapping out the evaluation through the report drafting, feedback and final submission periods. At the mid-term brief, the Evaluation Team should provide USAID with a comprehensive status update on progress, challenges, and changes in scheduling/timeline. Discussion of Preliminary Draft Evaluation Report: The Evaluation Team will submit a preliminary outline and plan to finalize the assessment report to the USAID Evaluation Program Manager prior to final Mission debriefing. Debriefing with NTP: The team will present the major findings of the evaluation to NTP Indonesia through a PowerPoint presentation after submission of the draft report or outline and plan and before the team’s departure from country. The debriefing will include a discussion of achievements and issues as well as recommendations for the future activities, designs and implementation. Debriefing with USAID: The team will present the major findings of the evaluation to USAID Indonesia through a PowerPoint presentation after submission of the draft report or outline and plan and before the team’s departure from country. The debriefing will include a discussion of achievements and issues as well as recommendations for the future activities, designs and implementation. The team will consider USAID comments and revise the draft report accordingly, as appropriate. Debriefing with Partners: The team will present the major finding of the evaluation to the partners of USAID/Indonesia (as appropriate and as defined by USAID) through a PowerPoint presentation prior to the team’s departure from country. The debriefing will include a discussion of achievements and activities only, with no recommendations for future program. The team will consider partner comments and revise the draft report accordingly, as appropriate. Draft Evaluation Report: A draft report of the findings and recommendations should be submitted to the USAID Evaluation Program Manager prior to the team’s departure from Jakarta at the times outlined above. The written report should clearly describe findings and conclusions. Recommendations for future programming will be addressed in a separate internal memo. USAID will provide written comments on the draft report within 10 working days of receiving the document. Draft “Future Directions” Internal Memo: The Evaluation Team will prepare a draft internal USAID memo that focuses on “Future Directions,” with recommendations for future project designs. The intent of this memo is to provide USAID/Indonesia with procurement-sensitive information that cannot be distributed or shared with implementers or partners as part of the final evaluation report. USAID will provide comments within five working days of receiving the document. Final Report: The Evaluation Team will submit a final report that incorporates responses to Mission comments and suggestions no later than five working days after USAID/Indonesia provides written comments on the team’s draft evaluation report (see above). This report should not exceed 30 pages (for Indonesia) in length (not including appendices, lists of contacts, etc.). The format will include an executive summary, table of contents, glossary, methodology, findings, and conclusions. The report will 76 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION be submitted in English, electronically, and then disseminated within USAID/RDMA and/or USAID/ Indonesia for final approval. The report will be disseminated within USAID and to stakeholders according to the dissemination plan developed by USAID. Final “Future Directions” Internal Memo: The team will submit a final memo that incorporates responses to Mission comments and suggestions no later than three working days after USAID/ Indonesia provides written comments on the team’s draft memo (see above). Expanded Executive Summary: The team will submit an expanded executive summary to accompany the final report that will include a background summary on the evaluation purpose and methodology, and an overview of the main data points, findings, and conclusions. The expanded executive summary should be easy to read for wide distribution to local audiences and the partners are encouraged to look for creative presentation styles, formatting and means of dissemination. The expanded executive summary will be submitted in English, in hard copy (10 copies) and electronically. The report will be disseminated within USAID and to stakeholders according to the dissemination plan. Data Sets: All data instruments, data sets, presentations, meeting notes and final report for this evaluation will be presented to USAID on three (3) flash drives to the Evaluation Program Manager. All data on the flash drive will be in an unlocked, editable format. Reporting Guidelines: The final report should be a comprehensive analytical evidence-based evaluation report: • Detail and describe results, effects, constraints, and lessons learned from USAID CEPAT TB partners and other stakeholder-supported activities. • Identify gaps in CEPAT TB program implementation, including programmatic, leadership, funding, and geographic gaps. • Evaluate the level of coordination among CEPAT partners, between partners and host governments and other stakeholders. • Evaluate the level of sustainability/replication/adaptation of CEPAT program. • Provide recommendations and lessons on aspects related to factors that contributed to or hindered: attainment of program objectives, sustainability of program results, innovation, and replication. The report shall follow USAID branding procedures. An acceptable report will meet the following requirements as per USAID policy: • The evaluation report should represent a thoughtful, well-researched and well organized effort to objectively evaluate what worked in the project, what did not and why. • The evaluation report should address all evaluation questions included in the scope of work. • The evaluation report should include the scope of work as an Annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline shall be agreed upon in writing by the USAID Mission. • Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex to the final report. • Evaluation findings will assess outcomes and impacts using gender-disaggregated data. • Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable USAID INDONESIA 77 MID-TERM PROGRAM PERFORMANCE EVALUATION differences between comparator groups, etc.). • Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions. • Findings should be specific, concise and supported by strong quantitative or qualitative evidence. • Sources of information need to be properly identified and listed in an Annex, including a list of all individuals interviewed. • Recommendations need to be supported by a specific set of findings. Recommendations should be action-oriented, practical and specific, with defined responsibility for the action. • The annexes to the report shall include: • The Evaluation Scope of Work. • Any “statements of differences” regarding significant unresolved difference of opinion by funders, implementers, and/or members of the evaluation team. • All tools used in conducting the evaluation, such as questionnaires, checklists, survey instruments, and discussion guides. • Sources of information, properly identified and listed. • Disclosure of conflicts of interest forms for all evaluation team members, either attesting to a lack of conflict of interest or describing existing conflict of interest. Timeline: The total expected duration of the evaluation will be no more than 30 days. The team is authorized and expected to work a six-day week. Beside the actual field mission, members of the evaluation team are expected to work from their home-based offices and communicate among themselves and with USAID and other stakeholders electronically. The duration of assessment will be no more than 30 days, potentially starting from: April 11, 2016. The proposed provinces in Indonesia for the evaluation include: North Sumatra, West/East Java, and Papua. Estimated Length of Efforts: • Preparation, desk review, writing of a summary of desk review findings (Team): 3 days. • Meeting at USAID Indonesia, Jakarta, and other preparatory work: 2 days. • Finalization of methodology/work plans: 2 days. • Meeting with MoH, CSO and NTP for pre-departure site visit: 1-2 days. • Site visits/data collection in 3 provinces: 10-12 days. • Initial draft of Major Findings and Recommendations: 4 days. • Discussion of Preliminary Findings/Recommendations: 1 day. • Debriefing with USAID, CEPAT partners, and GOI: 1 day. • Draft of Evaluation Report, respond to comment with revised draft and finalization of report: 5 days which must be done by May 13, 2016. 7. TEAM COMPOSITION The areas of technical expertise shall be reflected on the composition of evaluation team to address the technical foci of the project being evaluated: 78 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION • Knowledge of TB prevention, control and treatment (but not limited to): • Standard of TB prevention, control and treatment, including DOTS, MDR-TB, and TB-HIV; • Behavior change communication; • Public private partnership; and • Community mobilization and development. • Knowledge of organizational development that includes (but not limited to): human resources development; financial, budgeting and management system; monitoring and evaluation; • Global, regional and national expertise; • Economics related to disease and disease control; and • Working experience in Indonesia. USAID encourages the participation of local experts on evaluation teams. USAID encourages participation of related respective government institutions or other stakeholders in Indonesia when their participation would be beneficial for skill development and not present a conflict of interest or a threat to validity, or their engagement in the evaluation would help to ensure the use of evaluation results within USAID. All attempts should be made for the team to be comprised of an equal number of male and female members. The Evaluation Team shall include technical specialists (one international and one local consultant) and one interpreter/translator: • These international consultants will cover areas targeted for evaluation as outlined previously; • The evaluation must include: one local technical expert with an excellent understanding of the Indonesian public health and health system and policy as well as USAID or development programs, who are fluent in English and excellent writing skill, and one translator/logistics support to assist the international consultant during interview and presentation, and to handle the travel-related logistics and provide administrative support to the Evaluation Team members; and • The Evaluation Team should also have adequate expertise in monitoring and evaluation. USAID INDONESIA 79 MID-TERM PROGRAM PERFORMANCE EVALUATION ANNEX FIVE: DATA COLLECTION INSTRUMENTS EVALUATION OF CEPAT PROGRAM Coversheet for In-depth Interviews Date of Interview: Interviewee Name, Title: Team: Time Start: Time End: Interviewer(s): Location: INTRODUCTION: (Note to interviewer - be sensitive with use of the term ‘TB CEPAT Program’, depending on the respondent) Good morning/afternoon and thank you for taking the time to speak with us today. As mentioned during our interview request, we are conducting an evaluation of the TB CEPAT Program. The evaluation will inform stakeholders of how CEPAT contributes to community-based TB strategies, will generate a broad range of lessons learnt from partners and will provide recommendations for future community￾based TB programs in Indonesia. Our team has had the opportunity to review some background documents to get a better sense of the design and implementation of the project, however, these documents can only tell us so much. We would like to discuss the project with you today to hear about your experience to help us better understand how this project looks and functions. The interview will last around 45 minutes. You can stop the interview at any time you wish. We will take notes of the interview and, with your consent, a recording will also be made using a digital voice recorder to aid us in completing our notes after the interview. CONFIDENTIALITY PROTOCOL • We will collect information on individuals’ names, organizations and positions, but those names and positions will not be associated with any particular findings or statements in the report. • We may include quotes from respondents in the evaluation report, but will not link individual names, organizations or personally identifiable information to those quotes, unless express written consent is granted by the respondent. Should the team desire to use a particular quote, photograph or identifiable information in the report, the evaluators will contact the respondent(s) for permission to do so. • All data gathered will be used for the sole purpose of this evaluation, and will not be shared with other audiences or used for any other purpose. • Your participation in this interview is voluntary and if you do not feel comfortable answering a particular question, please let us know and we will simply go on to the next question. Once again, thank you for taking the time to speak with us today. Do you have any questions for us before we get started? After the interview ends, thank the participant and explain to them that we might need to follow up with phone or email. 80 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION 1. INTERVIEW GUIDELINES FOR USAID INTRODUCTORY QUESTIONS: 1. How are you connected to the TB Program in Indonesia? 2. How long have you been involved and observed, particularly in the community-based TB Program? KEY QUESTIONS: Question set 1 EQ 1. To what extent have the program activities strengthened the National TB Program and the national level response to TB at community level? • Is the context analysis for the program still relevant and appropriate? [SOW EQ1] • Please tell us about the current implementation of National TB Program? [SOW EQ1] • In what ways does the program strategy contribute to the strategic goals of USAID? • What is the added value of CEPAT Program in the overall National TB Program? [SOW EQ1] Question set 2 EQ 2. To what extent have the program activities increased TB detection, prevention and treatment in Indonesia, and been aligned with CEPAT goals and the TB CSO strategy for Indonesia? • Could you briefly explain the achievement of each partner in regard of those performance indicators? [SOW EQ2] • What have been the main challenges and opportunities that might have affected the level of achievements of CEPAT Program objectives? [SOW EQ2] • How effective has been the monitoring system that was put in place to follow progress towards the desired results? Do changes need to be made to the monitoring systems? If so what? [SOW EQ2] Question set 3 EQ 3. To what extent has the program activity strengthened capacity of the national and sub-national (province and district) government, private sector, community and other stakeholders? • In your opinion, do the local partners have adequate capacity to achieve their program goals? [SOW EQ3] • Has the implementation of the CEPAT Program in any way developed the capacity of local partners to achieve their TB program goals or to become more involved with TB activities? If so, how? [SOW EQ3] Question set 4 EQ 8. Does the CEPAT Program align with the country’s other community TB efforts (e.g. Aisyiyah) and other community health efforts (i.e. government-funded community health cadres and community organization in other health areas in Indonesia?) • Could you tell us about community-based health programs assisted by other development partners USAID INDONESIA 81 MID-TERM PROGRAM PERFORMANCE EVALUATION in Indonesia that have relevance to TB prevention, control or treatment? [SOW EQ8] • Could you tell us about the characteristics of this (community-based program)? [SOW EQ8] Question set 5 EQ 7. How effective has the collaboration/coordination among the program been in maximizing efforts and achieving greater results. • Is there a purposeful synergy between the projects (partners) i.e. in what ways do the various projects complement each other in achieving goals of the program? (geographically, target groups, thematically, sharing of lessons learnt etc.) [SOW EQ7] l If there is little or no synergy, what can be done to strengthen linkages? Question set 6 EQ 9. How can local and national ownership and future commitment to continued implementation of good practices/lessons learned be enhanced? • To what extent do you think that community-based TB program development, achieved through CEPAT, will be sustained after USAID funding has finished? [SOW EQ9] • Is there a transition strategy developed for CEPAT partners to best manage cessation of USAID funding? If so, what would this entail? [SOW EQ9] OTHER QUESTIONS: (Not related directly with Evaluation Question at SOW, but it might be helpful to enrich the evaluation). • What could be some of the key lessons drawn from the Community-based TB Program? • Do you have any other comments, suggestions or concerns about the Community-based TB Program you work with that you would like to share with us? • Please describe the program’s gender inclusion efforts. How do project activities ensure participation by both men and women? 82 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION 2. INTERVIEW GUIDELINES FOR NTP, MOH INTRODUCTORY QUESTIONS: 1. How are you connected to the National TB Program? 2. How long have you been involved and observed, particularly in the Community-based TB Program? KEY QUESTIONS: Question set 1 EQ 1. To what extent have the program activities strengthened the National TB Program and the national level response to TB at community level? • Please tell us about the current implementation and the focus of National TB Program? [SOW EQ1] (Probe whether the respondent refers to the Community-based TB program.) • Is the context analysis for the program still relevant and appropriate? [SOW EQ1] • What is the added value of (Community-based TB program) in the overall National TB Program? [SOW EQ1] Question set 2 EQ 2. To what extent have the program activities increased TB detection, prevention and treatment in Indonesia, and been aligned with CEPAT goals and the TB CSO strategy for Indonesia? • Would you like to share some best practices related to TB detection, prevention and treatment in Indonesia? [SOW EQ2] • In your opinion what are the key factors that lead to this success? [SOW EQ2] • Do you see any factors that hampered the success / challenge the success? [SOW EQ2](Probe respondent whether below factors affect the success of TB control program.) l What about the decentralization? l What about Community Welfare / Health Network, Jaminan Kesehatan Nasional (JKN)? l According to you what have been the main challenges of involving CSOs in TB control program? [SOW EQ2] Question set 3 EQ 3. To what extent has the program activity strengthened capacity of the national and sub-national (province and district) government, private sector, community and other stakeholders? • According to you, do local partners have adequate skills or capacities to contribute to the achievement of the (national) program goals? Has the CEPAT program approach in any way enhanced the effectiveness of local partners to achieve their TB program objectives/goals? [SOW EQ3] • Does the CEPAT partner have adequate capacity to be able to influence local and national authorities to strengthen policies related to improving community-based TB control programs? For example, improving access to TB services? [SOW EQ3] USAID INDONESIA 83 MID-TERM PROGRAM PERFORMANCE EVALUATION Question set 4 EQ 8. Does the CEPAT Program align with the country’s other community TB efforts (e.g. Aisyiyah) and other community health efforts (i.e. government-funded community health cadres and community organization in other health areas in Indonesia? • Could you tell us about other community’s TB efforts projects in Indonesia? [SOW EQ8] l What do you think about that? (Probe: do they complement each other? Or compete?) • Could you share the strength and weakness of each community’s TB efforts? [SOW EQ4] Question set 5 EQ5. How replicable, adaptable, and sustainable are the program’s/program components? EQ 9. How can local and national ownership and future commitment to continued implementation of good practices/lessons learned be enhanced? • Do you think that the Community-based TB Program will sustain? (Or to what extent that you see the Community-based TB Program will sustain.) [SOW EQ5] • Should any Community-based TB Program assisted by external funding come to an end, do you have any plans to ensure that this approach will be sustained? [SOW EQ9] OTHER QUESTIONS: • What could be some of the key lessons drawn from the Community-based TB Program? • Do you have any other comments, suggestions or concerns about the Community-based TB Program you work with that you would like to share with us? 84 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION 3. INTERVIEW GUIDELINES FOR DISTRICT HEALTH OFFICE INTRODUCTORY QUESTIONS: 1. How are you connected to the TB Program (roles and position)? 2. How long have you been involved and observed, particularly in the Community-based TB Program? 3. Could you tell us about the current TB situation in your district? (Epidemiological status, community empowerment, etc.) KEY QUESTIONS: Question set 1 EQ 1. To what extent have the program activities strengthened the National TB Program and the national level response to TB at community level? • Please tell us about the current implementation and the focus of National TB Program? (Probe whether the respondent mentions about community-based TB program.) [SOW EQ1] • Is the context analysis for the program still relevant and appropriate? [SOW EQ1] • What is the added value of (Community-based TB Program) in the overall National TB Program? [SOW EQ1] Question set 2 EQ 2. To what extent have the program activities increased TB detection, prevention and treatment in Indonesia, and been aligned with CEPAT goals and the TB CSO strategy for Indonesia? • Would you share some success stories regarding TB control program in your district? [SOW EQ2] • Do you see any factors that hampered the success / challenge the success? [SOW EQ2](Probe respondent whether below factors affect the success of TB control program.) l What about the decentralization? l What about JKN? • According to you what have been the main challenges of involving CSOs in TB control program? [SOW EQ2] Question set 3 EQ 3. To what extent has the program activity strengthened capacity of the national and sub-national (province and district) government, private sector, community and other stakeholders? • Did you gain new knowledge about the community-based TB program after the involvement with CEPAT programs? [SOW EQ3] • Has the program approach in any way enhanced the effectiveness of the district to achieve its goals on TB program? (Probe on the technical assistance or capacity building they received from CEPAT or any other partners in the district.) [SOW EQ3] USAID INDONESIA 85 MID-TERM PROGRAM PERFORMANCE EVALUATION Question set 4 EQ 4. What is the contribution of each project to the overall USAID CEPAT program goals? • Could you elaborate more about the contribution of CEPAT Program in your district? [SOW EQ4] Question set 5 EQ 8. Does the CEPAT Program align with the country’s other community TB efforts (e.g. Aisyiyah) and other community health efforts (i.e. government-funded community health cadres and community organization in other health areas in Indonesia? • Could you tell us (if any) about other community’s TB efforts projects in your district? [SOW EQ8] • What do you think about that? (Probe: Do they complement each other? Or compete?) [SOW EQ8] • Could you share the strength and weakness of each community’s TB efforts? [SOW EQ3] Question set 6 EQ 9. How can local and national ownership and future commitment to continued implementation of good practices/lessons learned be enhanced? • How can local and national ownership and future commitment to continued implementation of good practices/lessons learned be enhanced? [SOW EQ9] Question set 7 EQ 5. How replicable, adaptable/adoptable and sustainable are the program’s components? EQ 9. How can local and national ownership and future commitment to continued implementation of good practices/lessons learned be enhanced? • Could you tell us about the commitment of the district on the TB program in general? In what ways? [SOW EQ9] • Do you think that the Community-based TB Program will sustain? (Or to what extent that you see the Community-based TB Program will sustain.)[SOW EQ5] • Should any Community-based TB Program assisted by external funding come to an end, do you have any plans to ensure that this approach will be sustained? [SOW EQ5&9] OTHER QUESTIONS: • What could be some of the key lessons drawn from the Community-based TB Program? • Do you have any other comments, suggestions or concerns about the Community-based TB Program you work with that you would like to share with us? 86 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION 4. INTERVIEW GUIDELINES FOR CEPAT GRANTEES (LKNU, RCD, JKM) INTRODUCTORY QUESTIONS: The questions will be asked depending on the role of the participant in her/his organization: 1. Describe your role in your organization. How long have you been in your position? 2. Please describe your organization’s vision and mission. 3. Could you explain the involvement of your organization with CEPAT (type of engagement, activities conducted)? KEY QUESTIONS: Question set 1 EQ 1. To what extent have the program activities strengthened the National TB Program and the national level response to TB at community level? • Please tell us about the current implementation and the focus of National TB program? (The quality of National TB Program, etc.)[SOW EQ1] • Is the context analysis for the program still relevant and appropriate? [SOW EQ1] • In your own words, what is the objective or goal of CEPAT Program you are engaged with? [SOW EQ1] • What do you think is the added value of CEPAT Program in the overall National TB Program? [SOW EQ1] Question set 2 EQ 4. What is the contribution of each project to the overall USAID CEPAT Program goals? • Were there other achievements of this program that you want to mention? (related with key indicators.)[SOW EQ4] Question set 3 EQ 2. To what extent have the program activities increased TB detection, prevention and treatment in Indonesia, and been aligned with CEPAT goals and the TB CSO strategy for Indonesia? • According to you what have been the main challenges of involving communities in the TB control program? [SOW EQ2] • In your work in CEPAT Program, what were the challenges you faced as an organization? [SOW EQ 2] How did you deal with these difficulties? l What are the constraining factors regarding effort in national TB control program? l What are the difficulties of working with Indonesian policy makers at national level? l What are the difficulties of working with Indonesian policy makers at local level? • Regarding the achievements that you mentioned above, what led to this success? What hampered the success? [SOW EQ2] • Do you see JKN implementation affect the achievements that you mention above? USAID INDONESIA 87 MID-TERM PROGRAM PERFORMANCE EVALUATION • Do you see decentralization also affect the achievements that you mention? l Why do you think there are variations of achievements across districts? [SOW EQ2] • How did you deal with sensitive issues which may involve various actors (i.e. NTP, other development partners or CSOs, local government, private providers)? [SOW EQ2] Question set 4 EQ 3. To what extent has the program activity strengthened capacity of the national and sub-national (province and district) government, private sector, community and other stakeholders? • What are the most significant changes in the Community-based TB Program at the national and local level that you are involved with? [SOW EQ3] • Have any practices been introduced to the Community-based TB Program as the result of CEPAT Program/approach? [SOW EQ3] • How did your organization build partnerships with policy makers at national and local level? [SOW EQ3] • According to you, do local partners have adequate skills or capacities to contribute to the achievement of the (national) program goals? Has the CEPAT Program approach in any way enhanced the effectiveness of local partners to achieve their TB program objectives/goals? [SOW EQ3] • Does your organization have adequate capacity to be able to influence local and national authorities to strengthen policies related to improving community-based TB control programs? [SOW EQ3] Question set 5 EQ 8. Does the CEPAT Program align with the country’s other community TB efforts (e.g. Aisyiyah) and other community health efforts (i.e. government-funded community health cadres and community organization in other health areas in Indonesia? • Could you tell us about other community’s TB efforts projects in Indonesia? [SOW EQ8] • What do you think about that? (Probe: do they complement each other? Or compete?) [SOW EQ8] Question set 6 EQ 6. How has program design, management and implementation among the programs been in maximizing efforts and achieving greater results? • Do you think that the budget you received from USAID was sufficient for the activities you were asked to complete/impact you were asked to have? [SOW EQ6] • Is there anything that you wish USAID did differently? In which aspect? [SOW EQ6] • Were there any items in the contract that burdened/challenged the completion of the program? [SOW EQ6] • How responsive is USAID to the needs identified by your organization? [SOW EQ6] • Does the support affect in a positive way your capacity to implement its project? [SOW EQ6] • Has USAID and the program approach contributed to the organizational strengthening of your organization? In what ways has this been manifested? [SOW EQ6] • What were the most beneficial institutional support activities received by your organization from USAID? 88 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION l What type of support did you want more of? • Does your organization have the right or adequate skills to contribute to the achievement of the program goals? Has the program approach in any way enhanced the effectiveness of the programs? In what ways? [SOW EQ6] • Could you tell us about the relationship with your subgrantees? (What kind of agreement, what roles do the subgrantees have in this CEPAT Program?) [SOW EQ6] • In what ways do the subgrantees support your activities/work in CEPAT Program? [SOW EQ6] Question set 7 EQ 7. How effective has the collaboration/coordination among the programs been in maximizing the efforts and achieving greater results? • How do you see the coordination/collaboration among CEPAT programs? In what ways? [SOW EQ7] • Has the project stimulated relations between partners in any significant ways? Has this in turn had an effect on the project? In what ways? [SOW EQ7] • Do you have any suggestions or comments about the collaboration to maximize the efforts and achieve greater results? [SOW EQ7] Question set 8 EQ 9. How can local and national ownership and future commitment to continued implementation of good practices/lessons learned be enhanced? • How can local and national ownership and future commitment to continued implementation of good practices/lessons learned be enhanced? [SOW EQ9] Question set 9 EQ 5. How replicable, applicable/adoptable and sustainable are the program components? • Do you think the work you have been doing with CEPAT Program will sustain? [SOW EQ5] • Will your organization continue working in this issue, and how? [SOW EQ5] • What strategies of sustainability have you learned from USAID? [SOW EQ5] • In your opinion, what is the most crucial step in ensuring sustainability of your organization work in this sector (TB program)? [SOW EQ5] • Apart from USAID’s support of program sustainability, what are your strategies to make sure gains/ achievements are sustained? [SOW EQ5&9] OTHER QUESTIONS: • What could be some of the key lessons drawn from CEPAT TB Program? • Do you have any other comments, suggestions or concerns about the Community-based TB Program you work with that you would like to share with us? • Please describe the program’s gender inclusion efforts. How do project activities ensure participation by both men and women? USAID INDONESIA 89 MID-TERM PROGRAM PERFORMANCE EVALUATION 5. INTERVIEW GUIDELINES FOR SUBGRANTEES (E.G. URC, WESTAT) INTRODUCTORY QUESTIONS: 1. Describe your role in your organization. How long have you been in your position? 2. Please describe your organization’s vision and mission. 3. Could you explain the involvement of your organization with CEPAT (type of engagement, time of engagement, activities conducted)? KEY QUESTIONS: Question set 1 EQ 1. To what extent have the program activities strengthened the National TB Program and the national level response to TB at community level? • Please tell us about the current implementation of the National TB Program? [SOW EQ1] • Is the context analysis for the program still relevant and appropriate? In your own words, what is the objective or goal of CEPAT Program you are engaged with? [SOW EQ1] • What is the added value of (Community-based TB Program) in the overall National TB Program? [SOW EQ1] • Were there other achievements of this program that you want to mention? (related with key indicators) [SOW EQ1] Question set 2 EQ 2. To what extent have the program activities increased TB detection, prevention and treatment in Indonesia, and been aligned with CEPAT goals and the TB CSO strategy for Indonesia? • According to you what have been the main challenges of involving communities in the TB control program? [SOW EQ2] • In your work in CEPAT Program, what were the challenges you faced as an organization? How did you deal with these difficulties? [SOW EQ2] l What are the constraining factors regarding effort in the national TB control program? l What are the difficulties of working with Indonesian policy makers at national level? l What are the difficulties of working with Indonesian policy makers at local level? • Regarding the achievements that you mentioned above, what led to this success? What hampered the success? [SOW EQ2] Question set 3 EQ 3. To what extent has the program activity strengthened capacity of the national and sub-national (province and district) government, private sector, community and other stakeholders? • What are the most significant changes in the community-based TB program at the national and local level that you are involved with? [SOW EQ3] 90 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION • Have any practices been introduced to the community-based TB program as the result of the CEPAT program/approach? • According to you, do the local partners have adequate skills or capacities to contribute to the achievement of the (national) program goals? Has the program approach in any way enhanced the effectiveness of the local partners to achieve their goals on TB program? Question set 4 EQ 8. Does the CEPAT Program align with the country’s other community TB efforts (e.g. Aisyiyah) and other community health efforts (i.e. government-funded community health cadres and community organizations in other health areas in Indonesia? • Could you tell us about other community’s TB efforts projects in Indonesia? [SOW EQ8] l What do you think about that? (Probe: do they complement each other? Or compete?) Question set 5 EQ 6. How has the program design, management and implementation among the programs been in maximizing efforts and achieving greater results? • Do you think that the budget you received from CEPAT grantees was sufficient for the activities you were asked to complete/impact you were asked to have? [SOW EQ6] • Is there anything that you wish CEPAT partners did differently? In which aspect? [SOW EQ6] • What are the challenges working with CEPAT partners? Could you elaborate more? [SOW EQ6] Question set 6 EQ 5. How can local and national ownership and future commitment to continued implementation of good practices/lessons learned be enhanced? • How can local and national ownership and future commitment to continued implementation of good practices/lessons learned be enhanced? Question set 7 EQ 6. How replicable, applicable/adoptable and sustainable are the program components? • Do you think the progress and achievement that CEPAT Programs are making will be sustained? OTHER QUESTIONS: • What could be some of the key lessons drawn from CEPAT TB Program? • Do you have any other comments, suggestions or concerns about the Community-based TB Program you work with that you would like to share with us? USAID INDONESIA 91 MID-TERM PROGRAM PERFORMANCE EVALUATION 6. INTERVIEW GUIDELINES FOR RELIGIOUS ORGANIZATIONS (AISYIYAH) INTRODUCTORY QUESTIONS: 1. Describe your role within the organization. 2. Does your organization receive funding for providing TB-related programs or activities? If so, then what is funding for? KEY QUESTIONS: Question set 1 EQ 3. To what extent has the program activity strengthened capacity of the national and sub-national (province and district) government, private sector, community, and other stakeholders? • Please tell us about your organization’s involvement with TB programs or activities. [SOW EQ3] • Please tell us what other organizations are involved with TB activities in areas that your organization operates within and how you collaborate with them. (Probe for other religious organizations.) [SOW EQ3] • Are there organizations providing support for HIV/AIDS interventions? If so, do these organizations provide TB-related activities? [SOW EQ3] • Does your organization also provide HIV/AIDS or other health-related programs or activities? (Probe-if they do then is TB support aligned with it?) [SOW EQ3] • What are the main barriers to collaborating with other potentially useful stakeholders? [SOW EQ3] • Can these be overcome? If so, how? [SOW EQ3] Question set 2 EQ 5.How replicable, adaptable/adoptable and sustainable are the program components? • Much achievement in TB control in Indonesia has been through community-managed TB programs and activities, however, maintaining effort can be a challenge. Can you suggest ways that communities can become more committed to addressing TB? [SOW EQ5] Question set 3 EQ 9. How can local and national ownership and future commitment to continued implementation of good practices/lesson learned be enhanced? • What do you think are some ways that community-based TB programs and activities can be improved/increased in communities that your organization operates in and nearby communities? (Probe for local leadership, broader reach through existing communication channels, increased engagement from community organizations) [SOW EQ9] • Religious organizations and leaders are important partners that can be utilized for addressing and advocating for TB control in Indonesia. Do you have any suggestions about how to increase the engagement of religious organizations for TB control? [SOW EQ9] 92 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION OTHER QUESTIONS: • What are the main lessons learnt from your organization’s involvement with TB programs and activities? • Do you have any other comments or suggestions that you would like to share with us? USAID INDONESIA 93 MID-TERM PROGRAM PERFORMANCE EVALUATION 7. INTERVIEW GUIDELINES FOR COMMUNITY LEADERS INTRODUCTORY QUESTION: 1. Describe your role with your community. KEY QUESTIONS: Question set 1 EQ 3. To what extent has the program activity strengthened capacity of the national and sub-national (province and district) government, private sector, community, and other stakeholders? • Since your community has engaged with the CEPAT partner, what has been the result? [SOW EQ3] (Probe for involvement of the community and stakeholders with a focus on TB-related activities, for example greater promotion of TB services, increase in the number of TB cadres.) • Please tell us what organizations are involved with TB activities in the community (Probe for CSOs, faith based, patient groups, international NGOs, schools etc.) [SOW EQ3] • Are there organizations providing support for HIV/AIDS interventions? If so, do these organizations provide TB-related activities? [SOW EQ3] Question set 2 EQ 5.How replicable, adaptable/adoptable and sustainable are the program components? • Much achievement in TB control in Indonesia has been made through community-managed TB programs and activities, however, maintaining effort can be a challenge. Can you suggest ways that communities can become more committed to addressing TB? [SOW EQ5] Question set 3 EQ 9. How can local and national ownership and future commitment to continued implementation of good practices/lesson learned be enhanced? • What do you think are some ways that community-based TB programs and activities can be improved in your community and nearby communities? (Probe for local leadership, broader reach through existing communication channels, increased engagement from community organizations.) [SOW EQ9] • Local leadership support for addressing TB is very important. Can you tell me about leadership support for TB in your community? (Probe for possible ways to increase support for TB from leaders.) [SOW EQ9] OTHER QUESTIONS: • What are the main lessons learnt from your community’s involvement with TB programs and activities? • Do you have any other comments or suggestions that you would like to share with us? 94 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION 8. INTERVIEW GUIDELINES FOR OTHER USAID TB PROJECTS: (CHALLENGE TB/JSI/DELIVER) INTRODUCTORY QUESTIONS: 1. Describe your role within the organization. 2. Would you like to share the work that your organization does under USAID funding related to TB control program? KEY QUESTIONS: Question set 1 • Please tell us about the current implementation and the quality of the National TB Program? [SOW EQ1] • What do you think about the added value of this CEPAT Program (Community-based TB Program) in the overall National TB Program? [SOW EQ1] • Have there been any positive synergies between USAID TB projects that strengthened the National TB strategy? If so, what are they? [SOW EQ7] • Do you have any suggestions how to improve the synergy among USAID TB projects? [SOW EQ7] • According to you what have been the main challenges of working or assisting the Government of Indonesia in the TB control program? [SOW EQ1] • In your work related to TB control program, what were the challenges you faced as an organization? How did you deal with these difficulties? [SOW EQ1] l What are the constraining factors regarding effort in the national TB control program? l What are the difficulties of working with Indonesian policy makers at national level? l What are the difficulties of working with Indonesian policy makers at local level? • Has JKN implementation affected the national TB achievement? [SOW EQ1] • Has decentralization also affected the national TB achievement? [SOW EQ1] • How did you deal with sensitive issues, for example, those that may involve various actors (NTP, other development partners or CSOs, local government, private providers)? [SOW EQ7] OTHER QUESTIONS: • What are the main lessons learnt from your organization’s involvement with TB programs and activities? • Do you have any other comments or suggestions that you would like to share with us? USAID INDONESIA 95 MID-TERM PROGRAM PERFORMANCE EVALUATION EVALUATION OF CEPAT PROGRAM COVERSHEET FOR ALL FOCUS GROUP DISCUSSIONS Date of Interview: Interviewee Name, Title: Team: Time Start: Time End: Interviewer(s): Location: Introduction: Good morning/afternoon and thank you all for taking the time to participate with this focus group today. We are working to conduct an evaluation of the CEPAT Project. The evaluation will inform stakeholders of how CEPAT contributes to community-based TB strategies, will generate a broad range of lessons learnt from partners and will provide recommendations for future community￾based TB programs in Indonesia. Our team has had the opportunity to review some background documents to get a better sense of the design and implementation of the project, however, these documents can only tell us so much. We would like to hear about your experiences today in order to help us better understand more about how the CEPAT project functions. The focus group will last around 60 minutes. We will take notes of the group discussion and, with your consent, a recording will also be made using a digital voice recorder to aid us in fully completing our notes after the discussion. Confidentiality Protocol • We will collect information on individuals’ names, organizations and positions. A list of key informants will be made available as an annex to the final evaluation report, but those names and positions will not be associated to any particular findings or statements in the report. • We may include quotes from respondents in the evaluation report, but will not link individual names, organizations or personally identifiable information to those quotes, unless express written consent is granted by the respondent. Should the team desire to use a particular quote, photograph or identifiable information in the report, the evaluators will contact the respondent(s) for permission to do so. • All data gathered will be used for the sole purpose of this evaluation, and will not be shared with other audiences or used for any other purpose. • Your participation in this group discussion is voluntary and if you do not feel comfortable answering a particular question, please let us know and we will simply go on to the next question. • Once again, thank you for taking the time to speak with us today. Do you have any questions for us before we get started? Thank participants at the end of the discussion. 96 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION ADDITIONAL GUIDANCE FOR FOCUS GROUP FACILITATORS: Be sure to: • Set up the room to facilitate a participatory discussion. • Introduce yourselves and also the note taker. • Explain why you are conducting the focus group session. • Ask the participants to do the same. • Give a verbal agenda and length of the meeting. • Set goals. • Clarify your role as a facilitator. FOCUS GROUP FACILITATOR’S GUIDELINES FOR EFFECTIVE DISCUSSIONS: • Everyone is clear on the topic. • Everyone participates; no-one dominates the discussion. No speeches! • One person talks at a time. • Comments and discussion stay on the topic. • Comments should be to the whole group - no side discussions. • Respect time limits. • Write down unanswered questions. • No divisive or confrontational language or tone. • Ask participant’s consent to use the digital audio recorder. • Note taker should write the dynamic of the discussion and write down the question and key answers. 1. INTRODUCTION (10 MINUTES) See above • Reaffirm points of the meeting. • Welcome participants. • Set 60- 90 minute timeline. • Introduce the evaluation, give verbal agenda, objectives (as stated above). USAID INDONESIA 97 MID-TERM PROGRAM PERFORMANCE EVALUATION 2. SETTING RULES (5 MINUTES) Before the participants begin dealing with issues and ideas, the participants should agree on a set of rules that define how a group will function and how the participants will interact. • Each group member has the right to participate. • The opinion of each group member is important and should be respected. • Group members should be tolerant of different ideas. • Each group member is important. FOCUS GROUP DISCUSSION GUIDELINES FOR COMMUNITY CADRES 1. Introductory question: How long have you been in your position? Describe your role as CEPAT cadre? 2. Introductory question: Could you tell me why did you decide to become a CEPAT cadre? 3. (SOW EQ1): Could you tell me about the TB situation in your area including available support? (Probe for changes in the situation over recent times.) 4. (SOW EQ1,3): For a moment, think of yourself as a TB patient or a person suffering from TB symptoms. Can you tell me what barriers and opportunities there may be for accessing and obtaining successful treatment? Has there been any changes over recent times i.e. the past few years? (Probe for stigma, level/type of support being provided.) 5. (SOW EQ3): Are you familiar with the USAID CEPAT Program? Could you explain to me more? 6. (SOW EQ1,3,6): Has support from the CEPAT project assisted your work as a cadre? If so, how? If no, why? (Probe for capacity building.) 7. (SOW EQ9): Could you share about what you like most and least about your work as a CEPAT cadre? (This question seeks to establish sustainability issues/opportunities.) 8. (SOW EQ6): Have you encountered any challenges during your involvement as CEPAT cadre? If so, how would you suggest this be improved? (Probe for administrative burden/forms that they should fill in, approaching communities etc.) 9. (SOW EQ3,7,8): What TB services are there in your community? How much collaboration do you and the CEPAT project have with them? FOCUS GROUP DISCUSSION GUIDELINES FOR TB PATIENT GROUPS 1. (SOW EQ1,2,3): What support for your TB condition do you receive from community members, volunteers, or organizations? (Probe for support type, who provides it and organizations that are active in the community, including those providing communication.) 2. (SOW EQ6): Do you have any suggestions about what kind of additional support may be needed by you or for people you may know who you suspect have TB but are not being treated? 98 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION 3. (SOW EQ6): Can you tell me about any things in your community that may encourage or discourage people from seeking treatment for TB symptoms? 4. (SOW EQ3,6): Has support for TB patients changed over recent times? (Probe for treatment options, increased communication about TB, greater community awareness.) 5. (SOW EQ6): In what ways can support for TB patients or people with TB symptoms be improved? (Probe for improved case finding, access to service delivery, awareness.) 6. (SOW EQ1,2): Where and how did you receive information and understanding about TB? (Probe for types of communication channels - radio, interpersonal, health worker, cadres.) 7. (SOW EQ6) In what ways do you think TB prevention can be improved? (Probe for increased awareness.) 8. (SOW EQ9): What do you think about your community leaders and decision makers’ awareness and involvement towards addressing TB in your community? (Probe for changes in awareness, level of engagement from them, need for greater advocacy to engage them.) USAID INDONESIA 99 MID-TERM PROGRAM PERFORMANCE EVALUATION ANNEX SIX: RESPONDENTS CONSULTED CEPAT PARTNERS AND SUBGRANTEES No. Informant Institution Position IDI/FGD 1. Ir. Esty Febriani MKes LKNU Chief of Party FGD 2. Mr. Hartono LKNU Community Development Expert FGD 3 Mr. Makky Zamzami LKNU M&E Manager/TB Advisor FGD 4. Ms. Nurfina Bachtiar LKNU-URC Field Operation Manager FGD 5. Mr. Neeraj Kak URC Senior Vice President IDI 6. Mr. Syaiful LKNU District Coordinator, Central Jakarta FGD 7. Ms. Neng Apriani LKNU Advocacy Officer, Central Jakarta FGD 8. Ms. Ani Yuliani LKNU Finance Officer, Central Jakarta FGD 9. Ms. Lia Zul LKNU M&E Officer, Central Jakarta FGD 10. Mr. Ajat Sudrajat LKNU District Coordinator, Cirebon FGD 11. Mr. Moh. Amrun Rosikh LKNU M&E Officer, Cirebon FGD 12 Mr. Aziz Akbar LKNU Finance Officer, Cirebon FGD 13 Ms. Sueni LKNU Community Facilitator FGD 14 Ms. Ade Kurniawati LKNU Community Facilitator FGD 15. Watiah LKNU Advocacy Officer, Cirebon FGD 16. Dr. Delyuzar Sp PA(K) JKM Chief of Party IDI 17. Dr. Masrul Sp GK PhD JKM/University of Andalas Advocacy Expert IDI 18. Dr. Juliandi Harahap MA JKM TB Advisor FGD 19. Dr. R. Lia Kusumawati Sp MK(K) JKM/USU Laboratory Expert FGD 20. Mr. Alwin Harahap JKM Training Manager FGD 21. Mr. Harianto JKM COD Province Coordinator FGD 22 Mr. Samara Yudha JKM COD Medan FGD 23. Mr. Indun JKM COD Deli Serdang FGD 24. Mr. Nanang JKM COD Serdang Bedagai FGD 25. Ms. Dini Ully Adriani JKM Grant Manager FGD 26. Ms. Elly Arsam JKM M&E Manager FGD 27. Ms. Mekkla Thompson Westat Management Advisor & Capacity Building Expert IDI (Skype) 28. Mgr. John Philip S. RCD Chief of Party IDI 29. Mr. Ernest Vincentius Sia RCD Deputy Chief of Party IDI 100 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION No. Informant Institution Position IDI/FGD 30. Mr. Frederik Yosep Wela￾fubun RCD Advocacy Advisor FGD 31. Mr. Moriana Hungun RCD TB Technical Officer FGD 32. Mr. Ridho Yanuardi RCD Administration and M&E Officer FGD 33. Mr. Rudolph Kambayong RCD District Coordinator, Timika FGD 34. Ms. Elsye Madellu RCD/Perdakhi M&E Officer, Timika FGD 35. Mr. Martin RCD District Coordinator, Sorong FGD 36. Ms. Esti Yempormase RCD M&E Officer, Sorong FGD 37. Ms. Dian Satubandule RCD TB Officer, Sorong FGD 38. Mr. Budi Hermawan PETA-LKNU Subgrantee Vice Director IDI 101 GOVERNMENT OFFICIALS No. Informant Institution Position IDI/FGD 1. Ms. Dyah Erti Mustikawati DDS MPH MoH Directorate of NCD, Dir Gen of Disease Control / Former Head of TB Subdirector￾ate, MoH IDI 2. Ms. Budiarti Setyaningsih SKM Mkes MoH Focal Point of Planning, TB Subdirectorate, MoH IDI 3. Dr. Usma Polita DHO Medan Head of District Health Office IDI 4. Dr. Yuditha Endah P. Mkes DHO Central Ja￾karta/Department, Central Jakarta Head of District Health Office/Head of Department FGD 5. Dr. Myrna K DHO Central Jakarta/ Depart￾ment, Central Jakarta Head of Disease Control and Prevention, DHO Central Jakarta FGD 6. Ms. Dwi Rizky DHO Central Jakarta/ Depart￾ment, Central Jakarta Deputy TB Supervisor/ TB Supervisor FGD 7. Mr. Subhan DHO Cirebon District Deputy TB Supervisor/ TB Supervisor IDI 8. Mr. Nanang Ruhyana DHO Cirebon District Head of Disease Control Section IDI 9. Ms. Titis Puskesmas Susu￾kan, Cirebon TB Programmer FGD 10. Mr. Ahmad Hidayat Puskesmas Susu￾kan, Cirebon Health Promotion Officer FGD 11. Mr. Didi Junaedi Puskesmas Susu￾kan, Cirebon Laboratory Analyst FGD 12. Ms. Prihartiningsih Puskesmas Menteng, Central Jakarta TB Programmer IDI 13. Mr. Mastoyo Cirebon Head of Buyut Village IDI 14. Mr. Brahmana Puskesmas Muly￾orejo TB Programmer IDI 15. Dr. H.E. Sihombing DHO Sorong Head of DHO IDI 16. Ms. Ludya Wattimena DHO Sorong Deputy TB Supervisor/ TB Supervisor IDI 17. Mr. Reynold DHO Timika DHO Secretary / Head of AIDS Commis￾sion Timika IDI 18. Ms. Maya DHO Timika Deputy TB Supervisor/ TB Supervisor IDI 19. Ms. Yacoba Puskesmas Kota, Timika TB Programmer IDI 20. Mr. Maikel M. Puskesmas Doom Head of Puskesmas FGD 21. Ms. Barbalina Sawor Puskesmas Doom TB Programmer FGD 22. Mr. E. Bagaskoro Puskesmas Doom Laboratory Analyst FGD 102 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION No. Informant Institution Position IDI/FGD 23. Mr. M. Djaules Doom Village, Sorong Head of Village FGD 24. Drg. Surya Pulungan Puskesmas Medan, Marelan Head of Puskesmas IDI 25. Mr. Maryono Puskesmas Tanah Abang TB Programmer IDI 26. Mr. Hendra Wijaya Subdistrict Sung￾gal, Deli Serdang Head of Subdistrict IDI OTHER TB PARTNERS/STAKEHOLDERS No. Informant Institution Position IDI/FGD 1. Mr. Jeff Muschell Global Fund Global Fund Advisor IDI 2. Dr. Esty Maitiana R. Aisyiyah Chief of Party FGD 3. Ms. Marlina Harahap Aisyiyah Community Development Specialist FGD 4. Mr. Thoriq H. Aisyiyah M&E Manager FGD 5. Dr. Jhon Sugiharto Challenge TB Technical Director IDI 6. Ms. Yoana Anandita WHO Program Officer IDI COMMUNITIES No. Informant Institution Position IDI/FGD 1. Mr. Paul Tan, Priest Roman Catholic Diocese (RCD), Sorong Head of RCD Sorong IDI 2. Mr. Amandus Rahadat, Priest Tiga Raja Church, Timika Priest of the church IDI 3. Mr. Burhan KMP As-Syifa Coordinator of Koalisi Masyarakat Peduli TB, Susukan Cirebon IDI 4. Mr. Rudin Alert Village (Desa Siaga), Buyut Head of Alert Village, Buyut, Cirebon IDI 5. Mrs. Agustina Village (Kelurahan Soop), Sorong TB Cadre IDI 6. Mrs. Tati Suharti Subdistrict Tanah Abang TB Cadre FGD 7. Mrs. Sri Mulyani Subdistrict Tanah Abang TB Cadre FGD 8. Mrs. Asma Subdistrict Tanah Abang TB Cadre FGD 9. Mrs. Rukyah Subdistrict Tanah Abang TB Cadre FGD 10. Mrs. H. Nurheli Subdistrict Tanah Abang TB Cadre FGD USAID INDONESIA 103 MID-TERM PROGRAM PERFORMANCE EVALUATION No. Informant Institution Position IDI/FGD 11. Mrs. Kurniati Subdistrict Tanah Abang TB Cadre FGD 12. Mrs. Heddy Subdistrict Sung - gal, Deli Serdang TB Cadre FGD 13. Mrs. Tiar Subdistrict Sung - gal, Deli Serdang TB Cadre FGD 14. Mrs. Sugiarti Subdistrict Sung - gal, Deli Serdang TB Cadre FGD 15. Mrs. Sarinah Subdistrict Sung - gal, Deli Serdang TB Cadre FGD 16. Mrs. Yetty Subdistrict Sung - gal, Deli Serdang TB Cadre FGD 17. Mrs. Ida Subdistrict Sung - gal, Deli Serdang TB Cadre FGD 18. Ms. Nadia Subdistrict Sung - gal, Deli Serdang TB Cadre FGD 19. Ms. Neng Subdistrict Sung - gal, Deli Serdang TB Cadre FGD 20. Mrs. Desi Subdistrict Sung - gal, Deli Serdang TB Cadre FGD 21. Mrs. Suci Subdistrict Sung - gal, Deli Serdang TB Cadre FGD 22. Mrs. Tugiyem Subdistrict Sung - gal, Deli Serdang TB Cadre FGD 23. Mrs. Salmiah Subdistrict Sung - gal, Deli Serdang TB Cadre FGD 24. Mrs. Suarsih Subdistrict Sung - gal, Deli Serdang TB Cadre FGD 25. Mrs. Suratmi Subdistrict Sung - gal, Deli Serdang TB Cadre FGD 26. Mrs. Kiki Subdistrict Medan, Marelan, Medan TB Cadre FGD 27. Mrs. Fitri Subdistrict Medan, Marelan, Medan TB Cadre FGD 28. Mrs. Windi Subdistrict Medan, Marelan, Medan TB Cadre FGD 29. Mrs. Dita Subdistrict Medan, Marelan, Medan TB Cadre FGD 30. Mrs. Kusmawati Subdistrict Medan, Marelan, Medan TB Cadre FGD 31. Mrs. Wartini Subdistrict Medan, Marelan, Medan TB Cadre FGD 32. Mrs. Nella Numberi Kelurahan Doom, Sorong TB Cadre FGD 33. Mrs. Angel Kore Kelurahan Doom, Sorong TB Cadre FGD 104 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION No. Informant Institution Position IDI/FGD 34. Mrs. Nunung Siti Rohmatin Cirebon District TB Cadre FGD 35. Mrs. Juniah Subdistrict Ciwar￾ingin, Cirebon FGD 36. Mrs. Suniah Subdistrict Kali￾wed, Cirebon TB Cadre FGD 37. Mrs. Nani Yuhanid Subdistrict Asta￾najapura, Cirebon TB Cadre FGD 38. Mrs. Endawati Subdistrict Susu￾kan, Cirebon TB Cadre FGD 39. Mrs. Yulianah Subdistrict Weru, Cirebon TB Cadre FGD 40. Yance Tipsnaby Kelurahan Doom, Sorong TB patient - Cured FGD 41. Yuraida Kelurahan Doom, Sorong TB patient - Cured FGD 42. Mr. Korila Timika TB patient IDI 43. Mrs. Paskalina Subdistrict Lo￾pong, Timika TB Cadre FGD 44. Mr. Hengki Subdistrict Lo￾pong, Timika TB patient FGD 45. Mrs. Rosalina Subdistrict Lo￾pong, Timika TB patient FGD 46. Mr. Karel Subdistrict Lo￾pong, Timika TB patient - Cured FGD 47. Ms. Ruyana North Jakarta MDR-TB Cadre IDI USAID No. Informant Institution Position IDI/FGD 1. Dr. Artha Camelia MPH USAID Senior Infectious Disease Specialist IDI 2. Dr. Alia Hartanti USAID TB Officer IDI USAID INDONESIA 105 MID-TERM PROGRAM PERFORMANCE EVALUATION ANNEX SEVEN: DISCLOSURE OF ANY CONFLICTS OF INTEREST 106 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION USAID INDONESIA 107 MID-TERM PROGRAM PERFORMANCE EVALUATION ANNEX EIGHT: STATEMENT OF DIFFERENCES There were no significant or unresolved differences of opinion by funders, implementers and/or members of the ET during the development of this evaluation. 108 USAID INDONESIA MID-TERM PROGRAM PERFORMANCE EVALUATION REFERENCES: Collins, D.H. 2013. The Economic Case for Investing in TB in Indonesia. Medford, Massachusetts: Management Sciences for Health. Ministry of Health (MoH) (The National Institute of Health Research and Development [Badan Litbangkes], Directorate General Disease Control and Environment) and WHO. 2004. The National TB Prevalence Survey 2004. Ministry of Health (MoH) (The National Institute of Research and Development Centre). 2012. Health Facility Research – Puskesmas – 2011. Final report. Jakarta. Ministry of Health (MoH) (Directorate General Disease Control and Environment). 2014. National Guidelines on TB Control 2015-2019. Statistics Indonesia (Badan Pusat Statistik—BPS), National Population and Family Planning Board (BKKBN), and Kementerian Kesehatan (Kemenkes—MoH), and ICF International. 2013. Indonesia Demographic and Health 2012. Jakarta: Statistics Indonesia. World Health Organization (WHO). 2010. Global Tuberculosis Control 2010. Geneva: World Health Organization.d World Health Organization (WHO). 2015. Global Tuberculosis Report. Geneva: World Health Organization. 110 USAID INDONESIA U.S. AGENCY FOR INTERNATIONAL DEVELOPMENT 1300 PENNSYLVANIA AVENUE, NW WASHINGTON, DC 20523 MAY 2016 This report is made possible by the support of the American People through the United States Agency for International Development (USAID). The contents of this report is the sole responsibility of the authors, Angus CR Walker and Isma Novitasari, and do not necessarily reflect the views of USAID or the United States Government.