USAID INDONESIA 1 PERFORMANCE EVALUATION REPORT: THE COMMUNITY EMPOWERMENT OF PEOPLE AGAINST TUBERCULOSIS PROGRAM, INDONESIA MAY 2019 This report is made possible by the support of the American People through the United States Agency for International Development (USAID). The contents of this report are the sole responsibility of the authors, Jenne Roberts, Octavery Kamil and Isma Novitasari, and do not necessarily reflect the views of USAID or the United States Government. May 12th, 2019 #AID-497-A-13-00003 #AID-497-A-13-00007 #AID-497-A-13-00011 PERFORMANCE EVALUATION REPORT: THE COMMUNITY EMPOWERMENT OF PEOPLE AGAINST TUBERCULOSIS PROGRAM, INDONESIA USAID INDONESIA iii PERFORMANCE EVALUATION REPORT DISCLAIMER This report is made possible by the support of the American People through the United States Agency for International Development (USAID). The contents of this report are the sole responsibility of the authors, Jenne Roberts, Octavery Kamil and Isma Novitasari, and do not necessarily reflect the views of USAID or the United States Government. iv USAID INDONESIA PERFORMANCE EVALUATION REPORT USAID INDONESIA v PERFORMANCE EVALUATION REPORT CONTENTS DISCLAIMER iii CONTENTS iv ACRONYMS vi MID-TERM PROGRAM PERFORMANCE EVALUATION vii ACKNOWLEDGMENTS ix CEPAT PROJECT BACKGROUND 1 THE EVALUATION PURPOSE, DESIGN AND LIMITATIONS 2 METHODOLOGY 2 FINDINGS AND RESULTS 3 RESULTS 4 CONCLUSIONS 7 KEY RECOMMENDATIONS 9 ANNEXES 10 Annex 1: Evaluation Terms of Reference 10 Annex 2: Evaluation Work Plan 19 Annex 3: Evaluation Participants 20 Annex 4: Evaluation Questions 24 Annex 5: Reference Documents 34 vi USAID INDONESIA PERFORMANCE EVALUATION REPORT USAID INDONESIA vii PERFORMANCE EVALUATION REPORT ACRONYMS APBD Anggaran Pendapatan dan Belanja Daerah/ District Budget AIDS Acquired Immune Deficiency Syndrome Bappeda Badan Perencanaan Pembangunan Daerah/ Regional Development Planning Agency BOK Bantuan Operasional Kesehatan/ Health Operational Funding CA Cooperative Agreement Cadres Community health volunteer CDCS Country Development Cooperation Strategy CDR Case Detection Rate CEPAT Community Empowerment of People Against Tuberculosis COP Chief of Party COR Contracting Officer Representative CSO Civil Society Organization DHO District Health Office DOTS Direct Observation Treatment Short-Course ET Evaluation Team FGD Focus Group Discussion Gerdunas Gerakan Terpadu Nasional/ National Movement against TB GFATM Global Fund to Fight AIDS,Tuberculosis and Malaria GHI Global Health Initiative HIV Human Immunodeficiency Virus IDI Ikatan Dokter Indonesia/ Indonesia Medical Association JKM Jaringan Kesehatan/ Kesejahteraan Masyarakat / Community Welfare/Health Network KAP Knowledge, Attitude, Practice KII Key Informant Interview KMP Koalisi Masyarakat Peduli TB / Community Coalition for TB KNCV Yayasan KNCV Indonesia (YKI). (formerly Koninklijke Nederlandse Chemische Vereniging (Royal Dutch Chemical Association). KPI Key Performance Indicators LKNU Lembaga Kesehatan Nahdlatul Ulama/ Nahdlatul Ulama Health Institute MDR-TB Multi Drug Resistant-Tuberculosis viii USAID INDONESIA PERFORMANCE EVALUATION REPORT MID-TERM PROGRAM PERFORMANCE EVALUATION MoH Ministry of Health NGO Non-Governmental Organization/s NTP National Tuberculosis Program PBNU Pengurus Besar Nahdlatul Ulama/ Executive Board of Nahdlatul Ulama PCA Patient-centred approach PCTC Patients’ Charter for Tuberculosis Care PDPI Persatuan Dokter Paru Indonesia/ Indonesia Pulmonologist Association) PETA Pejuang Tangguh (Strong Fighter)/ MDR-TB Patient Support Group PKK Pembinaan Kesejahteraan Keluarga/ Family Welfare Movement PMDT Programmatic Management of Drug-resistant TB PMP Project Monitoring Plan Pokja Kelompok Kerja/ Working Group PPNI Persatuan Perawat Nasional Indonesia/ Indonesian National Nurses’ Association Puskesmas Pusat Kesehatan Masyarakat/ Community Health Centre RCD Roman Catholic Diocese RFA Request for Application SGD Small Group Discussion SoW Scope of Work SoW EQ Scope of Work - Evaluation Question SUBDIT TB Sub-direktorat Tuberkulosis (the TB sub-directorate at the Ministry of health) TB Tuberculosis TBCTA Tuberculosis Coalition for Technical Assistance UI University of Indonesia URC University Research Co. LLC. USAID United States Agency for International Development USU Universitas Sumatera Utara/ University of North Sumatra Wasor TB Wakil supervisor tuberkulosis (TB Program Manager) USAID INDONESIA ix PERFORMANCE EVALUATION REPORT ACKNOWLEDGMENTS The evaluation team (Jenne, Vita, Very and Vanda) would like to thank all the former leaders and staff from each of the CEPAT Implementing Partners (LKNU, JKM and RCD - Timika), who provided the team with information and access to former staff, partners and Cadres. This evaluation was enriched by stories shared with us and the opportunity to visit sites and see firsthand where Cadres are continuing to conduct community mobilization and supporting TB patients. Thank you to the many stakeholders who shared thoughts, opinions and time, your recollections and insights enabled us to produce this evaluation. A special acknowledgement goes to USAID/Indonesia staff (and former staff) for their extensive support and willingness to reflect on their experiences, BANTU Project staff (Sumana Brahman, Mita Witjaksono and Ester Hutabarat) for their tireless assistance with coordination and communication, William Wells and Adi Sarininggar for their comments and guidance and other USAID staff who provided input into the evaluation. x USAID INDONESIA PERFORMANCE EVALUATION REPORT USAID INDONESIA 1 PERFORMANCE EVALUATION REPORT CEPAT PROJECT BACKGROUND Indonesia has the third highest (after India and China) tuberculosis (TB) burden in the world (an estimated 842,000 new cases in 2017)1 and has one of the highest burdens of multidrug resistant (MDR) TB. MDR-TB is one example of antimicrobial resistance (AMR) which is a growing threat to health security in Indonesia and the region. Reducing the threat posed by TB, including MDR, requires an effective partnership between the Government of Indonesia (GOI), the private sector and the community, as well as international and cross border collaboration. Consistent with the principals of the USAID Forward Initiative (2010 -2016) to promote sustainable development through high-impact partnerships and local solutions2 , USAID /Indonesia provided funding directly to three local non-government organizations to address TB by mobilizing the community to increase demand for, the quality of and resources for TB in areas where they were well established. This approach was expected to build community’s engagement, trust and ownership of the TB program and simultaneously serve the USAID Forward goal of developing local capacity and fostering a sustainable model for development3 . The Community Empowerment of People Against Tuberculosis (CEPAT) program, in partnership with Indonesia’s National Tuberculosis Program (NTP) was launched in 2012. The implementing Partners (IP) were chosen for their extensive links with the community, their commitment to community empowerment and their genuine desire to reduce the burden of TB. CEPAT activities were organized into Components 1 and 2 as described below. Activities under Component 1 focused on the community’s role in: a) Improved case finding; b) Increased access to service delivery; c) Improved health seeking behavior to quality TB services; d) Increased knowledge and awareness of key TB information; e) Increased contact tracing of TB patients; and f) Improved and expanded community social support systems for patients during treatment. Activities under Component 2 aimed to: a) Increase the district budgets and allocated resources for TB; b) Increase advocacy at the local levels for TB services and resources; c) Increase the knowledge of TB among district leaders and other key stakeholders; and e) Reduce local policies inconsistent with national NTP guidance. USAID awarded three separate cooperative agreements (CA) to the following IP: • Lembaga Kesehatan Nahdlatul Ulama (LKNU) (funded from November 2012 - November 2017); • Jaringan Kesehatan/ Kesejahteraan Masyarakat (JKM) (funded from December 2012 – December 2017); and • the Roman Catholic Diocese of Timika (RCD) (June funded from 2013 – June 2018). The total funding of all three awards amounted to USD 7,304,028 over the life of the project. CEPAT was USAID Indonesia’s first experience with directly issuing cooperative agreements (CA) to local NGO. This approach coincided with the introduction of Indonesia’s universal Health Coverage Scheme, which was launched on 1 January 2014. The objectives of the scheme include two that are relevant two and the context in which CEPAT was implemented: 1 WHO Report, 2017 2 https://www.usaid.gov/usaidforward 3 https://www.usaid.gov/news-information/fact-sheets/usaid-forward-five-years-and-counting 2 USAID INDONESIA PERFORMANCE EVALUATION REPORT • To enable people accessing healthcare services without financial hardship. • To prioritize preventive and promotive measures in rendering healthcare services to reduce prevalence of diseases, lower the numbers of sick-people with efficient healthcare services. THE EVALUATION PURPOSE, DESIGN AND LIMITATIONS The purpose of this evaluation is to assess the extent to which CEPAT succeeded in empowering community action related to TB by (a) mobilizing communities to support improved TB care; and (b) advocating for increased TB resources and improved TB services. It also evaluates the contribution of direct funding to strengthening local capacity for development; and makes recommendations to help USAID continue to target efforts designed to enhance community engagement to support the national TB program. The evaluation objectives are to: • determine extent to which the USAID CEPAT Program supported the NTP goal of “achieving universal access to quality and early tuberculosis (TB) diagnosis and treatment through community empowerment, “ specifically by (1) mobilizing communities to support improved TB care; and (2) advocating for increased TB resources and improved TB services; • assess and describe the successes, challenges, and lessons learned from the program’s implementation; • evaluate the merits of the CEPAT approach in building community’s engagement, trust and ownership of the TB program; • evaluate the extent to which activities related to CEPAT contributed to the USAID Forward goal of developing local capacity and fostering a sustainable model for development, and describe how these capacity building efforts affected the results; • evaluate the alignment between CEPAT’s use of community health volunteers and Indonesia’s utilization of community health volunteers in general and identify whether CEPAT IP missed opportunities to align and if there are opportunities to strengthen the alignment and effective use of community health volunteers in TB programming in the future; • make recommendations to help USAID better target efforts designed to enhance community engagement in order to support the Government of Indonesia’s National Tuberculosis Program (NTP). The full Terms of Reference for this evaluation can be found in Annex 1. METHODOLOGY The evaluation team (ET) used mixed methods to answer the evaluation questions and elicit lessons learned. The assessment took place from October 2018 to January 2019. During this timeframe, the team conducted a document and performance data review and interviewed a range of stakeholders. The ET also made site visits to observe project activities that have been sustained and to meet with former CEPAT staff, volunteers (Cadres) and other key informants to capture the diversity of opinions and perceptions about the CEPAT Program’s implementation and impact. Additionally, the ET also met with National, provincial and district officials, puskesmas staff; other National TB Program stakeholders (CCM, Aisyiah, Challenge TB, other local NGOs); Academics; and community empowerment experts working in HIV and TB. The qualitative data collection included field observations, 73 in-depth USAID INDONESIA 3 PERFORMANCE EVALUATION REPORT interviews (IDI) and small group interviews and 10 focus group discussions (FGD) was conducted in DKI Jakarta, North Sumatera (Medan City and Deli Serdang), West Sumatera (Padang and Solok), West Java (Cirebon District), Papua (Jayapura, Timika and Nabire) and West Papua (Sorong City). The evaluation team included Ms. Jenne Roberts, an international public health specialist and TB and HIV community engagement expert who served as Team Leader for this assessment; Dr. Isma Novitasari Yusadiredja, a researcher and public health and evaluation specialist from Bandung; and Dr. Octavery Kamil, a research associate at the AIDS Research Center at Atma Jaya University with expertise in empowerment and civil society participation in programming. The team was supported by Ms. Vanda Moniaga, who provided interpreting, translation and cultural brokerage services, as well as administration and scheduling support to the team. For more details of the evaluation project, please see Annex 2 for the work plan for the evaluation; Annex 3 for the list of participants interviewed for the evaluation; and Annex 4 for a full list of the original list of the evaluation questions. Annex 5 provides a list of reference documents consulted by the evaluation team. The sites visited by the ET included: • DKI Jakarta and districts • Cirebon, Solok (West Java) • Medan and Deli Serdang (North Sumatra) • Timika and Nabire (Papua) • Sorong (West Papua) • Blitar and Kediri (East Java) FINDINGS AND RESULTS The three Implementing Partners and their roles in CEPAT are described below: A. Lembaga Kesehatan Nahdlatul Ulama (LKNU – award from November 2012 - November 2017): The Pengurus Besar Nahdlatul Ulama (PBNU), also known as the Nahdlatul Ulama (NU) is the largest Muslim organizations in Indonesia. One of its institutions is the Lembaga Kesehatan Nahdlatul Ulama (LKNU or ‘Health Institute’). LKNU was granted a CEPAT award from November 2012 until November 2017 to work in 10 districts in three provinces: DKI Jakarta, West Java, and East Java. LKNU trained and established community cadres (community health extension workers) to raise awareness of TB symptoms among the community, conduct contact-tracing, refer people with presumptive cases for TB testing; and provide patient support to people with confirmed cases while they underwent treatment and received Directly Observed Treatment Short-course (DOTS). This required coordination with the puskesmas/local health center TB diagnostic and treatment units. LKNU also advocated for increased local budgets for TB care and treatment; and for strengthening local health regulations/policies to improve the effectiveness of TB programs, and expand TB-related services among government health providers. The program also provided technical assistance to strengthen community groups and patient support groups to so they could better engage with local governments around TB control; and initiated new approaches for community support groups to ensure complete treatment of Multi Drug-Resistant (MDR) TB patients in collaboration with Persahabatan Hospital in Jakarta. 4 USAID INDONESIA PERFORMANCE EVALUATION REPORT B. Jaringan Kesehatan/Kesejahteraan Masyarakat (JKM – award from December 2012-December 2017): JKM or Networking for Community Welfare and Health, is a local NGO headquartered in Medan that deals primarily with community health projects throughout the province of North Sumatra. It was established in 2001 by a group of doctors and health practitioners in Medan. JKM has since developed into a well-known organization recognized for its leading role in community-based public health initiatives. JKM’s CEPAT program operated in North Sumatra, West Sumatra and DKI Jakarta provinces, and focused on community participation in TB programs and other activities designed to reduce the burden on the health system, and on reaching marginalized people and undetected cases. C. Roman Catholic Diocese of Timika (RCD – award from June 2013 – June 2018): RCD, a Christian faith-based organization, received the third CEPAT award to implement activities in Papua and West Papua. RCD operated CEPAT in Timika/Mimika, Paniai, Nabire, and Jayawijaya districts and Sorong city. The program supported strengthening the roles of civil society, the Roman Catholic Church and traditional leaders in TB control, as well as advocating for improved resources, services and TB policy in Papua. RESULTS During implementation, in 25 districts across seven provinces, CEPAT has contributed to empowering community action related to TB by providing community based screening which led to the health system testing 54,395 people with TB symptoms, among whom 6,583 TB patients initiated treatment. In total 9,288 TB patients enrolled in treatment for TB received support services from the project. Of these, 4,887 people were successful in completing treatment for TB. The table below shows the total achievement of each CEPAT IP over the life of the program, in relation to the four key performance indicators established in the CA: PERFORMANCE INDICATOR LKNU * JKM ** RCD *** CEPAT total 1. Total number of people with TB symptoms tested 16,134 20,392 17,869 54,395 2. Total number of smear-positive initiating treatment 1,942 3,050 1,591 6,583 3. Total number of TB cases provided with support services 3,877 2,933 2,478 9,288 4. Total number of TB cases successful in completing Direct Observation Treatment Short Course (DOTS) 831* 2,766 1,290 4,887 *Year 2-4 data only available Source: * Annual Reports CEPAT LKNU Year 1 -5 ** Final Report CEPAT JKM 2012 -2017 *** Final Annual Report CEPAT RCD Oct 2013 – June 2018 Most case finding by public health facilities was passive, until the TB Cadres were mobilized and started bringing in people suspected of having TB for testing. CEPAT utilized many community mobilization activities to actively find cases. The approaches that were most successful in extending services into the USAID INDONESIA 5 PERFORMANCE EVALUATION REPORT community, case finding, increasing access to service delivery and expanding social support systems for patients during treatment include: • Utilizing community health volunteers, religious leaders and village leaders to inform the community about how to recognize signs of TB infection, where to go for free and effective testing and treatment and how to access support for patients. This raised awareness and reduced stigma related to TB and treatment seeking; • Extending access to treatment into underserved, remote locations via TB Village Posts. This resulted in people with no prior access to TB services being able to access information about TB, where to get tested and treatment; • Establishing local forums for information sharing, problem solving and feeding back information to inform service improvements. This worked to focus attention and effort on TB programming and facilitated communication and the opportunity to build trust between the puskesmas and the community; and • Facilitating access to patient support groups and patient support systems amongst TB patients. This reduced stigma, provided adherence support and helped people to share information about TB. Multiple activities raised awareness about the spread and symptoms of TB and the IP facilitated and participated in forums where attention was drawn to TB. For example, awareness raising and calls-to￾action were issued through campaigns such as marches on World TB Day, ‘Knock on the Door’ and TB Free Village campaigns and ‘blusukan’, and by integrating TB into the SiAga Village and contributing to Gerakan Terpadu Nasional/ National Movement against TB (Gerdunas) actions via radio and songs. Enabling the leaders to mobilise the community was an effective way to increase knowledge and awareness, as TB information was integrated into fatwa, sermons on daily responsibilities and the leaders actively addressed stigma. LKNU did not originally include a counselling component in CEPAT support activities, but recognized the need for this and advocated that NTP community activities supported by the Global Fund include counseling. This is a work in progress, but LKNU has been successful in getting other aspects of the CEPAT approach adopted by the NTP in places where GFATM funds community activities. The NTP and GOI TB wasor were instrumental in developing the resource and training materials, and they have all received a full collection of all the resources produced by CEPAT. JKM conducted local studies that indicated that TB control problems stem from a low level of awareness and knowledge and with stigma in the community because people consider TB to be a “curse”. This was especially true amongst the many people who still hold traditional beliefs and primarily approach traditional healers when they are sick. Fear of discrimination was causing people to delay seeking help and treatment and negatively affecting adherence to the treatment for fear of being “found out” by neighbors. As a result, JKM developed and distributed a range of TB education materials, conducted a TB campaign (i.e., on-air TB talk show and discussion on local radio stations, and many music and traditional stage performances); and reached out to traditional healers to find ways to improve their understanding of TB. The radio talk shows utilized credible and well-known Professors, who described the epidemiology of TB in West Sumatra province, discussed facts and myths about TB to educate people about TB symptoms, and how it is spread. They called on the listeners to visit the closest health service or puskesmas to get tested. JKM and RCD worked with the traditional healers in several locations, including the remote Mentawai Island and mountainous villages subject to landslides. The traditional healers were diagnosing people by conducting autopsies on sacrificial chickens and through other similar means. The CEPAT teams worked with the healers to devise a way of ensuring the patient was reassured and supported to maintain 6 USAID INDONESIA PERFORMANCE EVALUATION REPORT the strength to purse treatment by the traditional healer, while receiving diagnostic and treatment services from the health system. Respondents regularly remarked that Cadres bringing people in to the puskesmas for testing was effective in combatting stigma, because they made it clear they were not afraid of people with TB, did not regard the family as cursed and were confidant they could be cured. Similarly, the TB Free village initiative in Papua, and other door￾knocking efforts provided the opportunity for the puskesmas staff to build bridges with communities by entering with the trusted Cadres. The IP took the opportunity to try a range of community mobilization strategies, and embed an appreciation for the complex and collaborative effort required to reduce the burden from TB. Prior to CEPAT many people in the community were not aware that free TB services and treatment are available through the puskesmas and hospitals. Cadres often acted as patient navigators, assisting with decision making, coordinating access to services, advocating for the patient and promoting patient centered care to the health service providers. They also played a range of other roles, often depending on their skills and interests, and the needs of their community. These roles include health promoter, community screening , advocate for resources, problem solver and case coordinator. The evaluation team found that the level of support and appreciation for CEPAT was consistently high at the sub-national level, and that the relationships among individuals, forged during the implementation of CEPAT, remain beneficial. RCD established 47 Village TB Posts in Papua and West Papua. These facilities are often located in the home of a Cadres or village leader which would have adequate space for drug storage, and for counseling the patients. They are prominently located in communities due to highly visible TB signage. Village TB Post are operated by villagers under the technical supervision of a government health facility. This results in improved health service access and, at the same time, empowers the community. RCD also initiated a Mobile TB outreach program to find TB cases, which involved the cadres and Government health staff working together to provide information about TB and reduce stigma, screen people, collect sputum samples and transport them for testing. This program reached remote villages, prisons, dormitories, and HIV hot spots. An internal evaluation showed that this activity did not yield very many TB cases, so it was stopped and replaced with a more intensive door-knocking and blusukan approach in areas where smear-positive people were identified and contact investigation interventions underway. RCD community empowerment activities were more obviously underpinned by a clear vision and commitment to increasing local capacity than was evident by the other two IPs. RCD CEPAT activities were planned, sometimes using a co-design process that actively engaged people from local areas or former TB patients. The RCD project coordinated and communicated regularly with the village officials and also the District Village Empowerment office, under the Ministry of Villages. CEPAT demonstrated that it is feasible to screen for TB symptoms in the general population followed by sputum smear microscopy to detect TB - even in rural and remote settings. “…It is very hard to change the stigma among the adults so we started by changing this in children, hence the coloring competition that reached 1000 children with a story about being a health hero. We encouraged the children to talk about TB with their parents and teachers as well as each other. We tried to reduce any fear of TB in the environment by starting with the children.” [Professor Rizanda Macmud, Dean of the Faculty of Nursing, Andalas University] USAID INDONESIA 7 PERFORMANCE EVALUATION REPORT CONCLUSIONS The strengths of the community mobilization approach include: • bringing TB healthcare services closer to TB patients, especially in Papua and West Papua and other locations where DOTS services are limited • prompting villages to plan collectively to prevent and respond to TB • engaging community health volunteers who were active in a range of roles (health promotor, patient navigator, case manager, support person), and worked to their strengths and interests • improving the quality of TB services through the establishment, facilitation of and participation in the pokja/working groups • linking TB with the concept of ‘amal’, a Muslim concept that values the willingness to do morally good deeds, comply with religious teachings, and help others • working together resulted in the staff of primary health care centers learning to value the contribution of the NGO and the Cadres, especially the training, and the ability to walk together in the community to find and support people with TB. While the IP met or exceed the targets related to screening and testing, bringing collectively, over 50,000 presumptive cases for testing at local health services, only 6,583 cases were detected amongst the population of more than 25 million people in the combined catchment areas. Of these only 4,887 people completed treatment. Given the project expenditure of over 7.3M USD, and a low case detection rate, it is clear that CEPAT developed a model that is too expensive to be replicated and scaled up in Indonesia. Some of the challenges faced by CEPAT include: • High turnover and distance from the NTP national program staff in Jakarta compromised the opportunity to engage with the GOI key decision makers and the CCM early and often, and to regularly update them on progress and ensure the program was measuring what mattered to them. • The IP and their collaborators in District and Provincial Health Offices had limited capacity to use program and other data to drive a systematic approach to mobilizing the community, target screening and identify where investment of the community’s resources were most needed or getting the best outcomes. • CEPAT emphasized community empowerment, but discussion within the program was largely focused on community mobilization. Empowerment activities were often sacrificed in order to focus on activities that resulted in measurable outputs. • There were high costs associated with running the program, largely due to the extensive capacity building that was required to enable Indonesian non-government organizations to receive direct funding from USAID and to manage this new type of activity. The local advocacy efforts increased the attention TB received from local authorities and decision makers, and resulted in the allocation of resources from local government offices, including from the Development Planning Agency, some Mayors, District health offices, Head of Community health centers, and village chiefs. 8 USAID INDONESIA PERFORMANCE EVALUATION REPORT In many locations, the IP were successful in securing small amounts of ongoing financial support for the mass community awareness raising, case finding and patient support activities after CEPAT funding finished. However, funding was usually not secured for recruitment and training of new volunteers, ongoing training for community leaders and resource development for guidance notes and other education materials. CEPAT Chief of Party and other key personnel successfully leveraged their networks to influence key decision makers, and gained critical support for proposals to maintain community level activities from District Health Office staff. The role of TB Cadres was not universally adopted by the health system, so these activities may not be continued, unless they are funded through local budgets or the Global Fund for HIV, TB and Malaria. The CEPAT activities were well-aligned with the NTP objectives, in that they aimed to contribute to active case finding, greater enrolment in and completion of treatment and reduced stigma around TB. The CEPAT program’s use of community health volunteers is well aligned with Indonesia’s utilization of Cadres in the maternal and child health program, however it is easier to get community volunteers involved in warm, rewarding programs like supporting mothers and promoting good health for babies. Working on a stigmatized infectious disease like TB is very difficult, so the incentives, support and training for volunteers requires a greater investment. The pattern of implementation, where not all sub districts were covered, did not allow for CEPAT to demonstrate better outcomes in the program areas than elsewhere. However, the burden of disease is high, the health budget is limited, and there may be greater appreciation for the role of community mobilization in the future. The key aspects of the CEPAT design, management, and implementation that worked well in terms of achieving the stated program goals are the close links of the IP with their communities, the IP ability to mobilize Cadres/community health volunteers, and drive local joint planning and problem solving forums and the passion and growth mindset the IP bought to their work. The program’s success was limited by contextual factors, such as the lack of precedent for engaging the community in partnership with the NTP, limited organizational capacity and ad hoc approaches to capacity building. The emphasis on gaining the knowledge and skills to meet USAID financial management and reporting requirements was prioritized, and although this was necessary, additional capacity building was also needed. USAID Mission staff stepped up in this regard, in many generous ways, however a comprehensive capacity building strategy was required to achieve ongoing quality and capacity improvement. The RCD approach, which embodied building the soft skills, (including initiative, self-directed learning, leadership and problems solving skills), provides a good example of what worked. The lessons that can be learned from directly funding local NGO in Indonesia include: • Substantial capacity building, guided by thorough annual capacity assessments and plans, is needed to ensure local NGO can work effectively with USAID, and be able to attract ongoing foreign donor support. • USAID staff must be supported and available to provide the sort of ‘back office’ functions normally provided by the International NGO headquarters and regional offices. If this is not feasible, other arrangements must be made, and people with a deep understanding of capacity building in the development sector, around financing, governance, management and technical aspects of programming, will be needed. • Direct funding to local partners will result in poor outcomes if the local NGO are not adequately supported to develop their effective strategies, expertise. leadership and capacity. This investment is likely to take longer than five years in countries and organizations starting from a very low base. There is an opportunity for NGO working on issues such as TB to learn from other local NGO working in sectors where capacity is greater, for example, climate change, human rights and HIV sectors. USAID INDONESIA 9 PERFORMANCE EVALUATION REPORT KEY RECOMMENDATIONS Any future community mobilization programs should be set up to explicitly innovate and test models, with dedicated resources for evaluating impact and acceptability. This will require buy in from the planning stage from the NTP leadership, sufficient resources, and a partnership between the NGO implementing the community based activities and a university or other organization with the expertise to conduct rigorous studies. Any similar programs should establish during conception what evidence will be needed and valued by key national decision makers, and then select the most useful indicators to support decision making re future investment. The local IP selected for CEPAT improved their capacity to collect data, but were in no position to be able to generate systematic evidence of what works. 1. Implementing partners require the capacity to use data to drive effective community screening and maximize case detection, and to inform the national program by answering key questions, including: what intervention and support in the community works to increase case detection and support good treatment outcomes? 2. IP need significant support and capacity building to focus on generating evidence of both cost effectiveness and impact. With sufficient support and a clear intention at the commencement of the funding, the IP could implement and evaluate a range of activities to answer the following questions: a) what is the average number of people needed to screen to detect one case of active TB?; b) how do population characteristics affect the number needed to screen? c) how does the underlying prevalence of TB and/or HIV affect the screening strategy ; and d) how do different screening tools and strategies affect the number needed to screen to secure a feasible level of return on investment in screening? The role of generating evidence may be considered by the USAID funded KNCV program, which has a TB focus and capacity to conduct research, or by a follow on activity. 3. The evaluation team did not find evidence that CEPAT actively engaged in discussions related to expanding activities in the community at the NTP level. The IP were largely disengaged from the NTP at the national level, and therefore unable to influence national policy. In future, strong links between USAID IP and the NTP must be established and maintained, to avoid USAID being accused of ‘doing their own thing’. 4. Support a partnership between the NTP and NGO to pilot innovative and risky approaches, whereby USAID will assist with: • Building capacity for co-design, monitoring, evaluation, accountability and learning, and build performance and progress indicators into contracting arrangements • Identifying appropriate financing mechanisms and incentives • Evaluating the cost and impact as well as ongoing potential for impact • Strategizing for roll out and going to scale once the pilot has been evaluated • Upskilling the workforce for a multi sectoral response 10 USAID INDONESIA PERFORMANCE EVALUATION REPORT USAID INDONESIA 11 PERFORMANCE EVALUATION REPORT ANNEXES ANNEX 1: EVALUATION TERMS OF REFERENCE Final Assessment and Document Lessons Learned from the Community Empowerment of People against Tuberculosis (CEPAT) Program 1. BACKGROUND: 1.1. USAID/Indonesia’s Tuberculosis (TB) Program and Strategy Tuberculosis (TB) is the leading infectious disease killer in Indonesia and poses a serious national public health threat. TB causes about 300 deaths every day – with more than one million new cases occurring each year. Indonesia has the second highest number of TB cases in the world, yet only 21 percent of Indonesian know of the risks posed by this disease, or its proper care and treatment. Though it is completely treatable, TB remains highly-stigmatized, and affects people in their most productive years. USAID’s programs support the reduction of TB infection rates, and the improvement of universal access to quality TB care. Since 2000, USAID has supported the Indonesia National TB Program (NTP). Initially, USAID’s support for Indonesia’s NTP was channeled through the TBCTA (Tuberculosis Coalition for Technical Assistance) Project, working closely with the NTP to implement DOTS in nine provinces at the primary health care level. TB control expanded rapidly in Indonesia between 2006 and 2010, reaching more communities through expanded health facilities (hospitals, clinics, etc.) and integrated TB-HIV services. From 2000-2006, USAID’s Tuberculosis Control Assistance Project (TBCAP) was implemented in 11 provinces to address capacity building, quality of care, health system strengthening, integration, coordination and sustainability. A five-year follow-on program, called TB CARE (2008-2013) provided a broad range of support to the NTP including: TB care and treatment (DOTS expansion and strengthening); expansion of programmatic management of drug resistant TB; TB/HIV care and treatment; and health systems strengthening. USAID’s current flagship TB project (Challenge TB 2014-2019) operates in 16 districts and focuses on five technical areas including: (a) Ensuring Universal Access by integrating TB in the National Health Insurance System (JKN), and securing increased local government funding for TB; (b) Increasing case detection: Intensified Case Finding to address the current gap in notification; (c) Ensuring the quality of treatment and care for TB, Drug resistant TB and TB/HIV co-infection; (d) Expanding the network of diagnostic services; and (e) Strengthening M&E, surveillance and operations research. 1.2. CEPAT Program in Indonesia In 2012, USAID/Indonesia launched the Community Empowerment of People against Tuberculosis (CEPAT) in partnership with NTP, which focused on enhancing technical capacity and skills related to TB detection and treatment services. The main objective of CEPAT was to support NTP in “achieving universal access to quality and early tuberculosis (TB) diagnosis and treatment through community empowerment.” To do this, CEPAT used two approaches: a. Mobilizing communities to support improved TB care including better access to and quality of TB services, increased community knowledge of TB, improved health seeking behavior, and increased support to patients on treatment. b. Advocating for increased TB resources and improved TB services. 12 USAID INDONESIA PERFORMANCE EVALUATION REPORT USAID awarded separate cooperative agreements to three Indonesian non-government organizations, including: Lembaga Kesehatan Nahdratul Ulama (LKNU), Jaringan Kesehatan/Kesejahteraan Masyarakat (JKM) and Roman Catholic Diocese of Timika (RCD). CEPAT was implemented in North Sumatra, West Sumatra, West Java, DKI Jakarta, East Java, Papua and West Papua provinces. Partnering with these NGOs, the program focused on increasing people’s awareness on TB prevention and early detection, as well as fostering local commitment to support the national program. CEPAT activities are targeted to improve TB care access and services to the most vulnerable populations including people living in urban slums, remote islands, malnourished people, the poor, and contacts of TB patients. Partnership with communities affected by TB is a critical element of the overall national TB control activities in Indonesia. The total funding of all three awards amounted to nearly $10 million over the life of project. Additionally, CEPAT was designed to address a key objective of the USAID Forward initiative -- to promote sustainable development through high-impact partnerships and local solutions. Thus, USAID sought to strengthen the capacity of the Indonesian NGOs selected to implement CEPAT. The goal of the capacity building was to strengthen the NGO so that it could better contribute to both the NTP’s strategic plan and CEPAT’s outreach efforts. USAID funded these capacity building efforts separately for each organization, based on their individual needs. CEPAT Awards in Indonesia Based on its two main components: (1) Mobilizing communities to support improved TB care; and (2) Advocating for increased TB resources and improved TB services. Activities under Component 1 focused on: a) Improved case finding; b) Increased access to service delivery; c) Improved health seeking behavior to quality TB services; d) Increased knowledge and awareness of key TB information; e) Increased contact tracing of TB patients; and f) Improved and expanded community social support systems for patients during treatment. Activities under Component 2 focused on: a) Increase the district budgets and allocated resources for TB; b) Increase advocacy at the local levels for TB services and resources; c) Increase the knowledge of TB among district leaders and other key stakeholders; and e) Reduce local policies inconsistent with national NTP guidance. Synopsis of the CEPAT Awards: A. Lembaga Kesehatan Nahdratul Ulama (LKNU) • LKNU is an organization under the PBNU (Pengurus Besar Nahdatul Ulama), and was granted a CEPAT award from November 2012-November 2017. LKNU worked in 10 districts in three provinces: DKI Jakarta, West Java, and East Java. • LKNU trained and established community Cadres to conduct contact-tracing, to ensure the appropriate testing of presumptive TB; and following up confirmed cases with Directly Observed Treatment Short-course (DOTS). This meant strong coordination with the puskesmas/local health center diagnostic and treatment units, and linking LKNU Cadres to ensure follow up and provision of community support for TB patients. LKNU also advocated for increased local USAID INDONESIA 13 PERFORMANCE EVALUATION REPORT budgets for TB care and treatment; and for strengthening local health regulations/policies to improve the effectiveness of TB programs, and expand TB-related services among government and private health providers. The program also provide technical assistance in strengthening community groups and patient support groups to engage with local governments in TB control; and initiated new approaches for community support groups to ensure complete treatment of Multi Drug-Resistant (MDR) TB patients in collaboration with Persahabatan Hospital in Jakarta. B. Jaringan Kesehatan/Kesejahteraan Masyarakat (JKM) JKM was the second local organization that received CEPAT award. The program ran from December 2012-December 2017. JKM operated in North Sumatra, West Sumatra and DKI Jakarta provinces. The program focused on community empowerment and ownership of TB programs as they play a key role in TB management, reducing burden on the health system, and reaching out to those who are not being treated or marginalized. As per December 2015, JKM trained more than 2,600 TB Cadres’ in14 districts in all three focus provinces. C. Roman Catholic Diocese (RCD) • RCD is implementing the third CEPAT award in Papua (June 2013-June 2018). RCD operates in Sorong city, Timika/Mimika, Paniai, Nabire, and Jayawijaya districts. The program supports strengthening the roles of civil society, the Church and traditional leaders in TB control, as well as advocating for improved resources, services and TB policy in Papua. • The CEPAT mid-term performance evaluation4 summarized the different CEPAT partner achievements as follows: Activity LKNU JKM RCD TOTAL People with TB symptomps tested 10,979 17,794 7,949 36,722 Confirmed smear positive for TB 1,353 2,463 869 4,685 Patients who were provided with support services 3,036 2,463 484 5,983 Patients successfully completed DOTS 1,621 1,298 192 3,111 2. OBJECTIVE OF THE CONSULTANCY: The objective of this short-term consultancy is to assess the overall performance of the CEPAT program and to extract and synthesize the key learnings from its implementation. This final assessment will provide insights and important feedback to USAID and its partners, to better understand both the strengths as well as areas where technical, administrative, and management could have been improved. Additionally, as this is USAID Indonesia’s first experience with directly issuing cooperative agreements to local organizations, it is important to document the related lessons learned. This assessment will provide evidence and learning for improving USAID’s future program designs and strategies. This assessment will: • Provide information on the lasting impact made by each award in (1) mobilizing communities to support improved TB care; and (2) advocating for increased TB resources and improved TB services; including relevant issues, sustainability, and cost effectiveness; • Determine to what extent the USAID CEPAT Program met its objectives; and what successes, challenges, and lessons learned can be drawn its implementation; 4 CEPAT Mid-Term Program Performance Evaluation. May 2016. 14 USAID INDONESIA PERFORMANCE EVALUATION REPORT • Examine whether implementation of these programs contributed to the goals of the Indonesian governmental (National – districts) policies and programs; • To determine how the USAID Forward objective related to building high-impact local partnerships was implemented and achieved under CEPAT, and describe how these capacity building efforts affected the organizational development, changes, and results; and • Make recommendations to help USAID better target efforts designed to enhance community engagement in order to support the national TB program. 2.1. Audiences and Intended Uses The primary audience of the evaluation report will be The USAID/Indonesia Mission, specifically the Health Office Team, the USAID/Washington, CDC Indonesia, Government of Indonesia, and other donors. The executive summary, expanded executive summary, and final report, will be provided to the GOI ministries and stakeholders. USAID Indonesia will integrate the assessment recommendations to the future activities aimed at leveraging community engagement, and share lessons learned and best practices with the related stakeholders. 3. METHODOLOGY OF THE ASSESSMENT: 3.1. Key Questions: • What influence did CEPAT have on the National TB Program? How well-aligned were CEPAT activities with NTP’s objectives? To what extent did CEPAT activities increase TB detection, prevention and treatment in the project areas? Did CEPAT strengthen the links between the community and health providers? • Did CEPAT strengthen capacity of the national and sub-national (province and district) government, private sector, community, and other stakeholders? • Have CEPAT community engagement approaches been sustained/replicated/adapted? • What aspects of CEPAT’s program design, management, and implementation worked well in terms of achieving the stated program goals? What elements hindered the attainment of program objectives, sustainability of results, innovation and replication? • How effective was the collaboration/coordination among CEPAT partners, between partners and host government and other stakeholders in maximizing efforts and achieving greater results? • What were the capacity strengthening interventions of CEPAT partners? What were the outcomes (intended and unintended), and overall results of these efforts on the different organizations? 3.2. Key Tasks: a) Conduct a desk review of international and national literature and related experiences; b) Conduct a desk review of relevant project background materials, including previous assessments and other documentation of each program component (if available); c) Review existing monitoring and evaluation data to build a detailed understanding of the project methodology and implementation experience; d) Attend a 2-day team planning meeting (TPM) at the start of the consultancy, which allows for the development of a common understanding of the assignment and how it fits within broader program and project activities; identifies clients involved in the assignment, understanding their relationships and interests, and a clear sense of how the consultant fits into this picture; and provides clarity of the team and individual scopes of work; USAID INDONESIA 15 PERFORMANCE EVALUATION REPORT e) Attend an assessment launch meeting. This meeting will provide the platform for the assessment team to a) to initiate discussions with implementing partners and stakeholders; b) to clarify the purpose and expected outcome of the review/evaluation; c) to ensure that implementing partners and evaluation team members are starting from the same frame of reference on the Indonesia situation; and d) to allow for an open and transparent discussion of USAID needs; f) Design primary data collection tools as needed, to respond to the key assessment questions. This will include applying a combination of qualitative and quantitative methods; g) Conduct focus group discussion, semi-structured discussions, and individual interviews in the different CEPAT program intervention sites; h) Conduct field visits: i. North Sumatera ii. DKI Jakarta iii. West/East Java iv. Papua/West Papua i) Analysis of data and other program information to develop evidence-based conclusions that are reliable, easily understood, useful, and applicable to USAID Indonesia technical program managers. 3.3. Interviews and Site-visits The Assessment Team will conduct in-depth interviews and focus group discussions involving the user and community, with the following organizations/staff: • Government: Ministry of Health, including selected province and district health office, selected provincial and district government in Indonesia, and NTP. • Beneficiary: patient and community (e.g. volunteers/Cadres, religious/local leaders). • Ex-Project staff: COP, Technical Officers, Provincial/District Coordinators. • Other stakeholders: relevant local CSOs, private sectors, universities, sub-awardees/grantees. Proposed provinces for the site visits are: North Sumatera, DKI Jakarta, West/East Java, and Papua. The team may be accompanied by a staff member from USAID/Indonesia, as appropriate, to observe interviews and field visits. A list of interviewees and key stakeholders will be provided by USAID prior to the assignment’s inception. 16 USAID INDONESIA PERFORMANCE EVALUATION REPORT 4. SPECIFIC ROLES AND RESPONSIBILITIES: KEY ROLE SPECIFIC RESPONSIBILITIES 1. Undertake a comprehensive review of the three CEPAT models of community engagement • Conduct a thorough desk review of relevant literature (international and country-level), as well as review of project documentation including monitoring and evaluation data, progress reports, mid-term assessments, and other relevant reports • Compare and contrast the different community engagement models implemented under the three CEPAT programs • Propose appropriate evaluation methodology including sample sizes for both quantitative and qualitative data collection; tools and steps for data collection and analysis (for review and approval by USAID) • Develop qualitative and quantitative data collection and analysis protocols, methodologies and tools, time frames, etc., • Conduct field visits to project areas and collect primary data using participatory approaches and a variety of qualitative methodologies (focus groups, key informant interviews, etc.) • Interview key stakeholders and groups within the project implementation sites including (but not limited to) CEPAT staff, senior management, outreach village Cadres, government counterparts, CSOs, and other groups as instructed by USAID • Compile all data instruments and data sets, and organize, analyze, and synthesize the data, including determining the need for additional information 2. Conduct a comprehensive, practical, and health program￾focused analysis of the CEPAT program • Detail and describe results, effects, constraints, and lessons learned from USAID CEPAT TB partners and other stakeholder-supported activities. • Identify gaps in CEPAT TB program implementation, including programmatic, leadership, funding, and geographic gaps. • Evaluate the level of coordination among CEPAT partners, between partners and host governments and other stakeholders. • Evaluate the level of sustainability/replication/adaptation of CEPAT program. • Develop recommendations and lessons on aspects related to factors that contributed to or hindered: attainment of CEPAT objectives, sustainability of program results, innovation, and replication. USAID INDONESIA 17 PERFORMANCE EVALUATION REPORT 3. Documentation, communication, and dissemination • Plan and conduct weekly update meetings and minutes to inform USAID of the status of the technical consultancy • Prepare summary reports on stakeholder consultation reports, as appropriate, with key objective, activities, findings and conclusions. • Prepare and present key findings to USAID and other stakeholders (including summary slides, full draft document) • Develop a final draft report (maximum 20 pages plus appendices) which will include a detailed analysis of key findings, recommendations, and conclusions • In-briefing meetings with USAID • Briefing meetings and presentations with MOH partners and other stakeholders as directed by USAID • Prepare an internal “future directions” memo for USAID with recommendations for future program design. • Finalize report based on comments from USAID. 5. COMPOSITION OF ASSESSMENT TEAM: The areas of technical expertise to address the technical foci of CEPAT include: • Knowledge of TB prevention, control and treatment (but not limited too): - Standard of TB prevention, control and treatment, including DOTS, MDR-TB, and TB-HIV; - Behavior change communication; - Public private partnership; - Community mobilization and development. • Knowledge of Organizational Development that includes (but not limited to): human resources development; financial, budgeting and management system; monitoring and evaluation; • Global, regional and national expertise; • Economics related to disease and disease control; and • Working experience in Indonesia. 5.1. Team Composition The Evaluation Team shall include technical specialists (one international and two local consultants). The team should have excellent skills and experience in monitoring and evaluation of health programs. • One international consultant with excellent expertise in planning and managing TB programs, with a particular focus on evaluation methodologies and community engagement; • Two local technical experts must have with an excellent understanding of the Indonesian public health and health system and policy as well as USAID or development programs, and be fluent in English with excellent writing skills • The evaluation team should have strong expertise in monitoring and evaluation of health programs. • In addition, there will be one translator/logistics support to assist the international consultant during interviews and presentations, and to handle the travel related logistics and provide administrative support to the evaluation team members; 18 USAID INDONESIA PERFORMANCE EVALUATION REPORT 5.2. Required Skills, Knowledge and Experience: This TOR will require a three-person team including one Team Leader/Senior Health Technical Expert and two local health technical experts Team Leader: • Master’s Degree (PhD preferred) in Public Health, Medicine, Social Sciences, or related field • At least 10 years of relevant work experience in health program design, management and evaluation, particularly in TB programs (preferably in Indonesia). • Extensive experience in leading studies/evaluations using a range of quantitative and qualitative data collection and analysis methods. • Extensive experience in working with community based outreach programs. • Proven ability to establish positive working relationships with a wide variety of stakeholders and demonstrated success in managing complex issues, with competing priorities, while still achieving positive results. • Highly-developed analytical and problem solving skills including a proven track record of creating practical solutions to complex problems; and analyzing data to develop strategically-focused reports with recommendations. • Excellent English writing, presentation and public speaking skills. • Ability to gather data, investigate and analyze information while being able to draw conclusions and issue relevant program recommendations. • Ability to create, compose and edit correspondence, written materials and/or complex reports. Two Assessment Team Members: • Master’s degree in public health. • Proven experience with research and analysis, with excellent expertise in designing and conducting monitoring and evaluation of programs. • Excellent skills and experience in the development of data collections tools and other instruments, and analysis of quantitative and qualitative data. • Fluency in Bahasa Indonesia. • Excellent English speaking and writing skills. 6. LEVEL OF EFFORT AND WORKING ARRANGEMENTS: This work will be awarded under individual consultant agreements and will be based in Jakarta, Indonesia, with field visits as agreed upon with USAID. We anticipate that this consultancy would require up to 3 months of time in-country. 7. SUPERVISION, REPORTING, AND PERFORMANCE EVALUATION: • Technical supervisor USAID: The Team Leader will report directly to the USAID Health Office Director, who will provide day to day oversight and provide input on the performance of the team. The Health Office Director may further delegate supervision to other technical steering groups as needed. • Team Leader: The Team Leader will be responsible for managing and delegating tasks to the two Indonesian assessment team members and ensuring the completion of all deliverables in a timely manner and to the complete satisfaction of USAID. USAID INDONESIA 19 PERFORMANCE EVALUATION REPORT 8. LOGISTICAL AND ADMINISTRATIVE SUPPORT: • Management, administrative and logistical support will be provided through the USAID BANTU project, including managing local travel; organizing any meetings or stakeholder consultations; providing and managing USAID-approved allowances, etc. 9. EXPECTED DELIVERABLES AND TIMELINES: TASK DELIVERABLE DUE DATE (based on start date) 1 Comprehensive review of data and development of data collection methodology and data analysis plan. Team planning meeting and development of detailed plan for field visits, stakeholder consultations, data analysis and report writing 1 week after start 2 Consultative process with stakeholders at different levels Field visits and data collection Data analysis and identification of additional data needs Months one and two 3 Synthesis of the major challenges and opportunities related to the different CEPAT outreach models Analysis of findings and development of draft report Review and revision of report Months two and three 4 Final report Presentations to USAID and other stakeholders In-briefing memos Data plan and sets Month three 20 USAID INDONESIA PERFORMANCE EVALUATION REPORT ANNEX 2: EVALUATION WORK PLAN Week Of 01.10 08.10 15.10 22.10 29.10 5.11 12.11 19.11 26.11 3.12 10.12 17.12 24.12 7.1 14.1 21.1 28.1 Consultant on Boarding Document Review Travel to Jakarta (TL &TM) Team Planning Meeting Development of data collection and Work Plan Logistics (With Assistance of CEPAT &USAID) Submit data collection draft 23 Data Collection Jakarta Data Collection North and West Sumatera Data Collection Cirebon and East Java Data Collection Papua&West Papua Data analysis and report writing Dry-run with BANTU 30 Outbriefs with USAID 5 Team Meeting 6/7 Travel from Jakarta (TL&TM) 12 Data analysis and report writing Draft report submission 18 USAID Review Report Finalization Nov 18 Dec 18 15/16 Jan 19 Field Work Analysis and Reporting Oct 18 Start Up and Coordination Key Completion of tasks Field Work X Submission of deliverable X USAID INDONESIA 21 PERFORMANCE EVALUATION REPORT ANNEX 3: EVALUATION PARTICIPANTS CEPAT Partners Nr. Name Gender Organization Position IDI/FGD 1. Esty Febriani F LKNU Former Chief of Party IDI 2. Ajat Sudrajat M LKNU Former District Coordinator - Cirebon IDI 3. Aziz Akbar M LKNU Former Finance Staff - Cirebon IDI 4. Wijiyanto M LKNU Former District Coordinator - Blitar IDI 5. Niswatu Syarifah F LKNU Former District Coordinator - Kediri IDI 6. Alfiati F LKNU Former Community Facilitator-Kediri IDI 7. Delyuzar Haris M JKM Former Chief of Party FGD 8. Juliandi Harahap M JKM Former TB Advisor FGD 9. Elly Arsam F JKM Former M&E Officer FGD 10. Ryzanda Machmud F JKM Former Advocacy Advisor FGD 11. Masroel M JKM Former Advocacy Advisor FGD 12. Dini Ardiani F JKM Former Office Manager IDI 13. Firdaus M JKM Former COD-Padang IDI 14. Moch.Irvan Rosiadi M JKM Former COD-West Jakarta IDI 15. Samhara Yudha M JKM Former COD-Medan IDI 16. Zul Hendrison M JKM Former COD-Solok IDI 17. Ernest Vincentius Sia M RCD Former Chief of Party IDI 18. Thea Hutanamon F RCD Former M&E Officer IDI 19. Alfred Petupetu M RCD Former Finance Manager IDI USAID Staff Nr Name Gender Organization Position IDI/FGD 1. Irene Koek F USAID Deputy Assistant Administrator/Former Director of Health-Indonesia Mission Phone interview 2. William Wells M USAID Senior TB Technical Advisor - USAID Global Health Phone interview 3. Mary Linehan F USAID Former Infectious Disease Team Leader – USAID Indonesia Health Office Phone interview 4. Jonathan Ross M USAID Former Director of Health-Indonesia Mission Phone interview 5. Artha Camelia F USAID Former Infectious Disease Senior Officer/ COR CEPAT TB IDI 6. Alya Budi Hartanti F USAID Former COR CEPAT TB IDI 7. Candyana Yohan F USAID Former COR CEPAT TB IDI 8. Pamela Foster F USAID Deputy Director of Health – USAID Indonesia IDI 9. Deddy Setiawan M USAID Financial Analyst, Office of Finance and Management, USAID Indonesia IDI 10. Yulian Christianti F USAID A&A Specialist; Office of Acquisition & Assistance, USAID Indonesia IDI 11. Adi Sarininggar F USAID TB Project Management Specialist IDI 12. Adhi Sanjaya M USAID TB Consultant IDI 13. John Langenbrunner M USAID Health Economist IDI 22 USAID INDONESIA PERFORMANCE EVALUATION REPORT Government of Indonesia Nr. Name Gender Organization Position IDI/FGD 1. Budiarti Setyaningsih F MoH Former NTP Officer IDI 2. Beeri Wopari M PHO - Papua Chief of AIDS, TB, Malaria (ATM) Control IDI 3. Hery Kurnia M DHO – Deli Serdang Chief of Infectious Disease Control FGD 4. Zurjani M DHO – Deli Serdang TB Vice Supervisor FGD 5. Titin Suwartini F DHO -Medan TB Vice Supervisor IDI 6. Reynold Ubra M DHO -Mimika Executive Secretary IDI 7. Alfred Delambey M DHO -Nabire Chief of Infectious Disease Control IDI 8. Eko Wijayanto M DHO-Blitar Head of Infection Disease Section IDI 9. Nur Munawarah F DHO-Kediri Head of Infection Disease Section IDI 10. Muhamad Subhan M DHO-Cirebon TB Vice Supervisor IDI 11. Ludya Wattimena F D H O - S o r o n g City Former TB Vice Supervisor IDI 12. Francisca Kilangin F DHO-Mimika Head of Infection Disease Section IDI 13. Maya Samuel F DHO-Mimika TB Vice Supervisor IDI 14. Supriyadi M DHO-Solok Chief of Disease Prevention and Control Section FGD 15. Aida Halima F DHO-Solok Head of Disease Prevention and Control Division FGD 16. Yake Iriantono M DHO-Nabire DHO Staff FGD 17. Asih F DHO-Nabire TB Program Staff FGD 18. Jeffrie Suska M Puskesmas Head of Puskesmas Labuan Haji, Deli Serdang IDI 19. Yanto Jumame M Puskesmas Head of Puskesmas Tanjung Kasuari, Sorong City IDI 20. Emma Kurnia Iriani F Puskesmas Head of Puskesmas Siriwini, Nabire FGD 21. Budi Irianto M Puskesmas Tb Program Officer, Puskesmas Siriwini Nabire FGD 22. Andi Ridwan M Puskesmas Head of Puskesmas Susukan, Cirebon FGD 23. Titis Nur’anisa F Puskesmas TB Program Officer, Puskesmas Susukan FGD Academics/experts Nr. Name Gender Organization Position IDI/FGD 1. Ignatius Praptoharjo M Atmajaya University Research Coordinator/Community Expert IDI 2. Pandu Riyono M University of Indonesia TB Expert IDI 3. Carmelia Basri F CCM TB Expert IDI USAID INDONESIA 23 PERFORMANCE EVALUATION REPORT External partners Nr. Name Gender Organization Position IDI/FGD 1. Yoana Anandita F WHO Indonesia National TB Officer IDI 2. Samhari Baswedan M CCM Executive Secretary IDI 3. Barry Adhitya M CCM TB officer IDI 4. Mariani Reksoprodjo F PPTI Executive Secretary of PPTI IDI 5. Daniel Marguari M Spiritia Foundation Executive Director IDI 6. Elis Wilden F UNAIDS Indonesia National Officer IDI 7. Totonta Kaban M Medan Plus Director IDI 8. Badurani Lubis M Galatea Director IDI 9. Sanusi M Suar Kediri Director IDI 10. Eva Damanik F PKBI West Sumatera TB-HIV Program Coordinator, PKBI West Sumatera FGD 11. Dahlia Taviano F PKBI West Sumatera Finance Staff FGD 12. Deri Rizal M Aisyiyah -West Sumatera Global Fund SR Coordinator, FGD 13. Rozi Rahmayuni F Aisyiyah -West Sumatera Global Fund SR Finance and Administration FGD 14. Rila Haryanti F Aisyiyah -West Sumatera Global Fund SR M&E FGD 15. Yunita Nazar F Aisyiyah -West Sumatera Global Fund SR FGD 16. Nurlatifah F Aisyiyah -West Sumatera Global Fund SR Finance and Administration FGD 17. Bey Sonata M KNCV Technical Director IDI 18. Erman Varella M KNCV Community Officer IDI Communities Nr. Name Gender Location Position IDI/FGD 1. Suhartini F Medan Baru Cadre FGD 2. Agusnawati F Medan Baru Cadre FGD 3. Ernawati Lubis F Medan Baru Cadre FGD 4. Gusna Tanjung F Medan Baru Cadre FGD 5. Jamilah F Medan Baru Cadre FGD 6. Elvina F Medan Baru Cadre FGD 7. Henny Hariani F Medan Baru Cadre FGD 8. Syarifah F Medan Baru Cadre FGD 9. Lita F Medan Baru Cadre FGD 10. Setiani Harefa F Medan Baru Cadre FGD 11. Susilawati F Medan Baru Cadre FGD 12. Anita F Medan Baru Cadre FGD 13. Irna Ningsih F Medan Baru Cadre FGD 14. Isnani F Deli Labuan Cadre FGD 15. Rosnita F Deli Labuan Cadre FGD 24 USAID INDONESIA PERFORMANCE EVALUATION REPORT Nr. Name Gender Location Position IDI/FGD 16. Eliyani F Deli Labuan Cadre FGD 17. Dhyra F Deli Labuan Cadre FGD 18. Ribiah F Deli Labuan Cadre FGD 19. Sumijah F Deli Labuan Cadre FGD 20. Endang Sriwahyuni F Talun Cadre FGD 21. Sri Agustina F Talun Cadre FGD 22. Uswatun Hasanah F Talun Cadre FGD 23. Umi Haryanah F Talun Cadre FGD 24. Siti Afifah F Semen Cadre FGD 25. Siti Zahro F Semen Cadre FGD 26. Binti Masruroh F Semen Cadre FGD 27. Iswati F Semen Cadre FGD 28. Rokatul Jannah F Semen Cadre FGD 29. Chusbiatun F Semen Cadre FGD 30. Nena F Tambora Cadre FGD 31. Iim Apriyani F Tambora Cadre FGD 32. Mardiana F Pemacungan Cadre FGD 33. Syamsul Chan M Pemacungan Cadre FGD 34. Darmeni F Solok Cadre FGD 35. Marlina F Solok Cadre FGD 36. Malinar F Solok Cadre FGD 37. Febri Astuti F Solok Cadre FGD 38. Delvita Kasmarini F Solok Cadre FGD 39. Martina Insyur Boseren F Siriwini,Nabire Cadre FGD 40. Osiana Gorogoro F Siriwini,Nabire Cadre FGD 41. Sueni F Susukan Cadre FGD 42. Amronah F Susukan Cadre FGD 43. Marhama F Susukan Cadre FGD 44. Evi F Susukan Cadre FGD 45. Endawati F Susukan Cadre FGD 46. Hj. Upi F Susukan Cadre FGD 47. Ani Latua F Susukan Cadre FGD 48. Sunengsih F Susukan Cadre FGD 49. Khunaenah F Susukan Cadre FGD 50. Halima F Tj.Kasuari Cadre IDI 51. Jarwasih F Semen Former TB Patient IDI USAID INDONESIA 25 PERFORMANCE EVALUATION REPORT ANNEX 4: EVALUATION QUESTIONS Key Questions for CoP/Managers of Implementing Partners (LKNU/ JKM/ RDC) Interviewer (s) Date Location Interviewee Gender Role Organization Religious affiliation Note to interviewer: choose from these questions according to the level of understanding of and responsibility for CEPAT of the interviewee. Purpose of the interview: To gain a deeper understanding of the CPEAT program, implementation, management and activities from the IP perspective. Introduction 1. How long were you the COP/local manager for CEPAT? 2. What training or other capacity strengthening did you receive while you preformed this role? 3. Did you work in this role full-time? What other roles did you hold at the same time? 4. In your opinion, how good was the fit between your NGO and USAID for the CEPAT Program? 5. What did you do to address the key challenges to engaging the Community in TB Control? Prompts:  Lack of public knowledge regarding TB symptoms and place to go for examination, diagnosis, and quality guaranteed treatment  Limited involvement of the patients, former patients and families and the wider community in TB control  Limited geographical coverage of civil society organizations and other stakeholders  Socio-economic barriers TB patients to access TB care  Persistent low public awareness about the rights and responsibilities of TB patients  High stigma among healthcare worker as well as in communities 26 USAID INDONESIA PERFORMANCE EVALUATION REPORT Key Questions on Management 1. Prior to CEPAT, had you ever received funding from an international donor? Was it under a similar arrangement as the CEPAT contracting arrangement? Or different? 2. Were you familiar with what entering into a Cooperative Agreement would involve? How prepared were you for this type of funding arrangement and the expected management and reporting? 3. Were there any aspects of CEPAT’s program design, or implementation that worked well in terms of mobilizing the community to respond to TB? Can you help me understand that a bit better? 4. Was there anything you think didn’t work well? Why? Can you think of an example that would help me understand this better? 5. Did you have any difficulty in recruiting or retaining the staff you needed? 6. From your involvement in CEPAT, what have you learned about undertaking work with financial support from an international donor/development partner? 7. Did you use your CEPAT data to reflect on how the program was progressing? Can you help me understand how you used the key performance indicator data? Capacity Building 1. Please tell us about your work with Capacity Building Partners. How did you select the CBP, identify the organizational capacity development needs and develop plans, what kind of support did you receive? PROMPT For detailed examples of any CB, including:  direct training to build capacity (financial, procurement, reporting);  mentoring;  capacity building partners/technical assistance partners;  joint program monitoring and field trips; and  manuals on how to work with USAID. 2. In your opinion, did (your organization) have the capacity to absorb the TA and support they received? Why/why not? 3. Over the five years you implemented CEPAT, did you observe any significant changes in capacity? What could you do more of? Or did you become better at? 4. In your view, has the above capacity development helped your organization in getting support from other funding sources and implementing the similar activities after CEPAT ended? If not, why? 5. How were USAID involved in ensuring your staff could meet USAID requirements? And any other training or capacity building? 6. Are there any lessons learned about how to strengthen capacity in the future? USAID INDONESIA 27 PERFORMANCE EVALUATION REPORT Community mobilization and empowerment 1. Was there a vision of what an empowered community would look like? Or what strategies were designed to empower the community? 2. Were your project staff able to do anything innovative to ensure that the community activities reached all kinds of people? Did you plan to include hard to reach populations? What was the result? 3. Did you do anything to ensure you were successful at meeting the specific needs of males and females and transgender or gender non-binary people? 4. Since CEPAT ended, have you seen any change in the community response to TB? If yes, what are they? If no, in your opinion, why? 5. Have CEPAT community engagement or empowerment approaches been sustained/replicated/ adapted? How? Advocacy 1. How effective was the collaboration/coordination among CEPAT partners, between partners and host government and other stakeholders in maximizing efforts and achieving greater results? How it could be improved? 2. Could you tell me more about advocacy component of your program? 3. Are there any examples of CEPAT advocacy that resulted in activities being funded or supported locally? 4. What do you think about the sustainability of results, innovation and replication? 5. What could have been done to make the advocacy more effective? What kind of support would you have needed to do this? Impact and Sustainability 1. Were you able to achieve what you set out to do with the CEPAT funding? PROMPT: Were there any circumstances that helped you to achieve the CEPAT objectives? 2. Did you experience any difficulty in reaching your CEPAT targets? PROMPT: were there any circumstances that made it difficult to do what you intended? Can you think of any activities included in your workplan that you found it difficult to complete? 3. When you think about your efforts to support the community, was there anyone you had difficulties reaching or supporting? Why? Can you help me understand that a little better? 4. Was there anything that you or the Cadres were able to do, or change, that made it easier for people to use TB services? 5. What were your key achievements? What key factors contributed to your achievements? 6. To what extent do you think CEPAT activities increased the engagement of the community in TB 7. Has CEPAT made any contribution to the National and local TB Program? What makes you think that? 8. In your opinion, is there any lasting impact made by CEPAT in empowering people and mobilizing communities to support improved TB care? 28 USAID INDONESIA PERFORMANCE EVALUATION REPORT Lessons learned Is there anything you think the CEPAT program could have done differently? 1. Based on your experience, do you have any suggestions/ ideas to improve similar TB program in the future? 2. Do you have any other comments regarding CEPAT or other community based TB Program? THANK YOU USAID INDONESIA 29 PERFORMANCE EVALUATION REPORT INTERVIEW/GROUP DISCUSSION GUIDE FOR CADRES of Implementing Partners (LKNU/ JKM/ RDC) Interviews or small group discussions Interviewer (s) Date Location Interviewee/s Gender Role Organization Religious affiliation Purpose of the interview: The purpose of interviewing Cadres is to gain an understanding of 1) how the chosen strategy (providing outreach and support services delivered by Cadres) contributed to mobilizing communities to support improved TB care including better access to and quality of TB services, increased community knowledge of TB, improved health seeking behavior, and increased support to patients on treatment; 2) any lasting changes in empowerment of people against TB, and sustain community activities in TB program after CEPAT closed. Introduction • How long were you a Cadres for CEPAT/TB program? Are you still a cadre now (post CEPAT ends)? • How did you become involved with CEPAT? Why? What motivated you to get involved in responding to TB? Was there any benefit of being a cadre (monetary AND/ non-monetary)? Questions for cadres (Interview or small group discussions) • Can you tell us about your experience working on TB in the community? What did your role as a TB Cadres involve? • Did your experience as a cadre increase your skills and expertise at working with the community? What could you do more of? Or did you become better at? • PROMPTS: Check if they think the program strengthened their capacity in any of the following, and how. Try to get specific examples of any changes: o Improving case finding, expanding into new networks, reaching hard to reach populations; o Increasing access to service delivery; o Promoting health seeking behavior, encouraging people to use TB services; o Increasing knowledge and awareness of TB o Improving their contact tracing of TB patients; 30 USAID INDONESIA PERFORMANCE EVALUATION REPORT o Improving and expanding community social support systems for patients during treatment o advocacy • Did you gain any additional knowledge of TB yourself, from being involved in CEPAT? Has this been of benefit to you, or your family, or your community? • Can you describe your role in contact investigation of TB patients? • Were there any circumstances that helped you to succeed with your CEPAT activities? • was there anyone you had difficulties reaching or supporting? Why? Can you help me understand that a little better? • Was there anything that you or the other Cadres could do, or change, that made it easier for people to use TB services? • What are your thoughts about the impact of CEPAT? Can you describe any changes you saw in the community as a result of your work/ CEPAT? • Can you describe any opportunities you had to review how well your activities were working? Were you able to use the monitoring data to see if [name initiatives, for e.g. Mobile TB/mass screening] were working? • What would happen to someone with TB if you couldn’t provide them with support to get screening, or treatment? • Can you provide any examples of sustained community-based TB activities that have continued since CEPAT closed? • Can you share with me any examples of feedback you received from the community about CEPAT program? • Was there anything you think didn’t work well? Why? • Did you face any challenges and difficult situations when you were doing your work? Please share some examples – • What kind support do you think would make your work to be more effective? Closing • Do you have any other comments regarding CEPAT Program or other community-based TB Program? • Based on your experience, do you have any suggestions/ ideas to improve similar TB program in the future? USAID INDONESIA 31 PERFORMANCE EVALUATION REPORT Interview Guide for Puskesmas TB Program Staff Interviewer (s) Date Location Interviewee Gender Role Organization Religious affiliation Purpose of the interview: To gauge information or insight into the experience of the key informants regarding TB CEPAT Program implementation as well as coordination with Puskesmas. Key Questions 1: Introduction 1. How are you connected to the TB Program (Roles and Position) and for how long? 2. What TB diagnostic and treatment services does this puskesmas offer? 3. What patient support services do you offer? During the time that CEPAT was being implemented and collaborating with your puskesmas, were there any other community-based partners also working on TB with you? 4. Current situation or update of the TB program in your Puskesmas area/? (how many CSO working on TB program). Key Questions 2: (Case detection and patient support) • In your catchment area, what type of people are most likely to be infected with TB? • Without active case finding, what path do patients usually take to find their way to this puskesmas (do they see a drug seller in the market? A pharmacist? A private doctor? A traditional healer?) • Prior to CEPAT partnering with your health center, did you do any active case finding in the community? What changed after CEPAT started partnering with you? • Can you tell me about the things CEPAT did to screen and refer patients and link them to your service? Did your staff plan or carry out any of these activities with CEPAT staff? • How has CEPAT contributed to reducing stigma related to TB? What about the stigma by health professionals? • Could you tell us, what was the role of CEPAT Program in contributing to the achievement of overall TB program in your area? (Probe: e.g. Probe: What were the most successful aspects of the CEPAT approach? What worked well? • Were there any activities that CEPAT assisted with that you think may have reduced stigma? Did you see any changes? • (Ask the staff to show the IEC materials that address the stigma in the community? Do any materials target health professionals?) • Could you tell me about your impression of the TB Cadres (what did they do, how often, how 32 USAID INDONESIA PERFORMANCE EVALUATION REPORT well? Were there problems as Cadres turned over?) • Based on your experience, is there any benefit to the Puskesmas in having a community partner? Key Questions 3: (TA) • What sort of technical assistance or capacity building was provided by CEPAT for health staff (remind re TB diagnosis, safe sputum collection, MDR TB, communication & counselling) • Was training provided in a place/at a time convenient for your staff? • How useful was the training offered? How useful was the training your staff participated in? • Looking back, is there anything you think would have been useful to include in the training? • Looking back, are there any changes you would make to how training was provided? Any lessons learned? Key Questions 4: (Sustainability) • Have you been able to continue any of the activities that were working well or developed under CEPAT? • Probe: o If yes, which ones, why and how was this done? o If not, what were the reasons and the obstacles. • Did you work with CEPAT n any planning or activities to advocate for resources to continue the community-based activities after CEPAT finished? How did that go? Key Questions 5: • How was the communication between you and CEPAT? • Were there any regular coordination meetings held? Who participated in them? How often? Any results? • Has the Tb program been able to make any links with the HIV program? Is there routine Tb screening of HIV patients? Closing • Was there anything you think the CEPAT program could have done differently in order to achieve better results? • Do you have any other comments regarding CEPAT Program or other community-based TB Program? THANK YOU USAID INDONESIA 33 PERFORMANCE EVALUATION REPORT Interview Guide Community Organizational Development Officer Interviewer/s Date Location Interviewee Gender identity Role Former Community Organizational Development Officer Organization Purpose of the Interview: To better understand the IP approach to ensuring the Cadres, primary health care staff, parliamentarians and others have the requisite knowledge and skills to empower their community to take action in relation to TB, which includes improved engagement of the provincial- and district-level health leaders; increased resources for the TB program; and an improved local legal and regulatory environment for TB. Interview questions 1. Can you outline what your role as a Community Organizational Development involved? 2. What training or other capacity strengthening did you receive while you preformed this role? 3. Did you work in this role full-time? What other roles did you hold at the same time? 4. What did you do to address the key challenges to engaging the Community in TB Control? • Prompts: • Lack of public knowledge regarding TB symptoms and place to go for examination, diagnosis, and quality guaranteed treatment • Limited involvement of the patients, former patients and families and the wider community in TB control • Limited geographical coverage of civil society organizations and other stakeholders • Socio-economic barriers TB patients to access TB care • Persistent low public awareness about the rights and responsibilities of TB patients • High stigma among healthcare worker as well as in communities Empowerment 1. Did (CEPAT IP) develop a definition of ‘empowerment’ or hold a clear vision of how empowerment would come about? 2. How is this different from community mobilization? Is there a gap between what was done to achieve short term community mobilization and what is needed to achieve long term empowerment? 34 USAID INDONESIA PERFORMANCE EVALUATION REPORT 3. Based on your experience, what do you think (CEPAT IP) learned about how to effectively empower people to respond to TB in the community? 4. In your opinion, did the CEPAT program do everything necessary to result in sustained empowerment of the community against TB? (Lessons learned about what else the project could have done or should have done differently) 5. What role did the Advisory Committee play? Who was most actively involved in the Committee? What affect, if any, did this committee have in promoting and sustaining the community mobilization in the respective areas? in what way? If it was ineffective, why? 6. What do you think the CEPAT Program contributed to the National TB Program goal of Engaging the Community in TB Control? 7. Can you identify any improvements in the local legal and regulatory environment for TB that CEPAT contributed to? How was this improvement achieved? 8. What were the most successful methods used by (CEPAT IP) to engage the provincial- and district-level health leaders? 9. Do you have any evidence that the changes introduced by CEPAT have been sustained or even built upon? Note re COD: The role of the COD includes: training people in the community to draft Dana Desa proposal to include an allocation of funds for TB from the village funds, including increasing health promotion activities, and proposals to secure an allocation for TB from the health budget allocation. They met with and trained the community leaders, Puskesmas officers, community cadres, medical and psychology students to conduct community outreach and increase knowledge of TB and reduce stigma and discrimination. Note re: JKM in Sumatra had a COD and advisory committee, including a provincial health office representative, an NTP representative, and district office representatives from each province. Their role was to provide advice to the CEPAT team on community mobilization and advocacy. The COD coordinator, provincial CODs, district CODs, and advocacy personnel in each province organized the committees. JKM implemented CEPAT in West Sumatra Province (six districts): Solok, Kota Padang, Tanah Datar, Pasaman, Padang Pariaman, and Kepulauan Mentawai and DKI Jakarta USAID INDONESIA 35 PERFORMANCE EVALUATION REPORT ANNEX 5: REFERENCE DOCUMENTS • Ministry of Health Republic of Indonesia, 2016. National Strategic Plan of Tuberculosis Control 2016- 2020 • LKNU Program Monitoring Plan, 2013 • LKNU Annual Report Final Year 1, 2013 • LKNU Annual Report Final Year 2, 2014 • LKNU Annual Report Final Year 3, 2015 • LKNU Annual Report Final Year 4, 2016 • LKNU Annual Report Final Year 5, 2017. • LKNU Annual Work Plan Year 1, 2013 • JKM Final Work Plan, 2013 • JKM Annual Report Final Year 1, 2013 • JKM Annual Report Final Year 2, 2014 • JKM Annual Report Final Year 3, 2015 • JKM Annual Report Final Year 4, 2016 • JKM Annual Report Final Year 5, 2017 • Capacity Assessment Report, Institutional Assessment Mission for CEPAT JKM, March 2015. • Organizational Development Component 1: Strengthened Financial Systems, Tools, and Standard Operating Procedures CEPAT JKM, Yayasan Penabulu, 2015 • JKM. 2015. Organizational and Human Resources Standard Operational Procedure. • RCD Program Monitoring Plan • RCD Annual Report Final Year 1, 2015 • RCD Annual Report Final Year 2 2015 • RCD Annual Report Final Year 3. 2016 • RCD Annual Report Final Year 5, 2018 • RCD CEPAT Program Proposal, 2012 • USAID Indonesia. 2015. Data Quality Assessment Report: Findings and Recommendations. • Mid-Term Program Performance Evaluation, Community Empowerment of People Against Tuberculosis, USAID, 2016. • USAID-Indonesia. 2011. Request for Full Applications (RFA) No. RFA 497-11-000005 Amendment No.1, Community Empowerment of People Against Tuberculosis, CEPAT • USAID, 2014. Engaging NGOs and other CSOs in TB Control in Indonesia. Country Strategy: April 2014. • USAID, 2014. Local Systems: A Framework for Supporting Sustained Development. • USAID, 2016. Strengthening Local Capacity in Southeast Asia: Approaches and Experiences. • Anna Wetterberg, Derick W. Brinkerhoff & Jana C. Hertz (2015) From compliant to capable: balanced capacity development for local organisations, Development in Practice, 25:7, 966-985, DOI: 10.1080/09614524.2015.1073224 U.S. AGENCY FOR INTERNATIONAL DEVELOPMENT 1300 PENNSYLVANIA AVENUE, NW WASHINGTON, DC 20523 MAY 2019 This report is made possible by the support of the American People through the United States Agency for International Development (USAID). The contents of this report are the sole responsibility of the authors, Jenne Roberts, Octavery Kamil and Isma Novitasari, and do not necessarily reflect the views of USAID or the United States Government.