July 2019 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by the Ethiopia Performance Monitoring and Evaluation Service (EPMES) of Social Impact, Inc. EVALUATION Final Performance Evaluation of the Challenge Tuberculosis Activity Photo by Social Impact, Inc. Patient collects sputum at Addis Aketema Health Center, Addis Ababa DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. FINAL PERFORMANCE EVALUATION OF THE CHALLENGE TUBERCULOSIS ACTIVITY Final Evaluation Report July 2019 Evaluation Mechanism Number: AID-663-C-16-00010 Ethiopia Performance Monitoring and Evaluation Service for USAID/Ethiopia Activity Cover photo by USAID/Challenge TB Authors/Evaluation Team Dr. Beulah Jayakumar, Team Leader Dr. Ezra Shimeles Dr. Amsalu Bekele Dereje Mamo i Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity ABSTRACT In 2014, the United States Agency for International Development (USAID) funded the KNCV Tuberculosis Foundation, Management Sciences for Health and the World Health Organization to implement the five-year Challenge TB (CTB) Activity to support Ethiopia’s National Tuberculosis Program (NTP) in reducing TB incidence and related deaths by providing capacity building and technical assistance to the government, agencies, and health facilities to strengthen program management and service delivery. USAID/Ethiopia contracted Social Impact, Inc. to conduct a performance evaluation to examine Activity effectiveness, intervention challenges, and provide recommendations to guide decisions on future TB activity design and implementation. The evaluation team used mixed methods including document review, secondary data analysis, key informant interviews, and observations to identify findings that addressed USAID’s evaluation questions. The evaluation found that the range and reach of CTB interventions and technical assistance were comprehensive, high quality, and responsive to NTP needs. However, design and implementation challenges impeded efficiency (implementation delays, lack of depth in use of data for decision making), and effectiveness (variability in adherence to standards for case finding, treatment, and community-based interventions). While CTB excelled in supporting the expansion of critical diagnostic services, management of drug-resistant TB, and research, there was evidence of inconsistent practice of active case finding. Further, while capacity building activities were especially robust at zone and woreda level, the expectation of continued support from CTB staff for financial support and overall activity management limits long-term sustainability. The report makes eleven recommendations directed to USAID and the NTP on priorities for future design and implementation. ii Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity ACKNOWLEDGMENTS The evaluation team expresses their sincere appreciation to various individuals who made significant contributions to the success of this final performance evaluation. Special thanks go to Awoke Tilahun, Mission Monitoring and Evaluation Specialist of USAID/Ethiopia and EPMES Contracting Officer's Representative (COR), and Dr. Yared Kebede Haile, Challenge TB Activity Manager, and Dr. Yewulsew Kassie for their technical contributions to the evaluation design and the report. Furthermore, special thanks go to the staff of Challenge TB for their unreserved support of this evaluation, without which this evaluation would not have been successful. They freely shared Activity documentation and databases, which were essential to the design and secondary analysis of the Activity results; helped the evaluation team with the scheduling of interviews; and willingly participated as key informants during the interviews. The team is extremely grateful for the direction, oversight, and support received from Francis Ogojo Okello, Chief of Party, EPMES. The team is also thankful for the participation of Worku Ambelu, Senior Evaluation Specialist/Evaluation Manager, EPMES, and Henok Metaferia, MIS Specialist, EPMES in data collection, compilation, and analysis. The team is grateful for the logistical support provided by Mahlet Yared, Deputy Chief of Party, EPMES and other staff at the EPMES office during the time the team spent in Addis Ababa and during field visits. Special thanks go to Tasneem Nahar/SI Program Director who served as the Project Director for this evaluation; Leah Ghoston/SI Technical Specialist for her support in all stages of the evaluation; Mike Pressl/SI Senior Program Associate, who served as the Project Manager; Nora Chamma/SI Senior Program Assistant, who, in addition to providing operational support, also worked with Mike Pressl to provide excellent editorial support. Finally, the evaluation team thanks the staff of the national TB program, national agencies, partner organizations and professional associations, government and partner staff in the regions visited, staff of health facilities, and patients who served as key informants and allowed us to observe services provided and received, and shared their perspectives. This evaluation would not have been successful without their participation. April 2019 The Evaluation Team iii Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity CONTENTS Abstract........................................................................................................................................... i Acknowledgments......................................................................................................................... ii Contents........................................................................................................................................ iii List of Tables and Figures............................................................................................................................................ v Acronyms...................................................................................................................................... vi Executive Summary....................................................................................................................viii Activity Background................................................................................................................................................... viii Evaluation Purpose..................................................................................................................................................... viii Evaluation Design, Methods, and Limitations...................................................................................................... viii Key Findings and Conclusions................................................................................................................................... ix Evaluation Question 1: To what extent did CTB’s technical assistance and management approach support CTB cooperative agreement objectives? ........................................................................................... ix Evaluation Question 2: To what extent did CTB implementation approaches use international standards and proven strategies?.......................................................................................................................... x Evaluation Question 3: Was the information generated by CTB used to help achieve objectives and outcomes and, if so, how?...................................................................................................................................... x Evaluation Question 4: What were CTB’s main achievements and challenges? ...................................... xi Evaluation Question 5: To what extent did CTB’s methodologies, interventions, and management set the stage for sustainability and ownership of project outcomes?....................................................... xiv Key Learning and Implementation Adaptation.................................................................................................... xiv Recommendations....................................................................................................................................................... xv Evaluation Background................................................................................................................. 1 Activity Background...................................................................................................................................................... 1 Context....................................................................................................................................................................... 1 Activity Overview..................................................................................................................................................... 2 Evaluation Purpose........................................................................................................................................................ 4 Evaluation Questions.................................................................................................................................................... 4 Evaluation Design, Methods, and Limitations......................................................................................................... 4 Document review..................................................................................................................................................... 4 Primary data collection methods.......................................................................................................................... 5 Sampling for primary data collection ................................................................................................................... 7 Secondary data analysis........................................................................................................................................... 8 Data preparation and analysis................................................................................................................................ 8 Ethical considerations.............................................................................................................................................. 8 Gender and social analysis plan............................................................................................................................. 8 Limitations.................................................................................................................................................................. 9 Findings and Conclusions............................................................................................................ 10 iv Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity Evaluation Question 1: To what extent did CTB’s technical assistance and management approach support CTB cooperative agreement objectives?............................................................................................... 11 Findings...................................................................................................................................................................... 11 Conclusions.............................................................................................................................................................. 17 Evaluation Question 2: To what extent did CTB implementation approaches use international standards and proven strategies?............................................................................................................................. 18 Findings...................................................................................................................................................................... 18 Conclusions.............................................................................................................................................................. 20 Evaluation Question 3: Was the information generated by CTB used to help achieve CTB objectives and outcomes and, if so, how?................................................................................................................................. 20 Findings...................................................................................................................................................................... 20 Conclusions.............................................................................................................................................................. 22 Evaluation Question 4: What were CTB’s main achievements and challenges?.......................................... 23 4.1. Achievements and challenges in supporting TB program management capacity....................... 23 Findings...................................................................................................................................................................... 23 Conclusions.............................................................................................................................................................. 26 4.2. Achievements and challenges in improving diagnostic services..................................................... 26 Findings...................................................................................................................................................................... 26 Conclusions.............................................................................................................................................................. 31 4.3. Achievements and challenges in improving case finding.................................................................. 32 Findings...................................................................................................................................................................... 32 Conclusions.............................................................................................................................................................. 37 4.4. Achievements and challenges in supporting improving case management.................................. 40 Findings...................................................................................................................................................................... 40 Conclusions.............................................................................................................................................................. 43 4.5. Achievements and challenges in TB prevention activities............................................................... 43 Findings...................................................................................................................................................................... 43 Conclusions.............................................................................................................................................................. 45 High impact interventions at a glance................................................................................................................ 45 Evaluation Question 5: To what extent did CTB’s methodologies, interventions, and management set the stage for sustainability and ownership of project outcomes?.................................................................... 46 Findings...................................................................................................................................................................... 46 Conclusions.............................................................................................................................................................. 48 Overall Conclusions .................................................................................................................... 51 Key Learnings and Implementation Adaptations .................................................................... 53 Recommendations ...................................................................................................................... 56 Annexes........................................................................................................................................ 59 Annex I: Evaluation Statement of Work .................................................................................. 60 v Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity Annex II: Detailed Description of Evaluation Design and Methods ....................................... 74 Annex III: Evaluation Design Matrix.......................................................................................... 78 Annex IV: Data Collection Tools............................................................................................... 87 Annex V: Information Sources ................................................................................................ 140 Annex VI: Disclosures of Conflict of Interest ......................................................................... 143 Annex VII: Evaluation Team Member Profiles and CVs ....................................................... 148 Annex VIII: Operations Research and Presentations ............................................................ 149 List of Tables and Figures Table 1: Categories of key informants.......................................................................................................................... 5 Table 2: List of sites and samples................................................................................................................................... 7 Table 3: Achievement by Activity objective .............................................................................................................. 10 Table 4: CTB indicators: achievement against targets............................................................................................ 10 Table 5: Program decisions resulting from ORs...................................................................................................... 21 Table 6: Functioning of GeneXpert in sites visited.................................................................................................. 28 Table 7: Changes in access to diagnostic services................................................................................................... 31 Table 8: Outcomes of CI, CTB areas (regions and city administrations) .......................................................... 33 Table 9: Outcomes of mass screening among KPs.................................................................................................. 34 Table 10: Summary of discussion on case-finding strategies................................................................................. 40 Table 11: High impact interventions at a glance....................................................................................................... 46 Table 12: Status of key program areas at the start and end of CTB................................................................... 49 Table 13: Selection of regions using criteria ............................................................................................................. 74 Table 14: Selection of zones......................................................................................................................................... 74 Table 15: Selection of districts and facilities in selected zones............................................................................. 75 Table 16: Assumptions of facility types/locations for estimating maximum number of respondents......... 75 Figure 1: Trends in TB incidence and TB mortality during CTB period .............................................................. 1 Figure 2: CTB operational areas.................................................................................................................................... 2 Figure 3: CTB objectives and sub-objectives.............................................................................................................. 3 Figure 4: CTB timeline and inclusion of geographic areas..................................................................................... 17 Figure 5: Xpert tests - total and positive, national .................................................................................................. 29 Figure 6: Distribution of types of DS TB cases notified, national ........................................................................ 30 Figure 7: DS TB case notification trend, national..................................................................................................... 35 Figure 8: Trend in case detection rate, national....................................................................................................... 36 Figure 9: DR TB cases notified, against targets and estimates, national............................................................. 36 Figure 10: Treatment outcomes, bacteriologically confirmed DS TB patients, national ................................ 41 Figure 11: Treatment outcomes, DR TB patients, national................................................................................... 42 Figure 12: U5s among all contacts of DS TB cases screened, CTB areas.......................................................... 44 Figure 13: Screening and IPT for U5s, CTB areas.................................................................................................... 45 vi Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity ACRONYMS AHRI Armauer Hansen Research Institute ALERT All Africa Leprosy Regional Training (Hospital) ART Antiretroviral Therapy CDCS Country Development Cooperation Strategy CI Contact Investigation CNR Case Notification Rates CRC Clinical Review Committee CSO Civil Society Organization CTB Challenge Tuberculosis CTBC Community Tuberculosis Care DOT Directly Observed Treatment DR TB Drug -resistant Tuberculosis DS TB Drug -sensitive Tuberculosis DSM Drug Supply and Management DST Drug Susceptibility Testing ECG Electrocardiogram EPHI Ethiopian Public Health Institute EPMES Ethiopia Performance Monitoring and Evaluation Service EPS Ethiopian Pediatric Society EQA External Quality Assurance FMoH Federal Ministry of Health GF Global Fund GHC Global Health Committee GLRA German Leprosy/Tuberculosis Relief Association GoE Government of Ethiopia HEAL TB Helping Ethiopia Address Low Tuberculosis Performance HEW Health Extension Worker HF Health Facility HIV Human Immunodeficiency Virus HMIS Health Management Information System HSS Health Systems Strengthening HQ Headquarters IEC Information, Education, and Communication IP Infection Prevention IPLS Integrated Pharmaceutical Logistics System IPT Isoniazid Preventive Therapy KII Key Informant Interview KNCV Koninklijke Nederlandse Centrale Vereniging KOICA Korean International Cooperation Agency KP Key Population LPA Line Probe Assay LTBI Latent Tuberculosis Infection LTFU Lost to Follow-up LQMS Laboratory Quality Management System M&E Monitoring and Evaluation MDR Multidrug-Resistant MSH Management Sciences for Health MTB/RIF Mycobacterium Tuberculosis/Resistance to Rifampicin vii Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity NSP Revised National Strategic Plan for 2013-’20 NTP National Tuberculosis Program OPD Outpatient Department OR Operations Research PEPFAR President’s Emergency Plan for HIV/AIDS Relief PSA Pharmaceutical Supply Agency PIH Partners in Health PLHIV People Living with HIV PMDT Programmatic Management of Drug-resistant Tuberculosis RDQA Routine Data Quality Assessment RHB Regional Health Bureau RR TB Rifampicin-resistant tuberculosis SL-LPA Second-line Line Probe Assay SNNPR Southern Nations, Nationalities, and Peoples’ Region SOC Standards of Care SOP Standard Operating Procedures SOW Statement of Work STR Short Treatment Regimen STTA Short-term Technical Assistance TA Technical Assistance TB Tuberculosis TB IC Tuberculosis Infection Control TFC Treatment Follow-up Center TIC Treatment Initiation Center TRAC Tuberculosis Research Advisory Committee TWG Technical Working Group USAID United States Agency for International Development USD United States Dollars WHO World Health Organization ZHD Zonal Health Department viii Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity EXECUTIVE SUMMARY Activity Background Ethiopia continues to be among the world’s 30 high-burden countries for tuberculosis (TB), TB HIV, and multidrug-resistant (MDR) TB. With nearly 30 percent of TB cases and 75 percent of estimated MDR TB cases being missed, the Revised National Strategic Plan for 2013-’20 (NSP) calls for robust case￾finding strategies and rapid diagnostic technologies and comprehensive patient-centered TB care with the engagement of community health providers and the private sector. Challenge TB (CTB) supported the Government of Ethiopia (GoE) to improve access to and provision of services for TB, MDR TB, and TB HIV. CTB was aligned with the U.S. government TB strategy, the World Health Organization (WHO) End TB Strategy and Ethiopia’s Health Systems Transformation Plan (HSTP) targets for TB prevention, care, and control. CTB had three primary objectives: Objective 1. Improve service use and patient-centered treatment and care for TB, TB HIV, and drug-resistant TB (DR TB) Objective 2. Prevent transmission and disease progression Objective 3. Strengthen TB platforms for health systems strengthening (HSS) Implemented by the KNCV Tuberculosis Foundation, Management Sciences for Health (MSH) and WHO from 2014 to 2019 with a budget of ~$10 million United States Dollars (USD) a year, CTB operated at the national level and in four major regions (Amhara; Oromia; Southern Nations, Nationalities, and Peoples’ Region [SNNPR]; and Tigray), three city administrations (Addis Ababa, Dire Dawa, and Harari), and two developing regions (Benishangul Gumuz and Gambella), covering over 90 percent of the country’s population. CTB collaborated with the National Tuberculosis Program (NTP) and major national institutions to provide technical assistance and to help the regional health bureaus (RHBs), zonal health departments (ZHDs), and woreda (district) primary health care units to take ownership of TB, TB HIV, and MDR TB program management. CTB aimed to increase case notification and TB service decentralization and help strengthen Ethiopia’s health system by supporting the national laboratory network; building capacity of regional, zonal, and woreda health office program management teams and health facility staff; and improving woreda planning, drug supply management (DSM), and TB infection control (TB IC). Evaluation Purpose This performance evaluation aimed to evaluate CTB performance by examining achievements, implementation challenges, and intervention effectiveness and relevance, and provide recommendations to guide decisions on future activity design and implementation in line with the United States Agency for International Development (USAID)/Ethiopia Country Development Cooperation Strategy (CDCS). The primary users of the evaluation findings will be USAID/Ethiopia, the Federal Ministry of Health (FMoH), other GoE entities, and other donors. Evaluation Design, Methods, and Limitations The evaluation used a mixed-methods design, involving document review, primary data collection through key informant interviews (KIIs), observation, and exit interviews, and secondary data collection. Multi-stage purposive sampling was used to collect primary data. The evaluation team reviewed a range of documents on CTB design, implementation, and evidence; NTP policy, strategy, guidelines and training material, and global resources. The evaluation team conducted ix Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity 106 KIIs involving 175 individuals from USAID and partners, CTB, and government stakeholders at national, regional, zonal, woreda, and facility levels including health posts. The evaluation team conducted 44 observations in service units in facilities to assess adherence to standards of care and interviewed 12 patients on TB treatment to assess care-seeking patterns and their perception of TB services. As the primary data was collected from a purposive sample, findings may not apply to sites not visited by the evaluation team. However, the limitation was mitigated by conducting a secondary analysis of CTB project data and NTP data. The evaluation team is confident that analysis of the secondary data would fill any gap arising from the non-representative sampling done for primary data. Key Findings and Conclusions Evaluation Question 1: To what extent did CTB’s technical assistance and management approach support CTB cooperative agreement objectives? CTB’s sub-objectives and activities addressed key gaps identified in the NSP in the areas of laboratory services, treatment, infection control, drug supply management, human resource capacity, and governance. CTB’s strategic approach was to support the NTP, regions, zones, and woredas with evidence-based technical direction, improved program management capacity, and quality-assured service delivery. CTB’s design carefully considered and built on the achievements of USAID’s past investments in TB in Ethiopia, and continued implementation approaches from past mechanisms that were shown to be effective in expanding access and improving quality, such as the specimen transportation system and external quality assurance (EQA) for lab services. The emphasis of CTB design and implementation was on improving facility-based service delivery. Community-based interventions were a thematic area of CTB and although they were critical to reaching the overall objective of supporting the NTP to improve case finding, the level of effort was low in terms of the type and intensity of planned activities, even in the 100 woredas that received intensified support (as a high impact intervention) from Year 4. The geographic scope of CTB was expanded twice – at the beginning of Year 2 and towards its end. The areas added later had a relatively higher need for TB program support and poorly functioning health systems. Thus, the expansions served to extend support to areas with higher need and helped utilize resources more efficiently, but CTB lost significant implementation time in re-doing fully developed workplans. which necessitated the involvement of all staff teams, Reprogramming of budgets was also needed to prioritize a set of high impact interventions in Year 4 to improve case finding, and to support the rollout of the Xpert Mycobacterium Tuberculosis/Resistance to Rifampicin (MTB/RIF) assay as the initial diagnostic test. CTB’s technical approach included strengthening the national technical working group (TWG) to provide NTP with technical leadership in updating guidelines, roadmaps, standard operating procedures (SOPs), and tools for TB, TB HIV, and MDR TB, including key thematic areas – diagnostics and lab network; programmatic management of drug-resistant TB (PMDT) TB; childhood TB and contact investigation (CI); TB infection control (TB IC); community TB care; key populations (KPs) including prisons, people living with HIV (PLHIV), urban poor and mine workers; drug supply management (DSM); and data management and operations research (OR). CTB designed and provided international short-term technical assistance (STTA) that identified gaps related to the above thematic areas. Actions following STTA recommendations resulted in considerable improvements in diagnosis and treatment for drug-sensitive (DS) and drug-resistant (DR) TB and prevention and assisted in updating resources and designing OR. CTB built human resource capacity through cascaded, quality-assured training for government staff at national, sub-national, and facility levels covering the full range of thematic areas, and through mentoring and supportive supervision. The training events were repeated continually to compensate for attrition. x Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity CTB’s management approach was informed by its objectives and thematic areas. CTB’s central and regional teams had experts in all thematic areas who worked closely with their government counterparts and other partners to identify gaps and develop work plans that were fully aligned with NTP plans. CTB supported regular data-led program reviews to assess progress and feed into subsequent planning cycles. CTB and other partners coordinated with the NTP, and with each other through the TWG, to avoid duplication and increase efficiency. In conclusion, CTB’s design and strategic approach were informed by the gaps that needed to be addressed to reach NSP targets, and by the contours of the government health system. The technical assistance was comprehensive, relevant to needs, and contributed to improved quality of services and expanded access. CTB’s management approach was informed by its objectives and the technical approach to capacity building. Staff teams were competent to support all thematic areas. Workplans were well aligned with identified needs and NTP plans. CTB was perceived as strong collaborators, transparent, and committed to avoiding duplication. The lack of a strategic approach to geographic scope resulted in areas with higher need receiving the least duration and intensity of support. Re￾planning reduced the time available for implementation. Evaluation Question 2: To what extent did CTB implementation approaches use international standards and proven strategies? Review of resources that CTB supported in producing and the support provided to service delivery show that CTB used the WHO guidance documents and updated policies in the various thematic areas of TB detection, diagnosis, patient management, TB IC, management of LTBI, TB HIV, data management, and research as the cornerstone in all its implementation strategies. CTB provided technical assistance (TA) and financial support through the TWG to update national guidelines, training manuals, algorithms and tools to support the incorporation of these international standards into national guidelines. CTB also supported adherence to the updated national standards by disseminating updated and revised manuals, algorithms and other tools and mentoring facility staff in their use, providing supportive supervision using Standards of Care (SOC) as the main tool to assess gaps in adherence, and regular program reviews. Directly observed treatment (DOT) was observed to be practiced in all sites during intensive and continuation phases of treatment with exceptions in two sites visited, for the continuation phase. Implementation of prospective quarterly CI for contacts of index TB patients, provision of Isoniazid preventative therapy (IPT), Xpert assay as the initial diagnostic, and all aspects of PMDT were consistent with national standards, which were in turn consistent with ISTC. Adherence was variable in the areas of active case finding in facilities, CI, certain aspects of TB IC, TB diagnosis in people living with HIV (PLHIVs) and community TB care (CTBC). The reasons for variable adherence include poor coordination with the HIV program. The support to facilities (through the zonal and woreda systems) was mostly limited to supporting the TB units, lab, and HMIS. There was little intentional engagement of ART clinics, OPDs and inpatient wards, where evidence shows that case finding efforts are likely to bear fruit. In conclusion, a key outcome of CTB interventions has been the updating of national guidelines and algorithms to align with international standards. Current national guidelines are consistent with global standards for TB detection, diagnosis and patient management including for DR TB, TB IC, LTBI, TB HIV, and M&E. CTB uses high-quality, globally recognized parameters for quality and has rolled out a mechanism to assess and assure quality through the SOC tool. Evaluation Question 3: Was the information generated by CTB used to help achieve objectives and outcomes and, if so, how? Key sources of data for CTB were: i) routine patient-level data captured through the national health management information system (HMIS) by program staff, ii) a comprehensive set of output and process xi Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity data that CTB staff routinely compiled using the SOC tool and iii) nonroutine data from assessments and OR. CTB did not set up a parallel data collection mechanism for patient-level data. Instead, it supported facilities and woredas to improve the quality of data that flowed into the HMIS. The SOC tool included a range of standard outcome indicators from WHO as well as those on critical outputs and processes. CTB built capacity in the health system to use the SOC to assess the performance of each facility, identify gaps, training and mentorship needs, and also to monitor gaps in supplies and equipment. Performance thresholds built into the tool helped identify low-performing facilities. CTB designed and conducted ORs and used the evidence for program decisions. The evaluation team noted several missed opportunities for optimal use of data for decision making. The rollout of Xpert as the initial diagnostic was a massive undertaking but was carried out before the pilot could be completed and evaluated. There is no evidence of critical review and reflection on output-level data from high impact interventions aimed at finding missed cases in order to re-evaluate assumptions and take corrective action. A midterm evaluation could have benefited CTB, given the breadth of its scope of interventions. Care-seeking patterns and barriers were not studied in depth. Sex-disaggregated data was not available for output-level data for potential insights into gender-related barriers to care￾seeking. In conclusion, it is evident that CTB generated and used some of the evidence it collected to re-focus its interventions. CTB also contributed to improved data quality and use in the management of the program at all levels. There is evidence that the available data was not put to optimal use and potential opportunities for evidence-generation were missed. Evaluation Question 4: What were CTB’s main achievements and challenges? Achievements and challenges in supporting TB program management capacity CTB built on earlier efforts to advocate for the creation of government-paid staff positions for TB thematic areas in regional, zonal and woreda health teams. Through training and on-site mentorship, CTB has successfully built capacity in zonal and woreda TB staff to collect, interpret and use data in the SOC tool to take remedial action. CTB expanded the number of labs that serve as centers for external quality assurance (EQA) and networked peripheral labs these EQA centers. CTB also worked with regional health teams to define the catchment areas of hospitals and linked the health centers with hospitals based on the catchment. CTB facilitated similar networking of treatment follow-up centers (TFCs) and treatment initiation centers (TICs) for managing DR TB patients. CTB strengthened DSM at all levels in quantification and rolled outpatient kits for first-line drugs by training, mentoring staff at the national and regional hubs of the pharmaceutical supply agency (PSA), DSM staff at regional and zonal health offices and pharmacy staff of facilities. The evaluation team noted shortages and excesses in two hubs, which indicate problems with quantification and supply. CTB continued support to the TB research advisory council (TRAC) to train researchers in regions in OR and supported research and dissemination activities. CTB supported RHBs to prioritize key populations (KPs) and also supported screening and DOT in mining areas, prisons, urban settlements, and shelter homes. CTB supported the costs for most of these activities. Zonal and woreda TB program staff approach CTB staff for problem￾solving rather than to their supervisors in the health system, as they find this to be more efficient since CTB is technically, financially, and logistically competent to resolve issues. This indicates ongoing dependence on CTB staff for routine functioning. CTB staff promptly respond to issues by initiating strategic or ad hoc measures or by escalating them appropriately. Regional and zonal program staff report that the demarcation of roles and responsibilities between them and CTB teams became blurred during day-to-day implementation, such that CTB teams actually did the work, rather than support program staff to do the work. CTB did not have a clear exit plan developed during design/re-design to identify and agree on specific actions for all stakeholders to carry out in order to transfer specific management functions to government counterparts. xii Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity In conclusion, CTB successfully continued and scaled up the efforts by earlier mechanisms to build capacity at zone and woreda levels for data-led program management – through training, mentorship, facilitating the networking of hospitals with health centers, peripheral labs with EQA centers and TFCs with TICs as well as supporting RHBs in revitalizing expansion of access to services to KPs. CTB support has resulted in improved DSM for PMDT drugs and supplies, but quantification issues persist. CTB’s role in problem-solving led to it directly managing these issues and this threatens the sustainability of the gains made. Achievements and challenges in improving diagnostic services CTB supported the expansion of sputum smear microscopy services and its external quality assurance mechanism. CTB provided TA and payments for expansion of culture drug sensitivity testing (DST) services, especially line probe assay for second-line line probe assay (SL-LPA) drugs including certifications. TA and financial support from CTB contributed to the progress of the national reference lab, housed in the Ethiopian Public Health Institute (EPHI) towards becoming a supranational reference lab. The evaluation team noted that data on the coverage of DR TB patients with baseline SL-LPA has not been captured and compiled by the lab system for all patients: 59 percent (432/731) of DR TB patients notified from CTB areas in 2018 have information on SL-LPA testing, and 70 percent of them (296/432) have had the SL-LPA test CTB supported all aspects of the expansion of the Xpert assay service and continues most of the direct financial and technical support for maintenance. However, power supply interruptions hamper the service. Utilization of GeneXpert has steadily increased, supported by extensive training, maintenance support, and mentoring provided by CTB, but there is insufficient data to verify if access to universal DST has been achieved. Case notification has not shown an increasing trend, which was another purpose of Xpert expansion. CTB also continued support to specimen transportation, thereby further expanding access to Xpert and culture DST to remote facilities. In conclusion, CTB has successfully supported all levels of the national laboratory system to improve its capacity to provide high-quality diagnostic services for TB, including DST, and the rollout of Xpert assay. CTB continues financial and technical support to most aspects of Xpert services; and available data has not been utilized sufficiently to assess if improving access to a better diagnostic will improve case finding, which is a key assumption underpinning Xpert expansion. Achievements and challenges in improving case finding Screening of all patients in outpatient departments (OPD) for TB was found to be consistently practiced in health centers but nearly non-existent in most hospitals visited. In hospitals that did conduct screening, the results were recorded in OPD registers by facility staff who did not necessarily observe OPD consultations. Furthermore, where screening did take place, its quality was poor in some locations and acceptable in others. CTB rolled out the tools and training for CI for index cases. Documentation of prospective screening in some facilities was incomplete and yielded lower than expected numbers of presumptives across all CTB areas. Training of health extension workers (HEWs), their prioritization of community case finding, assessment of coverage of households and the yield of presumptives were all low in the woredas that received intensified support from CTB. Involvement of civil society organizations (CSOs) has been minimal. CTB-supported mass screening among workers in mines, urban settlements, shelter homes, and health workers has given a high yield, however, the participation of HIV staff in facilities in CTB training and joint supervision has been low. Patients interviewed by the evaluation team reported delayed care-seeking, considerable out-of-pocket expenses, initial care-seeking at private facilities, and low index of suspicion of TB in private facilities. xiii Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity Case notification for drug-sensitive TB (DS TB) remained between 60 and 67 percent of all estimated cases during CTB implementation., in the context of a continuing decline in the WHO-estimated burden of TB in the country. Based on current estimates of DR TB cases, only a quarter of patients were notified and enrolled in treatment every year during CTB. These estimates will be revised once results from the recently concluded drug resistance survey become available. Despite the successful support to change the algorithm to test all those with TB symptoms with Xpert, targets of CTB for Xpert testing for new and re-treatment patients has not been achieved., in large part because of the insufficient funding for the TB program as a whole, which resulted in funding shortfalls for Xpert cartridges. In conclusion, the only activity among those prioritized for improving case finding – mass screening among KPs – has given a high yield. CI has improved considerably over the years of CTB implementation, although the coverage of U5 is less than expected numbers (based on population estimates). The others - screening of outpatients in hospitals, and screening of PLHIVs – has not been implemented well. Notification of DS TB and DR TB cases continue to remain at baseline levels. Achievements and challenges in supporting improving case management CTB supported improvements in patient-centered care, by strengthening referral networks, mentorship of facility staff and monitoring using SOC. Adherence to directly observed treatment (DOT) was widespread. Patients interviewed by the evaluation team had a high awareness of cough hygiene and adherence, which is a critical aspect of patient management. Treatment success rates have been consistently at or above 90 percent during CTB years. Cure rates for DS TB have also steadily increased. CTB supported training and mentorship for all areas of PMDT and patient management. CTB also provided equipment, several supplies needed for monitoring DR TB patients on treatment and patient support costs. The treatment success rate for DR TB patients has been consistently at 70 percent through CTB years, which is higher than the global average of 55 percent, and there has been a steady rise in cure rates. Some TICs visited lack equipment needed for ECG and audiometry and staff, and some TFCs require training and support. Almost all TB patients are tested for HIV, with close mentorship provided by CTB. However, CTB’s support for HIV clinics has been low due to poor coordination with the national HIV program at all levels, and hence the coverage of TB screening of PLHIVs is low. In conclusion, CTB’s mentorship and follow up using SOC and hospital initiative has contributed to improved treatment adherence and completion among DS TB and DR TB patients. Achievements and challenges in TB prevention activities CTB supported rollout and mentorship for TB IC measures. The evaluation team found infection prevention (IP) committees, but sub-optimal coverage of screening of health workers. Ventilation of patient waiting and consultation areas and the identification and fast-tracking of coughers among outpatients were implemented well in low volume facilities and poorly in high volume facilities. CTB’s engagement with facility management has been low. CTB supported the rollout of tools and mentoring for CI of index cases and latent TB infection (LTBI). Their coverages have improved considerably. However, the proportion of U5 children among contacts screened has been less than expected, based on population estimates. IPT coverage for eligible PLHIVs has also increased over CTB years. In conclusion, CTB’s engagement of facility management and implementation TB IC measures remain sub-optimal in high volume facilities. xiv Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity Evaluation Question 5: To what extent did CTB’s methodologies, interventions, and management set the stage for sustainability and ownership of project outcomes? The evaluation team found evidence for increased political commitment to health, and TB control in the form of prompt adoption of international standards and updating of national guidelines and tools, creation of staff positions for TB thematic areas in the government, and increasing utilization of integrated management systems for diagnostics and supplies. Domestic funding for TB remains low, and hence the NTP is heavily dependent on external funding. CTB supported a range of strategic, one-time investments that are likely to have a lasting impact on the program. These are related to TWGs, guidelines and tools, equipment and infrastructure. Technical and financial support for EQA, specimen transportation in Oromia and Amhara and procuring of new drugs for DR TB are likely to be taken over by NTP or other partners. CTB established competencies for program management including data-driven mentorship and to some extent, capacity for evidence generation, however, most maintenance related to Xpert and supplies related to culture DST, and most aspects of PMDT and DR TB patient support continue to be paid for by CTB. There appears to be no plan for the government to take over CTB’s financial support for these routine activities that are critical for program management, diagnosis, care, and prevention including the critical area of cascading supervision. Competencies have not been established in other areas such as maintenance of equipment for culture DST and Xpert assay, and they are still primarily managed by CTB staff. Data-driven program management and mentorship have also not been established in developing regions. Government staff at zones and woredas are dependent on CTB teams for problem-solving and assume the continuation of donor support. Political commitment to End TB targets is high, but nearly half of the NSP is not funded. In conclusion, low levels of domestic funding resulted in CTB continuing routine payments and other support, and hampered efforts to implement CTB’s exit plan, such that there is continued dependence on donor support. Key Learning and Implementation Adaptation • Alignment with government priorities: CTBs’’ design and strategic approach were built around meeting the gaps and needs identified in the NSP. CTB has gathered a reputation for identifying gaps in program implementation and service delivery and adapting its plans to meet the need. Examples include the continual gap-filling training carried out to ensure that new staff is trained and procuring supplies related to Xpert testing to continue support to the NTP in rolling out the changed algorithm. • Reprogramming to expand coverage: CTB demonstrated flexibility in expanding its coverage to regions that were not in its initial plan. The revised geographic coverage included regions that had previously received partner support as well as regions with no prior support. The evaluation team found an improved quality of services and expanded access in both sets of locations, but that the developing region visited needed more intense and continued support in the form of mentoring, supportive supervision and guidance for data-led program management. A bolder and strategic approach to deciding the geographic scope at the start of the Activity could have helped avert repeated reworking of budgets and given more implementation time for developing regions. • The role of USAID: The Mission has provided advice, direction, review, and support to the CTB team in ensuring that lessons from HEAL TB implementation are appropriated, developing regions are included, and priority is given to finding missed cases through the high-impact xv Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity interventions. As USAID has been the sole in-country bilateral supporter for the NTP for over two decades, its involvement in CTB has ensured that institutional learning is preserved. • Best practices and innovation: CTB supported a range of innovations and local solutions to overcome specific operational issues. These include: solar lamps for microscopes in locations with poor power supply; ISTAT as an efficient diagnostic mechanism for DR TB patients on treatment; using the messaging platform Viber (in facilities in Addis Ababa) and an mHealth platform (in St Peter’s hospital) to follow up DR TB patients’ adherence to treatment; using the Xpert assay on stool samples to diagnose childhood TB; the TB-HIV one-stop shop, to provide comprehensive in one place; the blended learning center that provides virtual training sessions, including those from experts from abroad, streamed to woredas through the WoredaNet system. These innovations need to be assessed for feasibility before they are scaled up. • Limitations learned from the evaluation: There is no mechanism established, using existing or new data capture systems, to verify the assumptions that underlie the expectation that expanding Xpert as the primary diagnostic would lead to improved case finding, increase bacteriological confirmation of cases and provide universal DST. There has been very little effort to gain an in-depth understanding of social and gender-related barriers to care seeking. This is significant because CTB aimed to support NSP targets by finding missed cases. CTB’s support to improve services in facilities have been limited to the TB clinic, lab, the pharmacy, and the HMIS unit. The OPD, HIV clinic, inpatient wards and biomedical engineers (in large hospitals) were not optimally engaged in appropriate thematic areas. Creative ways must be developed to engage all relevant service units in facilities. Recommendations The evaluation recommends the following priority investments, design and implementation considerations for future similar programming: 1. Active case finding in high volume facilities: Facilitate zones and woredas to engage facility staff at all outpatient and inpatient units more frequently and around specific actions that support routine active case finding and document findings that are valid and verifiable. Institute appropriate support mechanisms, especially, direct observations of the screening activity by supervisors. Review the yield of presumptives and confirmed cases after a time of quality￾assured implementation against the level of effort and redesign the implementation strategy. 2. Maintaining and expanding PMDT: This is critically dependent on external support, in order to maintain high-quality service delivery and the steady enrollment of patients into care. Continue payments related to routine program management (such as TFC-TIC reviews), routine supplies (such as ancillary drugs), patient support costs, clinical reviews and top-up training of staff teams, until such time that the government at national and regional levels is able to provide for them. Assess all TICs for barriers to optimal functioning and build the needed capacity, including infrastructure, equipment, supplies, and training. 3. Active case finding in communities: There is a need for intentional, evidence-based and time-bound engagement with careful review and management of results. Consider one of these two options for such intentional engagement: a) Intensified support for routine TB screening in the 100 woredas already identified by CTB. Train all HEWs in basics of TB and TB screening protocol, develop a detailed supervision plan for woreda TB officers and the health extension program supervisors to ensure 85 percent of household coverage is maintained; b)Conducting month-long nationwide campaigns, once or twice a year, aiming for a heightened focus on TB at all levels in the system. Work with regional, zonal and woreda TB teams to develop supervision plans to ensure high coverage of households and high quality of screening. Whichever approach is selected, the NTP and the future mechanism should build in the collection, compilation, and analysis of data on household coverage and yield; build in a mechanism for detailed xvi Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity documentation of challenges, responsive actions, and lessons learned at the health post, woreda, and zonal levels; conduct a detailed review of all data each quarter, and decide on scaling up or abandoning the effort based on the yield, cost-benefit, and operational challenges. In addition, commission well-designed studies such as cluster-randomized designs that will help in better understanding of the potential yield from various approaches, and the feasibility (including cost￾benefit) of each, and qualitative studies to gain insight into people’s preferences, and barriers to seeking care, and social or gender-based inequities in access. 4. Contact investigation: Support zones and woredas to follow up the coverage of household contacts, especially U5 contacts, in the frequency provided in the national guideline. Support the assessment of yield and its cost-effectiveness and titrate support accordingly. 5. Routine program management: There is a need for the future funding mechanism to maintain the competencies already built in program and facility teams to generate and use high￾quality service data, as new staff come in and to keep up with new evidence and guidance. This should be done through technical and financial support for training and supportive supervision and is required until such time when the NTP can take over these critical functions. 6. Continuing universal access to Xpert assay: Xpert has made universal drug susceptibility testing a possibility and its operational challenges are part of this new reality. The future funding mechanism should provide resources for training, mentoring, supplies and external technical support, including but not limited to ensuring GxAlert functioning. 7. Expanding to developing regions: Titrate the level of support for human resources and institutional capacity with the available staffing and infrastructure, but with a clearly planned and executed sustainability strategy. Such support is especially needed to test and adapt strategies for case finding and patient-centered care in these regions. Include the design considerations listed below. 8. Program design: Clearly identify, articulate and periodically review evidence for program assumptions; build in a timed exit plan, and a communication strategy to all stakeholders about the nature and duration of support; develop a clear plan for the transfer of responsibilities to the government at all levels, especially related to Xpert services. 9. Performance measurement, evidence generation, use, and accountability: Develop key performance measures that reflect the core functions of the support mechanism, rather than outcomes expected from NTP; revamp data collection, compilation and reporting of adequately disaggregated data for KP activities, CI; require implementing partners to build in time for critical reflection on evidence. Examples are measures of institutional capacity such as competencies of program staff to effectively use tools and guidelines, maintain equipment and independently manage supply system. 10. Include the following technical interventions/support: Improving collaboration with the national HIV program at all levels; facility-wide support rather than sole focus on TB units; expand the use of solar lamps, All Africa Leprosy Regional Training (ALERT) blended learning center, collaboration with mining companies for screening, mHealth support for DR TB patients. 11. The NTP has a strong team of staff to support the various thematic areas, with technical advice from the TWG. Key findings from the evaluation include continued ownership of the Xpert system by CTB and varying levels of adherence to national standards for service delivery. a. Political commitment to End TB targets needs to translate into domestic efforts to close the current funding gap. Even a modest increase in domestic funding will go a long way to complement donor funding and technical support and take over critical functions as outlined in the earlier recommendations and prevent erosion of the gains made. The evaluation recommends that the NTP work with technical partners to develop economic and cost-effectiveness analyses to convince policymakers of the relative benefits of investing in TB control and translating the commitment to End TB targets into increased funding. xvii Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity b. The evaluation recommends that the NTP work proactively with the future funding mechanism in order to assume ownership of the functioning of GeneXpert machines and their optimal utilization. c. The evaluation also recommends that the NTP lead a multi-stakeholder team in assessing adherence to all national standards at service delivery levels. 1 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity EVALUATION BACKGROUND Activity Background Context Challenge TB (CTB) is the flagship global mechanism for implementing the strategy of the United States Agency for International Development (USAID) to meet the global post-2015 goal of a world free of tuberculosis (TB). The project supports the introduction, scale-up, and sustainability of high impact TB interventions, primarily in 26 high-burden TB, multidrug-resistant (MDR) TB, and TB human immunodeficiency virus (HIV) countries, including Ethiopia. According to the 2018 World Health Organization (WHO) Global TB Report, approximately 10.0 million people developed TB in 2017: 5.8 million men, 3.2 million women, and 1.0 million children. In 2017, TB caused an estimated 1.3 million deaths among HIV-negative people and 0.3 million deaths from TB among people living with HIV (PLHIV). Ethiopia is the second most populous nation in Africa, with a projected population of 105 million1 in 2018, and continues to be among the world’s 30 high-burden TB, TB HIV, and MDR TB countries. Figure 1 below gives the trends in TB incidence and TB mortality per 100,000 population over the period of CTB implementation, along with the uncertainty ranges around the estimates. The incidence and mortality of TB are estimated to be 164 and 24 per 100,000, respectively. With case detection and notification at 65.5 percent out of estimated incidence, a third of TB cases are missed. Case detection has been declining by three to eight percent every year2 for the past five years. The annual incidence of MDR and Rifampicin-resistant (RR) TB is estimated to be 5.2 per 100,000, of which only 25 percent are diagnosed and enrolled in treatment. The global report also estimated the TB HIV co-infection rate at seven percent. Figure 1: Trends in TB incidence and TB mortality during CTB period 1 UN Estimates 2 National TBL update, March 2019, email communication 200 192 177 164 32 25 25 24 0 50 100 150 200 250 300 2014 2015 2016 2017 # per 100,000 population Incidence rate Mortality rate 2 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity Source: WHO Global TB reports Both men and women are vulnerable to TB in their young and productive ages with slightly more men and women among estimated cases.3 Gender disparities in biological susceptibility and health-seeking behavior, such as gender-related delay in TB diagnosis, treatment interruption, and stigma and discrimination remain poorly understood. Ethiopia achieved the millennium development goals set for TB in 2015. In alignment with the Global End TB Strategy of WHO, the National Tuberculosis Program (NTP) aims to end the TB epidemic by reducing TB-related deaths by 95 percent, cutting TB incidence by 90 percent, and ensuring that no family is burdened with catastrophic expenses due to TB. The Revised National Strategic Plan for 2013- ’20 (NSP)4 calls for robust case-finding strategies and rapid diagnostic technologies to address the threat of missed cases and of drug-resistant, comprehensive, patient-centered TB care with the engagement of community health providers and other actors, including the private sector. The estimated national TB financing was 93 million United States Dollars (USD) in 2017, of which 11 percent was from domestic sources, 33 percent international, while 56 percent of the NSP was unfunded. The national TB and leprosy program (NTP) under Federal Ministry of Health (FMoH) has been expanding TB services through both public and private health facilities to improve case finding and provide care, with support from USAID and other donors and stakeholders. Specifically, the Private Health Sector Support Program supports the NTP in improving case detection and management by the private sector. Activity Overview CTB Ethiopia is a cooperative agreement with an annual budget of US $10million started operations in October 2014 and is ending in September 2019. It is implemented by the Koninklijke Nederlandse Centrale Vereniging Tuberculosis Foundation (KNCV) as the in-country lead, along with Management Sciences for Health (MSH) and the World Health Organization (WHO). CTB collaborates with NTP, major national institutions (Ethiopian Public Health Institute [EPHI], Armauer Hanson Research Institute [AHRI], Pharmaceutical Supply Agency [PSA]), and other local institutions and helps the regional health bureaus (RHBs), zonal health departments (ZHDs), 3 WHO Global TB Report, 2018 4 Federal Democratic Republic of Ethiopia, Federal Ministry of Health. National Strategic Plan, Tuberculosis and Leprosy Control 2013-’20, with Updates for 2018-’20. November 2017 Figure 2: CTB operational areas 3 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity and woreda (district) primary health care units to take ownership of TB, TB/HIV, and MDR TB program management. Through this assistance, the RHBs, ZHDs, and woreda primary health care units have been fully engaged to ensure improved and sustainable program management capacity. CTB focuses on increasing case notification and finding missed cases, improving the quality of TB services and the collection and use of data for management. CTB is also helping to strengthen Ethiopia’s health system by supporting the national laboratory network; building capacity of regional, zonal, and woreda health office program management teams and health facility staff; and improving woreda planning, drug supply management, and TB infection prevention standards. CTB covers more than 92 percent of Ethiopia’s population and operates at the national level and in four major regions (Amhara, Oromia, Southern Nations, Nationalities, and Peoples’ Region [SNNPR], and Tigray), three city administrations (Addis Ababa, Dire Dawa, and Harari), and two developing regions (Benishangul Gumuz and Gambella), shown in Figure 2. CTB aligns with the U.S. government TB strategy, the WHO End TB Strategy and Ethiopia’s Health Systems Transformation Plan (HSTP) targets for TB prevention, care, and control. The CTB approach has three objectives and eleven sub-objectives as shown in Figure 3. Figure 3: CTB objectives and sub-objectives The evaluation team did not find a narrative statement of a theory of change in CTB documents and reports. The team put together the following statement based on the descriptions found in CTB documents: A third of the estimated incident TB cases are missed; finding and treating them is critical for achieving the national strategic goals for TB. If high-quality, patient-centered care is provided and the use of such services is improved, alongside efforts to reduce transmission and disease progression and strengthen political leadership and commitment to these goals by strengthening the supply chain, human resource capacity, and the generation and use of evidence, they will help close the gaps in capacity to find and treat missed cases. These will result in sustainable contributions to reaching national strategic goals and post-2015 global goals for eradicating TB. Objective 1. Improve service utilization and patient-centered treatment and care for TB, TB HIV, and DR TB Sub-objective 1: Enabling environment (KNCV) Sub-objective 2: Comprehensive, high-quality diagnosis (MSH) – EQA, GeneXpert Sub-objective 3: Patient-centered care and treatment (KNCV with assistance from WHO) Objective 2. Prevent transmission and disease progression Sub-objective 4: Targeted screening for active TB (KNCV) Sub-objective 5: TB IC infection control (KNCV) Sub-objective 6 Management of latent TB infection (KNCV) Objective 3. Strengthen TB platforms or Health Systems Strengthening (HSS) Sub-objective 7: Political commitment and leadership (WHO) Sub-objective 8: Comprehensive partnerships and informed community involvement (WHO) Sub-objective 9: Drug and commodity management systems (MSH) Sub-objective 10: Quality data, surveillance and M&E and OR (KNCV) Sub-objective 11: Human resource development (KNCV with assistance from WHO, MSH) Sub-objective 12: Technical supervision (HQ level, from all three partners, esp. STTA, in areas such as PMDT) 4 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity Evaluation Purpose The purpose of this evaluation was to evaluate the performance of the CTB Activity, implemented by the KNCV, MSH, and WHO. The evaluation was meant to examine CTB achievements, implementation challenges, and intervention effectiveness and relevance and provide recommendations to guide decisions on future activity design and implementation, in line with the USAID/Ethiopia Country Development Cooperation Strategy (CDCS). The primary users of the evaluation findings will be USAID/Ethiopia, the Federal Ministry of Health (FMoH), other GoE entities, and other donors. Evaluation Questions This evaluation addressed the following five evaluation questions, drafted by USAID/Ethiopia in the initial Statement of Work (SOW) and finalized by the Ethiopia Performance Monitoring and Evaluation Service (EPMES) in collaboration with USAID/Ethiopia (For details of each evaluation question, please refer to the evaluation SOW in Annex I: Evaluation Statement of Work). 1. To what extent did CTB’s technical assistance and management approach support CTB cooperative agreement objectives? 2. To what extent did CTB implementation approaches use international standards and proven strategies? 3. Was the information generated by CTB used to help achieve objectives and outcomes and, if so, how? 4. What were CTB’s main achievements and challenges? 5. To what extent did CTB’s methodologies, interventions, and management set the stage for sustainability and ownership of project outcomes? Evaluation Design, Methods, and Limitations The evaluation, which ran through January 2019 – July 2019, used a mixed-methods design that included document review and primary data collection through key informant interviews (KIIs), observations, and exit interviews with TB patients. KIIs collected in-depth information from experts most informed about CTB programming. An evaluation design matrix linking each of the approaches with the evaluation questions is included as Annex III: Evaluation Design Matrix. Document review A range of documents was reviewed to establish evidence on activity design, implementation, progress toward goals, and lessons learned, and to help understand the CTB programming context. Below is a list of categories of documents that were reviewed. • Documents related to CTB design: CTB request for application 2013, CTB Cooperative Agreement with USAID and technical proposal 2014 and CTB Monitoring, Evaluation and Reporting Plan • Baseline assessments in SNNPR and Tigray, HEAL-TB midterm evaluation report, 2014 • Documents related to CTB implementation, including but not limited to annual work plans and reports, short-term technical assistance (STTA) reports, manuals, guidelines, standard operating procedures (SOPs) and tools developed for NTP with technical assistance from CTB, published operations research (OR) reports, Job descriptions and CVs of select staff • Global strategies, guidelines, and reports: WHO End TB strategy, International Standards for TB Care, USAID Global TB strategy, Global TB report 2018, 5 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity • National policy, strategies, and reports: Revised National Strategic Plan, 2013-’20 with an update for 2018-’20; National guidelines for TB, DR TB, and leprosy in Ethiopia, 2017; Health and Health Indicators, 2017-’18; presentations made by NTP on annual progress, TB prevalence survey report • Review of literature focusing on a range of thematic areas including active case finding and community TB care. Primary data collection methods Primary data were collected through (a) KIIs with CTB implementers, key GoE counterparts at various levels in the health system, USAID/Ethiopia, and other stakeholders involved with CTB implementation; (b) observations of TB care and control activities in select health facilities; and (c) exit interviews with patients using TB services in health facilities. The team divided into two sub-teams, each with a TB technical specialist and evaluation specialist. Key informant interviews KIIs addressed the following topics: • CTB: Objectives, technical assistance design, implementation approaches and strategies, management and staffing, key achievements, challenges and adaptations applied to address them, alignment with international standards, coordination, sustainability and scale-up of innovations, and perceptions of future initiatives • GoE: Perceptions of key CTB achievements and challenges, alignment with and adoption of international standards, service delivery quality, coordination and collaboration, sustainability and scale up • USAID: CTB design considerations, key CTB management achievements and challenges, sustainability, and scale-up of innovations • Stakeholders: Coordination, collaboration, and perceptions of CTB • Facilities: Service delivery quality, adherence to standards, capacity-building achievements and challenges and how challenges were addressed A total of 108 KIIs (63 at facilities and 45 of CTB and partners) were conducted, involving 175 participants. Table 1 gives a list of stakeholders interviewed, and Annex V: Information Sources gives details of the KII respondents. Table 1: Categories of key informants Key Informant Categories Stakeholders • USAID/Ethiopia • Other stakeholders German Leprosy and TB Relief Association (GLRA) Global Health Committee (GHC) Partners in Health (PIH) Global Fund team at FMoH • Civil society organization (CSO) representatives o Voluntary Health Services, Organic health • Professional associations Ethiopian Thoracic Society Ethiopian Pediatric Society (EPS) • NGOs o REACH TB CTB Staff o o o o o o 6 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity • Group discussion with key informants: country director, technical director, monitoring and evaluation (M&E) director, regional directors • Leads of sub-partners – MSH and WHO • Regional and zonal coordinators • Operations director FMoH • NTP manager • Health Management Information System (HMIS) Unit Focal Person • National Reference Lab - EPHI • Pharmaceutical Supply Agency (PSA) • TB Research and Advisory Committee (TRAC) • Specialized Hospitals o St Peter’s Hospital o All Africa Leprosy Regional Training Center and Hospital (ALERT) Regional Health Bureau • TB coordination unit • Regional laboratory • PSA regional hubs Zonal/Sub city Health Department • TB focal person Woreda/Town Health Office • TB focal person Health facility level • Head of the facility • Outpatient department clinician • Laboratory in charge • Clinician in TB clinic (including MDR TB treatment initiation centers in specialized hospitals) • Clinician in HIV clinic Health Post/Community level • Health extension workers Observations The evaluation team observed facility-level TB service delivery in the following units of specialized hospitals, general/district hospitals, and health centers to assess adherence to national and global standards for TB care and control: • Outpatient department: adherence to standards for TB infection control (TB IC), screening for TB, diagnosis, referral, documentation, and patient education • TB clinic: adherence to standards for TB IC, diagnosis, referral, treatment (including drug regimens and adherence to treatment), contact tracing and screening, HIV testing for TB patients, and recording and reporting • MDR TB treatment initiation and in-patient care: adherence to standards for MDR TB management, contact screening, and TB IC, recording and reporting, and confirming MDR TB service expansion • HIV clinic: adherence to standards for TB IC, screening HIV patients for TB, including patient education, and recording and reporting • Laboratory: adherence to standards for TB IC, including handling of specimens, adherence to standards for equipment functionality, handling, and maintenance, confirming the expansion of 7 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity new diagnostics, external quality assurance (EQA) and sample referral services and recording and reporting The evaluation team used field-tested observation checklists that included questions related to the above elements, including workplaces observations, patient-provider interactions, provider activities in handling drugs, records, equipment, and specimens, and reviewing of records and reports. Observations were completed across facilities including hospitals, health centers, and specialized hospitals in Addis Ababa and also in EPHI and regional labs. The locations of facilities in which observations were carried out are listed in Table 2. Exit interviews In order to get patients’ opinion with regard services, their knowledge about TB, quality of service delivered (including advice on adhering to their treatment, side effects, and follow-up dates) and their perspectives on improving access to TB services exit interviews were conducted with TB patients. These interviews were held in the TB clinics of the sampled health facilities after the patients obtained the day’s medication under the directly observed treatment (DOT) strategy. Annex V: Information Sources gives demographic information of these patients. Sampling for primary data collection The evaluation team applied a multistage purposive sampling approach to select regions and city administrations to carry out KIIs, observations, and exit interviews. In the first stage, regions were selected, and in the second, zones were selected within those regions based on the duration of support by CTB, overlap with HEAL TB, developing region, logistic convenience and security issues. These parameters ensured that the sites represent not only CTB performance but also the cumulative effect of long-term investment by USAID/Ethiopia in TB control, along with the feasibility of visiting the sites. In addition to these sites, three hospitals were added as they were locations where CTB had implemented its best practices and innovations. Data were collected between February 14 and March 5, 2019. A total of 106 KIIs (61 at facilities and 45 of CTB and partners), 44 observations at 11 facilities (five hospitals, four health centers, two specialized hospitals in Addis Ababa) and 12 exit interviews were conducted (see Annex II: Detailed Description of Evaluation Design and Methods for list of sites and samples and Annex IV: Data Collection Tools for data collection tools and sampling methodology used.) Table 2: List of sites and samples Level KIIs KII Facilities Observations Exit Interviews Regional Oromia West Arsi zone Arsi Negele district 5 6 5 2 Amhara Awi zone Dangila Zuria district 5 6 5 2 SNNPR Sidama zone Wondo Genete district 5 6 5 2 Yirgalem hospital - 3 1 - Jinka hospital - 6 4 - Tigray East zone Adigrat district 5 6 5 2 Mekele hospital - 3 1 - Addis Ababa 4 6 4 2 8 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity Level KIIs KII Facilities Observations Exit Interviews Addis Ababa sub city Woreda 7 Gambella 3 5 5 2 Subtotals - Regional 27 47 35 12 National GoE + National Reference Lab 6 1 1 - Stakeholders 3 - - - USAID/Ethiopia 1 - - - Professional Associations 2 - - - ALERT, St Peter hospitals - 13 8 CSOs 2 - - - CTB Staff 4 - - - Subtotals - National 18 14 9 0 Totals 45 61 44 12 Secondary data analysis The evaluation team analyzed secondary quantitative data from sources including the CTB M&E database, data on key performance indicators, CTB summary data (on process and outputs) obtained from its various interventions, and national HMIS data. The primary purpose of the secondary data analysis of service statistics was to establish the extent of service delivery and utilization and CTB performance against targets and point to areas for further in-depth inquiry. The assessment also aimed to analyze trends in performance of key indicators from before CTB and over CTB implementation periods in case notification (trends, yield from various active case finding interventions), treatment outcome by type of outcome, disaggregation by age, sex, and other social parameters, and data related to expansion of services such as new diagnostics, MDR TB diagnosis, and treatment. This analysis did not directly associate the observed results in service utilization with CTB work, in the absence of a comparison group. However, it is logical to assume that CTB contributed to the observed results through its mobilization, capacity building, logistical support, and other key interventions. Data preparation and analysis The evaluation team transcribed recordings of KIIs and organized their notes around the evaluation questions and themes identified in the design matrix. Thematic analysis noted emerging sub-themes, categories and variations between them. Qualitative data from exit interviews and observations were added to this analysis. Primary quantitative data from exit interviews and observations were analyzed using Excel. Secondary data were analyzed for trends, variance, and yield of cases. Data disaggregated by sex, geography, and facility level was obtained and analyzed, where available. Ethical considerations All interview protocols and informed consent forms associated with this performance evaluation were submitted to Social Impact Inc.’s in-house Institutional Review Board and were approved via expedited review. The evaluation team obtained written informed consent from the KII participants and from service providers and verbal consent from patients included in observations and exit interviews. Gender and social analysis plan The evaluation reviewed sex-disaggregated data wherever it was available, and gender-related barriers to accessing care and adhering to treatment were explored through the various data collection methods. Data collection tools and the analysis plan included an assessment of CTB results on key 9 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity populations already identified by the NTP and CTB: prison inmates, mining workers, urban poor, and PLHIV. The sample of exit interviews included equal numbers of male and female patients. Limitations Primary data was collected from a purposive sample of respondents and geographic locations. The findings, therefore, may not reflect CTB performance in CTB sites not visited, and therefore may not be representative of CTB’s work in its entire target population. However, findings were supported by a secondary analysis of CTB project data and NTP data. The sample sizes of patient exit interviews and observations were limited due to time constraints. Participants provided information based on their informed consent. The team ensured adequate interviewing practices and correct phrasing and rephrasing of questions, but biases could have resulted from a lack of information or poor understanding of the questions. Sex-disaggregated data was not available at the level of process and outputs, where it would have enabled meaningful analyses. Limitations anticipated at inception regarding the inability to visit certain sites due to security concerns, and lack of availability of respondents did not inhibit data collection or analysis. 10 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity FINDINGS AND CONCLUSIONS This section describes findings from the evaluation and conclusions by evaluation question. It also includes an assessment of potential case finding strategies. Preceding these detailed descriptions are Table 3 that gives a snapshot of overall achievements and key limitations by objective, and Table 4 that gives the achievement of the four key performance indicators of CTB. Table 3: Achievement by Activity objective Objective Key Accomplishments Limitations 1. Improved access to high￾quality patient￾centered TB, MDR TB, and TB HIV services • National standards aligned with global recommendations • Adherence to national standards at service delivery improved, assured • Access to high-quality diagnostics considerably improved • Patient-centered care for drug-sensitive TB (DS TB) and DR TB considerably improved • Cure rates for DS TB and DR TB increasing • Case finding stagnant for DS TB and DR TB, and assumptions unverified • TB HIV collaboration sub￾optimal 2. Prevent TB transmission and disease progression • Contact investigation (CI) established and coverage improving • TB infection control (TB IC) measures in facilities strengthened • Coverage for latent TB infection (LTBI) for children and PLHIVs improving • Low implementation of key TB IC measures 3. Strengthen TB service delivery platforms • Strong national and regional technical working groups (TWGs) advising NTP • Drug-supply and management (DSM) improved at all levels • Data quality improved and data-led program management established • Staff capacity at program management and service delivery improved • Programmatic Management of Drug-resistant Tuberculosis (PMDT) expanded and high quality maintained • Domestic funding remains low • Community and CSO involvement minimal Table 4: CTB indicators: achievement against targets Performance indicator Baseline Target Achievement Notified cases DS TB all forms CTB areas, number 119,592 522,954 416,654 (79.6 percent of target) Notified and enrolled in treatment DR TB, CTB areas, number 557 3,359 2,779 (82.7 percent of target) Treatment success rate DS TB, CTB areas, percent 92 95 94 (98.9 percent of target) Cure rate DS TB, CTB area, percent 82 85 90 (105.8 percent of target) Source: HMIS data, summarized by CTB 11 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity Evaluation Question 1: To what extent did CTB’s technical assistance and management approach support CTB cooperative agreement objectives? Findings Design of CTB Activity This section provides an assessment of the adequacy and appropriateness of the design and re-design of CTB to meet its objectives and is intended to provide context for the findings related to the evaluation question. CTB’s overall design was aligned with GoE efforts to address gaps in NTP and its strategic approach was to build national and sub-national capacity to deliver services. Under each of the three key objectives of the cooperative agreement, the sub-objectives and activities of CTB addressed key gaps and needs articulated in the national strategic plan (NSP) in the areas of laboratory services, treatment, infection control, drug supply management, human resource capacity, and governance. CTB’s design also took into account the support provided by the PHSP Activity. CTB’s strategic approach was to build NTP’s capacity at national, regional and implementation levels to reach the NSP targets. At the national level, CTB support aimed to strengthen available platforms to provide strong, evidence-based technical direction to the NTP, the national agencies EPHI and PSA on diagnostic services and drug supply management (DSM) respectively, focusing on those thematic areas where national guidance was weak or non-existent. National level support also extended to St. Peter’s and ALERT Hospitals, to develop them as Centers of Excellence (CoE) for MDR TB management. At sub-national levels, CTB supported coordinated program management, including planning, managing supplies, training, and supervision and monitoring of service delivery. At the service delivery level, CTB’s approach included mentoring facility staff on the various aspects of TB services. Overall, CTB support was ambitious and in line with End TB targets, to which the CTB cooperative agreement aims to contribute. CTB aimed to function alongside the program teams at all levels to build capacity for management, coordination, and implementation, but its support also included the provision of routine supplies and paying for routine activities right up to the concluding months of CTB implementation. Such support was deemed necessary to keep the program operational, due to very low levels of domestic funding for NTP and government ownership. Emphasis on demand generation was low. While the design of CTB encompassed all thematic areas of TB, the emphasis was clearly on facility-based service delivery and strengthening of health systems. CTB documents, KIIs and a review of secondary data show that the level of effort assigned for community-based interventions in the general population, and engaging community and civil society organizations was too low to lead to meaningful change, even after efforts to intensify support to community TB care as a high impact intervention. Interventions were designed to reach the woreda level, and not beyond. Support to woredas did not include capacity building of woreda level supervisors to assess the population coverage and quality of community-based case finding activities. CTB design did not include systematic inquiry into barriers for care-seeking, including gender and social barriers. These are significant in light of the basic premise of CTB to find and treat missed cases. Performance measures did not directly reflect the core functions of CTB. CTB’s key performance indicators were patient-level outcomes (case detection and treatment success), rather than measures of institutional capacity such as competencies of program staff to effectively use tools and guidelines, maintain equipment and independently manage supply systems. This is discussed further 12 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity under Evaluation Question 4 as a likely cause for continued dependence of program staff on CTB for program-level problem-solving. Multiple revisions to the design demonstrate CTB’s agility to meet emerging needs and put resources to more efficient use, but indicate the lack of a strategic approach, especially to geographic coverage. The geographic scope of regional level support was expanded twice. The initial plan of CTB was to cover Tigray, SNNPR, and the city administrations of Addis Ababa, Harari, and Dire Dawa. These were areas with minimal prior support from partners for TB, and did not have significant security issues and had reasonably functioning health systems. This plan was revised in Year 2 when two additional regions, Oromia and Amhara, that were under the support of HEAL TB were included, following the closeout of the latter. This merger was done as a step towards more efficient use of available resources, compared to a possible new Activity. Later the same year, Benishangul Gumuz and Gambella were added. The latter regions have considerably weaker health systems, lower utilization of health services, large (but undocumented) populations of refugees and frequently experience political instability. Gambella also has a higher HIV prevalence than in other regions. The high need for intervention in these two regions clearly justified their inclusion but including them late in programming left CTB with little time for implementation. These two sets of expansion have resulted in more intensified support for sub-national levels and more efficient use of available resources and paved the way for the improvements that CTB support was able to achieve. However, the efficiencies and outcomes could have been greater if there had been a strategic approach to the ultimate geographic scope. CTB’s geographic scope for regional and implementation support did not include the two pastoralist regions of Afar and Somali (which have higher TB prevalence rates than the national average) due to resource limitations and the likely high cost of implementing TB interventions, in the context of weak health infrastructure and basic health services in these regions. In Year 4, USAID requested CTB and worked with CTB to prioritize a set of interventions aimed at improving case finding, which had leveled off at 66 to 68 percent of estimated cases annually. With the support of international technical assistance (TA) from implementing partners, the CTB team identified eight high impact interventions and developed mini operational plans for each by reprogramming its budget. Another significant reprogramming took place in Year 3 when the NTP decided to adopt and roll out a change to the diagnostic algorithm, recommended by WHO. This involved the scale-up of use of the Xpert™ Mycobacterium Tuberculosis/Resistance to Rifampicin (MTB/RIF) assay (Cepheid, Sunnyvale CA., USA), an automated, cartridge-based nucleic acid amplification test that uses the multi-disease GeneXpert™ (Cepheid, Sunnyvale CA., USA) platform, as the primary diagnostic test. This expansion was meant to serve two purposes: i) provide universal access to drug sensitivity testing (DST), and ii) improve case finding. The latter was based on the assumption that cases are being missed due to less accurate conventional diagnostic methods. When CTB undertook the task of setting annual targets for its key indicators in 2017 (listed in Table 4 above), it reviewed the trend of the preceding years and found NSP targets (revised the same year) to be unrealistically high. So, CTB used the rate of decline of 7 percent between the actual cases found in 2016 and '17 as the base, and estimated that, with its support, the rate of decline would reduce to 5, 4, and 3 percent, respectively, for the next three years, nationally, and that CTB areas would contribute to 90 percent of those cases. Over the period of 2017-'19, the targets that CTB set in this manner account for 82 percent of the targets set in the revised NSP. For DR TB cases, CTB estimated that the total cases countrywide would increase from 2016 to '17 by 30 percent (due to expanded access to Xpert testing), and thereafter, by 10 percent year on year, and 13 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity that CTB would contribute to 70 percent of these cases in '17 and 90 percent of the total cases in '18 and '19. Finally, the targets thus set for 2019 were reduced to 75 percent of the set targets, as CTB was closing one quarter earlier CTB design and implementation built on past achievements, especially of USAID investments. The design of CTB carefully considered and built on the achievements of past USAID investments in TB in Ethiopia – The TB Coalition for Technical Assistance (TB CTA), 2000-’05; the TB Control Assistance Program (TB CAP), 2005-’10; TB CARE (2010-’15), and HEAL TB, 2011-’16. The evaluation team found this continuity especially in the areas of PMDT, laboratory services, and support to the TB Research Advisory Committee (TRAC). HEAL TB strategies and approaches such as the SOC tool, EQA for sputum smear microscopy, specimen transport, and the catchment area approaches were taken up in new areas at bigger scale through CTB. Most staff of HEAL TB were absorbed into CTB. The USAID Mission led the process of merging HEAL TB with CTB, ensuring continuity of interventions and advising on expansion to new geographies. Continuity in the position of TB technical staff at the USAID Mission contributed to the ability of the Mission to ensure that the design of CTB Activity complemented previous programming cycles. Role of CTB technical assistance in reaching CTB objectives. CTB’s overall technical approach was generic across all geographies and the three objectives and included training, one-on-one mentorship, supportive supervision for data-led management of service delivery, and program review. Interventions were tailored to the need of specific regions, where required, as with retrospective contact investigation, which was carried out in non-HEAL TB regions. New interventions (such as the diagnostic algorithm for childhood TB) were piloted before scale up. CTB’s support to the NTP and national agencies was channeled through the national technical working group (TWG). CTB worked with USAID-supported staff in the NTP team to develop terms of reference for the national TWG, including the establishment of sub-groups for MDR TB, laboratory services, DSM, TB IC, community TB care and childhood TB. The TWG, in turn, set the agenda for updating national guidelines and manuals. CTB provided technical, logistical, and financial support to TWG activities and advocated through the national TWG to include indicators on contact investigation (CI) and Isoniazid preventive therapy (IPT) for treatment of LTBI children under five years of age (U5) in the national HMIS. CTB also took the lead role in revitalizing regional TWGs. Strengthening TWGs is part of CTB Objective 3 (strengthen TB platforms), and the TWGs, in turn, were CTB’s strategic mechanism to implement changes required to align national guidelines, training material and tools with international standards of care, which enabled the achievement of CTB Objectives 1 and 2: providing high-quality diagnosis, care and preventing TB transmission and disease progression. International TA was of high quality and relevant to in-country needs. Twenty of 58 planned short-term technical assistance (STTA) visits were completed. The rest were canceled due to extended planning that took place in Year 2 to incorporate changes to the geographic scope, and security issues in Year 3. Reports of completed STTA visits show that each visit had clearly defined objectives and resulted in specific and time-bound recommendations. Areas of support included quality assurance mechanisms for all diagnostics, specimen transport, quality of treatment, CI, TB IC, key populations5 (KP), programmatic management of drug-resistant TB (PMDT) including the rollout of the short treatment regimen (STR) for MDR TB and for designing OR studies. The majority of the recommendations were implemented by the CTB team, resulting in considerable improvements in culture and drug susceptibility testing (DST) 5 The Global Plan to End TB (2016-’20) defines key populations as those who are vulnerable, underserved or at risk for TB infection and disease. 14 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity services, EQA for microscopy, specimen transport, and TB IC, all of which were reviewed and/or observed by the evaluation team. Recommendations that were not fully implemented include: capacity building of program teams for maintenance of lab equipment (lack of funding and ownership by the government, discussed under evaluation question 4) and rolling out an electronic recording and reporting system for DR TB patients (still being piloted). Frequent STTA for lab support compensated for the lack of a senior international lab expert within the CTB team. Several of the canceled visits were related to establishing capacity for culture and DST in labs. However, the fact that CTB has contributed extensively to achieving the targeted capacity for this service (discussed in detail under evaluation question 4) indicates that the TA provided for this area has been adequate. One technical area for which all planned visits were canceled was intensified case finding of all risk groups. While some of the risk groups have been covered under TA for KPs, some risk groups inadequately reached include PLHIV and refugees. CTB staff were competent to carry out activities under all CTB objectives, but competencies related to demand generation were lacking. The profiles of CTB staff meet or exceed the minimum requirements in the respective job descriptions. RHB staff were involved in recruiting staff for CTB positions. The technical rigor and relevance of resource material that CTB supported in producing, the quality of TA provided, and perceptions of stakeholders indicate that CTB teams were highly competent in applying technical knowhow and evidence to developing guidance, SOPs, and tools. KIIs with program and facility staff showed that CTB staff were competent in training in all thematic areas under CTB Objectives. Stakeholders also commend the high levels of commitment of CTB staff in improving TB program management and service delivery. During the transition planning meeting, a regional TB staff is said to have commented: “How do you transition commitment?” Government counterparts at all levels and other partners supporting NTP considered CTB staff competent in identifying and addressing gaps, mentoring facility and program staff, and helping them gain both technical and programmatic skills that enabled them to provide or support high-quality services. They also perceived CTB’s TA to be highly relevant, guided by needs, prioritized and with a strong focus on building capacity. However, all of these competencies pertain to program management and facility-based services, and not to demand generation, as noted earlier. CTB teams have not critically assessed barriers and enablers to care-seeking and reviewed evidence in order to design appropriate strategies to improve it. Capacity building was relevant and of high quality, especially for program management, DSM, TB and MDR TB clinics in facilities. It did not sufficiently address capacity building needs of HIV clinics and OPDs.: CTB built human resource capacity through developing and disseminating resource material and through training and mentorship. Training events were cascaded from international or national levels to peripheral levels for all thematic areas included in CTB Objectives – comprehensive TB, TB HIV, lab, DSM and PMDT. They were led by national and international experts and made use of standardized training material that was developed by FMoH with technical and financial assistance from CTB and tools that were to be used during implementation. CTB and program reports indicate that all program staff at national, regional, zonal and woreda levels, and TB-specific staff of facilities were trained in these thematic areas. In addition, “gap-filling” training events were repeated throughout CTB’s implementation period to compensate for attrition in program and facility teams. CTB brought international trainers for training in clinical management and PMDT at the national level and supported international training for government lab staff. PMDT training was cascaded by a few highly competent trainers and with fewer tiers, so as to minimize dilution. CTB also provided technical and financial assistance to update, print and disseminate guidelines, algorithms, recording and reporting formats. Ongoing learning was reinforced through mentorship and supervision, which 15 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity contributed to the improvements in program management capacity and the quality of service delivery. These are discussed in detail under evaluation question 4. The evaluation team also observed that training and mentorship did not include all facility staff such as OPD clinicians and staff of the HIV clinic, HMIS, and pharmacy. For the OPDs, CTB provided orientation to the OPD heads (matrons) and expected them to cascade it to all clinicians in OPDs. The evaluation team observed that most OPD clinicians were unfamiliar with active screening for TB and prioritization of patients with cough. These are discussed in detail under evaluation question 4. CTB adapted the comprehensive Standards of Care (SOC) tool used in HEAL TB and introduced it as the main tool to conduct supportive supervision of service delivery. CTB trained and mentored zonal and woreda TB focal persons in using the tool and interpreting the data in order to identify gaps in performance and facilitate corrective action. This is described further under Evaluation Question 3 below. CTB mentored staff in TB clinics, MDR TB treatment initiation centers (TICs), wards, and treatment follow-up centers (TFCs) in all aspects of case management, recording, and reporting, and HMIS staff in facilities, woredas, and zones in compiling and reporting accurate data. CTB provided technical and financial support for quarterly supportive supervision from regions to zones, zones to woredas and woredas to facilities, and also trained woreda officers and lab staff of facilities in EQA for sputum smear microscopy. CTB’s contribution to developing and disseminating material for information, education, and communication (IEC) has been minimal. The 12 patient interviews that were held indicated adequate education on treatment adherence and cough hygiene through TB clinic staff but the education of all OPD visitors and inpatients on recognition of symptoms and cough hygiene were not observed. CTB provided training and mentoring for PSA national and regional teams and pharmacy staff of facilities on quantification and review of requests. However, CTB teams also directly transported drugs and supplies between facilities, when supply issues were brought to their attention. CTB procured and donated high-end medical equipment, as well as supplies, office furnishing, and printed resource material and tools. CTB provided logistic and financial support for a range of program management activities. The achievements and challenges resulting from CTB’s capacity building efforts are discussed under Evaluation Question 4. Role of CTB management approach in reaching CTB objectives CTB’s management approach was the same across the three objectives and 12 sub-objectives. The overall approach included a clear assessment of gaps in TB program management and service delivery at national, regional and facility levels, and developing responsive workplans. Annual planning cycles included the NTP and other TB partners and used the gap analysis from the preceding year. CTB’s staff structure was similar to that of the government. The teams were smaller than those of HEAL TB in order to efficiently use resources and wean-off support but ended up having higher travel and work burden. CTB had central, regional and zonal teams. The central team included advisors for all thematic areas included in CTB sub-objectives. Stakeholders from government and partners stated that the central CTB team provided high-quality technical assistance to the NTP and national agencies and guided regional teams in planning and implementation. Regional teams were similar in structure to the central team and they worked closely with their counterparts in the RHB to implement program management activities, and train, supervise and mentor program managers in zones and woredas and the facility staff in their respective thematic areas. CTB teams in the regions had dedicated office location and logistics, due to the size of the team. RHBs expressed that this was a missed opportunity for closer working relationship and skills transfer, referring “CTB made us ready for the job” - Head of a zonal health department 16 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity to other donor-funded mechanisms that were co-located with their government counterparts, as good examples. Some RHBs and one partner expressed concern about the lack of technical accountability (towards the joint plans and deliverables) of CTB to the RHB counterpart. However, CTB made this deliberate choice to enable its teams to provide focused TA related to all its sub-objectives and rapidly build capacity in national, sub-national and facility teams. CTB aimed to continue the support that HEAL TB provided zones, woredas, and facilities at a similar intensity but with fewer staff - HEAL TB had TB and lab advisors for each zone, while in CTB, the same staff cover three to five zones each. This arrangement intended to efficiently use available resources and intended to wean off support. However, the expectations from TB staff in RHB, ZHDs, and facilities, as well as the well-intentioned efforts of CTB staff to achieve CTB targets led to these smaller teams trying to provide the same level of support as before. This led to a higher travel and work burden on staff with only a minimal reduction in dependence of government staff on partner support. The implementation arrangement included KNCV, MSH, and WHO, where KNCV and MSH were assigned sub-objectives and geographies and worked collaboratively with WHO. MSH and WHO came on board as partners when the expansion to additional regions took place in Year 2. In addition to overall leadership, KNCV had primary responsibility for developing guidelines and tools, and the thematic areas of targeted screening, case management, prevention, and monitoring and evaluation (M&E). It also continued in SNNPR and the three urban locations and Gambella was added in Year 3. MSH had primary responsibility for diagnostics and drug supply management (DSM). It continued in Oromia and Amhara and took over Tigray, and later, Benishangul Gumuz in Year 3. WHO was responsible for the sub-objectives on political commitment and comprehensive partnerships, and supported KNCV in human resource development and interventions related to patient management. STTA was provided by headquarters (HQ) staff of all three organizations. Based on the extent of support required in regions, allocated budgets were allocated in the ratio of 52:45:3 for KNCV, MSH and WHO respectively. All three partners reported that they worked collaboratively and were part of the senior management team. The technical director drew from the expertise of all partners to identify needs and responses and to cost them; partners practiced mutual accountability and had common norms for operations, such as providing travel and per diem for CTB and program staff. CTB maintained high burn rates following geographic expansion in Year 2, through regular critical review of spending, joint annual planning and feedback to one another. Through the TWG and under the leadership of NTP, CTB coordinated and collaborated with all stakeholders. The TWG is the main modality for coordination between partners. Secretarial support for the TWG was rotated amongst partners in some regions, but in others, it is mainly supported by CTB. The NTP provided leadership to the TWG. Partners also came together with the NTP for annual planning, and this has helped avoid duplication. Additionally, efforts to avoid duplication extended to implementation levels. GHC and CTB worked closely in TICs in SNNPR to apportion technical, equipment and patient support amongst them. GLRA and CTB collaborated in developing the SOP for prisons and related tools. Isolated incidents of lack of clarity at the initial stages of CTB were resolved through a consultative process. Understandably, the design of CTB support to facilities has been focused on the TB clinic, and support to other service units was commensurate with intended objectives. However, the evaluation team found that the facility management and other units, in particular, the HIV unit, were not optimally engaged in the training, onsite support and review activities of CTB. “CTB staff respond to NTP’s questions clearly and reasonably. They are highly familiar with the national program” - Director of a partner organization 17 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity A key challenge was the reduced time available for implementation. CTB experienced significant delays and loss of time during implementation. CTB received formal approval five months after its inception date. The merging of HEAL TB in Year 2 and addition of Gambella and Benishangul Gumuz in Year 3 necessitated significant revisions of work plans and budgets, all requiring additional time. Additionally, planning took longer than expected due to prioritization of high impact interventions. Year 5 is closing three months earlier than planned. CTB staff stated that Year 4 was the only year with full implementation, as seen in Figure 4. Figure 4: CTB timeline and inclusion of geographic areas Conclusions CTB’s design was aligned with the efforts of GoE to address needs and gaps in the NTP. Its strategic approach was informed by the contours of the country’s health system and the NTP. The thematic range of CTB’s interventions was comprehensive and its targets ambitious, in keeping with CTB being USAID’s flagship TB program, and in line with End TB targets. CTB’s strategic approach was to build national and sub-national capacity to deliver high-quality services, but low levels of domestic funding and government ownership led to CTB providing routine supplies and funding routine management activities. CTB built on the achievements of previous investments of USAID in TB control in the country. The Mission’s leadership has been vital and consistent for ensuring smooth continuity in support to the NTP. The emphasis of CTB’s design was clearly on facility-based service delivery and strengthening of community-level interventions to find and treat missed cases were too little done too late. Key performance indicators of CTB do not directly reflect its core function as a support mechanism. The implications of having such measures are discussed under evaluation question 4. Revisions to the geographic scope of CTB’s implementation-level support indicate that a strategic approach to coverage was constrained by extraneous factors. As a result, while the geographic scope turned out to be much larger than what was initially envisioned, regions that needed the highest duration and intensity of support received the least. Additionally, pastoralist regions that have TB prevalence higher than the national average have not been included in CTB’s regional and zonal support. The major revisions to re￾prioritize eight interventions aimed at finding missed cases and to support the expansion of Xpert assay as the primary diagnostic test demonstrate the agility of CTB to reprogram its workplan and budgets based on emerging needs. CTB’s technical assistance combined strong central and field-level TA to focus its support on clearly identified needs and gaps in the NTP at national, regional and peripheral levels in program management 18 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity and facility-based services in all thematic areas under CTB objectives. STTAs have been of high-quality, relevant to need, and resulted in clear recommendations. Action taken on the basis of STTA recommendations led to the expansion of quality-assured laboratory services, especially culture DST and excellence in PMDT. Under NTP leadership, CTB worked with partners to ensure that the various TWGs function optimally and that they help set national technical agenda and serve as platforms to develop policy and strategies. CTB built the capacity of the national TB program at all levels through high-quality training, mentorship, tools, equipment, and supplies. However, high staff turnover and ongoing support for routine activities threaten the high level of functionality that CTB established. CTB’s management approach was informed by its objectives and strategic approach. The composition and skill mix in CTB teams has helped provide support in all thematic areas. Staff were competent for their job profiles and had high levels of commitment. These competencies are reflected in the extent of capacity built in program management and service delivery. CTB worked with the NTP and other partners rather than in isolation, and were perceived as strong collaborators, transparent and committed to avoiding duplication. Continuing the implementation approaches of HEAL TB with reduced staff helped CTB successfully address needs in the program at zone and woreda levels and build capacity but led to staff overburden and only a slight reduction in dependence. Additionally, delays caused by revisions in geographic scope reduced the effective implementation time. Evaluation Question 2: To what extent did CTB implementation approaches use international standards and proven strategies? Findings CTB supported NTP efforts to ensure that national standards were fully aligned with international standards.: CTB provided TA and financial support through the TWG to update national guidelines, training manuals, algorithms and tools to support the incorporation of the International Standards for TB Care6 into national guidelines. These international standards describe principles of accepted levels of care for TB that are the same worldwide. They are supported by WHO guidelines and policy statements, that have been developed using rigorous methods. These Standards are generally considered to be feasible now or in the near future, given the rapid deployment of new technologies and approaches in most developing country contexts. As well as being essential for good patient care, they are the foundation of the public health response to TB. The resources that CTB supported in developing include: • Guidelines: Comprehensive TB, TB HIV, and Leprosy (two editions in 2016 and 2018 respectively); Programmatic management of DR TB; Management of TB in children; Lab services and EQA; Airborne infection prevention and control; Engage TB (for community TB care); Finding and treating missed TB cases; • Manuals for training facilitators and participants: Comprehensive TB, TB HIV and Leprosy; Nutritional assessment, care and support for TB and DR TB; Critical care in DR TB management; Audiometry for monitoring treatment for DR TB patients; Training HEWs in TB and TB HIV; Woreda-based programming for KPs; Specimen transportation; Sputum smear microscopy; New pediatric fixed-dose formulations; TB Managers’ training; 6 TB CARE I. International Standards for Tuberculosis Care, Edition 3. TB CARE I, The Hague, 2014. 19 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity • Standard operating procedures (SOP) and job aids: Supportive supervision; CI and IPT; screening DR TB patients for hearing; culture DST lab quality improvement process towards accreditation; job aids for sputum collection and microscopy; TB screening, diagnosis and treatment in prisons; courier transportation and waste disposal; TB screening tool; diagnostic and treatment algorithms • Registers for the TB unit, CI and IPT, DR TB, DR TB follow up, referral, smear microscopy, culture DST, prison screening; recording and reporting forms for the full range of actions related to specimen transportation; treatment supporters • Checklists for regions, zones and woredas; TICs and TFCs; health facilities, health posts, regional and facility labs Observations, document reviews, and KIIs showed that these resources are consistent with the international standards for TB detection, diagnosis, patient management, TB IC, management of LTBI, TB HIV and monitoring and evaluation (M&E). KIIs indicated that CTB ensured the use of these resources in all its training and mentorship activities. The evaluation team also observed that the national guidelines list the three treatment options recommended by WHO for LTBI and the advantages of each but does not provide clear guidance on which option is recommended currently, and why, and under what conditions the other options would be recommended. CTB support emphasized quality in all aspects of management and service delivery, addressing the key dimensions of quality: assessment, action and follow up using high￾quality data, collected through the SOC tool. SOC includes standard indicators for case finding, diagnosis, treatment choices and adherence to treatment for drug-sensitive (DS) TB and DR TB, CI, TB HIV, community TB care, lab, data quality, TB IC and DSM. The SOC tool and its use by CTB in ensuring quality are explored fully under evaluation questions 3 and 4. CTB supported adherence to national standards in all thematic areas of service delivery: CTB printed and distributed the updated resources listed earlier, and mentored facility staff in their use, supported government TB staff to provide mentoring using the SOC tool, and regular program reviews. The following are specific standards for which adherence is now nearly universal due to CTB support: • DOT implementation: In all but two of the eight sites where the evaluation team assessed DOT, it was being practiced both in the intensive and continuation phases of treatment. In the two sites, drugs were given directly to patients every week in the continuation phase. For such patients, there was no evidence that one of the patient-centered treatment monitoring approaches- either treatment supporter or HEW- was available for observing the treatment. • Prospective quarterly CI for contacts of index TB patients and provision of IPT. • Xpert assay as the initial diagnostic: CTB was the major supporter in ensuring the rollout of the changed algorithm. CTB provided technical and financial support for developing, printing and distributing guidelines and tools for the use of Xpert. CTB staff continually supported lab staff in the use of Xpert and in low-level maintenance. CTB also procured 12 machines, all related supplies, and high-level curative maintenance support. • PMDT implementation: CTB supported training in PMDT and equipped TICs, provided almost all gap-filling supplies related to DR TB lab services, patient monitoring, and socio-economic support. It also provided financial and technical support to link TICs and TFCs to ensure continuity in patient care, such that PMDT implementation is fully consistent with national standards. Adherence to national standards was variable in the areas of active case finding in high caseload facilities in all regions visited, CI, TB IC, TB diagnosis in PLHIVs and community TB care. These are described under evaluation question 4. 20 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity Conclusions A key outcome of CTB TA has been the updating of national guidelines and algorithms to align with international standards. Current national guidelines are consistent with global standards for TB detection, diagnosis and patient management including for DR TB, TB IC, LTBI, TB HIV, and M&E. CTB uses high-quality, globally recognized parameters for quality and has rolled out a mechanism to assess and assure quality through the SOC tool. CTB support has also ensured that all mechanisms are in place for national standards to be adhered to in service delivery. In particular, DOT has been expanded and is universally applied, especially in the intensive phase. Adherence to standards is uniformly high in all areas except in DOT provision in the continuation phase in some sites, community and facility screening, CI, TB IC and TB diagnosis among PLHIVs. By disseminating these resources and supporting training and mentorship to ensure that the resources are put to use, CTB has set up the foundations for continued adherence to standards in TB care. Evaluation Question 3: Was the information generated by CTB used to help achieve CTB objectives and outcomes and, if so, how? Findings The following were key sources of data for CTB: 1. Patient-level data captured routinely in facilities, processed through the national HMIS and compiled every month. 2. A comprehensive set of 36 data points captured routinely in facilities and compiled by program staff and CTB staff using the SOC tool every quarter. These include patient-level data as well as those related to the coverage and quality of services such as lab and DSM. CTB has established this routine practice. 3. Non-routine generation of evidence conducted or supported by CTB: OR and assessments HMIS/DHIS2 data: Review of documents and secondary data and KIIs with program and facility staff showed that CTB did not set up a parallel mechanism to collect data for TB indicators included in HMIS. Instead, it worked to reduce errors in the capture of data in recording and reporting forms, as well as errors in data compilation, by training and mentoring TB and HMIS teams in facilities. CTB included the recounting of data for selected HMIS indicators in the SOC tool, so that woreda and zonal program staff can track and rectify errors. The evaluation team found a steady reduction in errors in data compilation and capture over the period of CTB support. The four key performance indicators of CTB are included in the HMIS and so CTB’s actions to improve HMIS data quality also improved its ability to track its performance. DHIS2 has replaced HMIS over the past year. CTB supported the transition, through training, mentoring and supportive supervision. The SOC tool: Indicators for the tool were taken from WHO standards and the national HMIS. It also included process indicators such as those related to specimen transport, lab EQA, CI, TB IC, and DSM and data quality, and output indicators such as the yield from outpatient department (OPD) screening, and sputum conversion. The tool was designed to assess the performance of each facility, identify gaps, training and mentorship needs, and also to monitor gaps in supplies and equipment. Each indicator had built-in thresholds for performance, and indicators that showed poor performance were assessed further for causes, remedial action was planned and their progress monitored. KIIs with program staff and review of completed SOC tools showed that data from the tool was compiled at woreda, zonal and regional levels and used in TB program reviews. The evaluation team found this to be routine practice in 21 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity all the sites visited. CTB also used SOC data to prioritize locations for sending the supplies and to re￾design the route of the cold chain van. Assessments: Baseline assessments in regions not covered by HEAL TB helped CTB identify gaps and design/prioritize interventions. Most of the gaps identified, such as the lack of triaging for cough in OPDs, low uptake of Xpert assays, non-use of patient kits and low levels of CI were found in all regions where the assessments were conducted. CTB also used data from screening activity among KPs to discontinue this activity among some KP groups and continue others. Evidence generation through ORs: CTB designed and completed 13 ORs. Four more are underway and three were canceled due to budget constraints. Table 5 gives a list of decisions taken by CTB based on evidence generated by ORs. Ethiopia was early in adopting the new diagnostic algorithm, supported by a modeling study that not only showed the effectiveness of Xpert as the initial diagnostic but also raised concerns about its affordability. The NTP commissioned a pilot to assess the feasibility of Xpert rollout, but the pilot was not fully implemented or evaluated before national scale up. Thus, the potential insights on operational issues that could have been gained from the pilots were not realized. Annex VIII: Operations Research and Presentations contains a comprehensive list of ORs and presentations conducted by CTB. The evidence generated helped further the achievement of all three objectives of CTB, in terms of TB thematic areas (Objectives 1 and 2) and health systems strengthening (Objective 3). Table 5: Program decisions resulting from ORs OR topic Result Action taken/limitations A cross-sectional study evaluating routine CI in Addis Ababa Recording and reporting routine CI was not being practiced consistently, minimal, despite routine CI being part of national guidelines SOP and tools for CI were developed and rolled out Modeling the impact of the new diagnostic algorithm for the diagnosis of pulmonary TB in Addis Ababa Of the eight diagnostic algorithms studied, the model of full rollout of Xpert showed the highest reductions in TB burden but entailed prohibitive costs. The model of targeted use of Xpert along with other methods was more affordable. NTP commissioned a pilot to study the feasibility of full rollout of Xpert, and CTB supported full scale up before evaluating the pilot Comparison of the yield of TB among contacts of MDR TB and drug-sensitive TB patients in Ethiopia using Xpert as a primary diagnostic test Yield of TB among contacts was higher using Xpert than in the general population This contributed to and supported the decision to roll out Xpert as the initial test. The comparison, however, has not been with another diagnostic for the same population A cross-sectional study on TB screening among hospital inpatients in a large referral hospital in Oromia TB was missed among several inpatients CTB included screening for TB at all entry points including inpatient wards, under its “Hospital Initiative” 22 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity OR topic Result Action taken/limitations Impact of technical support on TB IC in hospitals in two regions After a year, some TB IC practices improved, but managerial measures remained sub-optimal This is not new evidence, but it helped CTB advocate for a renewed focus on TB IC Household infection control status among MDR TB patients in SNNPR Houses of MDR TB patients were poorly ventilated and crowded Served to reinforce the need for patient education on infection control Evaluating the feasibility of online self-screening approach for TB case finding among students in Kotebe University Use of this tool was found to be feasible but did not yield any case CTB did not proceed with scaling up this intervention There is no evidence that CTB critically reviewed and optimally utilized output-level service data - especially in finding missed cases - in assessing the outcome of Xpert rollout and to gain insights into demand for TB services. A critical review of output-level data: CTB compiled data from its high impact interventions on case finding among KPs, contacts, and from door-to-door screening. A review of the data showed high variability from quarter to quarter in the yield of presumptives from these interventions and the proportion of confirmed cases from presumptives. However, there is no evidence that CTB critically analyzed, revisited its assumptions and drew specific conclusions based on the data regarding their impact on finding missed cases, which was a principal purpose of the high impact interventions. Summaries of these data points are presented and discussed under evaluation question 4. The ambitious expansion of Xpert testing was expected to serve the two purposes of universal access to DST testing and improving case finding. There is no compilation of data to verify the extent to which these two purposes have been achieved, even though the data points are available in the paper-based recording system. CTB did not conduct or support research aimed at gaining insight into the demand for TB services, such as barriers for seeking care, and the influence of gender roles and perceptions in care-seeking. This is a significant lapse because a community randomized trial that deployed HEWs in SNNPR, Ethiopia in 2008 in active case finding found more women patients,7 which is an unusual feature in TB epidemiology. Instead, as described under evaluation question 1, CTB’s efforts to find and treat missed cases have been heavily focused on improving facility-based services. Sex disaggregation was not available for output-level data on case finding such as data on mass screening, CI, OPD screening and community-based case-finding activities. Conclusions CTB generated and used evidence to re-focus its interventions as exemplified by the prioritization of KP groups. CTB also contributed to improved data quality and use in the management of the program at all levels and improving service delivery. The SOC tool has been the cornerstone of program improvement support. It helped CTB to routinely monitor progress along all thematic areas and take corrective action. CTB collected a wealth of output-level data, but not all of it was used for critical analysis and to trigger remedial actions in time, especially to understand the vexing question of plateaued case detection over the past several years. Barriers to care-seeking and the potential influence of gender roles and 7 Datiko DG, Lindtjørn B (2009) Health Extension Workers Improve Tuberculosis Case Detection and Treatment Success in Southern Ethiopia: A Community Randomized Trial. PLoS ONE 4(5): e5443. doi:10.1371/journal.pone.0005443 23 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity perspectives in care-seeking have not been studied in depth, and these are all the more perilous in being unrecognized. Adoption of the new diagnostic algorithm was rolled out before the pilots could be completed, and hence did not benefit from the potential insights on operational issues that could have been gained from evaluation of the pilot. Sex-disaggregated data was not available at the level where the information is likely to provide useful insights on care-seeking patterns. Given the breadth of work supported by CTB and the multiple mid-course revisions, a mid-term evaluation would have helped determine value and change and determine what is working and what is not, and what changes could make things work better. Lack of assessment of the impact of Xpert rollout on case-finding is another missed opportunity. Evaluation Question 4: What were CTB’s main achievements and challenges? The main achievements and challenges of CTB are presented under the themes of program management capacity, case finding, treatment outcomes, and infection control, as outlined in the sub-questions of evaluation question 4. The eight high impact interventions, prioritized in Year 4, are marked where they appear. 4.1. Achievements and challenges in supporting TB program management capacity Findings CTB successfully advocated for government staff positions for TB, required to bring about improvements in program management, but high staff turnover has been an impediment. Through the national and regional TWGs, CTB continued and ramped up advocacy for creating government-paid staff positions for TB thematic areas, an initiative that was supported since 2013 by earlier USAID-funded mechanisms. Now there are TB focal persons and advisors for laboratory, DSM, M&E in some regions and zones. All regions except Tigray have TB focal persons in woredas. These staff positions have been critical for the improvements in TB program management that CTB planned and implemented. Other regions have smaller TB teams, and there is only one TB staff in one region. Government and CTB staff at all levels report that turnover of staff has been an impediment to capacity building. Based on findings from all KIIs, it is evident that turnover has been high in the national team and in facilities and relatively lower in regions, zones and woredas. KIIs with program staff revealed that these positions are likely to continue after the closure of CTB, as they are paid for by the respective levels of government. CTB built capacity for routine, data-led program management at all levels through the use of the SOC tool; however, CTB continues to pay for most of these activities. CTB zonal and sub-city staff have trained woreda TB officers to complete the SOC tool for facilities, support evidence￾based action, and compile this data for use at the zonal level. CTB staff have also successfully mentored zonal TB officers to use woreda-level data from the SOC tool to support woreda officers in implementing remedial action. This is done through quarterly one-to-one meetings between the zonal and woreda levels, that was initiated during HEAL TB, and is now expanded to all CTB regions. The evaluation team found that the zonal program staff in all sites visited interpret SOC data, provide feedback and recommend remedial action. Through training and mentorship, CTB worked with TB clinic and HMIS staff in facilities to identify errors in data capture (in registers), and data compilation (in HMIS formats). The SOC tool was used to track progress in these areas. CTB also trained TB and HMIS staff on TB indicators and their calculation. CTB financially supported woreda-level HMIS meetings in some locations. CTB provided financial and technical assistance to RHBs and ZHDs to carry out assessments of data quality in facilities using 24 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity WHO’s Routine Data Quality Assessment (RDQA) tool. A review of RDQA reports showed verifications of the level of data accuracy and completeness. Compilation of data from SOC forms shows a reduction in variance between 2017 and 2018 from -1.1 to 0.2 for the number of TB patients cured, from 7.2 to 2.6 for the number of confirmed TB cases referred by HEWs. While these are averages across CTB regions, Gambella continues to have high variance, showing the need for continued support in that region for data quality improvement, among others. CTB also successfully continued the support introduced in HEAL TB, to RHBs and ZHDs to map geographic catchment areas for hospitals, EQA labs, and TICs. This helped identify the health centers that fall under each hospital’s area, the labs that fall under each EQA lab, and the TFCs that were within a TIC’s area, and thus clearly defined, without overlaps, the referral pathways between hospitals and health centers, labs and EQA centers and TICs and TFCs. CTB also pays for the catchment area meetings and mentors facility staff leading these meetings. Because of this activity, hospitals, EQA centers, and TICs interact with and support the health centers, labs and TFCs respectively, in their area, and conversely, the health centers, labs and TFCs know where to refer patients and samples to, and overlaps are avoided. CTB provided technical and financial support for integrated and TB program reviews at national, regional and zonal levels to be regularly conducted, but prioritized TB reviews, as they enabled them to discuss programmatic issues in detail and develop remedial action. CTB is the sole supporter of TB program reviews. The evaluation team did not find evidence for the government’s plans to take over paying for these activities mentioned above. CTB support has contributed to considerable improvements in DSM, but there is a need for further capacity building. CTB supported the rollout of patient kits of first-line drugs (FLDs) in regions not covered during HEAL TB. CTB has supported training and mentoring of PSA hubs in using QuanTB and the Integrated Pharmaceutical Logistics System (IPLS) to assess and update stock status and redistribute based on updates from facilities. CTB seconded one staff to oversee support to the three PSA hubs in Amhara, by following up the stock situation continually, evaluate requests received at these hubs from facilities, and carry out supportive supervision of pharmacies in facilities. CTB regional staff coordinate stock assessment and redistribution with supply chain case teams in RHBs. KIIs with PSA staff in the hubs visited revealed that these activities have resulted in improved quality of requests for supplies and reports from facilities, improved communication between service delivery teams and the hubs, better coordination of redistribution of supplies when needed. PSA hubs in the center and in the regions visited by the evaluation team reported significant improvement in supply management. Drugs and supplies for MDR TB management are now part of IPLS. KIIs revealed that two hubs had a shortage of patient kits two months prior to the evaluation, as well as brief stockout in some facilities, which were promptly refilled through the emergency response mechanism from another hub that had excess supply. The team also observed expired FLDs and history of wastage in a facility in Tigray, because the drugs received were close to expiry. Temperature maintenance is a challenge due to high humidity and power interruption in Gambella. CTB built in-country capacity to generate and use evidence, but continued financial and technical support is required. CTB continued and enhanced the support that earlier USAID mechanisms provided to TRAC, which is the national mechanism that brings together the TB program practitioners, researchers, and academia. TRAC is recognized by the International Union for TB and Lung Diseases and by WHO as a model for in-country collaboration in TB research. CTB continued funding the annual TRAC conferences, training and funding researchers from RHBs and universities to design and conduct OR and in setting up regional research teams. An STTA supported the development of the national OR roadmap for TB, which has a prioritized list of research questions. 25 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity Although CTB staff have mentored regional program staff in developing research questions, obtaining ethical approval, and conducting research, and RHBs have learned much from TB programs over the past ten years, CTB staff reported that program staff will require continued financial and technical support to be able to do this on their own. CTB supported expansion of access to KPs (high impact intervention) but continues to support costs for most activities. CTB has successfully supported the NTP in assessing and prioritizing KPs from a comprehensive list and in drafting and launching the National Strategic Framework for Key Populations and an operational guide, with STTA from KNCV HQ; SOP for TB Prevention and Care in Prisons, Detention Centers and Prisons; and a Roadmap for Ending Childhood and Adolescent TB. The NTP now considers reaching key populations to be a high priority intervention. CTB conducted three-day regional training events to disseminate the KP guidelines, however, CTB did not directly support case finding strategies in among nomadic pastoralist communities. Sensitization, provision of tools, and monitoring support from CTB have built the capacity of facilities within 52 prisons to screen inmates at entry and exit and in OPDs, and link patients with DOT centers closer to their homes, upon release from the prison. GLRA supports the remaining prisons in the country. CTB enhanced the support that began in HEAL TB for mining areas in Oromia and SNNPR, supported these RHBs and the woreda offices of these locations to transfer sputum specimen through the diagnostic network, and link diagnosed patients with community DOT, and continued woreda-level staffing in these locations that were hired during HEAL TB to coordinate these activities. CTB also provided financial and technical support for twice-yearly mass screening and one-time screening of contacts of index cases in these mining areas, as well as one-time screening in three urban settlements and homes for the elderly in Addis Ababa, screening of university students and holy water sites, and mining locations in Gambella. Although the facilities, including health posts in these locations, are capable of carrying out these activities, the costs are fully supported by CTB, which is a threat to sustainability. Program and facility staff continue to depend on CTB for problem-solving. KIIs with program and CTB staff in Addis Ababa, Amhara, Tigray, and Gambella revealed that zonal and woreda TB program staff and facility staff interact more frequently with CTB staff than their respective government counterparts for problem-solving for issues such as shortage of supplies or GeneXpert functioning. Zonal and sub-city TB focal persons report that they find approaching CTB staff rather than their supervisors to be more efficient because they know their supervisor will likely take the issue to their CTB counterpart anyway, since CTB is technically, financially, and logistically competent to resolve issues. National-level government staff also acknowledged this dependence on CTB for problem-solving. Program staff at all levels in all sites visited report that CTB staff promptly respond to issues by initiating strategic or ad hoc measures or by escalating them appropriately. Regional and zonal program staff report that the demarcation of roles and responsibilities between them and CTB teams became blurred during day-to-day implementation, such that CTB teams actually did the work, rather than support program staff to do the work. A member of the CTB central team remarked, “We are inside the system”. In one region, the evaluation team found that regional CTB staff follow up regularly on individual MDR TB patients and on the functioning of GeneXpert machines in each site. The idea that CTB is a time-bound support mechanism does not appear to have been clearly communicated frequently and intentionally, especially to facility staff, given the high turnover of facility staff. “CTB helped us work with practitioners and expand to regions. They (CTB) know what it means to operationalize even a minor change to the guidelines. They are excellent in understanding field realities, unlike us.” - Co-founder, TRAC “We are panicking ever since we heard about CTB closing” - Regional lab staff 26 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity High staff attrition and limited resources within the public health system meant that a certain level of direct management by CTB staff was necessary to maintain routine management functions for TB. These constraints notwithstanding, CTB could have proactively worked to reduce dependence, by carrying out the following: 1. Developed a clear exit plan as part of its design/re-design and implemented it, for routine supplies, payments, and management activities. Issues brought to the regional TB staff required technical, financial and logistical capacities which were located within the CTB system and required approvals along the CTB chain of command, and there was no strategy or plan to transfer these capacities to the program. This step could have helped build specific problem￾solving capacities in program teams. This would also have required financial commitments from the government for program teams, without which, dependence on support mechanisms seems inevitable. 2. Included, as performance indicators, measures of institutional capacity such as competencies of program staff to effectively use tools and guidelines, maintain equipment and independently manage supply systems. Inclusion of these process measures at the design stage could have provided the needed direction to CTB teams to prioritize building capacity in all areas over seeking to improve outcomes. High levels of staff attrition and low level of resources for TB programming in the public system could have hampered meaningful measurement of such processes, but just the emphasis on the processes is likely to have provided more direction and clarity for the work of CTB teams. 3. Developed clear lines of roles and responsibilities for CTB staff that gives clear direction for day-to-day functioning. Conclusions CTB successfully continued and scaled up the efforts by earlier mechanisms to build program management capacity, especially at zone and woreda levels. The level of competencies for routine program management observed at the zonal and woreda levels was unequivocally high. The program staff at zonal and woreda levels are able to collect, analyze and interpret high-quality data and use it for decision making on their own. CTB also supported RHB in revitalizing expansion of access to services based on evidence, to a range of KPs. CTB advocacy for government staff for TB thematic areas gave CTB counterparts to work with. CTB also played a key role in supporting TRAC to maintain research on the agenda of the national TB program, in line with the End TB strategy. CTB support has contributed to improved DSM for PMDT drugs and supplies, to the extent that supplies have been largely uninterrupted and optimally managed through CTB’s assistance for redistribution where needed. However, isolated instances of shortages, overstock, and stock-outs of FLDs point to persisting issues with quantification. 4. While the close follow-up and prompt support by CTB staff have helped resolve issues rapidly, they constitute direct management rather than support. This has created dependence on CTB staff in many places visited and replaced routine channels of communication and problem￾solving. This is a critical issue that threatens the sustainability of the gains made. Quantification and distribution have improved significantly and PMDT supplies integrated with PSA, but quantification issues persist in some regions. 4.2. Achievements and challenges in improving diagnostic services Findings CTB has successfully supported all levels of the national laboratory system to improve its capacity to provide high-quality diagnostic services for TB. This includes drug sensitivity testing and national-level 27 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity training in lab supply quantification. CTB is building EPHI lab capacity to obtain accreditation as a supranational reference laboratory. CTB supported the expansion of quality-assured sputum smear microscopy services, but there is a need for continued efforts to maintain microscopy skills. CTB procured 97 light emitting diodes (LED) fluorescent microscopes to initiate diagnostic services in high volume facilities. CTB used the catchment area approach to extend EQA services to all 3,012 facilities with functional microscopy services, by expanding the number of EQA hubs from 99 (across HEAL TB regions Amhara and Oromia) to 158 (across all CTB areas). Woreda TB officers are capable of managing the entire process, ensuring the management of discordant results through coordination between the peripheral labs and the EQA centers. CTB also trained nurses in 676 out of 950 facilities without microscopy services to fix and refer slides to a microscopy center in their network. These facilities were selected based on caseload and location. Slide-fixing has been a valuable stop-gap arrangement that has helped facilities continue providing diagnostic services to remote locations that do not have access to testing with Xpert. Sputum smear microscopy continues to have an important place among diagnostic methods. With Xpert becoming the primary diagnostic test, much of the microscopy work will be reserved for treatment monitoring. This, along with a declining incidence of TB will result in fewer and fewer opportunities for lab staff to view positive slides. CTB pays costs related to EQA in all CTB regions, except in Oromia and SNNPR where the government pays overtime for lab staff in EQA centers to carry out the additional work. CTB also continued regional-level training of incoming lab staff in sputum smear microscopy in Year 5. This illustrates the need for continued support for training government staff, in the most basic of services. CTB support was critical in the rollout of a new diagnostic algorithm with Xpert assay (high impact intervention) and the successful provision of the service, but it has not led to an increase in case finding, and its impact on access to DST is not yet known. Maintenance functions are not yet owned by the program. CTB played a significant role in expanding the use of Xpert as the test of choice for eligible conditions and subsequently, as the initial diagnostic tool for all persons with signs and symptoms of TB. CTB facilitated the updating of national guidelines, purchased 12 GeneXpert machines, conducted extensive and cascaded training of lab staff, supported the installation and use of GeneXpert in all sites through continual mentoring and troubleshooting, used the specimen transportation system (described below) for transporting specimens to Xpert testing sites, and is in the process of connecting all machines using GxAlert, a web-based data connectivity application. When supplies related to Xpert assay fell short of the demand, CTB reallocated its budget to purchase cartridges and falcon tubes, the most critical supplies needed for uninterrupted Xpert service. CTB also facilitated their distribution through the PSA system, carefully rationing the supplies based on caseload and location of facilities. CTB is the only agency procuring refurbished modules and calibration kits. In particular, procurement of cartridges was made necessary because of the acute shortage in supplies due to a mismatch between demand and the planned procurement through the Global Fund (GF) grant. This helped keep the Xpert tests going but has not been sufficient to meet the full demand for all machines. Supplies of cartridges are expected to last until September 2019 for maintaining the current level of service. CTB also procured power backup systems for sites with frequent interruptions in power supply. 28 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity CTB has worked to keep all GeneXpert machines running, through extensive maintenance support and training lab staff to carry out low-level preventive maintenance. CTB staff and some regional lab staff provide higher-level maintenance and fix issues like stuck cartridges and malfunctioning modules, which happen quite often in some locations, leading to interruptions in the service. In one site, cartridges got stuck thrice in the span of one year. CTB and regional lab staff who are able to resolve these issues are too few to promptly attend to them in all locations. Lab staff at some sites are trained in handling higher-level maintenance but don’t carry out those tasks because of fear of damaging the expensive machine. EPHI manages the relationship with the local service provider, who, along with CTB-seconded staff in EPHI, manage calibration and module change in all sites. CTB covered the cost of the contract for the current year. The service provider does not stock modules and spare parts; the manufacturer’s plans to open a workshop for East Africa in Addis Ababa have not yet been realized. CTB tried to offset this problem partly by procuring refurbished modules. One STTA had raised the centralized maintenance support as a concern, but the issue continues to date. Power supply interruptions are frequent and lead to loss of cartridges and interruptions in service. These were issues that facilities could not manage on their own, and hence the Xpert assay service was suspended. Lack of air conditioning leads to shut-down of the machine and this has been the cause for lower utilization of Xpert in Gambella. In these sites, staff run Xpert tests only at night. Some labs received locally-made falcon tubes (purchased and distributed by PSA) which had serious quality/safety issues. This was recognized promptly by facility staff and PSA has since discontinued procuring and distributing material from this source. GxAlert, when fully operational, will allow for remote, centralized and real-time monitoring of inventory, maintenance, utilization of the machines as well as disaggregated results, but the uncertain network connectivity in the country is likely to hamper its uninterrupted use. With the changed diagnostic algorithm, the utilization of GeneXpert machines has steadily increased, with but with large variability between sites. One facility conducted about 700 tests per quarter for three consecutive quarters, with the machine running day and night on some days, while a regional lab used its GeneXpert only as a backup. Table 6 outlines the level of functioning of GeneXpert machines in the sites that the evaluation team visited. Table 6: Functioning of GeneXpert in sites visited Level of site visited Modules: Functioning/Total Reason for not functioning National 12/16 Maintenance issue CoE 2/4 Limited cartridges CoE 4/8 Need Module change Regional 5/8 Maintenance issue Regional 7/10 Maintenance issue; Shortage of falcon tubes/sputum cups Regional 4/4 Regional 4/4 Facility 5/8 Maintenance issue Facility 4/4 Facility 2/4 Maintenance issue; no cartridge Facility 2/4 Maintenance issue; no cartridge “The Xpert system is fully dependent on CTB. The government does not yet own it.” – National government staff 29 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity About 10 to 14 percent of all Xpert tests are positive in CTB areas after the test became the primary diagnostic. These rates also vary considerably between sites, ranging from 4.2 percent in one site that received a large number of referred specimen (indicating that a large proportion of the tests were for initial diagnosis) and going up to 22 percent in another that does not serve as a referral center for specimens. Figure 5 gives the national data on Xpert tests and their outcome for the period for which this data is available, although Xpert assay was launched as a public service in 2013. Figure 5: Xpert tests - total and positive, national Source: CTB summary data Adopting Xpert as the initial diagnostic test was meant to serve two purposes. The first was to provide universal DST access, and the second, to increase case notification. Data on the proportion of all bacteriologically confirmed cases who had an Xpert test is not available, but using data presented in Figure 5, and the total number of new and retreatment cases notified, it can be deduced that about 27 percent of these cases were tested for RR during January-June 2018. There has been no change in the downward decline in case notification, either in CTB areas or nationally, as described under evaluation question 4.3 below. There is no data on the incremental yield above sputum smear microscopy, but the proportion of DS TB cases that are bacteriologically confirmed has been steadily increasing over the years, as Figure 6 shows. It is not possible to say if these additional, bacteriologically confirmed cases could not have been diagnosed in the absence of Xpert testing, or if these would have been clinically diagnosed. 30 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity Figure 6: Distribution of types of DS TB cases notified, national Source: HMIS CTB supported the expansion of culture and DST services (high impact intervention), especially for second-line drugs, but maintenance functions are highly centralized. Documents of CTB and KIIs with CTB staff and national/regional lab managers show that at the start of CTB, national and regional labs were in varying stages of preparedness to provide culture and DST services. Major infrastructure and equipment were already in place. Reports from STTA visits showed that STTA provided and supported by CTB identified persisting gaps in infrastructure and equipment, as well as training needs. CTB supported the costs of national and international training of lab staff in culture DST and line probe assay (LPA), quantification of supplies, and the lab quality management system. CTB procured services for certification of biosafety cabinets and negative pressure systems, and panels for proficiency testing conducted by the supranational reference lab in Uganda. CTB also procures basic supplies for culture and DST, both for routine services and for participation in the recently concluded drug resistance survey. A staff member seconded to EPHI coordinates the support required in all these sites. In the past three years, all nine labs have been implementing the lab quality management system (LQMS). Of particular significance is the expansion of LPA for second-line drugs (SL-LPA) to all DR TB cases, in line with WHO recommendation. As the equipment for this service was already available in all regional labs, CTB provided the needed training, supplies and mentorship for this service to be provided. As a result, the number of labs providing SL-LPA has increased from five to nine. With CTB support, eight of these participated in the most recent proficiency testing. CTB continues to support efforts to certify EPHI as a supra-national reference lab (ongoing) through financial contribution and STTA. CTB supported EPHI and regional labs to receive accreditation from the African Society for Lab Medicine. Two have been certified by the Ethiopian National Accreditation Office, and CTB is currently supporting the others for this. Data on SL-LPA testing is not complete. According to CTB data, only 59 percent (432/731) of DR TB patients notified from CTB areas in 2018 have information on SL-LPA testing, and 70 percent of them (296/432) have had the SL-LPA test. As with GeneXpert, the capacity for maintenance work is highly centralized. The biomedical engineer who is a staff member of a center with culture DST had not been trained in the maintenance-related skills. Six additional labs are under construction with support from the Global Fund and Centers for Disease Control and Prevention and they will require similar support to make them functional. 31 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity CTB strengthened specimen transportation (high impact intervention) in HEAL-TB regions, further improving access to diagnostic services. CTB provided financial and technical assistance to expand the services of the eight cold chain vans procured during HEAL TB to cover 28 TICs and 208 high-burden facilities. It also helped expand the scope of this service, to include specimen for Xpert assay, culture DST and for a range of other investigations such as viral load testing for HIV. CTB supported the coordination of specimen collection, submission to the testing facility and delivering results back. This cold chain transport system has been designed to complement and not overlap the postal transport system, which is supported by the government, through the GF, for specimen transportation across the country. CTB continued support to the eSpecimen referral system for electronic delivery of requests and delivery of results, but its implementation has been heavily constrained by network connectivity issues in the country. CTB also supported EPHI in developing an integrated specimen transport guideline with SOP for specimen transportation, including the postal system for other regions and EPHI-supported vans for Addis Ababa. In 2017, the CTB zonal clusters/teams focused on Xpert sites with the aim of supporting all means of specimen transportation, including the postal transport system extending to the level of rural information communication technology (ICT) officers, cold chain couriers, and CTB field vehicles, in addition to the cold chain vans. The various specimen transportation mechanisms now seamlessly connect health facilities with GeneXpert testing, SL-LPA, culture and DST and laboratory monitoring tests, through concerted efforts of CTB and Global Fund. The average time for collection and delivery of specimens from facilities to the culture DST lab fell from 7 days at the start of CTB to 1 day in 2017. The average sample rejection rate for specimen transported by the cold chain vans reduced from 3.4 percent in 2016 to 0.3 percent for samples received at the Adama regional laboratory in 2017. A total of 31,392 specimens were transported through the cold chain vans in 2017 and ’18. CTB pays the drivers, fuel, and maintenance for the cold chain vans and procured triple packaging for specimens. Oromia and Amhara RHBs are ready to take over running costs of the cold vans, with technical support from EPHI. An STTA recommended a single integrated specimen transportation system for the entire country, managed by EPHI, but that has not taken place. Table 7 below gives an overview of the improved access to diagnostic services over the period of CTB. Table 7: Changes in access to diagnostic services Service Baseline Current Number of sites Population coverage per site Number of sites Population coverage per site Sputum microscopy 2269 1 for every 43,587 2794 1 for every 39,420 Xpert 97 1 for every 1 million 314 1 for every 350,764 Culture DST 8 1 for every 12.3 million 10 1 for every 11 million Source: NTP records; the baseline coverage assumes that the country’s population of 98.9 million and the current coverage, 110 million. Conclusions CTB and the national and regional labs have worked together well to move towards accreditation. CTB supported strengthening laboratory network, specimen referral, EQA decentralization, slide fixing for non-diagnostic sites, and gap-filling procurements. EQA centers can carry out the process independently and assure the quality of diagnosis. CTB continues to financially support these processes, but RHBs are likely to prioritize funding allocation for this activity. Slide fixing has functioned well as a gap-filling intervention and will continue to be needed in remote locations and in areas with a staff shortage. The continued commitment of lab staff to conduct the additional tests is another factor determining the 32 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity continuation of this activity. Data on the proportion of DR TB patients receiving baseline SL-LPA tests are incomplete. Specimen transportation has been an effective mechanism to ensure that labs are networked and access to early diagnosis is improved, patient costs are minimized, and the national algorithm is properly implemented. CTB has played a vital role in the expansion of Xpert testing through installation, training, procuring supplies and maintaining the equipment and has ensured that GeneXpert systems are functional and optimally utilized. Xpert utilization rate is on the rise, but there are local differences, arising from differences in demand amongst facilities and the interruptions to Xpert assay services. KIIs with CTB staff, program staff and documents reviewed show that CTB adapted quickly to the spike in demand for Xpert-related supplies and procured all supplies that are in circulation now. Xpert expansion has been plagued by serious operational challenges: supply of consumables, electrical power, and maintenance issues. These this could have been managed better had the Xpert pilot been duly completed and evaluated. These issues will likely be better managed in future if GxAlert becomes fully operational and is supported by uninterrupted network connectivity. CTB’s procurement and distribution of consumables (cartridges and falcon tubes), while not ideal, has been timely and has helped in continued utilization of Xpert in key locations. However, the supply that CTB provided has not been sufficient to fill the entire gap in current supply. Expansion of Xpert as a primary test could lead to loss of microscopy skills among lab staff. Thus, with Xpert testing already expanded countrywide, it provides the country with a sound mechanism to test all cases found. There has been no increase in the proportion of estimated cases found, but the proportion of cases that are bacteriologically confirmed has increased. This is either an indirect measure of the accuracy of clinical diagnoses of TB prior to Xpert expansion, or the result of more cases being found because of Xpert expansion. The extent of achievement of the two purposes for which Xpert expansion was supported – universal access to DST and improved case finding – have not been objectively assessed yet, as there is no system for compiling the data available in the recording system. Thus, the assumption that providing access to an improved diagnostic will result in finding more cases is as yet unverified. 4.3. Achievements and challenges in improving case finding Findings Screening of Outpatients is generally implemented well in low caseload facilities and poorly in high caseload facilities (High impact intervention). CTB’s support for this area was part of its Hospital Initiative, which aimed to improve the full range of TB services in order to improve case finding and treatment outcomes in the 210 hospitals across CTB areas. At the implementation level, CTB zonal staff provided on-site sensitization for facility staff. Observations of patient-provider interactions and interviews with clinicians during field visits showed the practice of symptom-based screening of all OPD patients, as prescribed in national guidelines, to be highly variable in adult OPDs and under-five clinics. In all the health centers visited across all regions and in one CoE, OPD patients were screened and the outcome recorded in the OPD register, but the quality of symptom-based screening was not consistent. In most hospitals visited across all regions and in one CoE, the practice was either inconsistent or non-existent. In two locations, clinicians reported that they “screen” patients with symptoms. The evaluation team also observed that for those patients presenting with cough of fewer than two weeks, and for those with chronic respiratory illness, the OPD clinicians did not provide a clear follow-up plan. The team did not find registers in family planning and antenatal care clinics capturing the results of the screening. The reported turnover of OPD clinicians is 33 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity high. In most facilities that the evaluation team visited, the medical directors were aware of the work of CTB but clinicians had not participated in orientation or training related to TB. OPD registers in these facilities are filled at the end of the day by nurses, using details from patient cards, which typically do not contain information on the screening outcome. This is the data that gets compiled as the OPD abstract, which is, then used by HMIS staff of the facility to report data on screening. CTB staff compile data (for the SOC tool) from the OPD abstract and not necessarily by observing the OPD processes. Contact investigation (CI) of index cases (High impact intervention) is now a routine function, due to extensive support from CTB.; CTB enhanced the support for CI that had begun in HEAL TB. It supported the development of SOPs for CI and treatment of LTBI, updated national guidelines to include CI and IPT, piloted the CI and IPT registers for approval by the NTP, and distributed these tools to all facilities. Two indicators related to CI and IPT have also been included in the national HMIS. CTB included CI and IPT in its mentorship of program and facility staff and tracked these through the SOC tool. The evaluation team found CI taking place in all facilities visited. Eligibility for CI appears to be what the SOP describes as priority index cases (those with smear-positive pulmonary TB) and not all index cases, but they were documented as index cases. Documentation was poor in most facilities: names of contacts were entered only for some priority index cases, and the outcome of screening them every quarter was not entered in the registers. The evaluation team did not directly observe the CI process. About half the estimated proportion of presumptives have been identified among those screened, and this is likely due to the low quality of screening. CTB also supported retrospective CI in regions not covered by HEAL TB, which were expected to have had low coverage of prospective CI. This was a one-time activity, aiming to cover all index cases from the preceding three years, and implemented through the sub-grant with REACH Ethiopia. The support included three woreda-level staff and one zonal staff dedicated for CI, per diems for HEWs and an initial orientation with community leaders and HEWs in three woredas in SNNPR. CTB also supported prospective CI of DR TB cases in all TICs. Data from TICs in CTB areas for 75 percent of enrolled DR TB patients were included for CI and 70 percent of the estimated household contacts were screened for TB. The outcomes for prospective and retrospective CI as given in Table 8 show that case notification rates (CNR) are much higher than national CNR of 115 per 100,000 population.8 Table 8: Outcomes of CI, CTB areas (regions and city administrations) CI approach # screened for TB # presumptives tested # cases identified CNR per 100,000 Prospective, CTB areas, Jul ’17 - Sep ‘18 129,369 2,444 546 422 Retrospective, 2 zones 7,144 1,387 178 2,491 Prospective, DR TB cases, CTB areas, 2017- ‘18 1,466 78 16 1,091 Source: CTB summary data Active case finding in communities (High impact intervention) is poorly implemented, even in woredas receiving intensive support from CTB. CTB’s level of effort for all community TB care interventions was low, to begin with. When community TB care became prioritized as a high impact intervention in Year 4, CTB supported the development of operational guidelines for the 8 From national HMIS data for 2017- ‘18 34 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity ENGAGE TB approach for engaging communities and CSOs in TB care and control and organized training of trainers at national and regional levels to train HEWs. CTB selected 100 poor performing PHCUs for intensified support, which included sensitizing woreda TB officers, and supporting reviews with HEWs. There was no direct support for training, mentoring or supervising HEWs. The evaluation team found that in PHCUs receiving intensified support, the HEWs had not been trained in TB in the past four years. HPs are given annual targets for the number of presumptive cases that HEWs are expected to identify during their routine household visits, using national incidence estimates. One HP met only ten percent of this target last year, and in another, the HEW reported that only 15 percent of the Women’s Development Army conducted sensitization meetings in communities. HEWs in all PHCUs visited reported having competing priorities such as maternal and child health and immunizations. The TB screening register at the HP had records of household members screened and the outcomes, but there was no verification or assessment of the coverage of households in the catchment area. Nor was there any report of assessing why the targets given to the HEWs were not reached. Involvement of CSOs has been minimal. One CSO interviewed by the evaluation team lamented the lack of a framework for engaging CSOs and the heavy emphasis on strengthening health systems. Another CSO was grateful for the opportunities given by CTB to conduct awareness activities in public areas such as schools, as well as counseling MDR TB patients. CTB successfully supported mass screening among key populations by regional TB program teams. CTB support for mass screening in mining areas, urban settlements, shelter homes and among health workers resulted in yields up to ten times the national average, as shown in Table 9. The yield in prisons is not as high, presumably because the prisons also conduct routine entry and exit screening. Table 9: Outcomes of mass screening among KPs Key population # screened for TB # presumptives tested # cases identified CNR per 100,000 Workers in mines, 2 zones, Jul '16- Sep '18 50,586 5,370 585 1,156 Households in mining areas, Jul '16- Sep '18 6,729 687 59 877 44 prisons, 2017- ‘18 64,618 2,114 55 85 1 urban settlement, April – June 2018 3,500 90 6 171 2 shelter houses, 2016- ‘17 2,258 357 23 1,019 Health workers, hospitals in Addis Ababa and Dire Dawa, January￾March 2018 385 35 5 1,299 Source: CTB summary data Screening PLHIVs for TB has not improved and CTB support has been minimal. The evaluation team found that the HIV teams in facilities visited were not included in CTB interventions such as training, onsite support and program reviews. Familiarity with CTB was low among them. Joint reviews by TB and HIV program staff were not taking place in most facilities. There is low participation of HIV staff in TB TWG in national and regional levels, even though this is part of the Terms of Reference for TB TWG. The national TB TWG reports that efforts to ensure the participation of the national HIV program, the HIV/acquired immune deficiency syndrome (AIDS) Prevention and Control Organization (HAPCO) in TWG meetings have not been successful. 35 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity Childhood TB: CTB provided a subgrant to the Ethiopian Pediatric Society to develop and test a diagnostic algorithm for detecting TB in children. EPS piloted this in 30 facilities in Addis Ababa. Results from the pilot are not available and it is yet to be assessed for scale up. Perspectives of patients on care-seeking and diagnosis: All 12 patients on TB treatment who were interviewed by the evaluation team had symptoms lasting two weeks to four months and had visited two to four facilities before they were diagnosed, incurring expenses ranging from USD 4 to USD 266 in the process. Most sought care initially at a private facility, preferring these over the crowded public facilities. However, none of the private facilities they attended suspected TB, even after the patient himself suggested that to the private provider in one instance. This patient had observed a relative who had TB and learned from that experience. One patient reported a delay of four days in receiving the results of his sputum test. There was no significant difference between the experiences and perspectives of men and women patients or across sites. DS TB case notification in CTB areas has been declining over the past four years. CNR have been declining over the past four years, both in CTB areas and in the rest of the country, as shown in Figure 7. CTB regions included in Figure 7 included Oromia and Amhara for all years, and therefore the CNR in CTB regions reflects the combined effect of HEAL TB and CTB support over the first two years shown. Figure 7: DS TB case notification trend, national Source: Global TB Report 2018 (Incidence), HMIS (Number of cases) and the Ethiopia Central Statistical Agency (Population estimates) The CNR given above represents about 60 percent of all the estimated annual number of cases in the country for the respective years. The trend in case detection rate, or the proportion of estimated cases detected, is given in Figure 8. 207 192 177 164 151 135 125 115 155 137 127 114 0 50 100 150 200 250 2014-15 2015-16 2016-17 2017-18 Incidence rate CNR National CNR CTB regions Linear (CNR CTB regions ) 36 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity Figure 8: Trend in case detection rate, national Source: NTP presentation during TB program review, March 2019 DR TB case notification has remained low, but estimates are likely to be revised. Based on current estimates, only about a quarter of the estimated DR TB cases are being identified and put on treatment, as shown in Figure 9. These estimates will be revised once results from the recently concluded drug resistance survey become available. One of the STTAs recommended analysis of the pathway from presumptive DR TB to Xpert testing, documentation of DR TB and initiation of treatment. This was not carried out due to lack of documentation. In order to enhance DR TB case finding, CTB set a target of testing 50 percent new cases and 100 percent retreatment cases with Xpert. Service data verified and triangulated by CTB show that 80 percent of retreatment cases were tested with Xpert in July-Dec 2018 in CTB areas, while there is no data available for new cases. Figure 9: DR TB cases notified, against targets and estimates, national Source: Notified: CTB summary data, verified with TICs; Targets & Estimates: NSP 37 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity Conclusions Screening for TB symptoms at OPDs is implemented to varying levels. Generally, they are done well in health centers visited and poorly to very poorly at hospitals, the very places targeted by the CTB hospital initiative for improving active case finding. Among the two COEs, one of them implemented this exceedingly well, and the other, very poorly. Thus, potential symptomatics that come to facilities continue to be missed in the very facilities where they are expected to be present in large numbers. The overall SOC data on a number of OPD patients screened for TB is suspect. Hence the reported low yield from screening is because the denominator is an overestimate. The variable quality of screening could be another cause for the low yield from facility screening. CTB has made a significant contribution to improving CI, in the form of training, recording/reporting tools, and was part of the effort to include this as an indicator in the national system. Thus, CI has become part of the routine package of care. A review of facility records indicates the need to improve the quality of screening (to increase the yield of presumptives) and the quality of documentation. Prospective screening yielded over three times that of the national rates. CTB identified community TB care as a high impact intervention and worked intensively in 100 woredas. However, the scope and intensity of the intervention have been very low, especially in light of lessons learned from REACH TB and HEAL TB mid-term evaluation. Even in woredas with intensive support, the coverage of households by HEWs (for TB screening) is not verified and the yield of presumptives and cases has been low. A higher geographic scale is only possible through the involvement of local CSOs and CBOs and patient groups which are restricted in this context. Patients continue to prefer private facilities and incur high out-of-pocket expenditure from treatment shopping. Diagnostic delays are significant. There is a low index of suspicion for TB among private providers. Patients had ways of acquiring knowledge about TB that warrant in-depth study and utilization. The proportion of estimated cases detected and put on treatment during CTB’s lifetime has remained stagnant, at 60 to 66 percent and has remained thus since 2007. It is also likely that CTB support has enabled the CDR to be sustained at this level, without which, the rate could have dropped further. Only a quarter of estimated DR TB cases were identified and put on treatment during CTB’s lifetime. There is no coherent mechanism to compile data on the proportion of notified new and re-treatment patients who had a DST. Analysis of current case-finding strategies and options for improvement Based on assessment findings from the evaluation and a review of literature, the evaluation team presents the following discussion on potential case finding strategies and their relative merits. Table 10 at the end of this section provides a summary of the discussion. Active Case Finding in facilities: Care-seeking at public facilities (for any reason), is lower than the WHO norm of at least two OPD visits per person per year but has been increasing over the years. In 2017, there were 0.6 OPD visits per person per year or 60,725,801 visits.9 Assuming three percent of them will have a cough of over two weeks, the screening activity should result in the identification of 182,274 presumptives every year. This is a strong reason to further strengthen the screening of all patients at all OPDs (adult, U5, antenatal, family planning, HIV testing, and chronic care clinics) and inpatient wards. This is weakly implemented at present in adult and U5 OPDs, especially in hospitals visited by the evaluation team. Screening needs to be routine and has to be documented under the supervision of the clinician. Those 9 FMoH. Health and health related indicators, 2017 38 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity presenting with cough of fewer than two weeks should be given a clear follow-up plan, which is also documented in their treatment cards. Findings from the OR on missed cases in inpatient wards of hospitals emphasize the potential of this activity. Direct observations and interaction with OPD and inpatient clinicians by zonal and woreda TB staff are likely to help assess gaps and plan responsive action. Regular review of findings from these observations, and the OPD screening results, at facility management meetings; improving data quality; orienting newly appointed clinicians on screening for TB; inclusion of this activity in their performance assessment; and critical review of screening data every quarter are potential actions that could lead to better performance. A review of available literature showed several good practices carried out in similar contexts to improve routine screening for TB in OPDs. An intervention study conducted in Pakistan to evaluate the outcome of systematic screening in private facilities deployed and trained lay workers to carry out screening and obtain sputum samples and paid them a monthly salary10. The intervention achieved a case notification rate of 190. Another intervention study conducted in high volume facilities in Ghana, in 2013 trained OPD staff and produced SOPs to guide the screening process11. Yet another intervention study conducted in Uganda in 2015-’16 by the USAID Applying Science to Strengthen and Improve Systems (ASSIST) project provided onsite coaching for facility teams to review their performance, identify reasons for low levels of screening and come up with innovative solutions. Some of the solutions implemented as part of their interventions were: including the outcome of TB screening (as codes) alongside the diagnosis, sensitizing health workers at all entry points, assigning staff to oversee screening at OPDs, daily review of OPD registers12. These are excellent practices that can be adapted and tested for their effectiveness in the Ethiopian health systems context. CI for index cases: While the yield is high with the current level of implementation, improving the coverage of index cases, reaching all contacts, and U5 contacts, in particular, repeating the screening every quarter for two years and improving the quality of screening are likely to further improve the yield. A meta-analysis of studies on CI found a strong association between the proportion of TB cases found through contact screening and increased coverage of index cases, with an odds ratio of 1.6 for each ten percent increase in coverage.13 Screening among KPs: The high yield from the mass screening of mining workers combined with poor access to routine services makes for a compelling case to continue twice-yearly screening activity among this mobile population. Mass screening in shelter homes has also given a high yield, and it will be useful to assess and address barriers to care-seeking. Screening PLHIVs for TB: With 73 percent of all PLHIVs enrolled in care,14 it can be expected that 10,950, out of the 15,000 who 10 Creswell J, Khowaja S, Codlin A, et al. An evaluation of systematic tuberculosis screening at private facilities in Karachi, Pakistan. PLoS One. 2014;9(4):e93858. Published 2014 Apr 4. doi:10.1371/journal.pone.0093858 11 Ohene SA, Bonsu F, Hanson-Nortey NN, et al. Provider initiated tuberculosis case finding in outpatient departments of health care facilities in Ghana: yield by screening strategy and target group. BMC Infect Dis. 2017;17(1):739. Published 2017 Dec 1. doi:10.1186/s12879-017-2843-5 12 Nakibuuka S, Kisamba H, Karamagi E. 2018. Improving TB screening at 9 TB Diagnostic Treatment Units (DTU): Tested Changes and guidance from Uganda. Published by the USAID ASSIST Project. Chevy Chase, MD: University Research Co., LLC (URC). 13 Blok L, Sahu S, Creswell J, Alba S, Stevens R, et al. (2015) Comparative Meta-Analysis of Tuberculosis Contact Investigation Interventions in Eleven High Burden Countries. PLOS ONE 10(3): e0119822. https://doi.org/10.1371/journal.pone.0119822 14 UNAIDS AIDSinfo, http://aidsinfo.unaids.org/ accessed March 23, 2019 “In remote (pastoralist) areas, we see children with neck abscesses. It (TB) is likely more prevalent there (in pastoralist areas) than we think.” - Senior CTB staff 39 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity are estimated to develop TB every year are already in antiretroviral therapy (ART) care. Service records show that about 8,000 are already on TB treatment, and hence, the remaining 2,950 PLHIVs are in ART care, are co-infected with TB and are not detected every year. The remaining 4,000 co-infected are likely not in the system. Thus, about 7,000 cases are likely missed every year. Improved collaboration with the HIV program at TWG, zonal, woreda and facility levels, including better access to data; joint review and joint supportive supervision are likely to improve the screening of PLHIVs already in ART clinics. Community-based case finding: Routine, year-round screening does not seem to be a priority for HEWs, and its coverage is not monitored. There is no feedback or corrective action when the targets are not met, and hence this does not receive much attention by the HEW. Other areas such as family planning or immunization activities are prioritized, in the absence of any push for prioritizing TB screening during routine household visits. These appear to be some of the reasons for the weak implementation of community-based case finding. Several case-finding strategies have been found to be effective in other contexts. In a study in Nepal using mobile vans equipped with GeneXpert, 19 percent of those screened were presumptives and two percent of presumptives had TB.15 A cluster-randomized trial in a high HIV prevalence context in Zimbabwe resulted in a 4.6 percent yield using mobile vans in communities and three percent yield using door-to-door visits.16 It will also be useful to learn from the mobile van experience of the Korean International Cooperation Agency (KOICA) and REACH Ethiopia’s experience in door-to-door screening. A qualitative study that looked at barriers to implementing active case finding strategies in South Africa in 201717 found that people had low knowledge of TB symptoms and mistook the symptoms for that of the flu. The study also found a lower preference for targeting certain households within communities for screening, but higher acceptance of screening all households within the community. These could be due to the association of TB with HIV in that setting. Similar, well-designed studies such as cluster￾randomized designs will help in better understanding of the potential yield from various approaches, and the feasibility (including cost-benefit) of each. Qualitative studies are also helpful in providing insight into people’s preferences, and barriers to seeking care, and any social or gender-based inequities in access. Prior to scaling up any community TB care approach, it will be important to carefully evaluate the operational feasibility of the approach, including careful costing. 15 Khanal, S, Baral, S, Shrestha, P et al. (7 more authors) (2016) Yield of intensified tuberculosis case-finding activities using Xpert® MTB/RIF among risk groups in Nepal. Public Health Action, 6 (2). pp. 136-141. ISSN 2220- 8372 https://doi.org/10.5588/pha.16.0015 16 Corbett EL et al. Comparison of two active case-finding strategies for community-based diagnosis of symptomatic smear-positive tuberculosis and control of infectious tuberculosis in Harare, Zimbabwe (DETECTB): a cluster-randomised trial. Lancet 2010; 376: 1244–53 DOI:10.1016/S0140-6736(10)61425- 17 Kerrigan et al. Improving active case finding for tuberculosis in South Africa: informing innovative implementation approaches in the context of the Kharitode trial through formative research. Health Research Policy and Systems (2017) 15:42 DOI 10.1186/s12961-017-0206-8 40 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity Table 10: Summary of discussion on case-finding strategies Case finding strategy Potential for contribution to finding missed cases Potential improvements in implementation Active case finding in facilities High Direct observation and feedback Mandatory orientation and performance measure for clinicians Regular review by facility management Data quality improvement; sex-disaggregated data Critical assessment of data Prospective CI for index DS TB and DR TB cases High Improved coverage of index cases Focus on quality of screening (to yield expected numbers of presumptives) Complete documentation and follow up of quarterly screening Sex-disaggregated data Screening KPs Yield among mining workers and shelter houses high Support twice-yearly mass screening at mines and shelter homes; routine screening at prisons Include compilation of sex-disaggregated data Screening PLHIVs in ART care High Joint planning, review at TWG, zonal, woreda levels Joint supervision to ART clinics: direct observation and feedback Include ART clinic staff in TB training Sharing of data between the two programs at all levels Community case finding Evidence from other contexts indicates high potential for contribution In-depth study of barriers and enablers to care￾seeking Learn from KOICA and REACH Ethiopia experiences 4.4. Achievements and challenges in supporting improving case management Findings CTB support has contributed to improvements in the quality of care for DS TB (high impact intervention) and maintenance of high treatment success rates. CTB helped strengthen referral networks between hospitals and health centers through its hospital initiative and developed regional directories of contact persons in all facilities. CTB also provided mentorship and monitoring using the SOC tool. KIIs of facility and program staff and CTB teams revealed that establishing these networks have helped streamline referrals of patients from hospitals to health centers for DOT, and for referring them back to the hospitals for monitoring of treatment. The evaluation team found facility DOT happening even during weekends and public holidays, with the facility management paying overtime for the TB clinic staff. Community DOT through HEWs and DOT with treatment supporter were also widely practiced in facilities visited in Oromia, Amhara, Tigray, and Gambella. However, in facilities visited in SNNPR and Addis Ababa, the drugs were being given to patients without an accompanying treatment supporter. Exit interviews (n=12) showed that patients on DOT were knowledgeable about infection control practices, and all of them were highly satisfied with the care received after diagnosis. All patients who were interviewed were on facility DOT. 41 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity The treatment success rate (TSR)18 has been consistently above 90 percent throughout CTB years as shown in Figure 10, while the global average has been 82 percent19. The proportion of patients cured has steadily increased over the years. Figure 10: Treatment outcomes, bacteriologically confirmed DS TB patients, national Source: HMIS CTB supported the expansion of high-quality care for DR TB (high impact intervention) and improved programmatic management of DR TB, leading to treatment success rates well above the global average. CTB supported two international training events in country, cascaded these training events to entire MDR TB case management teams and also supported several rounds of gap-filling training in programmatic management of drug-resistant TB (PMDT), new drugs and short treatment regimen, and drug safety monitoring and management. CTB has seconded a clinician to ALERT, supervised and mentored clinicians and TIC teams across all its locations and supported joint supervision along with government counterparts to TICs with the aim of transferring skills in clinical mentorship. KIIs with national and regional program staff and TIC clinicians attest to the high quality of the international training conducted in-country and the mentorship provided by CTB. CTB also provided office infrastructure and medical equipment (such as audiometry) for patient monitoring, office equipment, supplies and running costs for all 19 TICs. CTB also reimbursed all DR TB patients their costs related to lab tests and imaging done from private laboratories and provided socio￾economic support (house rent, travel, food) after discharge. This supplements the government’s financial support for the patients during the intensive phase. CTB introduced ISTAT, a reliable, user-friendly point-of-care monitoring of electrolytes, blood counts and renal function for DR TB patients in ten TICs. This is more efficient than centralized testing at the International Clinical Laboratory. These machines are also being used for patients in other units in these 18Treatment success rate = treatment completion rate + cure rate 19 WHO Global TB Report, 2018 42 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity facilities. However, this is a one-time contribution, and cartridges for ISTAT are expensive and have a short shelf life. CTB supported new drug introduction in two of the eight centers that currently provide this service. CTB’s financial support has enabled the optimal functioning of the Clinical Review Committee (CRC) in clinical review and management of patients on new drugs, including travel costs of patients. Treatment is ambulatory and is managed by TFCs. Supply of new drugs has been assured through other funding sources. CTB continues to advocate for setting up regional CRCs. Treatment success rate has been above 70 percent throughout CTB years, while the global average is 55 percent20. There has been no decline in mortality, which is 12 percent for the most recent cohort, which also has seen a relatively high number lost to follow-up (LTFU) at eight percent (56 patients), as depicted in Figure 11. About 29 percent of the most recent cohort has had unsuccessful outcomes. Figure 11: Treatment outcomes, DR TB patients, national Source: CTB summary data The evaluation team found seven out of nine TICs visited to be fully equipped, functional, and adherent to national standards for PMDT. The team found gaps in infrastructure and lack of equipment (such as audiometry and ECG), and a shortage of ancillary drugs in two TICs visited. Data on baseline SL-LPA for DR TB patients is incomplete. The team also found gaps in infection control in some locations. One TIC that does not yet have separate premises, receives diagnosed patients from other locations, and follows them up in the TB clinic, raising the risk of transmission to staff and other DS TB patients and their supporters. In another TIC, MDR extra-pulmonary cases requiring inpatient care were kept in close proximity to pulmonary MDR TB pulmonary patients. Most TICs visited are dependent on one trained clinician, and there is frequent turnover of clinicians trained in PMDT. Not all TICs function optimally. There are 7 TICs in Tigray region, but only one of them has full capacity to initiate and manage treatment, such as a dedicated internist for MDR TB and patient monitoring equipment. Therefore all MDR TB patients in the region are referred to this facility for inpatient care, most of whom have advanced disease and require a prolonged stay in the facility. Most patients prefer to 20 WHO Global TB Report, 2018 43 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity stay at the facility until the end of the intensive phase of treatment. Thus, the optimization of workload and improved patient convenience that was intended through TIC expansion has not taken place. In two regions, the evaluation team found that MDR TB patients were referred to TFCs without adequate preparation of the latter in terms of sensitizing facility management and staff. Because of this, these TFCs refuse to accept the patients for follow up, creating the potential for LTFU. CTB fixed this issue in one region by supporting TIC teams’ visits to TFCs to sensitize and mentor the latter. This is another example of dependence on CTB teams for problem-solving. There is no strategy developed yet for long-term support of patients with permanent serious adverse effects after treatment. In one region, the deteriorating security situation deterred follow up visits, leading to the death of one patient. CTB supported expansion of HIV screening of TB patients. CTB support in the form of training, mentorship and supervision using the SOC tool has led to increases in HIV testing of TB patients, and linkages with ART services. HMIS/DHIS2 data shows that HIV testing of TB patients increased from 79.6 percent to 95 percent. Close mentorship using findings from the SOC tool appears to have contributed to this improvement. However, as mentioned under 4.3 above, collaboration between TB and HIV programs is less than optimal. The evaluation team found in facilities in Amhara and Gambella regions that PLHIVs with symptoms suggestive of TB but with negative Xpert test were being put on a trial of antibiotics to rule out TB, instead of further testing. This is not in line with current TB HIV guidance. The TB HIV one-stop shop was piloted in 14 sites. It needs evidence on effective delivery of patient￾centered care, as well as feasibility and operational costs before it is considered for scale-up. Conclusions CTB’s mentorship and follow up using SOC and hospital initiative has contributed to improved adherence to DOT and hence improved treatment adherence. DOT is universally applied, especially in the intensive phase, and with some inconsistency in the continuation phase. CTB support has also led to LTFU reducing to negligible levels. These have together led to high treatment success rates and increasing cure rate. PMDT is arguably the most successful area of CTB support. With few exceptions, patient preparation, initiation, follow-up and socio-economic support have been excellent. Implementation of PMDT is fully consistent with standards. There is increasing enrollment with minimal delay after diagnosis. However, not having a large enough pool of highly trained MDR TB clinicians leaves the current staff with a large burden. HIV testing of TB patients and linkage to ART services is improving. However, collaboration with HIV is not optimal at TWG level and facilities. 4.5. Achievements and challenges in TB prevention activities Findings CTB supported improvements in managerial aspects of TB Infection control (TB IC) and use of personal protective equipment, but administrative controls continue to be poor. CTB supported all the three aspects of TB IC – managerial, administrative and environmental, through guideline development, training and mentorship using the SOC tool. CTB provided orientation for facility OPD in-charge and some clinicians in triage, OPD screening, and recording. The evaluation team found that directors of all health facilities visited were knowledgeable about the various aspects of TB IC. Infection prevention (IP) committees exist in all facilities, but at varying levels of functionality, ranging from highly functional to non-functional, in the facilities visited. 44 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity Screening of healthcare workers happen in most sites and includes all staff in health centers, but in hospitals, it is restricted to staff working in the TB clinic. Staff in all TICs visited report that they are screened for TB but not given voluntary counseling and testing services, as that might further deter new staff from joining. CTB engaged medical students in partnership with the Ethiopian Medical Students’ Association to sensitize them on their risk for acquiring TB infection and on TB IC measures. While coughers are identified at triage in most facilities visited, most hospitals did not have a mechanism to ensure that those identified as presumptives are seen on a priority basis at the OPD. Presumptive TB registers were either not present or not updated in hospitals. One of the two CoEs and three hospitals did not have a designated sputum collection area/enclosure, while other hospitals and all health centers visited had them. Many facilities have well-designed ones. The evaluation team found a lack of TB-related IEC material in many health facilities; while patients on DOT interviewed by the evaluation team were knowledgeable about TB IC actions and reported that they practice them, they also report poor knowledge about TB prior to being diagnosed. OPD waiting areas, consultation rooms, and TB clinics in hospitals visited did not have cross ventilation, while those in health centers did. TB and HIV clinics are well separated from each other in all locations visited. The use of N95 masks by health workers and visitors and surgical masks by patients in MDR TB Ward and TICs was uniformly high in all sites visited. CTB supported the management of LTBI (high impact intervention), and coverage of identified contacts is nearly universal. The SOP developed for CI with CTB support includes LTBI for children under five years of age (U5). Data from CTB areas shows that nine to ten percent of all contacts of index cases that were screened were U5s, as depicted in Figure 12. The Demographic and Health Survey of 2016 shows that 15 percent of the national population is in the age group of 0-4 years. Figure 12: U5s among all contacts of DS TB cases screened, CTB areas Source: CTB summary data National coverage for those U5 who were screened and were eligible for IPT increased from 60.3 percent in Oct-Dec 2016 to 72 percent in Jul-Sep 2018. Data for IPT coverage for CTB areas is shown in Figure 13. 45 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity Figure 13: Screening and IPT for U5s, CTB areas Source: CTB summary data The evaluation team found nearly universal coverage of IPT for identified, eligible U5 contacts. An exception was a large hospital that had no stock of isonicotinylhydrazide (INH). IPT coverage for PLHIVs was at 65.7 percent in July – September 2018. There is considerable variation in coverage between regions. There were challenges in obtaining buy-in from senior clinicians for LTBI standards of care, but with repeated sensitization, compliance has improved. Conclusions Managerial steps for TB IC and personal protection use are uniformly high. However, administrative controls are variable. Implementing TB IC in high volume facilities takes strong leadership and commitment at the level of hospital management to ensure that the chain of actions needed to ensure IC remains unbroken, in the midst of handling a large number of cases on a daily basis and with a high turnover of staff. Training of clinicians on TB IC has not kept pace with rapid turnover of this cadre, especially in high caseload facilities. IPT coverage among eligible U5s shows increasing trends, but the proportion of U5 among those screened is below expected numbers. High impact interventions at a glance Table 11 below captures the key achievements, challenges, and limitations observed during the one year of implementation of the high impact interventions. 46 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity Table 11: High impact interventions at a glance High impact intervention Achievement Challenges/Limitations Specimen transportation Expanded access for more facilities to Xpert and culture DST; includes HIV and other specimens; two RHBs likely to take over special van costs Expanding Xpert Xpert sites expanded, utilization increased Universal access to DST not assessed; case notification has not increased; CTB continues to provide TA and payments; support is centralized Culture DST SL-LPA services expanded; Accreditation progressing at various levels Supplies and processes funded by CTB; support is centralized Key populations Routine screening in prisons; High yield from mass screening in other KPs Mass screening funded by CTB PMDT High-quality management and patient care; cure rates increased Several processes and supplies still funded by CTB CI and LTBI Processes established. CI and IPT coverage increased Coverage of U5 children in screening is low Hospital initiative Patient-centered care for DS TB improved; cure rates increased; TB IC mechanisms established Case finding in hospitals low Community TB care Low level of effort for planned activities Activities initiated too late Evaluation Question 5: To what extent did CTB’s methodologies, interventions, and management set the stage for sustainability and ownership of project outcomes? Findings Political commitment to TB has increased over the past four years, but domestic funding has not. As discussed under evaluation question 4, CTB’s advocacy work through national and regional TWGs has resulted in the creation and staffing of government-paid positions such as advisors in EPHI, supply chain case teams in RHBs and TB focal persons in zones and woredas. Ethiopia has been an early adopter of WHO recommendations such as Xpert as the primary diagnostic test. These indicate a growing political commitment to TB control. The State Minister’s participation in national TB program reviews and TRAC conferences, improved capacity in national agencies to use integrated management systems and expansion of health infrastructure across the country, including the health extension program are additional indicators of a growing commitment to TB control. The NTP team has been proactive in engaging with the TWG to adopt global policies and standards for TB care and control. 47 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity Domestic funding for TB remains low at 11 percent (including health infrastructure) of the total cost of the NSP21; only 50 percent of the NSP is funded, and hence the NTP heavily dependent on donors. The evaluation team found that government staff at all levels anticipate and assume continued partner support. Strategic investments made by CTB are likely to continue after its time. CTB supported the following one-time interventions that are likely to have a lasting impact on the program. Outputs resulting from these interventions are: 1. Strengthening of national and regional TWGs and their TORs that serve as a government￾endorsed coordination mechanism to oversee and support the NTP in the implementation of the national strategic plan. 2. Development and dissemination of roadmaps, guidelines, SOPs, operational plans and training manuals, and recording and reporting tools, ensuring consistency with the International Standards for TB Care. This has provided the foundation for ongoing support, mentorship by the government and partners. 3. International expertise in all thematic areas that provided the needed direction to make diagnostic systems, PMDT, and TB IC functional. Actions taken in response to recommendations arising from the STTAs have led to the establishment of systems and processes for streamlined and high-quality care. 4. Improvements to DSM, particularly in quantification, review, and redistribution, and proficiency in the use of the QuanTB tool will ensure uninterrupted supplies. 5. Medical and office equipment such as power backup for GeneXpert systems, equipment for PMDT patient monitoring, and basic furnishing for TICs; set up and equipment for critical care units in the two CoEs. Although these are strategic investments, there is no mechanism in place yet, for replacement and maintenance of equipment. CTB established several competencies critical for program management and service delivery but continues to pay for these activities. Program and clinical staff at all levels report that the following processes and systems that CTB established the capacity for, are also largely paid for by CTB. Although these have been included in CTB’s list of interventions to be transitioned to the program, there has been no step in that direction. These activities are therefore likely to be discontinued after CTB, without external support: 1. Data-driven program management, mentorship and follow up 2. Diagnosis: Seconded staff; some costs related to EQA; all supplies related to Xpert assay and culture DST; costs related to maintenance of GeneXpert systems and culture DST 3. PMDT: Seconded staff; ancillary drugs and supplies related to patient monitoring; routine management activities (such as TFC-TIC meetings); travel costs for expert patients; costs related to CRC reviews of patients on new drugs 4. MDR TB patient care: reimbursement of imaging and lab tests, travel and living expenses 5. Community TB care: payments for HEWs for active case finding activities such as contact investigation 6. Training: top-up training in basic TB management and clinical services, and training related to PMDT 7. Evidence generation: Training in operations research, supporting OR design and implementation and annual TRAC conferences 21 The share of domestic funding for national TB programs in the nine (out of 30) high burden countries designated as low income countries ranges from 15 percent in Central African Republic to less than 2 percent in Zimbabwe, per the Global TB Report, 2018 48 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity Competencies have not been established for some critical activities. The evaluation team found that the following processes and capacities that have been integral to the achievements of CTB are still primarily managed by CTB staff. This is primarily because the costs involved came from CTB (national and all regions, or because there was not enough time to build the competencies (developing regions). These continue to be fully dependent on CTB. 1. Maintenance of equipment for culture DST and Xpert assay 2. Data-driven program management and mentorship in developing regions Some activities have already taken over by NTP. The following functions have already taken over by the program or RHBs: 1. EQA-related technical and financial support in some locations 2. Specimen transportation using cold chain van in Oromia and Amhara 3. Procuring new drugs for STR, through the support of other partners Other challenges to sustainability: Attrition is highest among national and regional program staff and clinical staff in facilities. Staff trained by CTB move out of the program and even the health system for various reasons. CTB has continued to provide top-up “gap-filling” training to compensate for this, but there is no plan as yet to continue this mechanism. CTB does not have a sustainability strategy. The transition plan, developed in Year 5 only outlines those areas for which support has to be taken over by other sources. Reports of meetings with RHBs indicate the willingness of RHBs to prioritize TB program reviews and supervision. KIIs with USAID, NTP, the Global Fund coordinator and other partners and a review of the financing of the NSP show that the NTP’s dependence on external funding is likely to continue and even increase, with a potential decline in the Global Fund’s contribution in the coming year. Conclusions The improvements in human resources for health, health infrastructure and ensuring an enabling environment for partners indicate growing government commitment to healthcare and specifically TB control. However, there has been no increase in domestic or alternate sources funding. Strengthening national and regional TWGs, developing and disseminating roadmaps, guidelines and tools for thematic areas, equipment, and infrastructure improvements are strategic investments of CTB that are likely to have a lasting impact on the program. CTB established competencies for program management including data-driven mentorship, PMDT, patient care, community TB care, and to some extent, capacity for evidence generation, but these continue to be paid for by CTB. There appears to be no plan for the government to take over CTB’s financial support for these routine activities that are critical for program management, diagnosis, care, and prevention. Competencies have not been established for the maintenance of equipment for culture DST and Xpert assay, and they are still primarily managed by CTB staff. Data-driven program management and mentorship have also not been established in developing regions. Technical and financial support for EQA, specimen transportation in Oromia and Amhara and procuring of new drugs for STR are likely to be taken over by NTP or other partners. “Support to FMoH and PFSA should continue. We are still resource constrained - manpower, awareness, skills, efficiency and finances. CTB should stay in the region for longer. We are still not able to go on our own.” - Regional PSA Director 49 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity Government staff at zones and woredas are dependent on CTB teams for problem-solving. Donor dependence is high. Program staff assume the continuation of donor funding and consider it indispensable for the program. Table 12: Status of key program areas at the start and end of CTB Program area Status before CTB (2014/15) Achievement during CTB (2017/18) Program management Limited frequency and intensity of training, supportive supervision and reviews CTB conducted several rounds of training, TA, supported the regions, zones to conduct regular data-led supportive supervision using SOC and regular reviews Diagnostic services EQA for microscopy Irregular system, limited coverage and highly centralized (at regional labs) A well-functioning decentralized system carried out by an expanded network of centers based out of hospitals, providing quality-assured microscopy service Access to diagnostics, and in particular, Xpert assay Access to Xpert limited to one per 1 million population Expanded to about one per 350,000 population; potentially providing access to universal DST Access to sputum microscopy and culture DST also have been expanded Labs with culture DST (number) Five Nine Case finding Reaching key populations Identified in NSP as a critical gap; technical and financial capacity to reach them was lacking Systems, including guidelines, SOPs and tools are now in place to reach these populations Finding missed cases of DS TB About a third of estimated cases were missed The gap persists, and CNR has remained at about 66 per 100,000 population; it is likely that CTB support has enabled the case notification rate from declining further Finding missed cases of DR TB Only about a quarter of estimated cases were diagnosed and put on treatment The gap persists at the same levels as baseline, based on current estimates of the burden of DR TB Case management Cure rate, DS TB (percent) 78 90 Number of TICs 10 19 Number of TFCs 183 212 Cure rate, DR TB (percent) 47 58 ART for TB HIV co￾infected 79.6 95 TB prevention IPT for eligible U5 (percent) 45 (2016) 72.3 (2018) 50 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity Source: HMIS, NTP reports, and CTB records Table 12 above gives a snapshot of the key gaps that existed at the start of CTB, and their status towards the end of the Activity. It is important to note that while CTB’s efforts were critical in achieving the improvements noted, all the changes cannot be fully attributed to CTB’s efforts alone. 51 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity OVERALL CONCLUSIONS This performance evaluation aimed to evaluate CTB performance by examining achievements, implementation challenges, and intervention effectiveness and relevance, and provide recommendations to guide decisions on future activity design and implementation of TB control activities. CTB’s design and strategic approach were informed by the gaps that needed to be addressed to reach NSP targets, and by the contours of the government health system. The technical assistance was comprehensive, relevant to needs, and contributed to improved quality of services and expanded access. CTB’s management approach was informed by its objectives and the technical approach to capacity building. Staff teams were competent to support all thematic areas. Workplans were well aligned with identified needs and NTP plans. The lack of a strategic approach to geographic scope resulted in areas with higher need receiving the least duration and intensity of support. The emphasis of CTB’s design was clearly on facility-based service delivery and strengthening of community-level interventions to find and treat missed cases were too little done too late. Thus, CTB’s technical assistance and management approach have supported CTB cooperative agreement objectives for the most part, with the exception of supporting community-based interventions for case finding. All of CTB implementation approaches used international standards and proven strategies. A key outcome of CTB interventions has been support to NTP to update national guidelines and algorithms to align with international standards. Current national guidelines are consistent with global standards for TB detection, diagnosis, and patient management. Much of the information generated by CTB is used to help achieve objectives and outcomes. CTB used high-quality, globally recognized parameters for quality and rolled out a mechanism to assess and assure quality through the SOC tool. It is evident that CTB generated and used some of the evidence it collected to re-focus its interventions. However, available data was not put to optimal use and potential opportunities for evidence-generation were missed. CTB’s main achievements and challenges in program management: CTB successfully continued and scaled up the efforts by earlier mechanisms to build capacity at zone and woreda levels for data-led program management – through training, mentorship, facilitating the networking of hospitals with health centers, peripheral labs with EQA centers and TFCs with TICs as well as supporting RHBs in revitalizing expansion of access to services to KPs. CTB support has resulted in improved DSM for PMDT drugs and supplies, but quantification issues persist. CTB’s role in problem-solving led to it directly managing these issues and this threatens the sustainability of the gains made. CTB’s main achievements and challenges in supporting improvements in diagnostic services: CTB has successfully supported all levels of the national laboratory system to improve its capacity to provide high-quality diagnostic services for TB, including DST, and the rollout of Xpert assay. CTB continues financial and technical support to most aspects of Xpert services, however, there is insufficient data to assess if improving access to a better diagnostic will improve case finding, which is the key assumption underpinning Xpert expansion. CTB’s main achievements and challenges in supporting improved case finding: The only activity among those prioritized for improving case finding that has given a high yield is mass screening among KPs. The others have not been implemented well. Notification of DS TB and DR TB cases continue to remain at baseline levels. CTB’s main achievements and challenges in supporting case management and prevention: CTB’s mentorship and follow up using the SOC tool and the hospital initiative have contributed to 52 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity improved treatment adherence and completion among DS TB and DR TB patients. CTB’s engagement of facility management and implementation TB IC measures remain sub-optimal in high volume facilities. The improvements in human resources for health, health infrastructure and ensuring an enabling environment for partners indicate growing government commitment to healthcare and specifically TB control. However, there has been no increase in domestic or alternate sources funding. Strengthening national and regional TWGs, developing and disseminating roadmaps, guidelines and tools for thematic areas, equipment, and infrastructure improvements are strategic investments of CTB that are likely to have a lasting impact on the program. Technical and financial support for EQA, specimen transportation in Oromia and Amhara and procuring of new drugs for STR are likely to be taken over by NTP or other partners. However, CTB continues to pay for the implementation of program management activities, for which it established competencies. There appears to be no plan for the government to take over CTB’s financial support for these routine activities that are critical for program management, diagnosis, care, and prevention. Competencies have not been established for the maintenance of equipment for culture DST and Xpert assay, and they are still primarily managed by CTB staff. Data-driven program management and mentorship have also not been established in developing regions. Overall, donor dependence is high, and program staff assume the continuation of donor funding as indispensable for the program. CTB’s methodologies, interventions, and management have set the stage for sustainability and ownership of some of its outcomes, while others continue to be dependent on external support. In conclusion, CTB has successfully supported improvements in data-led management of the TB program, particularly at zonal and woreda levels, improved access to the full range of TB diagnostic services, supported the systems needed for reaching KPs and for improving treatment adherence for all patients. CTB has provided excellent support for PMDT which is now adherent to international standards. However, the success of case finding strategies, and use of evidence to test key assumptions have been suboptimal. Several functions continue to be critically dependent on external financial support. 53 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity KEY LEARNINGS AND IMPLEMENTATION ADAPTATIONS The evaluation established key learning from CTB implementation and the adaptations that the Activity made to maintain relevance, efficiency, and effectiveness. This section outlines the learning and adaptations made, as well as the limitations of CTB in implementing the adaptations. Alignment with government priorities and plans CTBs’ design and strategic approach were built around meeting the gaps and needs identified in the NSP. One of CTB’s objectives was to strengthen existing platforms for TB, and this resulted in a strong and active TWG that advises the NTP on technical matters. Regular program reviews at national and regional levels, supported by CTB helped identify gaps and they fed into the subsequent planning cycle. Annual plans were made in full consultation with the NTP and partners. CTB has gathered a reputation for identifying gaps in program implementation and service delivery and adapting its plans to meet the need. Examples include the continual gap-filling training carried out to ensure that new staff are trained, and procuring supplies related to Xpert testing, to continue support to the NTP in rolling out the changed algorithm. CTB also collaborated with NTP and partners, working through the TWG, to adapt and update guidelines, manuals, and SOPs. NTP staff report that the technical competency of the CTB central team and their willingness to collaborate made these achievements possible. Reprogramming to expand geographic scope CTB demonstrated flexibility in expanding its coverage to regions that were not in its initial plan. The regional program team in Gambella reported improved planning, support to facilities through the use of the SOC tool, and better management of supplies, with two years of implementation. KIIs and observations in Amhara and Oromia revealed the benefit of continued partner support to establish regular program reviews, supportive supervision, logistics and expand services such as lab EQA and screening of KPs. All this learning was made possible because CTB was willing to revise its geographic scope and redo its workplans. However, a more strategic and bold approach that included developing regions earlier on would have avoided the need for repeated reprogramming and would have provided developing regions with longer implementation time. Role of USAID Mission The USAID Mission in Ethiopia bought into CTB’s design with the merger of HEAL TB in Year 2. Since then, the Mission has provided advice, direction, review, and support to the CTB team in ensuring that lessons from HEAL TB implementation are appropriated, developing regions are included, and priority is given to finding missed cases through the high-impact interventions. USAID has been the main supporter of TB care and control in the country since the year 2000. As there has been very high turnover of NTP staff – with three NTP managers in the past three years – the institutional memory of the local Mission supported CTB to build on the experience of previous mechanisms. The SOC tool, decentralizing EQA, specimen transportation, defining catchment area for hospitals, TICs, and EQA centers are some examples of practices that worked in HEAL TB that CTB took up and expanded to new regions. Best practices and innovation CTB adapted to emerging needs through innovative local solutions or adopting practices that worked well in other contexts. The following is a list of such solutions, and their feasibility for continued use and scale up: Solar lamps for microscopy: CTB procured lamps and solar panels for 25 diagnostic centers in Gambella that have a minimal power supply. This practical solution has helped continue microscopy services in this 54 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity region. The panels supply 6-7 hours of power. Thus, they are an effective adaptation that can be scaled up with minimal investment in the form of continued support for procuring spare parts. ISTAT: This is a handheld, point-of-care testing device was brought in by CTB as a potential solution for the multiple diagnostic needs of MDR TB patients. CTB says that the ten sites that use ISTAT were meant to be a demo of what is possible through this system so that the NTP can make an informed decision on supporting its scale up. Viber™ group: CTB successfully encouraged staff in TICs and TFCs in Addis Ababa to use this popular instant messaging platform to keep others in the group informed of patient referrals. The impact of its use has not been fully assessed, but staff report reductions in LTFU. There are no added costs, and it is feasible to be used in locations where internet connectivity and smartphone ownership are high. TB HIV one-stop shop: This is an effort to provide patient-centered care that enables TB and HIV patients to receive the full package of care at one place. The pilot, implemented in 14 facilities across six towns, used TB and HIV index cases to screen contacts for both HIV and TB, both in facilities and in clinics. The pilot has not yet been fully evaluated. Xpert stool test for children: This was carried out as an OR in sites in Addis Ababa, with the aim of understanding the effectiveness and feasibility of using an accessible specimen to diagnose TB in children but was suspended due to stock out of cartridges. A similar study conducted in Adama found that the test had a sensitivity of 96 percent and specificity of 99.5 percent but has not yet provided information on operational details and costs.22 ALERT blended learning site: CTB continued the infrastructure support for this center that was provided by HEAL TB and supported hardware, software, connectivity and staff time to set up this video conferencing facility. Training sessions and patient consultation for MDR TB between ALERT training center and woreda/facilities/TICs have taken place by linking to the government-sponsored WoredaNet. Virtual sessions are combined with pre-reading assignments and face-to-face sessions for skills building. CTB also supported virtual training sessions in audiometry by an international expert. The ALERT team plans to develop a fee-based model for extending this learning platform to HCWs pursuing credits under continuing professional development. Improving treatment adherence through mHealth: CTB supported this mHealth application (installed in the phones of the clinician and the MDR TB patient) to follow up on treatment. Costs include the agreement with the telecom company. This initiative is six months old and covers patients of St. Peter’s hospital at Addis Ababa. There is no evidence yet on treatment outcomes. Weaknesses in design and implementation This evaluation has brought out learning in areas of design and implementation that would be useful for future programming cycles: Unverified assumptions: The change in algorithm to Xpert assay as the primary diagnostic was intended to provide universal access to DST and improve case detection. The rollout of this changed algorithm was a massive exercise and this was ably supported by CTB. But there was no mechanism set up to analyze routine data and periodically verify the extent to which Xpert testing is being used as the primary diagnostic (proportion of presumptive cases that had an Xpert test). Therefore, the extent of this rollout is not yet known. The extent of change in access to DST (proportion of bacteriologically confirmed cases that had an Xpert test) has not been assessed, either. Most importantly, the expectation that access to a more sensitive and specific test will improve case detection rested on the assumption that adequate numbers of presumptives are being identified, but are not diagnosed/confirmed due to less accurate 22 Abdella et al. Evaluation of GeneXpert in detecting suspected PTB from stool sample for children <15 years, Adama, East Shoa Zone, Ethiopia, 2019 (Abstract in TRAC conference 2019). 55 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity tests. There was no mechanism set up to verify this assumption, either before or after the rollout of Xpert. The increasing trend in the proportion of bacteriologically confirmed cases is cited as a possible contribution of expanded access to Xpert, but it is not known if these cases would have been diagnosed clinically in the absence of access to Xpert. Low emphasis on understanding barriers to care-seeking: Much of the existing body of evidence around care seeking is from contexts with high HIV prevalence. Social norms, taboos and inequities in those contexts (where the public associate TB with HIV) could be very different and not readily applicable to low HIV prevalence contexts. Additionally, gender-related vulnerabilities to TB, both social and biological, need further study, in the context of stagnant case detection. As the major supporter of the NTP, CTB was best placed to invest in better understanding of demand. Lack of focus on all service delivery units in facilities: CTB’s support to improve services in facilities have been limited to the TB clinic, lab, the pharmacy, and the HMIS unit. The facility management was engaged to varying levels across the facilities visited. The evaluation team’s observations and KIIs showed that the OPD, HIV clinic, inpatient wards and biomedical engineers (in large hospitals) were not optimally engaged in appropriate thematic areas. It is ambitious to expect the same level of intense engagement of all of these units, through training and mentorship. Creative ways of continually engaging all units must be worked out, and zonal/regional teams involved in designing the engagement. 56 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity RECOMMENDATIONS CTB has successfully established program management capacity for TB care and control at national and sub-national levels and has set up processes and tools needed for high-quality facility-based TB services. The need to protect and further advance these gains in the past five years, the considerable level of financial and technical support that CTB continued to provide the NTP at all levels and the enduring issue of missed cases necessitate another cycle of assistance for TB care and control in the country. The following are priority investments for the future mechanism of donor support to the NTP: 1. Active case finding in high volume facilities: a. Support RHBs, ZHDs and Woreda offices to intensify their engagement with management teams of facilities. b. Facilitate the development of specific actions to engage facility staff at all outpatient and inpatient units to routinely carry out active case finding and document findings that are valid and verifiable. c. Support the plans with needed tools, orientation/training, supervision, review, and action based on data. Specifically, institute direct observations of the screening activity by facility supervisors, woreda and zonal staff and onsite mentoring of clinicians and incorporate the activity as a measure of staff performance. d. Review the yield of presumptives and confirmed cases after a time of quality-assured implementation against the level of effort and redesign the implementation strategy. 2. Maintaining and expanding PMDT: This is critically dependent on external support, in order to maintain high-quality service delivery and the steady enrollments of patients into care. a. Continue payments related to routine program management (such as TFC-TIC reviews), routine supplies (such as ancillary drugs), patient support costs, clinical reviews and top￾up training of staff teams, until such time that the government at national and regional levels is able to provide for them. These are critical for the continued functioning of the program. b. Assess all TICs for barriers to optimal functioning and build the needed capacity, including infrastructure, equipment, supplies, and training. 3. Active case finding in communities: There is a need for intentional, evidence-based and time￾bound engagement with careful review and management of results. a. Consider one of these two options for such intentional engagement: i. Intensified support for routine TB screening in the 100 woredas already identified by CTB: train all HEWs in basics of TB and TB screening protocol, develop a detailed supervision plan for woreda TB officers and the health extension program supervisors to ensure that at least 85 percent households are covered and that the quality of screening is maintained. The advantage of this approach is that it does not need much lead time as working relationships are already built in these woredas. ii. Conducting month-long nationwide campaigns, once or twice a year, aiming for a heightened focus on TB at all levels in the system. Work with regional, zonal and woreda TB teams to develop supervision plans to ensure high coverage of households and high quality of screening b. Whichever approach is selected, the NTP and the future mechanism should: i. Build in collection and compilation of verified numbers of households in the HP area, households covered with screening, along with data on presumptives and confirmed cases. 57 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity ii. Build in a mechanism for detailed documentation of challenges, responsive actions, and lessons learned at the health post, woreda, and zonal levels and for these to be compiled and processed as they move up the system iii. Conduct a detailed review of all data each quarter, and decide on scaling up or abandoning the effort based on the yield, cost-benefit, and operational challenges, after two or three quarters. c. Commission well-designed studies such as cluster-randomized designs that will help in better understanding of the potential yield from various approaches, and the feasibility (including cost-benefit) of each, and qualitative studies to gain insight into people’s preferences, and barriers to seeking care, and social or gender-based inequities in access. d. Continue support for twice-yearly mass screening in mining areas and urban shelter homes 4. Contact investigation: Support zones and woredas to follow up the coverage of household contacts, especially U5 contacts, in the frequency provided in the national guideline. Support the assessment of yield and its cost-effectiveness and titrate support accordingly. 5. Routine program management: There is a need to maintain the competencies already built in program and facility teams to generate and use high-quality service data, as new staff come in and to keep up with new evidence and guidance. This should be done through technical and financial support for training and supportive supervision and is required until such time when the NTP can take over these critical functions (please see below). Continue financial support for: a. National and regional TWGs to regularly meet for reviews and updates; setting DR TB related targets, once results from the drug resistance survey are available. b. Program reviews at national, regional and zonal levels, supportive supervision using the SOC tool and one-to-one meetings at woredas and facilities c. Functions related to TRAC, expanding research activities in regions and ensuring the priorities identified are included d. Training and mentorship for DSM, in the use of QuanTB and IPLS e. Diagnostic services: supplies, training, mentorship, and management costs for culture DST and EQA for microscopy 6. Continuing universal access to Xpert assay: Xpert has made universal drug susceptibility testing a possibility, and its operational challenges are part of this new reality. Provide resources for training, mentoring, supplies and external technical support, including but not limited to ensuring GxAlert functioning. 7. Expanding to developing regions: Titrate the level of support for human resources and institutional capacity with the available staffing and infrastructure, but with a clearly planned and executed sustainability strategy. Such support is especially needed to test and adapt strategies for case finding and patient-centered care in these regions. Include the design considerations listed below. The design of future cycles of support can benefit from learning gleaned from the implementation and evaluation of CTB. Future mechanisms should consider the following overarching issues in their design and implementation, for all thematic areas: 1. Program design: a. Clearly identify and articulate assumptions that underpin the program theory of change, and regularly review evidence for or against these assumptions, leading to redesign as appropriate b. Build in a timed exit plan and a communication strategy to all stakeholders about the nature and duration of support 58 Ethiopia Performance Monitoring and Evaluation Service Final Evaluation Report, Challenge Tuberculosis Activity c. For all program operations that are supported, develop a matrix of the various operational details and the source of current financial and technical support and come up with a clear plan (with timelines) to transfer responsibilities to the government at all levels. This is particularly needed for GeneXpert management. 2. Performance measurement, evidence generation, use, and accountability: a. Develop key performance measures that reflect the core functions of the support mechanism, rather than outcomes expected from the national program being supported. Review the performance of each team against these measures. Examples are measures of institutional capacity such as competencies of program staff to effectively use tools and guidelines, maintain equipment and independently manage supply systems. b. Revamp the collection, compilation, and reporting of adequately disaggregated data for KP activities, CI, and case finding in communities c. Require implementing partners to build in time for critical reflection on evidence, developing responsive action plans and holding staff accountable to those plans 3. Include the following technical interventions/support in the design: a. Specifically, consider improving collaboration with the national HIV program at all levels b. Consider facility-wide support rather than a sole focus on TB units c. Expand solar lamps for microscopy centers in areas with low power supply d. Link the ALERT blended learning center with payments-based continued professional development for government health staff e. Build in support for negotiations and planning for including TB-specific skills development in medical, nursing and laboratory technician courses f. Build in support for negotiations with large mining companies to collaborate in mass screening g. Support detailed planning and implementation to scaling up mHealth support to all MDR TB patients. h. Support lobbying for the provision of support from hospital management for ISTAT￾related supplies 4. The NTP has a strong team of staff to support the various thematic areas, with technical advice from the TWG. Key findings from the evaluation include continued ownership of the Xpert system by CTB and varying levels of adherence to national standards for service delivery. a. Political commitment to End TB targets need to translate into domestic efforts to close the current funding gap. Even a modest increase in domestic funding will go a long way to complement donor funding and technical support and take over critical functions as outlined in the earlier recommendations and prevent erosion of the gains made. The evaluation recommends that the NTP work with technical partners to develop economic and cost-effectiveness analyses to convince policymakers of the relative benefits of investing in TB control and translating the commitment to End TB targets into increased funding. b. The evaluation recommends that the NTP work proactively with the future funding mechanism in order to assume ownership of the functioning of GeneXpert machines and their optimal utilization. c. The evaluation also recommends that the NTP lead a multi-stakeholder team in assessing adherence to all national standards at service delivery level 59 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB ANNEXES 60 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB ANNEX I: EVALUATION STATEMENT OF WORK STATEMENT OF WORK FINAL PROJECT PERFORMANCE EVALUATION OF CHALLENGE TB ETHIOPIA I. PURPOSE OF THE EVALUATION The overriding purpose of this performance evaluation is to independently assess the overall performance of the CTB project—in terms of achievements and challenges faced during implementation—and provide insight into effectiveness and relevance of the intervention strategies that have been implemented. Further, the evaluation should generate evidence, and document lessons learned aimed at improving future TB program strategies to inform the design and implementation of similar projects in line with the USAID/Ethiopia Country Development Cooperation Strategy. The primary users of the evaluation results/findings will be USAID/Ethiopia, the Federal Ministry of Health, other GOE entities, and other donors to provide information on approaches to addressing TB. USAID’s evaluation policy encourages independent external evaluation to increase accountability, to inform those who develop programs and strategies, and to refine designs and introduce improvements into future efforts. In keeping with that aim, this evaluation will be conducted to evaluate the performance of the USAID-funded Challenge-TB activity being implemented by KNCV TB Foundation, Management Sciences for Health (MSH) and World Health Organization (WHO). II. SUMMARY INFORMATION Activity Name USAID/Challenge TB Prime Implementing partner KNCV Tuberculosis Foundation, MSH Sub partners Management Sciences for Health and WHO Cooperative Agreement AID-OAA-A-14-00029 Total Estimated Cost $10 million annual budget with $525 million global TEC amount Life of Activity October 14,2014- September 30,2019 Activity Geographic Regions National and selected regions (Tigray, Amhara, Oromia, B/Gumuz, Gambella, SNNP, Harari, Dire Dawa, and Addis Ababa) Mission DO Increased Utilization of Quality Health Services USAID Office Health Activity Manager Yared Kebede Haile Activity Funding Global Health III. BACKGROUND A. Problem and Theory of Change Ethiopia, with a projected population of 96 million in 2018, is administratively divided into nine Regional States and two City Administrations. The Regional States and City Administrations are further divided into 103 zones, 800 woredas (districts), and approximately 16,000 kebeles (each with a population of 5,000 on average). There is extensive decentralization of service delivery, with relatively autonomous regions. The health system of Ethiopia is organized in a three-tier system, composed of primary health care units (District Hospital, Health Centers and Health Posts), Zonal Hospitals and Referral/Regional Hospitals. The woreda is the basic administrative unit, and it is further divided into kebeles with at least 61 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB one health post in each kebele. In each of the administrative levels, there are Regional Bureaus, Zonal Departments and woreda offices to administer health and social services. The Health Extension Program, which is focused at the Kebele level, is implemented by health post based health extension workers who also conduct community outreach activities. Ethiopia continues to be among the 30 High TB, MDR TB, and TB/HIV burden countries. According to 2018 WHO Global TB Report, the incidence and mortality of TB are estimated to be 164 and 24 per 100,000 populations respectively. TB case notification out of estimated incidence was 68 percent indicating more than 30% of TB cases are missed. The incidence of MDR/ RR-TB is estimated to be 5.2 per 100,000 populations with low treatment coverage of 25%. In the same report, 2.7% of New TB cases and 14% of previously treated TB cases are estimated to have MDR/RR-TB with 42% and 100% access to rifampicin resistance testing respectively. Among the total 117,705 notified DS-TB Cases in 2017, 69% are pulmonary TB case, and only 58% of the total cases notified are bacteriologically confirmed. The treatment success rates of new DS-TB cases enrolled in 2016 and MDR/RR-TB cases enrolled in 2015 were 90% and 75% respectively, which is above the global average. The global report also estimated the TB/HIV co-infection rate to be 7% with 92% treatment ART treatment coverage of TB/HIV co-infected patients and 45% of newly enrolled PLHIV were put on TB Preventive Therapy (TPT). Both men and women are vulnerable in their young and productive ages of 20-30 years. Gender disparities in biological susceptibility and health-seeking behavior contribute to differences in vulnerabilities especially in the gender-related delay in TB diagnosis, treatment interruption, and gender￾based TB stigma and discrimination. Low detection rate and treatment for TB among women have been seen due to delay in accessing care, poor compliance to treatment, and belief in alternative treatment; in some settings, men may delay care seeking for TB more than women. From 2017 WHO TB report, the estimated number of incident TB cases in Ethiopia is slightly higher in males (94, 000) than in females (78, 000). TB, MDR-TB and TB/HIV still continue to inflict tremendous harm with morbidity and mortality in Ethiopia today. The Ministry Health of Ethiopia in partnership with USAID and other donors and partners has been focused on expanding TB services. The strategy to provide care in both public and private health facilities by engaging all care providers with the aim to improve case finding through community-based and facility-based health services including private health providers continues to be a focus. However, in Ethiopia, the comprehensive TB program support is pursued by the PHSP activity, not by Challenge TB Ethiopia. As of June 2018, all public hospitals and public health centers provide TB services, and community TB care is rolled out in most health posts with the role to provide presumptive TB referral and treatment follow-up services. In CTB supported regions a total of 3324 health facilities are equipped to provide TB diagnostic services. However, 793 of these health facilities currently lack laboratory technicians making nearly 25% of them non-diagnostic. Through the PHSP activity, PPM￾DOTS services started in 2006 and, currently, 546 private health facilities providing different TB service mix; 247 diagnostic and treatment service providers (full PPM-DOTS service, 32 TB diagnosis and referral service and 267 presumptive TB referral service. With the engagement of less than 3 percent of private health facilities, the PPM-DOTS initiative is contributing more than 10 percent of TB cases notified nationally. The federal ministry of health has contextually adapted global End TB strategies to ease the TB, MDR and TB/HIV burden. With more than 30 percent of DS TB and 75 percent of DR TB cases not detected or reported by the health system, the ministry's strategy is geared towards detecting the missed cases through the expansion of comprehensive and patient-centered TB care with the engagement of community health providers and other actors including the private sector. With this strategy, the ministry has prioritized key populations and settings highly affected and vulnerable to TB. In addition, TB prevention including management of latent TB is one of the strategies to end TB in the country. Health 62 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB system strengthening including monitoring and evaluation, drug supply management, evidence generation and human resource management are the key cross-cutting strategy to improve TB program performance. A. Activity Goal and Objectives The USAID/Challenge TB Ethiopia (CTB) Project is a 5-year global mechanism active since October 2014 and ending in September 2019. In Ethiopia, it is being implemented by KNCV TB Foundation as the in-country lead, along with Management Sciences for Health (MSH) and the World Health Organization (WHO). Additional partners who are in the global coalition but not active in-country in Ethiopia are the American Thoracic Society, FHI 360, and Interactive Research and Development (IRD). Challenge TB Ethiopia supports a comprehensive package of tuberculosis (TB) interventions. Challenge TB Ethiopia collaborates with The National TB Program (NTP), major national institutions (EPHI, AHRI, PSA and FMHACA), Ethiopian universities and other local institutions and helps the regional health bureaus (RHBs), zonal health departments (ZHDs), and woreda (district) primary health care units to take ownership of TB, TB/HIV, and multidrug-resistant TB (MDR-TB) program management. Through this assistance, the RHBs, ZHDs, and woreda (district) primary health care units have been fully engaged to ensure improved and sustainable program management capacity. Challenge TB Ethiopia focuses on increasing case notification and decentralization of TB services to communities through Health Extension Workers (HEW) and expanding access to additional health facilities (HF) in rural and urban areas. The project is also helping to strengthen Ethiopia’s health system by supporting the national laboratory network, program management teams of regional, zonal and woreda health offices and health facility staff capacity, woreda planning and improved drug supply management and TB infection prevention standards. Challenge TB Ethiopia covers more than 80% of the total population of Ethiopia. It implements activities at the national level and in four major regions (Tigray, Amhara, Oromia, SNNPR), three city administrations (Addis Ababa, Dire Dawa, and Harari) and two developing regions (Gambella and Benishangul Gumuz). The strategic framework of Challenge TB Ethiopia activities focus on the U.S. Government TB strategy, the WHO’s Global Post-2015 (The End TB) Strategy and Ethiopia’s Health Systems Transformation Plan (HSTP) Targets for Tuberculosis Prevention, Care and Control. Through the CTB activity, USAID/Ethiopia in collaboration with USAID/Washington has been implementing activities by ensuring that our investments are coordinated with the investments of other USAID TB activities (HEAL TB and PHSP), USAID Health Systems Strengthening activities (e.g., HRH2030 and HSFR/HFG (Health Care Financing), PEPFAR TB/HIV activities, national health programs, and the Global Fund to generate efficiencies and enhance the combined efforts for impact and sustainability. USAID through Challenge TB Ethiopia activities intensified its efforts to identify and invest in programs and policies that have the greatest potential to end the TB epidemic. Challenge TB Ethiopia has been implementing activities through leveraging interagency and partners support and strong country ownership using innovative approaches and evidence-based practices and policies with strongly integrated implementation research/evidence generation for TB prevention, care, and treatment. With the U.S. global strategy, WHO’s End TB Strategy, and the GoE‘s Health Sector Transformation Plan, USAID together with the GOE worked to achieve a paradigm shift in TB program activities and strategies to meet the Global End TB targets set for 2025. To achieve this ambitious target, the USAID funded TB program activities focused on the following primary objectives: 1. Improve access to high-quality patient-centered TB, DR-TB and TB/HIV services 2. Prevent tuberculosis transmission and disease progression 63 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB 3. Strengthen health system platforms for the provision of standard and quality TB service 4. Accelerate research and innovation to better inform programs with better tools and evidence with a focus on implementation/operational research to generate evidence for program improvement and policy guidance I. EVALUATION QUESTIONS This evaluation will answer the following evaluation questions that are closely linked to the activities’ development hypothesis depicted in the Results Framework (see Annex C. Statements in parenthesis are included in the questions to help clarify the types of information that could be included in an analysis to respond to the main question. 1. To what extent has CTB’s technical assistance to Government at different levels and Activity management approach supported the objectives of Challenge TB cooperative agreement?  What and how was the capacity building support delivered at different levels of the health system  What are the perceptions of the stakeholders on the quality of the technical assistance to FMHO, RHBs, ZHDs, Woreda Health Offices and Health facilities from Challenge￾TB?  What are the main achievements and challenges of CTB in at the national, sub-national and facility levels in strengthening the local capacity of the TB system to deliver strategic and effective TB programming and TB services?  Is the management structure of CTB and of sub-partners--including the type and levels of supportive supervision and coordination, optimal to ensure that Challenge-TB performance is meeting the objectives of the cooperative agreement?  Are the staff competencies at all levels of the Challenge-TB activity (Technical expertise, administration, trainers, etc.) sufficient in order to meet the objectives of the cooperative agreement?  What opportunities and challenges were experienced by the Activity in relation to: o o o Collaboration and coordination with stakeholders such as the Global Fund, Government of Ethiopia and other TB implementing partners (HEAL TB, PHSP, PEPFAR programs) Implementation Management 2. To what extent did Challenge TB implementation approaches use international standards and proven strategies?  Are Challenge-TB implementation strategies and core approaches based on international standards?  Did Challenge-TB apply internationally recognized best practices? How did the Activity incorporate context-specific policies and strategies?  How was quality measured in the key components (TB case finding including Community TB care, diagnostics, patient management, follow-up, drug supply management, data recording, and reporting) of Challenge-TB interventions such as TB, TB/HIV, and MDR TB services?  Has Challenge TB improved National Standards of TB care and treatment; if so, how? 3. Has the information generated by the Activity been used to support the achievement of objectives and outcomes (i.e., to make adaptations during implementation) and, if so, how? (Information Generation and Use)  Are performance indicators and feedback mechanisms applied to ensure effective program learning and adaptation? What evidence exists on the utilization of the 64 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB performance data and feedback for continuous improvement of the Activity performance?  Is monitoring and evaluation implementation optimized to ensure that the system generates evidence on implementation progress and key lessons (examine also as to who generates the information/evidence, from where is the information generated and was it comprehensive enough to inform all the activity objectives)?  Does the M&E system document both successes and failures at National, Regional, Zonal, and Woreda levels? 4. What are the main achievements and challenges of CTB related to:  Improving TB program management capacity and sustainably  Increasing case notification  Decentralizing TB services to communities including key affected/vulnerable populations and settings  Expanding access to health facilities in rural and urban areas  Improving quality of care for TB, TB/HIV and DRTB cases  Implementation of TB prevention activities including management of latent TB  Strengthening the national laboratory network  Generating and using evidence 5. To what extent are the activity’s methodologies, interventions, and management setting the stage for future sustainability, and ownership of project outputs and outcomes:  Have Challenge-TB capacity building initiatives improved the sustainability of the project's activities and continued results in terms of political commitment, overall ownership, institutional strength (National, Regional, Zonal and Woreda health institutions) and human capacity built (health professionals) in managing TB Programs in the country?  Are CTB Ethiopia’s interventions designed so that it is feasible to scale them nationwide, within the existing health system?  Did CTB Ethiopia’s approach take into account, and successfully build upon, the broader changes in the country’s health system at different levels (National and sub-national)?  What lessons and best practices from CTB implementation (elements of the activity) could be standardized across other regions/kebeles of the country and why?  What are the main criteria for the allocation of funding to implementing partner?  What is the cost incurred to implement each expected result? Is the cost reasonable when compared with the achievements?  What support did the activity give to sub-partners on financial management? How did the activity monitor the financial management practices of the sub-partners? What improvements in financial management occurred among the sub-partners as a result of the activity support. The evaluation team must examine the extent to which gender issues are addressed across all evaluation questions, particularly in relation to making TB-related technical assistance and services accessible and equitable to both men and women. II. EVALUATION DESIGN AND METHODOLOGY A. Evaluation Design The evaluation team will be responsible for developing an evaluation strategy and methodologies that include a mix of qualitative and quantitative data collection and analysis and have to be discussed and finalized with the USAID team. The team should review the evaluation questions matrix presented in 65 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB this SOW and present a revised one with any other relevant additions showing the source of data, the method of data collection and also the tool to be used to answer each of the evaluation questions. The methodology will be presented as part of the draft work plan as outlined in the deliverables below and included in the final report. The evaluation team will have available for their analysis a variety of activity implementation documents, and reports. Methodology strengths and weaknesses, as well as measures taken to address those weaknesses, should be identified and discussed. All data collected and presented in the evaluation report must be disaggregated, as appropriate, by sex and geographic areas. The following section provides illustrative suggestions for evaluation design and methodology which the Contractor may take into consideration, or propose alternative methods with justification. B. Methodology (i) Data Collection and Sources The end of this project evaluation will mainly use desk review, analysis of existing quantitative data, and qualitative data collection methods. Data sources for this evaluation will include, but not be limited to, Challenge-TB agreement document, work plans, field trip, and STTA reports, periodic activity reports, health institutions at different operation areas, focus group discussions, key informants, service delivery point records, household/individuals selected from communities, etc. Qualitative inputs should be sourced from the mission, project staff, other donors and partners, government counterparts and beneficiaries at all levels of the health system. C. Data analysis plan While developing the work plan, the Contractor is also expected to provide data analysis plan; for example, what quantitative and qualitative analysis techniques will be used, how focus group responses will be documented and analyzed, etc. As this evaluation mainly collects qualitative data, the Contractor must employ an appropriate analysis tool for qualitative data in order to categorize, rank and rate the responses of the interviewees and discussants. Very insightful observations from interviewees and discussants should be quoted as appropriate to highlight findings. All person-level data must be disaggregated by sex as appropriate and differential outcomes on men and women sufficiently analyzed. The following table provides a summary of evaluation design and methodology and supplements the narrative section above. Evaluation Question Suggested Data Sources Suggested Data Collection Methods Data Analysis Methods 1. To what extent has CTB’s technical assistance to the Government at different levels and Activity management approach supported the objectives of Challenge TB cooperative agreement? ● CTB( central, regional and zonal) ● FMOH/EPHI, RHB/RRLs, ZHD, WoHO, HFs ● Other TB partners (PIH, GHC, GLRA) ● To be determined by the evaluation team. 66 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB ● USAID (Mission + WDC) 2. To what extent did Challenge TB implementation approaches use international standards and proven strategies? ● CTB ( central ,regional and Zonal) ● FMOH/EPHI, RHB/RRL, ZHD, WoHO, HFs ● Other TB partners (PIH, GHC, GLRA) ● USAID (Mission + WDC) ● To be determined by the evaluation team. 3. H has the information generated by the Activity been used to support the achievement of objectives and outcomes (i.e., to make adaptations during implementation); and if so, how? (Information Generation and Use) ● CTB ● FMHOH, TRAC, RHBs, ZHD, WoHO and HFs ● Other TB partners (PIH, GHC, GLRA) ● USAID (Mission + WDC) ● To be determined by the evaluation team. 4. What are the main achievements and challenges of CTB management and interventions? ● CTB ( central, regional & zonal) ● FMOH, RHBs/RRL, ZHD, WoHO and HF ● Other TB partners (PIH, GHC, GLRA) ● USAID (Mission + WDC) ● To be determined by the evaluation team. 5. To what extent are the activity’s methodologies, interventions, and management setting the stage for future sustainability, and ownership of project outputs and outcomes: ● FMOH, EPHI, RHB/RRL, ZHD, WoHo, HFs ● CTB ● Other regional partners (PHSP, WHO) ● USAID (Mission + WDC) ● To be determined by the evaluation team. III. DELIVERABLES AND REPORTING REQUIREMENTS 67 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB 1. Within 48 hours of the availability of the evaluation team in the Country, the evaluation team will have an in-brief meeting with USAID/Ethiopia’s Program Office and HAPN Office for introductions; presentation of the team’s understanding of the assignment and initial assumptions. Following this, the evaluation team shall present an evaluation work plan/evaluation design to USAID within twelve (12) working days of the initial introductory meeting. 2. Evaluation Work Plan/Inception Report: Within five working days following the in-brief presentation, the Contractor shall submit the evaluation work plan (evaluation inception report) to USAID/Ethiopia's Program Office and HAPN Office. This work plan/inception report will include: (a) the overall evaluation design, including the proposed methodology, data collection, and analysis plan, and data collection instruments; (b) a list of the team members and their primary contact details while in-country, including the e-mail address and mobile phone number for the team leader; and (c) the team’s proposed schedule for the evaluation;. USAID offices and relevant stakeholders are asked to take up to three working days to review and consolidate comments through the EPMES COR. Once the evaluation team receives the consolidated comments on the work plan/inception report, they are expected to return with a revised work plan/inception report within two working days. The revised work plan shall include the list of potential categories of interviewees and sites to be visited. USAID Offices send their final comments/say on the Contractor's re-submitted documents/work plan within two working days of receipts revised documents/work plan from the Contractor, and the Contractor proceeds accordingly. 3. Fieldwork Debrief: The Team or the Contractor is expected to provide the COR for EPMES and the Activity Manager for Challenge-TB Activity with periodic written briefings and feedback on the team's findings. If desired or necessary, weekly briefings by phone can be arranged with the Program Office and the HAPN Office to provide updates on field progress and any problems encountered. Immediately after the team's completion of the fieldwork, the team shall provide a debrief to USAID to discuss on and learn about field-level data collection experiences as well as the evaluation team members' preliminary impression on evaluation findings. 4. Final Exit Presentation (PowerPoint Presentation) to USAID and relevant partners that will include a summary of key findings and key conclusions as these relate to the evaluation's questions and recommendations to USAID. To be scheduled as agreed upon during the in-briefing, and before expats departure from Country. A copy of the PowerPoint file will be provided to the Program Office, at least three days before the final exit presentation day. The COR for EPMES shall compile comments from participants in this presentation and submit it to the Contractor for consideration during the preparation of the report. 5. Draft Evaluation Report: The content of the draft evaluation report is outlined in Annex A below, and all formatting shall be consistent with the USAID branding guidelines. The focus of the report is to answer the evaluation questions and may include factors the team considers to have a bearing on the objectives of the evaluation. Any such factors can be included in the report only after consultation with USAID. The Contractor will submit the draft evaluation to the Program Office within 10 working days after exit presentation and should incorporate comments made during the exit presentation. USAID's Program Office, HAPN Office, and other partners will have 15 working days to review and comment on the draft report, and the Program Office shall submit consolidated comments to the Contractor. The Contractor will then have 10 working days to make appropriate edits and revisions to the draft and re-submit the revised final draft report to USAID. The Program Office, HAPN Office, and other partners will have 10 working days after the submission of the second revised draft to again review and send any final comments. 6. Final Evaluation Report will incorporate final comments provided by the Program Office. The length of the final evaluation report should not be more than 45 pages, not including Annexes and Executive Summary. The Contractor should submit the final report to the Program Office within 10 days of receipt of comments. The Final Evaluation Report submission should also include a Two￾pager briefer on key qualitative and quantitative findings and conclusions relative to the evaluation 68 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB questions—to be given to the appropriate government counterpart(s) so that they have the opportunity to review evaluation findings and share them with the larger community. 7. All project data and records will be submitted in full and shall be in electronic form in easily readable format; organized and fully document for use by those not sufficiently familiar with the project or evaluation; and owned by USAID and made available to the public, barring rare exceptions, on the USAID Development Experience Clearinghouse (http://dec.usaid.gov). IV. EVALUATION TEAM COMPOSITION, ROLES AND RESPONSIBILITIES MEMBERS The evaluation team shall consist of one (1) independent international expert who serves as the team lead and three (3) high-level Ethiopian experts. The opportunity of involving a senior TB expert from USAID/Washington in the evaluation will be explored depending on availability. The evaluation team leader, in consultation with other team members, will be responsible for team coordination and performance and for ensuring the timeliness and quality of deliverables. USAID may propose other internal staff members from USAID/Ethiopia to accompany the team during site visits or participate in key parts of the evaluation (specific event participation to be determined in conjunction with the contractor and the team leader), and the internal USAID/Ethiopia staff and/or Washington based USAID staff are expected to provide written inputs to the draft report prior to Washington based staff’s departure from country. The evaluation team is not expected to identify or contract this person and the one from Washington. Expenses for possible USAID staff participation will not be included in the Evaluation Team’s budget. All the team members must be fluent in English and have strong writing skills. The Ethiopian experts should also be proficient in Amharic. A statement of potential bias or conflict of interest (or lack thereof) is required of each team member. Team Lead (1): The team lead will be an international consultant with more than 10 years of experience including some work in Africa. He/she will be responsible for team performance and for ensuring the timeliness and quality of deliverables. The team leader candidates have led at least two external performance evaluations. He/she should have technical knowledge in the areas of TB priority technical approaches, clinical, laboratory and facility/community tuberculosis treatment support, and in the design and management of donor￾funded technical assistance projects to achieve impact. Strong writing, evaluation methods, and analytical skills are required of the expert. The consultant will hold conference calls with the other team members and USAID/Ethiopia representatives/EPMES before and after the visit to Ethiopia to develop the evaluation methodology and take the lead in developing the evaluation report. The team leader together with other team members is expected to present preliminary findings of the evaluation to USAID/Ethiopia and Challenge-TB staff prior to departure from the country. Local Consultants (3): The local consultants should possess the following skills, though each does not need to possess all listed skills, all listed skills should be contained within the local consultants. o o o o o o o Experience with tuberculosis programming in Ethiopia Experience with quantitative and qualitative data collection (developing evaluation methodologies/tools and performing data collection, management, and analysis). Experience with the specific issues affecting genders differently in health programs and specifically in Tuberculosis programming in Ethiopia Understanding the local health system and structures (FMOH, RHBs, etc.) Design and management of donor-funded technical assistance projects to achieve impact Fluency in written and spoken Amharic Strong English language presentation and writing skills 69 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB USAID/Washington TDY (1): If possible, the involvement of the USAID/Washington technical staff person would complement the evaluation team by looking at the project's approach in addressing USAID policies, strategic approaches, and priorities as well as global standards. V. EVALUATION SCHEDULE The estimated time period for undertaking this evaluation is 83 working days, of which about 60 days should be spent in Ethiopia. The ideal start time for the evaluation team is January 22nd, 2019. However, the date will be finalized between USAID and the Contractor. The evaluation team is required to work six days a week, but with no premium pay for the sixth day. The team is required to travel to selected zones of the representative sample regions where the activity is being implemented. At least 30% of the consultants' time will be spent on conducting interviews with activity staff, government partners, and project beneficiaries. The evaluation team will prepare an exit briefing and presentation of the findings, which it will deliver to USAID staff before the expat departs Ethiopia. This evaluation should be completed before May 2019 so that sufficient collaboration and inputs could be sought from the implementing partner before the Activity’s period ends and also the findings and recommendations can be used for future similar Activity design. Illustrative Level of Effort (LOE) in person-days and timeline Activity Team Lead (1) Technical Specialist/Evaluation specialists (3) Total LOE Period of performance (estimated dates) Kickoff call meeting 0.5 1.5 2 Jan 2, 2019 EQUI® training 1.5 1.5 3 Jan 2-3, 2019 Desk Review of documents 3 9 12 Draft evaluation design work, data collection tools s; plan logistics 3 9 12 Jan4-8, 2019 Travel to country 1 0 1 Jan 9, 2019 Pre-in brief meeting with USAID 0.5 1.5 2 Jan 11, 2019 Evaluation teamwork on the evaluation work plan, design, methodology, and data collection tools 4 12 16 Jan 12-16, 2019 PowerPoint preparation on evaluation design and work plan 1.5 4.5 6 Jan 17-18, 2019 Evaluation team present the evaluation design and work plan to USAID 0.5 1.5 2 Jan 22, 2019 Evaluation team finalize the evaluation work plan, design, methodology, and data collection tools (including incorporating USAID comments), and logistics for field work) and submit the design report to USAID 1 3 4 Jan 23, 2019 Evaluation team address all comments from USAID 2 9 11 Jan 24-25, 2019 70 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB Meetings with select offices in Addis Ababa 2 6 8 Jan 28-29, 2019 Fieldwork including travel days 25 75 100 Jan 30-Feb 27, 2019 Preliminary Data analysis 10 36 46 Mar 1-12, 2019 Prepare power point presentation 2 6 8 Mar 13-14, 2019 Presentation to EPMES and addressing comments 1 3 4 Mar 15, 2019 Preliminary findings and recommendation presentation to USAID 0.5 1.5 2 Mar 18, 2019 Write 1st draft of the report 11 18 32 Mar 19-30, 2019 Complete a post-evaluation After Action Review with EPMES 1 3 4 April 1, 2019 Team leader departs the country 2 0 2 April 2-3,2019 EPMES/SI internal review of the draft report 0 0 0 April 4-5, 2019 Evaluation team address internal comments 3 3 6 April 8-10, 2019 EPMES submit draft report and USAID review 0 0 0 April 12-26, 2019 Evaluation team address USAID comments 3 3 6 April 29 -May 1, 2019 EPMES/SI reviews the second draft report 0 0 0 May 2-6, 2019 Team Leader address EPMES/SI comments 3 3 6 May 7-9 EPMES/SI and Team leader check formatting and completeness 1 0 1 May 10-13, 2019 EPMES/SI submits final draft report to USAID 0 0 0 May 14, 2019 USAID reviews the report and approves it 0 0 0 May 15-31, 2019 Total LOE 83 210 293 VI. MANAGEMENT Social Impact, the Contractor managing the Ethiopia Monitoring and Evaluation Service (EPMES) activity and conducting this evaluation will identify and hire the evaluation team, pending the Contracting Officer’s Representatives (COR’s) and relevant technical office’s concurrence and CO approval, assist in facilitating the work plan, and arrange meetings with key stakeholders identified prior to the initiation of the fieldwork. The evaluation team will organize other meetings as identified during the course of the evaluation, in consultation with EPMES’s Contractor and USAID/Ethiopia. The EPMES Contractor is responsible for all logistical support required for the evaluation team, including arranging accommodation, security, office space, computers, Internet access, printing, communication, and transportation. The evaluation team will officially report to the Ethiopia Monitoring and Evaluation Service (EPMES) Contractor, Social Impact. The EPMES Contractor is responsible for all direct coordination with the USAID/Ethiopia Program Office through the EPMES COR. From a technical management perspective, the evaluation team will work closely with the Activity Manager for the Challenge-TB Activity seated in 71 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB Health, AIDS, Population and Nutrition (HAPN) Office. In order to maintain objectivity, all final decisions about the evaluation will be made by the Program Office. VII. LOGISTICS The contractor will be responsible for all travel and logistics associated with conducting the evaluation. However, the Contractor may also seek assistance from the Activity’s AOR and implementing partner(s) to identify stakeholders for interviews. VIII. ANNEXES Annex A: USAID Evaluation Report Standards 1. Identify the evaluation as either an impact or performance evaluation per the definitions in ADS 201. 2. Include an abstract of not more than 250 words briefly describing what was evaluated, evaluation questions, methods, and key findings or conclusions. The abstract should appear on its own page immediately after the evaluation report cover. 3. Include an Executive Summary 2–5 pages in length that summarize key points (purpose and background, evaluation questions, methods, findings, and conclusions). 4. State the purpose of, the audience for, and anticipated use(s) of the evaluation. 5. Describe the specific strategy, project, activity, or intervention to be evaluated including (if available) award numbers, award dates, funding levels, and implementing partners. 6. Provide brief background information. This should include the country and/or sector context; specific problem or opportunity the intervention addresses; and the development hypothesis, theory of change, or how the intervention addresses the problem. 7. Identify a small number of evaluation questions. 8. In an impact evaluation, identify questions about measuring the change in specific outcomes attributable to a specific USAID intervention. 9. Describe the evaluation method(s) for data collection and analysis. 10. Describe the limitations of the evaluation methodology. 11. In an impact evaluation, use specific experimental or quasi-experimental methods to answer impact evaluation questions. 12. Include evaluation findings and conclusions. 13. Recommendations should be included separately from findings and conclusions. 14. Address all evaluation questions in the Statement of Work (SOW) or document approval by USAID for not addressing an evaluation question. 15. Include the following annexes: ● Evaluation SOW. If the SOW is revised, the evaluation report should include the updated SOW as an Annex rather than the original SOW. ● Description of evaluation methods (if not described in full in the main body of the evaluation report). ● All data collection and analysis tools used, such as questionnaires, checklists, survey instruments, and discussion guides. ● All sources of information— correctly identified and listed. ● Any “statements of differences” regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team. ● Signed disclosures of conflicts of interest from evaluation team members. ● Abridged bios of the evaluation team members, including qualifications, experience, and role on the team. 72 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB 16. Include enough information on the cover of the evaluation report so that a reader can immediately understand that it is an evaluation and what was evaluated. The evaluation cover should: ● Include a title block in USAID light blue background color. ● Include the word “Evaluation” at the top of the title block and center the report title underneath that. The title should also include the word “evaluation.” ● Include the following statement across the bottom of the cover page: “This publication was produced at the request of the United States Agency for International Development. It was prepared independently by [list authors and organizations involved in the preparation of the report].” For an internal evaluation team, use the following statement: “This publication was produced at the request of [USAID/Mission] and prepared by an internal evaluation team comprised of [list authors and affiliation].” ● Feature one high-quality photograph representative of the project being evaluated and include a brief caption on the inside front cover describing the image with photographer credit. ● State the month and year of the report. ● State the individual authors of the report and identify evaluation team leader. Annex B: USAID Criteria for Quality Evaluation ● Evaluation reports should represent a thoughtful, well-researched, and well-organized effort to evaluate the strategy, project, or activity objectively. ● Evaluation reports should be readily understood and should identify key points clearly, distinctly, and succinctly. ● The Executive Summary should present a concise and accurate statement of the most critical elements of the report. ● Evaluation reports must address all evaluation questions included in the SOW, or the evaluation questions subsequently revised and documented in consultation and agreement with USAID. ● Evaluation methodology must be explained in detail and sources of information adequately identified. ● Limitations to the evaluation must be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). ● Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or merely the compilation of people's opinions. ● Findings and conclusions should be specific, concise, and supported by strong quantitative or qualitative evidence. ● If evaluation findings address person-level outcomes and impact, they should be assessed for both males and females. ● If recommendations are included, they should be supported by a specific set of findings and should be action-oriented, practical, and specific. Annex C: Results Framework 73 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB IV. REFERENCES See the Official Websites ● http://www.usaid.gov// ● Challenge-TB RFA ● Challenge-TB Cooperative Agreement ● WHO End TB Strategy ● WHO Global TB Report 2018 Goal: Reduced Incidence and mortality of TB to achieve the global End TB Strategy milestones and targets for 2025 and 2035 in Ethiopia*. Purpose: Increased demand, access to, and utilization of TB, MDR-TB and TB/HIV services in target communities IR 1.IMPROVE ACCESS TO HIGH￾QUALITY, PATIENT￾CENTERED TB, MDR￾TB, AND TB/HIV SERVICES Sub-IR s 1. Create enabling environment 2. Comprehensive, high-quality diagnostic network 3. Patient-centered care and treatment IR 2 PREVENT TB TRANSMISSION AND DISEASE PROGRESSION. Sub-IR s 1. Targeted screening for active TB. 2. Infection Control. 3. Management of latent TB infection IR 3.STRENGTHEN TB SERVICE DELIVERY PLATFORMS Sub-IR s 1. Political commitment and leadership Strengthened 2. Comprehensive partnerships and informed community involvement 3. Drug and commodity management systems 4. Quality data, surveillance, and monitoring and evaluation. 5. Human resource development. IR 4. ACCELERATE RESEARCH AND INNOVATION Sub-IR s 1. New tools/approaches to prevent, detect and treat TB more effectively. 2. Performance and policy improvements through implementation research. 74 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB ANNEX II: DETAILED DESCRIPTION OF EVALUATION DESIGN AND METHODS Sampling procedure for primary data collection Site selection The evaluation team applied a multistage purposive sampling approach to select regions and city administrations from where primary data for the evaluation was collected in the form of KIIs, observations, and exit interviews. In the first stage, regions were selected, and in the second, zones were selected within those regions, following the parameters listed below. These parameters ensured that the sites better represent CTB performance and the cumulative effect of long-term investment by USAID/Ethiopia in TB control, along with the feasibility of visiting the sites. Using these parameters, the regions covered by CTB were assessed, and four agrarian regions (Amhara, Oromia, SNNPR, and Tigray), one developing region (Gambella), and one city administration (Addis Ababa) were selected. The assessment of regions and the selection outcome are listed in Table 13 below. Table 13: Selection of regions using criteria Regions Criteria Selected Long duration of support by CTB Overlap with HEAL TB Developing region Logistic convenience No security issues Oromia ✔ ✔ ✔ ✔ Amhara ✔ ✔ ✔ ✔ ✔ SNNPR ✔ ✔ ✔ ✔ Tigray ✔ ✔ ✔ ✔ Addis Ababa ✔ ✔ ✔ ✔ Diredawa ✔ Harrar ✔ Gambella ✔ ✔ ✔ Benshangul ✔ Within each of the selected regions, one zone was selected based on criteria listed earlier, as given in Table 14 below. An additional hospital was selected in SNNPR, in South Omo district, to include a remote location in the sample. Table 14: Selection of zones Regions Zones Criteria for selection Best practice/innovation sites Oromia West Arsi CDR higher than median for region Logistic convenience Amhara Awi CDR lower than median for region Logistic convenience SNNPR South Omo CDR lower than median for region REACH TB sites Yirgalem hospital SNNPR Jinka (South Omo) CDR lower than median for region Remote location 75 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB Tigray East CDR lower than median for region Logistic convenience Mekele hospital Addis Ababa Addis Ababa Subcity Best practice site All facilities in the sub-city Gambella Gambella Logistic convenience Security In each selected zone, one district that was part of CTB’s intensified support for CTBC was selected to conduct KIIs and document review at the district office and in one health post. In addition, in each selected zone, either the zonal hospital or the district health center of the selected district was selected for facility-based KIIs, document review, and observations. The reason for alternating between zonal hospital and district health center was to make the total number of KIIs and observations feasible. These selections are presented in Table 15 below. Table 15: Selection of districts and facilities in selected zones Regions Zones Hospitals Districts Health Centers Health Posts Oromia West Arsi Arsi Negele Arsi Negele 1 Amhara Awi Injebara Dangila Zuria 1 SNNPR South Omo Jinka Wondo Genete* 1 Tigray East Adigrat Adigrat 1 AA Addis Ababa Subcity Woreda 7 Addis Ketema 1 Gambella Gambella Gambella Total 6 3 5 3 5 *This is in Sidama but is selected for logistic convenience and presence of intensive support The following sites of CTB innovations were added to the sample to assess the innovations’ scalability. The data collection included KIIs and observation of relevant units in these sites. These sites, along with the sites already selected, cover all the major innovations and best practices that CTB introduced, except for the interventions related to key populations (prisons and mines), which have not been included for logistic and security reasons. 1. Mekele Hospital (I STAT for adverse drug reaction monitoring, New DR TB drug introduction) 2. Yirgalem Hospital (I STAT for adverse drug reaction monitoring) 3. ALERT laboratory (Culture DST for quality improvement) 4. REACH TB sites in Sidama, SNNPR (Community TB Care) 5. Addis Ababa (Integration with child health services, mass screening of the urban poor) Distribution of KIIs in regions, zones, and facilities In each selected region, a group discussion was held with key informants from CTB staff and another with the regional TB and lab staff of the program. Observations and KIIs were also held in the regional lab. The assumptions used to estimate the maximum sample size is given in Table 16 below. Table 16: Assumptions of facility types/locations for estimating maximum number of respondents Level KII - CTB KII￾Program Regional Lab Observation Facility Observations Facility KIIs Exit Interview Region 1* 1* 1 Zone 1 1 District 1 Hospital# 4 6 2 Health center# 4 6 2 Health post 1 1 Totals 2 76 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB *Group discussion with key informants # Either hospital or health center in each zone The final sample size obtained was in full alignment with the sampling plan. Data Preparation Qualitative data During fieldwork, the evaluation team divided into two parallel teams. One team covered Addis Ababa, Oromia, and SNNPR, and the other covered Amhara, Gambella, and Tigray. Both teams divided up the national-level KIIs between them. Each team had a TB specialist and an evaluation specialist. Members of each team prepared summary notes of the KIIs, observations, and exit interviews for each region and identified underlying themes emerging from the interviews and discussions. Descriptive data from exit interviews and observations were added to the other primary qualitative data for thematic analysis. After completion of fieldwork, the team transcribed recordings and organized the notes captured during data collection. The evaluation questions and identified themes served as the organizing framework for the notes. To ensure quality and consistency, the leaders of each subteam held calls daily to review progress and findings during fieldwork. The evaluation team leader provided daily updates to EPMES. Quantitative data (primary and secondary) Primary quantitative data from exit interviews and facility observations and secondary quantitative data from NTP and CTB were entered in separate spreadsheets for basic analysis of trends and variance. To the extent possible, data was disaggregated by sex, geography, and facility level. Data Analysis Transcripts from KIIs, observations, and exit interviews were analyzed manually based on the themes and subthemes contained in the data collection tools. Descriptive data were coded by both teams using a standard analysis matrix that included pre-identified themes and subthemes (from the evaluation design matrix) and accommodated emerging subthemes. Emphasis was placed on comparing the responses across participant categories, cities/towns, regions, and facility type to identify similarities and differences in the opinions obtained. Primary and secondary quantitative data will be analyzed in MS Excel. Although these data points are quantitative, they are not drawn from a representative sample, so the findings were not generalized. Any distinctive findings by gender and geography were actively looked for in analysis of both descriptive and quantitative data. Ethical considerations The evaluation team, as well as EPMES and HQ staff, followed professional and ethical guidelines to ensure that the evaluation was carried out with honesty and integrity, respondents are protected, and data security is ensured. Respondents' confidentiality and privacy were protected during data collection through procedures for obtaining informed consent and proper procedures for handling personal information. The location and timing of exit interviews with patients ensured that the confidentiality related to the patient’s illness and reasonable privacy was maintained. DOT provision typically takes place in the early hours of the day before the OPD opens. The team reviewed the list of patients scheduled for DOT on that day, selected two DS TB patients, and waited outside the TB clinic for these patients to arrive and receive DOT. The team also asked the TB clinic staff to call them in when the selected patients arrived. The team introduced the study briefly to the patient inside the TB clinic, and if the patient consented, escorted them to the identified interview location (in an open space within the facility’s premises) and read the consent form, and if the patient agreed, the team proceeded with the interview. 77 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB Evaluation team members undertook personal protection measures as appropriate and as provided for in national guidelines, but also ensuring that patient confidentiality is not compromised. 78 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB ANNEX III: EVALUATION DESIGN MATRIX Evaluation question Main themes Potential questions Data collection method(s) Data source/type of respondent Data analysis method EQ 1: To what extent has CTB’s technical assistance and management approach supported the CTB cooperative agreement objectives? 1.1. Design and alignment of technical assistance with CTB objectives, with objectives/strategy of GoE and needs/gaps identified How aligned was CTB’s technical assistance with its objectives/RF? What types of technical assistance were planned? How were these decided (alignment with GoE objectives and gaps/needs identified) What, if any, are areas of non-alignment between the CTB cooperative agreement and CTB Ethiopia objectives, and what are the reasons for that? Document review, KII Cooperative Agreement, CTB APA narrative reports; IP staff at central level; USAID Thematic analysis, including pre￾determined and emerging subthemes; triangulate data from multiple sources 1.2. Technical assistance support provided How did CTB approach technical assistance? What types of STTA were planned and provided to GoE at all levels? Document review; KII CTB APA narrative reports; IP staff at central level Thematic analysis, including pre￾determined and emerging subthemes; triangulate data from multiple sources 1.3. Content, quality, and delivery of capacity building What initial assessments and analyses did CTB conduct to inform its capacity-building approach and strategy? What types of capacity-building support were provided? How did CTB ensure relevance of its capacity-building support? How did CTB ensure that the support was effective? Document review; KII CTB APA narrative reports; lists of training and other capacity-building support provided; IP staff at all levels; NTP manager; regional, zonal and district TB staff; lab and HMIS staff; CSOs and Thematic analysis, including pre￾determined and emerging sub themes; triangulate data from multiple sources; anecdotal evidence, where available; analysis by CTB objective. 79 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB Evaluation question Main themes Potential questions Data collection method(s) Data source/type of respondent Data analysis method professional associations 1.4. Management structure To what extent did the technical support objectives inform the management structure of CTB? Is the staff structure optimal for carrying out the various planned activities? What considerations were used to develop staff profiles and hire staff? How was the attrition in the project and how was it managed? In what ways did it affect optimal delivery of capacity-building support? Document review; KII CTB APA narrative reports; IP staff at central and regional levels; NTP manager, Regional TB staff, USAID Thematic analysis, including pre￾determined and emerging subthemes; triangulate data from multiple sources; anecdotal evidence, where available 1.5. Staff competencies What was CTB’s strategy for building the competencies of its own staff? To what extent do you think the competencies of CTB staff were sufficient and relevant to meet the technical support and capacity￾building objectives of the CTB? What steps did you take when staff competencies were judged insufficient? Document review; KII APA narrative reports; IP staff at central and regional levels; NTP manager; regional TB staff; USAID; stakeholders Thematic analysis, including pre￾determined and emerging subthemes; triangulate data from multiple sources; anecdotal evidence, where available 1.6 Perceptions of the stakeholders How relevant/appropriate and effective was CTB’s technical assistance at all levels? KII NTP manager; PSA, EPHI, regional, zonal and district TB and laboratory Thematic analysis, including pre￾determined and emerging subthemes; 80 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB Evaluation question Main themes Potential questions Data collection method(s) Data source/type of respondent Data analysis method What difference did CTB make, in terms of capacity at all levels? staff; facility staff; USAID and stakeholders; professional associations triangulate data from multiple sources; anecdotal evidence, where available; analysis by CTB objective 1.7. Main achievements and challenges of CTB – capacity building What are the achievements that CTB helped happen in capacity? What challenges did CTB face and how were these addressed? Document review; KII CTB APA narrative reports; NTP manager; PSA, EPHI, regional, zonal, and district TB and laboratory staff; facility staff; USAID; CSOs and professional associations Thematic analysis, including pre￾determined and emerging subthemes; triangulate data from multiple sources; anecdotal evidence, where available; analysis by CTB objective 1.8. Coordination and collaboration How did CTB collaborate and coordinate with these stakeholders, especially the Global Fund, PHSP, PEPFAR, GLRA? What improvements or efficiencies resulted from these collaborations? What opportunities do you think were missed, What challenges did you encounter in the collaborations? How did you address them? Document review; KII CTB APA narrative reports; IP staff at all levels; NTP manager; regional, zonal and district TB staff, lab and HMIS staff; CSOs and professional associations, USAID, Stakeholders Thematic analysis, including pre￾determined and emerging subthemes; triangulate data from multiple sources; anecdotal evidence, where available 1.9. Building on past achievements In what ways did CTB leverage the achievements of the past initiatives (TB CARE and HEAL TB) and build on them? Document review; KII CTB APA narrative reports; IP staff at all levels Thematic analysis, including pre￾determined and emerging subthemes; 81 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB Evaluation question Main themes Potential questions Data collection method(s) Data source/type of respondent Data analysis method NTP manager, Regional, zonal and district TB staff, lab and HMIS staff; USAID, Stakeholders triangulate data from multiple sources; anecdotal evidence, where available; EQ2: To what extent did CTB implementation approaches use international standards and proven strategies? 2.1 Alignment with international standards What are the national standards and strategies? What is the overall approach that CTB used to ensure that international standards and best practices in TB care and control are used by the NTP? Document review; KII International best practices; CTB APA narrative reports; national guidelines related to TB; IP technical staff at central level; NTP manager, PSA, EPHI/TRAC, USAID Thematic analysis, including pre￾determined and emerging subthemes; triangulate data from multiple sources; anecdotal evidence, where available; analysis by CTB objective 2.2 Customizing to national context How did CTB help the national program contextualize/adapt international standards and best practices? Document review; KII CTB APA narrative reports; national guidelines related to TB; IP technical staff at central level; NTP manager, PSA, EPHI/TRAC, USAID Thematic analysis, including pre￾determined and emerging subthemes; triangulate data from multiple sources; anecdotal evidence, where available; analysis by CTB objective 2.3. Parameters for quality What parameters did CTB use to define quality in all its TA areas? How did it apply these parameters in its work? Document review; KII CTB APA narrative reports; performance framework IP M&E staff at central level Thematic analysis, including pre￾determined and emerging subthemes; triangulate data from multiple sources; anecdotal evidence, where available; 82 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB Evaluation question Main themes Potential questions Data collection method(s) Data source/type of respondent Data analysis method analysis by CTB objective 2.4. Adherence to national standards In what ways did CTB help improve adherence to national standards and protocols at all levels of service delivery? What would the adherence levels be, had CTB never been there? Document review; KII; observations, exit interviews CTB APA narrative reports; IP staff at regions and zones; facility staff, including specialized hospitals; patients Thematic analysis, including pre￾determined and emerging subthemes; triangulate data from multiple sources; anecdotal evidence, where available; analysis by CTB objective EQ3. Has the information generated by CTB been used to support the achievement of objectives and outcomes and, if so, how? 3.1 Generation of evidence How does CTB generate evidence? Who were the key players in generating evidence? What role do local stakeholders play in generating evidence? To what extent has national capacity in generating evidence been built due to CTB efforts? Document review; KII CTB APA narrative reports; performance framework with achievements IP M&E staff at central level Thematic analysis, including pre￾determined and emerging subthemes; triangulate data from multiple sources; anecdotal evidence, where available 3.2. Using performance data for learning/decision making What are CTB’s measures of progress toward objectives? How were performance indicators used to learn, adapt, and improve CTB performance? What is the process for documenting lessons? 3.3. Documenting successes and failures Does the M&E system allow for documentation of successes and 83 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB Evaluation question Main themes Potential questions Data collection method(s) Data source/type of respondent Data analysis method failures at all levels? Cite examples. 3.4. Use of data for program improvement To what extent did CTB’s use of data help improve program management? EQ4: What are CTB’s main achievements and challenges? 4.1. Achievements and challenges in supporting TB program management capacity What are CTB’s achievements in supporting sustainable improvements in TB program management capacity? Probe for generating and using evidence; decentralizing services to key populations; expanding access. What, among these, would have taken place had CTB not been implemented? What difference did CTB make? What were the challenges? Document review; KII CTB APA narrative reports; IP staff at center and regions; NTP manager, Regional TB staff; USAID Thematic analysis, including pre￾determined and emerging subthemes; triangulate data from multiple sources 4.2. Achievements and challenges in supporting increasing case notification What are CTB’s achievements in supporting sustainable improvements in case notification? Probe for contact tracing, lab network, intensified case finding. What, among these, would have taken place had CTB not been implemented? What difference did CTB make? What were the challenges? Document review; secondary data analysis; KIIs CTB APA narrative reports; program data; IP staff at center and regions; NTP manager; regional, zonal, and district TB staff; EPHI/TRAC; AHRI; regional laboratories; USAID Thematic analysis, including pre￾determined and emerging subthemes; triangulate data from multiple sources 4.3. Achievements and challenges in supporting What are CTB’s achievements in supporting case management? Probe for quality of care for TB, Document review; CTB APA narrative reports; program Thematic analysis, including pre￾determined and emerging subthemes; 84 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB Evaluation question Main themes Potential questions Data collection method(s) Data source/type of respondent Data analysis method and improving case management MDR TB, TB HIV, retrieving LTFU, improving adherence. What, among these, would have taken place had CTB not been implemented? What were the challenges? secondary data analysis; KIIs data; IP staff at center and regions; NTP manager; regional TB staff; facilities; USAID triangulate data from multiple sources 4.4. Achievements and challenges in TB prevention activities (TBIC, LTBI) What are CTB’s achievements in supporting implementation of TB prevention activities? Probe for TBIC, LTBI, IEC related to these? What, among these, would have taken place had CTB not been implemented? What were the challenges? Document review; KIIs; observations; exit interviews CTB APA narrative reports; IP staff at all levels; NTP manager; regional, zonal, and district TB staff; facility staff, including specialized hospitals; TB patients; USAID Thematic analysis, including pre￾determined and emerging subthemes; triangulate data from multiple sources 4.5. High-impact interventions What was the rationale behind this prioritization? In what ways did it help achieve CTB objectives Document review; KII IP staff at all levels; USAID EQ 5. To what extent are CTB’s methodologies, interventions, and management setting the stage for future sustainability and ownership of project outputs and outcomes? 5.1. Improved political commitment To what extent do you think the political commitment to TB has grown over the past five years? The proportion of NTP/health budget allocation from govt sources; components of NTP funded by the govt; trends; inclusion of TB in HSTP; operationalizing evidence-based policies and strategies (from EQ 3) Document review; KIIs HSTP document; National health accounts; CTB APA narrative reports; IP staff at central level; USAID and stakeholders; NTP manager Thematic analysis, including pre￾determined and emerging subthemes; triangulate data from multiple sources 85 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB Evaluation question Main themes Potential questions Data collection method(s) Data source/type of respondent Data analysis method 5.2. NTP’s ownership of CTB initiatives To what extent do you think the improvements brought about by CTB (in the various technical areas - from EQ 4) will continue beyond its lifetime? Which improvements are most likely to continue, and why? Which improvements are least likely to continue, and why? To what extent is the NTP able to successfully manage their own progress (from EQ 4)? Document review; KIIs CTB APA narrative reports; IP staff at central level; NTP manager and regional TB managers; PSA; EPHI; USAID Thematic analysis, including pre￾determined and emerging subthemes; triangulate data from multiple sources 5.3. Scaling up innovations/best practices What are the innovations and best practices piloted by CTB and what makes them such? What factors make them feasible for scale up, what factors hinder? Document review; KIIs CTB documents related to innovations; PIH; IP staff at central level and regions with innovation pilots; NTP manager; regional TB staff with innovation pilots; USAID Thematic analysis, including pre￾determined and emerging subthemes; triangulate data from multiple sources 5.4. Adapting to changes in health system What changes took place in the last five years, and how did CTB adapt to the changes? Document review; KIIs CTB APA narrative reports; IP staff at central and regional levels; NTP manager; PSA; EPHI Thematic analysis, including pre￾determined and emerging subthemes; triangulate data from multiple sources 5.5. Allocating funds to sub- stakeholders What were the main criteria? What are the perceptions of IP and sub-stakeholders on these? Document review; KIIs CTB APA narrative and IP at central Thematic analysis, including pre￾determined and 86 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB Evaluation question Main themes Potential questions Data collection method(s) Data source/type of respondent Data analysis method level; MSH; WHO; USAID emerging subthemes; triangulate data from multiple sources; analysis by CTB objective 5.6. Reasonableness of project costs What have the burn rates been? If good, how did they achieve and maintain it? If not, what steps did they take to improve it? (Subject to provision of this information by CTB) Any creative cost-cutting measures in training, procurement? Any leveraging? Document review; KIIs CTB APA narrative and IP at central level; MSH; WHO Thematic analysis, including pre￾determined and emerging subthemes; triangulate data from multiple sources; specific anecdotal evidence, where available 5.7. Supporting sub￾stakeholders’ financial management practices What support was provided? What is the evidence for improved capacity? Document review; KIIs CTB APA narrative and IP at central level; MSH; WHO Thematic analysis, including pre￾determined and emerging subthemes; triangulate data from multiple sources Cross-Cutting Issue: Gender Equity 6.1. Equitable access How has gender equity improved under CTB assistance? Document review; secondary data analysis; exit interviews; KIIs CTB APA narrative reports; program and CTB data; IP at central and regional levels; patients Thematic analysis based on themes and sub themes, with specific anecdotal evidence, where available; triangulate data from multiple sources 87 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB ANNEX IV: DATA COLLECTION TOOLS KEY INFORMANT INTERVIEW CONSENT FORM Title: Challenge TB – Final Evaluation Investigators: Beulah Jayakumar, Ezra Shimeles, Amsalu Bekele, Dereje Mamo, Worku Ambelu, Henok Metaferia Sponsor: USAID/Ethiopia Introduction Hello, my name is----------. I am part of a team from Social Impact (SI) currently conducting an independent evaluation of the “Challenge TB (CTB) Activity." SI is an international consulting company with its headquarters in Arlington Virginia, USA and with a Field Office in Addis Ababa, Ethiopia. SI works to improve development effectiveness around the world through evaluation, capacity building and strategic planning. CTB is a USAID-funded activity which supports the government of Ethiopia to support the National TB Program to improve the quality of and access to TB services in the country CTB Activity is implemented by a consortium led by KNCV. This evaluation is intended to measure the achievements of this activity and to obtain opinions about how such an activity can be improved in the future. I would like to request you to read (or have read to you) this Consent Form. This is to make sure that you are fully informed about this evaluation. After I have introduced this evaluation to you and have gone through what is expected of you, I will ask you to sign this form if you agree to participate. SI Internal Review Board has approved this evaluation. We will give you a copy of this form if you would like. This consent form might contain some words that are unfamiliar to you. Please ask us to explain anything you may not understand. I want to be sure that you understand the purpose of this valuation and your responsibilities before you decide if you want to be in it or not. Please ask me to explain any words or information that you may not understand. Information about the evaluation If you agree to be part of this evaluation, we are going to ask you and other key informants about the interventions of CTB Activity that you may know, such as capacity building, coordination and your perceptions of their results. We will also ask you about the successes and challenges CTB Activity encountered and how the activity could be improved to achieve more significant results. We will also review some reports in your facility. We plan to conduct interviews similar to this with about 90 respondents across the locations where CTB works. The information you share will be kept confidential and will not be disclosed to anyone in a way that can be linked to you. Although we will share the opinions you give us in a report to other entities outside of the evaluation team, all your answers will be treated with confidentiality and will be anonymized in the report. We want to record your interview to help us transcribe what you said accurately. The audio recording will be deleted after transcription. Additionally, your decision to participate or not to participate in this evaluation will in no way affect the services you currently receive or provide or the support you receive from CTB Activity. You have the right to refuse to answer any questions or to stop the interview at any time. Your participation in this evaluation will take about 1 to 2 hours. I will not write down your name on this form and your name will not appear in when we analyze the data or in the report we write so that the answers you give cannot be linked to you. You have the right to tell whomever you choose about this evaluation. You may stop participating in this interview at any point during our discussion. Again, 88 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB I want you to be aware that accepting to participate or ending your participation will not affect the services you provide or the support you may be receiving from CTB Activity. Possible risks We do not anticipate any significant risks to you or your organization/facility because of your participation in this evaluation. However, please note that should you choose to participate in this study, you will be taking time away from your regular activities, which may affect your routine tasks. I wanted you to be aware of these possible aspects of the interview that might affect your feelings before accepting to participate in the discussion. Possible benefits The results of this evaluation are expected to inform USAID’s planning and decision-making, assess the results of CTB Activity, improve strategies for more significant public health impact and to guide broader inter-sectoral learning and collaboration. Your participation in this evaluation will therefore be beneficial to current and future similar programs. By participating in this evaluation, you will, however, get no immediate and direct personal benefit. If you decide not to participate in this evaluation You are free to decide if you want to participate in this evaluation or not. If you decide not to participate, we will accept your decision without holding anything against you. Your relationship with CTB Activity or other organizations that provide similar services or will use the evaluation results will not be affected at all. Confidentiality We will protect information about you and your involvement in this evaluation to the best of our ability. We will not record your name in our data collection tools or notes, but only in this consent form, which we will keep separately from the notes and transcripts of this interview. We will also not indicate your name in the any of the reports we prepare, but only your official designation and place of work in an annex to the evaluation report. We will not tell your peers, supervisors, or friends about your participation or about the information you give. Leaving the interview You may end your participation in the interview at any time. We will not hold anything against you should you choose to leave before the end of the interview. Duration of interview We anticipate that this interview may take up to 2 hours. If you have a question about the evaluation If you have any questions about this evaluation, you may contact Biruk Belayneh via his email address BBelayneh@socialimpact.com or phone number 0912503019. You can also contact the Social Impact Internal Review Board. The contact person is Leslie Greene Hodel; Address is: 2300 Clarendon Blvd, Suite 1000, Arlington, VA 22201; phone number 703-465-1884; email address: irb@socialimpact.com. VOLUNTARY AGREEMENT I certify that the nature and purpose, the potential benefits, and the possible risks associated with participating in this research have been explained to me. __________________________________________________ ____________________ Signature of study participant Date EXIT INTERVIEW CONSENT FORM 89 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB Title: Challenge TB – Final Evaluation Investigators: Beulah Jayakumar, Ezra Shimeles, Amsalu Bekele, Dereje Mamo, Worku Ambelu, Henok Metaferia Sponsor: USAID/Ethiopia Introduction Hello, my name is----------. I am part of a team from Social Impact (SI) currently conducting an independent evaluation of the “Challenge TB (CTB) Activity." SI is an international consulting company with its headquarters in Arlington Virginia, USA and with a Field Office in Addis Ababa, Ethiopia. SI works to improve development effectiveness around the world through evaluation, capacity building and strategic planning. CTB is a USAID-funded activity which supports the government of Ethiopia to support the National TB Program to improve the quality of and access to TB services in the country. CTB Activity is implemented by a consortium led by KNCV. This evaluation is intended to measure the achievements of this activity and to obtain opinions about how such an activity can be improved in the future. I would like to request you to read (or have read to you) this Consent Form. This is to make sure that you are fully informed about this evaluation. After I have introduced this evaluation to you and have gone through what is expected of you, I will ask you if you agree to participate. SI Internal Review Board has approved this evaluation. We will give you a copy of this form if you would like. I want to be sure that you understand the purpose of this evaluation and your responsibilities before you decide if you want to be in it or not. This consent form might contain some words that are unfamiliar to you Please ask me to explain any words or information that you may not understand. Information about the evaluation We are aiming to gather information about the kinds of services offered to patients like yourself in this region to get a better idea about how activities like CTB respond to the needs of persons with TB. If you agree to be part of this evaluation, we are going to ask you your opinion about the services and treatment you have received, as well as any other information you may be interested to share with us about your experience with TB and the quality of care patients receive. This information will be very helpful to us in determining how activities like CTB can best serve people with TB, now and in the future. We will conduct such interviews with a total of 12 patients from six facilities in different parts of the country. The information you share will be kept confidential and will not be disclosed to anyone in a way that can be linked to you. Although we will share the opinions you give us in a report to other entities outside of the evaluation team, all your answers will be treated with confidentiality and will be anonymized in the report. Additionally, your decision to participate or not to participate in this evaluation will in no way affect the services you currently receive or provide or the support you receive from CTB Activity. You have the right to refuse to answer any questions or to stop the interview at any time. Your participation in this evaluation will take about half an hour. I will not write down your name on this form so that the answers you give cannot be linked to you. You have the right to tell whomever you choose about this evaluation. You may stop participating in this interview at any point during our discussion. Again, I want you to be aware that accepting to participate or ending your participation will not affect the services you provide or the support you may be receiving from CTB Activity. Possible risks We want to conduct this interview in a location where the chances of other patients or clinic staff coming to know of your illness is minimal. We do not anticipate any significant risks to you or your household or your organization because of your participation in this evaluation. However, please note that should you choose to participate in this study, you will be taking time away from your regular activities, which 90 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB may affect your routine tasks. I wanted you to be aware of these possible aspects of the interview that might affect your feelings before accepting to participate in the discussion. Possible benefits The results of this evaluation are expected to inform USAID’s planning and decision-making, assess the results of CTB Activity, improve strategies for more significant public health impact. The evaluation findings could lead to continued and better delivery of TB activities. Your participation in this evaluation will therefore be beneficial to current and future similar programs. By participating in this evaluation, you will, however, get no immediate and direct personal benefit. If you decide not to participate in this interview You are free to decide if you want to participate in this interview or not. If you decide not to participate, we will accept your decision without holding anything against you. Your relationship with CTB Activity or other organizations that provide similar services or will use the evaluation results will not be affected at all. Confidentiality We will protect information about you and your involvement in this evaluation to the best of our ability. We will not record your name in our data collection tools or notes or anywhere else, We will also not indicate your name in the any of the reports we prepare. We will not tell your peers, supervisors, family members, caretakers, or friends about your participation or about the information you give. Leaving the interview You may end your participation in the interview at any time. We will not hold anything against you should you choose to leave before the end of the interview. Duration of interview We anticipate that this interview will take no more than 30 minutes. If you have a question about the evaluation If you have any questions about this evaluation, you may contact Biruk Belayneh via his email address BBelayneh@socialimpact.com or phone number 0912503019. You can also contact the Social Impact Internal Review Board. The contact person is Leslie Greene Hodel; Address is: 2300 Clarendon Blvd, Suite 1000, Arlington, VA 22201; phone number 703-465-1884; email address: irb@socialimpact.com. VOLUNTARY AGREEMENT The participant read the consent form (or the form was read to him/her) and gave consent to be interviewed. __________________________________________________ ____________________ Signature of Interviewer Date OBSERVATION/KEY INFORMANT INTERVIEW CONSENT FORM – HEAD OF FACILITY Title: Challenge TB – Final Evaluation Investigators: Beulah Jayakumar, Ezra Shimeles, Amsalu Bekele, Dereje Mamo, Worku Ambelu, Henok Metaferia Sponsor: USAID/Ethiopia 91 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB Introduction Hello, my name is----------. I am part of a team from Social Impact (SI) currently conducting an independent evaluation of the “Challenge TB (CTB) Activity." SI is an international consulting company with its headquarters in Arlington Virginia, USA and with a Field Office in Addis Ababa, Ethiopia. SI works to improve development effectiveness around the world through evaluation, capacity building and strategic planning. CTB is a USAID-funded activity which supports the government of Ethiopia to support the National TB Program to improve the quality of and access to TB services in the country. CTB Activity is implemented by a consortium led by KNCV. This evaluation is intended to measure the achievements of this activity and to obtain opinions about how such an activity can be improved in the future. I would like to request you to read (or have read to you) this Consent Form. This is to make sure that you are fully informed about this evaluation. After I have introduced this evaluation to you and have gone through what is expected of you, I will ask you to sign if you agree to participate. SI Internal Review Board has approved this evaluation. We will give you a copy of this form if you would like. This consent form might contain some words that are unfamiliar to you. Please ask us to explain anything you may not understand. I want to be sure that you understand the purpose of this evaluation and your responsibilities before you decide if you want your facility to be in it or not. Please ask me to explain any words or information that you may not understand. Information about the evaluation If you agree for your facility to be part of this evaluation, we are going to observe some of the services that are offered in this facility that are related to TB, including sitting in on two consultations each, in the OPD, TB clinic and HIV clinic (and in the ward – for Specialty Hospitals). We are also going to interview some of the staff of this facility who provide TB-related services and two patients receiving DOT. These observations and interviews are related to the capacity building they received, coordination and their perceptions about the various aspects of TB services. We will also ask them about the successes and challenges CTB Activity encountered and how the activity could be improved to achieve more significant results. This information will help stakeholders assess CTB and also help improve similar activities in the future. We will carry out this set of exercises in a total of eight facilities, including this facility. The information they share and that we observe will be kept confidential and will not be disclosed to anyone in a way that can be linked to you or your facility. Although we will share the opinions your staff give us and our observations in a report to other entities outside of the evaluation team, all your answers will be treated with confidentiality and will be anonymized in the report. The name of your facility will appear in the evaluation report as part of the list of facilities visited, but we do not plan to associate the findings from this facility with the name of the facility, in our report. However, we may provide such information to USAID if we deem that to be necessary for the improvement of services offered here. Additionally, your decision to participate or not to participate in this evaluation will in no way affect your work or that of the staff of this facility. You and your staff have the right to refuse to answer any questions or to stop the observation at any time. The entire observation and interviews will take about 4-5 hours. I will write down your name on this form but your name will not appear in the data collection forms or in the report so that the answers your staff give or our observations cannot be linked to you. You have the right to tell whomever you choose about this evaluation. You may stop our interviews or observations at any point during our discussion. Again, I want you to be aware that accepting to participate or ending your participation will not affect your position in this facility or the support you may be receiving from CTB Activity. Possible risks 92 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB We will take conduct exit interviews in a location where the chances of others in the facility coming to know of the illness of the patients will be minimal. We do not anticipate such risk for patients whose consultations we will observe. We do not anticipate any significant risks to you or your staff or this facility because of your participation in this evaluation. However, please note that should you choose to participate in this study, you will be taking time away from your regular activities, and your staff (whom we interview) will take about half an hour’s time each away from their regular activities, which may affect your routine tasks. I wanted you to be aware of these possible aspects of the interview that might affect your feelings before accepting to participate in the discussion. Possible benefits The results of this evaluation are expected to inform USAID’s planning and decision-making, assess the results of CTB Activity, improve strategies for more significant public health impact. Additionally, the results may be presented or disseminated at regional, national and international meetings to support planning aimed at mobilizing support for similar activities. The findings and recommendations from this evaluation will generate critical information that can be used by planners to determine and implement activities that support the delivery of TB services. Your participation in this evaluation will therefore be beneficial to current and future similar programs. By participating in this evaluation, you will, however, get no immediate and direct personal benefit. If you decide not to participate in this evaluation You are free to decide if you want to participate in this evaluation or not. If you decide not to participate, we will accept your decision without holding anything against you. Your relationship with CTB Activity or other organizations that provide similar services or will use the evaluation results will not be affected at all. Confidentiality We will protect information about you and your involvement and your facilities’ in this evaluation to the best of our ability. We will record your name on this form, but not in our data collection tools and notes. We will not indicate your name in the any of the reports we prepare. We will not tell your peers, supervisors, or friends about your participation or about the information you give. The name of this facility will appear in the report, but findings from here will not be ordinarily associated with the name of this facility. However, we would provide information related to this facility to USAID, if providing such information will help improve the services provided here. Leaving the interview You may end your and your facility’s participation in the evaluation at any time. We will not hold anything against you should you choose to do so. Duration of interview We anticipate that the interviews and observations in your facility will take 4 to 5 hours. If you have a question about the evaluation If you have any questions about this evaluation, you may contact Biruk Belayneh via his BBelayneh@socialimpact.com or phone number 0912503019. You can also contact the Social Impact Internal Review Board. The contact person is Leslie Greene Hodel; Address is: 2300 Clarendon Blvd, Suite 1000, Arlington, VA 22201; phone number 703-465-1884; email address: irb@socialimpact.com. VOLUNTARY AGREEMENT I certify that the nature and purpose, the potential benefits, and the possible risks associated with participating in this research have been explained to me. __________________________________________________ ____________________ 93 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB Signature of head of facility Date OBSERVATION CONSENT FORM - PATIENT Title: Challenge TB – Final Evaluation Investigators: Beulah Jayakumar, Ezra Shimeles, Amsalu Bekele, Dereje Mamo, Worku Ambelu, Henok Metaferia Sponsor: USAID/Ethiopia Introduction Hello, my name is----------. I am part of a team from Social Impact (SI) currently conducting an independent evaluation of the “Challenge TB (CTB) Activity." SI is an international consulting company with its headquarters in Arlington Virginia, USA and with a Field Office in Addis Ababa, Ethiopia. SI works to improve development effectiveness around the world through evaluation, capacity building and strategic planning. CTB is a USAID-funded activity which supports the government of Ethiopia to support the National TB Program to improve the quality of and access to TB services in the country. CTB Activity is implemented by a consortium led by KNCV. This evaluation is intended to measure the achievements of this activity and to obtain opinions about how such an activity can be improved in the future. I would like to request you to read (or have read to you) this Consent Form. This is to make sure that you are fully informed about this evaluation. SI Internal Review Board has approved this evaluation. We will give you a copy of this form if you would like. This consent form might contain some words that are unfamiliar to you. Please ask us to explain anything you may not understand. I want to be sure that you understand the purpose of this valuation and your responsibilities before you decide if you want to be in it or not. Please ask me to explain any words or information that you may not understand. Information about the evaluation If you agree to be part of this evaluation, we are going to observe the interaction you have with the clinician/other service provider. We will not ask any questions of you, so that we can better understand how services are provided in this facility and what we can do to make them better. We will observe such interactions with five other patients in this facility and also carry out such observations in a total of eight facilities including this one. The information that we record from this observation will be kept confidential and will not be disclosed to anyone in a way that can be linked to you. Additionally, your decision to participate or not to participate in this evaluation will in no way affect the services you currently receive or provide or the support you receive from CTB Activity. You have the right to refuse to answer any questions or to stop the interview at any time. Your participation in this evaluation will last as long as your interaction with the clinician/nurse lasts. I will not write down your name on this form or anywhere else so that what we observe or hear cannot be linked to you. You have the right to tell whomever you choose about this evaluation. You may stop us from observing the interaction at any point during your time in this consultation room. Again, I want you to be aware that accepting to participate or ending your participation will not affect the services you provide or the support you may be receiving from CTB Activity. Possible risks Our observation of the services you receive will be confined to this consultation room. We do not anticipate any significant risks to you or your household or your organization because of your participation in this evaluation. You will not have to spend any additional time here or anywhere else, in order to 94 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB participate in this study. I wanted you to be aware of these aspects as these might affect your feelings about participating in this study. Possible benefits The results of this evaluation are expected to inform USAID’s planning and decision-making, assess the results of CTB Activity, improve strategies for more significant public health impact. Additionally, the results may be presented or disseminated at regional, national and international meetings to support planning aimed at mobilizing support for similar activities. The findings and recommendations from this evaluation will generate critical information that can be used by planners to determine and implement activities that support the delivery of TB services. Your participation in this evaluation will therefore be beneficial to current and future similar programs. By participating in this evaluation, you will, however, get no immediate and direct personal benefit. If you decide not to participate in this evaluation You are free to decide if you want to participate in this evaluation or not. If you decide not to participate, we will accept your decision without holding anything against you. Your relationship with CTB Activity or other organizations that provide similar services or will use the evaluation results will not be affected at all. Confidentiality We will protect information about you and your involvement in this evaluation to the best of our ability. We will not record your name in our data collection tools or notes or anywhere else. We will also not indicate your name in the any of the reports we prepare. We will not tell your peers, supervisors, family members, caretakers, or friends about your participation or about the information we collect. Leaving the interview You may end our observation at any time. We will not hold anything against you should you choose to stop the observation before your interaction with the clinician/nurse is over. Duration of interview We anticipate that this observation will take as long as your interaction with the clinician/nurse last. If you have a question about the evaluation If you have any questions about this evaluation, you may contact Biruk Belayneh via his email address BBelayneh@socialimpact.com or phone number 0912503019. You can also contact the Social Impact Internal Review Board. The contact person is Leslie Greene Hodel; Address is: 2300 Clarendon Blvd, Suite 1000, Arlington, VA 22201; phone number 703-465-1884; email address: irb@socialimpact.com. VOLUNTARY AGREEMENT The participant was read the consent form and gave consent to the observation. __________________________________________________ ____________________ Signature of Interviewer Date KEY INFORMANT INTERVIEW – IMPLEMENTING PARTNER Challenge TB Evaluation KII – Implementing Partner Interviewee: Country Director, Technical Director, Regional Directors, M&E Director 95 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB EQ 1. To what extent has CTB’s technical assistance to Government at different levels and Activity management approach supported the objectives of Challenge TB cooperative agreement? Design and alignment of CTB technical assistance 1. How aligned was CTB’s technical assistance with its objectives/RF? 2. What types of technical assistance was planned? How were these decided? (probe for: alignment with GoE objectives and gaps/needs identified) 3. Please describe how the objectives and strategies set forth in the cooperative agreement informed the design of CTB in Ethiopia? Were there any points of divergence, and why? Technical assistance support provided 4. How did CTB approach technical assistance? What considerations went into designing a technical assistance strategy for CTB? 5. What types of STTA were planned and provided to GoE at all levels? Content and quality of capacity-building support 6. What are the major types of capacity building provided? 7. How did CTB make sure that the capacity building support it provided was relevant to the needs? 8. How did CTB ensure the effectiveness of its capacity building support? Optimizing management structure to provide TA 9. To what extent (or how) did the technical support and capacity building objectives inform the management structure of CTB? 10. What considerations were used to develop staff profiles and hire staff? 10.1.In what ways did attrition/vacant posts affect optimal delivery of capacity building support? 10.2.What considerations went into the staffing structure in Regions and Zones? Staff competencies 11. What was CTB’s strategy for building the competencies of its own staff? 12. To what extent do you think the competencies of CTB staff were sufficient and relevant to meet the technical support and capacity building objectives of the CTB? 12.1.What steps did you take when staff competencies were judged insufficient? 13. Did the project review and adapt its capacity building approach and plan based on ground realities and changes in the health system? If yes, what changes did you make? 14. How did CTB make sure that the technical support and capacity building it provided was relevant to the needs of the program at all levels? Main achievements and challenges in capacity building? 15. What are CTB’s main achievements in building the capacity of the national stakeholders? 16. What challenges did CTB face in making these happen? How did you overcome those? Opportunities and challenges in coordination 17. There are several players in the field of technical support to the national TB program. How did CTB collaborate and coordinate with these stakeholders, especially the Global Fund, PHSP, PEPFAR, GLRA? 17.1.What improvements or efficiencies resulted from these collaborations? 17.2.What opportunities do you think were missed? 17.3.What challenges did you encounter in the collaborations? How did you address them? 17.4.What was CTB’s involvement in national planning exercises for TB? Building on past achievements 18. CTB itself is a continuation of a long line of initiatives that built national capacity in TB, including the most recent HEAL TB. In what ways did CTB leverage the achievements of the past initiatives and build on them? 18.1.Could you give specific examples for this? 96 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB EQ 3. Has the information generated by the Activity been used to support the achievement of objectives and outcomes (i.e., to make adaptations during implementation) and, if so, how? Generation of evidence 19. How does CTB generate evidence? Who were the key players in generating evidence? 20. What role of local stakeholders play in generating evidence? To what extent has national capacity in generating evidence been built due to CTB efforts? Using performance data for learning and decision-making 21. What are CTB’s measures of progress towards objectives? 22. What is the process of utilizing the data and for what purposes do you use them? 23. Could you please give us a couple of instances where CTB used the evidence it generated to improve its own performance? Documenting successes and failures 24. What is the mechanism used to gather successes and limitations, and document them? 25. What is the process for documenting lessons? How are they utilized? Using data for program improvement 26. How was the evidence generated from CTB used to improve the performance of the national TB program, at all levels? 26.1.What was the process to share your data with the national program? 26.2.Could you give some examples? Other – Ops research 27. What were the ORs that CTB design and conduct? Could you please describe the key learnings from these initiatives? EQ 4. What are the main achievements and challenges of CTB in TB program management capacity, increasing case notification, decentralizing services, expanding access, prevention of TB, strengthening lab network, and generating and using evidence? 28. What would you consider as the top three achievements of CTB, and why? 29. What would you consider as the top three areas where CTB could have done better? Please give reasons 30. What were the main limitations and challenges that CTB faced? How did the team go about addressing them? 30.1.To what extent do you think have they been addressed? 31. What were the main opportunities that you took advantage of, for your success and for addressing challenges? 32. What opportunities are still not fully utilized to improve the national TB program, which future initiatives can make use of? 33. What was the rationale behind the list of high impact interventions? To what extent has that objective been achieved? 34. What are the main achievements of CTB in supporting sustainable improvements in TB program management capacity? 34.1. In generating and using evidence 34.2. Decentralizing services to key populations – Prisons, mines, 34.3. Expanding access to – agrarian, urban areas, pastoralist areas 34.4. Ensuring that women and girls are able to access services 97 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB What difference did CTB make? 35. What were the main challenges in each? How did you address them? Achievements and challenges in supporting increases in case notification 36. What are the main achievements of CTB in supporting sustainable improvements in case notification? 36.1. Case notification 36.2. Strengthening the lab network 36.3. Intensified case finding What difference did CTB make? 37. What were the main challenges in each? How did you address them? Achievements and challenges in supporting improved case management 38. What are the main achievements of CTB in supporting sustainable improvements in case management? 38.1. In quality of care for TB, DR TB, TB HIV 38.2. Retrieving lost to follow up 38.3. Improving adherence to treatment What difference did CTB make? 39. What were the main challenges in each? How did you address them? Achievements and challenges in supporting prevention activities 40. What are the main achievements of CTB in supporting sustainable improvements in prevention activities? 40.1. TB Infection Control 40.2. Latent TB Infection treatment 40.3. Information, education and communication What difference did CTB make? 41. What were the main challenges in each? How did you address them? EQ 5. To what extent are the activity’s methodologies, interventions, and management setting the stage for future sustainability, and ownership of project outputs and outcomes Political commitment 42. To what extent do you think the political commitment towards TB has increased over the past 5 years? National ownership/capacity to successfully manage their progress 43. To what extent do you think the improvements brought about by CTB will continue beyond its lifetime? 43.1.Which improvements are most likely to continue, and why? 43.2.Which improvements are least likely to continue, and why? 44. Overall, do you think that the NTP can successfully manage their own progress? Scaling up CTB innovations 45. CTB tested a few innovations/best practices successfully. To what extent do you think they are feasible for national scale up? 45.1.What factors in the innovations are conducive for scale up? 45.2.What factors are likely to hinder scale up? CTB adaptations to changes in the system 46. Overall, how did CTB respond and adapt to changes in the national health system at different levels? 98 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB 46.1.Could you cite some examples for this please? 46.2.How easy or difficult was it for CTB to make these adaptations? Funding allocation to stakeholders 47. What criteria did you use to allocate funding to the implementation stakeholders MSH and WHO? 47.1.What process did you use in setting these criteria? 47.2.How were MSH and WHO engaged in this? 47.3.What was their ownership of the process and criteria? Reasonableness of costs 48. How does the resourcing of CTB compare with the expected results? 49. How does the resourcing of CTB compare with the actual results achieved? 49.1.How reasonable were the actual expenditures compared to the results achieved? 50. We have come to the end of this discussion. Is there anything else you would like to tell us at this time, about Challenge TB, or about the TB program in general? Thank the respondent for his/her time Challenge TB Evaluation KII – Implementing Partner Interviewee: Operations Director, CTB EQ 5. To what extent are the activity’s methodologies, interventions, and management setting the stage for future sustainability, and ownership of project outputs and outcomes Allocating funds to sub partners 1. We would like to discuss finance and accounting matters of CTB, beginning with the criteria that CTB USED to allocate funding to the implementation partners MSH and WHO. In your opinion, to what extent do these criteria reflect the resource needs of partners’ activities? 1.1. What process did you use in setting these criteria? 1.2. How were MSH and WHO engaged in this process? 1.3. What was their ownership of the process and criteria? Reasonableness of costs 2. How reasonable were the actual expenditures compared with the projected expenditures? 3. What was the burn rate over the years of CTB? 3.1. (If low), What mechanisms did you put in place to fix it? 3.2. (If high), what are the best practices that helped CTB maintain/achieve a good burn rate? 4. What are top three successful or best practices you have used, that can be replicated in other projects? 5. What were the top three challenges you faced in your work, and how did you address them? 6. Were there any creative cost-cutting measures used, such as leveraging other partners’/govt input? Supporting sub-partners’ financial management 7. How did the CTB central office help build the financial and accounting capacity of MSH and WHO? Please cite examples. 8. We have come to the end of this discussion. Is there anything else you would like to tell us at this time, about Challenge TB, or about the TB program in general? Thank the respondent for his/her time Challenge TB Evaluation KII – Implementing Partner Interviewee: Director – Regions: MSH 99 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB EQ 1. To what extent has CTB’s technical assistance to Government at different levels and Activity management approach supported the objectives of Challenge TB cooperative agreement? Technical assistance support provided 1. How did CTB approach technical assistance? What considerations went into designing a technical assistance strategy for CTB? 2. What types of STTA were planned and provided to GoE at all levels? 3. What was MSH’s involvement, if any, in designing the technical assistance package? Optimizing management structure to provide TA 4. To what extent (or how) did the technical support objectives inform the management structure of CTB – the MSH part? Building on past achievements 5. CTB itself is a continuation of a long line of initiatives that built national capacity in TB, including the most recent HEAL TB, implemented by MSH. In what ways did CTB leverage the achievements of the past initiatives and build on them? 5.1. Could you give specific examples for this? EQ 5. To what extent are the activity’s methodologies, interventions, and management setting the stage for future sustainability, and ownership of project outputs and outcomes Funding allocation to partners 6. What are your perceptions about the funding allocation criteria for MSH? 6.1. How involved was MSH in setting the criteria? 7. We have come to the end of this discussion. Is there anything else you would like to tell us at this time, about Challenge TB, or about the TB program in general? Thank the respondent for his/her time Challenge TB Evaluation KII – Implementing Partner Interviewee: WHO (Policy/GF) EQ 1. To what extent has CTB’s technical assistance to Government at different levels and Activity management approach supported the objectives of Challenge TB cooperative agreement? Technical assistance support 1. What types of technical assistance of CTB was WHO involved in? To what extent did they meet GoE objectives and needs? 2. What was WHO’s involvement, if any, in designing the technical assistance package? Content and quality of capacity-building support 3. What are the major types of capacity building provided from WHO? 4. How did CTB make sure that the capacity building support it provided was relevant to the needs? 5. How did CTB ensure the effectiveness of its capacity building support? Optimizing management structure to provide TA 6. To what extent (or how) did the technical support objectives inform the management structure of CTB – the WHO part? EQ 5. To what extent are the activity’s methodologies, interventions, and management setting the stage for future sustainability, and ownership of project outputs and outcomes 100 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB Funding allocation to partners 7. What are your perceptions about the funding allocation criteria for WHO? 7.1. How involved was WHO in setting the criteria? 8. We have come to the end of this discussion. Is there anything else you would like to tell us at this time, about Challenge TB, or about the TB program in general? Thank the respondent for his/her time Challenge TB Evaluation KII – Implementing Partner Interviewee: Regional team, CTB (Region and zone) EQ 1. To what extent has CTB’s technical assistance to Government at different levels and Activity management approach supported the objectives of Challenge TB cooperative agreement? Content and quality of capacity-building support 1. What are the major types of capacity building provided in your Region? 2. How did CTB make sure that the capacity building support it provided was relevant to the needs of this Region? 3. How did CTB ensure the effectiveness of its capacity building support? Optimizing management structure to provide TA 4. To what extent (or how) did the technical support objectives inform the management structure of CTB staff in the Region? Staff competencies 5. What was CTB’s strategy for building the competencies of its own staff in this Region? 6. To what extent do you think the competencies of CTB staff were sufficient and relevant to meet the technical support needs? 6.1. What steps did you take when staff competencies were judged insufficient? Opportunities and challenges in coordination (where applicable) 7. How did CTB collaborate and coordinate with these stakeholders, especially the Global Fund, PHSP, PEPFAR, GLRA? 7.1. What improvements or efficiencies resulted from these collaborations? 7.2. What opportunities do you think were missed? 7.3. What challenges did you encounter in the collaborations? How did you address them? 7.4. What was CTB’s involvement in national planning exercises for TB? Building on past achievements 8. CTB itself is a continuation of a long line of initiatives that built national capacity in TB, including the most recent HEAL TB. In what ways did CTB leverage the achievements of the past initiatives and build on them? 8.1. Could you give specific examples for this? EQ 4. What are the main achievements and challenges of CTB? Achievements and challenges in supporting program management capacity 9. What are the main achievements of CTB in supporting sustainable improvements in TB program management capacity in your Region? 9.1. In generating and using evidence 9.2. Decentralizing services to key populations – Prisons, mines, 9.3. Expanding access to – agrarian, urban areas, pastoralist areas 9.4. Ensuring women and girls have access to services? 101 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB What difference did CTB make? 10. What were the main challenges in each? How did you address them? Achievements and challenges in supporting increases in case notification 11. What are the main achievements of CTB in supporting sustainable improvements in case notification your Region? 11.1. Case notification 11.2. Strengthening the lab network 11.3. Intensified case finding What difference did CTB make? 12. What were the main challenges in each? How did you address them? Achievements and challenges in supporting improved case management 13. What are the main achievements of CTB in supporting sustainable improvements in case management in your Region? 13.1. In quality of care for TB, DR TB, TB HIV 13.2. Retrieving lost to follow up 13.3. Improving adherence to treatment What difference did CTB make? 14. What were the main challenges in each? How did you address them? Achievements and challenges in supporting prevention activities 15. What are the main achievements of CTB in supporting sustainable improvements in prevention activities in your Region? 15.1. TB Infection Control 15.2. Latent TB Infection treatment 15.3. Information, education and communication What difference did CTB make? 16. What were the main challenges in each? How did you address them? Probe for High Impact Interventions where applicable: 1.CI and associated TB Preventive therapy for under five children 2. Expanding use of GeneXpert as a primary test 3. Support sample referral strategies & C/DST 4. Ensure finalization and implementation of KP strategies – prison, mines, urban PLHIV 5. Enhance CBTC activities. 6. Introduction and expansion of the MDR- TB STR, new drugs, and aDSM 7. Identify Hospital as one of the priority initiatives to improve referral linkage and service integration 8. Support slide fixing and referral EQ 5. To what extent are the activity’s methodologies, interventions, and management setting the stage for future sustainability, and ownership of project outputs and outcomes National ownership/capacity to successfully manage their progress 17. To what extent do you think the support system set up by CTB will continue beyond its lifetime? 18. To what extent do you think the improvements brought about by CTB will continue beyond its lifetime? 18.1.Which improvements are most likely to continue, and why? 18.2.Which improvements are least likely to continue, and why? 102 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB 19. Overall, do you think that the NTP can successfully manage their own progress? Scaling up CTB innovations 20. CTB tested a few innovations/best practices successfully. To what extent do you think they are feasible for national scale up? 20.1.What factors in the innovations are conducive for scale up? 20.2.What factors are likely to hinder scale up? CTB adaptations to changes in the system 21. Overall, how did CTB respond and adapt to changes in the national health system at different levels? 21.1.Could you cite some examples for this please? 21.2.How easy or difficult was it for CTB to make these adaptations? 22. We have come to the end of this discussion. Is there anything else you would like to tell us at this time, about Challenge TB, or about the TB program in general? Thank the respondent for his/her time KEY INFORMANT INTERVIEWS – NTP (NATIONAL/REGIONAL/ZONAL) AND AGENCIES Challenge TB Evaluation KII – National TB Program Respondents: NTP Manager, HMIS (Policy & Planning Directorate), Laboratory (EPHI) and DSM (PSA) (Individual tools for each interviewee will be developed for data collection) Introductory questions 1. Please tell us about the status of national TB control program 1.1. What are its key achievements? What are the success factors? 1.2. What major Challenges/gaps does the NTP face? What are the major reasons? 1.3. What opportunities lie before the program? 1.4. What threats does it face? Transition Questions 2. Can you please tell us briefly about the various stakeholders working with national TB program, at different levels? 2.1. Please mention names and type of support they provide EQ 1: To what extent has CTB’s technical assistance to Government at different levels and Activity management approach supported the objectives of Challenge TB cooperative agreement? Perception of stakeholders: content, quality and delivery of TA 3. Please tell us about the role of CTB, in more detail. 3.1. What types of technical assistance are provided by CTB? How? 3.2. How relevant/appropriate and effective was CTB’s technical assistance? Probe 3.2.1. Is the support provided by CTB aligned/complementary with national priorities? 3.2.2.To what extent does CTB discuss its priorities and plans with the NTP at different levels? Please Probe for: Regional, zonal, district levels 3.3. What are your perspectives on the quality of technical assistance provided by CTB? 103 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB 3.4. In your opinion, has the support provided by challenge TB improved capacity for TB service? How? Management structure 4. Can you please tell us your opinion about the management structure of CTB, if it is optimal to ensure that project’s performance is meeting the objectives? Please Probe 4.1. Central, regional, zonal level Staff competency 5. To what extent do you think the competencies of CTB staff were sufficient and relevant to meet the technical support and capacity building objectives of the CTB? Please probe for: Local staff of CTB, International short-term TA providers of CTB 5.1. What could have been better? Main achievements and challenges 6. What are the main achievements that CTB helped happen in strengthening local capacity of the TB program to deliver strategic and effective TB programming and services? Probe for: 1.1. At national, 1.2. Regional/zonal 1.3. Health facility level, etc. 7. What about the limitations/weakness/challenges of CTB, in strengthening local capacity? 8. What difference did CTB make? Please explain Coordination and collaboration 9. How did CTB collaborate and coordinate with other stakeholders, especially the Global Fund, PHSP, PEPFAR, GLRA? Please probe for: During planning, harmonization, implementation, monitoring, review meetings 10. How did CTB collaborate and coordinate with these stakeholders, especially the Global Fund, PHSP, PEPFAR, GLRA? 10.1.What improvements or efficiencies resulted from these collaborations? 10.2.What opportunities do you think were missed? 10.3.What challenges did you encounter in the collaborations? How did you address them? 10.4.What was CTB’s involvement in national planning exercises for TB? Building on past achievements 11. Do you think CTB leverage the achievements of the past initiatives (TB CARE and HEAL TB) and build on them? In what ways? EQ 2. To what extent did Challenge TB implementation approaches use/support the use of international standards and proven strategies? Alignment with international standards 12. Let’s now move to the issue of standards and best practices. What is the overall approach that CTB used, to ensure that international standards and best practices in TB care and control are used by the NTP? Customizing to national context 104 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB 13. How did CTB help the national TB program contextualize/adapt international standards and best practices? Could you illustrate with examples, please? 14. In what ways did CTB help improve adherence to international standards and protocols at all levels of service delivery? EQ 4: What are the main achievements and challenges of CTB related to: Achievements and challenges in supporting TB program management capacity 15. What are CTB’s achievements in supporting sustainable improvements TB program management capacity? Probe for generating and using evidence; decentralizing services to key populations, expanding access, prison TB program, etc. ● What, among these, would have taken place, had CTB not been implemented? ● What were the challenges? Achievements and challenges - in supporting increasing case notification 16. What are CTB’s achievements in supporting sustainable improvements in case notification? Probe for contact tracing, lab network, support sample referral strategies & C/DST, intensified case finding, identifying hospital as a priority initiative to improve referral linkage and service integration, ● What, among these, would have taken place, had CTB not been implemented? ● What were the challenges? Achievements and challenges in supporting improving case management 17. What are CTB’s achievements in supporting case management? Probe for – quality of care – TB, MDR TB, TB HIV, retrieving LTFU, improving adherence ● What, among these, would have taken place, had CTB not been implemented? ● What were the challenges? Achievements and challenges - TB prevention activities (TBIC, LTBI) 18. What are CTB’s achievements in supporting implementation of TB prevention activities? Probe for TBIC, LTBI, IEC? ● What, among these, would have taken place, had CTB not been implemented? ● What were the challenges? EQ 5: To what extent are the activity’s methodologies, interventions, and management setting the stage for future sustainability, and ownership of project outputs and outcomes? Improved political commitment 19. To what extent do you think the government commitment towards TB has increased over the past 5 years? Please probe for : The proportion of NTP/health budget allocation from Govt sources, Components of NTP funded by the Govt, trends, Inclusion of TB in HSTP, etc. 19.1.How do you see the contribution of challenge TB for this to happen? NTP’s ownership of CTB initiatives 20. To what extent do you think the improvements brought about by CTB will continue beyond its lifetime? Please probe for 20.1.What mechanisms are there to ensure sustainability? 20.2.Which improvements are most likely to continue, and why? 20.3.Which improvements are least likely to continue, and why? 105 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB 21. To what extent will the NTP be able to manage its own progress in TB control? Scaling up innovations/best practices 22. Can you please tell if there are any innovations designed/supported/piloted by CTB? 22.1.Are these CTB support/interventions/innovations designed so that it is feasible for scale up through the health system? Please Probe for 1.3.1. Factors that make them feasible for scale up, 1.3.2. Factors that might hinder scale up, etc., ask for examples. Adapting to changes in health system 23. What is your opinion on CTB’s approach in taking into account, or aligning with these broader changes in health system? Please probe for: 23.1.Can you mention examples of changes in health system? 23.2.Did CTB adapted its approach to those changes? How? General 24. What are the unmet NTP priorities for future support? 1.4. Does NTP need any similar support in the future? If so, do you have any recommendation on the type of support NTP requires; areas to be covered, modalities of support (project approach, seconding staff, TA, logistic support) 25. We have come to the end of this discussion. Is there anything else you would like to tell us at this time, about Challenge TB, or about the TB program in general? Thank the respondent for his/her time Challenge TB Evaluation KII – Regional and Zonal TB, Laboratory, HMIS Staff (Individual tools for each interviewee will be developed for data collection) Introductory question 1. Please tell us about the status of the TB control program in your Region 1.1. What are its key achievements? 1.2. What major Challenges/gaps does the program face in your Region 1.3. What opportunities lie before the program in the Region 1.4. What threats does it face in the Region? Transition Question 2. Can you please tell us briefly about the various stakeholders working with national TB program in the Region? 2.1. Please mention names and type of support they provide EQ 1: To what extent has CTB’s technical assistance to Government at different levels and Activity management approach supported the objectives of Challenge TB cooperative agreement? 3. Please tell us about the role of CTB in the Region, in more detail. 4.1 What are the areas of support provided by CTB? 4.2 Is the support provided by CTB aligned/complementary with your Region’s priorities? 106 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB 4. To what extent does CTB discuss its priorities and plans with the Regional Health Bureau at different levels? Please Probe for: Regional, zonal, district levels 5. What are the main achievements of challenge TB in strengthening the Region’s capacity to deliver strategic and effective TB programming and services? Please Probe for: regional, zonal and facility levels. 6. What about the limitations/weaknesses/challenges of CTB, in strengthening the Region’s capacity? Please describe what could have been done better? 7. Do you think the management structure of challenge TB and its role in supportive supervision optimal to meet its objectives? How? 8. What do you think of the staff competencies of CTB, to provide high quality technical assistance? What could have been better? Please Probe for: 8.1. Local staff of CTB 8.2. International short-term TA providers of CTB 9. What are your perspectives on the quality of technical assistance provided by CTB? 10. What was the role of CTB in collaborating and cooperation with other stakeholders and stakeholders in the Region? Please Probe for: 10.1.During planning, harmonization, implementation, monitoring, review meetings, etc. 11. In what ways do you think the support provided by CTB has made an impact on the Region’s TB program? 11.1.What do you think the Region’s program would have been had CTB never been implemented? Could you give some examples please? EQ 4: What are the main achievements and challenges of CTB related to: 12. Can you please share your opinion/perception on the main achievements of CTB in supporting the following areas in your Region? 12.1.Improving TB program management capacity and sustainability 12.2.Increasing case notification 12.3.Decentralizing TB services to communities including key affected/vulnerable populations and settings 12.4.Expanding access to health facilities in rural and urban areas 12.5.Improving quality of care for TB, TB/HIV and DRTB cases 12.6.Implementation of TB prevention activities including management of latent TB 12.7.Strengthening the national laboratory network 12.8.Generating and using evidence 13. Please also tell us about the challenges that CTB faced in supporting the Region’s TB program in these areas: EQ 5: To what extent are the activity’s methodologies, interventions, and management setting the stage for future sustainability, and ownership of project outputs and outcomes 14. Can you please tell if there are any innovations designed/supported by CTB in your Region? 14.1.Are CTB support/interventions/innovations designed so that it is feasible for scale up through the health system? Please Probe for factors that make them feasible for scale up, factors that might hinder scale up, and ask for examples. 15. In your opinion, will the Region be able to maintain the support provided by CTB and the improvement gained, after phase out of the project? 107 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB 15.1.What mechanisms are there to ensure sustainability? 16. We want to know your general opinion about CTB, in providing support to the Region 16.1.Is challenge TB viewed as a supporter of the program, or parallel to it? 16.2.What are the overall strengths of the project? 16.3.What do you see as weakness of the project? 17. What are the unmet Regional priorities for future support? 17.1.Does the Region need any similar support in the future? If so, do you have any recommendation on the type of support the Region requires; areas to be covered, modalities of support (project approach, seconding staff, TA, logistic support) 18. We have come to the end of this discussion. Is there anything else you would like to tell us at this time, about Challenge TB, or about the TB program in general? Thank the respondent for his/her time KEY INFORMANT INTERVIEW – STAKEHOLDERS (Organizations) Challenge TB Evaluation KII – GLRA, PIH, GHC, GF EQ 1. To what extent has CTB’s technical assistance to Government at different levels and Activity management approach supported the objectives of Challenge TB cooperative agreement? Content and quality of capacity-building support 1. What are the major types of capacity building that CTB provided that you are aware of? 2. What are your perspectives on the relevance of CTB’s capacity building? What areas were not relevant, and why? 3. What are your perspectives on the effectiveness of CTB’s capacity building? What areas were not effective, and why? Staff competencies 4. To what extent do you think the competencies of CTB staff were sufficient and relevant to meet the technical support and capacity building objectives of the CTB? In what areas was it not sufficient or relevant, according to you? Perceptions of stakeholders 5. How relevant, appropriate and effective was CTB’s technical assistance and capacity building? 6. What difference did CTB make, in these areas? Main achievements and challenges in capacity building Opportunities and challenges in coordination 7. There are several players in the field of technical support to the national TB program. How did CTB collaborate and coordinate with these stakeholders, especially the Global Fund, PHSP, PEPFAR, GLRA, PIH and GHC? 7.1. What improvements or efficiencies resulted from these collaborations? 7.2. What opportunities do you think were missed? 7.3. What challenges did you encounter in the collaborations? How did you address them? 7.4. What was CTB’s involvement in national planning exercises for TB? 108 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB 7.5. What challenges, as per your knowledge, did CTB encounter in the collaborations? How did you address them? 7.6. What was CTB’s involvement in national planning exercises for TB? EQ 5. To what extent are the activity’s methodologies, interventions, and management setting the stage for future sustainability, and ownership of project outputs and outcomes Political commitment 8. To what extent do you think the political commitment towards TB has increased over the past 5 years? National ownership/capacity to successfully manage their progress 9. To what extent do you think the support system set up by CTB will continue beyond its lifetime? 10. To what extent do you think the improvements brought about by CTB will continue beyond its lifetime? 10.1.Which improvements are most likely to continue, and why? 10.2.Which improvements are least likely to continue, and why? 11. Overall, do you think that the NTP can successfully manage their own progress? 12. We have come to the end of this discussion. Is there anything else you would like to tell us at this time, about Challenge TB, or about the TB program in general? Thank the respondent for his/her time KEY INFORMANT INTERVIEW – CSOs & PROFESSIONAL ASSOCIATIONS Challenge TB Evaluation KII – CSOs and Local NGOs (REACH Ethiopia, VHIS, Organic Health, EPS, ETS) EQ 1. To what extent has CTB’s technical assistance to Government at different levels and Activity management approach supported the objectives of Challenge TB cooperative agreement? Content and quality of capacity-building support 1. What are the major types of capacity building that CTB provided, that you are aware of? 2. What are your perspectives on the relevance of CTB’s capacity building? What areas were not relevant, and why? 3. What are your perspectives on the effectiveness of CTB’s capacity building? What areas were not effective, and why? 4. In what areas did CTB build the capacity of your organization and your staff? 5. To what extent were these relevant and appropriate for your capacity needs? Staff competencies 6. To what extent do you think the competencies of CTB staff were sufficient and relevant to meet the technical support and capacity building objectives of the CTB? In what areas was it not sufficient or relevant, according to you? Perceptions of stakeholders 7. How relevant, appropriate and effective was CTB’s technical assistance and capacity building? 8. What difference did CTB make, in these areas? Main achievements and challenges in capacity building 9. What were CTB’s main achievements in capacity building? 10. What were the main challenges that CTB face, in building capacity? 109 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB Opportunities and challenges in coordination 11. There are several players in the field of technical support to the national TB program. How did CTB collaborate and coordinate with these stakeholders, especially the Global Fund, PHSP, PEPFAR, GLRA? 11.1.What improvements or efficiencies resulted from these collaborations? 11.2.What opportunities do you think were missed? 11.3.What challenges did you encounter in the collaborations? How did you address them? 11.4.What was CTB’s involvement in national planning exercises for TB? 11.5.What challenges, as per your knowledge, did CTB encounter in the collaborations? How did you address them? 11.6.What was CTB’s involvement in national planning exercises for TB? EQ 5. To what extent are the activity’s methodologies, interventions, and management setting the stage for future sustainability, and ownership of project outputs and outcomes Political commitment 12. To what extent do you think the political commitment towards TB has increased over the past 5 years? National ownership/capacity to successfully manage their progress 13. lifetime? 14. To what extent do you think the improvements brought about by CTB will continue beyond its lifetime? 14.1.Which improvements are most likely to continue, and why? 14.2.Which improvements are least likely to continue, and why? 15. Overall, do you think that the NTP can successfully manage their own progress? 16. We have come to the end of this discussion. Is there anything else you would like to tell us at this time, about Challenge TB, or about the TB program in general? Thank the respondent for his/her time KEY INFORMANT INTERVIEW – USAID MISSION Challenge TB Evaluation KII – USAID Mission EQ 1. To what extent has CTB’s technical assistance to Government at different levels and Activity management approach supported the objectives of Challenge TB cooperative agreement? Design and alignment of CTB technical assistance 1. How aligned was CTB’s technical assistance with its objectives/RF? 2. What types of technical assistance was planned? How were these decided? (probe for: alignment with GoE objectives and gaps/needs identified) 3. Please describe how the objectives and strategies set forth in the cooperative agreement informed the design of CTB in Ethiopia? Were there any points of divergence, and why? Technical assistance support provided 4. How did CTB approach technical assistance? What considerations went into designing a technical assistance strategy for CTB? Content and quality of capacity-building support 5. What are the major types of capacity building that CTB provided, that you are aware of? 110 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB 6. What are your perspectives on the relevance of CTB’s capacity building? What areas were not relevant, and why? 7. What are your perspectives on the effectiveness of CTB’s capacity building? What areas were not effective, and why? 8. In what areas did CTB build the capacity of your organization and your staff? 9. To what extent were these relevant and appropriate for your capacity needs? Optimizing management structure to provide TA 10. To what extent (or how) did the technical support objectives inform the management structure of CTB? 11. What considerations were used to develop staff profiles and hire staff? 11.1.How was the attrition in the project? How did you manage it? 11.2.In what ways did attrition/vacant posts affect optimal delivery of capacity building support? 11.3.What considerations went into the staffing structure in Regions and Zones? Staff competencies 12. To what extent do you think the competencies of CTB staff were sufficient and relevant to meet the technical support and capacity building objectives of the CTB? In what areas was it not sufficient or relevant, according to you? Perceptions of stakeholders 13. How relevant, appropriate and effective was CTB’s technical assistance and capacity building? 14. What difference did CTB make, in these areas? Main achievements and challenges in capacity building 15. What were CTB’s main achievements in capacity building? 16. What were the main challenges that CTB face, in building capacity? Opportunities and challenges in coordination 17. There are several players in the field of technical support to the national TB program. How did CTB collaborate and coordinate with these stakeholder, especially the Global Fund, PHSP, PEPFAR, GLRA? 17.1.What improvements or efficiencies resulted from these collaborations? 17.2.What opportunities do you think were missed? 17.3.What challenges, as per your knowledge, did CTB encounter in the collaborations? How did you address them? 17.4.What was CTB’s involvement in national planning exercises for TB? Building on past achievements 18. CTB itself is a continuation of a long line of initiatives that built national capacity in TB, including the most recent HEAL TB. In what ways did CTB leverage the achievements of the past initiatives and build on them? 18.1.Could you give specific examples for this? EQ 2. To what extent did Challenge TB implementation approaches use/support the use of international standards and proven strategies? Alignment with international standards 19. Let’s now move to the issue of standards and best practices. Could you please describe the overall management approach that CTB used, to ensure that international standards and best practices in TB services? Probe for: case notification (contact tracing, diagnostics), case management, prevention, data management. Customizing to national context 20. How did CTB help the national program contextualize/adapt international standards and best practices? Could you illustrate with examples, please? EQ 4. What are the main achievements and challenges of CTB? Achievements and challenges in supporting program management capacity 111 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB 21. What are the main achievements of CTB in supporting sustainable improvements in TB program management capacity? 21.1. In generating and using evidence 21.2. Decentralizing services to key populations – Prisons, mines, 21.3. Expanding access to – agrarian, urban areas, pastoralist areas What difference did CTB make? 22. What were the main challenges in each? How did you address them? Achievements and challenges in supporting increases in case notification 23. What are the main achievements of CTB in supporting sustainable improvements in case notification? 23.1. Case notification 23.2. Strengthening the lab network 23.3. Intensified case finding What difference did CTB make? 24. What were the main challenges in each? How did you address them? Achievements and challenges in supporting improved case management 25. What are the main achievements of CTB in supporting sustainable improvements in case management? 25.1. In quality of care for TB, DR TB, TB HIV 25.2. Retrieving lost to follow up 25.3. Improving adherence to treatment What difference did CTB make? 26. What were the main challenges in each? How did you address them? Achievements and challenges in supporting prevention activities 27. What are the main achievements of CTB in supporting sustainable improvements in prevention activities? 27.1. TB Infection Control 27.2. Latent TB Infection treatment 27.3. Information, education and communication What difference did CTB make? 28. What were the main challenges in each? How did you address them? High Impact Interventions 29. Please describe the rationale for and the process for arriving at the list of high impact interventions. 30. What difference has CTB made, in these areas? 31. What is your overall perception of this approach of prioritizing? EQ 5. To what extent are the activity’s methodologies, interventions, and management setting the stage for future sustainability, and ownership of project outputs and outcomes Political commitment 32. To what extent do you think the political commitment towards TB has increased over the past 5 years? In terms of resource allocation, capacity improvements National ownership/capacity to successfully manage their progress 33. To what extent do you think the improvements brought about by CTB will continue beyond its lifetime? 33.1.Which improvements are most likely to continue, and why? 112 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB 33.2.Which improvements are least likely to continue, and why? 34. Overall, do you think that the NTP can successfully manage their own progress? 35. We have come to the end of this discussion. Is there anything else you would like to tell us at this time, about Challenge TB, or about the TB program in general? Thank the respondent for his/her time OBSERVATION TOOL FOR FACILITIES Challenge TB Final Evaluation Observation Tool for Health Facilities (Regional lab, Hospitals, Health Centers, Health Posts) Region-------------------Zone---------------------Woreda/City Administration----------------- Name of Hospital/Health Center/Health Post---------------- Position------------------------------ Length of service in this position ------- years # Data Points Responses Yes/No Other Reasons, perceptions, opinions National & Regional Laboratory 1 Does the number of staff appear adequate? 2 Is laboratory facility equipped with necessary equipment & supplies available & functional? ● Light microscope ● LED microscope ● Xpert ● Supplies for Xpert ● Culture &DST ● supplies for culture &DST ● Other supplies (Reagents, cartridge, Sputum cups…) 3 Observe the process of sputum sample handling ● Proper sputum sample reception/collection ● Labeling of the sample container ● Proper processing of samples (Smear preparation, staining, fixing & reading) ● Record result ● Disposal of remained sputum sample 4 Is culture & DST done regularly? ● Liquid/solid? 113 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB ● Turnaround time? ● Contamination rate? ● How do you do quality control? ● How do you maintain negative pressure? ● What are the main isolates? ● What are the main challenges? ● What is your level of Laboratory(1,2,3)? What helps to assume this level? 5 Is there TB Lab register? Does it have complete information? 6 Is there most recent Lab result report (Quarterly, annual..) 7 Is your lab receiving fixed slid es for AFB microscopy? If yes, ● How is the quality of fixed slides? ● What is the positivity rate? ● How long it take to get the result? ● What is the wastage rate/Unreadable slides? 8 Is Xpert used in your facility for TB diagnosis? What is the yield, What is the turnaround time? 9 Is there any change in terms of case detection after starting to use Xpert? ● Comment on the contribution of Xpert, verify with report? 10 Is there tissue crusher available for tissue sample preparation for Xpert? 11 Is the working environment safe in the lab? ● Negative pressure ● Natural ventilation ● Availability of Personal protection utilization (N95) ● If you are using negative pressure, How do you maintain negative pressure? 12 Is LPA (for 1st & 2nd ) available and routinely performed? 13 Are you collect sample send to national Lab/ abroad for quality control? 114 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB 14 Are you receiving sputum sample collected at other facility? How is the sample transportation done ? ● By VAN/Cold chain ● Postal services ● How efficient the system? ● What was the turnaround time? For the result ● Challenges? 15 Are there technical guidelines, SOPs, manuals available? Check utilization ? 16 Are you using national algorithm for TB diagnosis? 17 Is performance data displayed on wall by graphs or tables? 18 Is the result communicated to clinicians timely? Indicate How? See report 19 Is there EQA in place? How often you do EQA? 20 Is the most recent EQA report available? See the report? Check if regularly done? 21 Is TB related aid available and utilization? ● Lab Register &its completeness ● Sample transportation form ● Internal Quality control report 22 Is this laboratory accredited internationally? 23 Are you conducting supportive supervision to facilities? How is the frequency & your feedback mechanism, see report Position------------------------------ Length of service in this position ------- years # Data Points Responses Yes/ No Other Reasons, perceptions, opinions Outpatient Department (OPD) Is the general compound of the health facility clean? Is there any screening for cough at triage, prioritization (Prompt identification of coughers, segregation and fast tracking)? 115 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB Is there any method for identifying presumptive TB cases at registration/triage? Are identified presumptive TBs (suspects) sent for AFB or gene x pert? (Check last 10 presumptive TB cases) Is there a recording system for triage results? 1 Is there a dedicated waiting area for coughers? What is the condition of cougher waiting area? ● Ventilated (Window open…) ● Spacious Is TB related health education given regularly? Verify by looking the schedule? Are IEC materials displayed in this area? 1 Is there a general waiting area for other patients? 2 What is the condition of general waiting area? ● Ventilation (Window open…) ● Spacious 3 Are IEC materials displayed in this area? 4 Is there TB screening at OPD consultation? 5 Is there a screening tool available? 6 Is the screening tool being used? 7 Is there a TB screening register? 8 Is there a TB suspect register? Is it complete? 9 What is the condition of the consultation room? ● Ventilated ● Spacious 10 Is there a diagnostic algorithm? If yes its utilization? 11 Are the following job aids/tools available in the OPDs? Comment on utilization. ● TB screening tools ● TB diagnostic algorithms ● AFB laboratory request form ● GeneXpert Request form ● GeneXpert eligibility criteria ● TB referral form 12 Is adequate information given to the TB suspected patient about the next step? 116 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB 13 Is adequate information given to confirmed TB patients? ● Correct Interpretation ● Proper disclosure of the results ● Appropriate linkage to TB clinic 14 Is there a practice of TB screening in under- five OPDs? Write % of TB screening and evaluate--- Register Position------------------------------ Length of service in this position ------- years # Data Points Responses Yes/ No Other Reasons, perceptions, opinions Laboratory 1 Is there a designated sputum collection area? 2 Is sputum collection area ventilated? 3 Process of sputum sample handling? ● Proper sputum sample reception/collection ● Labeling of the sample container ● Proper processing of samples (Smear preparation, staining, fixing & reading) ● Record result ● Disposal of remained sputum sample 4 Is there TB Lab register? Does it have complete information? 5 Is there most recent Lab result report (Quarterly, annual..) 6 Is staff number adequate for the services? 7 Is laboratory facility equipped with necessary equipment &supplies available &functional? ● Microscopy (Light, LED) ● Xpert ● Culture &DST ● Other supplies (Reagents, cartridge, Sputum cups…) 8 Is staff number & qualification adequate for your facility? 117 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB If number of staff is not adequate what else you do or any task shifting? 9 Is your lab receiving fixed slid es for AFB microscopy? If yes, ● How is the quality of fixed slides? ● What is the positivity rate? ● How long it take to get the result? ● What is the wastage rate/Unreadable slides 10 Is Xpert your first line test for TB? What is the turnaround time? 11 Is there any change in terms of case detection after starting to use Xpert? Comment on the contribution of Xpert, verify with report? 12 Is the working environment in the lab? ● Ventilated ● Spacious ● Availability of Personal protection utilization (N95, face mask Glaves) 13 Are you collect sample and send for GeneXpert test and Culture &DST? 14 If sample is collected how is the sample transportation? ● By VAN/Cold chain ● Postal services ● How efficient the system? ● What was the turnaround time? For the result ● Challenges? 15 Are there technical guidelines, SOPs, manuals available? Check utilization ? 16 Are you using national algorism for TB diagnosis? 17 Is data showing performance displayed on wall by graphs or tables? 18 Is the result communicated to clinicians timely? Indicate How? 19 Is there EQA in place? 20 Is the most recent EQA report available? See the report? Check if regularly done? 118 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB 21 Is TB related aid available and utilized? ● Lab Register & its completeness ● Sample transportation form ● Internal Quality control report Position------------------------------ Length of service in this position ------- years # Data Points Responses Yes/ No Other Reasons, perceptions, opinions TB Clinic 1 Is TB clinic present? 2 Is DOTS routine? If yes, observe and comment 3 Location of TB clinic, close to ART clinic 4 Is the TB clinic environment ● Ventilated (natural ventilation, window opened) ● Spacious 5 Is IEC materials available? Displayed, or used during patient education 6 Is the TB patients well received by staff in TB clinic? 7 Is TB patient informed about the disease, treatment, and possible side effects and cough hygiene? 8 Is the patient offered to ask questions? 9 Is TB treatment algorithm present & used? 10 Are all TB patients tested for HIV (HIV test result documented)? If no, why, Is test kit adequate? 11 Is CPT provided to TB/HIV patients ? 12 Is ART provided at TB clinic for patients with TB/HIV Co-infection? 13 Are all forms of TB index cases getting their family members/contacts screened for TB? (Comment if they do for SM+ only) 14 Is there are register for contact tracing? 15 Is health education given Daily/Regularly in TB clinic for TB patients? Verify by seeing health education schedule 16 Is TB register available? 17 Is this register regularly filled? completeness 119 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB 18 Is the register stored and maintained properly? 19 Are there mechanisms to trace defaulters? Specify, if yes. 20 Is anti-tb Drugs supply available: ● FDC /Lose formulation ● Available ● Stored in place protected from sunlight & moisture ● Any expired drug Patient Box available 21 Is there lockable cabinet for storage of TB drugs, free of moisture? 22 Is the TB Clinic staff comfortable with the use of TB Patient Kit? observe 23 Are national guidelines available? ● Programmatic Management of TB,TB/HIV and DRTB ● Infection control ● Comprehensive HIV management 24 Are there the following job aides/tools in the TB clinic? ● TB diagnostic algorithm ● AFB laboratory request form ● TB patient referral form ● GeneXpert eligibility criteria ● Gene Xpert request form ● TB treatment follow up card ● TB drug dose chart ● Recording & reporting TB tools 25 Are N95 & surgical mask available (MDR-TB treating facilities) 26 If N95 & surgical mask are available, are they used regularly? (MDR-TB) 27 Review TB Treatment cards (sample randomly). Observe for completeness. Note gaps under Remarks 28 Check Quarterly report, trends, copies of most recent Are they available? 29 Is TB performance data analysis displayed on the walls? 30 Is there a comprehensive TB trained designated TB focal person? 120 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB 31 Is there Multi-Disciplinary/IP committee in the HF? 32 Is TB focal person IP committee member? 33 Does IP committee meet regularly (at least monthly) & address TB/TB IC issues? (Verify minute logbook) Position------------------------------ Length of service in this position ------- years # Data Points Responses Yes/ No Other Reasons, perceptions, opinions DR TB Treatment Initiating Center 1 Is there triaging for smear/culture negative vs positive patients at OPD level? 2 Check for TB IC administrative control (Cough hygiene, separation, fast track) 3 Does the examination room have? ● Ventilated (natural or mechanical using fan/exhaust fan) ● UVGI ● Negative pressure ● Spacious 4 Are patients: ● Reassured ● Politely communicated ● Questions addressed 5 Is adequate information given to the patient regarding: ● Duration of treatment ● Importance of adherence ● Possible consequence if he/she stopped the treatment ● Possible side effects ● Cough hygiene ● Contact/family member investigation 6 Is chance given to the patient to ask his/her concerns? 7 Do staff use Personal protection (N95, surgical mask) 8 Is TB IEC Present? Is it displayed, or used during patient education? 121 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB 9 Is there an MDR-TB treatment initiation Panel? Do they have regular meeting to review the cases? Check register 10 Is there a TB/MDR-TB register? Are the columns complete? 11 Is initial assessment of MDR-TB done before initiation of second line? (Base line, ECG, Audiometry TSH, CBC, LFT, RFT) 12 Are MDR-TB management algorithms available? 13 Is second line Anti-TB & ancillary drug available? 14 Is Audiometry available? Training to use & interpretation? 15 Is ECG available for monitoring & initiation of second line? Any training how to interpret, How do you manage Hypokalemia 16 Are the following guidelines present? ● PMDRTB ● TBIC ● Comprehensive HIV care 18 Is there infection prevention committee? 19 Is MDR-TB focal person part of IP committee? If yes is a committee support TBIC activity? Check minutes 20 Is there regular surveillance of staff for TB? 21 Is there a comprehensive MDR-TB trained designated MDR￾TB focal person? Inpatients observation 1 Is the room having: ● Ventilation (natural, mechanical using fan/exhaust fan) ● UVGI ● Negative pressure ● Adequate space 2 Are patients: Reassured; Politely communicated Questions addressed 3 Is contact tracing done for new TB patients? 4 Is second line & new drugs available ? Adequacy of second line drugs & ancillary drugs? 5 Is HIV screening done for MDR-TB patients? If positive ART started 6 Is there a register? 1 7 122 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB Is the information entered complete? 7 Review and verification of DRTB Treatment card (sample randomly). ● Completeness 8 Is there a comprehensive MDR-TB trained designated MDR-TB focal person? 9 Is surgical management of TB/DR-TB done? Set up? 10 Is this site a new-drug initiating site? Is adverse effect monitoring tools & functionality (ECG, Audiometry, chemistry machine, Xpert, CBC,RFT) 11 Are there criteria to start new drug clear and available? 12 Is there any guideline for new drug initiation? If available usability? 13 Is CRC decision & feedback timely and help full? How? ,probe; what is the challenges? 14 Is there any drug safety monitoring (DSM)? How it function, feedback ? Position------------------------------ Length of service in this position ------- years # Data Points Responses Yes/ No Other Reasons, perceptions, opinions HIV Clinic 1 Location of TB clinic-closer to ART clinic? Is the HIV clinic environment ● Ventilation(window open ● Spacious 2 Are IEC materials available? Displayed, or used during patient education? 3 Is HIV patients informed about the symptoms of TB to look and report if they have? 4 Is HIV patient screed for cough? 5 Are those patient with TB symptoms investigated for TB with Xpert? 6 Is the investigation done at ART clinic or Patient referred to TB clinic? 7 Is IPT given regularly for HIV patients? 8 Are HIV patients informed about TBIC? 123 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB 9 Are national guidelines available? ● Comprehensive HIV management ● Programmatic Management of TB,TB/HIV, DR TB ● Infection control 10 Is HIV register available in the clinic? 11 Is this register regularly filled? Sample for completeness; note gaps 12 Review and verify HIV Treatment card (sample randomly). Observe for completeness, note gaps 13 Are there the following job aides/tools in the HIV clinic? ● TB diagnostic algorithm ● AFB laboratory request form ● TB patient referral form ● Gene Xpert request form ● Gene Xpert eligibility criteria ● Recording & ● Reporting TB tools 14 If N95 & mask is available, is it used regularly? 15 Review quarterly report, trends, copies of most recent reports 16 HIV data analysis displayed on the walls Thank the participants for their time Document Review: Supportive supervision and review meetings Data item EC 2010 Planned Performed Number of review meetings conducted Supportive supervision visits conducted by RHB/Zonal/Woreda Formal written feed backs provided by RHB/Zonal Is there a supervision checklist? (verify and collect sample) • Yes, verified • Reported yes, could not verify • Not available Is there a supervision logbook? (Verify) • Yes, verified • Reported yes, could not verify • Not available How did supervisors communicate their findings? • Written feedback immediately after supervision • Written feedback after supervision 124 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB • Other (specify) RDQA conducted - verify RDQA report? EC 2010 Yes, verified Reported yes, could not verify Not available Q 1 Q 2 Q 3 Q 4 Review meeting conducted (quarter 1-4) verify Report, PPT and minute key findings? EC 2010 Yes, verified Reported yes, could not verify Not available Q 1 Q 2 Q 3 Q 4 Thank the participants for their time KEY INFORMANT INTERVIEW – FACILITIES Challenge TB Final Evaluation KIIs in Hospitals & Health Centers General Information from Hospital/Health Centre (As part of Introduction and Greetings) 1. What is the catchment area population for this facility? Population:__________ 2. What kinds of TB services are provided in your facility? a. TB Diagnostics • AFB • Xpert • CXR • FNA • LPA • Second line LPA • TB Culture and DST b. Treatment • DOT • IPT • TB HIV Collaborative service • Treatment Center for DRTB • In patient services for DRTB c. Prevention • TBIC • Health education 125 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB KII - CEO of Specialized Hospitals Region-------------------Zone---------------------Woreda/City Administration----------------- Name of Hospital/Health Center/Health Post---------------- Position------------------------------ Length of service in this position ------- years Introductory questions 1. What kinds of TB services are provided in your facility? a. Diagnosis: Smear microscopy (Light/LED), Xpert, CXR, FNA, biopsy, cytology, Cultur & DST b. DOTS (susceptible TB &MDR-TB), inpatient services EQ1. To what extent has CTB’s technical assistance to Government at different levels and Activity management approach supported the objectives of Challenge TB cooperative agreement? 2. When did CTB implement its interventions in your institution? 3. What Capacity building support you got from CTB, How was it delivered? 4. What is your perception regarding the technical assistance of CTB ? 5. Would tell us the overall contribution of CTB in improving TB services in your facility? 6. What have the main challenges been, for CTB’s work in your facility? 7. What, among these, would have taken place, had CTB not been implemented? Conclude 8. This covers the questions that we have prepared for our interview. Is there anything else you would like to add, regarding CTB, or the TB program in general? Thank the respondent for his/her time. KII – OPD Clinician, Specialized Hospital Region-------------------Zone---------------------Woreda/City Administration----------------- Name of Hospital/Health Center/Health Post---------------- Position------------------------------ Length of service in this position ------- years Introductory questions 1. What mechanisms does your facility use, to find TB cases? 2. Are there TB related job aids and current TB guidelines for TB/MDR-TB management available ? 3. Are you using national guideline algorism for TB/MDR-TB diagnosis? Probe: for what is the first line test you are using for TB diagnosis?, other additional test you need, When do you consider Antibiotic trial/not 4. Would you tell us if there is integration of TB services with other services like NCD /other 126 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB 5. Any integration of TB services in Under-five clinic; What is the contribution of this integration in case finding, contact tracing 6. When was CTB implementation program started in your institution? EQ1. To what extent has CTB’s technical assistance to Government at different levels and Activity management approach supported the objectives of Challenge TB cooperative agreement? 7. What additional contribution is made by CTB to increase screening of Presumptive TB cases? 8. Based on what you know, what activities does CTB support in your facility? In what ways have they added value to your ongoing services? Please describe for: 8.1. Capacity building: training 8.2. Health facility development, provide DOTS 8.3. Health posts available, HPs engaged in DOTS 8.4. Prevention and control of drug resistance: Trained HW, TICs, TFCs 8.5. Case finding 8.6. Onsite supervision 9. What is your perspective on the capacity building/support given by CTB? Were they frequent enough? What changes did you observe in your staff/services after the capacity building? 10. In your opinion, what is the quality of technical assistance provided by CTB? Technical skill/Capacity, supportive supervision, training facilitation…? 11. What would you consider to be the main achievements of CTB in strengthening the capacity of your facility? 12. What would you consider to be the main challenges of CTB in strengthening the capacity of your facility? 13. What, among these, would have taken place, had CTB not been implemented? EQ 2. To what extent did Challenge TB implementation approaches use international standards and proven strategies? 14. What is the contribution of CTB in supporting adherence to national standards at service delivery points? 15. What would the adherence levels be, had CTB never been there? EQ 4. What are the main achievements and challenges of CTB related to: 16. What are the main achievement & challenges of CTB in supporting improving quality of care for TB,TB/HIV and DRTB cases? 17. What are the achievements and challenges of CTB in supporting TB prevention activities including management of latent TB? 18. What, among these, would have taken place, had CTB not been implemented? Conclude 19. This covers the questions that we have prepared for our interview. Is there anything else you would like to add, regarding CTB, or the TB program in general? Thank the respondent for his/her time. KII – OPD Clinician – Hospital/Health Centre 127 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB Region-------------------Zone---------------------Woreda/City Administration----------------- Name of Hospital/Health Center/Health Post---------------- Position------------------------------ Length of service in this position ------- years EQ 1. To what extent has CTB’s technical assistance to Government at different levels and Activity management approach supported the objectives of Challenge TB cooperative agreement? 1. What TB related manuals you have 2. Would you tell us how do you diagnose TB? Are you using national algorisms? 3. What is your first line of test for TB in your facility? 4. Any integration of TB services in to other clinical services Like NCD in your institution, how does that help? 5. When was CTB implementation program started in your institution? 6. Any integration of TB services to Under Five Clinic in your facility? Screening under five children for TB, How that helps in Case finding, contact tracing and improve overall care? 7. What additional contribution is made by CTB to increase screening of Presumptive TB cases? 8. Based on what you know, what activities does CTB support in your facility? In what ways have they added value to your ongoing services? Please describe for: 8.1. Capacity building: training 8.2. Health facility development, provide DOTS 8.3. Prevention and control of drug resistance: Trained HW, TICs, TFCs 8.4. Case finding 8.5. Onsite supervision 9. What is your perception on the capacity building support given by CTB? Were they frequent enough? What changes did you observe in your staff/services after the capacity building? 10. In your opinion, what is the quality of technical assistance provided by CTB? Technical skill/Capacity, supportive supervision, training facilitation…? 11. What would you consider to be the main achievements of CTB in strengthening the local capacity of your facility? 12. What would you consider to be the main challenges of CTB in strengthening the local capacity of your facility? EQ 2. To what extent did Challenge TB implementation approaches use international standards and proven strategies? 13. What is the contribution of CTB in supporting adherence to the national standards at your facility? 14. What would the adherence levels be, had CTB never been there? EQ 4. What are the main achievements and challenges of CTB related to: 15. What are the main achievement & challenges of CTB in supporting improving quality of care for TB,TB/HIV and DRTB cases? 16. What are the main achievements and Challenges of CTB in supporting in implementation of TB prevention activities including management of latent TB ? 17. What, among these, would have taken place, had CTB not been implemented? Conclude 128 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB 18. This covers the questions that we have prepared for our interview. Is there anything else you would like to add, regarding CTB, or the TB program in general? Thank the respondent for his/her time. KII – Lab staff – Hospital/Health Center Region-------------------Zone---------------------Woreda/City Administration----------------- Name of Hospital/Health Center/Health Post---------------- Position------------------------------ Length of service in this position ------- years Introductory questions 1. What kind of TB diagnostics you have in this facility? 1.1. Microscopy, Xpert, culture LPA 1.2. Utilization of existing diagnostics? Are they functioning? 1.3. How you manage to get the lab supplies such as Slides, sputum cup, reagents, cartilages, N95 masks? From where? 1.4. How often do you have stock outs? How do you manage stock outs? 2. How many laboratory staff works in this facility? How many of them are dedicated TB lab staffs? 2.1. What is the total number of patients sent for TB test per day and what is yield of the test? What is the proportion of smear positive, smear negative, Extra Pulmonary TB and total TB cases of last quarter (see Quarterly and annual report)? 2.2. IF your Institution is one of the Gene Xpert site in this region and how long has been using this machine? 2.2.1.How many test done per day? 2.2.2.What is the yield of Xpert machine in your facility? For detection of drug sensitive TB /RR￾TB, Contamination rate, error rate?(See quarterly or annual report) 2.2.3.Where do you get the cartilages? Any Shortage? Explain 2.2.4.How many laboratory technicians are trained to use the machine? 2.2.5.Have you received any training to do easy maintenance and troubleshooting? 2.2.6.How frequently is maintenance work done on the machine? 2.2.7.What is the turnaround time? 2.2.8. Is Xpert your primary means of diagnosis? How does it help in case notification ? 2.2.9.What do you think the overall contribution of Xpert in TB diagnosis in your facility? 2.2.10. Is there regular training given to the staff? Who is giving, how frequent, is it adequate? 2.2.11. Is there trained biomedical engineer in the facility for equipment maintenance? 2.2.12. Do you need an Outside expert for maintenance of Biosafety / able to do yourself? 2.2.13. How frequent you need for TA from outside of the country and how that is done? 3. What are the main challenges 4. What is your level of Laboratory(1,2,3)? What helps to assume this level? EQ 1. To what extent has CTB’s technical assistance to Government at different levels and Activity management approach supported the objectives of Challenge TB cooperative agreement? 129 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB 5. Based on what you know, what laboratory activities does CTB support in your facility? In what ways have they added value to your ongoing services? Please describe for: 5.1. Training: Type of training, frequency, number trained 5.2. Onsite supportive supervision: How often, by whom, and the person’s qualification 5.3. Equipment maintenance & skill training (microscopy, Xpert) 5.4. Renovation 5.5. Supply management 5.6. Sample transportation, if there sample transported for XPert, Culture and DST, What is the turnaround time? 6. What is your perception on the capacity building/support given by CTB? Were they frequent enough? What changes did you observe in your staff/services after the capacity building? 7. In your opinion, what is the quality of technical assistance provided by CTB? Technical skill/Capacity, supportive supervision, training facilitation…? 8. What would you consider to be the main achievements of CTB in strengthening the capacity of your facility? 9. What would you consider to be the main challenges of CTB in strengthening the capacity of your facility? EQ 2. To what extent did Challenge TB implementation approaches use international standards and proven strategies? 10. What is the contribution of CTB to supporting adherence to national standards at your laboratory? 11. How do you assess the overall contribution of CTB in improving TB diagnosis and maintain the standards? EQ 4. What are the main achievements and challenges of CTB related to: 12. What are the main achievements & challenges of CTB in supporting improving quality of diagnosis of TB,TB/HIV and DRTB cases? 13. What are the achievements and challenges of CTB in supporting TB prevention activities including management of latent TB? 14. What, among these, would have taken place, had CTB not been implemented? Conclude 15. This covers the questions that we have prepared for our interview. Is there anything else you would like to add, regarding CTB, or the TB program in general? Thank the respondent for his/her time. KII – Clinician, TB Clinic Region-------------------Zone---------------------Woreda/City Administration----------------- Name of Hospital/Health Center/Health Post---------------- Position------------------------------ Length of service in this position ------- years Introductory questions: 130 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB 1. What are the main sources of TB patients? OPD, HIV clinic, transfer in, chronic disease clinic, private? 2. What is the total number of TB,TB/HIV & RRTB ,MDR-TB cases seen in the last quarter SP,SN,EP and sum of all, (see also TB annual, quarterly report for 2-4 quarter based on availability)? 3. What is the total number of TB patients screened for HIV? How many turned positive and linked to ART clinic? 4. What is your opinion on the adequacy of staff in your clinic? 5. What are the TB outcomes in your facility, generally? Susceptible TB (Cure, defaulter, transferred out , died, relapse) 5.1. What are your perspectives on the mortality related to TB? 5.2. And also for DR-TB treatment outcome? 6. Would you tell us about Supportive supervision mechanism? a. Check for Frequency? b. Who was coming? c. Would do have a supervision logbook? d. How did supervisors communicate their findings? 7. Would you tell about review meeting ?How frequent, who organize, ? (quarter 1-4) verify Report, PPT and minute key findings? EQ 1. To what extent has CTB’s technical assistance to Government at different levels and Activity management approach supported the objectives of Challenge TB cooperative agreement? 8. Based on what you know, what activities does CTB support in your facility? In what ways have they added value to your ongoing services? Please describe for: 8.1. Training: Type of training, frequency, number trained 8.2. Onsite supportive supervision: How often, by whom, and the person’s qualification 8.3. Renovation 8.4. Supplies (N95 and surgical masks) 9. What is your perspective on the capacity building/support given by CTB? Were they frequent enough? What changes did you observe in your staff/services after the capacity building? 10. In your opinion, what is the quality of technical assistance provided by CTB? Technical skill/Capacity, supportive supervision, training facilitation…? 11. What would you consider to be the main achievements of CTB in strengthening the capacity of your facility? 12. What would you consider to be the main challenges of CTB in strengthening the capacity of your facility? EQ 2. To what extent did Challenge TB implementation approaches use international standards and proven strategies? 13. What is the contribution of CTB in supporting adherence to national standards at your laboratory? 14. What would the adherence levels be, had CTB never been there? EQ 4. What are the main achievements and challenges of CTB related to: 15. What is the achievement and challenges of CTB in supporting contact tracing for DRTB in your facility? 131 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB 16. What are the main achievement & challenges of CTB in supporting improving quality of care for TB,TB/HIV and DRTB cases? 17. What are the achievements and challenges of CTB in supporting TB prevention activities including management of latent TB? 18. What, among these, would have taken place, had CTB not been implemented? Conclude 19. This covers the questions that we have prepared for our interview. Is there anything else you would like to add, regarding CTB, or the TB program in general? Thank the respondent for his/her time. KII – Clinician, MDR TB Treatment Initiating Center/Specialized Hospitals Region-------------------Zone---------------------Woreda/City Administration----------------- Name of Hospital/Health Center/Health Post---------------- Position------------------------------ Length of service in this position ------- years Introductory questions: 1. What are the Number of HCWs working in MDR TB Center - Number trained on PMDT; TBL and TB/HIV; IC, ? 2. What TB Infection Control activities are implemented in your facility? 2.1. How many professional & administrative staff have been trained in IP/TB IC ? Trained HCW in service outlet, How functional is the facility IP/TB IC committee? 2.2. How do you assess IC risk and develop Plan for the facility? 2.3. What infrastructure improvement (renovations) have been done? 2.4. What administrative measures are being implemented in the facility? 2.5. What is the status of utilization of N-95 and surgical mask in the centers 2.6. How is the Coordination and linkage with the TB/MDR-TB control program at zonal, regional and facility level ? 3. Do you have an MDR TB Panel? How frequently does it meet? Is there any documented minutes of the panel team? 4. Do you conduct mentoring and supervisory visits to TFCs? What is the frequency? Does the center receive technical support from RHB or FMOH or stakeholders? How do you utilize the funds sent from the TB control program? 5. Briefly explain the current status of Referral linkage with TFC and other TICs; RL and Xpert sites; RHB 6. Would you tell us your perspectives on the DRTB treatment outcomes in your facility? (See report) 7. How and from where the center receive SDLS? Where are SDLs stored? Storage situation (Space, fridge, shelf, record system, RRF use)? Any Supply interruptions for SLD 8. What is the Frequency and mechanism of dispensing to MDRTB ward; TFC ? Do you have stock of ancillary drugs and/ or opportunistic infections drugs? 132 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB 9. Would you tell us the feedback from CRC for new drug initiation and adverse effect management? Probe; How frequent they visit facility, Is there feedback adequate, what are the main challenges; what do think will be the best to improve the services? EQ 1. To what extent has CTB’s technical assistance to Government at different levels and Activity management approach supported the objectives of Challenge TB cooperative agreement? 10. Based on what you know, what activities does CTB support in your facility? In what ways have they added value to your ongoing services? Please describe for: 10.1.Training; 10.2.On site supervision 10.3.Data generation &utilization 10.4.Recording &reporting 10.5.Renovation 10.6.Supplies (N95,Surgical mask) 11. What is your perception on the capacity building/support given by CTB? Were they frequent enough? What changes have you observe in your staff/services after the capacity building? What changes have you observed in service delivery in the past five years? 12. In your opinion, what is the quality of technical assistance provided by CTB? Technical skill/Capacity, supportive supervision, training facilitation…? 13. What would you consider to be the main achievements of CTB in strengthening DRTB management in your facility? 14. What would you consider to be the main challenges of CTB in strengthening DRTB management in your facility? EQ 2. To what extent did Challenge TB implementation approaches use international standards and proven strategies? 15. What is the contribution of CTB in supporting adherence to national MDR-TB treatment standards? 16. What would the adherence levels be, had CTB never been there? EQ 4. What are the main achievements and challenges of CTB related to: 17. What is the achievement and challenges of CTB in supporting contact tracing for DRTB in your facility? 18. What are the main achievement & challenges of CTB in supporting improving quality of care for DRTB cases? 19. What are the achievements and challenges of CTB in supporting TB/DR-TB prevention activities including management of latent TB? 20. What, among these, would have taken place, had CTB not been implemented? Conclude 21. This covers the questions that we have prepared for our interview. Is there anything else you would like to add, regarding CTB, or the TB program in general? Thank the respondent for his/her time. KII – Clinician, HIV Clinic 133 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB Region-------------------Zone---------------------Woreda/City Administration----------------- Name of Hospital/Health Center/Health Post---------------- Position------------------------------ Length of service in this position ------- years Introductory questions: 1. Would you tell us please what TB related activities are being done in this clinic? 2. Would you screen HIV patients for TB symptoms? How many HIV patients screened for TB symptoms? How many symptom positive and sent for TB investigation and turned positive/HIV? 3. What is the practice of IPT in your clinic? How many eligible cases receive IPT? (See also report) EQ 1. To what extent has CTB’s technical assistance to Government at different levels and Activity management approach supported the objectives of Challenge TB cooperative agreement? 4. Based on what you know, what activities does CTB support in your facility? In what ways have they added value to your ongoing services? Please describe for: 4.1. Training 4.2. On site supervision 4.3. Recording & reporting 4.4. Supplies (N95, Surgical mask, other) 5. What is your perspective on the capacity building/support given by CTB? Were they frequent enough? What changes have you observe in your staff/services after the capacity building? What changes have you observed in service delivery in the past five years? 6. In your opinion, what is the quality of technical assistance provided by CTB? Technical skill/Capacity, supportive supervision, training facilitation…? 7. What would you consider to be the main achievements of CTB in strengthening the capacity of your facility? 8. What would you consider to be the main challenges of CTB in strengthening the capacity of your facility? EQ 2. To what extent did Challenge TB implementation approaches use international standards and proven strategies? 9. What is the contribution of CTB in supporting adherence to national MDR-TB treatment standards? 10. What would the adherence levels be, had CTB never been there? EQ 4. What are the main achievements and challenges of CTB related to: 11. What is the achievement and challenges of CTB in supporting screening of HIV patients for TB symptoms in HIV clinic? 12. What are the main achievement & challenges of CTB in supporting improving quality of care for TB/HIV collaborative activities? 13. What are the achievements and challenges of CTB in supporting TB prevention activities for HIV patients including management of latent TB? 14. What, among these, would have taken place, had CTB not been implemented? 134 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB Conclude 15. This covers the questions that we have prepared for our interview. Is there anything else you would like to add, regarding CTB, or the TB program in general? Thank the respondent for his/her time. KII – Pharmacy Staff Region-------------------Zone---------------------Woreda/City Administration----------------- Name of Hospital/Health Center/Health Post---------------- Position------------------------------ Length of service in this position ------- years Introductory questions: 1. Would you please tell us from where you get anti-TB medication & supplies? (FLD, SLD, cartridges, N95 masks, ancillary drugs) Where are SLDs stored, Storage situation (Space, fridge, shelf, record system, RRF use? 2. How do you manage your stock of anti-TB drugs? How frequent are stockouts? How about SLDs? 3. Would you tell us the Availability and adequacy of ancillary drugs and other supplies? 4. Would you get adequate INH (IPT) drugs and availability in the stock? 5. Does your health facility get all requested TB drugs according to its request? EQ 1. To what extent has CTB’s technical assistance to Government at different levels and Activity management approach supported the objectives of Challenge TB cooperative agreement? 6. Based on what you know, what capacity building does CTB support in your facility? In what ways have they strengthened drug supply management? 7. What is your perception on the capacity building/support given by CTB? Were they frequent enough? What changes have you observe in your staff/services after the capacity building? What changes have you observed in service delivery in the past five years? 8. What would you consider to be the main achievements of CTB in strengthening drug supply management in your facility? 9. What would you consider to be the main challenges of CTB in strengthening drug supply management in your facility? EQ4. What are the main achievements and challenges of CTB? 10. What are CTB’s achievements in supporting case management? Probe for – quality of care – TB, MDR TB, TB HIV, retrieving LTFU, improving adherence? 11. What were the challenges in supporting drug supply management? 12. What, among these, would have taken place, had CTB not been implemented? Conclude 13. This covers the questions that we have prepared for our interview. Is there anything else you would like to add, regarding CTB, or the TB program in general? 135 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB Thank the respondent for his/her time. KII – HMIS Staff Region-------------------Zone---------------------Woreda/City Administration----------------- Name of Hospital/Health Center/Health Post---------------- Position------------------------------ Length of service in this position ------- years Introductory questions: 1. How is TB, TH/HIV/ and DR-TB data captured in HMIS ? How is the data used? (See report) 2. Is there a separate HMIS unit/focal person in the facility? 3. Is e-HMIS utilized for reporting TB & TB/HIV performance? 4. Do TB & HIV/ART focal persons work closely with the HMIS unit in monthly and quarterly TB/HIV report compilation and analysis? (Observe evidence/practice) 5. Is there functional performance monitoring team in the facility? Observe minute. Specify the frequency of meetings: a) Weekly b)Monthly c) Quarterly d)Annually e) irregular f)none 6. Is the TB focal person a member of this team? 7. Is the latest facility level TB,TB/HIV and MDR-TB performance data analyzed and displayed? EQ 1. To what extent has CTB’s technical assistance to Government at different levels and Activity management approach supported the objectives of Challenge TB cooperative agreement? 8. What is contribution of the CTB in improving the TB data system and its use, in your clinic? 9. Based on what you know, would you please tell us the capacity building support delivered by CTB in data capturing and utilization? 9.1. How it was done and how significant was it? 9.2. What changes have you seen in data capture and management due to CTB support? 10. In your opinion, how do you assess the quality of the technical assistance of CTB staff in data management? EQ4. : What are the main achievements and challenges of CTB? 11. What would you consider to be the main achievements of CTB in supporting data management in your facility 12. What would you consider as the main Challenges that CTB faced in strengthening data management in your facility? 13. What, among these, would have taken place, had CTB not been implemented? Conclude This covers the questions that we have prepared for our interview. Is there anything else you would like to add, regarding CTB, or the TB program in general? Thank the respondent for his/her time. 136 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB KII – Health Extension Worker Region-------------------Zone---------------------Woreda/City Administration----------------- Name of Hospital/Health Center/Health Post------------------------- Position------------------------------Length of service in this position ------- years Introductory questions: 1. Please tell us the about your TB related activities. 2. How do you identify Presumptive TB cases? 3. How would you do Contact Screening? 4. Do you have register for contact tracing? Are you providing DOTs? 5. Do you have adequate anti-TB drug if your center provides DOT? Have you had any stock out? 6. Would you tell us referral system for symptomatic patients, Registration of TB patients 7. How you been trained on sputum handling? Probe: Slide fixing EQ 1. To what extent has CTB’s technical assistance to Government at different levels and Activity management approach supported the objectives of Challenge TB cooperative agreement? 8. Based on what you know, would you please tell us the capacity building support delivered by CTB? 8.1. What were the training topics and how frequent were they? 8.2. What changes did you take up in your services as a result of the training 8.3. What about provision of supplies such as N95 masks EQ 2. To what extent did Challenge TB implementation approaches use international standards and proven strategies? 9. What is the contribution of CTB in supporting the adherence to national standards for TB services? 9.1. Contact screening 9.2. Referral of symptomatic persons for TB investigation? 9.3. Data processing EQ4: What are the main achievements and challenges of CTB? 10. What are CTB’s achievements in supporting identification of Presumptive TB cases? Probe for – retrieving LTFU, improving adherence? 11. What are CTB’s achievements in supporting implementation of TB prevention activities? Probe for TBIC, IEC? 12. What were the main challenges in implementation of CTB activities in your facility? 13. What, among these, would have taken place, had CTB not been implemented? Conclude 14. This covers the questions that we have prepared for our interview. Is there anything else you would like to add, regarding CTB, or the TB program in general? Thank the respondent for his/her time. Challenge TB Final Evaluation 137 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB KIIs in Facilities (Hospitals & Health Centers) where there is best practice implemented KII - Health facility with best practice Region-------------------Zone---------------------Woreda/City Administration----------------- Name of Hospital/Health Center/Health Post---------------- Position------------------------------ Length of service in this position ------- years Introductory questions 1. What kinds of TB services are provided in your facility? a. Diagnosis: Smear microscopy (Light/LED), Xpert, CXR, FNA, biopsy, cytology b. DOTS (susceptible TB &MDR-TB), inpatient services EQ 5. To what extent are the activity’s methodologies, interventions, and management setting the stage for future sustainability, and ownership of project outputs and outcomes? 2. Would you tell us, what best practice/innovation CTB activity implemented in this facility? Probe; For how long? Where was this innovation designed? (In country or adapted from elsewhere) 3. What capacity building activities have you done related to this innovation? 4. How has it contributed to the overall TB services? Have there been any change in the trends of related indicators? 5. Is there any harm or disadvantage to the service providers or patients or the environment in implementing this innovation? 6. What challenges have you faced in implementing it/using this technology? How did you address those? 7. What lessons have been learned from its implementation in this site? 8. Has the data from this innovation been included in reporting – project, program? 9. What do you think regarding your institution’s capacity at this stage, to effectively run and maintain best practice implemented without CTB? What challenges do you foresee? 10. What is your perception on the scalability of this best practice implemented by CTB? 11. Do you see any potential to introduce this innovation in non-TB services? Conclude 12. This covers the questions that we have prepared for our interview. Is there anything else you would like to add, regarding CTB, or the TB program in general? Thank the respondent for his/her time. EXIT INTERVIEW – PATIENTS/GUARDIAN OR PARENT Challenge TB Final Evaluation Exit Interview with Patient/Guardian or patient Facility Name: ----------------------------------Date: ----------------------------------------- Name of interviewer: ---------------------------- Patients Age: -------------------------------- Sex: Male --------------Female--------------- 138 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB General questions and care seeking 1. How far do you live from here? • More than 10 km • 5 to 10 km • Less than 5 km 2. How long did it take for you to come here? • More than 2 hours • 1 to 2 hours • Less than an hour 3. Is this your first time to get to this facility? • Yes • No 4. What was the major complaint/symptom you (or in your ward) come here for? • Cough • I was told I have TB / Referred by another facility or HEW • Other symptoms 5. Who first suspected/considered that you (or you ward) might have TB? • In this facility (OPD/Triage/other) • Came here with referral from HEW/other facility • Other……………………………. 6. How long did it take, after your symptoms began, for you (or your ward) to be diagnosed with TB? • Less than a month • 1-2 months • More than 2 months 7. Could you tell us about your experience you went through, from the time your (or your ward’s) symptoms began, until you (or your ward) were diagnosed with TB? Probe for: Number of places visited: Type of places of visited: Cost incurred: Patient Satisfaction/Quality of Care for TB 8. Could you tell us what you know about TB? Probe for: symptoms, spread, prevention, treatment, its duration and side effects etc. 9. What is your perception of services in this facility? Probe for: Opening hours of facility Waiting time Staff attitudes Privacy and comfort Allowing you to ask questions/concerns Addressing concerns Providing all relevant information about your illness Cleanliness of premises 139 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB 10. Could you describe for us the advice you (and your ward) received regarding your illness? Probe for: Advise on importance of continuing TB treatment Advise on cough hygiene Advise on bringing family members for screening 11. Please tell us about your experience in receiving TB treatment for yourself (or for your ward). Probe for: Which month of treatment are you in? How is/was the drug administered to you (or your ward) in the first two months of treatment – Swallow medicine every day at health facility (DOT), took medicines with the help of treatment supporter, collect drug daily and swallow it yourself (or give your ward) at home, collect drug for a few days or weeks and return for refill. 12. When and where did you first learn about TB? Probe for: work of HEW in his/her community Perspectives on improving access 13. To what extent, do you think people in your community know about TB and how and where to get tested and treated for it? 13.1.What can be done to further improve the awareness of people 14. What do you think can be done to improve the awareness of people on TB in your community? 15. What can you recommend on how to get people with symptoms of TB get tested early enough? Probe for – awareness, mobilization, location of facilities/distance, package of services offered, lab services, staffing, DOT and treatment support 16. What do you think you can do for others with TB, to get proper care early enough? Thank the patient/guardian or parent for his/her time 140 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB ANNEX V: INFORMATION SOURCES Respondents for Key Informant Interviews 1. NTP M&E Focal Person 2. NTP GF Seconded Staff 3. NTP USAID Seconded staff 4. NTP Manager 5. VHS Incharge 6. EPS Director 7. ETS Director 8. CTB M&E Director 9. CTB Ops Director 10. CTB Regional Director MSH 11. CTB Sr Program Advisor WHO 12. GLRA Director 13. National PSA (FMOH seconded staff) 14. EPHI Director 15. PIH Director 16. GHC Director 17. Organic Health Director 18. ALERT HMIS 19. ALERT Training Center 20. ALERT Director 21. St. Peter’s Director and Asst Director 22. St. Peter’s Pharmacy 23. Oromia Regional Health Bureau 24. Adama Regional Lab Head 25. PSA Adama Hub 26. West Arsi Zone Health Department 27. Arsi Negele Health Center head 28. District health office 29. Arsi Negele Health Center HMIS 30. Health Extension Worker (kersa Elala HP) 31. CTB West Arsi Zone Cluster Coordinator 32. Sidama Zone Health Department 33. Abaye Health Extension Worker 34. Wondo Genet District 35. CTB Sidama Zonal Coordinator 36. CTB regional Office SNNP 37. SNNP Regional Health Bureau TB focal person 38. Hawas PSA Hub 39. Reach Ethiopia 40. Addis Ababa RHB TB Coordinators 41. Addis Ketema Health Center head 42. Addis Ketema sub city health department 43. Addis Ketema Health Center HMIS 44. Health Extension worker Urban HEW Addis Ketema 45. CTB Coordinators Addis Ababa 46. CTB Coordinators Oromia 141 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB 47. Jinka Hospital Director 48. Jinka Hospital HMIS 49. Amhara CTB Team (5) 50. Amhara Regional TB Team (2) 51. Amhara Regional Lab - APHI (4) 52. Amhara Awi Zonal CTB Team 53. Amhara Awi Zonal TB team (2) 54. Amhara Injibara Facility Head 55. Amhara Injibara HMIS Focal person 56. Dangila Woreda TB focal 57. Dangila HP HEW 58. Regional TB case team (4), Tigray 59. CTB Regional team, Tigray (5) 60. Zonal CTB & Lab, Tigray 61. Woreda TB focal, Adigrat 62. HMIS focal (HIT), Adigrat HEALTH CENTER & Woreda 63. HEW Adigrat HP 64. Regional CTB team Gambella 65. Regional TB case team Gambella 66. Facility Head, Gambella Hospital 67. HMIS, Gambella Hospital Respondents for Key Informant Interview and Observation 1. ALERT OPD 2. ALERT TB Clinic 3. ALERT TB Lab 4. ALERT HIV Clinic 5. ALERT Pharmacy 6. ALERT Culture Lab 7. St. Peter’s Laboratory 8. St. Peter’s OPD 9. St. Peter’s HMIS 10. St. Peter’s TB clinic (MDR-TB) 11. Arsi Negele Health Center OPD 12. Arsi Negele Health Center Laboratory 13. Arsi Negele Health Center TB Clinic 14. Arsi Negele Health Center Pharmacy 15. Arsi Negele Health Center ART Clinic 16. Hawassa Regional Lab 17. Yiragelem Lab, TB clinic, MDR TB 18. Addis Ketema Health Center OPD 19. Addis Ketema Health Center Laboratory 20. Addis Ketema Health Center TB Clinic 21. Addis Ketema Health Center Pharmacy 22. Addis Ketema Health Center ART Clinic 23. Jinka Hospital Laboratory 24. Jinka Hospital OPD 25. Jinka Hospital Pharmacy 26. Jinka Hospital MDR TB clinic 142 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB 27. Jinka Hospital TB Clinic 28. Amhara Injibara TB clinic person 29. Amhara Injibara Pharmacy head 30. Amhara Injibara Labin charge 31. Amhara Injibara TIC clinician 32. Amhara Injibara HIV focal person 33. Regional Lab director, Tigray 34. Mekele best practice and MDR TB, Tigray 35. TB focal, Adigrat HEALTH CENTER 36. HIV focal, Adigrat HEALTH CENTER 37. Lab focal, Adigrat HEALTH CENTER 38. Pharmacy focal, Adigrat HEALTH CENTER 39. OPD focal, Adigrat HEALTH CENTER 40. Regional Lab Gambella 41. OPD Clinician Gambella Hospital 42. HIV focal Gambella Hospital 43. TB focal Gambella Hospital 44. Pharmacist Gambella Hospital List of TB patients interviewed # Age (in years) Sex Facility/Location 1 25 M Aris Negele Health Center 2 20 M Aris Negele Health Center 3 20 F Injibara Health Center 4 37 M Injibara Health Center 5 17 M Jinka Hospital 6 30 M Jinka Hospital 7 30 F Adigrat Health center 8 39 F Adigrat Health center 9 21 F Gambella Hospital 10 30 F Gambella Hospital 11 39 M Addis Ketema Health Center 12 35 F Addis Ketema Health Center 143 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB ANNEX VI: DISCLOSURES OF CONFLICT OF INTEREST 144 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB 145 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB 146 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB 147 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB 148 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB ANNEX VII: EVALUATION TEAM MEMBER PROFILES AND CVS Dr. Beulah Jayakumar, Team Leader, is a public health consultant with 20 years' experience in the design, implementation, and monitoring & evaluation of TB, maternal, newborn and child health, nutrition, and reproductive health programs. She has served as an international consultant since 2009, providing strategic technical leadership for high-quality programming, and MEL in health and development and has led the design and implementation of evaluations of large-scale multi-site programs, assessments, population surveys, research initiatives, and program design workshops in Ethiopia, Kenya, Zambia, Uganda, Malawi, Tanzania, South Sudan, Sierra Leone, Niger, Namibia, Cambodia, India, and Bangladesh. She has expertise in developing and evaluating conceptual, logical, and performance monitoring frameworks and theories of change; designing, conducting, and analyzing qualitative studies including barrier analysis, formative and operations research, behavior change programming, health systems strengthening approaches, and data quality audits; and quantitative (household and facility) surveys. She has led the development of national training curricula, high-value grant proposals. She was part of the team that assessed the effectiveness of Global Fund-supported programs in Ethiopia (2010 and 2017) and Namibia (2011). She holds an MD from Bharatiar University, Coimbatore, India and a Masters in Family Medicine from the National Board of Examinations, New Delhi, India. Dr. Ezra Shimeles holds an MD from Addis Ababa University and a Masters’ degree in Public Health from the Royal Tropical Institute (KIT), Amsterdam. He has over two decades of national and international experience in TB control. He served as Country Director for TBCARE/USAID project in Ethiopia. Prior to that, he served as a consultant for TB control, WHO Ethiopia, TB/HIV advisor, Columbia University-International Centre for AIDS Care and Treatment project, as an international consultant TB project monitor for the FIDELIS (Fund for Innovative DOTS Expansion through Local Initiatives to Stop TB) project of the International Union for TB and Lung Diseases, and as a WHO consultant for human resource development for the National TB Program of Ethiopia. He has authored peer-reviewed publications on TB and TB HIV. Dr. Amsalu Bekele holds an MD from Addis Ababa University and a degree in Pulmonary Medicine from St. John's Medical College, India. He currently serves as the Head of Research for the Division of Pulmonary and Critical Care Medicine, Department of Internal Medicine, College of Health Sciences, Addis Ababa University and as consultant pulmonologist at Black Lion Hospital, Addis Ababa. He is currently President of the Ethiopian Thoracic Society and Vice Chair of the TB Research and Advisory Committee. He is also Chair of the Clinical Review Committee for new drugs (in treatment of MDR TB) and a member of the Technical Working Group for TB, TB HIV and MDR TB of the FMoH. He has authored several peer-reviewed publications on TB and pulmonary medicine. Dereje Mamo Tsegaye has more than 19 years’ work experience and skills in planning, managing, leading, and coordinating projects in health information system, research monitoring and evaluation, and capacity building. He currently serves as Consortium Program Manager of the HPF II Program, led by Save the Children. In this capacity, he leads project planning, setting performance targets, and ensures adherence to technical standards. Prior to this, he was Health Systems Capacity Building Manager in South Sudan, monitoring & evaluation consultant with FHI 360, and as Director of the Policy, Planning, Monitoring and Evaluation Directorate, FMoH, among other positions. He holds a Masters’ degree in Public Health Monitoring and Evaluation from Jimma University. 149 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB ANNEX VIII: OPERATIONS RESEARCH AND PRESENTATIONS Operations Research Conducted by CTB 1. The magnitude of active tuberculosis disease among healthcare workers in Amhara region, Ethiopia (Ongoing) 2. Feasibility and outcomes of implementing 99DOTS as alternative TB treatment adherence support strategy in Ethiopia (Ongoing) 3. Routine validation of a simplified methodology for stool sample testing by GeneXpert MTB/Rif to increase access to bacteriological confirmation of childhood pulmonary tuberculosis in the primary health care setting in Ethiopia (Ongoing) 4. Geospatial Patterns of Drug Resistant Tuberculosis and Associated Risk Factors in Addis Ababa, Ethiopia 5. Quality of TB Care Service in Public Health facilities, Somali Region: A Facility Based Cross Sectional Study 6. Evaluation of the postal service for referral of specimen of drug resistance tuberculosis in Amhara region, Ethiopia 7. Intensive phase treatment outcome and contributing factors among patients treated for MDR￾TB in Ethiopia 8. Assessment of Sputum Specimen Quality & Associated Factors in Acid Fast Bacilli Smear Microscopy among Presumptive pulmonary Tuberculosis patients in Harari Regional State, Harar, Ethiopia 9. Modeling the patient and health system impact of alternative Xpert MTB/Rif algorithms for the diagnosis of PTB in Addis Ababa, Ethiopia 10. Missed pulmonary tuberculosis: a cross sectional study in the general medical inpatient wards of a large referral hospital in Ethiopia 11. Effectiveness of isoniazid preventative therapy in reducing incidence of active tuberculosis among people living with HIV/AIDS in public health facilities of Addis Ababa, Ethiopia: a historical cohort study 12. Firefighting in the MDR-TB control: household infection control status in Southern Region of Ethiopia 13. Mycobacterium tuberculosis infection in students of public higher education institutions of eastern Ethiopia: Molecular epidemiology and drug resistance patterns 14. Prevalence and Associated Risk Factors of Pulmonary Tuberculosis and Drug Resistant Tuberculosis among Prisoners in Benishangul Gumuz Region, Ethiopia 15. Evaluating the feasibility of online self-screening approach for tuberculosis case finding among University students in Ethiopia Presentations made by CTB Title Type of presentation In-patient wards as source of missed pulmonary TB cases: a case study in Ethiopia Oral Modeling demonstrates that targeted use of Xpert MTB/RIF is a cost effective and affordable option for Tuberculosis diagnosis in Addis Ababa, Ethiopia Poster Patient and provider delay in pulmonary tuberculosis patients: a cross-sectional study in Addis Ababa city, Ethiopia Poster Improved access to MDR-TB services via decentralized service delivery model in Amhara and Oromia regions of Eth. Oral 150 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB Burden of MDR TB among contacts of MDR TB cases: results from routine program implementation in two Regions of Ethiopia Oral Incidence of tuberculosis among health workers at public healthcare facilities in two regions of Ethiopia Oral Towards Zero Anti-TB Drugs Stock-Out: Focusing on System Strengthening Brought a Difference in two regions of Ethiopia Oral Electronic laboratory specimen (eSpecimen) referral system in Ethiopia: a feasible approach Oral Task shifting in TB laboratory service delivery: the experience of non-laboratory technicians in two regions of Ethiopia Oral Narrowing the gap between cure and treatment success over four years: sign of improved quality of TB treatment follow up E-poster Improved Tuberculosis contact investigation and Isoniazid Preventive Therapy among under-5 children in two regions of Ethiopia E-poster The yield of TB screening in over 16 million outpatient department visitors in two regions of Ethiopia Poster The yield of tuberculosis contact screening in Ethiopia: comparing between contacts of bacteriologically confirmed and clinically diagnosed index TB cases Poster Correlation of childhood TB cases' notification with bacteriologically confirmed pulmonary TB case notification: results from two regions of Ethiopia Poster Geographic variation of tuberculosis case notification in two regions of Ethiopia and its implication on TB program mgt Poster Factors affecting treatment outcome of childhood tuberculosis in two regions of Ethiopia Poster Improved TB/HIV collaborative activities via health system strengthening in two regions of Ethiopia Poster Tuberculosis and pregnancy in a cohort of women receiving antiretroviral therapy in Ethiopia Poster TB, HIV and diabetes mellitus tri-directional screening in four hospitals of Ethiopia Poster Risk scoring system and symptom-based screening as initial steps for detecting diabetes mellitus in TB and HIV clinics in Ethiopia Poster Yield of Tuberculosis among children with presumptive TB using GeneXpert MTB/RIF assay in two regions of Ethiopia Poster Experiences and challenges in the scale up of GeneXpert services in Oromia and Amhara regions, Ethiopia Poster The pattern of rpoB gene mutations from Mycobacterium tuberculosis isolates of pulmonary TB patients using Xpert® MTB/RIF in Ethiopia Poster Survival and Predictors of Mortality among Multi-drug Resistant Tuberculosis Patients on treatment in Two Regions of Ethiopia Poster Seasonality in Tuberculosis case notification rate and its implications for developing season-based case finding strategies in Ethiopia Oral Blended learning for capacity building of health care workers in TB-HIV: results of a comparative study in Ethiopia Oral Factors associated with unfavorable treatment outcomes among MDR-TB patients treated at a tertiary hospital in Tigray Region, northern Ethiopia Oral Drug resistant TB burden among contacts of drug resistant TB patients: results from routine program implementation in three regions of Ethiopia Oral Cold chain vehicle specimen transportation system for TB culture improved quality of laboratory service in Ethiopia Oral Treatment outcome patterns among exclusively health facility level versus community/health post level treated Tuberculosis patients in Ethiopia Oral Interventions led to improved contribution of community Tuberculosis (TB) care to TB case notification in two regions of Ethiopia Oral Setting the National Tuberculosis Research Agenda: The experience of Ethiopia Oral Sensitization for clinicians followed by weekly service monitoring contributes to improved uptake of GeneXpert service in Tigray Region, Ethiopia. Poster 151 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB Clinical, programmatic and epidemiologic significance of wide-scale implementation of tuberculosis contact investigation in Ethiopia Poster Improvements in isoniazid preventive therapy uptake rates in under-five children in Ethiopia: results from a five-year program implementation experience Poster Comparison of randomized blinded rechecking for Fluorescence light-emitting diode and Ziehl Neelson microscopy in three regions of Ethiopia Poster The use of same-day, spot-spot sputum testing leads to improved pre-diagnosis retention of Tuberculosis patients in public health facilities in Ethiopia Poster TB Case Notification Rates among various key population groups in Ethiopia Poster Comparable yield of TB in children among presumptive Tuberculosis patients who underwent GeneXpert testing in Ethiopia Poster Under diagnosis of drug resistant childhood TB compared to drug sensitive counterparts in three big regions of Ethiopia Poster Using GxAlert Report in Troubleshooting the Problems Encountered in Remote GeneXpert Laboratories Poster GeneXpert utilization rate and associated factors for suboptimal test uptake in South Nations Nationalities & Peoples Region, Ethiopia Poster TB and diabetes mellitus in high burden settings: implementation and research experiences from Asian, African, Caribbean and Latin American countries Symposium The experience of using video-conferencing technology for improving access to life￾saving services for TB patients in remote parts of Ethiopia Oral Family matrix-guided HIV and tuberculosis (TB) case finding using index patients as entry point at four urban sites in Ethiopia Oral Tuberculosis Contact Investigation and its Yield in <5 year Children over a Five Year Period in Amhara region, Ethiopia, 2013-2017 Oral "The experience of innovative specimen transportation and GeneXpert expansion in Ethiopia" Oral "Implementation of the childhood TB road map addressed the missed childhood TB cases in Addis Ababa" Oral "Xpert/RIF Utilization Rate Improved through Strong Monitoring and Innovative Interventions in Tigray Region, Northern Ethiopia" Oral "Impact of Strong Support in Improving Drug Supply Management of TB and Related Supplies in Tigray, Ethiopia" Oral Slide Fixing and Referral improved access to diagnosis: experiences of Oromia Region in Ethiopia Oral Tuberculosis (TB)-related knowledge of the general population: results from a cross￾sectional survey in 11 regions of Ethiopia E-poster The yield of tuberculosis mass screening among refugees in western Ethiopia Poster Xpert MTB/RIF implementation leads to more accurate diagnosis and rational child antibiotic use in in south-east Ethiopia Poster Satisfaction of tuberculosis (TB) patients by the service provided by the health system Poster A Decline in the Number of Notified TB Cases in the Last Seven Years Could Be Ascribed to the Decline in HIV Infection Rates in Ethiopia Poster Increasing Trend of Drug-resistant TB among New TB Patients and its implications in Two Large Regions of Ethiopia Poster Isoniazid preventive therapy uptake in <5 year Children using TB contact investigation as an entry point over a Five Year Period in Amhara region, Ethiopia, 2013-2017 Poster Sputum smear slide referral by non-laboratory professionals as an interim solution in remote areas, Amhara Region, Ethiopia Poster Factors associated with stigmatizing attitude towards tuberculosis (TB) patients in the general population of Ethiopia Poster Yield of retrospective versus prospective tuberculosis (TB) contact investigations: survey findings in four Ethiopian towns Poster Perceptions of family members of tuberculosis (TB) patients towards TB associated stigma in Ethiopia Poster 152 Ethiopia Performance Monitoring and Evaluation Service Final Performance Evaluation Report, Challenge TB Tuberculosis-related knowledge of family members of tuberculosis (TB) patients in Ethiopia Poster "The Contribution of Mass Screening to Overall Case Finding in 10 Prisons in Tigray Region, Northern Ethiopia" Poster "Coordination and partner support improved the performance of tuberculosis control in Tigray, Ethiopia" Poster Trends in treatment outcomes among MDR-TB patients in Tigray Region, Ethiopia Poster Targeted interventions led to improvements in TB case finding among selected mining shafts in six high priority districts in Ethiopia Poster GxAlert field implementation experience of Ethiopia Poster Demographic characteristics and geographic distribution of drug resistant TB patients enrolled in six treatment initiation centres of SNNPR, Ethiopia. Poster "Improved GeneXpert utilization through capacity building activities in Southern Nations Nationalities and Peoples Region, Ethiopia Poster Contribution of Mass Screening to Case Finding in USAID/CTB supported prisons in SNNPR, Ethiopia Poster "A Novel community based tuberculosis case finding mentorship intervention to find the Poster High-impact interventions for tuberculosis case finding: global updates and successful country examples: The Experience of Implementing High Priority Case Finding Strategies in Ethiopia Symposium The experience of contact investigation in Ethiopia Symposium U.S. Agency for International Development Entoto Street PO Box 1014 Addis Ababa, Ethiopia