HEAL TB ETHIOPIA MID-TERM EVALUATION 1 Evaluation: USAID/ETHIOPIA: HELP ETHIOPIA ADDRESS THE LOW TB PERFORMANCE (HEAL-TB) MID TERM EVALUATION NOVEMBER 2014 This publication was produced for review by the United States Agency for International Development. It was prepared by Fabio Luelmo, Abera Bekele Leta, Daniel Gemechu Datiko and Janet Phillips. 2 HEAL TB ETHIOPIA MID-TERM EVALUATION USAID/ETHIOPIA: HELP ETHIOPIA ADDRESS LOW TB PERFORMANCE (HEAL-TB) MID-TERM EVALUATION Final Draft: December 16, 2014 AID-663-O-14-00052 DISCLAIMER The views of the authors expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. HEAL TB ETHIOPIA MID-TERM EVALUATION 3 ACKNOWLEDGEMENTS The members of the evaluation team acknowledge the support received from the Ministry of Health, the country health staff at all levels, and the staff from HEAL-TB and from ABH Associates, as well as the guidance from USAID/Ethiopia and from IBTCI to carry out this evaluation. HEAL TB ETHIOPIA MID-TERM EVALUATION 1 TABLE OF CONTENTS ACKNOWLEDGEMENTS ........................................................................................................... 3 TABLE OF CONTENTS .............................................................................................................. 1 ACRONYM LIST ........................................................................................................................... 1 EXECUTIVE SUMMARY.............................................................................................................. 3 Evaluation Purpose and Questions ..................................................................................................... 3 Findings...................................................................................................................................................... 4 Conclusions .............................................................................................................................................. 6 Recommendations .................................................................................................................................. 7 I. INTRODUCTION ................................................................................................................ 9 II. BACKGROUND ................................................................................................................... 9 III. SCOPE AND METHODOLOGY .................................................................................... 12 Evaluation Purpose ...............................................................................................................................12 Key evaluation questions and main methods .................................................................................12 IV. STRENGTHS AND LIMITATIONS OF THE EVALUATION .................................... 14 V. FINDINGS ........................................................................................................................ 14 1. Overarching findings ........................................................................................................................14 2. Strategy and management ..............................................................................................................16 2.1 Case detection and diagnosis ......................................................................................................17 2.3. Treatment and drug supply .........................................................................................................22 2.4 MDR-TB...........................................................................................................................................25 2.5. TB/HIV .............................................................................................................................................28 2.6. Infection control............................................................................................................................29 2.7. Community-based TB care .........................................................................................................30 2.8. Supervision, monitoring and evaluation ...................................................................................31 3. Finance ................................................................................................................................................32 VI. CONCLUSIONS AND LESSONS LEARNED ................................................................. 32 VII. RECOMMENDATIONS ................................................................................................... 35 ANNEX A. STATEMENT OF WORK ...................................................................................... 37 ANNEX B. TABLES AND FIGURES ........................................................................................ 48 ANNEX C. SCHEDULE OF FIELD VISITS .............................................................................. 53 ANNEX D. MAIN PERSONS INTERVIEWED ........................................................................ 54 ANNEX E. INSTITUTIONS VISITED ..................................................................................... 56 2 HEAL TB ETHIOPIA MID-TERM EVALUATION ANNEX F. DATA COLLECTION INSTRUMENTS ............................................................... 57 ANNEX G. REFERENCES ......................................................................................................... 62 List of Figures and Tables Figure I. Reported TB cases in 1000s. Ethiopia, 2010-2013 ................................................................................. 10 Figure 2. Trend of all forms of TB cases notified in the Amhara Region 2002-2006 EFY .............................. 17 Figure 3. Number of new TB cases (all forms) reported in HEAL-TB original 10 zones, 2011-2013 ......... 18 Figure 4. Sites with external quality asessment of microscopy. HEAL-TB, July 2011-September 2014 ...... 21 Figure 5. Treatment outcomes in new smear positive PTB in the HEAL-TB original zones, 2010-2014. .. 23 Figure 6. Treatment outcomes in new smear positive PTB in the HEAL-TB new zones, quarter 4/2013 to quarter 2/2014. ................................................................................................................................................................. 23 Figure 7. Percent of registered TB patients tested for HIV through HEAL-TB, Oct. 2012-Sep. 2013, and target (90%). ...................................................................................................................................................................... 29 Table 1. TB case detection performance in one of the visited health center in Amhara ............................... 18 Table 2. Annual workload of TB microscopy, North Showa zone, 2006 EC (population 2.3 million) ....... 20 Table 3. Number of samples transported and confirmed MDR-TB. 2011-14. .................................................. 27 Table 4. Performance of MDR-TB diagnosis and treatment in Amhara Region, 2005-2006 ......................... 27 Table 5. HEAL-TB MDR and HIV activity, October 2013 - September 2014 .................................................. 28 Table 6. Yield of enhanced house to house active case finding in selected zones, October 2014 .............. 31 HEAL TB ETHIOPIA MID-TERM EVALUATION 1 ACRONYM LIST ACSM Advocacy, communication, and social mobilization AFB Acid-fast bacillus AIDS Acquired Immunodeficiency Syndrome ALERT All Africa Leprosy, Tuberculosis, Rehabilitation and Research Training Centre ART Antiretroviral therapy BCG Bacillus Calmette-Guérin (vaccine against tuberculosis) CBTC Community-based TB care CDS Community DOTS Supporter CDC Centers for Disease Control and Prevention CDR Case detection rate CNR Case notification rate DOT Directly observed therapy DOTS WHO-recommended TB strategy for TB control DSM Drug supply management DST Drug Susceptibility Testing EC Ethiopian calendar EFY Ethiopian fiscal year EQA External quality assurance EPHI Ethiopian Public Health Institute FLD First-line drugs FMOH Federal Ministry of Health GDF Global Drug Facility GFATM Global Fund to Fight AIDS, Tuberculosis and Malaria GLRA German Leprosy Relief Association GOE Government of Ethiopia HC Health Center HEAL-TB Help Ethiopia Address the Low TB Performance Project HEP Health Extension Program HEW Health Extension Worker HIV Human Immunodeficiency Virus HMIS Health Management Information System HP Health Post HSDP Health Sector Development Plan HSS Health systems strengthening IC Infection control IPLS Integrated Pharmaceutical Logistics System KAPTLD Kenya Association for the Prevention of Tuberculosis and Lung Disease LED Light-emitting diode MDR-TB Multidrug-resistant tuberculosis MSH Management Sciences for Health NTP National Tuberculosis Program NHIMS National Health Information Management System OPD Outpatient department PEPFAR President’s Emergency Plan for AIDS Relief PFSA Pharmaceuticals Fund and Supply Agency PHCU Primary Health Care Unit PMDT Programmatic management of drug-resistant TB 2 HEAL TB ETHIOPIA MID-TERM EVALUATION PPM Public-private mix PTB+ Smear-positive pulmonary tuberculosis EQA External quality assurance RHB Regional Health Bureau RPHL Regional Public Health Laboratory RS Respiratory symptoms, or respiratory symptomatic SNNPR Southern Nations, Nationalities, and People’s Region SOP Standard operating procedure TFC Treatment follow-up centers for MDR-TB TIC Treatment initiation center for MDR-TB TB Tuberculosis TB CAP Tuberculosis Control Assistance Program TB CARE Tuberculosis Care TSR Treatment success rate TTS Tuberculosis treatment supporters TWG Technical Working Group USAID United States Agency for International Development WHO World Health Organization WorHO Woreda (District) Health Office ZHD Zonal Health Department HEAL TB ETHIOPIA MID-TERM EVALUATION 3 EXECUTIVE SUMMARY EVALUATION PURPOSE AND QUESTIONS The USAID-funded Help Ethiopia Address the Low TB Performance (HEAL-TB) Activity is a 5-year cooperative agreement being implemented since July 2011 by Management Sciences for Health (MSH) in partnership with the Program for Appropriate Technology in Health (PATH), the All Africa Leprosy, Tuberculosis, Rehabilitation and Research Training Centre (ALERT), and the Kenya Association for the Prevention of Tuberculosis and Lung Disease (KAPTLD). HEAL-TB supports a comprehensive package of tuberculosis (TB) interventions to provide quality directly observed treatment, including strengthening referral linkages to the community; enhancing TB/HIV collaborative activities; strengthening programmatic management of drug-resistant TB (PMDT); and strengthening health systems. The assistance has been supporting the Federal Ministry of Health (FMOH) to implement comprehensive TB projects in 21 zones of the Amhara and Oromia regions, covering a population of over 40 million at the end of the second year. The goal of the HEAL-TB activity is to demonstrate that global targets could be reached in specified geographical areas with a package of TB interventions, with the objective of increasing the case detection rate to 75%, treatment success rate to 90%, cure rate to 80%, and placing multi-drug resistant TB (MDR-TB) cases on treatment. The present document reports on a USAID mid-term evaluation of the HEAL-TB activity, carried out from September to November 2014, to evaluate achievements and challenges and produce recommendations for the remaining two years, as well as to contribute to future project strategies. Ethiopia ranks eighth in the global list of high-burden countries with an estimated 210,000 incident cases of TB (224 per 100,000). In a recent Ethiopian national population based survey in 2011, the prevalence of bacteriologically-confirmed TB (smear and/or culture positive) for persons aged 15 years and above was 277 per 100,000. The prevalence of smear positive pulmonary TB was 108 per 100,000, or 63 per 100,000 including children. The population-based TB survey concluded that TB burden in Ethiopia was lower than previously thought, which may indicate better program performance. However, a high proportion of TB among young persons’ suggests that TB is circulating in the community and that there is a need for more efforts to limit the spread of TB disease1. The annual risk of infection fell from 3% in 1953-55 to 1.4% in 1987-90, a reduction of 2.2% between the two measurements2. Ethiopia has been implementing the DOTS strategy since the early 1990s, with nation-wide coverage since 2002. The number of TB cases reported to the National Tuberculosis and Leprosy Program (NTP) has declined in recent years, in spite of the increase in population, better access to TB diagnostic services and implementation of community-based TB care. HEAL-TB supports the NTP efforts with focus on four main technical areas: Strengthening and Expansion of DOTS; Emergency Response to MDR-TB; TB/HIV Collaboration and Health Systems Strengthening. The purpose of the evaluation was to independently assess the overall performance of the HEAL-TB activity in terms of achievements and challenges, and to identify actions to improve management during the remaining life of the activity. The findings will also inform the design and implementation of similar projects in line with USAID/Ethiopia Country Development Cooperation strategy. A mixed evaluation method was used including review of documents, interviews with health staff at national, regional, zonal, district and health facility levels, interviews with staff of HEAL-TB and other stakeholders, observation of TB care in health facilities and procedures for community participation. The evaluation team included one international expert, two local consultants and a USAID adviser. The main evaluation questions were: 1. Verify that HEAL-TB is using standard and proper strategies for implementation of TB-related activities. 1 Int J Tuberc Lung Dis. 2014 Jun; 18(6):635-9. doi: 10.5588/ijtld.13.0417 2 First Ethiopian national population based tuberculosis prevalence survey, Addis Ababa, July 2011, page 2 4 HEAL TB ETHIOPIA MID-TERM EVALUATION 2. Examine monitoring and implementation to see if the indicators used are appropriate and mechanisms are in place to ensure effective program learning. 3. Check if HEAL-TB activities support the objectives of the cooperative agreement: improved management (program integration, appropriate policy and norms, strengthened staff capacity, essential infrastructure and supplies, training and supervision, monitoring system and use of information for decision making); improved availability and implementation of TB quality care (community information and behavior, access to TB services, quality of case detection, diagnosis and treatment, management of TB/HIV and PMDT). 4. Examine financial procedures to see if USAID investments are used effectively and efficiently. 5. Examine the coordination with other organizations and their role to strengthen the TB program. FINDINGS The overall findings of this evaluation are very positive. All the government and hospital officials/health facility staff interviewed were satisfied with the level of communication and coordination between the TB program and the activity and also several noted that HEAL-TB is different from other projects in the way the activity has been developing mutual plans with the government and work hand-in-hand to focus on systems strengthening and capacity building. In addition, the level of technical expertise of the project staff was highly valued by many of the key informant respondents. There was an acknowledgement that the project has improved since start-up, since the activity has focused on the zones to build capacity rather than placing project staff at district level. 1. HEAL-TB follows the technical and operational guidelines of the NTP, which are in accordance with international recommendations. Planning and evaluation are conducted jointly with the Regional Health Bureau authorities of the Oromia and Amhara Regions. Project data and report reviews, qualitative findings from key informant interviews and field observations confirmed that the main achievements of the HEAL-TB activity were expanded access to TB diagnostic and treatment services; improved diagnostic quality of microscopy through external quality assurance (EQA); support to sustainable national capacity for monitoring and supervision; improvement of case detection activities through screening in health facilities and improvement of treatment outcomes. The project has expanded in Year 3 to cover the whole population of the two Regions, a total of 51 million.3 Training for capacity building is complemented by supervision with a mentoring system and provision of essential equipment and supplies. Detection of presumptive TB cases is through routine screening of outpatients in health facilities, contact tracing and community-based screening of presumptive TB cases using health extension workers (HEWs). Diagnosis is mainly with sputum microscopy. EQA has resulted in improved quality of microscopy. Treatment is ambulatory and directly observed following WHO recommendations; and patient kits are used for first-line drugs (FLD). Treatment of MDR-TB is ambulatory and directly observed. The reported number of TB cases has declined in spite of the increase in population, better access to TB services and the project support to case detection activities. The low detection can be attributed partly to a decrease in the TB prevalence, but also to the impact of the project successful treatment, to prohibitive barriers for the presumptive cases to reach diagnostic facilities and limited implementation of community￾based TB care. In the 10 initial zones, the number of TB cases reported (of all types) declined from 36,445 in 2011 to 33,644 in 2013 – an 8% reduction in two years – compared with 16% reduction for Ethiopia overall, suggests that the HEAL-TB activities increased case detection in the area and halted the declining case 3 HEAL TB Year III Quarter IV report HEAL TB ETHIOPIA MID-TERM EVALUATION 5 detection rate by 50% in activity areas compared to national level. In spite of interventions to increase detection for early diagnosis, the number of presumptive TB cases examined by microscopy is very low. Persons with symptoms do not attend health facilities, even when identified by HEWs and referred for care. This reduces the effectiveness of screening for cough in general outpatients which is one of the main WHO and project strategies for early diagnosis of infectious TB. The cure and treatment success rates of new smear-positive TB patients increased consistently and reached 89.8% and 95.5% in the original zones, from a baseline of 71% and 88% in 2011, over the planned targets. 2. The country has useful epidemiological information on TB transmission (tuberculin surveys in 1953/55 and 1987/90) and TB control (national TB prevalence survey in 2011, epidemiological review and NTP Review in August 2013). The indicator for case detection (case detection rate, CDR) used from the start of the activity4 is not appropriate, because there are no useful estimates of incidence at regional or lower levels in the country. There are major variations in TB prevalence among zones and since the real prevalence is declining, the targets may never be reached. The planning for case detection is based on 10 presumptive TB cases per pulmonary TB (PTB+) case, although the average smear positivity is closer to 6% (17 suspects per actual case). This could be one of the reasons for insufficient quantities of sputum cups and laboratory reagents, resulting in stock-outs in past years. The indicators for treatment are appropriate but only cure and treatment success are followed at all levels. Health facility staff is well-trained, and data and trends in case detection are visible in many facilities; this is not true for the microscopy laboratories which do not analyze available data on persons tested for diagnosis and positivity rate. Only a few health facilities and districts use maps for planning, supervision and monitoring, although this was a component of the Health System Expansion program. 3. Interviews with health managers at all levels indicated wide acceptance of the project, its aims and objectives and the norms and strategies used. However, limited engagement in community-based TB care below the district and primary health care unit levels remain a challenge. Program staff acknowledged and highlighted the HEAL-TB contribution to capacity building, sustainable human and institutional resources including equipment and supplies as well as problem-solving collaboration. In particular, HEAL-TB contributed to capacity building for PMDT and TB/HIV patient care and used strategies that can sustain the gains after the project ends such as health workers with technical capacity for improved diagnosis (microscopy, GeneXpert) and decentralized EQA, and decentralization of treatment and PMDT. HEAL-TB activity improved the implementation of community-based TB care. The activity supported the revision of National guideline for Community-Based TB Care and training of trainers to cascade the training for HEWs in the regions. It was observed that most heath centers have established referral mechanisms so that presumptive TB cases referred from the community have easier access and established mechanism for presumptive cases to directly visit the TB clinic to get expedited diagnostic process. However, community￾based activities for case detection are carried out by HEWs irregularly and inadequately, and had limited success: a large proportion of the presumptive cases identified do not reach the diagnostic health facilities for testing and the smear positivity is often very low. There is no mechanism to ensure that persons who did not reach the health center are traced. Health center staff and HEWs carry out health education and occasional population-based screening “campaigns” to identify more cases, however very few infectious cases are identified. Despite a huge contribution to the health system strengthening (HSS), TB, TB-HIV and PMDT, the limited engagement below the districts and PHCU, missing links between the districts and the community, inability of presumptive TB cases to reach diagnostic facilities and limited engagement of HEWs in implementing community-based TB care remain the key challenges. 4 Case detection rate: Number of smear positive cases reported by HEAL-TB over the WHO-estimated incidence for the country. WHO does not have estimates for regions. 6 HEAL TB ETHIOPIA MID-TERM EVALUATION 4. Based on a general review there were no serious financial management issues identified. HEAL-TB implements activity-based budgeting where activities included a detailed cost budget. Spending is monitored regularly along with the technical aspects of the work through biweekly management meetings. In addition, activity managers and technical advisors are tasked with regularly monitoring the activities in the technical areas both for quality of implementation and spending. At a central and regional level HEAL-TB develops quarterly activity plans to enable them to monitor the progress of activities and resolve delays in spending and implementation. The regional satellite offices are allocated funding which is monitored by regional financial officers and monthly vouchering. All partners are required to submit monthly vouchers in order to monitor spending. There were no major reported problems with getting timely and accurate financial reports. 5. Planning of activities is well-coordinated and harmonized between HEAL-TB and the various government levels. At national level, HEAL-TB participates in the main TB technical working group (TWG) where various partners meet to discuss activities with the NTP. In addition, the activity shares work plans and meets individually with the NTP to coordinate and update the NTP on activities on a periodic basis. At regional, zonal, and woreda levels, all of the health leaders interviewed were very satisfied with the level of coordination and collaboration between the HEAL-TB activity, and the fact that HEAL-TB staff often participated in the woreda-level planning activities. All of the government and hospital officials/health facility staff interviewed were positive about the level of communication and coordination between the national TB program and HEAL-TB. Coordination with the partners within the HEAL-TB activity is done formally on a quarterly basis with additional communication and meetings were called as needed. CONCLUSIONS 1. HEAL-TB is using appropriate strategies for TB control, consistent with international recommendations, and has strengthened national capacity in the two Regions. However, the WHO-recommended strategy of screening general outpatients with persistent cough cannot work effectively and have the expected impact on detection of presumptive cases because the population does not attend health facilities for any reason in sufficient numbers, even when referred from the community by HEWs. Most of HEAL-TB impact can be attributed to the expansion of microscopy services and excellent results of treatment achieved. The expansion of the network of sputum microscopy to all health centers and upgrading from light to light emitting diode (LED) microscopy, initially justified on the basis of population and distances, will not solve the low detection of presumptive cases. The workload per microscope, in particular the number of smear positive samples, is already too low to maintain attention and quality, and the problem will increase with reduction of the TB prevalence. The occasional population-based screening “campaigns” at the district level to detect more presumptive TB cases are not effective and cannot replace the regular screening of the population by HEWs. This was recognized and stopped to focus on system strengthening. Increased use of health facilities for any reason is out of the scope of TB program interventions. The CDR is not an appropriate indicator; WHO has recognized this in 2013 but has not yet proposed alternatives. However, its use should be continued because it was part of the original project plan. Rapid expansion of MDR-TB treatment facilities has significantly improved access to treatment. In these two regions, MDRTB treatment services has been expanded in the last three years from only 1 hospital with inpatient model of care (Gondar) to 14 ambulatory sites (5 in Amhara and 9 in Oromia). TB/HIV activities are well supported and coverage has increased (over 90% of TB patients tested, over 80% on CPT and 72% on ART by June 2014). 2. The institutional capacity to manage the TB program has improved with HEAL-TB support and its contribution to HSS was recognized at all levels. The mentoring strategy adopted in recent activity implementation appears to achieve sustainable gains in terms of building the monitoring and HEAL TB ETHIOPIA MID-TERM EVALUATION 7 evaluation capacity of zone and district experts. However, utilization of the capacity is limited to health centers and not adequately reaching the community. The indicators for treatment are appropriate. The indicators used by the health management information system (HMIS) were insufficient and were revised by the NTP to avoid the need for parallel data collection systems. Staff is well-trained and motivated at all levels. Gender is not routinely analyzed, but there is useful data from the national TB survey showing slightly higher prevalence in males, as in most countries. A recent HEAL-TB study in Oromia showed no difference in delay in diagnosis and treatment by sex. Data on prevalence of TB in contacts of index cases is available; with the high proportion of children. Screening and isoniazid prevention therapy (IPT) in contacts should be a priority intervention but is currently low (under 30%). 3. The HEAL-TB reports accurately reflect the achievements of the activity and its current status. The priorities are clear. An analysis of the low level of detection of presumptive TB cases shows that the problem is related to limited access and use of health facilities by the population for any reason, and will not be solved by expansion of the microscopy network. The methods to detect and screen persons with symptoms (presumptive TB case referral) should be re-evaluated to ensure sputum smear examination. Alternatives are community sputum collection and transport to diagnostic units (preferable) or fixing and transport of slides (easier transport but with higher work risk for community staff). 4. The financial mechanisms and coordination with partners and other stakeholders appear satisfactory. The coordination with the national TB program is now excellent, after two years of maturing and adjusting activity’s strategies. RECOMMENDATIONS 1. Maintain USAID support for expansion in Phase III and consider extending support to other regions in future projects. USAID should consider expanding the current HEAL-TB model to other regions and/or future projects. Priority should be given to areas with higher TB prevalence (estimated on the basis of smear positivity among outpatients with symptoms and community screening) HEAL-TB should:  continue to coordinate and collaborate with government partners as they have been doing since year 2 of the activity;  consider a more targeted assistance approach, reassessing the needs and tailoring activities specifically for each zone; and  include mapping in the training and support its use at all levels as an instrument for planning and supervision (managerial levels) and follow-up of patient and contacts (health facilities) and support facilities to upgrade the maps available. 2. HEAL-TB should analyze routinely the number of persons examined quarterly by microscopy for diagnosis and positivity (data available from the microscopy register) as an indicator for case detection, and train laboratory and program staff to analyze trends, display them visibly and use them for decision making in the health facility. The zonal and district information can be used to select geographical areas with higher prevalence and/or areas with a low number of cases to test appropriate strategies and adjust heath facility targets. HEAL-TB should implement household visits to register contacts of all TB cases and give priority to IPT for children contacts of index TB cases. HEAL-TB should rationalize the laboratory capacity and workload, taking into account that a small number of slides per week (particularly smear positive) will result in reduction of quality and increase of false 8 HEAL TB ETHIOPIA MID-TERM EVALUATION diagnosis; and should evaluate the need for new microscopy units versus strengthening of transport of samples to existing units;. 3. HEAL-TB should develop and test interventions which can reduce the burden to the patient and improve access to diagnostic units combined with treatment in the community. Interventions such as microscopic examination of presumptive cases without referral could be appropriate and could ensure examination, reduce health system and patient related barriers to access services. Further decentralize the engagement of districts and PHCU so that implementation of community-based TB care to contribute to improving access and quality of care. 4. Support GOE to screen HIV+ for active tuberculosis, through coordination with HIV/AIDS program. This could be done by strengthening TB diagnostic capacity where there are ART clinics, for instance using GeneXpert. Support the staff capacity to monitor and evaluate TB-HIV activities in the districts by regular review of activities and by supervision to health centers and health posts to improve data capturing for TB/HIV activities 5. Further decentralize the TIC role to support TFC and quality of care. Mentoring and catchment area meetings should be considered to decrease burden on current TICs and improve quality of care. This includes ensuring correct patient diagnosis, treatment, program management and continuous monitoring and evaluation of all program activities to provide feedback loop for quality improvements. Attention should be given in maintaining quality laboratory equipment, supplies and staff training to manage MDR-TB patient complications and side-effects, identifying sites with insufficient capacity or staff knowledge and responding efficiently to express needs, and ensuring that the TFCs have sufficient capacity to support the TICs. HEAL TB ETHIOPIA MID-TERM EVALUATION 9 I. INTRODUCTION Ethiopia is one of the world’s high tuberculosis (TB), TB/HIV and multi-drug resistant TB (MDR-TB) burden countries. Prevalence and incidence of TB are 210 and 224 per 100 000 population respectively5. The estimated TB case detection rate in Ethiopia has been consistently low; the WHO estimate is 64% for all forms. To address the low TB case detection, Help Ethiopia Address the Low TB Performance (HEAL-TB) activity was designed to support TB prevention and control efforts in the Oromia and Amhara Regions. The USAID funded HEAL-TB activity is a 5-year cooperative agreement awarded in July 2011 to Management Sciences for Health (MSH) in partnership with the Program for Appropriate Technology in Health (PATH), the All Africa Leprosy, TB, Rehabilitation, Research and Training Centre (ALERT), and the Kenya Association for the Prevention of Tuberculosis and Lung Disease (KAPTLD). HEAL-TB supports a comprehensive package of TB interventions to provide quality DOTS including strengthening referral linkages to the community; enhancing TB/HIV collaborative activities; strengthening programmatic management of drug resistant TB (PMDT); and strengthening health systems. The assistance supports the Federal Ministry of Health (FMOH) to implement comprehensive TB projects starting in 10 zones of the Amhara and Oromia regions and expanding to all zones, covering a population of 52 million by the third year. Due to the need for emergency treatment for hundreds of MDR patients, support for PMDT has also expanded in both regions. HEAL-TB collaborates with Ethiopian universities and other local institutions and helps the regional health bureaus (RHBs), zonal health departments (ZHDs), and woreda (district) primary health care units take ownership of TB, TB/HIV, and MDR-TB program management. Through this assistance, they are expected to have improved and sustainable program management capacity. HEAL-TB focuses on increasing case notification and decentralization of DOTS in communities through Health Extension Workers (HEW) and expanding TB diagnosis and treatment access to additional health facilities in rural and urban areas. HEAL-TB is also helping to strengthen Ethiopia’s health system by supporting and taking part in woreda-based planning, improving drug supply management, strengthening infection control, enhancing and improving TB laboratories, and strengthening monitoring and evaluation at all levels. The present document reports on a USAID mid-term evaluation of the HEAL-TB activity, carried out from September to November 2014, to evaluate achievements and challenges and produce recommendations for the remaining two years; and to contribute to future activity strategies. II. BACKGROUND Ethiopia, with a population of 94 million in 2013 according to the WHO Global TB report 2014 6, ranks eighth in the list of high burden countries with an estimated 210 000 incident cases of TB (224 per 100 000). According to the same WHO report, there were 200 000 prevalent TB cases in 5 Global TB WHO Report 2014 6 The national population estimate is 84 million. 10 HEAL TB ETHIOPIA MID-TERM EVALUATION Ethiopia in 2013 (211 per 100 000). Tuberculosis mortality is estimated at 30 000 deaths (excluding deaths among HIV positive TB cases) or 32 deaths per 100 000. In a recent Ethiopian national population based survey, the prevalence of bacteriologically confirmed TB (smear and/or culture positive) for persons aged 15 years and above was 277 per 100 000. The prevalence of smear positive pulmonary TB was 108 per 100 000, equivalent to 63 per 100 000 including children which are nearly 50% of the population. Prevalence has been declining for a long time as shown by the annual risk of infection which fell from 3% in 1953-55 to 1.4% in 1987-90, at 2.2% per year7. The number of TB cases reported to the National Tuberculosis and Leprosy Program (NTP) has declined in recent years in spite of an increase in population and access to diagnostic services (Figure1). Figure I. Reported TB cases in 1000s. Ethiopia, 2010-2013 Source: WHO Global TB Report 2014 Both men and women are equally vulnerable to tuberculosis in their productive ages of 20-30 years. Gender disparities in health-seeking behavior might contribute to differences in delay in TB diagnosis, treatment interruption, gender based TB stigma and discrimination8. Low detection rate and treatment for TB among women have been seen due to delay in accessing care, poor compliance to treatment, and belief in alternate treatments. The reporting system does not disaggregate data by sex; but a recent study in Oromia found no difference in diagnostic delay and treatment by sex. The delay in males was higher but not statistically different. Ethiopia has been implementing the DOTS strategy since the early 1990s, with nation-wide coverage since 2002. According to the Ethiopian Fiscal Year (EFY) 2005 (Gregorian calendar 2012/2013) Health Sector Development Plan (HSDP) IV annual performance report, the TB case detection rate was 58.9%, below the detection rate estimated in the 2011 TB prevalence survey (72%) and below the target set for the year (82.7%). In the same annual bulletin report of the FMOH, a total of 130 614 TB cases (all forms) were reported with a TB case notification rate of 152 per 100 000; this performance was below that observed in EFY 2004 (2011/2012), when 145 367 cases (all forms) were reported with a TB case notification rate of 172 per 100 000. Out of the 130 614 cases reported in 7 National tuberculosis prevalence survey, July 2011. 8 HEAL-TB/KAPTLD. Strategy to mainstream gender in tuberculosis care and control in Ethiopia. J.M. Chakaya et al. 155 157 145 132 46.6 49.6 47.2 43.9 55 55.5 49.4 45.5 0 20 40 60 80 100 120 140 160 180 2010 2011 2012 2013 Reported TB Cases (in thousands) Year NEW/REL NEW PTB+ NEW PTB￾EP TB HEAL TB ETHIOPIA MID-TERM EVALUATION 11 EFY 2005 (2012/2013) 33.4% were smear positive pulmonary TB, 34.5% were smear negative pulmonary TB, and 32.1% were extra pulmonary TB. As of 2013, the current national TB treatment success rate (TSR) is estimated to be 91.4% while the cure rate for the year 2012/2013 was 70.6%. According to the 2013 annual TB bulletin of FMOH, in 2011/2012 86% of the TB patients know their HIV status and 20% are estimated to be HIV infected. The HIV co-infection rate was higher than the previous year estimate which could be as a result of the new TB/HIV surveillance sites being located in more of urban sites. The 2014 WHO global report estimated the co-infection rate to be 11%. In 2011/2012, among TB/HIV co-infected patients 76% got co-trimoxazole preventive therapy (CPT) and 51% were put on antiretroviral therapy (ART). However the isoniazid preventive therapy (IPT) coverage was only 18%. TB, MDR-TB and TB/HIV continue to inflict a major impact on both morbidity and mortality rates in Ethiopia today. National and international attention is now focused on providing care in both public and private health facilities. The national program has planned to carry out comprehensive TB activities in order to improve the case detection rate and to build the capacity of Health Extension Workers (HEWs) to mobilize communities for early TB diagnosis and active TB case finding (to screen all TB suspects and start community DOTS service in each Kebele). As of June 2014 46 hospitals, 1618 health centers and 7 764 health posts (out of 15 000) provided DOT treatment services9. Also 1 941 health facilities provided diagnostic services (one diagnostic center for 43 000 inhabitants), with the government continually expanding the number of health centers and diagnostic sites. The Public-Private Mix (PPM-DOTS) is a strategy designed by the country to improve the TB case detection. PPM-DOTS started in 2006 and currently 317 PPM-DOTS facilities provide DOTS services. In the area of TB/HIV about 1 553 health facilities were providing TB/HIV services. HEAL-TB supports the national TB program efforts in two regions (Amhara and Oromia) focusing on four main technical areas including the TB program cross-cutting areas of gender as well as partnership and coordination.  Technical Area 1: Strengthening and Expansion of DOTS  Technical Area 2: Emergency Response to MDR-TB  Technical Area 3: TB/HIV Collaboration  Technical Area 4: Health Systems Strengthening The HEAL-TB strategies to achieve the activity aims include: o Increase community awareness of TB, improve referrals, and improve the quality of service through training of Health Extension Workers o Improve the organization of TB case detection within facilities by strengthening multidisciplinary teams and streamlining the flow of TB suspects and patients o Provide standard operating procedures, job aids, and posters to all health facilities and improve workers’ skills in TB diagnosis and management o Improve the documentation and monitoring of TB suspects o Strengthen TB/HIV collaborative activities and screen all TB patients for HIV and strengthen referral linkages for co-management of HIV positive TB patients 9 MSH HEAL TB Report April-June 2014 12 HEAL TB ETHIOPIA MID-TERM EVALUATION o Strengthen MDR-TB early diagnosis, referral and treatment capacity in the two regions through expansion of PMDT services. o Strengthen laboratory services management; o Improve the drugs and commodities management of the program and health facilities; o Involve all care providers, including private health facilities, and drug outlets. The profile of the HEAL-TB activity sites with treatment outcomes is included in Annex B Table 1, HEAL-TB coverage by October 2014 in Annex B Table 2 and the activity geography and scope is provided in Annex B Figure 1. III. SCOPE AND METHODOLOGY EVALUATION PURPOSE The purpose of the HEAL-TB mid-term evaluation was to evaluate achievements, identify best practices, areas for improvement and make recommendations for the remaining two years of the activity implementation. This midterm evaluation informs implementing partners to capitalize on best practices and make adjustments in the midterm and as well draws lesson to USAID/Ethiopia for its future USAID/Ethiopia country operational project strategies. KEY EVALUATION QUESTIONS AND MAIN METHODS The Scope of Work (Annex A) guided the evaluation methodology. The key methods employed to answer each evaluation question is provided below: 1. Verify that HEAL-TB is using standard and proper strategies for implementation of TB-related activities, based on evidence or best practices that meet international standards, and describe how quality is measured in the key components (diagnosis, patient management and follow-up, data recording and reporting). Methods: Comparison of activity norms and practices with NTP and internationally recommended norms through interviews with stakeholders and document review; interviews with health delivery staff; and observation of health facility procedures and information system; assessment of availability, quality and use of information on patient care through observation of TB suspect screening, diagnosis, treatment and outcomes; and review of data at facility and district level. 2. Examine monitoring and implementation to see if indicators are appropriate and mechanisms are in place to ensure effective program learning: Is M&E documenting evidence to ensure that key lessons to inform future projects? Has HEAL-TB capacity building improved sustainability of the activity regarding institutional strength and human capacity to manage TB programs? To what extent have gender metrics been incorporated into the M&E system? Methods: Review of HEAL-TB reports to USAID and additional data requested from MSH to evaluate if indicators are appropriate and sufficient; interviews with HEAL -TB and health management staff to see if the information is used for action; interviews with staff at all levels to see if the priorities for TB control are clear and appropriate and if the activity has developed staff capacity to sustain and expand the interventions after the activity HEAL TB ETHIOPIA MID-TERM EVALUATION 13 3. Check if HEAL-TB activities support the objectives of the cooperative agreement. Improved health system management: program integration, appropriate policy and norms, strengthened staff capacity, information for decision making, essential infrastructure and supplies, training and supervision, monitoring system and use of information; improved availability and implementation of quality care in tuberculosis; including community information and behavior, access to TB services, quality of case detection, diagnosis and treatment, management of TB/HIV and PMDT and treatment outcomes, and management of sub-partners, supervision and coordination. Is staff competence sufficient? Methods: Assess expansion of TB care services (health facilities implemented, diagnostic capacity, regular supplies) and human resource capacity (training and supervision) to increase population access to TB services through document review and interviews at district and regional level Assess activities to increase population access through interviews with staff at regional and district level and with partners. Review HEAL-TB reports to USAID and additional data requested from MSH to evaluate if indicators are appropriate and sufficient, interviews with HEAL-TB and health management staff to see if the information is used for action. Interviews with staff at all levels to see if the priorities for TB control are clear and appropriate and if the activity has developed staff capacity to sustain and expand the interventions after HEAL-TB ends. Observe procedures and data collection at health facilities to check if activities are implemented and registered correctly and monitored by the staff; and if data is used for action. 4. Examine financial procedures to see if USAID investments are used effectively and efficiently to achieve the objectives: Funds allotted to partners, monitoring and utilization of funding. Methods: Review of reports and interviews with MSH, partners and USAID 5. Examine the coordination with other organizations and their role to strengthen the TB program. How have other stakeholders contributed to achieve HEAL-TB results capacity. Methods: Interviews with MSH, NTP and other stakeholders10 The evaluation team included one international TB control program consultant (Fabio Luelmo, Team Leader), two national consultants (Abera Bekele Leta and Daniel Gemechu Datiko), and a USAID/Washington Advisor (Janet Phillips). Site visits were carried out in selected areas of the two regions (Amhara and Oromia). The selection of zones and facilities was done based on accessibility, sufficient population to be representative and duration of implementation (sites started in phase 1 and phase 2). Visits included general hospitals, health centers and health posts, MDR facilities and microscopy units. Field visit schedule is provided in Annex C. A priority was to estimate the impact of HEAL-TB activities to support the NTP to reduce TB transmission and the burden of disease; and improve on the access and quality of TB diagnosis, treatment and prevention. A list of persons contacted for interviews is provided in Annex D and institutions visited in Annex E. The guidelines for interviews and for observations in health facilities are included in Annex F. The tools/guidelines for interviews were tested during the field work and adjusted based on findings, to capture key technical and operational aspects that might influence activities and results. 10 Question 5, co-ordination, has been included under strategy and management. 14 HEAL TB ETHIOPIA MID-TERM EVALUATION Data obtained at Regional, Zonal, District and health facilities was compared with HEAL-TB reports for consistency and analyzed to estimate activity performance and epidemiological impact. This included the availability and trend of indicators on case detection, external quality assurance (EQA) of microscopy, and proportion of diagnosed TB cases according to site and microscopy and treatment outcomes. Quality of care included the proportion of cases examined for HIV infection and proportion of TB/HIV cases receiving CPT and ART, and the number of MDR-TB cases identified and treated and treatment outcomes. IV. STRENGTHS AND LIMITATIONS OF THE EVALUATION The team members did not have any conflicts of interest that might affect the evaluation. The national consultants contributed the knowledge of the country, their experience in the national TB program and activities and the local languages; and the international consultant and USAID advisor contributed their extensive experiences in TB programming from other countries and projects and USAID procedures. All team members participated at national, regional and zonal level visits and interviews, and separated in two teams for some district (Woreda) and health facility field visits, in each of the regions. HEAL-TB staff accompanied the team but was not present during interviews and observation of facilities. The number of persons interviewed and sites visited cannot be assured to represent the entire project; however they were used to validate the HEAL-TB reports. The time for preparation of the team, review of documents, planning logistics and development of data collection instruments was very limited. This limitation was partially compensated by revision of instruments and review data during field work. Future evaluations should consider the WHO procedures for conducting TB program reviews, adapted to project evaluations11, with a local consultant responsible for logistics in advance, in coordination with USAID. Part of the evaluation team was absent during the period following field work, due to other commitments (IUATLD Conference, Barcelona); only partially compensated by electronic consultation. V. FINDINGS 1. OVERARCHING FINDINGS HEAL-TB improved program management capacity, coordination, training of government staff and mentoring during implementation of TB activities, development and distribution of materials, strengthening staff capacity to conduct monitoring and evaluation, and advocacy to improve political commitment. HEAL TB placed staff at regional and zonal levels to work closely with and mentor MOH TB staff to improve quality, implementation, monitoring and supervision. Although there are no 11 Framework for conducting reviews of tuberculosis programmes. WHO, 2014 HEAL TB ETHIOPIA MID-TERM EVALUATION 15 direct measures of improved program management capacity with support of HEAL TB, supervision and review meetings took place regularly. Interviews with MOH supervisory staff and health center staff working on TB indicated that support from HEAL TB had increased motivation and interest in the TB program, and supervisory staff indicated that the additional support from HEAL TB allowed them to better do their job of supervision and data analysis. In addition, improved treatment success and cure rates can act as proxy indicator for improvements in the overall management and quality implementation of the TB program. The laboratory network was stregthened by procurement and distribution of supplies and equipment, training, implementation of decentralized EQA system, improvement of diagnostic capacity using LED and GeneXpert, repair and training on maintenance of microscopes. The drug and supply management was supported by the activity through PFSA, and HEAL-TB contributed to the redistribution of drugs and supplies during stock out and supported implemenation of patient kits. HEAL-TB actively supported the decentraliation of MDR-TB treatment initiating centers in the two regions, expanded from one (Gondar University Hospital) in 2011 to 16 sites in Sept. 2014 with more than 300 MDR-TB treatment follow up centers12; renovating facilities, training, mentoring, supporting sample tansport and laboratory networking and providing supplies and equipment.The activity participated in joint planning, staff capacity building and participated and supported joint supervision and regular review of activies. At regional, zonal, and woreda levels, all of the health leaders interviewed were very satisfied with the level of coordination and collaboration between the activity, and the fact that HEAL-TB staff often participated in the woreda-level planning activities. All of the government and hospital officials/health facility staff interviewed were positive about the level of communication and coordination between the TB program and HEAL-TB. Several people interviewed said that HEAL-TB is different from other projects in the way they develop mutual plans with the government and work hand in hand to focus on systems strengthening and capacity building. There is a very real sense that HEAL-TB is supporting the system and not operating as a stand alone activity. In addition, the level of technical expertise of HEAL-TB staff was highly valued partner by many of those interviewed. At all levels, there was an acknowledgement that HEAL-TB has improved since start-up. At the beginning there seems to have been some confusion about the structure of the activity as well as how the activity could best fit within the national priorities. The initial HEAL-TB strategic approach was modified following guidance from USAID, and based on internal monitoring and evaluation activities. The approach is being evaluated externally for the first time through this evaluation. Since HEAL-TB changed the model, focusing on zones with direct support to district teams to build capacity rather than placing activity staff at health facility/district level, most of the coordination issues appear to have been resolved and there is a clearer focus and understanding between the national TB program and HEAL-TB. The NTP manager indicated that the current strategy will be sustainable in the future with national resources. 12 MSH/HEAL-TB year 3 quarter 4 Report. “The project has reached maturity in its third year. It is more consistent and harmonized with the TB program. Managerial capacity-building at regional and zonal levels contribute to sustainability. The way forward is to continue the strategy, expand coverage and increase managerial capacity at facility level”. Addis Ababa, interview at national level. 16 HEAL TB ETHIOPIA MID-TERM EVALUATION 2. STRATEGY AND MANAGEMENT The interviews at all levels (national, regional, zonal, district, health facilities and stakeholders) showed good knowledge and recognition of the role and contribution of HEAL-TB to strengthen the response of the NTP to the TB problem. USAID and GF are the largest TB donors in Ethiopia. HEAL TB is the largest of the USAID TB programs along with TB CARE I, Private Sector Health Program (PHSP) as well as PEPFAR supported TB/HIV activities. Planning of activities is well-coordinated and harmonized between HEAL-TB and the various government levels. At the national level, HEAL-TB participates in the main TB technical working group (TWG) where various partners meet to discuss activities with the NTP. In addition, HEAL-TB shares work plans and meets individually with the NTP to coordinate and update the NTP on activities on a periodic basis. Since HEAL-TB provides significant support for laboratory activities, they are also very involved with the Ethiopian Public Health Institute (EPHI) as active participants in the National Laboratory TWG and the TB lab specific advisory group, as well as conducting monthly meetings between HEAL-TB and EPHI staff in order to coordinate activities and address gaps. The coordination and collaboration with the national TB program (all levels) includes sharing data and information. HEAL-TB shares all of the reports and data with the various levels of the TB program. HEAL-TB also funds and participates in quarterly coordination meetings at the zonal level to review data and address challenges. In the government system, these quarterly meetings and supervision are integrated and TB generally gets little time or attention. The district and zonal supervisors expressed that this type of meetings and data analysis and review is very useful and not something they have had the time or funding to do in the past. There was a sense that this acted as a motivating factor for the government staff. Coordination with Other Partners Coordination with other non-governmental partners is done, for the most part, at the national and regional TWG level. In addition to TWG meetings, USAID leads regular partner coordination meetings for the USAID-funded TB implementation partners (HEAL-TB, TB CARE 1 and PHSP). Coordination with the partners within the HEAL-TB activity is done formally on a quarterly basis with additional communication and meetings called as needed. PATH is the largest partner in HEAL-TB and is tasked with leading the community TB activities and strategy. Coordination with PATH is mainly through the Community TB Coordinator who is empoyed by PATH but sits with the HEAL￾TB MSH staff. In addition, there are quarterly meeting with the additional PATH/Ethiopia staff. With the exception of PATH, the other partners in HEAL-TB, KAPTLD and ALERT, have very specific and targeted activities and receive a small amount of funding per year so that a strong coordinating mechanism is not as critical as in other projects. In addition, MSH headquarters staff coordinate with the HEAL-TB staff on a regular basis through email, phone and regular field visits to Ethiopia. “No partner is comparable in the Region in joint work, planning and evaluation” - “Fully engaged with the government, better than other USAID￾funded projects”. Amhara, Regional reference laboratory interview. HEAL TB ETHIOPIA MID-TERM EVALUATION 17 The activity has increased political commitment. In Amhara, HEAL-TB designated TB clinical and lab officers at Zonal Health Department level and the government acknowledged the need and is considering establishing those posts in the regular government budget. In Oromia, the health authorities were worried about the sustainability of program activities without this staff when HEAL￾TB ends. 2.1 CASE DETECTION AND DIAGNOSIS Case detection includes screening of adults for persistent cough at the registration point in health facilities, screening for cough at community level, screening of contacts of TB cases (all forms) and community advocacy communication and social mobilization (ACSM). It was observed in multiple interviews that the population has very low utilization of health facilities and does not consider TB symptoms (particularly cough) as sufficient reason to attend for care, even when indicated by HEW in the community. The number of outpatient visits per person per year for any reason does not reach 0.3. In Quarter 2, 2014, 1448 government facilities in Oromia and Amhara received 1.65 million outpatient department (OPD) visitors, of which 1.4 million were screened for cough, with 41 500 presumptive TB; 87% were screened with 5.7% smear positive.13 Assuming a population of 40 millon, the number of visits per person per quarter was 0.04 (less that 2 per year). Even if adding non-government health visits doubles the number of visits for any reason (considering that the private sector accounts for 40% of the health curative and rehabilitative services in the country according to the World Bank), they are relatively low. As a result, the number of presumptive TB cases (persons with symptoms) examined for diagnosis is very low, one of the reasons for the low case detection. In this situation, the main strategy recommended by WHO of screening outpatients for cough in health facilities will have limited impact. However, the high cure/completion rates achieved justifies continuing this activity, as a much more cost-effective and patient-friendly strategy than population screening. The number of TB cases reported has declined rapidly in the regions covered by HEAL-TB (Figure 2 and Figure 3), in spite of increased access to diagnostic facilities, of active screening for cough in health facilities and communities, and of the population increase. The positivity in HEAL-TB activity areas has declined among presumptive cases examined as a result of screening more presumtive cases and probab;ly of reduction of real TB prevalence in the community (Table 1) Figure 2. Trend of all forms of TB cases notified in the Amhara Region 2002-2006 EFY 13 MSH HEAL TB Report, April-June 2014 18 HEAL TB ETHIOPIA MID-TERM EVALUATION Source: Amhara Regional Bureau, 2014 Figure 3. Number of new TB cases (all forms) reported in HEAL-TB original 10 zones, 2011-2013 Source: HEAL-TB report, Jan-Mar 2014. Case detection is measured through the case detection rate (CDR), an indicator based on WHO estimates of TB incidence for national level that is useful only in large populations but is applied by the NTP and by HEAL-TB also at district and facility levels. The facilities receive estimated annual and quarterly targets for TB cases to be found (smear positive pulmonary TB or PTB+ and smear negative pulmonary TB or PTB-) and for symptomatics to detect. The ratio of case/symptomatic used is 1:10 (10 persons examined to find one infectious case) equivalent to a microscopy positivity of 10%, although the average positivity is about 6% in outpatient facilities and in low TB prevalence areas or communty activities closer to 1% (18 to 100 persons need to be examined to find one infectious case). This could be one of the factors for shortage of reagents and suppplies and /or under-estimates of supplies for the laboratory (reagents and sputum cups) at national level, which resulted in shortages and stock-outs in past years. Table 1. TB case detection performance in one of the visited health center in Amhara Year Number of presumptive TB cases Number of PTB+ % 33 716 30 804 31 636 28 158 24 306 20000 2002 2003 2004 2005 2006 All forms of TB cases notified 36445 36061 33644 32000 33000 34000 35000 36000 37000 2011 2012 2013 Number of TB cases HEAL TB ETHIOPIA MID-TERM EVALUATION 19 2003 157 48 31 2004 73 14 19 2005 279 36 13 2006 644* 29 5 * A population screening “campaign” was carried out in quarter 2, with 427 suspects (1141 slides) of which 17 were positive In some cases government staff has tried to compensate the low CDR by “campaign” community activities, which increase the number of presumptive cases identified and examined with sudden overload of laboratory microscopy but with little or no advantage. For instance, in one health centre in Amhara a campaingn in 2006 doubled the the number of symptomatics identified but with a low number of smear positive cases detected (Table 1). One such campaign activity at health post level examined 95 respiratory symptomatic (RS) to detect 1 positive and 1 negative TB case; another one at health center (HC) level examined 250 persons with RS with no smear positive TB detected. These campaigns cannot replace the regular screening, may have the additional problem of increasing the proportion of cases diagnosed only through clinical methods, without laboratory confirmation, and are not recommnded. They could also have negative impact on staff as the persons identified do not go to be examined for diagnosis, results may not be available and staff will be discouraged. The most practical indicator of case detection activities -the number of persons examined for diagnosis by microscopy, available from the microscopy register- is not routinely compiled or used by the program. The information is available to HEAL-TB but not used routinely by the laboratory and program staff. The positivity rate of slides is used by HEAL-TB to identify areas (districts) with higher TB prevalence, particularly when analyzing pilot community activities. The TB prevalence variation among areas is very large: in one pilot zone 133 000 household members screened resulted in 1,870 referred to HC, of which 979 were evaluated (52%) and 29 PTB+ detected (3%); in two other zones 148 000 persons were screened, 3 800 were referred of which 2 083 were evaluated (55%) and 227 PTB+ were found (11%).14 HEAL-TB analysis of the number of PTB+ found among contacts of PTB+, PTB- and extra-pulmonary TB (EPTB) patients shows that the difference in risk of disease among these contacts is not very large; so screening of contacts is useful for all forms of TB and HEAL-TB has correctly started contact tracing for all cases of TB, different from the NTP or WHO-recommended policy. Contact tracing is generally passive, with the patient asked to bring household members rather than staff going to the household, which would be a better method. The average number of household contacts registered per TB case seems too low for a country with about 50% of the population under 15 years old, and the proportion of children indicated IPT is very low (under 30%). The proportion of PTB+ found in contacts of EPTB patients (2-4%) is an indication that the very large proportion of EPTB reported in the country (one third of the TB cases reported) may have correct diagnosis 14 HEAL TB, Oromia, August 2014 20 HEAL TB ETHIOPIA MID-TERM EVALUATION As per national norms, diagnosis of PTB is done on the basis of positive microscopy or clinical criteria including a short treatment with non-specific antibiotics. The proportion of confirmed cases is about 50% of PTB and one third of the total TB cases diagnosed. EPTB is much higher than the global average (among the highest in the world); and the proportion of PTB cases confirmed by microscopy is lower than global recommendations but within acceptble limits. It also corresponds to the proportion found in the prevalence survey, slightly lower than 50% of the bacteriologically positive cases using two sputum smears and one culture (note that the survey methods underestimate real prevalence of bacteriologically positive TB by about 20%, as one culture only identifies about 80% of culture positive samples15). Culture (usually Lowenstein Jensen) is generally not used for diagnosis due to the long waiting time and problems of transport, and it is not included in the government guidelines. Access to rapid methods is still very limited, both by the laboratory capacity and by the long distances and difficult transportation. Culture and drug sensitivity testing (DST) are expanding for PMDT diagnosis and follow-up. Use of GeneXpert is expanding slowly in the country (26 sites by 2013 according to the 2014 WHO Report) but is used mainly to detect MDR-TB. This technology would improve quality of diagnosis in smear negative cases, but is still too expensive for general expansion in the country, and its contribution to general case finding is limited. It could be considered for special groups such as HIV infected persons. 2.2. LABORATORY The HEAL-TB activity has supported the expansion of access to the diagnostic network by implementing microscopy units in health centers (target 100%), through supply of microscopes, Gene xpert and cartridges, reagents for microscopes and ancillary equipments, and staff training including microscope maintenance. The number of microscopy units per 100 000 population reported by the NTP to WHO in 2013 was 2.3 (one per 43 500 inhabitants). The activity area has one microscopy unit per 25 000, but about 20% of the facilities do not have a laboratory technician because of insufficient availability, so the active microscopy units cover a population of 33 000 on the average. HEAL-TB has introduced zonal project laboratory officer, that did not exist in the government structure; this may be a critical component for the success and sustainability of the activity. There is a large staff turnover and limited regional capacity outside HEAL-TB to train and mentor new staff. The large number of microscopy units facilitates access to diagnosis, but results in lower frequency of positive slides for the readers, with a substancial risk of loss of quality, particularly with decreasing prevalence of infectious TB in the population. The introduction of fluorescence microscopy has increased the capacity of technicians to process sputum microscopy samples (one technician can read a maximum average of 20 slides with light microscopy and three times more with LED) so there is now over-capacity of the microscopy units. Table 2. Annual workload of TB microscopy, North Showa zone, 2006 EC (population 2.3 million) Number of functional health facilities Number of slides examined in 4 quarters Slides per day per Slide positivity 15 Toman’s tuberculosis. Questions and answers. Second edition. WHO, 2004. HEAL TB ETHIOPIA MID-TERM EVALUATION 21 Diagnostic Non￾diagnostic Negative Positive Total microscope rate % 41 23 15 486 1 029 16 515 1.6 6.2 Source: HEAL-TB It was observed that the workload of microscopists is very low (Table 2). In some cases most of the work was for control of treatment, and in one case the sample for EQA consisted of 35 negative slides and one positive: 100% of the positive slides seen in the quarter. The problem will increase with reduction of TB prevalence and it is not the responsibility of the laboratory, but a consequence of low general use of health facilities by the population. A major achievement of HEAL-TB, reported at all levels, has been the estabishment of a system of microscopy EQA through blinded rechecking (Figure 4). The quality of results has gradually improved and field observation evidenced no false positives for several consecutive quarters in some sites. However the discordances, while diminishing, are still important (particularly for false positives, that can result in false diagnosis) and the activity should continue. When the positive slides are too few to maintain quality, on the job training using panels of slides may be necessary. It should be noted that in the TB prevalence survey a large proportion of results with only one smear positive and not confirmed by culture or X-ray (13/61, 20%) were discarded as not TB, indicating that smear examination screening might be misleading. Figure 4. Sites with external quality asessment of microscopy. HEAL-TB, July 2011- September 2014 Source: MSH HEAL-TB, November 2014 In contrast with the EQA development, limited direct supervision was observed, and at least in one case the laboratory register was not filled appropriately and no action was taken by supervisors. In general it appears that the laboratory staff take their clinical role and quality of work seriously, but do not have sufficient involvement in the public health aspects of TB control. Many sites visited did not have visible information on trends of number of persons or slides examined, or positivity rates. This 4 104 82 957 1 10 100 1000 EQA center EQA participating sites EQA sites expansion from July 2011 to Sept. 2014 Before HEALTB Currently 22 HEAL TB ETHIOPIA MID-TERM EVALUATION is in contrast with the trends in cases found and treatment outcomes, registered in graphs in the walls of Health Centres and Health Posts. The recent introduction of rapid molecular methods for diagnosis of TB and of resistance to rifampicin for MDR will improve the quality of diagnosis but cannot be expected to increase substantially the number of TB cases, as the number of clinically diagnosed should diminish. In 2013 the country reported 26 GeneXpert sites.16 A desirable increase of TB reported cases and more rapid initiation of treatment does not depend on the laboratory but mainly on the identification of persons with symptoms and improvement of referrals. The most effective way of improving detection would be to develop, test and implement methods for diagnostic microscopy that do not involve patient referral, such as sending sputum samples or prepared slides from the community as soon as a presumptive TB case is identified. The experience of TB REACH in the Sidama zone, recently published by the Stop TB Partnership, can be a useful model.17 2.3. TREATMENT AND DRUG SUPPLY Ethiopia uses short-course standard treatment regimens to treat TB patients under directly observed therapy (DOT). Health workers are responsible to ensure provision of recommended daily doses of drugs for appropriate duration. DOTS can be provided at the hospital, health center or health post, workplace, or at home. Anti-TB treatment duration is divided in to 2 months of intensive and 4 months of continuation phase. The patient must swallow anti-TB drugs in front of a health worker or HEW to ensure the daily intake of drugs, this treatment option is further decentralized to the community or home level through the engagement of TB treatment supporters. Patients should be given the option to choose the place of treatment which can be a health facility, health post or home and the supervisor for treatment that can be a health worker (if patient chooses a health facility) or a TB treatment supporter (if patient chooses home). The HEW collects anti-TB drugs from HC on monthly basis; the TB treatment supporter accompanies the patient and collects anti TB drugs weekly from the health post or health centers. However, there is limited implementation and large variations regarding involvement of treatment supporters in HEAL-TB area. For instance, in one of the HPs visited in Oromia region the team travelled 30 minutes, which could be over one hour for patents coming for daily treatment; here the use of TTS cold have been the best option. Although the national guidelines allow for the use of treatment supporters and HEWs, the evaluation team noted that treatment practices varied from the guidelines in many locations. Many providers seemed to prefer that patients come to the health facility throughout treatment, and many patients chose to receive treatment from the health center rather than the health post (through HEWs). According to providers and patients, there is a perception of better quality at the health center than the health post. This may also reflect underlying concerns about stigma if patients do not want to accept treatment at a community level. While the default rates remain low in Ethiopia, facility-level DOT puts a lot of costs and burden on the patient during treatment and follow-up. The use of patient kits is expanding in the activity, reaching full coverage. It was observed that the health post staff does not receive the full kits from the HC but only drugs for a week (as per NTP guidelines); this defeats one of the purposes of the kit which is to have all drugs available and 16 WHO Global TB Report, 2014 17 Improving Tuberculosis Case Detection. TB REACH, 2014, page 20. HEAL TB ETHIOPIA MID-TERM EVALUATION 23 overseen by the patient. Follow-up of the patient during the duration of the treatment is through monitoring of clinical assessment (signs and symptoms), weight measurement and follow up acid fast bacilli (AFB) sputum examinations at 2nd, 5th and 6th for new, and at 3rd, 5th, and 8th for retreatment patients. HEAL-TB supported the two regions in improving knowledge and skills of health care workers and health extension workers. Additional support was provided to monitor adherence to anti-TB treatment. This support included training to health care worker (HCW) and training of trainers (TOT) on integrated refreshment training for the health extension workers to improve skill and knowledge of TB treatment at health facilities and health post level. This training was included in the comprehensive TB and TB/HIV training modules for health workers and IRT for Health extension workers. HEAL-TB supported the Zonal Health Departments and woreda (district) health office to conduct joint supervision and mentoring to health facilities and selected health posts to monitor quality of DOT. HEAL-TB provides technical assistance (TA) and logistic support to perform supervision. Some facilities have received medical and ancillary equipment that help improve the quality of TB treatment. These includes shelves for health facilities, tables and microscopes HEAL-TB provided recording and reporting forms to health facilities to improve their recording and reporting practice. Additionally, facilities have received job aids, standard operating procedures (SOPs), guidelines, etc. A new patient register has been introduced recently by the NTP. The treatment form in use at Health posts and by HEWs to monitor dug intake/delivery and evolution of the patient does not include indication of the site of disease or initial result of the microscopy (although it has spaces for the follow up), and no place to record the treatment outcome and date. The treatment outcomes have improved consistently, both in the initial 10 sites of phase 1 and in the 11 of phase 2 (Figures 5 and 6). Figure 5. Treatment outcomes in new smear positive PTB in the HEAL-TB original zones, 2010-2014. Source: HEAL-TB report April-June 2014 Figure 6. Treatment outcomes in new smear positive PTB in the HEAL-TB new zones, quarter 4/2013 to quarter 2/2014. 88 91 92.2 95.5 71 77.6 85 89.8 70 75 80 85 90 95 100 2010/11 Q2/2012 Q2/2013 Q2/2014 % SUCCESS % CURE 24 HEAL TB ETHIOPIA MID-TERM EVALUATION Source: HEAL-TB report Observation of visited health facilities: The TB clinic at the health center is well ventilated and separated from the general OPD. Almost all health centers have a trained TB focal person working in the TB clinic. Patient data were well registered; TB patient kits were well labeled for each patient on DOT in health centers. Patient treatment cards are used for patient follow up. At the health post patients were provided DOT and daily doses were monitored using treatment cards. It was observed that the form to monitor treatment intake or delivery to the supporter (yellow treatment card) has lines to control the evolution of the sputum results but does not indicate the diagnosis or site of TB or the initial microscopy, and does not include a place to register the treatment outcome and date. Health posts visited in the Amhara region used TB treatment supporters (TTS) to further decentralize DOT, but none of the health posts visited in Oromia used TTS for decentralization. In general, they use them as an alternate to replace the HEW during their absence. Guidance to engage TTS in DOT was not clear in visited sites in Oromia. In general, the impression was that staff capacity to implement activities, team organization, coordination capacity and working with MOH is better in Amhara than in Oromia, but the reasons were not identified so no specific recommendation can be made. DOT in health posts is at its early stages in Oromia compared with Amhara. DOT is provided strictly either at health centers or health posts; health workers and health extension workers are well trained; treatment data is well documented in the registers and patient cards and outcome trends are plotted and visible in the walls; there are good infection prevention practices at health centers (with natural ventilation) but poor infection control for inpatients in some hospitals; reference documents and guidelines were available. The patient card lacks some basic information needed for patient follow up. MDR-TB treatment initiating centers in the two regions were rapidly expanded from one (Gondar University Hospital) in 2011 to 16 sites in September 2014, with more than 300 MDR-TB treatment follow up centers. The number of patients with MDR-TB on treatment has increased from 56 in year 1 to 254 in year 3. Drug supply management HEAL-TB has provided support to national TB control program and regional health bureaus in strengthening the anti-TB drug management system. At national level support was provided to the PFSA by introducing new quantification tools and providing training on the tools. They also closely work with PFSA during planning, quantification and distribution of anti-TB drugs. Additionally technical and financial support was provided at national and regional levels during the shift of the anti￾87.9 91.6 93.8 71 71.6 78.5 70 75 80 85 90 95 100 Q4/2013 Q1/2014 Q2/2014 % SUCCESS % CURE HEAL TB ETHIOPIA MID-TERM EVALUATION 25 TB drug supply system from blister form to patient kits, including guidelines and training to health facilities and program managers at different level during the shift.  Training on the integrated pharmaceutical and logistics management has been provided to health facilities in the regions. This helped PFSA to build the skill and knowledge of health workers in implementing the IPLS strategy to distribute drugs to facilities.  Capacity building to the PFSA at national and regional level which provide technical support and coordinate technical working groups activities  Supervision to PFSA to conduct joint supervision to health facilities in collaboration with regional health bureaus, supported review meetings for the drug supply management  TB patient kit implementation: financial and technical support was provided to RHB to provide training to health facilities, ZHD and woreda health offices to implement the shift from blisters to patient kits. Support was provided to PFSA to improve the distribution of first and second line drugs  Supported PFSA in conducting baseline assessment of DSM at the facility level, analysis and dissemination of assessment findings for further action  Procuring reagents on emergency basis while out of stock. Examples are the procumbent of reagents for florescence microscopy and electrolyte machine for the MDR-TB centers. Frequent stock outs of first line anti-TB drugs and laboratory reagents and sputum containers were reported by the health offices and health facilities visited. Stock-outs of reagents and sputum cups can result in delay of diagnosis and have a significant impact on the case detection and quality of treatment follow up; and they are probably due to improper planning and procurement. In most places each health center / woreda health office tried to fix the challenge by borrowing from near districts or health facilities. HEAL-TB reports indicate that the stock-out rate of selected tracer adult anti-TB drugs has decreased from what it was in 2011. 2.4 MDR-TB When MSH started implementation of HEAL-TB activity in 2011, the country had only two MDR-TB treatment centers, in St. Peter hospital in Addis Ababa and in the Gondar University hospital. By August 2011, 260 cases of MDR-TB were enrolled on treatment and more than 180 confirmed cases were on the waiting list for second line treatment. To strengthen the response to MDR‐TB, HEAL￾TB planned to support scale-up of MDR-TB case finding and treatment sites in the Oromia and Amhara regions, contributing to the national expansion plan. This included early identification of MDR-TB suspects; sputum specimen transportation for early diagnosis; developing a mentorship model for clinicians on MDR‐TB; implementing a capacity building plan for clinicians on MDR‐TB; and coordinating with other partners to put MDR‐TB cases on treatment. The engagement of HEAL-TB in supporting response to MDR-TB has increased year after year. They supported the NTP in implementing the shift of model of MDR-TB care from inpatient model of care (with hospital admission for at least 4-8 weeks) to ambulatory model of care - where patient receives treatment and care for MDR-TB as an outpatient at TICs or TFCs. TICs initiate MDR-TB treatment, follow patients until treatment is tolerated and maintain acceptable level of adherence. Once patients are stabilized, they will be linked to TFC. TIC will continue to serve as a monthly follow up center, clinical mentorship, and supportive supervision and also act as catchment area meeting place for their respective TFCs. At TFC, MDR-TB patients will continue to receive all the necessary care such as directly supervised treatment, screening and management of common side effects and refer patients with serious side effect or any other condition which need referral. The first sites for the ambulatory model of care in Amhara (Boru Meda and Debre Markos Hospitals) and Oromia (Nekemt Hospital) regions were supported by HEAL-TB, and they were fully renovated and equipped by HEAL-TB as referral centers. Currently, MDR-TB treatment initiating centers in two regions expanded from one (Gondar University Hospital) in 2011 to 16 sites in Sept. 2014 with more than 300 MDR-TB 26 HEAL TB ETHIOPIA MID-TERM EVALUATION treatment follow up centers. Number of patients with MDR-TB on treatment has increased from 56 in year 1 to 254 in year 3. HIV testing and counseling services were provided for MDR-TB patients at all TICs and TFCs. In 2014, HIV testing was done for 3338 MDR-TB cases and 68 cases (17.5%) were co-infected with HIV. All co-infected cases are linked with ART clinics but outcome data is not ready for the cohort. MDR-TB treatment outcomes (interim and final) are monitored at the TIC levels. Patient level data are electronically captured in all TIC in both regions using data base software provided by HEALTB. Final MDR-TB treatment outcome was evaluated for 17 patients enrolled in the Gondar University hospital (supported by USAID through TB CARE 1) two years ago and later the support continued through HEAL-TB. The cure and success rate was 59% and 76%, respectively. Interim treatment outcome was analyzed for 197 patients who completed their six months of treatment over the last two years. 93 (47.2%) had culture results that were negative, 5 (2.5%) had positive results, and 67 (34%) had unknown results. Patient follow up is coordinated by TICs. There is MDR-TB clinic day at TIC on monthly bases to monitor patient progress from all TFC in the catchment areas and provide psychosocial supports. Patients will be visited by zonal TB focal and zonal HEAL-TB officer on the MDR-TB day. Sample for culture, electrolyte and other tests are collected on this day. Sputum for the culture test is sent to regional laboratories through postal system. HEALTB has provided 60 electrolyte machines to almost all TICs to conduct electrolyte tests at TIC level. But some sites were waiting for installation of the machine. Rapid expansion MDR-TB treatment facilities have significantly improved access to treatment. However, our observation in some of visited TICs showed that staff in TIC was overstretched to support all TFC as per the guideline for mentoring and catchment area meetings. This practice will significantly compromise quality of care. Further decentralization of the TIC role for close patient monitoring should be considered. Additionally strong commitment of zonal and woreda level offices are very important both for detection of more drug resistance cases as well as improving quality of treatment and care. HEAL-TB provided support to regional health bureaus in strengthening M&E system for PMDT. Patient level data are electronically recorded in TIC using software for data management. Reports are compiled and sent to regional health bureau with support of HEAL-TB officers at zonal level. In the revised HMIS, MDR-TB related indicators are included to capture all information related to MDR-TB. The NTP and regional health bureaus strongly acknowledged the high level of support received from HEAL-TB to shift from the very expensive and less accessible hospital based care to a more decentralized, less costly and more patient accessible ambulatory care. HEAL-TB contributed to the expansion of TICs from nil to nine in the Oromia Region and from one to five in the Amhara Region. The following major MDR-TB specific supports were provided by HEAL-TB:  Financial and technical support was provided to the two RHBs to renovate hospitals to fulfill the minimum criteria required to initiate MDR-TB treatment services.  MDR-TB training was provided with support from HEAL-TB to staff from MDR-TB TIC and treatment follow up centers (TFC). For some facilities formal training was provided by other partners and HEAL-TB provided onsite training through supervision and mentoring  Financial and logistic support for the catchment area meetings at the beginning of the initiation of TIC. Recently the Global Fund provided financial support while HEAL-TB continued technical support.  Supported the regions in transporting sputum samples of presumptive MDR-TB cases from health facilities to the regional laboratory during the first and second years of the activity (Table 3 and HEAL TB ETHIOPIA MID-TERM EVALUATION 27 Table 4). This contributed to diagnosis of MDR-TB cases and follow up of patients on second line treatment. With expansion of laboratory capacity, the number of suspects examined has increased and the yield in 2013 has declined. Currently sample transportation has been outsourced to the Ethiopia postal system (EMS), a sustainable national mechanism. During the shift HEAL-TB supported regional laboratories to train TB focal persons from all woredas for sample collection and delivery to the postal system and processing at the regional laboratory. Table 3. Number of samples transported and confirmed MDR-TB. 2011-14. Year Number of tested suspects Number of confirmed MDRTB cases 2011 263 34 2012 450 55 2013 2534 39 Source: Adam Regional laboratory Results of the drug susceptibility tests are sent to the health facilities who requested the testing through the postal system. For samples with confirmed MDR-TB, the regional labs communicate the result to the requesting facility by telephone. Few health facilities have SMS printing mechanism to immediately communicate the results of MDR-TB cases Table 4. Performance of MDR-TB diagnosis and treatment in Amhara Region, 2005-2006 Indicator EFY 2005 EFY 2006 Number of presumptive MDR -TB cases for whom DST was performed 554 1,543 (Culture & DST: 77; GeneXpert: 757) Number of MDR-TB cases detected 126 130 (both MDR & RR) Number enrolled on SLD treatment 117 114 Treatment success rate N/A 83% HEAL-TB provided technical support through experts at the ZHD on technical and managerial issues during the catchment meetings of TIC with TFC. TICs received support from HEAL-TB to strengthen their capacity to provide quality service of MDR-TB treatment. This included supervision to improve practices on TB infection control and linkage with the TB program, TFC and regional laboratory. TIC received supplies required for routine activities such as respirators for infection control, job aids for health workers and for patient education, treatment cards, computers with software for data management, technical support to manage difficult cases and transportation of patients from health facilities to TIC for diagnosis. HEAL-TB supported the reparation for transfer of MDR TB cases from the TIC to TFC at health centers, including staff training,, mentoring, and provision of formats for recording and reporting Field observation noted that the majority of MDR-TB patients did not have previous exposure to second line drugs (SLDs), and therefore the probability of extremely drug resistant TB (XDR-TB) is very low and the probability of cure with current regimens should be high. MDR-TB is a challenge and needs to be addressed, although the proportion of MDR￾TB cases without previous treatment is still relatively low: 1.6% for new and 12% for re- 28 HEAL TB ETHIOPIA MID-TERM EVALUATION treatments cases in 1st survey, 2.3% and 17.8% in the 2 nd survey18 . MDR-TB patients were screened for HIV, and the results in the last year are included in Table 5. Table 5. HEAL-TB MDR and HIV activity, October 2013 - September 2014 Region MDR-TB under follow-up in FY2014 Tested for HIV HIV + MDR￾TB patients HIV positivity rate Availability of GeneXpert Oromia 157 151 23 15.2 3 Amhara 246 237 45 19.0 2 Total 403 388 68 17.5 5 Source: HEAL-TB, October 2014 The major challenges in MDR-TB are: 1. Delay in sputum sample transportation for culture and return of results from culture center to TICs has affected early decision making at health facilities. 2. High running cost for electrolyte machines. It requires continued reagent for 24 hours without interruption. If the machine is switched off for four days for any reason (power supply or lack of reagents), it requires maintenance. 3. Weak linkage between some TIC and TFC (for example Shashamene Hosptal). 4. Weak networking of facilities with postal system for sample transportation. Sometimes it takes too long before samples arrive at regional labs under poor storage system (reduces quality of sample). 2.5. TB/HIV The HEAL-TB activity provided technical support to improve service expansion and quality of care by sending mentors to the zones, districts and health facilities on quarterly basis. During mentorship, on￾job training was provided to health workers which include TB, TB-HIV and drugs and supplies management. . This is enhanced by regular joint supervision to HCs and districts using checklists comprising quality indicators and standards of care. In addition to monitoring the activities, joint supervision is intended to build the technical capacity of district office to develop the capacity to effectively supervise HCs. TB-HIV related capacity building was integral part of the training given to health workers on comprehensive TB training, to laboratory personnel to improve diagnostic skill and pharmacy professionals on IPLS to ensure service expansion and quality of care. They have provided job aids, SOPs, standards of care and screening tools to test TB patients for HIV. This is complemented by providing laboratory supplies were provided to the activity area to improve diagnostics, including microscopes, reagents for AFB and LED, GeneXpert and cartridges. As a result of the technical capacity building in the health system, the proportion of TB patients screened for HIV has increased to over 90%, patients are offered co-trimoxazole and linked to ART 18 Global Tuberculosis Control WHO Report 2009 HEAL TB ETHIOPIA MID-TERM EVALUATION 29 services19 and improved data quality (Figure 7). CPT coverage increased to over 80% and ART to 72% (June 2014). Field observation showed that the time from diagnosis of TB/HIV and the start of ART was very short (usually on the same day as the positive HIV result was received); and that in the facilities visited every co-infected patient was treated with ART. Figure 7. Percent of registered TB patients tested for HIV through HEAL-TB, Oct. 2012- Sep. 2013, and target (90%). Source: MSH HEAL-TB activity. Second annual progress report; October 1, 2012-September 30, 2013 2.6. INFECTION CONTROL HEAL-TB has contributed to infection control through the comprehensive TB and TB-HIV training which included infection control (IC) as one key component. They provided technical and logistic support and mentorship to healh facilities and supported zones and districts to implemet TB IC interventions. To improve TB IC in MDR-TB treatment intiating centres, HEAL-TB has renovated isolated rooms for TB screening and admission of cases. They have also provided IC supplies such as N-95 masks. In addition, the activity supported the development, printing and distribution of TB IC job aids to health facilities. During the field vists to health facilitie, it was observed and confirmed that the general understanding about TB IC has improved with the presence of IC committees, plans and imlementation in some facilties. Additionaly, all of the visted outpationent health facilties have adequate space and good natural ventilation: TB clinics are separated from the genearl OPDs. However, hospitals are overcrowded, the in-patients rooms are closer to general OPDs and the waiting area is shared by coughing and other patients. Outpatients are screened for persistent cough and given priority to consult the physician and being screened by microscopy. Sputum is collected in the open air, outside the facility. TB cases are treated with efective regimens on outpatient basis. Weather and open facilities contribute to reduce the risk of transmission in the health facilities. The microscopy laboratories showed reasonable precautions but could benefit from better organization of work. However, observation of procedures for inpatient care in some general hospitals showed serious deficiencies in the staff knowledge and 19 Observed during the field visits to health facilities 70 89 91 90 93 94 90 90 90 90 0 10 20 30 40 50 60 70 80 90 100 Baseline Year 1 Year 2, Q 1 Year 2, Q 2 Year 2, Q 3 Year 2, Q 4 30 HEAL TB ETHIOPIA MID-TERM EVALUATION practices regarding IC. Hospital TB patient wards shared common air with wards for general care, and there was no effort to achieve ventilation to the outside. In most cases correction needs only administrative procedures or minor changes in air flow or patient distribution, using local resources of low cost. Patient wards in MDR facilities (TIC) included new and treated patients - although this may a temporary situation as new facilities are under construction or renovation. In some of the facilties visited infection prevention (IP) committee is being established and did not yet start active engagement. 2.7. COMMUNITY-BASED TB CARE The activity implementation was started by realigning the proposed community-based TB care (CBTC) activities with NTP. The activity then supported the revision of National guideline for Community-Based TB Care and training of trainers to cascade the training for HEWs in the regions. They have also printed and distributed patient treatment card to be used by HEWs and TTS for the decentralized DOT and registers for presumptive case and contact tracing. This has contributed to increased engagement of HEWs. However, the understanding to decentralize treatment and use of TTS in the community and the practice varies among the visited zones. The health workers tend to prefer treating TB cases at health center than giving options to the patents to use decentralized treatment in the community in health posts despite the availability of clear guidance to implement. HEAL-TB provides technical and financial support to the zones to conduct supervision to districts. Most of the visits are limited to district level. Occasionally, the activity visits at most one health center and health post in some districts. The technical support is limited to the level of district and lacks the capacity to reach the PHCU, a clear missing link that makes support inadequate to embrace the community engagement. The district health office conducts integrated supervision to health centers which in general has limited focus on TB and does not even reach most of the health posts (HPs) in the catchment area. The districts also conduct general district based review of activities still with less attention to CBTC. There is a monthly reporting of presumptive cases from the community to health centers but there is no clear mechanism to document the number of presumptive cases identified, referred, examined and diagnosed to have TB. It was observed that most heath centers have established referral mechanisms so that presumptive TB cases referred from the community have easier access and established mechanism for presumptive cases to directly visit the TB clinic to get expedited diagnostic process. However, the patient pathway is complicated for presumptive cases referred and most referrals do not reach the HCs and did not access the services. HEWs identified and referred presumptive cases from most communities but far below expected. Even among these, most presumptive cases did not reach diagnostic units, inadequate feedback is given and there is no follow-up mechanism for those who did not reach the diagnostic units. In order to improve identification and referral of presumptive cases, HEAL-TB implemented different interventions including community based mobilization, TB campaigns, community TB week and active case finding. Despite this, about 50% of the identified and referred presumptive TB cases did not reach diagnostic health facilities, and of those who reached the health facilities, the positivity rate was very low. An analysis of house by house active case detection in selected zones of Oromia and Amhara shows significant differences in the two regions (Table 6). The average number of members screened by household was similar, but the proportion with symptoms was double in Oromia (average 2%). The proportion of referrals reaching the Health Center was similar (40% in Amhara and 53% in Oromia) but the yield of TB cases was 7.6% in Oromia and only 1.2% in Amhara; for smear positive cases respectively 6.1% and 0.8%. This shows a much higher prevalence of TB in Oromia, where 241 smear positive cases were identified in 340.000 persons screened (0.7 per 1000) than in Amhara (121 in 2 HEAL TB ETHIOPIA MID-TERM EVALUATION 31 million or 0.06 per 1000). Both results are much lower than the yield of patients attending health facilities for any reason, showing that community screening is more costly and less efficient than health facility screening. However, as said above, only a small part of the population attends health facilities for any reason. Collecting sputum samples in situ for transport to the diagnostic unit instead of referring suspects would double the yield in both regions. Table 6. Yield of enhanced house to house active case finding in selected zones, October 2014 Indicator Amhara Oromia Total Number of households visited 591,951 90,285 682,236 Number of household members screened 2,112,801 339,834 2,452,635 Number with presumptive TB 39,769 (1.88%) 12,493 (3.68%) 52,262 (2.13%) Number referred to health center 35,709 7,307 43,016 Number evaluated in HC 14,342 3,920 18,262 All forms of TB identified (% yield) 179 (1.2%) 300 (7.6%) 479 (2.6%) Smear positive TB cases identified 121 241 362 Smear negative TB cases identified 23 26 49 EPTB cases identified 35 33 68 Source: HEAL-TB HEWs collect drugs on weekly basis for TB cases while the patients visit HPs daily. TTS were not used in patient care in most of visited health posts. The national form to register patient drug intake or treatment supervisor receipt of TB drugs is not appropriate: it includes lines to register the microscopy results for control of treatment but does not indicate that the patient has TB, if the site is pulmonary or EP, and if PTB what is the initial microscopy status. It also does not indicate the outcome of treatment and date. In several facilities visited, the staff has registered this essential information anywhere in the form, but the procedure is not standard. Patient and community interviews identified that there is prolonged and complicated patient pathway and long distance to be travelled to take daily doses, and low community awareness about the disease which would be more complicated if TB related stigma prevails. In general, community based activities have contributed to an increase in the number of presumptive cases identified, referred and diagnosed to have TB, but not to the level that many people expected them to contribute. 2.8. SUPERVISION, MONITORING AND EVALUATION HEAL-TB has placed a TB clinical officer and laboratory professional in each zone to support activity implementation and provide mentorship for the government staff in those areas. HEAL-TB also provided a vehicle per zone to facilitate field work in the activity sites, and motorcycles for ditricts to ensure regular supervison. The activity has strengthened the capacity of the health system to supervise, monitor and evaluate TB activities, and by extension the general health system capacity. The recent integration of TB 32 HEAL TB ETHIOPIA MID-TERM EVALUATION information system into HMIS has resulted in reduction of the number of indicators, some of them crucial to evaluate program performance and impact.20 One instrument which is very useful for planning, supervision and evaluation and that was rarely seen in field observation is mapping (of geographical areas and communications at managerial levels, of health facilities and roads at service level).21 This is even more useful when there is frequent staff turnover as reported almost universally during field observation. Several years ago, the Health Expansion Program used this instrument, and maps indicating the health facilities, churches (as places for community mobilization) and roads were seen in some districts and health posts. The number of persons examined by microscopy (from the microscopy register) and the positivity rate are available but not utilized for monitoring of case detection or activity impact on the TB epidemiology. 3. FINANCE Although the financial management of HEAL-TB was reviewed, a detailed audit of the activity was outside the scope of this evaluation. HEAL-TB implements activity-based budgeting where activities included a detailed and costed budget. There do not appear to have been serious pipeline issues, and spending is monitored regularly along with the technical aspects of the work through biweekly management meetings. In addition, technical advisors are tasked with regularly monitoring of the activities in the technical areas both for quality of implementation and spending. At a central and regional levels, the activity develops quaterly activity plans to enable them to monitor the progress of activities and resolve delays in spending and implementatio. The regional satellite offices are allocated funding which is monitored by regional financial officers and monthly vouchering. All partners are required to submit monthly vouchers in order to monitor spending. KAPTLD and ALERT have targeted and specific activities and receive small amounts of funding per year, and the budget for PATH is fairly set from year to year as most of the budget is staffing and salaries. There were no major reported problems with getting timely and accurate financial reports, although ALERT has been slow in implementing and reporting on activities and finances. HEAL-TB staff have been more actively engaged in training them on grants managements, reporting and implementation of activities in order to resolve some of these issues. However, MSH does not perform regular internal audits on HEAL-TB. VI. CONCLUSIONS AND LESSONS LEARNED 1. Management: HEAL-TB activity is achieving its objectives. The activity follows appropriate strategies, plans coordinating with the NTP and the regional staff. Its support for technical capacity building is well recognized. It has achieved expansion of diagnostic and treatment capacity and improved quality of microscopy. The activity contributed to political commitment and HSS for sustainability. There is an epidemiological impact. 20 Revised HMIS incorporated more indicators 21 Management of Tuberculosis. Training for District TB Coordinators. WHO/HTM/TB/2005.347. HEAL TB ETHIOPIA MID-TERM EVALUATION 33 2. Case detection and diagnosis: USAID support to the Government of Ethiopia TB program through HEAL-TB activities has expanded the diagnostic laboratory capacity and implemented a system of EQA. Screening of general outpatients for microscopy has been implemented but it is not enough to increase case detection because the population recognition of illness and use of government health facilities is very limited. The population does not attend HF for general care and the suspects do not recognize cough as a major reason to seek care, even when identified and referred by HEW. HEAL-TB has implemented successfully decentralized EQA in the regions and improved quality of diagnosis by decreasing discordance rates. Community activities to identify presumptive TB cases and refer them to health facilities but do not result in sufficient work for the laboratory. The reduction of PTB+ prevalence results in insufficient microscopy work to maintain quality and interest (too few smear positive). Campaign initiatives to detect suspects and new cases in the community are not very effective and recommendable due to the great loss of persons referred to the HF for microscopy due to different barriers. The large proportion of children in the community (50% or more) suggests that control of contacts of TB patients and isoniazid prophylaxis in children would have a large impact to reduce future TB incidence. The CDR is not an appropriate indicator and target for populations small or with large variations in TB prevalence, as there are no appropriate estimates for Regional, Zonal or district levels. The best indicator is the number of persons examined for diagnosis, and the positivity rate, which can be easily obtained from the laboratory register and plotted in graphs to follow trends. 3. Treatment and drug supply: DOT is practiced strictly in health facilities (HC and HPs) but further expansion DOT to community level through the engaged of TTS is limited. The use of patient kits is expanded rapidly. There was irregular supply of FLD, with shortages partially solved through HEAL-TB collaboration to redistribute. There seems to be no reserve stocks of FLD. Treatment outcomes are very good (high success and cure rates, low deaths and failures) and graphs of performance trends are used by staff to self-evaluate. There is limited national and project data on the proportion of lost cases (transfer out and not-evaluated) in the cohort. At national level in 2010 100% of the reported cases were included in the cohort, with 10% not evaluated; in 2011 only 83% were included and 4% of them not evaluated.22 WHO has stopped reporting clearly the proportion of transfers and not evaluated in 2014 and produced no data for 2012 outcomes except success (91%). This could be used to find out ways for patients to start treatment in the health facilities they could finish treatment. Very good support was provided in the area of DSM which has evidenced with improved knowledge, skill and practices of health facilities. But still there are huge gaps to be addressed to challenges related to DSM system 4. Infection control: TB infection control at outpatient health facilities seems adequate due to the availability of well-ventilated spaces, but it needs much more attention in hospital inpatient settings, which have a high risk of transmission from TB patients to other inpatients and staff. Most of the interventions required are administrative and of low cost, such as adjustments to ensue air flow to the exterior and separation of TB and other patients, plus retraining of program staff to identify the 22 WHO Global TB Report 2013. 34 HEAL TB ETHIOPIA MID-TERM EVALUATION areas of risk. HEAL-TB supported the implementation of infection control by mentoring and supervision, provision of supplies and job aids. 5. TB/HIV: HEAL-TB has supported service delivery and improving quality of care and increased coverage of HIV screening, CPT and ARV in co-infected TB patients. The key challenges in the area of prevention and diagnosis of TB in HIV-infected: limited provision of IPT and CPT for adults, inadequate decentralization of ART clinics, limited diagnostic capacity for HIV+ clients and limited district capacity to support and monitor activities in the catchment area. Moreover, lack of indicators to capture IPT and CPT data has compromised the monitoring and evaluation. 6. PMDT: HEAL-TB has played key roles in rapidly expanding MDR-TB treatment sites (TICs and TFCs). In addition to site expansion, capacity of program management at regional, zonal and woreda level is built through contributed to capacity building by training, mentoring, and supportive supervision and review meetings. At regional level the coordination between HEAL-TB and RHB is good but HEAL-TB has promised to renovate some facilities and did not respond for several months. Some differences were noted between the two Regions. In Amhara visited zones were fully engaged in the PMDT and they have good knowledge required. In Oromia the level of engagement of visited ZHDs on MDR-TB was very limited and they were less clear about what to do for the management of MDR-TB. The fast growing number of TFC is becoming a challenge for TIC to provide frequent mentoring. In some TICs there is no laboratory service to properly monitor side effects of second line drug (e.g. absence of electrolyte test due to inability to install the machine or shortage of reagents). There is significant difference between the two regions in supporting TFCs. Health care workers at some TFC in Oromia received suboptimal training before they started supporting second line treatment. A high number of TFCs per TIC seems challenging for TIC in providing frequent support. There is absence of critical lab test at some TICs and the engagement of ZHD in PMDT activities in some places is minimal. Alternatives would be to strengthen TICs or increase their number; the decision should be made taking into account staffing and access (transport) and growing experience during expansion of the ambulatory model - the recommendation of this report to use mapping is also valid here. HEAL-TB has provided training on TB for the community, HC and district offices to ensure coordination and implementation of community based interventions. Presumptive case registration and referral formats are available in HPs. There is increased number of presumptive cases identified and referred but limited number that reached HCs for diagnosis. There is limited feedback and follow up for those who did not reach the HCs. Hot spot focused community based interventions were implemented including active search for presumptive cases. There is limited link below districts and HC to health post (HP) link functions suboptimal. 7. Community-based TB care: Implementation of community-based TB care is inadequate. It requires due attention to improve access to the patients identified from the community, ensuring access to diagnostic units and providing patient friendly services. This requires evaluation of available evidence and prompt action to reverse declining case notification. Transport of sputum samples or slides should complement the referral of persons with symptoms. Sputum transport requires less training, but is more difficult (risk of contamination, must reach the microscopy unit in less than 5 days); transporting fixed slides is easier and safer and slides can be kept for more time; but needs additional skills and puts the riskier job (selecting the sample and preparing the slide) in hands of non-laboratory staff. Both methods have been tried in Ethiopia. Sputum collection could be organized in such a way that on identifying the person with symptoms, sputum cups are left (with appropriate instruction) for the person to collect sputum for a fixed day, when the HEAL TB ETHIOPIA MID-TERM EVALUATION 35 CHW would return, collect a third (spot) sample and transport to the health center. This would imply only one additional visit by the CHW, who could combine visits to several households just before the visit to the diagnostic unit. VII. RECOMMENDATIONS 1. Planning and management: USAID should continue support and consider expansion to other regions of the country in future projects.  HEAL-TB should consider a more targeted assistance approach in some existing zones. Rather than continuing a general package of support that is the same in each zone, in the zones that have received support since the beginning of the activity, consider reassessing the needs and tailoring activities specifically for each zone.  HEAL-TB should include mapping in the training and support its use at all levels as an instrument for planning and supervision (managerial levels) and follow-up of patient and contacts (health facilities)  HEAL-TB staff should develop clear and practical input and output indicators to ensure sustainability and measure it before the end of the project period.  HEAL-TB should continue to coordinate and collaborate with government partners as they have been doing since year 2 of the activity.  USAID should consider expanding the current HEAL-TB model to other regions and/or future projects. Priority should be given to areas with higher TB prevalence (estimated on the basis of smear positivity among outpatients with symptoms and community screening). The other criteria of prioritization could be regions with similar health systems structure and with exisiting capacity including the Health Extension Workers program. 2. Case detection and diagnosis: HEAL-TB should test patient-friendly methods to screen suspects in the community without referral, such as sputum collection at the community level, to ensure examination of identified presumptive cases. HEAL_TB should prioritize control of contacts and implementation of IPT in children contacts. HEAL-TB should rationalize (use efficiently) the existing laboratory capacity and workload, taking into account that a small number of slides per week (particularly smear positive) will result in reduction of quality and increase of false diagnosis; and should evaluate the need for new microscopy units versus strengthening of transport of samples to existing units;. HEAL-TB should use laboratory data on the number of persons examined and the positivity rate to measure case detection activities and to select priority groups, and train program staff in this methodology. Calculate the predictive value of positive and negative results based on the average positivity rate in different areas and use it to plan retraining and additional methods such as panel testing when required. USAID should use the number of persons examined by microscopy and positivity rate as the main indicator for case detection in future projects (instead of the CDR). The same should be done for GeneXpert. 36 HEAL TB ETHIOPIA MID-TERM EVALUATION 3. Treatment and drug supply: HEAL-TB should support further decentralization of treatment to the health posts, community health worker (CHW) and treatment supporters; revise the patient treatment forms to include diagnosis (TB and site) and initial microscopy result in PTB, treatment outcome and date; and promote the use of community treatment supporters to improve access to TB treatment. Drug supplies: HEAL-TB should support the national TB program to improve efficiency during planning, quantification and distribution of drug supplies. 4. Infection Control: HEAL-TB should support improvement of IC practices in hospitals and strengthen the existing practices in health centers, with emphasis on ventilation and administrative interventions. The priority should be to implement IC administrative practices for inpatients in hospitals, mainly through minor adaptation of existing infrastructure to separate the air from TB infectious patients from other inpatients (doors, air flow, and ventilation). In MDR wards (TICs) separate patients according to infectiousness. 5. TB/HIV: HEAL-TB should support the staff capacity to monitor and evaluate TB-HIV activities in the districts by regular review of activities and by supervision to health centers and health posts to improve data capturing for TB/HIV activities; and support GOE to improve the capacity of the HIV program to screen HIV infected persons for TB (not a direct responsibility of HEAL-TB); and the capacity of ART clinics to diagnose TB, including the use of GeneXpert. 6. MDR-TB: Rapid expansion of MDR-TB treatment initiating centers and follow up centers has significantly improved access to service. However, more attention has to be given to improve quality of care. This includes ensuring correct patient diagnosis, treatment, program management and continuous monitoring and evaluation of all program activities to provide feedback loop for quality improvements. Attention should be given in maintaining quality of laboratory equipment, supplies and staff training to manage MDR-TB patient complications and side-effects, identifying sites with insufficient capacity or staff knowledge and responding efficiently to needs, and ensuring that the TFCs have sufficient capacity to support the TICs. 7. Community-based TB care: HEAL-TB should: - strengthen the links between the district and health centers, and between health centers and health posts which to improve care delivery; support and coordinate CBTC activities and test evidence￾based high impact interventions like innovative community based approaches and contact tracing; - Test and implement procedures to transport sputum samples (recommended) or fixed slides from the community to the diagnostic units, instead or in addition to referring persons with symptoms. Document the findings and best practices for the conditions of the region. - Increase decentralization of DOT to HPs, and find out alternative methods to improve adherence of presumptive cases or examination of presumptive cases sent from HPs. - Improve support at districts and health centers to enhance the performance of the community based activities. Decentralize the monitoring and evaluation by conducting district based review meeting and catchment area review meeting at health centers to strengthen the implementation of community based activities. HEAL TB ETHIOPIA MID-TERM EVALUATION 37 ANNEX A. STATEMENT OF WORK Mid-Term Performance Evaluation of HEAL-TB. USAID/Ethiopia Updated October 9, 2014 TABLE OF CONTENTS I. PERIOD II. BACKGROUND III. PURPOSE OF THE EVALUATION V. FUNDING SOURCE VI. EVALUATION DESIGN AND METHODOLOGY VI. LIMITATIONS OF EVALUATION DESIGN/METHODOLOGY VII. EVALUATION TEAM SKILLS AND QUALIFICATIONS VIII. LOGISTICS IX. DELIVERABLES & EVALUATION PROCEDURES 1. PERFORMANCE PERIOD The period of performance of the evaluation from the time the consultants are estimated to first begin reviewing materials to when they complete final deliverables is from September 29, 2014 – December 31, 2014. 2. BACKGROUND The USAID funded Help Ethiopia Address the Low TB Performance (HEAL-TB) Activity is a 5 year cooperative agreement being implemented since July 2011 by Management Sciences for Health (MSH) in partnership with the Program for Appropriate Technology in Health (PATH), the All Africa Leprosy, TB, Rehabilitation, Research and Training Centre (ALERT), and the Kenya Association for the Prevention of Tuberculosis and Lung Disease (KAPTLD). HEAL-TB supports a comprehensive package of tuberculosis (TB) interventions to provide quality Directly Observed Treatment - Short Course (DOTS) including: DOTS strengthening and expansion which includes strengthening referral linkages to the community; enhancing TB/HIV collaborative activities; strengthening programmatic management of Drug Resistant TB (PMDT); and strengthening health systems. Assistance supports the Federal Ministry of Health (FMOH) to implement comprehensive TB projects in 21 zones of the Amhara and Oromia regions covering a total population of above 40 million at the end of the second year. Due to the need for emergency treatment for hundreds of MDR patients, support for PMDT has also expanded in both regions. HEAL-TB collaborates with Ethiopian universities and other local institutions and helps the regional health bureaus (RHBs), zonal health departments (ZHDs), woreda (district) health offices (WrHOs), and primary health care units (PHCUs) take ownership of TB, TB/HIV, and multidrug-resistant TB (MDR-TB) program management. Through this assistance, the RHBs, ZHDs, WrHO and PHCUs are expected to have improved and sustainable program management capacity. HEAL-TB focuses on 38 HEAL TB ETHIOPIA MID-TERM EVALUATION increasing case notification and decentralization of DOTS in communities through Health Extension Workers (HEWs) and expanding access to additional health facilities (HF) in rural and urban areas. HEAL-TB is also helping to strengthen Ethiopia’s health system by supporting woreda planning and improved drug supply management and infection control. Context Ethiopia, with a population of 92 million, according to the WHO Global TB report 2013, ranks eighth in the list of high burden countries with an estimated 230,000 incident cases of TB, which is equivalent to incidence of 247 cases per 100,000 persons. According to the same WHO report, there were 210,000 TB cases in Ethiopia in 2013, which is equivalent to 224 cases per 100,000 persons. Case mortality is also estimated at 16,000 deaths (excluding deaths among HIV positive TB cases) which is equivalent to 18 deaths per 100,000 persons. In a recent Ethiopian national population based survey, the prevalence of bacteriological confirmed TB (smear positive and/or culture positive) for those age 15 years and above was 277 per 100,000 persons. Both men and women are equally vulnerable in their young and productive ages of 20-30 years. However, gender disparities in health-seeking behavior contribute to differences in vulnerabilities especially in gender related delay in TB diagnosis, treatment interruption, gender based TB Stigma and Discrimination. Low detection rate and treatment for TB among women have been seen due to delay in accessing care, poor compliance to treatment, and belief in alternate treatment Ethiopia has been implementing the DOTS strategy since the early 1990s, nation-wide coverage of which was achieved in 2002. According to the Ethiopian Fiscal Year (EFY) 2005 (Gregorian calendar 2012/2013) Health Sector Development Plan (HSDP) IV annual performance report, the TB case detection rate was 58.9%, below the detection rate estimated in the 2011 TB prevalence survey (72%) as well as below the target set for the year (82.7%). In the same annual bulletin report of FMoH, a total of 130,614 TB cases (all forms) were reported with a TB case notification rate of 152 per 100,000 population; this performance was below that observed in EFY 2004 (2011/2012), when 145,367 cases (all forms) were reported with a TB case notification rate of 172 per 100,000 population, 12% decline. Out of the 130,614 cases reported in EFY 2005(2012/2013), 33.4% were smear positive pulmonary TB, 34.5% were smear negative pulmonary TB, and 32.1% were extra pulmonary TB. As of 2013, the current national TB treatment success rate (TSR) is estimated to be 91.4% while the cure rate for the year 2012/2013 was 70.6%. According to the 2013 annual TB bulletin of FMoH, in 2011/2012 86% of TB patients know their HIV status and 20% of are estimated to be HIV infected. The HIV co-infection rate was higher than the previous year estimate which could be as a result of the new TB/HIV surveillance sites being located in more of urban sites. The 2013 global report estimated the co-infection rate to be 10%. In 2011/2012, among TB/HIV co-infected patients 76% got co-trimoxazole preventive therapy (CPT) and 51% were put on ART. However the Isoniazid Preventive Therapy (IPT) coverage was only 18%. TB, MDR-TB and TB/HIV still continue to inflict tremendous impact on both morbidity and mortality rates in Ethiopia today. National and international attention is now focused on providing care in both public and private health facilities. The national program has planned to carry out comprehensive TB activities in order to improve the Case Detection Rate and to build the capacity of HEWs to mobilize HEAL TB ETHIOPIA MID-TERM EVALUATION 39 communities for early TB diagnosis and active TB case finding (to screen all TB suspects and start community DOTS service in each Kebele). The Public-Private Mix (PPM-DOTS) is a strategy designed by the country to improve the TB case detection. As of June, 2011, 2,858 hospitals and health centers provide DOTS (treatment) service and out of 15,092 health posts, 3,844 (26%) of them provide treatment follow-up services. Also, 1,941 health facilities provide diagnostic services (one diagnostic center for 43,000 populations). PPM-DOTS started in 2006 and, currently, 317 PPM-DOTS facilities are providing DOTS service. In the area of TB/HIV services, as per the report of FMoH in 2011, about 1,553 health facilities were providing TB/HIV services. HEAL-TB Based on the TB prevalence estimated at the time the activity started, HEAL-TB has initially aimed to increase the Case Detection Rate in the supported zones from the estimated rate of 35.5% to 70% within two years. However, a national survey conducted in 2011 showed the CDR rate was at 72% which was also reflected in the WHO 2012 TB report. However, according to the FMOH report of 2012/13(2005 EFY), the CDR for the nation has declined to 59% and the current regional estimates for Amhara and Oromia are 63% % and 56.8% respectively. Though new cases identified are declining, WHO‘s TB incidence estimate is still high, therefore HEAL-TB continues to maintaining the target of reaching 70% CDR. HEAL-TB supports the national TB program efforts in the two regions of Amhara and Oromia with a focus on the four main technical areas including the TB program cross-cutting areas of Gender as well as Partnership and Coordination.  Technical Area 1: Strengthening and Expansion of DOTS  Technical Area 2: Response to Emergency of MDR-TB  Technical Area 3: TB/HIV Collaboration  Technical Area 4: Health Systems Strengthening The strategies for achieving the project aim include: o Increase community awareness of TB, improve referrals, and improve the quality of service through training of Health Extension Workers; o Improve the organization of TB Case Detection within facilities by strengthening multidisciplinary teams and streamlining the flow of TB suspects and patients; o Provide standard operating procedures, job aids, and posters to all health facilities and improve workers’ skills in TB diagnosis and management; o Improve the documentation and monitoring of TB suspects; o Strengthen TB/HIV collaborative activities by screening all people living with HIV for TB as well as screen all TB patients for HIV and strengthen referral linkages for co-management of HIV positive TB patients o Strengthen MDR-TB early diagnosis, referral and treatment capacity in the two regions through expansion of Programmatic Management of Drug-resistant TB (PMDT) services. o Strengthen laboratory services management; o Improve the drugs and commodities management of the program and health facilities; o Involve all care providers, including private health facilities, and drug outlets. 3. PURPOSE OF THE EVALUATION The overriding purpose of this performance evaluation is to independently assess the overall performance of the project—in terms of achievements and challenges faced during implementation— 40 HEAL TB ETHIOPIA MID-TERM EVALUATION and to identify actions aimed at improving management during the remaining life of the project. The findings are also necessary to inform the design and implementation of similar projects in line with the USAID/Ethiopia Country Development Cooperation Strategy. 4. EVALUATION QUESTIONS Examine implementation approaches to verify that HEAL-TB is using standard and proven strategies for implementation of TB-related activities. o Are the HEAL-TB implementation strategy and core approaches based on evidence (or best practices) that meet international standards and incorporate context specific policies and strategies? o Describe how quality is measured in the key components (diagnostics, patient management, follow-up, data recording and reporting) of HEAL-TB interventions such as TB, TB/HIV, and MDR TB services. Examine monitoring and evaluation implementation to determine if indicators are appropriate and mechanisms are in place to ensure effective program learning. o Is monitoring and evaluation implementation optimized (via PMP or other methods) to assure that the system to generate evidence is in place and key lessons from the project are being documented (both positive and negative and including management and operations at National, Regional, Zonal, and Woreda levels) to inform future similar project designs? o Have HEAL-TB capacity building initiatives improved sustainability of the project’s activities and continued results in terms of institutional strength (National, Regional, Zonal and Woreda health institutions) and human capacity built (health professionals) in managing TB Programs in the country? o To what extent have gender appropriate metrics been incorporated into the HEAL-TB monitoring and evaluation system? Examine the program management to determine if the activities (inputs) of HEAL-TB support the objectives of the HEAL-TB cooperative agreement. o Is management of sub-partners, including the type and levels of supportive supervision and coordination, optimal to ensure that HEAL-TB performance is meeting the objectives of the cooperative agreement? o Are the staff competencies at all levels of the HEAL-TB activity (Technical expertise, administration, trainers, etc.) sufficient in order to meet the objectives of the cooperative agreement? Examine financial management to determine if the USAID investments are being effectively/efficiently utilized to achieve the objectives of the cooperative agreement. o Financial management o What determines the amount of funds to be allocated to each sub-partner? o Describe the financial management support provided to sub-partners. What type of monitoring is in place? Are there areas for improvement? o Is the utilization of funds adequate to ensure full implementation and optimal pipeline? Examine HEAL-TB collaboration and coordination with other organizations to determine the role of other entities such as the Global Fund and other TB implementing partners in strengthening the national and regional TB program management capacity at all levels. How have collaboration, coordination, and planning efforts with GOE and other key stakeholders such as the Global Fund and TB implementing partners (TB CARE, PHSP, PEPFAR programs and others) contributed to HEAL-TB achieving its intended results? HEAL TB ETHIOPIA MID-TERM EVALUATION 41 5. FUNDING SOURCE The funding source will be Tuberculosis Program Development and Learning - USAID/Ethiopia. 6. EVALUATION DESIGN AND METHODOLOGY This performance evaluation will employ mixed methods (qualitative and quantitative methods). a. Methodology The methodology should include, but not be limited to the following:  Document Review  Key Informant Interviews  Focus Group Discussions (FGDs)  Site visits The methodology will be discussed and finalized with the USAID/Ethiopia team before site visits begin b. Evaluation Design This is a non-experimental design that will focus on measuring project results before and after project implementation using project monitoring and survey data. The “before” project data should be drawn from the baseline Survey Report. For some of the indicators whose baseline data is missing, the evaluation team is expected to reconstruct baseline data using a recall method (for instance, by asking individuals or groups to provide informtion on key project result measures before the start of the project) and by referring to secondary sources and administrative data. Gender disaggregation of any data collected and analyzed should take place to determine if impact due to the intervention is taking place differently by gender. Further, gender sensitive indicators should be collected and analyzed as appropriate to accurately reflect gender integration and gender-related activity indicators. c. Data Sources and collection methods Qualitative and quantitative methods are to be used to collect data from selected beneficiaries (using appropriate sampling method). Sample Selection and Size: The evaluation team will prepare a detailed assessment framework including sampling and instruments which will be reviewed and approved by USAID/Ethiopia. However, it is expected that tools and sampling strategies for target groups (partners, service providers and other health facility staff, beneficiaries) will be developed based on the previously stated evaluation questions. d. Data Sources and Collection Methods o HEAL-TB RFA o HEAL-TB Cooperative Agreement o Work plans o Field trip reports o Semi-annual and annual progress reports o Financial reports o Program documents o This will also include documents produced in the course of program implementation by USAID, Management Sciences for Health (MSH) and other entities. o Raw data as need to answer the proposed questions (including facility level data, reporting data) 42 HEAL TB ETHIOPIA MID-TERM EVALUATION Data sources and collection methods will be developed by the team based on the final evaluation questions. e. Data analysis plan Based on the data collection tools designed for the quantitative data, data will be summarized and descriptive analysis will be made using appropriate data analysis. Tabular, graphical and other relevant presentations of results can be used. The evaluation team will employ appropriate analysis tools for qualitative data in order to categorize, rank and rate the responses of the interviewees and discussants. Very insightful or special description of interviewees and discussants will be quoted word by word to corroborate findings from other data sources. The evaluation team will work with USAID/Ethiopia to finalize an analysis plan. 7. LIMITATIONS OF THE EVALUATION DESIGN AND METHODOLOGY The baseline survey may lack some of the indicators that this mid-term evaluation will assess and therefore the before-after analysis may not be possible. Evaluators will reconstruct the baseline values for such indicators using a retrospective approach when possible. The use of both qualitative and quantitative data collection methods and this will strengthen and reinforce the rigor of the data and validity of the findings. However, any data limitations associated with using either method should be clearly documented as part of the final evaluation report. 8. EVALUATION TEAM SKILLS AND QUALIFICATIONS A strong evaluation team will be comprised of three consultants with experience in the field of TB programming including community-based tuberculosis care, programmatic management of drug￾resistant TB, TB/HIV, as well as Health Systems Strengthening as related to TB programming. Team members should have a sound understanding of Ethiopia as a country and country-specific TB issues and challenges. A. Profile of Evaluation Team: Team Lead (1): The team lead will have the overall responsibility for the expected results of the performance evaluation. S/he will be an international consultant with more than 10 years of experience including some work in Africa. A second international consultant will serve as co-team lead and share the responsibilities of the team lead. The Team Leader will be responsible for team performance and for ensuring the timeliness and quality of deliverables. Strong team lead candidates will have led at least two external performance evaluations. Strong writing, evaluation methods, and analytical skills required of both international staff. The consultant will hold conference calls with the other team members and USAID/Ethiopia representatives before and after the visit to Ethiopia in order to develop the evaluation methodology and take the lead in developing the evaluation report. The team lead is expected to present preliminary findings of the evaluation to USAID/Ethiopia and HEAL-TB staff prior to departure from the country. HEAL TB ETHIOPIA MID-TERM EVALUATION 43 The team lead will be supported by two consultants whose skills should complement the evaluation. Technical knowledge in the areas of clinical, laboratory and community tuberculosis treatment support and evaluation methods will be necessary for a comprehensive team. o Experience leading performance evaluations, including development, implementation, and analysis o Experience implementing qualitative and quantitative evaluation methods o Experience in tuberculosis programming, research and evaluation in other countries; preferably as principal investigator (PI) or co-investigator with peer-reviewed publications validating experience o Experience with the issues affecting genders differently in health programs and specifically in Tuberculosis programming o Excellent demonstrated written language skills o Cultural sensitivity and ability to work in a cross-cultural team Local Consultants (2): The local consultants should possess the following skills, though each does not need to possess all listed skills, but all listed skills should be contained within the local consultants. Experience should include: o Experience with tuberculosis programming in Ethiopia o Experience with quantitative and qualitative data collection (developing evaluation methodologies/tools and performing data collection, management, and analysis). o Experience with the specific issues affecting genders differently in health programs and specifically in Tuberculosis programming in Ethiopia o Understanding of the local health system and structures (FMOH, RHBs, etc.) o Fluency in written and spoken Amharic o Strong English language presentation and writing skills The local consultants are required to communicate and work closely with the Evaluation Team lead for the successful completion of all deliverables outlined in this SOW. USAID/Washington TDY (1): The opportunity of involving a senior TB expert from USAID/Washington in the evaluation will be explored depending on availability. The involvement of the USAID/Washington staff person would complement the evaluation team by looking at the project’s approach in addressing USAID policies and priorities as well as global standards. B. Estimated Level of Effort (LOE): An agreed tentative schedule will be drafted and the working team(s) will work with designated experts from HEAL-TB. The actual task distribution will be finalized once the team is established and according to specific expertise of the consultants. A six-day work week will be approved when the consultants are working in country. Weekend travel may be necessary. Below is a list of the specific tasks to be accomplished by the consultant team, with an estimated level of effort and proposed timing for each task. 44 HEAL TB ETHIOPIA MID-TERM EVALUATION Illustrative Activities and Estimated Level of Effort Activity Person(s) Responsible Team Lead Consultant 1 Consultant 2 Total LOE Period of Performance (estimated dates are included) Mission sends background documents to Team Members USAID/Ethiopia HEAL-TB 0 0 0 0 The period of performance is from September 29- December 31, 2014 Review of documents Team Leader & Consultants 2 2 2 6 September 29-30, 2014 Draft evaluation work plan and survey instruments; plan logistics Team Leader & Consultants 2 2 2 6 October 1-2, 2014 Travel to country Team Leader & Consultants 1 0 0 1 October 3, 2014 Team prepares for in-brief and Team Planning Meeting Team Leader & Consultants 5 5 5 15 October 4-9 2014 In-briefing with USAID Team Leader & Consultants 1 1 1 3 October 10, 2014 Organize logistics for field work Logistics Company 0 0 0 0 Fieldwork including travel days Team Leader & Consultants 14 14 14 42 October 11-October 27, 2014 Data analysis Team Leader & Consultants 4 4 4 12 October 28-31, 2014 Write 1st draft of report Team Leader & Consultants 4 4 4 12 November 3-6, 2014 Debriefing with mission staff with draft findings and Team Leader & Consultants 2 2 2 6 November 7, 2014 HEAL TB ETHIOPIA MID-TERM EVALUATION 45 Activity Person(s) Responsible Team Lead Consultant 1 Consultant 2 Total LOE Period of Performance (estimated dates are included) recommendations. Team leader departs country Team Leader & Consultants 1 0 0 1 November 8, 2014 Review and comment on 1st draft report USAID/Ethiopia 0 0 0 0 November 10-17, 2014 Revise and submit 2nd draft of report Team Leader & Consultants 5 2 2 9 November 18-November 25, 2014 Review 2nd draft report and make final comments USAID/Ethiopia 0 0 0 0 November 26-December 5, 2014 Finalize report and submit for final approval Team Leader & Consultants 2 0 0 2 December 8-9, 2014 Approve final report USAID/Ethiopia 0 0 0 0 December 10, 2014 Send final approved report (soft copy) Team Leader 1 1 December 11, 2014 Total LOE 44 36 36 116 46 HEAL TB ETHIOPIA MID-TERM EVALUATION LOGISTICS Two purchase orders (to be written separately), including one for the team lead and one for logistics will be used to hire a company to manage logistics as well as to hire two local consultants. DELIVERABLES & EVALUATION PROCEDURES Based on the above stated purpose, objectives and key tasks, the evaluation team will participate in the following activities or provide the stated deliverables. 1. No more than 8 days after receiving background documents from USAID/Ethiopia, the evaluation team will submit a detailed evaluation framework/work plan including sample size and instruments which will be reviewed and approved by USAID/Ethiopia. 2. Before leaving the country provide a presentation including preliminary findings and recommendations for USAID/Ethiopia staff and other relevant stakeholders 3. Within 10 working days of leaving the country provide a Draft Evaluation Report, and 4. After providing USAID/Ethiopia the opportunity to review Draft Reports provide within 5 working days the Final Evaluation Report of not more than 35 pages that shall include findings, conclusions, and recommendations. a. Team Planning Meeting (TPM): The assignment will commence with Team Planning Meeting (TPM) to allow USAID/Ethiopia to clarify the purpose, expectations, and agenda of the assignment. In addition, the team will clarify roles and responsibilities; review and develop final survey questions; review and finalize the assignment timeline and share with USAID/Ethiopia; develop data collection techniques, instruments, tools and guidelines; review and clarify any logistical and administrative procedures for the assignment; and agree on procedures for resolving differences of opinion. No more than eight working days after USAID/Ethiopia submits background documents to the evaluation team, the evaluation team will submit a detailed work plan. The Evaluation work plan must include detailed design, approach/methodology, sample size, survey protocol, data collection tools, and plans for analysis and dissemination of findings. The Team Leader will submit the evaluation work plan to USAID/Ethiopia. USAID/Ethiopia will then review the proposed work plan/methodology and data collection tools and submit comments to the Team Leader prior to field work. The evaluation team will revise the work plan/methodology and data collection tools and send the final version to USAID/Ethiopia. The evaluation work plan must be finalized and approved prior to the initiation of the interviews and site visits. b. Interim briefings including status reports The Team Leader will provide weekly status reports on the evaluation plan implementation to USAID/Ethiopia. These should be 1-5 pages in length describing progress in the evaluation, and may be in e-mail form. Should there be any strategic or methodological changes to the evaluation work plan, the Evaluation Team Lead must propose these changes in writing USAID/Ethiopia and will be subject to further approval. c. PowerPoint Presentation and short debriefing report A presentation (in MS PowerPoint) and short report (not to exceed 10 pages) will be developed to be used during debriefing to USAID/Ethiopia staff focusing on the preliminary findings and HEAL TB ETHIOPIA MID-TERM EVALUATION 47 recommendations that address the evaluation objectives and associated questions. A template will be provided to the team lead. At least two working days before the presentation, the evaluation team must provide a draft slide deck (not to exceed 20 slides), for the presentation which will be approximately 1.5 hours in length, including discussion. An additional presentation may also be scheduled to include other evaluation stakeholders. d. Draft report The draft report in English must be submitted in English and no longer than 35 pages, excluding coversheets and appendix. The report shall follow the general format indicated below: (i) Coversheet indicating type of evaluation (ii) Table of Contents (iii) Acknowledgments (iv) Acronyms (v) Executive summary (vi) Introduction (vii) Background (viii) Scope and Methodology (ix) Strengths and Limitations of the Evaluation (x) Findings (xi) Conclusions (xii) Lessons learned (xiii) Recommendations (xiv) References (xv) Appendix (includes, but not limited to, SOW, data collection instruments, sources identified, and list of people contacted or interviewed, statements of differences regarding significant unresolved differences of opinion by funders, implementer, and/or members of the evaluation team, if any). The final report is expected to follow current USAID styles (fonts, size) as well as branding and marking. Further is available here: http://www.usaid.gov/branding. It is highly recommended that the Evaluation Team review and follow guidelines in the How-To Note on preparing Evaluation Draft Reports found here and the Evaluation Report branding template found here. e. Raw Data The evaluation team will provide raw data as well as any analysis or coding to USAID/Ethiopia. f. Final report The final report will address the Mission’s comments from the 1st and 2nd draft report. The Team Leader will submit the final report to USAID/Ethiopia within 2 working days after the evaluation team receives consolidated comments from USAID/Ethiopia. The team lead will provide the edited and formatted final document approximately 5 days after USAID/Ethiopia provides final approval of the content. Procurement sensitive information or other information as determined by USAID/Ethiopia PRM, DOC, and HAPN will be removed from the final report and incorporated into an internal USAID Memo. The remaining report will be submitted to the USAID Development Experience Clearinghouse (DEC) (http://dec.usaid.gov) for use as a public document as per Automated Directives System, Chapter 540. The team lead shall submit the edited and formatted final report electronically via CD-rom or email. 48 HEAL TB ETHIOPIA MID-TERM EVALUATION ANNEX B. TABLES AND FIGURES FIGURE 1. PROJECT GEOGRAPHY AND SCOPE Management Sciences for Health 5 PROJECT SCOPE Coverage: • Phase I: 10 zones, 691 health facilities, and 3 380 health posts • Phase II: 11 zones, 610 health facilities, and 3 320 health posts • Total phases I and II zones: 21 zones, 1 301 health facilities, and 7,000 health posts • Target Population: 40.3 million • Phase III: preparation completed for expansion to 7 additional zones • Target Population including Phase III: 50.8 million HEAL TB ETHIOPIA MID-TERM EVALUATION 49 Table 1. Profile of HEAL-TB activity Sites. S/No Phase of Implementation Zone Population Woredas Hospitals Health Centers Health Posts CDR all forms (%) Cure rate TSR Oct 2010- Sept 2011 Oct 2012- Sept 2013 Oct 2010- Sept 2011 Oct 2012- Sept 2013 Oct 2010- Sept 2011 Oct 2012-Sept 2013 1 Phase I North Gondar 3,018,326 24 3 133 522 78.2 80.5 32.6 73.4 77.5 86.4 2 Phase I South Gondar 2,278,555 15 1 93 393 43 50.5 42 75.4 98.9 89.4 3 Phase I East Gojam 2,397,876 20 2 101 402 56.2 55.6 70 83.7 91.89 93.1 4 Phase I North Wollo 1,535,744 13 2 61 248 32.3 41.7 19.39 74.6 87.36 86.4 5 Phase I South Wollo 2,797,445 24 4 134 502 54 53.9 57.64 84.3 95.69 91.2 6 Phase II North Showa 2044127 27 4 90 408 NA 49.2 NA 64.3 NA 78.5 7 Phase II Awi 1,098,652 11 1 41 185 NA 41.6 NA 74.66 NA 80.5 8 Phase II West Gojam 2618132 19 2 112 356 NA 56.4 NA 64.6 NA 78.6 9 Phase II Oromia 510295 7 0 26 104 NA 62.8 NA 86.34 NA 87.7 10 Phase II Waghimra 464575 7 1 29 125 NA 64.9 NA 28.36 NA 64.7 Amhara 18,763,727 167 20 820 3,245 11 Phase I Arsi 3,106,289 26 3 100 506 64.5 58.2 46.65 91.6 88.2 94.8 12 Phase I East Wollega 1,442,416 22 2 61 316 65.4 66.6 75.84 81.2 87.72 92.5 13 Phase I Jimma 3,083,421 18 3 99 545 44.4 41.8 40.4 80.8 91.6 89.9 14 Phase I West Harerghe 2,210,879 19 2 79 448 57 50.2 78.6 88.9 88.23 97.1 15 Phase I East Harerghe 3,215,513 16 3 110 550 56.7 61.1 87.99 91.7 46.62 97.7 16 Phase II Borena 1,137,081 14 3 67 259 NA 71.2 NA 78.47 NA 85.9 17 Phase II Guji 1,642,810 15 2 70 352 NA 72.3 NA 73.47 NA 76.1 18 Phase II West Wollega 1,610,609 21 3 63 518 NA 54.5 NA 81.69 NA 82.6 19 Phase II West Showa 2,315,782 19 3 88 561 NA 51.4 NA 78.96 NA 84.2 20 Phase II North Showa 1,388,617 14 2 61 269 NA 46.6 NA 72.17 NA 76.6 50 HEAL TB ETHIOPIA MID-TERM EVALUATION 21 Phase II Horu Gudru 705,507 10 1 50 195 NA 57 NA 56.63 NA 71.7 Oromia Total 21,858,924 194 27 848 4519 22 Phase III Illu Aba Bora 1,503,733 24 2 69 501 23 Phase III Kelem Welega 940,059 11 1 48 267 24 Phase III South West Shewa 1,079,814 12 2 55 268 25 Phase III West Arsi 2,384,180 14 3 83 343 26 Phase III Bale 1,661,816 20 4 86 369 27 Phase III East Shoa 1,836,115 15 2 70 299 28 Phase III Finfine Zuria 829,522 14 1 41 156 Oromia Total 32,094,163 304 42 1,300 6,722 Total Project Zones 50,857,890 471 62 2,120 9,967 HEAL TB ETHIOPIA MID-TERM EVALUATION 51 TABLE 2. Profile of HEAL-TB’s project-supported zones S/N Zone Population Woredas Hospitals Health Centers Diagnostic Health Centers Health Posts Phase I: HEAL-TB Supported Zones (Started on July 15, 2011) Amhara Region 1 North Gondar 3,018,326 24 3 133 110 522 2 South Gondar 2,278,555 15 1 91 82 393 3 East Gojam 2,397,876 20 2 100 79 402 4 North Wollo 1,535,744 13 2 61 40 248 5 South Wollo 2,797,445 24 4 132 86 502 Region Total 12,027,946 96 12 517 397 2,067 Oromia Region 6 Arsi 3,106,289 26 3 98 70 506 7 East Wollega 1,442,416 22 2 60 55 316 8 Jimma 3,083,421 18 4 112 86 545 9 West Harerghe 2,210,879 19 2 76 56 448 10 East Harerghe 3,215,513 16 4 115 79 550 Region Total 13,058,518 101 15 461 346 1,820 Total Phase I Total Phase I 25,086,464 197 27 980 743 3,887 Phase II: Expansion Zones (Started on August 2013) Amhara Region 11 North Showa 2,044,127 27 4 88 68 408 12 Awi 1,098,652 11 0 41 37 185 13 West Gojam 2,618,132 19 2 110 91 356 14 Oromia 510,295 7 0 27 27 104 15 Waghimra 464,575 7 1 29 12 125 Region Total 6,735,781 71 7 295 235 1,178 Oromia Region 16 Borena 1,137,081 14 3 67 25 259 17 Guji 1,642,810 15 2 71 69 352 18 West Wollega 1,610,609 21 3 63 48 518 19 West Showa 2,315,782 19 3 88 54 561 20 North Showa 1,388,617 14 2 61 41 269 21 Horo Guduru 705,507 10 1 50 50 195 Region Total 8,800,406 93 14 400 287 2,154 Total Phase II 15,536,187 164 22 695 522 3,332 Phase III: Expansion Zones (Started on August 2014) Oromia Region 52 HEAL TB ETHIOPIA MID-TERM EVALUATION Source: MSH HEAL-TB Report, Year 3 Quarter 4, July-September 2014 22 Finfine Zuria S/zone 829,522 14 1 41 24 156 23 East Shoa 1,836,115 15 3 70 59 299 24 South West Shoa 1,079,814 12 2 55 32 268 25 West Arsi 2,384,180 14 3 83 57 343 26 Illu Aba Bora 1,503,733 24 2 67 40 501 27 Kelem Wollega 940,959 11 1 47 25 267 28 Bale 1,661,816 20 4 86 48 369 Region total 10,236,13999 110 16 449 285 2,203 Total phase III 10,236,139 110 16 449 285 2,203 Amhara (Phase I, II) 18,763,727 167 19 812 632 3,245 Oromia (Phase I, II & III) 32,095,063 304 45 1,310 918 6,177 Project Total 50,858,790 471 64 2,122 1,550 9, 442 HEAL TB ETHIOPIA MID-TERM EVALUATION ANNEX C. SCHEDULE OF FIELD VISITS Region Zone Institution District Main areas of evaluation Oromia Travel to Adama and RHB Adama town Geda MDR TB TIC MDR TB TIC Regional reference laboratory Interview on the lab support TB culture, specimen referral and EQA Arsi zone Health Department Asela Hospital Interview on the support of the project DOT practice and AFB EQA Dhera HC Dodota district DOT practice and EQA Itaya HC Hetosa district DOT practice, EQA and woreda support Two health posts Dodota district DOT practice and EQA Two health posts Hetosa district DOT practice, EQA and woreda support Travel to Addis/Travel to Shashamane North Shoa zone Health department Fiche hospital Team one: North Shoa Shashemene hospital TIC Arsi Negele HC TFC Team two : Shashemane hospital and Health Centre District DOT practice, EQA and woreda support Two health posts District DOT practice, EQA and woreda support Addis Ababa Addis Ababa EPHI MSH EQA, LED, Xpert and culture support Amhara Baher Dar Town Flight to Bahir dar RHB RPHL Interview on the support TB culture, specimen referral and EQA East Gojam Health Department Debremarkos Hospital Interview on the support of the project and TIC Lumame HC Awabal district DOTS practice, EQA and woreda support Yegodana HP Awabal district Awi zone Health Department Wagusa district Interview on the support of the project Tilili HC Jata HP DOTS practice Meet with HEAL-TB team Flight to Addis from Bahir Dar 54 HEAL TB ETHIOPIA MID-TERM EVALUATION ANNEX D. MAIN PERSONS INTERVIEWED Organization/Place Name Position National Level USAID ETHIOPIA / Addis Ababa Advisor Josephine E.V. Francisco Program Officer, M&E Advisor, HAPN Yared Kebede Haile Sr. Infectious Disease Advisor, HAPN Joshua Karnes Family Health & Infectious Diseases Jeanne Rideout Deputy HAPN Office Chief Muluken Chanie Program Officer Endalkachew Meiese Sr. HIV/AIDS Advisor / Treatment Advisor MSH / Addis Ababa Muluken Melese HEAL-TB activity Director Degu Jerene HEAL-TB Deputy Project Director Habtamu Wudneh HEAL-TB Sr. Manager Project Finance National TB Program, FMOH Andargachew Kumsa NTP Manager Birru Shigut NTP Technical Advisor Oromia Region Regional Health Bureau/Addis Ababa Shallo Daba Head of Oromia Regional Health Bureau Adama Hospital Ahti Alemazelo Vice Clinical Director Desalesu Aremis Head, Laboratory Department District Health Office Hetosa District Megena Ursessa Head of District Ziyad Ridwan TB Focal Point Heteya Health Center Lem Negeso Arsi Zone Health Office Haji Abdela Egu Deputy Head Olmer Kasim Medical Lab. Technician Amhara Region Regional Health Bureau Mr. Mulusew Lijalem CRCP owner Mr. Wondimu G/Kiros TB expert HEAL TB ETHIOPIA MID-TERM EVALUATION 55 Regional TB laboratory Mr. Wondimu Gebeyehu Head Regional Public Health Laboratory Mr. Yacob G/Egzaber Diagnostic core process Mr. Mulu Melese Laboratory capacity building core process East Gojam ZHD Mr. Getaneh Tesema Head zone health department Mr. Tadele Mebrat TB-HIV focal person Mr. Melkam Getnet TB-HIV focal person Debre Markos Referral Hospital Mr. Tariku Belachew CEO of the hospital Dr. Selam Kinfe MDR Focal person Tebay Godena HP Mrs. Tsigereda Abeba Health Extension worker Zinde Yedeger HP Mrs. Emebet Hibstu Health Extension worker Gwanguwa Shikundad district Mr. Abayneh Nigat Head Tiliti HC Mr. Adugna Mese TB Focal person Mr. Adugna Mese TB Focal person Mr. Zemenu Tafere HC head Sr. Aster Girmaye Emergency OPD Mr. Belachew Bese TB clinic Mr. Alelign Fente Mr. Melaku Yilak Mrs. Tiru Meseret Awi ZHD Sr. Tsehaynesh Yirga Delegated ZHD head Mr. Habtamu Bitewu TB expert PFSA Mr. Abdisa Mengesha PFSA Branch Manager 56 HEAL TB ETHIOPIA MID-TERM EVALUATION ANNEX E. INSTITUTIONS VISITED Region Zone District Health facility Oromia RHB Adama RPHL Geda MDR TIC Arsi zone Asela Hospital Dodota district Dhera HC Hetosa district Itaya HC Sero HP Shashemene hospital MDR TIC West Arsi zone Arsi Negele HC Amhara RHB RPHL East Gojam ZHD Debre Markos Hospital Awbal Tebay Godena HP Zinde Yedeger HP Gwanguwa Shikundad East Gojam ZHD Tilili HC Awi ZHD Jibayitan HP PFSA HEAL TB ETHIOPIA MID-TERM EVALUATION 57 ANNEX F. DATA COLLECTION INSTRUMENTS QUESTIONAIRES I. For staff of HEAL-TB partners  Name, position, organization  Which are the priorities in TB in the current project (in order of importance)? - Increase the national capacity (training, lab equipment, community involvement) - Improve quality of care (diagnosis, treatment, TB HIV, PMDT, PPM) - Reduce TB transmission, morbidity and mortality (screening sources of infection, IC)  Have those priorities changed in time from 2011 to date? How?  How are the results measured (indicators, methodology)  Are the outcomes of the project satisfactory? On track?  Which have been the main achievements?  Which are the major challenges?  Was the TA well accepted by the regions?  Has the NTP used the experiences of the project to expand coverage?  Did other international partners participate? Was their strategy or priorities different?  Which should be the key priorities for the rest of the project? For future projects? II. For national TB program (NTP) and TB laboratory staff at all levels  Name, position, institution  What is the role of HEAL TB? Is it useful? How?  Are the plans discussed with the NTP at different levels? How?  How are the results measured? Which are the main indicators?  Has the TA made an impact in NTP of NGO capacity? How?  Has the NTP benefited by the experience in supported areas to revise technical or operational guidelines, adopt indicators for case detection and ACSM?  What are the unmet NTP priorities for future support? III. For zonal health department  Name, position, institution  What are the key activities or priorities of zone TB program?  Are the work plans discussed and aligned with zone health department? How?  How was the implementation of HEAL TB project initiated in the zone?  How are activities coordinated with HEAL TB?  How are the results measured? Which are the main indicators?  What activities does HEAL TB support in your zone? Is it useful? How?  Capacity building: training  Decentralization of DOTS: HCs available, provide DOTS  Health posts available, HPs engaged in DOTS  Decentralization of diagnostic units: available, expanded by HEAL TB  Logistic and supply management. Mechanisms available.  Performance measures: reports and tools available, frequency of monitoring  Quality assurance: what mechanisms are available? No of diagnostic units, facilities implementing EQA, feedback received and frequency, discordance rate. (FP, FN)  Mentoring: how is it conducted? Who is engaged?  Prevention and control of drug resistance: Trained HW, TICs, TFCs  Number of presumptive cases registered, tested, how are samples transported 58 HEAL TB ETHIOPIA MID-TERM EVALUATION  MDR cases detected, cases put on treatment,  What is the role of HEAL TB in strengthening TB-HIV collaborative activities?  Training on TB HIV: number trained  Screening TB cases for HIV: TB cases, No tested for HIV, No of HIV positive, ART  Screening of HIV + for active TB, IPT, CPT  What joint mechanisms are available to monitor project implementation?  Supervision: frequency, team, feedback  Review meetings and frequency: catchment area, districts, province  Reports: quarterly and feedbacks  In your opinion, has the support from HEAL TB improved the capacity for TB care in your zone? How?  Has the zone benefited by the experience in supported areas?  What are the key challenges for the zone TB prevention and control activities?  What do you feel are the unmet zone priorities for future support?  In your opinion, does HEAL TB support the priority TB control activities?  Is HEAL TB viewed as a supporter of the program, or parallel to it?  What, if anything, would you modify about HEAL TB?  In your opinion, after HEAL TB ends will the zone be able to maintain the same level of support for TB activities?  What mechanisms are available to ensure sustainability? V. For laboratory staff 1. Laboratory Services Provided 1. I Types of TB diagnostic tests  AFB Microscopy Service; GeneXpert (MTB/RIF Assay), FNAC  Mycobacterial culture. If yes, what methods are used?  Drug Susceptibility Testing (DST). Which method are used ? 1. 2 Is there monitoring of the number of persons examined for diagnosis (or number of slides for diagnosis) by the laboratory network, and positivity rate? 2. Quality Assurance 2.1. Does your lab perform EQA for other microscopy centers (EQA center)? If yes, which labs d does the laboratory control?  On-site visits: Yes ________ No ____. If yes, when was the last visit made? 2.3. Is your lab participating in EQA for AFB Smear Microscopy?  If Yes what is you concordance rate in last quarter blind rechecking? _________(%) 2.4. Are records kept? Yes _______ No ____________ (Check) 3. Records (check registers, request forms, log sheets) 4. What methods are used to confirm MDR suspects? Culture, LPA, GeneXpert  How many new cases have been confirmed? in 2011, 2012, 2013  Does the laboratory report MDR TB results immediately? If yes, specify how. To whom? IV. For MDR-TB service staff at treatment initiation center (TIC) level 1. Human resources HEAL TB ETHIOPIA MID-TERM EVALUATION 59  Number of HCWs working in MDR TB Center  Number trained on PMDT; TBL and TB/HIV; IC  Presence of guidelines, SOP, job aids. 2. TB Infection Control  IP/TB IC Trained HCW in service outlet  Functionality of Facility IP/TB IC committee  written down report of IC risk assessment and developed Plan for the facility  infrastructure improvement (renovations)  What administrative measures are being implemented during visit?  utilization of N-95 and surgical mask in the centers: 3. Coordination and linkage with the TB control program  Do you have MDR TB Panel Team? Is it functional? What is the frequency of meetings? Is there any documented minutes of the panel team?  Do you conduct mentoring and supervisory visits to TFCs? Frequency  Does the center receive technical support from RHB or FMOH or partners?  How do you utilize the funds sent from the TB control program? .  Briefly explain the current status of Referral linkage with TFC and other TICs; RL and Xpert sites; RHB 4. Case finding  TB DOT clinic: How do you practice Tb contact tracing? How do you practice contact tracing for confirmed DRTB patient? How do you access to DST service for presumptive MDR-TB case? Are there TB job aids and current TB guidelines?  TB UNIT register: Number of new TB patients in the last 3 years; number of smear positive by month six; DST performed; number confirmed with DRTB. Number of previously treated TB patients in last 3 years; DST performed; number confirmed with MDR-TB 5. Enrolment to service Enrolment: Number of DR TB patients initiated on SLD from the beginning; number of under five children enrolled in the period. Adherence preparation for the patient and treatment supporter:  How do you prepare the MDR-TB patient? Clinical assessment; baseline lab; psychosocial; economic needs  MDR-TB treatment supporter: When do identify MDR-TB supporter? Do you have criteria? How do you train? How do you monitor their continuous engagement?  DOT Practice: With what frequency do you administer SLD? During the three phases, who is observing treatment? Who and how observes inpatients? Any DOT challenges? 6. Laboratory support and pharmacy  Do you have additional clinical services? Which ones?  Sample transport arrangement: Does the center linked to other center for Culture and DST services? Linked to? Frequency of transport? Do you have log book to record and track this activity? Number of samples sent in past 3 quarters? Average TAT? Any challenge? Is the lab the member of panel team, CAM and mentoring support? 7. DSM  No of Personnel trained on PMDT.  How and from where the center receive SDLS? Where are SDLs stored?  Storage situation (Space, fridge, shelf, record system, RRF use)  Any Supply interruptions for SLD 60 HEAL TB ETHIOPIA MID-TERM EVALUATION  Frequency and mechanism of dispensing to MDRTB ward; TFC  Do you have stock of ancillary drugs and/ or opportunistic infections drugs 8. RR review and verification  Treatment card (sample randomly). Completeness, status, gap identified  Treatment register: Completeness, status, gap identified GUIDE FOR OBSERVATION OF HEALTH FACILITIES 1. Meet with head of institution or TB responsible officer: introduction of team and objectives 2. Basic information: name and type of facility (hospital, Health Center, post) - Population coverage, area - Daily outpatient load (any reason). % of children, % and number of adult males and females - TB services available (MD, microscopy, X-ray, treatment) 3. Observation - Is there screening of adults with cough? How is it organized? Is there a register? Are the information messages for general patients regarding cough in the waiting areas? - What is the route of a suspect (persons with respiratory symptoms or signs) - Who requests sputum microscopy? Where does the patient collect sputum? Infection control in sputum collection area and in patients waiting areas, exposure of other patients, ventilation. - Are there sufficient sputum containers? Stock outs? How are they obtained? - At the laboratory: staffing, equipment and environment. Method of microscopy (light, LED) - Lab register and quarterly reports: workload, average number of persons examined for diagnosis per day, positivity of persons examined. Where there changes in time? - Check microscopy results for quality of registration and consistency (quantification, % of scanty results) and for number of samples for diagnosis and for control. - Is the number of persons or slides for diagnosis counted / reported? To whom? Frequency. Is that information used to measure detection of suspects? Is it recorded? Is the data shown in wall graphs or tables? Are there changes in time (trends, data observation and staff interview) - Are technical guidelines available? - Is EQA done routinely? Does the lab receive reports on the results? Are they available? What is the proportion of false positive / false negative? Are there corrective actions (direct supervision, retraining)? By whom? - Are there sufficient reagents? Were there stock outs? How long, how often? How are the reagent needs estimated? - Is the microscope functional? Is anybody trained in microscope maintenance? - Does the lab collect samples for culture / DST? Where are they processed? How are they transported? What is the response time? - Actions for smear positive patients and for smear negative persons. - Treatment procedures. Check TB register and a sample of cards. How many patients diagnosed per quarter? - Who indicates treatment? How is the patient informed, trained. How is the decision of place and method done (facility based, CHW, supporter, type?) HEAL TB ETHIOPIA MID-TERM EVALUATION 61 - Proportion of S+PTB, S-PTB, EP in a sample of patients and from reports. Proportion of new and retreatment. - Is DOT routine? Treatment interruptions (check cards), actions to retrieve interruptors and defaulters. - Treatment outcomes in new PTB, from register and from quarterly reports (cured, completed, defaulted, failed, died, transferred out). Have these changed in the last 3 years? What can be attributed to HELP TB support? - Is the facility using blister packs? FDC? How are the needs estimated? Check dugs stocks - environment, validity. How are the drugs distributed to HP, to treatment supporters, to patients? - Are there sufficient FLDs? Were there stock outs? When, how often, how was the problem solved? - Are contacts followed? How? - Are patients tested for HIV? Procedures for HIV positive. - How is treatment organized (decision by staff, with patient, with treatment supported) How is follow up done? Check patient cards, patient register. - What were the treatment outcomes? Check reports, register. Are they compatible with reported success, cure rate, default rate? - How are drugs presented (patent kits, blisters, other). - How are probable MDR patients identified? What are the procedures? V. For other national and international partners  Name, position  How do you see HEAL TB support to TB control in country activities?  Is your organization collaborating directly with HEAL TB? How?  Which do you see as priorities for USAID country support to TB?  Which do you see as HEAL TB strengths and weaknesses?  How would you modify USAID technical assistance to be more effective in improving TB quality of care? In reducing the TB burden?  How would you modify WHO and STBP technical guidelines and assistance to be more effective in improving TB quality of care? In reducing the TB burden? 62 HEAL TB ETHIOPIA MID-TERM EVALUATION ANNEX G. REFERENCES 1. Help Ethiopia Address the Low TB Performance request for application. 03 Nov. 2010. USAID/Ethiopia Project 2. Cooperative Agreement No-663-A-11-00011. Help Ethiopia Address the Low TB Performance (HEALTB) Project 15 July 2011 3. Help Ethiopia Address the Low TB Performance (HEAL TB) Project Cooperative Agreement No. AID￾663-A-11-00011. Annual Work Plan (July 15, 2011-July 14, 2012) 4. Help Ethiopia Address the Low TB Performance (HEAL TB) Project Cooperative Agreement No. AID￾663-A-11-00011. Annual Work Plan (October 1, 2012-September 30, 2013) 5. Help Ethiopia Address the Low TB Performance (HEAL TB) Project Cooperative Agreement No. AID￾663-A-11-00011. Annual Work Plan (October 1, 2013 - September 30, 2014) 6. MSH/HEAL TB Annual Project Report July 15, 2011 – July 14, 2012 7. Second Quarter Report 2012 October 15, 2011- January 15, 2012 8. Management Sciences for Health (MSH) HEAL TB Project: Third Quarter Report 2012 9. Management Sciences for Health (MSH) HEAL TB Project Progress Report for July 15, 2012- Sept 30, 2012 10. Management Sciences for Health HEAL TB Project Progress Report for October 1, 2012–September 30, 2013 11. Management Sciences for Health HEAL TB Project Progress Report October 1, 2012 to December 31, 2012 12. Management Sciences for Health HEAL TB Project Progress Report January 1, 2013 - March 31, 2013 13. Management Sciences for Health HEAL TB Project Progress Report April 1-June 30, 2013 14. Management Sciences for Health HEAL TB Project Progress Report October 1, 2013 – December 31, 2013 15. Management Sciences for Health HEAL TB Project Progress. January 1, 2014-March 31, 2014 16. National Population Based Ethiopian Tuberculosis Prevalence Survey. July 2011 17. Epidemiological review mission report in preparation of National TB program review. 29th July to 9th August 2013 18. Report of a Comprehensive Joint External Mid–Term Review of the Ethiopia National TB Control Programme. 20-29 AUGUST 2013 19. Revised Strategic Plan Tuberculosis, TB/HIV, MDRTB and Leprosy prevention and Control 2006-2013 EC (2013/14 – 2020 20. Guidelines for Clinical and Programmatic Management of TB, Leprosy and TB/HIV in Ethiopia. Fifth Ed 21. WHO Global TB Report. WHO, 2014