January 2013 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Dr. Carolyn Mohan, Dr. Anna Cristina Carvalho, Dr. Silvia Kelbert, Mr. Valdir Pinto and Ms. Telva Barros through the GH Tech Bridge II Project. EVALUATION USAID/Brazil: Performance Evaluation of the Tuberculosis Portfolio, 2001-2012 EVALUATION USAID/Brazil: Performance Evaluation of the Tuberculosis Portfolio, 2001-2012 EXAMINING THE LONG-TERM OUTCOMES AND SUSTAINABILITY OF TB ACTIVITIES RECEIVING USAID SUPPORT FROM 2001 JANUARY 2013 Global Health Technical Assistance Bridge II Project (GH Tech) USAID Contract No. AID-OAA-C-12- 00027 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. This document (Report No. 12-02-016) is available in printed or online versions. Online documents can be located in the GH Tech website at www.ghtechproject.com. Documents are also made available through the Development Experience Clearinghouse (http://dec.usaid.gov). Additional information can be obtained from: GH Tech Bridge II Project 1725 Eye Street NW, Suite 300 Washington, DC 20006 Tel: (202) 349-3900 Fax: (202) 349-3915 www.ghtechproject.com This document was submitted by Development and Training Services, Inc., with CAMRIS International to the United States Agency for International Development under USAID Contract No. AID-OAA-C-12- 00027. USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 i ACKNOWLEDGMENTS The Evaluation Team would like to thank the National Tuberculosis Control Program of the Brazilian Ministry of Health; the personnel of the State TB Coordination Offices in Minas Gerais, Parana, Pernambuco, Rio de Janeiro, Sao Paulo, Santa Catarina, Rio Grande do Sul, and the Federal District; and the Municipal TB Coordination Offices of the numerous municipalities visited during this evaluation. We greatly appreciate your meeting with the Evaluation team and generous sharing of information in person, via email, Skype and phone. The Team also extends thanks to all of the implementing partners for their collegiality and transparency. Finally, special thanks to the USAID/Brazil Health Team, especially Ms. Nena Lentini, for tireless support of this process and product. ii USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 iii CONTENTS ACRONYMS ................................................................................................................................................. vii EXECUTIVE SUMMARY ................................................................................................................................. xi Lessons Learned......................................................................................................................................xiii Recommendations..................................................................................................................................xiii I. INTRODUCTION.....................................................................................................................................1 II. PROJECT BACKGROUND........................................................................................................................ 3 TB Control in Brazil, 2001 to 2012 ............................................................................................................ 3 The USAID/Brazil TB Program................................................................................................................... 9 III. EVALUATION OBJECTIVE, METHODS & LIMITATIONS..................................................................... 13 Objective .................................................................................................................................................13 Methods..................................................................................................................................................13 Limitations...............................................................................................................................................15 IV. FINDINGS, CONCLUSIONS & RECOMMENDATIONS........................................................................ 17 Overall Findings.......................................................................................................................................17 Illustrative Findings by Selected States................................................................................................... 28 Findings by Activity .................................................................................................................................35 Conclusions and Lessons Learned........................................................................................................... 48 Recommendations..................................................................................................................................49 ANNEXES Annex A. Scope of Work ............................................................................................................................. 53 Annex B. Persons Contacted....................................................................................................................... 65 Annex C. References...................................................................................................................................69 Annex D: Questionnaires developed .......................................................................................................... 77 Annex E: Illustrative Evaluation Questions................................................................................................. 89 Annex F: Implementing Partner’s Project Effectiveness............................................................................. 93 Annex G: The Unified Health System (SUS) and the National TB Control Program..................................135 Annex H: Summarized Hierarchical Organogram of CRPHF and FIOCRUZ ...............................................139 Annex I: Overall TB Drug Distribution Flow .............................................................................................. 141 Annex J: Disclosure of any Conflicts of Interest........................................................................................ 143 iv USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 FIGURES Figure 1: TB Incidence Rate in Brazil by State, 2011..................................................................................... 6 Figure 2: Tuberculosis incidence rate, mortality rate and treatment success percentage for smear‐ positive cases in Brazil, 2001 – 2010 (adapted)............................................................................................ 7 Figure 3: TB incidence rate and % of TB/HIV Co‐infection in Brazil by State, 2011...................................... 8 Figure 4: Location of USAID/Brazil‐supported TB Projects 2001‐2011......................................................... 9 Figure 5: USAID/Brazil Funding for TB, 2001‐2011 ..................................................................................... 10 Figure 6: USAID/Brazil TB Funding by Implementing Partners, 2001‐2011................................................ 10 Figure 7: Triangulation Process................................................................................................................... 15 Figure 8: Number of people trained in DOTS using USG funding, 2007‐2011............................................ 18 Figure 9: Number of people trained in DOTS using USAID funding, disaggregated by sex, 2007 ..............19 Figure 10: Percentage of USG supported lab performing TB microscopy with over 95% correct microscopy results, 2007‐2008................................................................................................................... 19 Figure 11: Case notification rate in new SS+ pulmonary TB cases per 100,000 pop in USG supported areas, 2006‐2011. .......................................................................................................................................20 Figure 12: Percentage of All Registered TB Patients Who Are Tested for HIV Through USG Supported Programs, 2006‐2011..................................................................................................................................21 Figure 13: Treatment Success Rate in USAID‐assisted DOTS Plus Programs to Treat MDR TB Patients, 2010‐2011 ...................................................................................................................................................21 Figure 14: Percentage of National TB Program Funding by Source, 2003‐2012 ........................................ 24 Figure 15: National TB Program Funding by Activity in 2012. .................................................................... 25 Figure 16: National TB Program Funding 2000‐2012.................................................................................. 25 Figure 17: Percentage of TB Patients on DOT in Carapicuiba and Guarulhos, 2006‐2011 .........................32 Figure 18: Percentage of Smear Microscopy, MTB Culture, and HIV Testing Among HIV Patients, Itaim Paulista, MSP 2006‐2009 ............................................................................................................................ 32 Figure 19: Cure Rate and Default Under DOT, Itaim Paulista, 2007‐2009.................................................. 33 Figure 20: Impact 1, Increased Number of Notified Cases......................................................................... 40 Figure 21: Impact 3, Treatment Outcome Improved.................................................................................. 41 Figure 22: Educational Campaign “Essa dupla nao combina” (TB and HIV ‐ A Mismatch Couple).............44 Figure 23: Stamp Questionnaire for TB Screening at SAE Level. ................................................................ 45 Figure 24: Number of PPD, PPD≥5mm and IPT Provided at IAPI, Porto Alegre, 2008 to mid‐2012...........46 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 v TABLES Table 1: Percent of Smear Positive Patients That Default. Selected Municipalities, Pernambuco State, 2007‐2011 ...................................................................................................................................................30 Table 2: Number and Percent of Health Facilities with DOT. Selected Municipalities, Pernambuco State, 2009‐2011. ..................................................................................................................................................30 Table 3: Smear Positive TB Cases Under DOT by Municipality ‐ 2007 – 2011. ........................................... 30 Table 4: TB Treatment Success Rate (%) in 14 Priorities Municipalities in Rio de Janeiro .........................34 Table 5: Results of KAPB intervention at Franco da Rocha Penitentiary (2011‐2012 ................................42    vi USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 vii ACRONYMS ACSM Advocacy, Communication and Social Mobilization ADS USAID’s directive program ART Anti-Retroviral Therapy ARV Anti-Retroviral BCC Behavioral Change Communication BCG Bacilli Calmette-Guerin vaccine, type of vaccine BEMFAM Sociedade Civil do Bem-estar Familiar no Brasil BFP “Bolsa Familia” Program BMGF Bill and Melinda Gates Foundation CBO Community Based Organization CDC U. S. Centers for Disease Control and Prevention CHA Community Health Agents CHWs Community Health Workers CRPHF Centro de Referência Professor Hélio Fraga (National Tuberculosis Reference Center) DAB Basic Attention Department DMIS Disease Management Information System for Di DOT Directly Observed Treatment DOTS Directly Observed Therapy, Short Course DRS Drug Resistance Survey DST Drug Susceptibility Test EQA External Quality Assurance E-TB Manager MDR Data Management Information System (International) FAP Fundação Ataulpho de Paiva FDC Fixed Dose Combination FHP Family Health Program FIND Foundation for Innovative New Diagnosis FIOCRUZ Oswaldo Cruz Foundation FLDs First Line Drugs FP Family Planning GFATM Global Fund to Fight AIDS, Tuberculosis, and Malaria GOB Government of Brazil viii USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 HCWs Health Care Workers HSS Health Systems Strengthening IAA Centers for Disease Control and Prevention Inter-Agency Agreement IUATLD International Union against Tuberculosis and Lung Diseases IEC Information, Education and Communication LTBI Isoniazid for Latent TB Infection INCQS National Quality Assurance Institute IPs Implementing Partners IPPF International Planned Parenthood Federation IPT Isoniazid Preventive Therapy JHU John Hopkins University JSB John Snow/Brazil Consultoria JSI John Snow, Inc. KAPB Knowledge, Attitude, Practice and Behavior Survey LABMOST Laboratory Management Model LACENs Central Laboratory of the State LGBT Lesbian, Gay, Bisexual and Transgender LOA Letter of Agreement MARPS Most At-Risk Populations MCH Maternal and Child Health MDGs Millennium Development Goals MDR-TB Multi Drug Resistant Tuberculosis M&E Monitoring and Evaluation MSM Men Who Have Sex with Men MODS Microscopic Observation for Detection and Susceptibility MOH Ministry of Health MOS Ministry of Social Development MOU Memorandum of Understanding MSH Management Sciences for Health NGOs Non-Governmental Organizations NPAT National Program against Tuberculosis NTP National Tuberculosis Control Program PACS Health Community Outreach Program USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 ix PAHO Pan-American Health Organization PHI Public Health Impact PLWHA People Living With HIV/AIDS RGS Rio Grande do Sul RPM Plus Rational Pharmaceutical Management Plus Program RH Reproductive Health RJ Rio de Janeiro AE HIV/AIDS Specialized Care Services SC Santa Catarina State SIAPS Systems for Improved Access to Pharmaceuticals & Services SINAN National Information System for Notifiable Diseases SITE-TB MDR Information System (latest version) SLDs Second Line Drugs SOW Scope of Work SP São Paulo SPS Strengthening Pharmaceutical Systems SVS Secretary of Health Surveillance SUS Sistema Único de Saúde (Unified and Decentralized Health System) TA Technical Assistance TB Tuberculosis TB CAP Tuberculosis Control Assistance Program TB CTA Tuberculosis Coalition for Technical Assistance TOT Training of Trainers UNION International Union against Tuberculosis and Respiratory Diseases USAID United States Agency for International Development USG United States Government WHO World Health Organization XDR-TB Extensively Drug Resistant TB x USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 xi EXECUTIVE SUMMARY USAID has provided development assistance to Brazil since the early 1960s. At least half of that overall technical and financial assistance has been provided to the health sector, including infectious diseases such as HIV/AIDS, tuberculosis (TB) and malaria. These diseases affect health, well-being, and productivity, especially among low income people. USAID worked closely with the Ministry of Health (MOH), civil society, and local institutions to develop a joint response to these public health priorities. From 2001 to 2012, USAID/Brazil invested approximately US$ 35 million in TB programs aimed at expanding Directly Observed Therapy Short-Course (DOTS), the WHO-recommended strategy for TB control, in selected priority areas with the highest burden of TB. USAID programs and projects also combated Multidrug-Resistant TB (MDR-TB), improved coordination between TB and HIV/AIDS Programs, and increased public awareness of TB. USAID TB funding for Brazil ended in September 2011, i.e., two years earlier than previously expected. All activities will terminate in 2013. It is essential to capture the Agency’s legacy in TB, share lessons learned and best practices, and inform the recommendations that will be provided to the Government of Brazil (GOB), other donors, and stakeholders. In October 2012, USAID/Brazil commissioned GH Tech Bridge II to conduct an independent, external performance evaluation of its TB portfolio, covering the period of fiscal year (FY) 2001 to FY 2012. This evaluation was designed and carried out in accordance with the USAID Evaluation Policy, launched in January 2011, and addressed the most important and relevant questions about the program’s performance. The evaluation was conducted in accordance with institutional aims of reinforcement of local capacity and respectful engagement with all partners and stakeholders. The team assigned by GH Tech II to carry out the external evaluation included: • Dr. Carolyn Mohan – Senior TB and Evaluation Expert (Team Leader) • Dr. Anna Cristina Carvalho – TB Specialist • Ms. Maria Etelvina Reis de Toledo Barros – Monitoring and & Evaluation Specialist • Dr. Silvia Kelbert – Monitoring and & Evaluation Specialist • Mr. Valdir de Souza Pinto – TB Specialist Listed below are the main questions of this performance evaluation: • What were the objectives of the USAID/Brazil TB program? • What were the objectives of the individual TB projects? • Did the program and projects meet their objectives? • How were the projects perceived and valued? • How were the projects managed and implemented? The evaluation began with an in-country, four-week visit, from October 15 to November 9, 2012, and a return visit one month later, from December 2 to December 9, 2012. The Evaluation Team was divided into two groups for site visits, with team members assigned according to their complementary skills and to prevent any possible conflict of interest. States and municipalities were chosen for visits based on the presence of a USAID-supported TB project either currently or in the past 10 years. Over 50 interviews, with more than 150 individuals across eight states and three time zones were performed during field visits. An additional 20 interviews were conducted by phone and Skype calls. The evaluation included a combination of quantitative and qualitative data collection and analysis, using primary and secondary data sources. The evaluation team conducted an extensive review of all resources obtained from USAID and its implementing partners, which included work plans, quarterly, xii USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 annual and final technical and financial reports, training curricula and information, education and communication materials. The evaluation team improved the internal and external validity of data by triangulating different data sets, methods and approaches. Interviews, site visits, project technical and financial reports, scientific publications, and public documents were collected and analyzed. Overall, USAID’s TB program was perceived as highly valuable, with strategic investment that leveraged and supplemented existing, larger technical, managerial and financial resources in the National TB Program. The main legacy of USAID support to TB control in Brazil is the adoption of DOTS as the national TB control policy, and implementation of quality DOTS programs in selected intervention sites. USAID support was key in influencing policy change for adoption of DOTS and in helping to implement and expand quality DOTS in priority high burden areas. By funding relevant research, USAID played a fundamental role in provided scientific evidence to this major policy change, which included Ethambutol in the TB treatment regimen, in accordance with international standards. USAID support to DOTS expansion projects also helped to integrate TB activities into the Family Health Program (FHP) and the Community Health Agents (CHA). In 2010, 56.3% of sputum smear positive cases were diagnosed in FHP clinics, as compared to only 22% in 2004. DOTS training for health care staff was crucial for the success of USAID/Brazil TB programs. Over 66,000 health workers and professionals were training in DOTS with USAID support. Mission support focused on building the capacity of national and state TB officials through training and direct technical assistance. Training topics included management skills, use of epidemiological data for program management, and effective planning and organization of TB services. USAID-supported activities also contributed to upgrades in the public laboratory networks by decentralizing procedures and implementing laboratory quality control programs that ensured increased access to TB diagnosis, improved drug quality testing and helped national and state reference labs qualify for ISO accreditation. In addition, USAID contributed to improved collaboration between TB and HIV/AIDS programs in order to address TB/HIV co-infection, particularly through programmatic interventions such as incorporation of screening for TB among HIV patients in HIV specialized care centers. Mission support also led to the development of a national TB/HIV plan. Finally, USAID supported the development and enhancement of disease management information systems, Web-based case and patient management platforms, and models for TB monitoring and supportive supervision. Local production of quality fixed-dose combination drugs was a direct result of technical support provided with USAID financing. Although USAID made a vital contribution to the Brazilian National TB Program in DOTS adoption, in addition to fostering significant increases in the national TB budget, the program did not, however, meet all of its objectives, in particular in improving DOTS quality in Rio de Janeiro state. Critical assumptions made at the strategic planning and project design phase were either not realized, or met to a limited degree, thus negatively impacting USAID and its partners’ ability to meet all their objectives. Despite the positive results obtained in DOTS expansion, the percentage of patients that successfully complete TB treatment is still below the target of 85%. Treatment default rates are still very high, at over 10%, in many urban areas, including USAID-assisted areas. In addition, political commitment – probably the singular most important determinant of success and sustainability of USAID’s support to TB control – is USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 xiii still weak in some priority areas, particularly at the state level. High staff turnover also poses challenges to the sustainability of USAID training investments. Still on sustainability, drug supply and management is likely to continue at high quality levels, given the Government of Brazil’s political commitment and increased local capacity, as well as effective monitoring by the quality control board. It is also worth noting that a substantial increase in the federal government’s budget for TB has been observed in recent years, making the Brazilian NTP one of the largest TB programs in the world. Several elements of USAID’s program may be sustained as they have been adopted by the NTP. This includes more reliable TB databases, laboratory quality control programs, diagnosis and treatment availability, increased social mobilization, and use of behavior change communication in program design. New diagnostic methods that are being validated with USAID support (e.g. GeneXpert) will soon be introduced. These new technologies will greatly accelerate diagnosis of both TB and multi-drug resistant TB. A summary of selected lessons learned and recommendations is listed below and further expanded in the full evaluation report. LESSONS LEARNED  Political commitment is extremely important, if not the most important factor for sustainability of results. Even if financial resources are limited, prioritizing TB as a public health concern in states and municipalities fosters public and private sector support to TB control efforts.  Integration of TB activities into the Family Health Program with adequate training, monitoring, and supervision, helps to increase access to quality DOTS, improves treatment outcomes, and reduces default.  The availability of laboratory support for TB diagnosis is still limited in many cities. Issues relate not only to the shortage of professionals and laboratories, but also to gaps in organization of labor that cause delays in results availability. Improvement of computerized systems for laboratory results management, coupled with integration of new diagnostic methods, represent good prospects for the future.  Supportive supervision is critical to ensure professional development, and to motivate and incentivize staff. It is as important, if not more so, than training. Staff morale and performance are improved with ownership of, and connection to, a larger program. Accountability and recognition of good performance encourage ownership of program outputs and outcomes.  One-year, or otherwise short-term projects, are insufficient to achieve sustainable results in TB control. Given the complex bureaucracy at all levels of government, projects that span for one year only are at a disadvantage, as they may be negatively impacted by the required start up negotiations. In addition, projects limited to one geographic site may create professional conflict and disruption among TB staff in non-intervention, neighboring areas. Geographically-comprehensive projects are more likely to produce sustainable results. RECOMMENDATIONS  Pursue a partnership between the Ministry of Health and the Ministry of Social Development to increase coordination at the policy level, including the continuation of incentives such as the Basic Food Basket, to promote TB treatment adherence, reduce default and increase treatment completion rates.  Mainstream TB in GOB poverty alleviation programs to address the needs of most vulnerable populations.  High turnover of health sector staff is a reality in many countries with high TB burden. Brazil is no exception. Therefore, a comprehensive human resource development and capacity building plan is xiv USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 necessary, in addition to the creation of monetary and non-monetary incentives for staff to remain engaged in TB programs.  Strengthen the partnership with the Ministry of Justice’s National Justice Secretariat to design and implement a comprehensive strategy for TB control in the penitentiary system.  Increase public-private partnerships with both national and international partners. Given limited public health resources, partnerships can help strategic leverage of available resources and provide critical technical assistance in areas such as program management and operations research.  Provide continuous training to state and municipal staff in the areas of strategic planning, annual work planning, and social mobilization. These skills are crucial to ensure the establishment of realistic goals and implementation of sustainable activities.  Provide additional technical support at the state level, including state-to-state exchange of ideas and experiences. Through concerted efforts, develop a plan for information sharing and provide fora for state officials to actively share lessons learned, best practices and challenges. Underperforming states should be provided with technical assistance from both the NTP as well as from high￾achieving States.  Prioritize support to vulnerable populations aimed at reducing default and ensuring treatment completion. This support requires interaction with different institutions and development of specific and complex professional skills.  Fully integrate DOTS into the FHP and CHA programs. Integration of DOTS into these primary health care programs addresses the needs of poor and vulnerable TB patient’s right at the communities where they live and work, thus contributing to better treatment outcomes and sustainability of activities.  Either maintain or increase the level of national funds for TB control, despite possible additional support from partners and donor agencies.  Continue to support and promote investments in alternative approaches for TB diagnosis, including availability of rapid tests such as GeneXpert that may facilitate the identification and treatment of drug resistant TB cases.  Foster increased operational research on TB. A comprehensive operations research plan should be developed by the NTP outlining timelines and responsibilities for each department, with adequate resources to carry out activities.  Continue to strengthen laboratory capacity countrywide, especially at the state level. The laboratory network is still weak, with resources and performance varying considerably from one location to another.  Continue to improve biosafety measures, especially administrative controls and personal respiratory protection.  Scale-up HIV rapid testing for HIV diagnosis among TB patients and TB screening among HIV patients.  Create and implement a TB/HIV co-infection information system. USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 1 I. INTRODUCTION USAID has provided development assistance to Brazil since the early 1960s. At least half of that overall technical and financial assistance has been provided to the health sector, including the areas of maternal and child health, family planning, reproductive health, health systems strengthening, and infectious diseases, including HIV/AIDS, tuberculosis (TB) and malaria. In the past ten years, USAID/Brazil increased efforts to fight the spread of infectious diseases through a focused program that addressed TB control and HIV/AIDS prevention. These diseases affect health, well-being, and productivity, especially among low income people. USAID worked closely with the Ministry of Health (MOH), civil society, and local institutions to develop a joint response to these public health priorities. USAID/Brazil’s TB programs were designed to expand Directly Observed Therapy Short Course (DOTS), the WHO-recommended strategy for TB control, in priority areas with the highest burden of TB in the country; combat Multidrug Resistant (MDR) TB; improve coordination between TB and HIV/AIDS programs; and increase public awareness of TB. In October 2012, USAID/Brazil commissioned GH Tech Bridge II to conduct an independent, external performance evaluation of its TB Portfolio, covering the period of 2001 to 2012. This evaluation was designed and carried out in accordance with USAID’s Evaluation Policy, launched in January 2011, and addressed the most important and relevant questions about the program’s performance. High methodological standards were followed from the design stage, ensuring that systematic methods were used to arrive at valid findings and conclusions. In addition, the evaluation was led by an external team of experts. Transparency was addressed by using and sharing publicly available reports and data provided by stakeholders, with a commitment to full and active disclosure. Furthermore, a comprehensive description of methods, key findings and recommendations was provided. The evaluation was conducted in accordance with institutional aims of reinforcement of local capacity and respectful engagement with all partners and stakeholders. USAID TB funding for Brazil ended in September 2011, i.e., two years earlier than previously expected. All activities will terminate in 2013. As such, it is essential to capture the Agency’s legacy in TB through a thorough evaluation of program performance and results achieved. This evaluation will also serve to share lessons learned and best practices, as well as to inform recommendations that will be provided to the Government of Brazil (GOB), other donors and stakeholders. The team assigned by GH Tech II to carry out the external evaluation included:  Dr. Carolyn Mohan – Senior TB and Evaluation Expert (Team Leader)  Dr. Anna Cristina Carvalho – TB Specialist  Ms. Maria Etelvina Reis de Toledo Barros – Monitoring and &Evaluation Specialist  Dr. Silvia Kelbert - M&E Specialist 2 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012  Mr. Valdir de Souza Pinto – TB Specialist. The report drafting committee was comprised of Drs. Carvalho, Kelbert and Mohan. The evaluation began with an in, four-week visit from October 15 to November 9, 2012, and a return visit one month later, from December 2 to December 9, 2012.The Evaluation Team was divided into two groups for site visits, with team members assigned according to their complementary skills and to prevent any possible conflict of interest. States and municipalities were chosen for visits based on the presence of a USAID-supported TB project currently or in the past 10 years. Over 50 interviews, with more than 150 individuals, across eight states and three time zones were performed during field visits. An additional 20 interviews were conducted by phone and Skype calls. Key informant interviews and site visits included:  USAID/Brazil current and former staff  Implementing partners Sociedade Civil do Bem-estar Familiar no Brasil (BEMFAM), Management Sciences for Health (MSH), Johns Hopkins University (JHU), the Pan-American Health Organization (PAHO), John Snow Inc. (JSI), and KNCV  The Centers for Disease Control in Brazil, and the Health and Human Services Office  Senior NTP Officials (NTP Coordinator, Technical Officers, and Strategic Information Officer)  State TB programs in Pernambuco, Rio Grande do Sul, Rio de Janeiro, Santa Catarina São Paulo, and the Federal District  Selected municipal TB programs in Belo Horizonte, Manaus, and Pernambuco, Rio de Janeiro, and São Paulo states  Municipal TB and HIV/AIDS Programs in Porto Alegre and São Leopoldo (Rio Grande do Sul state) and Florianópolis (Santa Catarina State)  Government officials and staff engaged in TB Control, as well as laboratory and health facility staff in target areas: Centro de Referência Professor Hélio Fraga (CRPHF, the national TB reference lab), Farmanguinhos and Fiocruz (Rio de Janeiro state)  Civil society organizations and activists (Transformarte, Rede Paulista, EPAH, John Snow do Brasil)  Health care workers, managers and TB patients.  Donors supporting TB activities, e.g. the Global Fund and Global Health Strategies and the Gates Foundation USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 3 II. PROJECT BACKGROUND TB CONTROL IN BRAZIL, 2001 TO 2012 The modern history of the Brazilian response to tuberculosis begins early in the 20th century, with the “Liga Contra a Tuberculose” (Leagues against Tuberculosis) promoting scientific advances, such as BCG vaccination in 1927. Since then, the role of the public sector in the organization and implementation of TB control activities increased gradually through the creation of the “Inspetoria de Profilaxia da TB” (TB Prophylaxis Inspection Services – 1920), the “Serviço Nacional de Tuberculose” (National TB Services – 1940), and the “Campanha Nacional Contra a Tuberculose” (National Campaign Against Tuberculosis – 1946). In 1979, a short-course, outpatient-based, self-administered treatment regimen was adopted. Beginning in 1981, the standard first-line regimen for treating TB consisted of Rifampicin, Isoniazid and Pyrazinamide (Hijjar, 2007; Kritski and Ruffino-Netto, 2000). In the late 80s, advances in TB therapy led the industrialized countries of North America and Western Europe to no longer consider this disease a public health priority. However, the increase in TB cases related to the HIV epidemic, and the growth of poverty, mainly in urban settings, associated with the insurgence of outbreaks of multi-drug resistant TB (MDR-TB) generated awareness of a global epidemic. In 1991, fewer than 15 countries worldwide were able to report TB treatment outcomes. Less than half of TB cases were covered by proper treatment services, and less than half of the cases treated were cured (Raviglione, 2003). In 1991, a World Health Assembly resolution recognized TB as a major global public health problem, establishing two targets for TB control worldwide: detection of 70% of new smear-positive cases, and cure of 85% of such cases by the year 2000 (WHO, 1993). In 1993, the World Health Organization declared TB a world emergency, proposing a new TB control policy as a response to the increasing number of cases all over the world. In 1995, this new policy was branded “DOTS”, standing for Directly Observed Therapy Short-Course. The DOTS strategy is based on five essential elements:  Political commitment with increased and sustained financing  Case detection through quality-assured sputum-smear microscopy  Standardized short-course treatment with supervision and patient support  An effective drug supply and management system  Recording and reporting system allowing assessment of treatment results. (Raviglione 2003; WHO, 2012). In the 1990s, Brazil’s public health system was marked by the implementation of the Unified and Decentralized Health System (SUS), a new health sector reform to which the TB program should be integrated. SUS implementation was based on the decentralization of health services, with transfer of health management to the municipal level. However, local health managers in municipalities had to deal with financial, institutional, and administrative difficulties right at the beginning of the transfer process (Ruffino and Villa, 2006). Decentralization occurred with little integration between federal, state and 4 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 municipal health services. Many municipalities lacked the minimum managerial capacity necessary to perform the tasks required (Kritski and Ruffino-Netto, 2000). The process was marked by crises of “health funding squeeze,” due to limitation of resources for clinical activities (Cordeiro, 2001). The immediate consequence of decentralization for the NTP was a lack of commitment and funding for TB control, leading to a reduction of NTP coverage, declining treatment success rate, and increasing treatment default. In addition, during the 1990s, discontinuation of the production of TB drugs in Brazil caused frequent stock outs and led to acquisition of drugs in the international market (Ruffino-Netto, 2001; Kritski & Ruffino-Netto, 2000). A review of NTP activities from 1990 to 1995 shows a 50% reduction in the number of TB diagnoses based on smear exams, with a concomitant increase in the number of cases diagnosed through radiological findings alone - most likely, a consequence of poor program management to ensure proper bacteriological diagnosis (Pio et al., 1997). Health sector reform in Brazil initially had a negative impact on the control of several infectious diseases, including TB (WHO, 1998). In 1998, with renewed political commitment by the Ministry of Health, the NTP was able to kick off another National Plan against Tuberculosis (NPAT), covering 5,500 municipalities. The National Disease Notification System (SINAN) received a major boost in 1998, with regulation of its use and mandatory, quarterly input of information on TB cases, in addition to feedback to the primary level for updating (Brazil, 2007). The NPT officially proposed the implementation of the DOTS strategy in 1998. Initial implementation of DOTS faced several problems, mainly related to the level of decentralization and organization of the health system at the regional, state and municipal levels (Ruffino e Villa, 2007; Levcovitz, 2001). Free access to TB diagnosis and treatment, availability of TB drugs in public health settings only, and the use of a standardized short-course therapy with Rifampicin were already in place, and formed a strong basis for DOTS implementation and expansion in Brazil. However, many TB control practitioners resisted the implementation of directly observed treatment (DOT). Critics of DOT pointed to the results of the first Drug Resistance Survey (DRS) (1996-1999) that identified a low level of resistance to first line TB drugs. These results seemed to indicate that self-administered therapy worked well, and that the role played by Brazil as a pioneer country in adopting short course, self-administered, outpatient TB treatment using Rifampicin was showing satisfactory results. In addition, some TB experts advocated that supervision of the patient’s therapy was a violation of human rights. However, the high default rate of self-administered treatment reported in the 1990s (14% country wide and 30–40% in selected urban areas) (Diniz et al, 1995), coupled with considerable bias identified in the DRS (e.g. exclusion of samples from the largest hospitals) became strong arguments in favor of urgent implementation of DOTS in Brazil. Initially, the WHO estimated that DOTS coverage in Brazil would increase from 0 over the 1994-1997 period, to 3% in 1998; 81% in 2004; and, finally, 100% coverage in 2005. However, DOTS coverage later reported by the Brazilian NTP was lower than expected: 34% in 2003, 52% in 2004, and 68% in 2005 (Stop-TB, 2006). Within a decade, almost all countries had adopted DOTS and there had been considerable progress towards global targets established for 2005: detection of 70% of the estimated number of smear-positive pulmonary cases, and successful treatment of 85% of those cases. In 2005, the number of cases reported worldwide reached over five million, while treatment success rates reached 85%. In 2000, all of the 22 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 5 TB High Burden Countries1 were implementing DOTS. Brazil presented the lowest coverage at 7%, while other countries such as Kenya and Tanzania reached 100% - although these numbers do not necessarily represent full geographical coverage or ensure access to quality services (Dye, 2003). In 2000, the GOB spent US$ 7.97 per capita on primary health care. Of those, only US$ 0.13 was spent on TB (Brazil, 2003). In that same year, an important achievement for decentralization of TB activities in Brazil was the creation, by the MOH, of the Basic Attention Department (DAB), which incorporated the NTP and other health programs (Hansen’s disease, hypertension, and diabetes, among others) into the Family Health Program (FHP). The FHP represents a concept of primary health care centered on the family and community, where a team of health professionals ensures access to health assistance. Teams are composed of physicians, nurses, assistant nurses and community health agents (CHA) who are responsible for providing health care to a specific number of families in a defined geographic area. With that, TB active case finding, diagnosis, treatment and follow-up of TB cases (mainly smear-positive pulmonary cases, that represent 57% of total TB cases diagnosed in Brazil) should be managed at the basic health care level. Community health agents work directly with communities through home visits. This represents a key resource in offering DOT to TB patients who cannot travel to distant health units, and to decrease both direct and indirect costs (BRAZIL, 2000, Brazil, 2012a). Tuberculosis was declared a national priority in 2003. TB treatment default rates were adopted as one of the 20 main sentinel indicators of the Primary Care Indicators’ Deal, a national tool to monitor health activities and services in primary care (Brazil, 2003). The NTP was gradually integrated into general health services. Activities were carried out by the three levels of government operating as a network. The creation of the Secretariat of Health Surveillance (SVS) within the Ministry of Health redesigned the TB program by combining surveillance, control, and prevention activities, making integration of health services network possible (Ruffino e Villa, 2007; Brazil, 2012a) Aiming to meet the Millennium Development Goals (MDG) of reducing TB prevalence and deaths by 50% by 2015 (in comparison with a 1990 baseline) and achieving the Stop TB Partnership targets for TB control , in 2006 the WHO launched a new policy named Stop TB. The Stop TB strategy is based on six main components:  Pursue high-quality DOTS expansion and enhancement  Address TB/HIV, MDR-TB and other challenges  Contribute to health systems strengthening  Engage all care providers  Empower people with TB, and communities  Enable and promote research (WHO, 2006). Since 2001, Brazil has been reporting significant improvements in TB indicators. The incidence rate dropped from 42.8 cases/100,000 inhabitants in 2001 to 36.0/100,000 in 2011, i.e., a 15.9% reduction over 10 years. The states of Amazonas and Rio de Janeiro present the highest incidence rates in the                                                              1 Eighty percent of all incident TB cases are found in 22 countries, with more than half the cases occurring in 5 Southeast Asian countries. These countries are classified as the 22‐High Burden TB Countries by the WHO. 6 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 country (62.6/100,000 and 57.6/100,000, respectively), while Goiás and the Federal District report the lowest rates at 13.6/100,000 and 11.1/100,000, respectively (Figure 2). A 22.6% reduction in TB mortality rates was observed from 2001-2010: from 3.1 deaths/100,000 inhabitants in 2001 to 2.4/100,000 in 2010. The absolute number of deaths was highest in the Southeastern region, while the highest death rates were observed in the Northeast region. Estimated TB case detection increased from 79% of cases in 1995 to 91% in 2011 (WHO, 2012). Improvement of treatment success rates is still a challenge: the percentage of all smear-positive TB patients that successfully completed treatment was 55% in 2001 and 74% in 2010, still far from the global target of 85% (Brazil, 2012b). Figure 3 presents TB incidence and mortality rates in Brazil, and treatment success percentages for smear-positive cases between 2001 and 2010. Figure 1: TB Incidence Rate in Brazil by State, 2011 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 7 Figure 2: Tuberculosis incidence rate, mortality rate and treatment success percentage for smear-positive cases in Brazil, 2001 – 2010 (adapted) The NTP recognized the importance of the Stop TB strategy to consolidate and expand the achievements obtained through implementation of DOTS. During the first decade of the 2000s, the NTP established and strengthened partnerships with national and international organizations, including Universities, the Brazilian Society of Pulmonology and Tuberculosis, Rede TB, the TB Global Coalition (STOP TB), the International Union against Tuberculosis and Respiratory Diseases (UNION), the United States Agency for International Development (USAID), WHO, and PAHO (Ruffino e Villa, 2007). Renewed political commitment and an increase in internal and external budgets allowed the NTP and state TB programs to implement training in DOTS for health personnel involved in TB control. Considerable investment was made in expanding and improving the management information system for MDR-TB nationwide. In 2009, the NTP introduced the four-drug regimen, which included Ethambutol, in response to the increase in Isoniazid resistance (from 4.4% to 6.0%), reported on the second Drug Resistance Survey (DRS, Brazil, 2011). Strengthening laboratory services and training laboratory technicians and managers became high priorities in that period. The HIV incidence rate is increasing in all Brazilian regions, and particularly in the South of the country, where reported incidence increased from 5.8/100,000 in 1997 to 14.1/100,000 in 2007 (Figure 4). The Southern Region also presents the highest increase in AIDS-related mortality in recent years. Additionally, Porto Alegre, the largest city in, and capital of, the state of Rio Grande do Sul, reports the highest TB mortality rate in Brazil per figure 5 below. HIV/TB collaborative activities have been reinforced in areas of high HIV/TB co-infection rates, and mainly in the Southern Region’s metropolitan areas. With USAID, other donors and private sector support, the NPT implemented operational research, including the GeneXpert validation study that is in progress in Rio de Janeiro and Manaus. The 8 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 results of this study will define the policy for countrywide implementation of this new diagnostic resource. Figure 3: TB incidence rate and % of TB/HIV Co-infection in Brazil by State, 2011. Between 2001 and 2010, the FHP coverage doubled in Brazil: from 25.4% to 52.2%. Smear-positive cases treated at the FHP increased from 50.1% in 2001 to 56.3% in 2010. In 2010, 71.5% of basic health units offered DOT for TB patients (Brazil, 2012a, Brazil, 2012b). With respect to the MDG objectives, Brazil has almost reached the target of 50% decline in TB mortality rates as compared to the 1990 rates (from 5.3/100,000 to 2.9/100,000, respectively). However, the target of halving TB prevalence is still far from being reached (Brazil, 2012a). The improvement of TB figures in Brazil was a consequence of complex interactions and strategies that included an increase in the NTP budget from US$ 5.4 in 2002 million to US$ 87.7 million in 2011, relying mostly on national contributions (Brazil, 2012b). In summary, the progress reported by the Brazilian NTP in the last decade was a consequence of varied and interconnected interventions based on the principles of the DOTS and STOP TB strategies. Decentralization of TB diagnosis and care to the primary health care network, and integration into the FHP; implementation of fixed-dose combination regimens; partnerships with national and international institutions for financial and scientific support; and strengthening of civil society participation are among the many outputs that contributed to advancing TB control in recent years. In 2011, the total number of USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 9 new TB cases was less than 70,000, falling from 71,790 in 2010 to 69,245 in 2011, i.e., a 3.5% reduction. This becomes even more significant when analyzed against a sharp increase in case detection rates. Brazil now ranks 17th among the 22 High Burden TB countries (WHO, 2012). THE USAID/BRAZIL TB PROGRAM Over the past decade, USAID/Brazil worked directly with the NTP and provided technical assistance in 46 of the 315 high-burden TB municipalities in Brazil. Figure 6 shows the geographic scope of USAID support to TB projects in Brazil, by state. Figure 4: Location of USAID/Brazil-supported TB Projects 2001-2011. Approximately $35 million was invested in TB control from 2001 to 2012 (Figure 6) through funding to a total of 17 TB projects (Figure 7). 10 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 Figure 5: USAID/Brazil Funding for TB, 2001-2011 Figure 6: USAID/Brazil TB Funding by Implementing Partners, 2001-2011 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 11 In September 2001, USAID/Brazil wrote a two-year TB strategy. That strategy was included in the Mission’s Annual Strategic Plan as a short-term initiative to be fleshed out in USAID/Brazil’s next Strategic Plan expected to cover the FY 2003-2010 period. This second strategy would build upon the lessons learned from the earlier two-year program. The TB program was expected to significantly expand over time in terms of budget and scope. The objectives of the 2001 TB Strategy were to:  Help strengthen political commitment at the national, state and municipal levels;  Support expansion of DOTS coverage in Rio de Janeiro state;  Assist the GOB in the initial implementation of improved mechanisms to monitor TB control activities; and,  Address co-infection of HIV/AIDS and TB as a cross-cutting issue through targeted activities, such as support to studies about co-infection rates and facilitation of improved coordination between HIV and TB programs. The September 2001 Strategy established the following intermediate results: improved national, state and municipal political and administrative support for TB control; expanded and improved implementation of DOTS; and, improved mechanisms to monitor TB control program activities and performance. By the end of two years of implementation, the expected outcomes in selected target areas were:  Increase in the detection of TB cases  Increase in the number of cured cases  Decrease in treatment default  Decrease in TB mortality  Decrease in TB incidence among key populations in selected municipalities of Rio de Janeiro state It is not possible to say, however, whether such significant epidemiological change in TB morbidity and mortality was achieved in pilot areas over the initial two years of program implementation, due to lack of evidence in documentation. The follow on, eight-year strategy was formulated based on activities developed and results achieved in 2002 and 2003. That strategy retained the objectives above and added support to DOTS coverage expansion and improvement of DOTS performance in selected areas. According to the new strategy, by the end of the program, USAID would have contributed to expand DOTS coverage to 75%, cure rates to 85%, and case detection rates to 70% in intervention areas. Choice of implementing partners was carefully considered. A large amount of funding was channeled through USAID central mechanisms in Washington, in order to reduce the Mission’s management burden. USAID/Brazil TB funds were strategically invested to address high burden priority areas and important gaps, such as quality DOTS, adoption of fixed-dose combination drugs, and improved information systems. 12 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 13 III. EVALUATION OBJECTIVE, METHODS & LIMITATIONS OBJECTIVE The objective of this evaluation is to measure the effectiveness of activities and operations of USAID’s US$ 35 million investment in TB control in Brazil from 2001 to 2012; identify lessons learned and best practices; and share recommendations with diverse stakeholders, based on over a decade of program implementation, in addition to capturing the Agency’s legacy to TB control in Brazil. METHODS Performance evaluations answer descriptive and normative questions. They often use monitoring data to aggregate outputs and outcomes in order to compare baseline and end-of-project indicators and targets, and determine whether targets have been met. However, they generally lack a rigorously defined counterfactual. Descriptive questions refer to “what is” questions, seeking to describe or understand a program or processes, or attitudes towards it, like a snapshot. Normative questions are typically a comparison between “what is” and “what should be”. Listed below are the main questions of this performance evaluation:  What were the objectives of the USAID/Brazil TB program?  What were the objectives of the individual TB projects?  Did the program and projects meet their objectives?  How were the projects perceived and valued?  How were the projects managed and implemented? In addition, the SOW asked for answers to the following questions:  What interventions, policies, financing and other enabling environments have been introduced by USAID/Brazil projects?  How were USAID/Brazil projects able to foster integration between programs, policies and service provision units for enhanced TB control?  To what extent have USAID/Brazil projects contributed to sustainable TB control through effective local capacity building within Brazil’s decentralized health system?  How effective was donor coordination and how was USAID/Brazil able to work in a complementary fashion in a multi-donor environment? Performance Evaluation can answer these questions: What did a particular program or project achieved?  How was it implemented?  How was it perceived and valued?  Did the expected outcomes occur?  Other questions pertinent to program design, management and operational decision making. 14 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 The evaluation included a combination of quantitative and qualitative data collection and analyses, using primary and secondary data sources. The evaluation team conducted an extensive review of all resources obtained from USAID/Brazil and implementing partners, which included work plans, quarterly reports, annual reports, final reports, official PowerPoint presentations, financial reports, cooperative agreements, contracts, training curricula, and information, education and communication (IEC) materials, where available. Questionnaires were developed for each specific audience (implementing partners, national TB program, state/municipal TB programs, health care workers, civil society, and patients) to ensure that consistent, reliable information was obtained. The data collection instrument was tested before full implementation and served to collect both quantitative and qualitative data, including information disaggregated by gender, where available. Open-ended questions, as well as the Likert scale were used. Respondents provided voluntary, verbal informed consent, and were assured of the confidentiality of their interview responses. Qualitative data from key informant interviews and site visits were transcribed, and a pattern analysis conducted. The evaluation team then interpreted and discussed the qualitative findings to extract meaning from the information gathered. Considering that this is a non-experimental evaluation, and there is limited baseline information available, data analysis largely focused on descriptive statistics. Analysis of qualitative data involves careful and theoretically informed coding to identify, in the data, indicators of underlying relevant concepts for the evaluation, looking for emerging key words, specific quotes, common patterns and phrases in the responses, as well as latent meaning behind responses. When used in combination, both quantitative and qualitative data yield a more complete analysis, complementing each other. For this reason, the team chose the mixed methods approach, integrating quantitative and qualitative data collection and analysis. Mixed method approach is more than simply collecting both quantitative and qualitative data: it indicates that data will be integrated, related or mixed at some stage. The underlying logic of mixing is that neither quantitative nor qualitative methods are sufficient in themselves to capture both trends and details of the situation. The evaluation improved internal and external validity by triangulating different data sets, methods and approaches (Figure 8). Interviews, site visits, project technical and financial reports, scientific publications, and public documents were collected and analyzed. Performance Evaluations cannot answer these questions  Cause-effect questions: Did the intervention cause the observed outcomes?  Could other explanations be eliminated as possible causes of the observed outcomes?  What difference did the intervention make?  What would have happened in the absence of the program or project (counterfactual)? USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 15 Triangulation is built on the assumption that using different “readings” of one phenomenon, in this case two or more source of data, improves accuracy. If discrepancies are identified, reference sources can be used to clarify and resolve any inconsistencies. Triangulation is also used to minimize bias that may arise from relying on single methods or sources of information. Figure 7: Triangulation Process Validity refers to the accuracy and truth of the data and findings that are produced. Internal validity refers to accurate measurement that produces correct findings. It is important for both performance evaluations and impact evaluations. External validity is the extent to which results are representative and applicable to other settings. It is the ability of a given design to produce results that can be held true for other cases, such as different people, places or times. Evaluation inferences are externally valid when they could be generalized from the sample to another population or setting. While the internal validity of this performance evaluation is good, its external validity is limited. The reader must be cautioned against broad application of results of the USAID/Brazil TB portfolio evaluation to other USAID Missions or other TB programs. LIMITATIONS Given the long time period covered by the evaluation, there was limited access to documentation and sources of information for projects that closed more than five years ago. This includes documentation 16 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 provided by USAID/Brazil and also reports received from implementing partners and states/municipalities. For current projects and/or projects that closed less than five years ago there was some missing information as well. Despite many attempts made by the evaluation team to obtain that information, especially quarterly and final reports, in addition to approved work plans, in some cases there was limited to no success. A convenience sample of patients and health care workers were interviewed in Sao Paulo state and municipality, based on their availability at the health centers visited by the Evaluation Team. This limits the team’s ability to generalize findings for these particular groups. IV. FINDINGS, CONCLUSIONS & RECOMMENDATIONS OVERALL FINDINGS The main questions for the evaluation of the USAID/Brazil TB Portfolio 2001-2012 are answered below. The following results are presented by major activity area and by selected intervention states that received Mission support (Pernambuco, Rio de Janeiro, and Sao Paulo). Did USAID/Brazil TB Program meet its objectives? Although the Mission made a vital contribution to the Brazilian National TB Program in the area of adoption of DOTS, in addition to fostering significant increases in national TB funding, it did not, however, meet all of its objectives, in particular in improving DOTS quality in Rio de Janeiro state. As mentioned earlier, the USAID/Brazil 2001 TB strategy included the following objectives: I. Help strengthen political commitment at the national, state and municipal levels Political commitment was effectively strengthened at the national level with the contribution of the USAID/Brazil TB program, particularly in the areas of promoting advocacy and appropriate allocation of funds. However, political commitment to TB control efforts varied considerably at the state and municipal levels. II. Support expansion of DOTS coverage in Rio de Janeiro state DOTS coverage in Rio de Janeiro state has been expanded, but the quality of implementation is not uniform in the 12 priority intervention municipalities. III. Assist the Government of Brazil (GOB) in the initial implementation of improved mechanisms to monitor TB control activities Improved mechanisms to monitor TB activities were implemented with USAID/Brazil support. In particular, training of the so called “DOTS nurses” helped to emphasize the importance of monitoring and supervision. Implementation of improved registration and reporting systems in intervention sites, as well as development of new software and management systems provided more reliable TB program data. IV. Address co-infection of HIV/AIDS and TB as a cross-cutting issue through targeted activities such as supporting studies on co-infection rates and facilitating improved coordination between programs USAID’s program included support to TB/HIV co-infection studies. Collaboration between TB and HIV/AIDS programs improved to a limited degree, with further progress required, particularly at the state and municipal levels. Importantly, a number of critical assumptions were made in the 2001 TB strategy:  The MOH has its mandate strengthened and allocates appropriate resources to establish guidelines and supervise standards of care for TB. 18 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012  The MOH ensures that the Family Health Program is appropriately staffed, trained and provided resources to undertake TB control activities.  The MOH’s Sexually-transmitted Infections (STI/HIV/AIDS Program and the NTP recognize and understand the need to improve coordination in order to address HIV/AIDS-TB co-infection.  Other local and international institutions participating in the NTP continue to collaborate with the MOH. The critical assumption of staffing, training and resources for the FHP was not realized, while that of the HIV/AIDS and NTP was met to a limited degree. While collaboration and coordination have improved at the national level, much improvement is still required at the state and municipal levels. Lack of fulfillment of these critical assumptions negatively affected the Mission’s ability to meet all of the objectives in its strategies. What were the results of the Program? DOTS Training for health care staff was crucial for the success of the USAID/Brazil TB program. Over 66,000 health workers and professionals were trained in DOTS with USAID support. Up until 2006, a total of 32,000 people were trained. Information disaggregated by year was unavailable. From 2007 to 2011, an additional 34,226 people were trained in DOTS (Figure 9), regularly exceeding the target established for each fiscal year. This increase was mainly due to acceleration of activities in both Sao Paulo city and state, as well as in a few municipalities in Rio de Janeiro, due to greater participation of local governments. In 2009, over 11,000 people were trained, which is more than 5 times the target for that year. Mission support focused on building the capacity of national and state TB authorities through training and direct technical assistance. Training topics included management skills, use of epidemiological data for program management, and effective planning and organizing TB services. Figure 8: Number of people trained in DOTS using USG funding, 2007-2011 Disaggregation by gender was not available for training information. USAID/Brazil’s Annual Report only shows number of people trained disaggregated by sex in 2007 (Figure 10). After 2007, training numbers are provided as a single, consolidated figure. 2007 2008 2009 2010 2011 Total Target 1,000 3,000 2,000 2,000 3,500 11,500 Results 3,859 5,764 11,084 3,586 9,933 34,226 0 5,000 10,000 15,000 20,000 25,000 30,000 35,000 Number of people trained Number of people trained in DOTS using USG funding, 2007‐2011 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 19 Figure 9: Number of people trained in DOTS using USAID funding, disaggregated by sex, 2007 Overall, the percentage of USG-supported labs performing TB microscopy with over 95% correct microscopy results in 2007 and 2008 was over 85% (Figure 11). From 2007 to 2008, there was a decrease in that percentage due to the inclusion of new, additional municipalities and laboratories required training to improve their performance. Figure 10: Percentage of USG supported lab performing TB microscopy with over 95% correct microscopy results, 2007-2008 From 2006 to 2008, the USAID program reported an increase by 70% in case notification rates in target areas, thus exceeding the target in 2008 (Figure 12). This may be due to an increase in incidence or, more likely, to improvement in data collected by newly trained and supervised TB nurses, as stated in USAID/Brazil’s 2008 Annual Report. In 2009, however, this rate decreased by over 50%. The evaluation team was unable to find a justification for this decrease in the following Men Woman Total Target 300 700 1000 Result 921 2,938 3859 0 500 1000 1500 2000 2500 3000 3500 4000 4500 Number of people trained Number of people trained in DOTS using USG funding in 2007, disaggregated by sex 70% 100% 100% 86% 0 20 40 60 80 100 2007 2008 Percentage of USG supported lab performing TB microscopy with over 95% correct microscopy results, 2007‐2008 Target Results 20 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 year’s annual report. It is possible to infer, however, based on the Mission’s reports, that the enormous variability of political commitment and infrastructure in high burden municipalities in Rio de Janeiro state may partially explain the steep decrease in case notification rates in target areas. In 2010, the target was revised downward, from 75/100,000 to 40/100,000. The results for that year exceeded the target, even though the number presented in 2010 is referred to as partial. The 2011 target of 50 cases notified per 100,000 inhabitants was not met. Figure 11: Case notification rate in new SS+ pulmonary TB cases per 100,000 pop in USG supported areas, 2006-2011. In 2009, USAID/Brazil introduced a new indicator to measure the percentage of new registered smear-positive pulmonary TB cases that were cured and completed treatment under DOTS (i.e. treatment success rate) in USAID supported areas. The target for that year was 80%, while and the actual result was 68%. No further information was available for this indicator in USAID/Brazil’s annual report. The percentage of all registered TB patients who are tested for HIV through USAID supported programs increased from 75% to 90%, from 2006 to 2008 (Figure 13). In 2009, a 40% decrease was observed in this figure, which might be partially due to incomplete data submitted for Sao Paulo state. In 2010, the actual result again failed to meet the target. In 2011, the target was revised and reduced from 85% to 45%. In that year, USAID/Brazil started a TB/HIV co-infection control project in the Southern region of Brazil. At the national level, USAID fostered the establishment of a TB/HIV Committee to promote integration between the two programs and support TB/HIV collaborative activities. Following the implementation of that project, a 59% increase was reported in the percentage of registered TB patients who are tested for HIV. This indicator was disaggregated by sex in 2008 only, with similar results for women and men. 2006 2007 2008 2009 2010 2011 Target 42 75 40 50 Result 42 44 71 34 56.23 42.65 0 10 20 30 40 50 60 70 80 Case Notif rate per 100,000 pop Case notification rate in new SS+ pulmonary TB cases per 100,00 pop in USG supported areas , 2006‐2011 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 21 Figure 12: Percentage of All Registered TB Patients Who Are Tested for HIV Through USG Supported Programs, 2006-2011. In 2010, USAID introduced a new indicator for treatment success in USAID-assisted DOTS Plus2 programs to treat MDR TB patients. No target was set for that year. In 2011, a 40% increase in the treatment success rate was observed when compared to 2010 (Figure 14). Figure 13: Treatment Success Rate in USAID-assisted DOTS Plus Programs to Treat MDR TB Patients, 2010-2011 The number of institutions with improved Management Information Systems (MIS) as a result of USAID assistance doubled from 65 in 2006 to 122 in 2007. This increase coincides with the                                                              2 DOTS Plus is the WHO-recommended treatment for MDRTB. 2006 2007 2008 2009 2010 2011 Target 80 85 70 80 45 Result 75 77 90 54 47 59 0 10 20 30 40 50 60 70 80 90 100 % of TB patient tested for HIV Percentage of all registered TB patients who are tested for HIV through USG supported programs, 2006‐2011 75 49.5 74 0 10 20 30 40 50 60 70 80 90 100 2010 2011 Treatment success rate in USAID assisted DOTS  plus programs to treat MDR TB patients, 2010‐2011 Target Result 22 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 development of a web-based surveillance system (Disease Management Information System - DMIS) in 2004-2005, which became operational in 2006, as a USAID response to a request from CRPHF for implementation of an MDR-TB management system. This initiative contributed to increased detection and cure of MDR-TB cases, and significantly strengthened clinical and case management practices at MDR-TB reference centers. From 2007 to 2009, no increase in the number of institutions using DMIS was reported. The number of centers increased to 138 in 2012. Other important components of USAID/Brazil-supported projects were advocacy and policy change. In 2010, the Mission’s Annual Report included a new indicator to capture the number of improved legislation, policies, regulations and guidelines drafted with USAID support to address access to, and use of, health services. In 2010 and 2011, five such documents were drafted with USAID support. Results were unavailable for most of USAID/Brazil’s custom indicators listed in annual reports, with the exception of the following:  Number of new MDR TB patients diagnosed and initiated on treatment: 474 in 2011, and a target of 720 in 2012; and  Percentage of successfully treated MDR TB (category IV TB) cases: results exceeded the target by 47% in 2011. No further information was available. What is the legacy of USAID support to TB control in Brazil? The main legacy of USAID support to TB control in Brazil is the adoption of DOTS as the national TB control policy, and implementation of quality DOTS programs in selected intervention sites. Investment by international donors, especially USAID and the Global Fund to Fight AIDS, Tuberculosis, and Malaria (GFATM) to build Brazil’s capacity in TB control has opened doors for policy change and awareness of the importance of TB control. USAID/Brazil’s investments in TB influenced relatively rapid policy change for adoption of DOTS, which was finally implemented in Brazil in 2004. USAID was also instrumental in producing scientific evidence to support that change. USAID support was key in helping to implement and expand quality DOTS in priority high burden areas, such as São Paulo (city and state), Manaus (Amazonas State) and Belo Horizonte (Minas Gerais State). DOTS supervision at all levels of the program was enhanced and reinforced (from national to state and state to municipal) with the contribution of USAID, PAHO and GFATM. Providing adequate, supportive supervision poses a challenge for the NTP, due to lack of human and financial resources. USAID support included funding to hire consultants and to train GOB staff, including nurse supervisors who formed a much lauded cadre of senior DOTS nurses. DOTS nurses implemented the strategy for TB case notification and reporting at the state and municipal levels. These nurses were employed by State TB programs but trained, mobilized and empowered in DOTS with USAID financial and technical support. DOTS nurses advanced DOTS in priority states and municipalities by providing training, monitoring and supervision. In Rio de Janeiro state, a similar approach was used with a group of 12 nurses in priority municipalities. In this latter case, consultant nurses’ salaries were paid with USAID/Brazil funds. USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 23 Mission support to DOTS expansion projects helped to integrate TB activities into the FHP and CHA programs. In 2010, 56.3% of sputum smear-positive cases were diagnosed in FHP clinics, as compared to only 22% in 2004. Brazil still has a lot to do to increase TB diagnosis in primary health care, but it is in the correct path to do it and has made significant progress towards this objective. The Mission also contributed significantly to improvements in TB control in Brazil through technical assistance provided by a medical epidemiologist placed in the NTP, mainly to provide operations research guidance. The USAID consultant led the second national Drug Resistance Survey (DRS), supported by USAID. The preliminary results of this survey showed an increase in drug resistance to Isoniazid with the use of the three-drug regimen then adopted by Brazil. Inclusion of a fourth drug, Ethambutol, in the TB treatment regimen, in accordance with international standards, was a major shift in the national TB control policy. Civil society involvement in TB control began with USAID support and expanded through the GFATM project, which began in May 2007 and terminated in April 2012. In fact, the Round 5 Brazil TB application to GFATM was written with USAID/Brazil support and awarded over $21 million for additional DOTS training; Behavior Change Communications (BCC); increased coordination and advocacy through creation of 11 Metropolitan Committees in 10 States, with representation from government and civil society; strengthen the laboratory network through implementation of TB lab quality control; and implementation of rapid HIV testing for TB patients. USAID improved information systems by developing the web-based SITE TB/E-TB Manager MDR￾TB disease management information system that integrates patient and case management in a single platform; refining the TB Web MDR management system in São Paulo state; and deploying WHO-recommended notification and reporting systems. This was an important contribution to generating accurate data for decision making, strategic planning and advocacy. In addition to tracking clinical and epidemiological information about patients, the E-TB Manager system can manage and control the forecast and distribution of second line drugs that are sent to different centers by the CRPHF pharmacy. The system monitors not only MDR TB cases and alternative schemes on a routine basis, but also non-tuberculosis mycobacteria, which is innovative in Brazil. USAID TB funding for Brazil averaged $2.9 million per year over the 10 year period under study. In 2012 alone, the NTP budget was over $77 million. Figure 15 below clearly shows that the vast majority of funding for TB control activities has been allocated by the national government. “The Brazilian TB program will be a showcase in a few years….” USAID/Brazil Implementing Partner 24 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 Figure 14: Percentage of National TB Program Funding by Source, 2003-2012 * Loans refer to the World Bank loan to develop the VIGISUS surveillance project. **Other Grants includes USAID. Note: before 2003, 100% of NTP funds were provided by the Brazilian Government, with no contributions from international donors. Close to half of NTP funding is used for program management, supervision and technical assistance, which includes hiring numerous consultants to fill the MOH’s staffing gap, as depicted in the pie chart in Figure 16 below. The next highest budget item is core staff costs, at 16% of the annual budget. USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 25 Figure 15: National TB Program Funding by Activity in 2012. ACSM (Advocacy, Communication and Social Mobilization); PPM (Public‐Public, Public–Private Mix). Over time, USAID support for TB contributed to increase allocation of government funds to TB at the national level. In 2001, the NTP program funding was $6.3 million. By 2011, it had increased by 14 times to a high US$ 87.7 million. In 2012, funding fell by almost $10 million to $77.4 million (Figure 17). Figure 16: National TB Program Funding 2000-2012. 26 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 However, state and municipal authorities reported that allocation of increased GOB funding to TB activities did not occur uniformly at all levels of government. How sustainable were the results? Overall, some results were sustainable, while others were not, depending on the existing level of political commitment of local counterparts at the three spheres of government. In terms of sustainability of USAID’s investment in TB activities, a substantial increase in Brazil’s federal budget for TB is observed in recent years, making the Brazilian NTP one of the largest TB programs in the world. Moreover, several elements of USAID’s program may be sustained as they have been adopted by the NTP. This includes more reliable TB databases, laboratory quality control programs, diagnosis and treatment availability, increased social mobilization, and use of BCC in program design. The MOH and local governments’ plans for sustaining USAID-supported systems and interventions include individual national, state and municipal agreements with PAHO to enhance financial and technical management of TB control activities, mainly by facilitating efficient use of resources and decreasing bureaucracy. For high priority cities, such as São Paulo, Rio de Janeiro, Manaus and Porto Alegre, these five-year agreements will ensure funding and technical assistance at the following levels:  NTP: US$ 4 million/year  Rio Grande do Sul state: US$3 million/year  Rio de Janeiro state: US$6 million/year  Manaus US$300,000/year (pending political decision)  New 5-year agreement with the state of Pernambuco covering TB among other neglected diseases - US$1 million.  New PAHO/Washington “TB in Big Cities” 5-year regional project, funded by USAID/Washington (includes the municipality of Guarulhos, in São Paulo state. The “Frente Parlamentar Contra a Tuberculose” (Congressional Front against TB) was formed on March 24, 2012, with participation of 220 Congress representatives. This has raised TB high on the national agenda through further congressional engagement and ongoing discussions around increased coordination between government agencies and ministries to enhance TB control. The National Front was inspired by the Rio de Janeiro State Front, fostered by the Rio NGO TB Forum, which was first convened with USAID/Brazil support. In Minas Gerais, the creation of a State Committee called “Respira Minas” (Breath, Minas) put TB high on the political agenda. This multidisciplinary committee includes national, state and municipal governments, academia, hospitals and NGOs. The Committee is responsible for developing an integrated approach to respiratory health care, including TB, which will increase access to the primary health care network. The formation of the Committee was an important milestone in the Minas Gerais State response to TB and received USAID support. Its success will be judged by the extent to which it raises awareness and supports state policy makers, municipalities, and “There was no perception of the end of the USAID project because the gains have been incorporated.” Nurse Manager, Health Unit, São Paulo USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 27 community-based organizations to take TB seriously, reduce stigma and discrimination, and implement long-term programs. It is a true effort to work collaboratively, harmonizing and coordinating individual agendas, to share responsibility and accountability. Ownership is also a very positive aspect in the case of Paranaguá, in the state of Paraná, where the local government assumed full responsibility for providing TB training that was previously supported by USAID/Brazil. Now, trained health professionals are responsible for replicating training, and the municipality is able to continue fighting TB without USAID support. Political commitment was the singular most important determinant of success and sustainability of USAID/Brazil’s support to TB control at the national, state and municipal levels. Political commitment was present at the federal level and in the majority of the States that received USAID support, thus successfully putting TB high on the local agenda, and drilling down to individual health care units. However, in some high priority areas, such as Rio de Janeiro and Rio Grande do Sul states, political commitment is still weak. This affects the sustainability of results achieved through USAID support. The high turnover of municipal Secretaries of Health also negatively affects DOTS implementation, as political commitment may change from one administration to the next. In areas where political commitment was an issue from the beginning, activities were discontinued after project completion Problems with insufficient and inadequate staffing were exacerbated by the lack of a contracting mechanism. In Rio de Janeiro and Rio Grande do Sul states, despite USAID’s substantial support to implement DOTS, municipalities with little political commitment find quality DOTS hard to sustain. In Rio de Janeiro state, the end of USAID support has made some TB activities less reliable (e.g. laboratory results), while the sustainability of program results is uncertain. “International projects reinforce political commitment.” (State TB Coordinator) “After the close out of the project, patients enrolled into DOT, will be left without follow up and supervision (….) as the nurse responsible for the activity will be reallocated to another function. What will happen with patient adherence?” Nurse in charge of DOT 28 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 The end of USAID funding for TB in Brazil will be felt mainly in activities considered as lower priority, such as social mobilization campaigns. Besides that, the absence of flexible, readily accessible funds (as was the case with USAID funds) will create bureaucratic difficulties, even if funding is available from local governments. Reduction in the frequency of scheduled activities, such as staff training and supervision is likely to occur. Loss of logistic support such as transportation of samples to the laboratory network and of health professionals to supervision visits will likely affect quality of care in intervention sites, and particularly in Rio de Janeiro, where local governments may not assume the costs of the program. How was the program valued and perceived? Overall, USAID’s TB Program was perceived as highly valuable, with strategic investment that leveraged and supplemented existing, larger technical, managerial and financial resources. Although challenges have been identified in implementation of individual projects, national counterparts worked well with USAID and its implementing partners. National, state and municipal TB authorities expressed appreciation for the financial, technical and advocacy support provided by the Mission. ILLUSTRATIVE FINDINGS BY SELECTED STATES Pernambuco State In Pernambuco, USAID/Brazil provided support to six TB priority municipalities: Recife, Cabo de Santo Agostinho, Camaragibe, Jaboatão dos Guararapes, Olinda and Paulista. USAID support was successful in garnering public and private sector collaboration in intervention areas, and to move TB higher in the state’s health agenda. The Pernambuco project implemented by the local NGO Sociedade Civil do Bem-Estar Familiar no Brasil (BEMFAM) from 2007 to 2011, was successful in ensuring political commitment of diverse partners, by using a participatory approach for work planning, decision making and project implementation. “The PAHO/USAID project helped a lot in [TB activity] implementation, mainly through the Nurse Task Force and vehicles that were used to collect lab samples from the municipalities. When the municipal project terminated, and without political commitment, the work flow for microscopy diminished, leaving behind a significant number of smear microscopies that were not done. As a consequence, supervision and monitoring of health units have not been done since September 2012.” Municipal TB Official, Rio de Janeiro state “Investment in training provided by the USAID project helped to create the culture of how to work with tuberculosis” Nurse, Epidemiological Surveillance Unit, São Paulo. USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 29 BEMFAM worked in a very collaborative manner with municipal and state authorities, in addition to involving other civil society organizations. Although partners acknowledged that this participatory work presented difficulties and delays, there is consensus that it conferred legitimacy to the project and promoted buy-in and ownership from stakeholders. In work planning sessions, each partner was given responsibility for particular indicators and could see how their contribution impacted the overall achievement of targets. BEMFAM also worked closely with PAHO and invited them to provide comments and feedback during work planning process. This collaboration helped BEMFAM to improve its technical expertise in TB with positive reflections on project implementation. In August 2006, the Mission invited a team of international reviewers to evaluate the Pernambuco project. The evaluation team concluded that the project contributed to the decentralization and expansion of DOTS in priority municipalities and was projected to achieve 50% DOTS coverage. In addition, reports indicate that the project helped to increase political commitment to DOTS by municipal health authorities, t and contributed to improved coordination and collaboration between the state and municipal programs. According to the report, the project increased the percentage of registered TB patients whose treatment outcome is evaluated, reduced the default rate, decentralized lab services, expanded quality control for smear microscopy, and effectively trained staff in DOTS. Criticism included little progress in detecting TB suspects and TB cases. The use of smear microscopy for both diagnosis and control continued to be low, as well as acceptance of HIV testing by TB patients. Further, presentation and analysis of data and cohort analysis was considered weak at all levels. Reportedly, the project performance did improve over time, but the extent to what recommendations of the 2006 evaluation were adopted and executed is still unclear. Overall results were somewhat sustainable. At the basic health unit visited by this Evaluation Team, DOTS was implemented by the FHP team. The nurses and community health agents interviewed were actively engaged in providing DOT and registering and reporting cases. At the secondary level (the policlinic), however, TB treatment was not in accordance with DOTS. Municipal TB authorities expressed their priority to have TB patients treated at the primary care level and to put in place, in 2013, a referral and counter-referral plan to ensure that TB patients receive quality care from the Family Health Program. The targets of the project, consistent with those of the State TB control program, were: Sustain TB case detection at 78% or higher There was not sufficient data available to the Evaluation Team to evaluate whether case detection targets were achieved. Cure at least 80% of patients with directly observed treatment (DOT) Treatment results are not disaggregated by DOTS/non-DOTS so it is not possible to single out cure rates for DOTS patients. “We have trained many. Now, in 2013, we will focus on monitoring.” TB Nurse, Pernambuco 30 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 Treatment default less than 10% Most municipalities have experienced a decline in the default rate, with only Recife reporting a default rate over 10% in 2011, as shown in Table 1 below. This target seems to have been sustained after the end of USAID support in 2011. Table 1: Percent of Smear Positive Patients That Default. Selected Municipalities, Pernambuco State, 2007-2011 Municipality 2007 2008 2009 2010 2011 Recife 13.3 13.8 13.1 17.2 12.6 Cabo de Santo Agostinho 21 12.7 6.9 12.8 6.4 Camaragibe 6.7 7.7 5.1 2.6 2.6 Jaboatão dos Guararapes 7.3 11.6 12.8 13.3 7.4 Olinda 6.8 9.0 10.1 11.2 6.8 Paulista N/A N/A N/A N/A N/A Implementation of DOTS in 50% of health facilities Statistics provided by the State TB Coordination are presented in Table 2 below. In 2011, all but one of the six priority municipalities that received USAID support had implemented DOT in at least 80% of health facilities, clearly indicating that gains were sustained after the end of USAID support. It should be noted, however, that, by definition, if a single patient is receiving directly observed treatment, the health facility can report to be providing DOT. Table 2: Number and Percent of Health Facilities with DOT. Selected Municipalities, Pernambuco State, 2009-2011. Source: Diagtube/SEVS/SES-PE 2009 2010 2011 Municipality Number DOT % Number DOT % Number DOT % Recife 131 115 87.8 140 123 87.9 140 123 87.86 Cabo 41 38 92.7 41 41 100.0 41 41 100.00 Camaragibe 43 43 100.0 43 40 93.0 43 40 93.02 Jaboatão* - - - 94 - - 94 32 34.04 Olinda 37 37 100.0 37 37 100.0 37 37 100.00 Paulista 47 37 78.7 50 40 80.0 50 40 80.00 When one examines the percentage of patients that receive DOT it becomes clear that it is actually much lower. Table 3 shows that the percentage of patients that receive DOT ranges from a low of 49.4% in Cabo de Santo Agostinho to a high of 84.2% in Camaragibe, with an average of 60.5% across the six municipalities in 2011. As such, “fifty percent or more of health facilities providing DOTS” does not necessarily translate into 50% or more of patients receiving directly observed treatment. Table 3: Smear Positive TB Cases Under DOT by Municipality - 2007 – 2011. Source: SINAN/ SEVS/SES – PE Municipality Year 2007 2008 2009 2010 2011 Recife 55.8 51.0 50.9 49.7 54.7 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 31 Cabo de Santo Agostinho 80.7 76.4 67.2 62.8 49.4 Camaragibe 89.6 84.2 76.7 55.9 84.2 Jaboatão dos Guararapes 23.3 24.7 30.8 30.5 46.6 Olinda 44.0 45.3 41.3 41.3 58.5 Paulista 62.0 56.3 71.4 69.3 69.4 Integrated TB-HIV services (100% of TB patients being tested for HIV) No data was made available to the Evaluation Team to evaluate whether this target has been achieved. São Paulo state USAID/Brazil supported PAHO’s technical assistance to the São Paulo State and Municipal TB coordinating offices from 2006 to 2009. This support sought to improve case management and laboratory capacity in the state. Interventions were planned to strengthen the State’s TB information system, TB Web, provide management training for TB program managers, train health care workers in DOTS, monitor and supervise DOT, as well as improve lab quality control through increased use of quality sputum smear microscopy and sputum cultures. Although the evaluation team was able to confirm that these activities occurred, it was not possible to determine if the target of training 4,000 health care workers was reached, or if fewer or more health workers were trained, mainly due to lack of access to project documentation. Documentation about other planned activities, such as examination of contacts and infection control measures were also unavailable to the Evaluation Team. Public resources from both the State and National TB program cover the majority of TB activities in the State. The State Health Secretariat was directly involved in project implementation. São Paulo benefited from a participatory work process, with integration between the state municipal programs, with highlight for the effective coordination between the State and the São Paulo city TB programs. The State Program’s commitment to intensive supervision provides essential support to municipal programs. According to government officials interviewed by the Evaluation Team, overall, USAID/Brazil support contributed to significant gains in TB control in São Paulo. DOTS training, along with sample transportation, logistic support and provision of internet access to TB datasheets contributed significantly to improving TB indicators. DOT was incorporated as an essential component of the TB program. From October 2006 to September 2007, an increase to 81% of the new smear positive cases on DOT, from 66.2% in the previous period, was reported. In the Figure below, a steady increase is observed in the percentage of TB patients on DOT from 2006 to 2011: from 49% to 83% in Carapicuiba, and from 41% to 78% in Guarulhos. 32 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 Figure 17: Percentage of TB Patients on DOT in Carapicuiba and Guarulhos, 2006- 2011 USAID/Brazil support to improve laboratory capacity in São Paulo by hiring contracting laboratory technicians contributed to improvements in smear microscopy, mycobacterium culture, and HIV testing. As shown in Figure XX below, from 2006 to 2009, smear microscopy increased from 60.9% to 93.6% in the municipality of Itaim Paulista. Particularly noteworthy is the remarkable increase in MTB culture during that period: from 4.4% to 73.8%. HIV testing among TB patients also showed important progress, increasing from 33.4% to 82.9 %. Figure 18: Percentage of Smear Microscopy, MTB Culture, and HIV Testing Among HIV Patients, Itaim Paulista, MSP 2006-2009 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 33 Cure rates among new cases were close to 80% in 2005 and 2006. Although still high at around 14%, default rates dropped considerably as compared to previous years. The Figure below shows treatment outcomes for DOT and DOT default moving in the wrong direction, but still within the target, at 86.6% and 7.3% respectively in Itaim Paulista. During site visits, all health professionals and TB managers interviewed mentioned that alcohol and drug users stand out among defaulters and tend experience negative treatment outcomes. Figure 19: Cure Rate and Default Under DOT, Itaim Paulista, 2007-2009 Limitations confronting the State TB Program include a reduction in training opportunities and delays in resource availability. High turnover of health professionals - mainly physicians presents an ongoing human resources challenge. Political commitment in peripheral municipalities is less consolidated. Vulnerable populations, such as the homeless and drug users, require collaborative efforts among different social institutions. This type of coordination, however, has not yet been implemented. Rio de Janeiro state In Rio de Janeiro state, USAID/Brazil funded PAHO interventions in: (1) development of DOTS implementation plans for 12 high-burden, priority municipalities; (2) training health professionals in the five elements of DOTS ; (3) increase detection of TB symptomatic cases ; (4) hiring 10 nurse supervisors to supervise, monitor, and evaluate implementation of DOTS in the 12 municipalities with the highest disease burden; (5) increasing use of sputum smear microscopy; (7) renting vehicles for supervision visits; (7) training staff in the use of the TB registry, treatment card and registration of symptomatic cases. The extent to which activities occurred is unclear, as some were not linked to targets that. This was the case, for instance, of the number of trainings actually delivered, which could not be cross-checked against any target. Political commitment for TB control varies across municipalities. The current State Health Secretary has committed to invest in TB. Collaboration between State and Municipal 34 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 levels has improved, particularly with the city of Rio de Janeiro. A Working Group for TB and AIDS Programs was created with civil society representatives to discuss an emergency plan. The initial proposition is a strategy seminar on TB and AIDS in the State of Rio de Janeiro in the first quarter of 2013. Although targets have not been met in all Mission-supported 12 municipalities, improvements in TB control are noted such as the information system, laboratories, logistics and the distribution of drugs. Staff capacity has increased with training in DOTS, as well as monitoring and supportive supervision through Mission support which was prioritized for high burden municipalities. DOTS as a control strategy has been heavily promoted and acknowledged in all intervention municipalities. In particular, DOT has not yet reached 100% of patients in these municipalities, but there has been change in a positive direction, as acknowledged by State TB authorities. The nurse supervisors hired with USAID/PAHO support fulfilled instrumental role supervision and monitoring of municipal actions, supporting and guiding towards improvement. This intervention was highly prized and appreciated by the State TB Coordination. Provision of DOT improved in all 12 municipalities. Whereas in 2005, only one of the municipalities provided DOT, all 12 municipalities provided DOT in 2010. Provision ranged from over 95% in four municipalities to less than 20% in another four municipalities. In 2009, in all but two of the 12 municipalities (Japeri and Queimados), case detection rate was 70% or better. As seen in the Table below, there was no clear trend of improvement of treatment rates in selected municipalities. Only three municipalities had a mean treatment success rate above 80% from 2006-2010, none reached the 85% target consistently. Likewise, only four of these municipalities had default rates lower than 10% during the same period. From 2005 to 2008, average default in the municipalities steadily decreased from 14.5% to 11.5%. Yet, in 2009, it increased again to 13.1%. Table 4: TB Treatment Success Rate (%) in 14 Priorities Municipalities in Rio de Janeiro 2006 2007 2008 2009 2010 Belford Roxo. Roxo 72 65 71 61 67 Duque de Caxias 74 69 69 65 67 Itaboraí 86 82 80 79 65 Itaguaí 89 85 77 84 86 Japeri 81 76 84 73 89 Magé 72 80 81 85 81 Mesquita 75 67 73 84 74 Nilópolis 53 63 66 60 47 Niterói 84 77 74 75 76 Nova Iguaçu 70 75 81 81 79 Queimados 81 74 68 73 70 Rio de Janeiro 68 62 61 67 61 S. Gonçalo 74 69 73 69 78 S. João de Meriti 67 66 72 62 62 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 35 Limitations confronting Rio de Janeiro include limited human resources, both in quantity and capacity. There is poor political commitment at some municipalities. Poor coverage of Family Health Program, with exception of Itaborai and Mage municipalities, (only 30-40% of TB patients) remains a concern. FINDINGS BY ACTIVITY DOTS Implementation Political commitment with increased and sustained funding As noted earlier, at the inception of USAID/Brazil’s TB program, the Brazilian government strongly opposed adoption of DOTS, including directly observed therapy and sputum smear microscopy for case detection. Concurrently, states had (and still have) a great deal of autonomy to implement their own health interventions. A multi-pronged approach was selected as the best method to encourage national adoption of DOTS. To influence policy, USAID and PAHO worked with federal authorities to promote adoption of the DOTS strategy. Programmatic interventions were implemented in selected high-burden states and municipalities as a means to encourage much needed change at the policy level. In 2004, when Brazil officially adopted DOTS as a national policy, standard operating procedures (SOPs), clinical manuals and guidelines were yet to be developed to guide implementation of the new national policy. In addition, staff training, mentoring and supervision of staff was still required to support the implementation of new procedures, including DOT, patient reporting and registration, and cohort analyses. USAID-supported implementation and expansion of DOTS training led to improved technical knowledge among participants, measured by pre- and post-test scores. All municipal TB coordinators interviewed by the Evaluation Team stressed the strategic contribution of the USAID and PAHO technical cadre to move the TB municipal agenda forward. Case detection through quality-assured bacteriology Laboratory networks were upgraded in USAID-supported areas. The decentralization of laboratory procedures was a key factor to strengthen the laboratory network in the cities supported by USAID/Brazil. In addition, USAID partnered with GFATM to implement lab quality control. The major contribution of USAID/Brazil to TB laboratory strengthening at the national level was implementation of lab quality systems that led to accreditation of the National Quality Assurance Lab according to ISO 17025, as well as improvements of state lab network. A new tool, called LABMOST, was developed in partnership with the National Quality Assurance Institute for implementation of quality systems for drug quality testing to help reference labs qualify for ISO accreditation. Consistent application of the tool led to accreditation of the National Quality Assurance Lab according to ISO 17025, and strengthening of the state lab network - 12 state labs now have the capacity to perform quality testing of TB, while none was in this position before implementation of USAID programs. The MOH now uses this model for other transmissible diseases’ programs, thus maximizing the impact of USAID’s investment. Mission support was also instrumental in increasing access to TB diagnosis through the reorganization of services, patient flow, and consultations in selected intervention areas. In Rio de Janeiro, USAID support improved TB diagnosis by providing vehicles that were used to collect patient samples and lab supplies. In São Paulo state, the number of sputum smear exams increased during the period of USAID support, when results were turned around in 24 hours. However, 36 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 after USAID support for laboratory personnel ended, a number of project-hired lab technicians left. The remaining staff had to deal with an increased demand for exams. Consequently, the time for results for sputum smear exams increased to 48 hours. Labs helped to increase diagnosis, but data was not available about overall effects of the program over a significant period of time. USAID strengthened the laboratory network and increased its capacity to diagnose and control TB through training programs coordinated by the states’ central laboratory network (LACENs) in target areas. In addition, USAID provided logistics and technical support in Manaus, Rio de Janeiro and São Paulo. Still, the quality of labs is not uniform countrywide. The level of performance of LACENs is lower than expected in some states that received USAID assistance, such as Rio de Janeiro, where culture and DST patients are subject to long waits. Despite significant progress over the past 10 years, the laboratory network remains as one of the weakest elements of TB control in Brazil at all federative levels. Standardized treatment with supervision, and patient support USAID support for DOTS expansion contributed to increased DOT coverage through DOTS training, as well as monitoring and supervision. Numerous communication campaigns reached out to hundreds of thousands, if not millions, in target sites, including prisoners, through radio, TV and virtual media. Despite the positive results obtained with DOTS expansion, the percentage of patients that successfully complete TB treatment is still below the target of 85%. In addition, treatment default rates are still unacceptably high, at over 10% in many urban areas. In fact, the mean default rate between 2006 and 2010 in the 14 high TB burden municipalities in Rio de Janeiro was higher than 10% in 10, or 71% of them reaching rates as high as 18% in São João de Meriti and 19% in Duque de Caxias. In fact, the mean default rate between 2006 and 2010 among the 14 high TB burden DOT is not uniformly implemented in either USAID intervention sites or throughout the country. DOT continues to be part of the national TB control strategy. With that, the best chance for sustainability of quality DOT for all patients is full integration into the FHP, with CHA providing treatment supervision and patient support. The poor coverage of the Family Health Program high￾burden Rio de Janeiro municipalities negatively impacted implementation of DOTS, resulting in only 30 to 40% of TB patients having access to DOT. Effective drug supply and management system USAID/Brazil support was instrumental in improving management and drug quality control at the national level. The NTP acknowledges the value of USAID support to the TB drug forecasting, management and distribution system. Technical assistance to CRPHF was particularly important to improve the management and delivery of second line drugs for treatment of MDR/XDR TB cases. USAID-support contributed to the improvement of drug management, implementation of drug quality control systems, and the development of SITE TB/E-TB Manager - an information system that was created in Brazil and expanded to more than 10 countries worldwide, including countries Eastern Europe and Asia. Mission support strongly encouraged the creation of a quality control board for TB drugs and provided necessary funding. Currently, the Gates Foundation is supporting this effort with logistics assistance. USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 37 Mission support was crucial in promoting discussion and encouraging the addition of Ethambutol in the NTP treatment regimen, in accordance with international standards. Local production of quality 4-in-1 and 2-in-1 fixed-dose-combination (FDC) drugs was a direct result of technical support provided with USAID financing. USAID supported collaboration with the Brazilian government’s medicine manufacturer, Farmanguinhos, to develop s 4-in-1 (RHZE) and 2-in- 1 (RH) tablets, in addition to , Isoniazid 300 mg tablets for latent TB infection (LTBI). Drug supply and management is likely to continue at high quality levels, given the GOB’s political commitment and CRPHF’s and Farmaguinhos’ increased capacity, as well as monitoring by the quality control board. Potential production for national and international markets is in discussion, thus positioning Brazil as a potential supplier for the PAHO Revolving Fund. Monitoring and evaluation systems and impact measurement USAID support was crucial to foster understanding of the importance of DOT monitoring and supervision. Through USAID assistance, Brazil developed a national model of monitoring and supportive supervision through the valuable work of the DOTS nurses recruited by PAHO in intervention sites. The creation of the nurse training and supervision team was instrumental to expand DOTS in priority municipalities. A similar approach was utilized in 12 priority municipalities in Rio de Janeiro to reinforce the importance of monitoring and supervision. The creation of the GFATM Monitoring and Evaluation group was also a product of this example. USAID/Brazil funding and technical assistance helped to strengthen monitoring and evaluation systems through investment in web-based TB database such as TB Web in São Paulo and DMIS at CRPHF, in Rio de Janeiro and countrywide. USAID support allowed the GOB to contract capable consultants and to offer training courses for effective use of those. USAID/Brazil’s financial and technical support was particularly effective in providing a cadre of temporary health professionals for efficient execution of strategic tasks, such as monitoring and supervision. USAID also provided support to revise SINAN, which was launched in 1994 as the national health surveillance system. Today, 54 diseases are reported in SINAN. Currently, the system requires too many variables, including some for which no value is required. This results in lots of data simply not being reported. The system includes 89 variables at the federal level but will undergo a radical revision in order to limit variables to a maximum of 36 federal, 44 state, and 52 variables at the municipal level. Adoption of USAID recommendations to improve the performance of SINAN resulted in an increase of information on treatment results, from 33% to 50%, from 2002 to 2003) “USAID's investment enabled an increase in the number of training and supervision visits from the TB programs; the involvement of those institutions made health professionals feel that they are not alone driving the actions of the program.”   TB Nurse 38 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 Vulnerable populations USAID/Brazil supported communication campaigns for TB patients and their families in general, as well as targeted campaigns for vulnerable populations, such as prisoners, people living with HIV/AIDS (PLWHA), TB patients, men who have sex with men (MSM), lesbian, gay, bisexual, transvestite and transsexual (LGBT) individuals, and sex workers. It was not possible to determine whether TB case notification and treatment outcomes improved among these groups after implementation of these campaigns. Although drug and alcohol use has become more frequent among TB patients, while these groups tend to report unfavorable TB treatment outcomes, USAID/Brazil’s TB projects did not report specific interventions for these particular groups. Partnerships In 2004, TB projects were considerably segmented, with no adequate coordination among different partners. USAID played no specific coordination role at that time. Reportedly, partners learned from each other in occasional meetings with the NTP in Brasilia. Over the last 5 years, though, greater synergy and integration has been observed, according to partners interviewed by the Evaluation Team. Formally and informally, USAID/Brazil has contributed to better coordination through regular meetings, sharing of professional contacts, joint work plans, and dissemination of preliminary findings and final reports, as well as community grants competition and invitations for training. USAID support helped establish partnerships between private and public laboratories and several international organizations. These included the MOH, CRPHF, the Brazilian TB Network, FIND, the BMGF, the Union, CRPHF, Beckton-Dickinson and BioMérieux, in addition to several other international laboratories. These partnerships have enabled the introduction of new rapid tests for TB using molecular biology, e.g., GeneXpert and HainlifeScience, both of which have been recommended by the WHO. Partnerships with GFATM and the Gates Foundation (represented in country by Global Health Strategies) were synergistic and involved support to DOTS expansion, quality diagnosis, increased civil society involvement, social mobilization, and behavior change communications. Moreover, USAID and GFATM were instrumental in supporting the inclusion of TB in the national policy agenda by advocating with stakeholders and providers. Reportedly, the Gates Foundation and GFATM work plans were also aligned with NTP priorities, in coordination with USAID-supported interventions, to address national gaps and strengthen existing initiatives. The NTP is open to collaboration and works in an inclusive manner, inviting interested stakeholders to actively participate in TB control. Partnerships are likely to expand in the coming years, as the NTP is making a conscious effort to increase advocacy for enhanced TB control. Programmatic Management of Drug Resistant TB (PMDT) USAID support improved laboratory and information systems through direct coordination and technical assistance to CRPHF. USAID/Brazil-supported technical assistance contributed to structural and operational improvements at CRPHF, which serves as the national reference laboratory for TB. USAID/Brazil support helped CRPHF better perform as the key lung disease reference for the national laboratory network, with the implementation of an external quality assessment program for sputum smear microscopy in 27 state labs, thus covering the entire Brazilian territory. This resulted in 12 out of the 27 public laboratories now being regularly supervised for microscopy and DST. As a result of these achievements, CRPHF’s accreditation as USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 39 the national TB reference laboratory was renewed in 2009. Additionally, USAID has provided technical assistance for CRPHF to obtain ISO 17025 accreditation, which is the single most important standard for calibration and testing laboratories around the world. Laboratories that are accredited to this international standard have demonstrated that they are technically competent and able to produce precise and accurate test and/or calibration data. The MDR information system (DMIS) was established in 2004 with USAID support. Since then, it has been significantly improved and renamed from TB MR to E-TB Manager (globally) and SITE TB (nationally). DMIS has been deployed and used in all MDR-TB centers over the country and is now the MOH’s official system for MDR-TB management. The system includes information on every type of TB drug resistance, in addition to special cases, such as hepatic-associated diseases which require alternative treatment schemes. Besides tracking clinical and epidemiological information about patients, DMIS manages and controls forecast and distribution of second line drugs through the CRPHF pharmacy. It also monitors non-tuberculosis mycobacteria, which is innovative in Brazil and improves the country’s surveillance capacity This system contributed to increasing detection and cure rates of MDR-TB cases, and significantly strengthened clinical and case management practices in MDR-TB reference centers. The DMIS implementation has also enriched the quality of information about TB patients, offering a comprehensive database for further research and scientific publications. Figure 18 below records increases in TB notification rates from 2000 to 2011. Figure 19 portraits continuous improvement in treatment outcomes for drug resistant cases through the years. DMIS is now managed by the health surveillance department at the National School of Public Health at the Oswaldo Cruz Foundation (ESPN/Fiocruz). Technical and managerial steps have been taken to ensure fully functional transfer of the system to Fiocruz and sustainability of the system as USAID support ends. It is important to highlight that Brazil has its own National Disease Notification System (SINAN), which includes information about all compulsory notification diseases, including TB, but no information on MDR TB. Currently, no automatic integration is observed between SINAN and DMIS. However, it is possible to manually check whether a TB case has been notified in both SINAN and DMIS based on the unique patient reference number. The NTP has adopted a decentralized approach for treatment MDR TB treatment. Better geographic coverage of MDR-TB centers has been achieved, with an increase from 63 MDR-TB Reference Centers registered in 2004 to 138 MDR-TB centers and treatment units in 2012. Ad￾hoc training was offered countrywide to build capacity in system procedures and flows, and to review the MDR-TB medical charts in order to update patient information in the DMIS database. However, MDR-TB management has not yet been fully decentralized in some states, e.g. Minas Gerais. 40 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 Figure 20: Impact 1, Increased Number of Notified Cases USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 41 Figure 21: Impact 3, Treatment Outcome Improved The Mission provided support for the validation of GeneXpert by providing technical assistance to the development of the study protocol. GeneXpert is a new, cartridge-based, automated diagnostic test that can simultaneously identify and detect Mycobacterium tuberculosis (MTB) and Rifampicin resistance, a marker for MDR TB. Results are available in only three hours. This represents an important advance in TB and MDR TB diagnosis, as results of tests currently available for drug resistance only can take more than 4 weeks, leading to higher mortality and further spread of MDR strains. National funds will be used to purchase 100 GeneXpert machines and 500,000 tests. Labs without GeneXpert will reportedly have liquid culture. Next year, Brazil will be the second country in the world to utilize GeneXpert machines. Gender In general, male and female clients have benefited equally from TB interventions in Brazil, according to national, state and municipal authorities interviewed. However, women tend to seek care more frequently than men despite equal access to health care. Health care providers interviewed at the national, state and municipal levels did not consider gender to be an issue for TB control. It is also worth noting that gender considerations have not been fully integrated into USAID/Brazil’s TB program and TB-related activities, although USAID has specific policies in this area. It is not possible to determine the different effects of the project on male and female beneficiaries, as data is not generally disaggregated by gender. Implementing partners reported that USAID/Brazil did not require gender-specific data in project design and work plan development. In addition, sex-disaggregated data for individual level indicators was inconsistently reported. 42 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 Advocacy, Communications, Social Mobilization (ACSM) The impact of ACSM project activities in improving main TB indicators is difficult to evaluate. During the period of project development other implementers (Global Fund, PAHO) also contributed to training in this area, making it difficult to attribute achievements to a particular intervention. The NTP does implement its own advocacy, communication and social mobilization campaigns. The USAID-funded BCC project, though, represented an innovative and strategic social marketing approach for TB control, with proven successfully methodologies and tools. The rationale for the intervention was based on the persistence of high default rates in São Paulo State, despite the reduction in incidence and mortality rates reported between 2003 and 2007. DOTS expansion in São Paulo State faced barriers to reach particularly vulnerable populations and high-risk groups, such as prisoners. The “De Peito Aberto” campaign, developed with USAID support, counted on input and participation of civil society, the São Paulo state and municipality TB program managers, and representatives from the of the Mayor’s communications office, the State Penitentiary System, and health providers. This participatory methodology made it possible to share ownership of the campaign with people most impacted by the disease, as well as by public health providers. The availability of indicators from intervention municipalities and districts involved in the BCC program is not uniform. A trend for increase in case detection and treatment completion rates applicable to all areas under study could not be verified. The default rate of TB treatment was not clearly reported among selected indicators. Aiming to increase the level of TB treatment knowledge, attitudes, and behaviors/practices (KAPB) of TB patients in prisons USAID supported an intervention in the Franco da Rocha Penitentiary in 2011-2012. The strategy was developed in two of the three Franco da Rocha units (security levels one – P1 - and two –P2) Security level three (P3) was used as a control site. In P1, significant increase in TB treatment knowledge, adherence, and social support was observed in the interventions before and after analysis. . Impact was lower in P2, though still significant when compared to the control unit (Table 5). Table 5: Results of KAPB intervention at Franco da Rocha Penitentiary (2011-2012) Prison Unit (Prisoners affected by TB) Analysis Model 1 (ex-ante/ex-post per unit) Analysis Model 2 (ex-ante - control Unit - P3) Analysis Model 2 ( ex-post - control – Unit - P3) Unit P1 5,5 (p-value-p<0,05)* 4,9 (p-value>0,05) 9,9 (p-value<0,001)* Unit P2 1,9 (p-value-p>0,05) 7,25 (p-value<0,05)* 9,3 (p-value<0,01)* Unit P3 - 0,15 (p-value-p>0,05) Since the 1980s, Brazilian civil society has been active and strong in AIDS control. By the early 2000s, there were over 1,000 community-based organizations (CBOs) throughout the country working on HIV/AIDS issues, but none in TB. In collaboration with the State of Rio de Janeiro, USAID organized the Rio NGO TB Forum, which is still very active today with core funds and includes about 100 organizations. In addition, USAID supported educational campaigns organized by local NGOs such as Rede Paulista and Espaço de Prevenção e Atenção Humanizada (EPAH). Campaigns focused on HIV counseling and testing with emphasis on HIV/TB co-infection among most-at-risk populations (MARPS), increased prison inmates’ adherence to TB treatment by changing the KABP of patients in prisons and their families at the Franco da Rocha Penitentiary in São Paulo. Another local NGO, Transformarte, focused on community outreach through youth mobilization. In a small area in the Rio de Janeiro slum, Transformarte’s work helped to increase cure rates from 70% to 90% and decrease default from 22% to 6%, according to a health officer USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 43 from the Municipal Health Secretary. Transformarte’s Project received a prize at the 6th National Exhibit of Successful (Expoepi), sponsored by the Ministry of Health. A lasting legacy includes the Rio de Janeiro State NGO TB Forum, set up with USAID support, which continues to operate until today. The Forum was created to address an expressed need of community organizations to more efficiently discuss ways to influence public policies and to monitor the use of public resources destined to TB programs. Today, the Forum holds monthly meetings that bring together an average of 70 NGOs to promote TB advocacy and engagement. An example of effective action is the Forum’s relevant participation in planning and organizing the annual commemoration of the World TB Day. The Forum attracts community leaders, and health professionals and managers. STOP TB Brazil is another forum that helps to increase social mobilization and to bring stakeholders together. The National TB Steering Committee is an advisory body to the NTP that approves TB-related policies and practices. The Committee includes civil society representation among its 19 members, one of which being USAID’s partner Management Sciences for Health. In the past, civil society organizations received funding and support from a variety of sources, including USAID, GFATM and donors working in HIV/AIDS. Currently, however, they are faced with severe lack of funding, particularly due to the international financial crisis. For this reason, funds to support TB NGOs are no longer available. While some NGOS receive limited support for TB activities from state or municipal authorities, the sustainability of these efforts is uncertain. Infection control Infection control, and especially administrative control, was part of USAID/Brazil’s general support to local efforts to control TB. Infection control measures were included in DOTS training and TB/HIV collaborative activities, among other types of support. Simple measures, such as ventilation of physician’s offices, and availability of an area outside the health units where patients can safely produce sputum were adopted by health units in intervention sites. Nevertheless, infection control measures still require enhancement, particularly in the area of personal protection. Reportedly, some health care workers resist working in TB for fear of infection. Staff receives hazard pay for working in the TB program, but that does not seem to be enough to compensate for the risk of exposure and infection. Staff in some health units visited by the Evaluation Team were observed wearing respirators incorrectly. Staff interviewed expressed the desire to acquire additional knowledge on personal protection measures. TB/HIV collaborative activities The Brazilian TB and HIV/AIDS programs have long been vertical, with little programmatic collaboration or coordination. Mission support to improve collaboration started with operational research to determine the percentage of TB patients who were tested for HIV. Support evolved “What USAID did was fundamental to see changes in TB in the country…We did a revolution, here in this country. There was absolutely nothing…no social awareness…we started from scratch.” Long‐time AIDS and TB Activist 44 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 to include programmatic interventions such as the incorporation of screening for TB among HIV patients in HIV specialized care centers (SAEs). Mission support led to the development of a national TB/HIV plan. In the Southern region, where TB/HIV co-infection rates are particularly high, USAID support allowed for the development of guidelines for TB/HIV care at SAE level. The absolute number of purified protein derivative (PPD) and latent TB infection (LTBI) tests increased in general, while TB diagnosis in HIV- specialized care centers improved in selected intervention sites. For example, in São Leopoldo, the absolute number of PPDs doubled from 2010 to 2011, and a 30% increase was reported in the first semester of 2012. The number of LTBI also increased two times from 2011 to mid-2012 in this setting. In Porto Alegre, similar results were found in terms of PPD comparing between 2010 and mid-2012. Complete information about PPD and LTBI was available, thus facilitating full analysis of this activity. USAID supported the development of an educational campaign based on the “3Is for tuberculosis” (Intensified TB case finding, Isoniazid preventive therapy, and Infection control for TB) (Figure 23) and produced educational materials (video, folders, posters, t-shirts, buttons, stickers, educational booklets, and bus posters) were created in partnership with PLWHA and community leaders in order to engage civil society and strengthen advocacy, communication and social mobilization Figure 22: Educational Campaign “Essa dupla nao combina” (TB and HIV - A Mismatch Couple). The MOH’s TB/HIV Working Group was officially created as a result of USAID-promoted coordination, representing a step forward in the integration of the TB and HIV programs at the national level. Another successful result of the TB/HIV integrated approach was the inclusion of TB/HIV co-infection in HIV/AIDS and TB national guidelines, and the inclusion of TB drugs into “We knew that we had to do something about TB/HIV but we did not know how. The project was important to show us how to do it"   State TB Coordinator USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 45 the HIV/AIDS drug management system (SICLON). In addition, TB drugs for TB treatment became available at SAEs. A TB clinical screening tool was developed and implemented at SAEs to ensure that TB suspects are diagnosed in HIV specialized care facilities (Figure 24). Figure 23: Stamp Questionnaire for TB Screening at SAE Level. The number of PPD and IPT increased in most of the selected facilities. Prior to 2011, very few PPD and IPT were provided in SAEs. The number of PPD tests provided in SAEs more than doubled from 2009 to 2012, increasing from 101 to 242. A similar trend was observed in Porto Alegre (Figure 25). 46 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 Figure 24: Number of PPD, PPD≥5mm and IPT Provided at IAPI, Porto Alegre, 2008 to mid-2012. Note: Total number of HIV patients each year was not provided. The current NTP manager previously worked in the HIV/AIDS national program in the past and is committed to furthering TB/HIV integration efforts. This bodes well for continued collaboration. The TB/HIV working group will continue with support from the MOH. Formal recommendations to integrate TB and HIV activities will be issued for all Brazilian states, based on the model adopted in Southern states. Besides this, campaign materials will be disseminated to all Brazilian states. Successes will soon be published. While much has been accomplished with USAID/Brazil support, a lot remains to be achieved, especially at the state level, to improve commitment to effective integration of TB and HIV services. Although Project support was instrumental in promoting dialogue and collaboration between the two technical areas at the national, state and municipal levels, its implementation was extremely short. There was insufficient time to observe major progress and results. Lack of political commitment and support at the state level constrained the integration of TB and HIV/AIDS programs. In the absence of this support, municipal leadership became the deciding factor for program integration. Operational Research USAID/Brazil directly supported operational research and trained staff on how to conduct operational research. Results of USAID-funded operational research were published in scientific “The main contribution of the USAID‐supported TB/HIV project was to help health care professionals, and especially infectologists to understand the importance of integrating TB and HIV care.”   State TB Coordinator USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 47 journals, including such topics as TB/HIV and DOTS expansion. Selected articles are listed in the references section of this report. USAID sought to provide scientific evidence to support policy and programmatic change, in accordance with international standards. As some stakeholders were highly resistant to implementation of DOTS s, unbiased, scientifically-sound and robust research findings were warranted. As previously mentioned, USAID supported the Second National DRS, which showed increased resistance to Isoniazid, from 4.4% to 6.0%. These findings highly influenced adoption of Ethambutol in the first line regimen. This brought Brazil in line with international standards and helped to improve treatment outcomes and prevent the emergence of drug resistant cases. Among all the activities undertaken, the DRS was undoubtedly the one that required more complex planning, personnel training, standardization of procedures and financial resources was the DRS. The results of DRS will be enormously important to define the policy for diagnosis and management of resistant TB cases in the country. The activities implemented through the DRS, such as training of technicians from local and state laboratories involved in the study, hiring of lab staff to support state laboratories and supervisory visits of WHO consultants, helped to improve the quality of work of TB laboratories in different parts of the country. Planning and implementation of a National TB/HIV co-infection survey The study of the prevalence of HIV infection in TB patients was conducted concomitantly with the DRS was also supported by TBCTA. The aim of the study was to determine the prevalence of HIV infection among TB patients and its association with resistance to anti-TB drugs. TB patients enrolled in the DRS were offered HIV testing. HCW taking part in the study was trained to offer pre and post counseling for HIV testing. The inclusion of TB-HIV co-infection surveillance into the DRS was supposed to strength the relationship between the TB and HIV/AIDS National programs. In the last CDC report of September 2006 no information was available regarding the results of this study. Evaluation of the current epidemiological situation of TB in Brazil Data collected through the SINAN database and the cohort study allowed for analysis of the epidemiological scenario of TB in Brazil between 2001 and 2003. The results of this study were presented at the Union World TB Conference in October 2003. National cohort study performed at the State level, using State-level data This study had the objective to analyze the results of TB treatment in a cohort of TB patients from the whole country who initiated TB treatment in 2001. The study was conducted in March￾April, 2004. Data collection and entry was completed in late l. Preliminary data showed low treatment completion rate of 62% and high rates of treatment default at 13%. Deficiencies in the registry, also known as the black book were identified. These findings helped the NTP to change the black book accordingly. Evaluation of the impact of antiretroviral (ARV) treatment on the incidence of TB in Brazil 48 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 The study showed that the use of highly active antiretroviral therapy (HAART) by HIV patients was associated with a significant reduction in the risk of TB disease; an 80% reduction in incident TB was observed in HIV-infected patients treated with HAART as compared to ART patients. A manuscript with results of this study was prepared and published in a scientific journal in 2007 (Miranda et al., 2007). CONCLUSIONS AND LESSONS LEARNED Below is a list of the lessons learned, based on the findings of the performance evaluation of the USAID/Brazil TB portfolio, 2001-2012:  Political commitment is extremely important, if not the most important factor for sustainability of results. Even if financial resources are limited, prioritizing TB as a public health concern in states and municipalities fosters public and private sector support to TB control efforts.  Integration of TB activities into the Family Health Program (FHP) with adequate training, monitoring, and supervision, helps to increase access to quality Directly Observed Therapy Short-Course (DOTS) services, improve treatment outcomes and reduce default.  Implementation of TB projects in a collaborative, participatory manner encourages stakeholders to express their positions, contributes to work planning and promotes buy-in and ownership of projects and their results. Stakeholders are more likely to take shared responsibility for reaching targets when planning, management and implementation are participatory and collaborative.  High turnover of health sector staff is a reality in many countries with high TB burden. Brazil is no exception. Therefore, a comprehensive human resource development and capacity building plan is necessary, in addition to creation of monetary and non-monetary incentives for staff to remain engaged in TB programs.  The availability of laboratory support for TB diagnosis is still limited in many cities, especially regarding mycobacterial culture and Drug Susceptibility Tests (DST). Issues relate not only to the shortage of professionals and laboratories, but also to the gaps in organization of labor that cause delays in results availability. Improvement of computerized systems for laboratory results management, coupled with integration of new diagnostic methods (e.g. GeneXpert) into the National TB Program represent good prospects for the future.  Supportive supervision is critical to ensure professional development, and to motivate and incentivize staff. It is as important, if not more so, than training. Staff morale and performance are improved with ownership of, and connection to, a larger program. Accountability and recognition of good performance encourage ownership of program outputs and outcomes.  Long-term TB technical assistance is most effective when implemented by experts and technicians who have had prior working experience in the host country. This ensures in-depth understanding of economic, social and cultural issues that may influence the success of a project, and builds upon established personal and professional networks to facilitate project implementation and identification of potential pitfalls and obstacles that may not be apparent to novices.  One-year, or otherwise short-term projects, are insufficient to achieve sustainable results in TB control. Given the complex bureaucracy at all levels of government, projects that span for one year only are at a disadvantage, as they may be negatively impacted by the required start up negotiations. Collaborative, participatory work requires sufficient time to ensure stakeholder engagement and buy-in. USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 49  Projects limited to one geographic site may create professional conflict and disruption among TB staff in non-intervention, neighboring areas. Geographically-comprehensive projects are more likely to produce sustainable results.  Delays in obligation and transfer of funds may not only cause technical delays in TB project implementation, but also contribute to poor morale of stakeholders. Local USAID partners were frequently unaware of USAID’s complex requirements and procedures for budget approval and obligation of funds. This created misunderstandings and raised disagreements between the parties in some projects.  Integrating DOTS into the primary health care network by partnering with the Family Health Program (FHP) is fundamental to improve performance. Supporting health teams with management systems and tools will increase success. RECOMMENDATIONS Below is a list of recommendations based on the findings of this performance evaluation and according to each audience. To USAID/Brazil  Strictly monitor projects in a timely manner to allow for continuous evaluation and eventually redirection of objectives and/or activities. Select appropriate indicators to measure project results. Baseline data should be required in final project proposals. Encourage the use of specific tools to evaluate the acquisition of skills and behavioral change, so as to best evaluate the impact of health professionals’ training. Require partners to develop specific, planned outputs that include targets and expected completion dates.  Where possible, address potential funding delays and create mechanisms to prevent interruption of project activities.  Prioritize medium to long term projects, e.g., a minimum of two years, and require well defined annual work plans.  Extend investments in operational research beyond target municipality or district limits, to include an expanded geographic focus. Explore opportunities for neighboring municipalities to benefit from research findings.  Maintain supporting documentation in official files relating to reported performance data, including approved work plans, and annual and final reports.  Promote cross-fertilization among different implementing partners (IP), and states and municipalities, fostering exchange/sharing of experiences and best practices, and enhancing communication between IPs and health professionals.  Encourage partners to work closely with government officials in order to support work flow and systems, rather than creating parallel systems that do not build local capacity;  Convene a meeting with the main IPs and the GOB to formally close out USAID assistance, share lessons learned and best practices, and provide recommendations to the GOB and other stakeholders. To National/State TB Programs  Pursue a partnership between the Ministry of Health (MOH) and the Ministry of Social Development (MOS) to increase coordination at the policy level, including the continuation of incentives such as the Basic Food Basket to promote TB treatment adherence, reduce default and increase treatment completion rates. Mainstream TB in GOB poverty alleviation programs to address the needs of most vulnerable populations. Include specific actions for TB 50 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 patients with a record of drug (including crack and cocaine) and alcohol abuse, as well as those with mental health problems.  Strengthen the partnership with the Ministry of Justice’s National Justice Secretariat to design and implement a comprehensive strategy for TB control in the penitentiary system. TB control efforts in prisons are currently limited. However, experience – including that of programs supported by USAID - demonstrates that education of prison staff, inmates and their families, along with appropriate screening and treatment, can produce highly successful prevention and treatment outcomes.  Involvement of TB program managers at all levels (from the National TB Program – NTP - to TB programs in peripheral municipalities), and participation of representatives of other programs involved in the diagnosis and treatment of TB patients (FHP, MOS, health coordination of the penitentiary system, and state laboratories (LACENs) are critical to project success.  Increase public-private partnerships (PPPs) with both national and international partners. Given limited public health resources, partnerships can help strategic leverage of available resources and provide critical technical assistance in areas such as program management and operations research.  Provide continuous training to state and municipal staff in the areas of strategic planning, annual work planning, and social mobilization. These skills are crucial to ensure the establishment of realistic goals and implementation of sustainable activities. Management skills varied greatly from state to state, and from municipality to municipality. Continued capacity building in this area is needed for program success, especially at the local level.  Provide additional technical support at the state level, including state-to-state exchange of ideas and experiences. The USAID-supported quality DOTS program in selected sites had limited spillover in non-intervention municipalities. Through concerted efforts, develop a plan for information sharing and provide fora for state officials to actively share lessons learned, best practices and challenges. Underperforming states should be provided with technical assistance from both the NTP as well as from high-achieving States, as feasible.  Develop a comprehensive human resources plan for the health sector, including TB. The major constraint found in most USAID-supported states was high turnover of HCW, which requires constant re-training. High turnover is a concern for sustainability of program results at both the provider and the management level. Allocate more resources to ensure maintenance of a continuous supervision plan at all levels (national, state, municipality).  Prioritize support to vulnerable populations aimed at reducing default and ensuring treatment completion. This support requires interaction with different institutions and development of specific and complex professional skills. In addition, patient incentives such as food baskets, transportation vouchers, cash transfers, and engagement in income generation initiatives, in addition to special activities for cases of compulsory hospitalization.  Fully integrate DOTS in the FHP and CHA programs. Integration of DOTS into these primary health care programs addresses the needs of poor and vulnerable TB patient’s right at the communities where they live and work, thus contributing to better treatment outcomes and sustainability of activities.  Maintain the level of national funds for TB control, despite possible additional support from partners and donor agencies.  Continue to support and promote investments in alternative approaches for TB diagnosis, including availability of rapid tests such as GeneXpert that may facilitate the identification and treatment of drug resistant TB cases. USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 51  Foster increased operational research on TB. With discontinuation of USAID funding, a key position for a medical epidemiologist in the NTP became vacant. As a result, operations research in now decentralized throughout individual departments. A comprehensive operations research plan should be developed by the NTP outlining timelines and responsibilities for each department, with adequate resources to carry out activities.  Continue to strengthen laboratory capacity countrywide, especially at the State level. The laboratory network is still weak, with resources and performance varying considerably from one location to another. A comprehensive laboratory network strengthening plan should be developed with expert technical assistance, in collaboration with Centro de Referência Professor Hélio Fraga (CPHRF).  Continue to improve biosafety measures, especially administrative controls and personal respiratory protection. Staff interviewed consistently reported interest in acquiring additional knowledge on personal respiratory protection to protect against possible infection. Improving biosafety may also help to maintain and recruit staff to the TB program. For TB/HIV Collaborative Activities  Scale-up HIV rapid testing for HIV diagnosis among TB patients;  Scale-up TB screening among HIV patients;  Increase awareness among health care workers (HCW) to prescribe Isoniazid preventive therapy (IPT) for eligible HIV patients;  Improve access to antiretroviral treatment (ART) for TB patients;  Expand TB treatment in HIV Specialized Care Centers (SAE);  Reinforce infection control at all health units;  Create and implement a TB/HIV information system; For Future Evaluations  Allocate time for desk review prior to the team’s arrival in country, in order to optimize site visits, interviews, and data collection. Draft evaluation tools prior to the evaluation team’s arrival in country. Specify the format for the final written report in advance, with provision of examples where available.  Share project documents with the evaluation team using a shared drive, organized in labeled folders, with clear orientation about the documents.  Global Health, the Regional Bureau and the Mission should provide orientation, including historical perspective and key issues, to the Evaluation Team prior to site visits.  Finalize an agenda for the team on the day of arrival, with all meetings and interviews programed.  Focus on the duration of the evaluation and evaluation questions. For an evaluation covering a period of over five years, it is advisable to evaluate overall program performance rather than details of individual projects. Given the long period covered by this evaluation, there was limited access to documentation and sources of information, including staff, for projects that closed more than five years ago.  Adequate and timely logistical support should be provided to the evaluation team by the contractor hired to carry out the evaluation. Such critical support includes arranging domestic flights, hotels and ground transportation, as appropriate. This will enable the team to focus on technical matters of the evaluation. 52 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 For Global Health Bureau (USAID/Washington)  Continue to provide technical support to USAID TB programs. The technical support provided by the Global Health Bureau to the Brazil Mission during in-country monitoring and evaluation visits, as well as virtual assistance to review TB strategies and other relevant planning, was very valuable and helped the Brazil Health Team make important decisions regarding programming and strategy. For Implementing Partners  Provide better communication and transparency in the management of resources, which can facilitate collaboration with national/state/municipal programs on future projects. To PAHO  Continue to work closely with the NTP and support its activities and ensure quality implementation of DOTS at the primary care level through the Family Health and Community Health Agents’ programs. To Civil Society  Promote operational research to evaluate the impact of civil society activities. In some cases, it is not possible to use quantitative indicators to evaluate social change. So, other methodologies should be applied to validate those findings, such as qualitative methods approach.  Provide additional support to marginalized groups (e.g. alcohol and drug users) in partnership with the Technical Advisory Group to identify their profile and integrate TB care for mental health patients.  Work closely with the NTP to provide social support to those more likely to default treatment. Consider other sources of funding for TB activities, such as public-private partnerships. USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 53 ANNEX A. SCOPE OF WORK GLOBAL HEALTH TECHNICAL ASSISTANCE BRIDGE II PROJECT GH Tech Contract No. AID-OAA-C-12-00027 SCOPE OF WORK Oct 9, 2012 I. TITLE: USAID/Brazil: Performance Evaluation of Tuberculosis Portfolio 2001-2012 II. CONTRACT: Global Health Technical Assistance Bridge II Project (GH Tech) III. PURPOSE OF EVALUATION The U.S. Agency for International Development in Brazil (USAID/Brazil) seeks to conduct a comprehensive external performance evaluation of USAID-supported Tuberculosis (TB) programs and implementing partners. The focus of this performance evaluation is to compile and analyze results achieved through USAID/Brazil investments between 2001 and 2012 and to measure the effectiveness of activities and operations of the Mission’s TB projects during this period of time. The evaluation will examine the long-term outcomes (direct and indirect), identify strengths and weaknesses of TB activities that received USAID support since 2001, and look at the sustainability of these efforts over time. The evaluation results will capture the accomplishments gained from USAID’s investment in TB activities in Brazil and gather lessons learned from the past decade. In addition, the evaluation results will serve to inform recommendations that will be provided to the Government of Brazil (GOB), other donors and stakeholders as USAID funding for TB in Brazil will be discontinued in September 2012, and all activities will terminate by mid-2013. USAID TB funding is ending earlier than previously expected, so it is essential to determine the sustainability of TB prevention, detection, treatment and cure in Brazil. It is also important to capture USAID’s legacy in TB in Brazil and to assess the sustainability of the change that the Agency affected. While it will be important to understand the effectiveness of individual activities, the main goal of this evaluation is to assess the overall success of the principal aspects of the program: policy change, DOTS adaptation, and project sustainability. The Statement of Work is for an external performance evaluation of the implementation of USAID/Brazil’s TB projects. USAID/Brazil expects the evaluation to begin in early October 2012 until December 21, 2012. The Contractor is required to form an evaluation team to assess the overall results of the USAID-funded program, including TB integration into the primary health care network, combined approaches to TB/HIV co-infection, capacity building across the spectrum of health care professionals involved in service delivery, health systems strengthening, and promotion of effective dialogue between the three spheres of government - leading to greater positive health outcomes within Brazil than the individual projects alone could have accomplished. 54 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 The primary audiences for this evaluation include: 1) USAID/Brazil; 2) The Latin America and Caribbean (LAC) and Global Health (GH) Bureaus; 3) USAID/Brazil implementing partners; 4) the Government of Brazil; and 5) other donors and stakeholders working in TB. The Contractor is required to ensure that the evaluation is compliant with USAID’s Evaluation Policy (available at www.usaid.gov/evaluation). Under the Agency’s current Evaluation Policy, performance evaluations are to “… focus on descriptive and normative questions: what a particular project of program has achieved (either at an intermediate point in execution or at the conclusion of an implementation period); how it is being implemented; how it is perceived and valued; whether expected results are occurring; and other questions that are pertinent to program design, management and operational decision making. Performance evaluations often incorporate before-after comparisons, but generally lack a rigorously defined counterfactual.” IV. PERIOD OF PERFORMANCE The period of performance is estimated to be from on/around October 10, 2012 through December 21, 2012, which as extended to January 22th, 2013. V. FUNDING SOURCE The evaluation will be funded with Mission TB earmarked funds. VI. BACKGROUND TB in Brazil USAID has provided development assistance to Brazil since the early 1960s. At least half of USAID’s overall technical and financial assistance to Brazil has been provided to the health sector, including the areas of maternal and child health (MCH), family planning (FP), reproductive health (RH), health systems strengthening (HSS) and decreased transmission of infectious diseases, including HIV/AIDS, Tuberculosis (TB) and malaria. In recent years, USAID/Brazil has increased efforts to fight the spread of infectious diseases through a focused program that addresses TB control and HIV/AIDS prevention. Brazil ranks 17th among the world’s 22 countries with the largest number of TB cases (the 22 high burden countries), accounting for 31% of TB cases in Latin America, with an estimated 71,000 new cases in 2010. However, the country has advanced considerably towards enhanced TB control by increasing TB and drug resistant (DR) and multi-drug resistant (MDR) TB detection rates, rapidly expanding the World Health Organization (WHO)-recommended Directly Observed Therapy Short-Course (DOTS) Strategy in the primary health care network, strengthening the reference laboratory network to decentralize diagnosis, developing the conditions for local production and improved logistics management of TB drugs, enhancing health information systems, and investing in social communication campaigns and behavior change communication (BCC) programs. Combined with activities developed by the GOB and the international donor community, these efforts have resulted in a decrease in TB mortality rates from 3.6/100,000 in 1999 to 2.4/100,000 in 2010, in addition to adoption of DOTS as the national policy for TB case management. USAID has been a key GOB and National Tuberculosis Control Program (Programa Nacional de Controle de Tuberculose - NTP) partner in these achievements since 2001. USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 55 USAID/Brazil TB Portfolio USAID/Brazil’s Tuberculosis programs were designed to expand and enhance the delivery of DOTS in Brazil during the period 2001-2012. DOTS is widely considered to be the most effective intervention for TB case management. Over the past decade, USAID has worked directly with the National TB Control Program and provided technical assistance in 46 of the 315 high-burden TB municipalities in Brazil with the goal to ensure that TB case management will be fully under DOTS and that Brazil will graduate from the list of 22 TB high-burden countries. Projects to be included in the evaluation are: Project Name Implementing Partner Agreement Number Agreement Value Obligation Date COR/AOR/ Activity Manager Centers for Disease Control and Prevention Inter-Agency Agreement (CDC IAA) Centers for Disease Control and Prevention AAG-P-00-99- 00006 $2,212,966 2001 – 2007 Jaime Rojas Centers for Disease Control and Prevention Inter-Agency Agreement (CDC IAA) Centers for Disease Control and Prevention GHN-T-00-06- 00001 $30,000 2006 – 2007 Jaime Rojas WHO Umbrella Grant World Health Organization AAG-G-00-99- 00005 $865,000 2005 – 2006 Jaime Rojas STOP TB Partnership World Health Organization GHA-G-00-08- 00005 $150,000 2008 Patricia Paine Strengthening Pharmaceutical Systems (SPS) Management Science for Health GHN-A-00-07- 00002 $3,748,000 2007 – 2011 Tony Boni Nena Lentini SPS Multi￾Country Associate Award (2010) Management Sciences for Health OAA-LA-10- 00002 $1,673,000 2010 – 2011 Tony Boni TB Control Assistance Program (TB CAP) TB Coalition GHS-A-00-05- 00019 $455,730 30 September 2005 – 30 September Cheri Vincent 56 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 Project Name Implementing Partner Agreement Number Agreement Value Obligation Date COR/AOR/ Activity Manager 2010 Global Health Fellows Program (GHFP) Public Health Institute GPO-A-00-06- 00005 $43,000 2006 – 2007 Jaime Rojas Management & Leadership Development Management Sciences for Health HRN-A-00-00- 00014 $354,631 2001 – 2002 Jaime Rojas Patricia Paine RPM PLUS Management Sciences for Health HRN-A-00-00- 00016 $2,998,000 2003 – 2010 Tony Boni Patricia Paine TB Coalition for Technical Assistance Royal Netherlands TB Assn.(KNCV) HRN-A-00-00- 00018 $1,852,000 2001 – 2003, 2006 Jaime Rojas Patricia Paine JHU Family Health and Child Survival Johns Hopkins University HRN-A-00-96- 90006 $1,100,000 2001 – 2006 Patricia Paine Systems for Improved Access to Pharma. & Services (SIAPS) Management Sciences for Health OAA-A-11-00021 $2,100,000 29 September 2011 – 30 September 2012 COR: Tony Boni Activity Manager: Nena Lentini Expansion of Directly Observed Treatment Short-Course Strategy in Brazil Pan American Health Organization (PAHO) 512-G-00-06- 00024-00 $9,000,000 29 September 2006 – 30 September 2013 Nena Lentini Social Communication Technologies for TB/HIV Co￾AIDSTAR One GHH-I-01-07- 00059 $1,110,000 29 September 2011 – 30 September Elizabeth Berard USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 57 Project Name Implementing Partner Agreement Number Agreement Value Obligation Date COR/AOR/ Activity Manager infection Control 2012 TB Drug￾resistance Survey, Nurse and Supervision Training BEMFAM 512-A-00-01- 00016-00 $3,194,556 1 October 2001 – 30 September 2008 Jaime Rojas; Patricia Paine Behavior Change Communication for Effective TB Control John Snow, Inc. (JSI) 512-TO-10-00001 $1,053,000 29 September 2010 – 30 September 2011 Nena Lentini Projects Summaries CDC IAA CDC primarily provides technical assistance to TB programs in the highest priority municipalities in the states of Rio de Janeiro and São Paulo. CDC also conducts TB research, tapping into its well-established competitive advantage. Under this interagency agreement, CDC completed a TB needs assessment for the states of Rio de Janeiro and São Paulo. WHO Umbrella Grant The WHO Umbrella Grant supported broad USAID-WHO collaborations in TB, malaria and other vector-borne diseases, surveillance, research, antimicrobial resistance, maternal and child health, and HIV/AIDS at WHO headquarters in Geneva as well as regional and country offices. The WHO was strong in all aspects of DOTS program design, policy dialogue, monitoring and evaluation, implementation, and advocacy, and provided assistance to countries to prepare plans for DOTS expansion and programs to address HIV/AIDS and TB co-infection. The WHO also developed (in partnership with other organizations) manuals, norms, guidelines, and training materials related to TB. STOP TB Partnership Founded in 2001, the Stop TB Partnership’s mission is to serve every person who is vulnerable to TB and ensure that high-quality treatment is available to all who need it. STOP TB’s 1,000+ partners work in 100 countries – including Brazil – in the fight against TB. They include international and technical organizations, government programs, research and funding agencies, foundations, NGOs, civil society and community groups and the private sector. The partnership works through the WHO and seven working groups whose role is to accelerate progress on access to TB diagnosis and treatment; research and development for new TB diagnostics, drugs and vaccines; and tackling drug resistant- and HIV-associated TB. 58 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 MSH – Strengthening Pharmaceutical Systems (SPS) SPS aimed to strengthen TB pharmaceutical logistics, developed and established quality assurance programs, carried out operational research on new rapid diagnostic tools for drug sensitive and drug resistant TB; supported the move to fixed dose combination regimens and supported the local production of new pharmaceutical formulas for first line TB treatment; strengthened drug resistant TB control and assisted National Program for Tuberculosis Control (NTP) to respond to extensively drug resistant TB challenges; and strengthened the TB laboratory network to improve TB and DR-TB control; to decrease the TB burden in People Living with HIV/AIDS (PLWHA) through the implementation of TB/HIV collaborative activities. TB Coalition – Tuberculosis Control Assistance Program (TB CAP) TB CAP provided technical assistance to USAID Bureaus, Missions, host country governments and other partners to prevent and control tuberculosis. TB CAP contributed to the national, global and Millennium Development Goals (MDGs) targets for TB control. MSH – Management & Leadership Development MSH provided technical and administrative support to the National TB Program (NTP) and the Rio de Janeiro State Secretariat of Health TB Program (Rio de Janeiro PCT). Through this project, MSH established joint work plans for activities to improve national, state and municipal political and administrative support for TB control, including DOTS. RPM PLUS RPM+, in conjunction with MSH, implemented a decentralized quality control program for MDR￾TB drugs (E-TB Manager) as well as a country wide management system for monitoring the diagnosis and treatment of MDR-TB patients. Additionally, RPM+ worked with Centro de Referência Professor Hélio Fraga to modify existing drug regimens based on the evidence made available by a USAID-funded, country wide evaluation of existing re-treatment regimens. TB Coalition for Technical Assistance The TBCTA is a unique coalition of the major international organizations in TB control. TBCTA programs in Brazil include: the acceleration implementation and expansion of the DOTS strategy in Brazil; developing institutional and technical capacity for TB control at federal, state and municipal levels; integrating TB control efforts within primary health care services more effectively; and strengthening linkages between the TB control program and HIV/AIDS prevention and treatment programs and MDR-TB efforts. JHU Family Health and Child Survival JHU worked to improve the political and administrative support for TB control through DOTS at the municipal, state and national levels. They expanded DOTS coverage and performance in selected targeted areas in the city of São Paulo and supported continuing DOTS efforts by expanding into new areas of Rio de Janeiro State. Another JHU project named “Expansion of DOTS for tuberculosis control in Rio de Janeiro, Brazil” was added during the data collection of this evaluation, based on complementary information. MSH – Systems for Improved Access to Pharmaceuticals and Services (SIAPS): SIAPS focuses on enhancing pharmaceutical services through patient-centered solutions while continuing to support essential supply chain functions and medical products supply security. The SIAPS goal is to support the achievement of improved and sustained health outcomes by applying USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 59 a systems strengthening approach consistent with the Global Health Initiative (GHI) principles. In Brazil, SIAPS supports local production of TB drugs through research and technology transfer, medicine quality control, laboratory strengthening, and development of health information systems, among others. Pan American Health Organization (PAHO) – Expansion of Directly Observed Treatment Short-Course Strategy (DOTS) for Tuberculosis in Brazil: The most effective public health strategy for reducing Tuberculosis transmission is the World Health Organization-recommended Directly Observed Treatment Short Course Strategy (DOTS). DOTS has several elements: government commitment to TB control, TB diagnosis by sputum smear microscopy for patients who report a cough lasting three weeks or longer, standardized and directly observed drug therapy, an efficient and reliable drug supply system, and an efficient recording and reporting system with assessment of treatment results. Through PAHO’s activities USAID aims to support the Brazilian National Tuberculosis Program strengthen all elements of the DOTS strategy. An important goal of PAHO’s project is the expansion of directly observed treatment. The expected outcome is the reduction of Tuberculosis transmission, morbidity and mortality in metropolitan areas of high TB disease burden. In order to achieve these results, PAHO is working through Brazil’s extensive Primary Health Care Network, Family Health Program and the Community Outreach Workers Program. This will increase the efficiency and effectiveness of TB control. JSI – AIDSTAR One – Social Communication Technologies for TB/HIV Co-infection Control: The AIDSTAR-One project provides technical assistance to USAID’S Office of HIV/AIDS and USG country teams across a broad range of technical areas. The project is available to implement programs in other bilateral countries across prevention, care and treatment, and for technical assistance to support USG teams in strategic planning, program or needs assessments and documentation of successful program models. In Brazil, AIDSTAR promotes behavior change communication (BCC) among patients and service providers for enhance TB/HIV co-infection control. BEMFAM – TB Drug-resistance Survey, and Nurse and Supervision Training BEMFAM (a Brazilian NGO) conducted TB activities, including all five elements of DOTS, in some of the highest-burden municipalities for TB. BEMFAM trained thousands of health professionals in DOTS, which led to the initiation of directly supervised treatment (DOT) in municipalities with the greatest absolute number of cases. JSI – TASC Three – Behavior Change Communication (BCC) for more Effective TB Control This activity aimed to 1) assist local governments to improve productive and social infrastructure; and 2) maximize the health impact of the water and sanitation infrastructure. JSI delivered training and supervision in DOTS through engagement of state and municipal program managers and civil society mobilization to increase case detection, treatment and cure. It also provided training in microscopy, lab quality control, infection control measures and TB drug management, and training in MDR-TB. Finally, this activity developed BCC strategies and provided peer-to-peer interventions for TB control, including Knowledge, Attitude, Behavior and Practice (KABP) 60 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 surveys, training in communications strategies, and development of information, education and communication (IE&C) materials. VII. SPECIFIC OBJECTIVES AND EVALUATION QUESTIONS Listed below are the principal questions to evaluate USAID interventions on TB control in Brazil. Quantitative indicators should be provided to the extent possible (i.e. before and after project intervention implementation output and outcome). For most evaluation questions, each TB project should be evaluated alone, as USAID’s objective is to obtain an evaluation that assesses the results of individual TB projects along with the total collective contributions of USAID TB projects’ package of interventions to the overall goals and objectives. In addition to the main questions below, the list of questions in Annex B will be reviewed and revised by the evaluation team, in consultation with USAID/Brazil, to inform a reasonable data collection tool. • How was USAID able to affect change and what change can be directly attributed to USAID interventions? Where were they not successful at achieving change? • What interventions, policies, financing and other enabling environment have been introduced by USAID projects? • How were USAID programs able to foster integration between programs, policies and service provision units for enhanced TB control? • To what extent have USAID projects contributed to sustainable TB control through effective local capacity building throughout Brazil’s decentralized health system? • How effective was donor coordination and how was USAID able to work in a complementary fashion in a multi-donor environment? VIII. METHODOLOGY The Contractor is required to gather information through reports and data review, and in￾country interviews with (but not limited to): government staff (PNCT, State and Municipal TB Programs), TB service providers (national, state and local), non-governmental organizations (NGOs), community-based organizations (CBOs), health facility managers, support groups, patients; USAID implementing partners; and USAID LAC, GH and Mission Brazil staff; among other partners and stakeholders. The Contractor is required to develop gender-sensitive data collection and evaluation tools for ensuring consistency of information for different partners and areas of TB control. The following are illustrative methods for generation of data and information for this evaluation: • Desk review - collect data before and after project implementation at sites. • Site visits to select USG-assisted sites and non-project sites to compare the outcomes and the performance of the two groups. • Focus Group discussions for data validation. Descriptions of sources of information and methods used are to be included in the final report. USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 61 1. Desk Review and development of data collection tool The Contractor is expected to carry out a desk review of materials generated by USAID/Brazil and each project prior to the country visit. The Contractor is expected to review various sources of documental information such as quarterly reports, annual reports, studies, publications, tools, and other relevant documents. This desk review will help to organize materials and to review progress, thus facilitating analysis information prior to arrival in Brazil maximizing time in country. The Contractor is expected to review historical program documents including USAID health strategy information and performance data that is available in USAID’s Development Experience Clearinghouse, the largest online resource of USAID-funded technical and programmatic documentation, through a searchable DEC Web site (http://dec.usaid.gov). For guidance please see ADS 540 partial revision http://transition.usaid.gov/policy/ads/500/540.pdf. Procurement￾sensitive and proprietary data is not available in reports in the DEC and will not be provided to the Contractor. The Contractor is expected to extract relevant components of the reports, project work plans and data, in addition to data from the national, state and municipal TB programs for the purposes of the evaluation. Analysis and confirmation of project-specific data will be done by the evaluation team either during the desk review or during the country visit. The Contractor is required to identify any gaps in publicly available data and to notify USAID/Brazil of such gaps. The Contractor is required to ensure gender responsiveness according to USAID’s Gender Equality and Female Empowerment Policy (March 2012). This Policy is available at http://transition.usaid.gov/our_work/policy_planning_and_learning/documents/GenderEqualityPoli cy.pdf The Contractor is expected to coordinate with USAID/Brazil during implementation of the performance evaluation, including during the desk review, and to participate in meetings/conference calls to: • Review the Statement of Work’s goals and objectives; • Review work plan, evaluation design and timeline; • Review background materials; • Review contractor performance including roles and responsibilities; • Review data collection methods and instruments; • Review in-country itinerary and schedule of interviews. The Contractor is expected to develop a data collection instrument based on an agreed list of outcome/output indicators and interview questions. The contractor is expected to test the data collection instrument before full implementation. This instrument will serve as the major tool to collect data (before and after USAID intervention; USAID-assisted sites versus non-USAID sites), both quantitative and qualitative. The Contractor is required to submit the final data collection instrument and work plan to USAID approval before the country visit. The Contractor is required to disaggregate data by gender for all people-level indicators. Given the extensive period of time to be covered by the evaluation, the evaluation team should expect to encounter limited documentation and sources of information for projects that closed 62 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 more than five years ago. Whenever document and data limitations impact the depth and breadth of the evaluation this should be noted in the final report. It is important to note that no baseline surveys were completed prior to 2001. 2. Data Collection and Analysis The Contractor is expected to coordinate with the Brazil TB Backstop Team in USAID/Washington before traveling to Brazil through an in-person meeting or a conference call or video conference, to obtain important background about activities implemented by USAID’s central and regional mechanisms, in addition to input about the performance of the Brazil TB program as a whole. The Contractor is expected to propose the time required in country for collection and analysis of data required to provide the expected results of the performance evaluation. However, it is estimated that this will require approximately three to four weeks in Brazil interviewing key stakeholders and partners and visiting project and non-project sites. The Contractor is expected to begin the country visit with a Team Planning Meeting with USAID/Brazil to: • Review the Statement of Work’s goals and objectives; • Review approved work plan; • Clarify any issues on the background material; • Review expectations regarding Contractor performance including roles and responsibilities; • Review and make last revisions to the data collection methods and instruments; • Review and finalize the country itinerary and schedule of interviews. The Contractor is required to include, but not be limited to collect data for analysis from interviews/visits as follows: • USAID/Brazil Mission • Other USG agencies working in health (CDC) • Senior PNCT Officials (PNCT Coordinator, Technical Officers and Strategic Information Officer); Senior National HIV/AIDS Department Officials (Deputy Director, Co-Infection Technicians); • State TB programs of Rio de Janeiro, São Paulo, Rio Grande do Sul, Santa Catarina and Paraná; • A selection of municipal TB programs to be extracted from the following illustrative list of municipal programs: o Itaboraí, Magé, Nova Iguaçu, Belford Roxo, Japeri, Queimados and São João do Meriti (Rio de Janeiro State); São Paulo, Guarulhos, Osasco, Carapicuíba and Franco da Rocha (São Paulo State); Manaus (Amazonas State); Belo Horizonte and Contagem (Minas Gerais); Recife, Olinda and Jaboatão dos Guararapes (Pernambuco State); USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 63 o Municipal TB and HIV/AIDS Programs of Porto Alegre, Alvorada and Canoas (Rio Grande do Sul); Florianópolis and Itajaí (Santa Catarina State); and Paranaguá (Paraná State) • Other Government officials and staff engaged in TB Control, and particularly laboratory and health facility staff in target areas; • Centro de Referência Professor Hélio Fraga, Farmanguinhos and FIOCRUZ (Rio de Janeiro); • USAID past and current implementing partners (BEMFAM, MSH, Stop-TB, JHU, PAHO, JSI); • Civil society organizations working on TB community mobilization (Transformarte, Rede Paulista, John Snow do Brasil); • Other donors supporting TB activities (Global Fund, Gates Foundation) • Health care workers and managers in the municipalities referred to above (M/F) • Clients, treatment partners (M/F) The Contractor is expected to develop a set of questions appropriate to specific interviewees. The different USAID TB projects cater to different clients. The Contractor is expected to conduct interviews designed to ensure that useful information can be generated to determine if desired outcomes have been achieved. The Contractor is at liberty to ask normative questions which are relevant to determine facilitating factors or barriers affecting implementation of the projects. USAID/Brazil expects the contractor to present strong quantitative and qualitative analysis of the data collected throughout the visit. The evaluation team should allow significant time for data analysis at the conclusion of the field visits. 3. Debrief/Report At the end of the country visit, the team will provide an oral/PowerPoint presentation on the major findings and analysis to USAID/Brazil, PNCT, and other relevant partners for validation prior to completion of the final report. Analysis should be based on the result of data review and interviews, vis-à-vis expert judgment and logical analysis. Participants’ bias should be determined in order to come up with more logical and objective analysis. The team will provide USAID/Brazil with a draft written report within three weeks of the presentation. USAID staff will provide comments and suggestions for incorporation into the final report. Dissemination of the final report will be conducted with PNCT and relevant stakeholders. IX. DELIVERABLES AND TIMELINE Deliverable Due Date Minutes of the meeting with GH and LAC backstop team in Washington DC 2 weeks after award Detailed evaluation plan, methodology, design and timeline for 2 weeks after award 64 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 Deliverable Due Date evaluation Final Data Collection Instruments and Method of Analysis 3 weeks after award Interview questions and site visit schedule 3 weeks after award Sequential trip reports per State with information about meetings held and interviews completed 6 weeks after award Debriefing and presentation of findings November 7, 2012 Draft report to USAID/Brazil January 7, 2013 USAID comments and suggestions submitted to GH Tech Bridge II Program Manager January 9, 2013 USAID provides final sign-off on draft report TBD Final edited and formatted report submitted to USAID January 22, 2013 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 65 ANNEX B. PERSONS CONTACTED 66 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 67 68 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012    USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 69 ANNEX C. REFERENCES GENERAL Brazil. Ministério da Saúde. Programa Nacional de Controle da Tuberculose (PNCT). Situação Epidemiológica. Available at: http://portal.saude.gov.br/portal/arquivos/pdf/taxa_mortalidade_tuberculose_1999_2010_30_05_ 2012.pdf. Brazil. Ministério da Saúde. Plano Estratégico para a Implementação do Plano de Controle da Tuberculose no Brasil no período de 2001-2005. Resumo Executivo. Brasília, 2000. Brazil. Ministério da Saúde. Secretaria de Atenção à Saúde. Departamento de Atenção Básica Pacto de Indicadores da Atenção Básica: instrumento de negociação qualificador doprocesso de gestão do SUS. Rev. Bras. Saúde Matern. Infant., Recife, 3 (2): 221-224, abr. / jun., 2003. Available at http://www.scielo.br/pdf/rbsmi/v3n2/a13v03n2.pdf. Brazil. Ministério da Saúde. Secretaria de Vigilância em Saúde. Departamento de Vigilância Epidemiológica. Sistema de Informação de Agravos de Notificação – Sinan: normas e rotinas / Ministério da Saúde, 2. ed. – Brasília : Editora do Ministério da Saúde, 2007. Available at http://bvsms.saude.gov.br/bvs/publicacoes/07_0098_M.pdf. Brazil. Ministério da Saúde. Secretaria de Vigilância em Saúde. Departamento de Vigilância Epidemiológica. Manual de recomendações para o controle da tuberculose no Brasil / Ministério da Saúde, Secretaria de Vigilância em Saúde, Departamento de Vigilância Epidemiológica. – Brasília:Ministério da Saúde, 2011. Brazil. Ministério da Saúde. Coordenação-Geral do Programa Nacional de Controle da Tuberculose. Especial tuberculose. Boletim Epidemiológico, volume 43, pages 2-12, March 2012a. Brazil. Ministério da Saúde. País atinge uma das metas do milênio com redução de mortes por tuberculose. 2012b. Available at http://www.brasil.gov.br/noticias/arquivos/2012/06/01/pais-atinge￾uma-das-metas-do-milenio-com-reduçao-de-mortes-por-tuberculose (November 27th 2012, date last accessed). Brasil. Ministério da Saúde. Programa Nacional de Controle da Tuberculose. Protocolo II Inquérito Nacional de Resistência a Drogas em Tuberculose. Brasil, 2004. Brasil. Ministério da Saúde. Secretaria de Vigilância em Saúde. Departamento de Vigilância Epidemiológica. Manual de recomendações para o controle da tuberculose no Brasil / Ministério da Saúde, Secretaria de Vigilância em Saúde, Departamento de Vigilância Epidemiológica. – Brasília : Ministério da Saúde, 2011. Diniz L S, Gerhardt G, Miranda J Á, Manceau J N. Efetividade do tratamento da TB em oito municípios de capitais brasileiras. Bol Pneumol. Sanit 1995; 3: 6–18. Hijjar, MA; Gerhardt, G; Teixeira, GM; Procópio, MJ. Retrospecto do controle da tuberculose no Brasil. Rev Saúde Pública 2007; 41(Supl. 1): 50-58. 70 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 Kritski AL, Ruffino-Netto A. Health sector reform in Brazil: impact on tuberculosis control. Int J Tuberc Lung Dis. 2000 Jul; 4(7):622-6. Levcovitz, E. et al. Política de saúde nos anos 90: relações intergovernamentais e o papel das normas operacionais básicas. Ciência e Saúde Coletiva. 2001; 6 (2): 269-291. Pio A, Luelmo F, Kumaresan J, Spinaci S. National tuberculosis programme review: experience over the period 1 990-95. Bulletin of the World Health Organization, 1997, 75 (6): 569-581. Stop￾TB partnership. Brazil – country profile. Available at: http://www.stoptb.org/assets/documents/countries/acsm/Brazil.pdf. Raviglione, MC. The TB epidemic from 1992 to 2002. Tuberculosis. 2003; 83, 4 Ruffino-Netto A, Villa, TCS. Tuberculosis treatment. DOTS implementation in some regions of Brazil. Background and regional features. Millennium Institute – Rede TB (Brazilian tuberculosis research network). ISBN 978-85-86862-18-2. Available at http://www.eerp.usp.br/geotb/Doc/livro_tuberculose_ingles.pdf. Ruffino-Netto, A. Programa de Controle da Tuberculose no Brasil: situação atual e novas perspectivas. IESUS 2001, 10 (3):129-138. Stop TB Partnership. Country profile – Brazil 2006. Available at: http://www.stoptb.org/assets/documents/countries/acsm/Brazil.pdf. WHO. Resolution WHA44.8. Tuberculosis control programme. In: Handbook of resolutions and decisions of the World Health Assembly and the Executive Board. Volume III, 3rd ed. (1985– 1992). Geneva, World Health Organization, 1993 (WHA44/1991/REC/1):116. WHO. Report of the AD Hoc Committee on the tuberculosis epidemic. Global Tuberculosis Programme. WHO/TB/98.245. Geneva: WHO, 1998. WHO. The five elements of DOTS. Available at: http://www.who.int/tb/dots/whatisdots/en/index.html. WHO. The Stop TB strategy. Building on and enhancing DOTS to meet the TB-related Millennium Development Goals.2006. Available at http://whqlibdoc.who.int/hq/2006/WHO_HTM_STB_2006.368_eng.pdf. WHO. Global tuberculosis report 2012. Available at: http://www.who.int/tb/publications/global_report/en/index.html. PROJECT-SPECIFIC MSH Activities Summary for USAID TB Evaluation Team, MSH, July 2003 Final Consultant Report on TB NGO Forum, MSH USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 71 The Role of the TB Forum, MSH/AIDS 2006 SPS Activity and Product Status Report, MSH, 2010 Strengthening Pharmaceutical Systems Program, MSH, 2012 Strengthening Pharmaceutical Systems Program: Work Plan for TB/HIV Activities, Oct. 2010- Sept. 2011 DOTS Implementation in Brazil, How Family Health Teams Have Made it Work Management and Leadership Program, Final Report 2005 MSH Management Development Assistance to Public Sector TB Programs in Brazil, 2001-2002 Strengthening Capacity of Tuberculosis Control Programs in Brazil Work Plan, July 2002- Sept. 2003 Strengthening Capacity of Tuberculosis Control Programs in Brazil Work Plan, Sept. Oct. 2002- Sept. 2003 MSH Training and Technical Assistance Activities in the Area of TB, October 1, 2002 to September 30, 2003 Deployment of Supervised Treatment for Tuberculosis, MSH Management and Leadership Plan for Performance Improvement, 2003 Logical Framework Matrix: Strategy Implementation of DOTS, 2003-2004 MSH Brazil, Final Report, Nov. 2003 RPM Plus/Brazil Annual Report FY 05, 30 November 2006 RPM Plus/Brazil Annual Report FY 06, 03 October 2007 RPM Plus/Brazil Proposal FY 07, December 2006 RPM Plus - USAID updated DECEMBER 2006 (Power Point presentation) Brazil FY08 SPS Final Report - 01 Oct 08-30 Sept 09 - Executive Summary -12 Nov 09 MSH-SPS Brazil - Executive Summary on FY09 results- October 2009-Sept 2010 20101 - final version Work Plan BrazilFY09 USAID MSH-SPS final version - revised 22 March 2010 72 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 MSH/SPS TB Activity Portfolio – Update on FY09 and Perspectives for FY10. USAID Meeting - Brasília– November, 3rd - 2010 (Power Point presentation) TB-HIV - SPS Workplan Brazil - last version v4 19-01-2011 Strengthening Pharmaceutical Systems Program, Brazil. Work Plan for TB/HIV Activities: October 2010 to September 2011 – (USAID Brazil Mission Funding). Printed November 2010 Strengthening Pharmaceutical Systems Program (TB/HIV). Brazil. FY10 Work Plan: USAID Mission SPS Activity and Product Status Report. Activity progress – TBHIV. Work plan Brazil TB Work plan Year 2010 Systems for Improved Access to Pharmaceuticals and Services, Brazil. Work Plan: FY11 – USAID Brazil Mission Funding. November/2011 Projecto MSH, Activities conducted & Overall Program Evaluation (Power Point presentation). Rio de Janeiro, November 5th - 2012 MSH activities and partnerships to support Tuberculosis / Drug Resistant TB control in Brazil: Overview of key interventions and results to date. USAID Evaluation Visit– Rio de Janeiro - 5 Nov 2012 (Power Point presentation) Implementação do controle da coinfecção TB/HIV na Região Sul do Brasil USAID Evaluation Visit– Rio de Janeiro - 5 Nov 2012 (ppt presentation) PAHO USAID Brazil Tuberculosis Profile, 2009 Successful Experiences in Control of Tuberculosis, USAID Project, 2005 Successful Experiences in Control of Tuberculosis, USAID Project, 2006-2008 Expansion of DOTS Strategy for Tuberculosis in Sao Paulo City, March 2008 Design Building Expansion Strategy and DOTS in the Metropolitan Region, 2009 Report of Activities, Expansion of DOTS in Brazil (OPAS-USAID), 2009 Plan of Action, Rio de Janeiro, 2009-2010 Work Plan Rio de Janeiro and Sao Paulo, 2010 Work Plan, USAID, 2011 Work Plan, USAID, 2012 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 73 Semester Plan, USAID, 2010 Work Plan, DOTS Expansion in Brazil, 2009 USAID/Brazil Participant Training Report for In-Country Activities, 2011 Improving the National Tuberculosis Program (NTP) management and laboratory capacity and expand the implementation of DOT in Rio de Janeiro State, PAHO/USAID, 2008 Rio de Janeiro Tuberculosis Control Program, June 2011 DOTS Expansion in Brazil, Oct. 1-Dec. 31, 2007, Report: January 2008 DOTS Expansion in Brazil, Jan. 1-March 31, 2008, Report: April 2008 Work Plan: DOTS expansion in the city of Rio de Janeiro through family health teams and community health workers, JHU/PAHO/USAID, 2004 Work Plan for USAID Project in Sao Paulo, 2006 Work Plan Components, Rio de Janeiro, Sao Paulo, MDR-TB, TB/HIV Co-infection, 2007-2008 PAHO Financial Report, Sept. 29, 2006-March 2007 PAHO Financial Report, April-June 2007 PAHO Financial Report July-Sept. 2007 PAHO Financial Report Oct.-Dec. 2007 PAHO Financial Report Jan.-March 2008 PAHO Financial Report Apr.-June 2008 PAHO Financial Report July-Sept. 2008 PAHO Financial Report Oct.-Dec. 2008 PAHO Financial Report Jan.-Mar. 2009 PAHO Financial Report Apr.- June 2009 PAHO Financial Report July- Sept. 2009 PAHO Financial Report Oct.-Dec. 2009 PAHO Financial Report June-Sept. 2010 74 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 PAHO Financial Report, Summary, 2006-2011 PAHO Financial Memorandum, 2011 PAHO Financial Report, Jan- Mar. 2011 s JSI John Snow, Inc. Monitoring and Evaluation (M&E) Plan Brazil Behavior Change Communication for More Effective Tuberculosis Control (BCC for TB) Project. 31 December 2010. John Snow, Inc. Geographical distribution of activities in the BCC for more effective TB control program (15 Municipalities in State of SP and 2 Districts in the Municipality of SP). September 2010-11. John Snow, Inc. BCC for More Effective Tuberculosis Control Program. Final Report. September 30, 2010 – September 29, 2011. Past performance narrative. Agreement Officer: Cynthia Shartzer, USAID/Peru. Period of performance: October 1st, 2010 to September 30, 2011. Quarter 2 performance report. AIDSTAR-1 Brazil Social Tech. Social Technologies for Sustainable Access to Tuberculosis Control and HIV/AIDS Interventions in Brazil (Social Tech). EPAH. Relatório (fase #2). Aumentar as opções de acesso a informação, educação e comunicação sobre coinfecção da tuberculose e prevenção do HIV. 1 de Novembro 2012 a 24 de novembro 2012. EPAH. Relatório (fase #3). Aumentar as opções de acesso a informação, educação e comunicação sobre coinfecção da tuberculose e prevenção do HIV. 24 de Novembro 2012 a 14 de dezembro 2012. John Snow do Brasil. Executive summary. Penitentiary intervention for more effective tuberculosis control in São Paulo, Brasil. AIDSTAR One Brasil Social Tech. December 2012. CDC Centers for Disease Control. Relatório anual de atividades – Novembro/2003. Cooperação USAID/CDC/MS. 2003. Centers for Disease Control. Annual activities report – November/2004 - USAID/CDC/MS. 2004. Centers for Disease Control. USAID/CDC/MOH - Project summary – June 2004. Centers for Disease Control. USAID activities report – September 2006. USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 75 Miranda A, Morgan M, Jamal L, Laserson K, Barreira D, et al (2007) Impact of Antiretroviral Therapy on the Incidence of Tuberculosis: The Brazilian Experience, 1995–2001. PLoS ONE 2(9): e826. doi:10.1371/journal.pone.0000826 PAHO/WHO. Second survey of drug resistant tuberculosis in Brazil. Progress report. June 18-26, 2007. Brasil. Ministério da Saúde. Programa Nacional de Controle da Tuberculose. Protocolo II Inquérito Nacional de Resistência a Drogas em Tuberculose. Brasil, 2004. Brasil. Ministério da Saúde. Secretaria de Vigilância em Saúde. Departamento de Vigilância Epidemiológica. Manual de recomendações para o controle da tuberculose no Brasil/Ministério da Saúde, Secretaria de Vigilância em Saúde, Departamento de Vigilância Epidemiológica. – Brasília : Ministério da Saúde, 2011. JHU Project Prefeitura da cidade do Rio de Janeiro. Secretaria de Saúde. Proposal for USAID Initiative. Expansion of DOTS for Tuberculosis Control in Rio de Janeiro, Brazil. Annual Report of Current Activities of USAID program in Brazil. Collaboration between the Municipal Health Secretariat of Rio de Janeiro and Johns Hopkins University. November 2003. Cavalcante SC, Soares ECC, da Rocha MC, de Oliveira JR, Dias SM, Pacheco AGF, Chaisson RE, Durovni B. Implementation of the DOTS strategy in the city of Rio de Janeiro. Pulmao RIO DE JANEIRO 2003;12:71-79. Soares EC, Pacheco AG, Mello FC, Durovni B, Chaisson RE, Cavalcante SC. Improvements in treatment success rates with directly observed therapy in Rio de Janeiro City. Int J Tuberc Lung Dis. 2006 Jun;10(6):690-5. Mohan CI, Bishai D, Cavalcante S, Chaisson RE. The cost-effectiveness of DOTS in urban Brazil. Int J Tuberc Lung Dis. 2007 Jan;11(1):27-32. Cavalcante SC, Soares EC, Pacheco AG, Chaisson RE, Durovni B; DOTS Expansion Team. Community DOT for tuberculosis in a Brazilian favela: comparison with a clinic model. Int J Tuberc Lung Dis. 2007 May;11(5):544-9. Cavalcante SC, Durovni B, Barnes GL, Souza FB, Silva RF, Barroso PF, Mohan CI, Miller A, Golub JE, Chaisson RE. Community-randomized trial of enhanced DOTS for tuberculosis control in Rio de Janeiro, Brazil. Int J Tuberc Lung Dis. 2010 Feb;14(2):203 Miller AC, Golub JE, Cavalcante SC, Durovni B, Moulton LH, Fonseca Z, Arduini D, Chaisson RE, Soares EC. Controlled trial of active tuberculosis case finding in a Brazilian favela. Int J Tuberc Lung Dis. 2010 Jun;14(6):720-6. 76 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 BEMFAM Modification of Assistance, BEMFAM/USAID, 2001 Modification of Assistance Contract, BEMFAM/USAID, 2006 BEMFAM Annual Report, 2011 BEMFAM Annual Report, Part 1, 2003 BEMFAM Annual Report, Part 2, 2003 BEMFAM Annual Report, Part 3, 2003 BEMFAM, Review of the Coalition against Tuberculosis, 2003-2007 BEMFAM, Evaluation of Anti Tuberculosis Coalition Project, 2005 Modification of Assistance, BEMFAM/USAID, 2002 BEMFAM, Annual Report, 2004 BEMFAM, Annual Report, 2005 BEMFAM, Annual Report, 2006 BEMFAM, Survey of Drug Resistance, June 2004 BEMFAM, Summary of Work, 2005-2006 BEMFAM, Review of Activities, 2004 BEMFAM, Narrative Report, 2005 Coalition against Tuberculosis - Promotion of Tuberculosis Control at the Municipal Level, BEMFAM, 2004-2005 Work Plan Proposal, DOTS Expansion, BEMFAM, 2005-2006 BEMFAM Recife Report for Final Fiscal year, 2005-2006 TBCTA/TB CAP Overview of TBCTA and TBCAP in Brazil Executive Summary, TBCAP Brazil, Oct. 2006- Sept. 2007 TBCAP F01 Brazil Work Plan APA1 Approved, Oct. 2005- Sept. 2006 TBCAP Brazil APA2, Oct. 2006-Sept. 2007 TBCAP Final Report, 2005-2010 The Tuberculosis Coalition for Technical Assistance, Final Report, 2000-2006 TBCTA Brazil (3) Work Plan, TBCTA Brazil Approved Work Plan, TBCTA, 2001-2006 Brazil Approved Work Plan 2, TBCTA TBCTA Budget, TBCTA USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 77 ANNEX D: QUESTIONNAIRES DEVELOPED INTERVIEW QUESTIONS: IMPLEMENTING PARTNERS Overarching Issues: 1. How have USAID-supported TB projects been coordinated with other partners (USAID/non-USAID, TB and other health and non-health partners)? (D) 2. Describe the synergies of the USAID-TB projects with other development partners working for TB in terms of activities, support, sites, etc. (D) 3. How and to what extent have partners’ capacity been strengthened? (E) General TB DOTS Interventions: 4. What attention has been paid to vulnerable populations, like indigenous groups, people with disabilities, geographically isolated and depressed areas, and congregate settings of the urban poor? (D) 5. Have TB case notifications or treatment outcomes in these groups improved? (A,B) Provide sex-disaggregated data for all people level indicators. Policy, Financing and Regulatory Environment: 6. Have the USAID-supported TB project partners able to develop and implement local policies to support the program? (A, D) 7. Have the USAID supported TB projects able to leverage resources provided by the national and local governments? (A, C) Programmatic Management of Drug Resistant TB (PMDT): 8. Have USAID-TB projects helped to increase the number of MDR-TB patients diagnosed? (A,B) 9. What social support interventions for PMDT, if any, have USAID-TB projects supported? What has been the impact? (C, D, E) TB Laboratory Strengthening and Diagnostic Network: 10. How have the USAID-TB projects contributed to TB laboratory strengthening nationally? Locally? (A,B) 11. How the USAID-TB projects contributed to the quality of TB smear microscopy labs in local settings? Are they staffed by trained microscopists? (A, B) 12. How the USAID-TB projects contributed to rational lab diagnostic strategies at national level? At regional levels? (A, B) 13. How the USAID TB projects contributed to TB suspects to have adequate access to TB diagnosis (labs, x-rays, etc.)? (A, B) Anti-TB Drug Supply and Network: 14. USAID-supported TB projects have contributed to strengthened: Topic (1) Strongly disagree (2) Disagree (3)Neutral (4)Agree (5) Strongly agree Performance Evaluation A. Project achievement B. Are expected results occurring? C. Questions pertinent to Program design D. Implementation process/ Questions pertinent to management/operational decision making E. How it is perceived and valued? 78 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 a. Information b. TB drug management system 15. On scale of 0-5, where 0 (zero) means “no change” and 5 (five) being “the most change”, how do you quantify the contribution of USAID-supported TB projects on: Topic 0 1 2 3 4 5 a. Forecasting b. Procurement c. Distribution d. Utilization 16. USAID- supported TB projects have contributed to improved: Topic (1) Strongly disagree (2) Disagree (3)Neutral (4)Agree (5) Strongly agree a. Staff capacity to monitor the availability and quality of anti-TB drugs b. Staff capacity has improved Gender: 17. How have gender considerations been integrated in USAID-TB programs and TB-related activities? (C,D) 18. What are the differential effects of the project on male and female beneficiaries? (C,D) 19. What gender issues were identified in the USAID-TB programs and how were they addressed? (C,D,E) 20. What was lacking in the project design and/or implementation that would have improved gender considerations? (C,D) Overall 21. Accomplishments gained from USAID’s investment in TB activities in Brazil and lessons learned from the past decade in terms of:  Policy change  DOTS adaptation  Project sustainability 22. What has been the effect of USAID-funded projects on:  TB integration into the primary health care network  Combined approaches to TB/HIV co-infection,  Capacity building across the spectrum of health care professionals involved in service delivery  Health systems strengthening USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 79 INTERVIEW QUESTIONS: STATE AND MUNICIPALITY What USAID-supported TB projects are/have been implanted in your state/municipality? Implementer: Project period: General TB DOTS Interventions: 1. How is DOTS defined in your TB state/municipality strategy: Please note the 5 topics: a) Political commitment with increased and sustainable financing b) Case detection through quality-assured bacteriology c) Standardized treatment with supervision and patient support d) An effective drug supply and management system e) Monitoring and evaluation system and impact measurement 2. Has TB case detection increased in USAID-supported TB project sites? Nationally? Provide baseline, target and actual data sets. 3. Have TB treatment outcomes improved in USAID-supported TB project sites? Nationally? Provide baseline, target and actual data sets. 4. What attention has been paid to vulnerable populations, like indigenous groups, people with disabilities, geographically isolated and depressed areas, and congregate settings of the urban poor? 5. Have TB case notifications or treatment outcomes in these groups improved? Provide sex￾disaggregated data for all individual level indicators. 6. Is there an adequate and skilled government health staff managing TB patient in USAID￾supported TB projects assisted sites? 7. Have USAID-supported TB projects been able to enhance information system at national and local levels? Programmatic Management of Drug Resistant TB (PMDT): 8. Have USAID-supported TB projects helped to increase the number of MDR-TB patients diagnosed? 9. Have USAID-supported TB projects helped to increase the number of MDR-TB patients initiating and successfully completing treatment? (Male/Female) 10. Have USAID-supported TB projects assisted in the expansion of treatment centers and in decentralization of MDR-TB management? 11. What social support interventions for PMDT, if any, have USAID supported TB projects supported? What has been the result? Advocacy, Communications, Social Mobilization: 15. Have misperceptions/misconceptions and stigma related to TB among the general population decreased? 16. Are more persons with TB symptoms seeking treatment at public facilities? (M/F) 17. Is communication material available at DOTS facilities? If so, has it made an impact? Performance Evaluation F. Project achievement G. Are expected results occurring? H. Questions pertinent to Program design I. Implementation process/ Questions pertinent to management/operational decision making J. How it is perceived and valued? 80 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 18. In site questions: a. Are providers, in USAID-supported TB projects, able to provide TB related information and counseling? b. What is the involvement of communities in USAID-supported TB projects? TB Laboratory Strengthening and Diagnostic Network: 19. How have the USAID-supported TB projects contributed to TB laboratory strengthening nationally? Locally? 20. How have the USAID-supported TB projects contributed to the quality of TB smear microscopy labs in local settings? Are they staffed by trained microscopists? 21. How have the USAID-supported TB projects contributed to rational lab diagnostic strategies at national level? At regional levels? 22. How have USAID-TB projects contributed to increasing access to TB diagnosis (labs, x-rays, etc.)? Anti-TB Drug Supply and Network: 23. Have there been any TB drug stock-outs during the 5-year life of the USAID-supported TB projects? 24. USAID-supported TB projects have contributed to strengthened: Topic (1) Strongly disagree (2) Disagree (3)Neutral (4)Agree (5) Strongly agree c. Information d. TB drug management system 25. On scale of 0-5, where 0 (zero) means “no change” and 5 (five) being “the most change”, how do you quantify the contribution of USAID-supported TB projects on: Topic 0 1 2 3 4 5 e. Forecasting f. Procurement g. Distribution h. Utilization 26. USAID- supported TB projects have contributed to improved: Topic (1) Strongly disagree (2) Disagree (3)Neutral (4)Agree (5) Strongly agree c. Staff capacity to monitor the availability and quality of anti-TB drugs d. Staff capacity has improved USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 81 Local TB Control Program Capacity Building/Strengthening: 27. What has been the support to the local TB programs in terms of capacity building and technical assistance (guideline, policy development, training, etc.)? a. What are the needs that have not been met? 28. What local policy instruments and operational improvement tools can be attributed to the USAID-supported TB projects? 29. What is the relationship between the local health facility staff and the various USAID TB projects? 30. To what extent have USAID-supported TB projects contributed to high quality technical assistance at the local level? 31. To what extent have USAID-supported TB projects contributed to capacity building of community based organizations (CBOs), non-government organizations (NGOs), and civil societies in participating in local TB control initiatives? Sustainability Mechanism: 32. Have the projects developed a process to ensure sustainability of the results after the project life? 33. What are the plans of the MOH and local governments for sustaining results developed under the projects? Gender 34. How have gender considerations been integrated in USAID’s TB programs and TB-related activities? 35. What are the differential effects of the project on male and female beneficiaries? 36. What gender issues were identified in the USAID TB programs and how were they addressed? 37. What was lacking in the project design and/or implementation that would have improved gender considerations? Overall 38. Accomplishments gained from USAID’s investment in TB activities in Brazil and lessons learned from the past decade in terms of:  Policy change  DOTS adaptation  Project sustainability 39. What has been the effect of USAID-funded projects on:  TB integration into the primary health care network  Combined approaches to TB/HIV co-infection,  Capacity building across the spectrum of health care professionals involved in service delivery  Health systems strengthening 82 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 INTERVIEW QUESTIONS: NATIONAL TB PROGRAMME General TB DOTS Interventions: 23. USAID-supported TB projects have contributed to improved capacity of PNCT to: Topic (1) Strongly disagree (2) Disagree (3)Neutral (4)Agree (5) Strongly agree a. Advocate for DOTS in Brazil b. Enforce Dots in Brazil c. Monitor Dots in Brazil 24. USAID supported TB projects have contributed to enhanced information system: Topic (1) Strongly disagree (2) Disagree (3)Neutral (4)Agree (5) Strongly agree a. At national level b. At local level Policy, Financing and Regulatory Environment: 25. USAID-supported TB projects have strengthened capacity of PNCT in its regulatory functions: (1) Strongly disagree (2) Disagree (3) Neutral (4) Agree (5) Strongly agree 26. How? TB Laboratory Strengthening and Diagnostic Network: 27. How have the USAID supported TB projects contributed to TB laboratory strengthening nationally? Locally? 28. How have the USAID supported TB projects contributed to the quality of TB smear microscopy labs in local settings? Are they staffed by trained microscopists? 29. How have the USAID TB projects contributed to rational lab diagnostic strategies at national level? At regional levels? Anti-TB Drug Supply and Network: 30. What USAID supported TB projects have been implemented to support the TB drug management and distribution system? Are the results of these projects sustainable? 31. USAID-supported TB projects have contributed to strengthened: Performance Evaluation K. Project achievement L. Are expected results occurring? M. Questions pertinent to Program design N. Implementation process/ Questions pertinent to management/operational decision making O. How it is perceived and valued? USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 83 Topic (1) Strongly disagree (2) Disagree (3)Neutral (4)Agree (5) Strongly agree e. Information f. TB drug management system 32. On scale of 0-5, where 0 (zero) means “no change” and 5 (five) being “the most change”, how do you quantify the contribution of USAID-supported TB projects on: Topic 0 1 2 3 4 5 i. Forecasting j. Procurement k. Distribution l. Utilization 33. USAID- supported TB projects have contributed to improved: Topic (1) Strongly disagree (2) Disagree (3)Neutral (4)Agree (5) Strongly agree e. Staff capacity to monitor the availability and quality of anti-TB drugs f. Staff capacity has improved PNCT Capacity Building/Strengthening: 34. What support has been provided to PNCT in terms of capacity building and technical assistance (guideline, policy development, training, etc.)? What are the needs that have not been met? 35. How have the USAID-supported TB projects contributed to the development of policy instruments and programmatic tools? 36. What is the relationship between the PNCT staff and the various USAID-supported TB projects? 37. To what extent have USAID-supported TB projects contributed to high quality technical assistance at the national level? Sustainability Mechanism: 38. Have the USAID-supported TB projects developed a process to ensure sustainability of the results after the project life? 39. What are the plans of the MOH and local governments for sustaining systems and interventions developed under the USAID supported TB projects? Gender: 40. How have gender considerations been integrated in USAID’s TB programs and TB-related activities? 41. What was lacking in the USAID supported TB project design and/or implementation that would have improved gender considerations? 84 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 Overall 42. Accomplishments gained from USAID’s investment in TB activities in Brazil and lessons learned from the past decade in terms of:  Policy change  DOTS adaptation  Project sustainability 43. What has been the effect of USAID-funded projects on:  TB integration into the primary health care network  Combined approaches to TB/HIV co-infection,  Capacity building across the spectrum of health care professionals involved in service delivery  Health systems strengthening USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 85 INTERVIEW QUESTIONS: PATIENT KAP QUESTIONNAIRE (INDIVIDUAL) (Write a sentence explaining about the interview, why, confidentiality, privacy, anonymity, possibility to finish at any time, no personal identification, no influence in the treatment) Project title: ________________________ Study period: ___________________________ Interview date: ____/____/____ Interviewer: ___________________________ Interview site: ______________ Questions: 1. Gender: M/F 2. Age (years): _________ Thinking about your visit at the TB Clinic today, please respond the questions bellow regarding your disease and your treatment: 3. How many days did you wait to begin TB treatment since you searched for care in this Unit for the first time? 5. More than 30 days 4. Between 15-30 days 3. Between 8-14 days 2. Between 3-7 days 1. Less than 3 days 4. How satisfied are you with the information you received from the health care worker about your disease? 5. Very unsatisfied 4. Unsatisfied 3.Neutral 4.Satisfied 5.Very satisfied 5. Did you understand the dose and time schedule of medications after speaking with the health care worker? 1. Poorly 2.Fairly 3.Neutral 4. Well 5. Very well 6. Have you received direct supervision of your TB treatment in the unit? 1. Never 2. Infrequently 3.Sometimes 4. Almost always 5. Always 7. How courteous and attentive during your visits have been the health care workers? 86 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 INTERVIEW QUESTIONS: HCW KAP QUESTIONNAIRE (Write a sentence explaining about the interview, why, confidentiality, privacy, anonymity, possibility to finish at any time, no personal identification and no influence in the treatment) Project title: ________________________ Study period: ___________________________ Interview date: ____/____/____ Interviewer: ___________________________ Interview site: ______________ Questions: 1. Gender: M/F 2. Age (years): _________ 3. Professional position: _______________ Thinking about your work activities at this health facility please answer the following questions: 4. How frequently do you have problems in laboratory that make it difficult to administer the treatment to the patient? 1. Never 2. Infrequently 3.Sometimes 4. Almost always 5. Always 5. Do your patients recognize and follow your instructions? 1. Never 2. Infrequently 3. Sometimes 4. Almost always 5. Always 6. Do you have problems to supervise the treatment of TB patients? 1. Never 2. Infrequently 3. Sometimes 4. Almost always 5. Always 7. Do you think the training courses provided by USAID were important to improve the quality of your daily work? 1. Poorly important 2. Fairly important 3. Important 4. Very important 5. Extremely important 8. How important do you think it is to have TB prevention and treatment campaigns? 1. Poorly important 2. Fairly important 3. lmportant 4. Very important 5. Extremely important USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 87 QUESTIONS FOR TRANSFORMARTE (CIVIL SOCIETY) 1. Can you please explain what is Transformarte and which kind of work you do (what is the focus of the activities of Transformarte)? 2. When Transformarte have started? In which states/municipalities do you work? 3. Please give us an overview of TB social mobilization in Brazil and situate Transformate in this scenario. 4. How is the mechanism to receive funds for the activities? 5. Which other sources of funding mechanism for Transformarte do you have? 6. Did you receive any grants from Global Fund for civil society and TB work? If yes, please provide details on the funding and the project. 7. To which USAID implemented partners other than PAHO, MSH and Benfam have Transformarte been working with in partnership? Regarding the work of Transformarte in partnership with PAHO, MSF, Benfam (and other, if applicable) supported by USAID (please make sure we are focusing in everything related with USAID/Brazil-TB supported activities. All activities that is not supported by USAID should not be included), can you please tell me for each project in collaboration with each IP: 8. What were the objectives of this TB project with Transformarte? 9. What were the results of the project? Were these results sustainable? 10.What was the collaboration process with each IP like? 11.How is the experience to be subcontracted as an NGO by PAHO, MSH, Benfam (and others, if applicable)? 12.What were the positive aspects of the collaboration with each IP with Transformarte? 13.What issues of the project did not happen as it was expected by Transformarte? 14.Overall, how was the experience to work with many partners (PAHO, Benfam, MSH, etc.) at the same time? In terms of coordination, overlapping activities and management? 15.Have any gender considerations been included in Transformarte TB social mobilization activities supported by USAID/Brazil? 16.USAID/Brazil-supported TB projects have contributed to improved capacity of Transformarte to: Topic (1) Strongly disagree (2) Disagree (3)Neutral (4)Agree (5) Strongly agree d. Advocate (TB social mobilization) in Brazil e. Manage TB of Transformarte projects f. apply and to use other funding mechanisms to support Transformarte activities 88 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 89 ANNEX E: ILLUSTRATIVE EVALUATION QUESTIONS PERFORMANCE AND IMPACT:  What has been the overall impact of USAID TB programs on TB case notification and treatment outcomes in Brazil?  Is TB prevention and control in Brazil better now than it was before USAID interventions began in 2001? Compare areas with and with no USAID TB programs.  How has each of the TB programs and TB-related activities contributed to meeting the goals and objectives of the PNCT? OVERARCHING ISSUES:  How have projects coordinated with other partners (USAID/non-USAID, TB, other health and non -health partners)?  Describe the synergies of the TB projects with other development partners working for TB in terms of activities, support, sites, etc.  How and to what extent have partners’ capacity been strengthened? GENERAL TB DOTS INTERVENTIONS:  Have TB case detection increased in project sites? Nationally? Provide baseline, target and actual data sets.  Have TB treatment outcomes improved in project sites? Nationally? Provide baseline, target and actual data sets.  What attention has been paid to vulnerable populations, like indigenous groups, people with disabilities, geographically isolated and depressed areas, and congregate settings of the urban poor? Have TB case notifications or treatment outcomes in these groups improved? Provide sex-disaggregated data for all people level indicators.  Have TB projects been able to improve the capacity of PNCT to advocate, enforce and monitor DOTS in Brazil?  Are there adequate and skilled government health staff managing TB patients in USAID assisted sites?  Have TB projects been able to enhance information system at national and local levels? POLICY, FINANCING AND REGULATORY ENVIRONMENT:  Were the TB project-supported partners able to develop and implement local policies to support the program?  Do TB project-supported partners have increased financing for TB?  Were the TB projects able to leverage resources provided by the national and local governments?  Are there more TB patients availing themselves of DOTS benefits in the project sites? (M/F)  Have TB projects strengthened the capacity of PNCT in its regulatory functions? How? 90 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 PROGRAMMATIC MANAGEMENT OF DRUG RESISTANT TB (PMDT):  Have USAID TB projects helped to increase the number of MDR-TB patients diagnosed?  Have USAID TB projects helped to increase the number of MDR-TB patients initiating and successfully completing treatment? (M/F)  Have USAID TB projects assisted in the expansion of treatment centers and in decentralization of MDR-TB management?  What social support interventions for PMDT, if any, have USAID TB projects supported? What has been the impact? ADVOCACY, COMMUNICATIONS, SOCIAL MOBILIZATION:  Are DOTS facilities better utilized?  Have misperceptions/misconceptions and stigma related to TB among the general population decreased?  Are more persons with TB symptoms seeking treatment at public facilities? (M/F)  Is communication material available at DOTS facilities? If so, has it made an impact?  Are participating providers able to provide TB related information and counseling?  Are communities in USAID assisted sites more involved in TB control and prevention activities? TB LABORATORY STRENGTHENING AND DIAGNOSTIC NETWORK:  How have USAID TB projects supported TB laboratory strengthening nationally? Locally?  Has external quality assurance of smear microscopy indicators improved over the past 5 years?  Do TB suspects have adequate access to TB diagnosis (labs, x-rays, etc.)?  Are there quality TB smear microscopy labs in local settings? Are they staffed by trained microscopists?  Have the projects supported rational lab diagnostic strategies at national level? At regional levels? ANTI-TB DRUG SUPPLY AND NETWORK:  Have there been any TB drug stock-outs during the 5-year life of the USAID-supported TB projects?  What USAID-supported interventions have been implemented to support the TB drug management and distribution system? Are these interventions sustainable?  Have the projects supported information and TB drug management systems (forecasting, procurement, distribution, utilization) strengthening to improve the capacity of the national and local staff to monitor the availability and quality of anti-TB drugs? PNCT CAPACITY BUILDING/STRENGTHENING:  What has been the support to PNCT in terms of capacity building and technical assistance (guideline, policy development, training, etc.)? What are the needs that have not been met? USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 91  What policy instruments and programmatic tools have been developed and can be attributed to the USAID TB projects?  What is the relationship between the PNCT staff and the various USAID TB projects?  To what extent have USAID TB projects contributed to high quality technical assistance at the national level? LOCAL TB CONTROL PROGRAM CAPACITY BUILDING/STRENGTHENING:  What has been the support to the local TB programs in terms of capacity building and technical assistance (guideline, policy development, training, etc.)? What are the needs that have not been met?  What local policy instruments and operational improvement tools can be attributed to the USAID TB projects?  What is the relationship between the local health facility staff and the various USAID TB projects?  To what extent have USAID TB projects contributed to high quality technical assistance at the local level?  To what extent have USAID TB projects contributed to capacity building of community based organizations (CBOs), non-government organizations (NGOs), and civil societies in participating in local TB control initiatives? SUSTAINABILITY MECHANISM:  Have the projects developed a process to ensure sustainability of the systems and interventions after the project life?  What are the plans of the MOH and local governments for sustaining systems and interventions developed under the projects?  What are the plans for sustaining and/or increasing the locally generated funding for TB control? GENDER:  How have gender considerations been integrated in USAID’s TB programs and TB-related activities?  What are the differential effects of the project on male and female beneficiaries?  What gender issues were identified in the USAID TB programs and how were they addressed?  What was lacking in the project design and/or implementation that would have improved gender considerations? LESSONS LEARNED:  What are the overall lessons learned from the implementation and evaluation of USAID TB and TB-related projects?  What are the best practices from TB programs that could be adopted and replicated by follow-on activities? 92 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 93 ANNEX F: IMPLEMENTING PARTNER’S PROJECT EFFECTIVENESS A. MANAGEMENT SCIENCES FOR HEALTH (MSH) PROJECT TITLE: MANAGEMENT AND LEADERSHIP PROJECT Cooperative Agreement Number: HRN-A-00-00-00014 Implementation Period: June 2001 to January 2004) Location: National; Rio de Janeiro Total Funding: $554,631 Context In June 2001, MSH began support to Brazil's TB program with mission funding. This project was one of the earliest TB interventions by USAID/Brazil. The mission provided field support to the Management and Leadership Project cooperative agreement managed at USAID headquarters in Washington, DC. Objectives of this project were to:  Strengthen the capacity of the MOH National TB Program to provide political and administrative support for TB control;  Strengthen the capacity of the Rio de Janeiro State Secretariat of Health TB Program (Rio de Janeiro SES PCT) in order to expand and implement DOTS. Findings The focus of this project was on building the capacity of national and state TB authorities. A total of eight workshops, seminars and conferences were sponsored, which trained approximately 500 national and state TB staff. Training topics included management skills, use of epidemiological data for program management, and planning and organizing TB services. The project worked with Duque de Caxias Municipal Secretariat of Health staff to implement DOTS in a municipal health center and a family health team health center. Performance improvement instruments were developed to operationalize and monitor DOTS in the health services and train staff for DOTS implementation. It was expected that these tools would be used by the State TB Program in the expansion of DOTS in other municipalities. It is uncertain if these tools were so used, as State authorities interviewed, for this evaluation had no knowledge of the project. After the project ended, the mission supported PAHO’s work in the municipality. No meeting took place between the two organizations to discuss the transfer. Further, the project staffer responsible for providing technical assistance to the municipality was supposed to be hired by the State TB coordination but did not get hired to continue the work. Lessons learned about integration of DOTS from the experience in Duque de Caxias include the importance of training family health teams and including them as partners in improving performance. Supporting health teams with management systems and tools will increase success. These lessons are still relevant today. This project did a great deal of work to improve staff capacity, particularly at the state and municipal level. However, due to high TB staff turnover, it is unclear to what extent the tools, systems and capacities developed were sustained by the government after project end. 94 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 Achievements A lasting legacy includes the Rio de Janeiro State NGO TB Forum set up with the support of the project that continues today. It was developed to address the expressed need by community organizations to create a community space for development of common community policies towards the public policy on TB and monitoring of public resources for TB. In fact, the Forum meets monthly with 70 NGOs to promote trainings and meetings about TB, including promotion of World TB Day. The Forum attracts the involvement of community leaders, health professionals and managers. Management Overall, coordination of this project encountered some difficulties. At times, it was difficult to meet with TB staff that had low wages at the time. Their various activities outside of work made collaboration challenging. Limitations Given the short time frame of the project, there was no opportunity for follow-up on the trainings to determine the extent to which behavior change occurred among staff. Given that this project ended in 2003, it was difficult to locate full documentation of the project activities. For example, it was not possible to obtain participant lists to request interviews with those who had received training. Financial reports on detailed expenditure of funds were unobtainable. Upon the completion of the cooperative agreement, project records were sent for storage at MSH offices in USA. Some work plans were provided to the evaluation team, but it was not possible to confirm that these planned budgets were approved and implemented by the project in the absence of final financial expenditure reports. PROJECT TITLE: RATIONAL PHARMACEUTICAL MANAGEMENT PLUS (RPM PLUS) Cooperative Agreement: HRN-A-00-00-00016-00 Obligation Period: 2003-2010 Implementation Period: 2004-2006 Location: National, Rio de Janeiro Total Funding: $2,998,000 Context In 2003, USAID/Brazil began support of the Hélio Fraga Tuberculosis Reference Center (CRPHF) which is responsible for developing, analysing and transferring technologies to combat 1st and 2nd line drug resistant resistance TB infection in the country and to monitor the cases of multi-drug resistant TB (MDR-TB). USAID/Brazil started providing technical assistance to CRPHF through the Rational Pharmaceutical Management Plus Program (RPM Plus) managed by Management Sciences for Health (MSH) in 2004, followed by SPS and SIAPS projects. Main Objectives (1) Decentralize the diagnosis, treatment and case management of MDR-TB patients to state reference centers in 27 states; (2) Develop a product quality testing system for TB drugs already distributed within the Ministry of Health system; (3) Switch to fixed dose combination (FDC) TB products which strongly promotes Direct Observed Therapy Short-course (DOTS) (4) Replace the TB regimen for re-treatment cases with less toxic TB drugs. USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 95 Findings A web-based surveillance system (DMIS) was created in 2004-2005 that started implementation in 2006 as a response of CRPHF demand for a MDR-TB management system. As MDR-TB patient numbers increased, the national epidemiological surveillance model needed to be decentralized to state and regional reference centers, which occurred progressively using this web system. Since its creation in 2004, DMIS has been improved and the name has been changed: from TB MR to e￾TB manager and SITE TB, which is the latest version in 2011. RPM Plus provided technical assistance to CRPHF through local experts knowledgeable in computer manipulation, clinical aspects of MDR-TB and trainings to carry out implementation of the management information system for improving diagnosis, treatment and management of MDR-TB cases. The components of DMIS included case notification data sheet, trimester follow-up data sheet, follow-up data sheet after treatment success (“cure”), request for MDR-TB medicines, trimester report on stock turn over for MDR-TB medicines, standards and customized reports and data extraction tool. Trainings were conducted on an ad-hoc basis countywide to increase capacity building on system procedures and flows and to review the MDR-TB medical charts in order to update the patient information in the DMIS database (Annex Figure 1). Annex Figure 1: Evolution of the data entries in the new MDR TB system (from RPM Plus - USAID updated December 2006) The current validated functions of DMIS are being used to extract epidemiological information (reports, tables and graphics) to be employed in trainings and conferences and also for use by the MoH for MDR-TB surveillance as well as to be published in national and international scientific journals. Besides tracking clinical and epidemiologic information on patients, DMIS can manage and control the forecast and distribution of second line drugs that are sent to different centers by the CRPHF Pharmacy. It is very helpful to monitor not only TB MDR cases and alternative schemes on a routine basis, but also non-tuberculosis mycobacteria, which is innovative in Brazil. Some constraints observed were not directly associated with DMIS, but related with health professionals’ gaps during the hard copy filling of the TB sheets that later on are inserted in the electronic system. However, in some states, slow internet connection is an issue that difficult the use of DMIS. It is important to highlight that Brazil has its own National Disease Notification System (SINAM), which includes information of all compulsory notification diseases including TB, but no information of TB MDR. Currently, no automatic integration is observed between SINAN and DMIS. Even 96 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 though, it is possible to manually check whether a TB patient had SINAN and DMIS notification based on the unique patient reference number. Responding to an urgent request of CRPHF and National Quality Assurance Institute (INCQS) to implement quality systems according to ISO certification norms in its built facility, RPM Plus started working with the reference laboratory at CRPHF in partnership with INCQS since 2004, based on MOU with CRPHF and INCQS. A new tool was developed in partnership with INCQS for Quality Systems Implementation in Drugs Quality Testing Labs helping moving reference labs to ISO accreditation, called LABMOST. Consistent application of the tool led to the accreditation of National Quality Assurance Lab according to ISO 17025 and strengthening of State Lab Network - 12 state labs have the capacity to perform quality testing of TB drugs permanently. Moreover, the MoH working group uses this model created for TB for other programs. So, it represents an important support for not only TB, but also other transmissible diseases, thus maximizing the impact of USAID’s investment. For quality control, Farmanguinhos, supported and funded by the Project conducted bio￾equivalence studies for 2 in 1 FDCs in partnership with Institute of Research Evandro Chagas (IPEC/Fiocruz). RPM also supported MoH for the publication and dissemination of new TB control recommendation. Training material elaboration (handbooks, clinical cases for exercises and folders, Annex Figure 2) for supporting roll-out of new TB treatment guidelines countrywide and 5 clinical management regional workshops were conducted for each region of Brazil (North, Northeast, Central West, Southeast and South). A total of 29 seminars and 2.884 (723 men & 2161 women) health workers were capacitated. Annex Figure 2: Handbooks and training materials to disseminate new TB control recommendations. Main Achievements  Developed LABMOST in partnership with INCQS. This tool helps to move reference labs towards ISO accreditation.  RPM Plus contributed to strengthening technical interaction between national producers and National Regulation Authority for product quality testing.  RPM supported MoH for the publication of training materials (handbooks, clinical cases for exercises and folders) in order to implement new TB treatment guidelines countrywide.  Collaboration with the Brazilian national medicine manufacturer, Farmanguinhos, and develop FDCs: 4 in 1 (RHZE tablets), 2 in 1 RH tablets (150/75 and 300/150 mg) and Isoniazid 300 mg tablets for latent TB infection treatment (ILTB). USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 97  Potential production for national and international markets is in discussion, putting Brazil as a supplier for PAHO Revolving Fund. Sustainability Farmanguinhos has made several enhancements to its facility, such as acquisition of new higher performance manufacturing equipment to produce FDCs and paediatric sachets and validation of new methodologies like capillary electrophoresis to support further process of FDCs production. PROJECT TITLE: STRENGTHENING PHARMACEUTICAL SYSTEMS (SPS) Cooperative Agreement: GHN-A-00-07-00002-00 Obligation Period: 2007-2011 Implementation Period: 2007-2011 Location: National Total Funding: $3,748,000 Context SPS project represent the continuation and consolidation of RPM plus project funding aiming to improve TB/DR- TB control in Brazil and overall drug management. Main Objectives 1. Support the move to fixed dose combination (FDC) regimens and the local production of new pharmaceutical forms for first-line TB treatment. 2. Strengthen DR-TB control and assist NTP and key partners to respond to upcoming MDR/XDR challenges 3. Strengthen the laboratory network to improve TB and DR-TB control Findings Objective 1  The Project, with the NTP, created a Task Force to develop and conduct interactive training in all 27 states to support the changes to the treatment regimen.  The Project provided technical assistance for successful development of new formulations for 2 in 1 and 4 in 1 FDC with the manufacturer Farmanguinhos  Instituted a product quality assurance testing program for FLD and SLD. Objective 2  Since its creation in 2004, DMIS has been improved and the name has been changed: from TB MR to e-TB manager and SITE TB, which is the latest version in 2011. It has been implemented and used in all Brazilian DR-TB centers over the country and is now the MoH’s official system for DR-TB management at a national level  This initiative contributed to increasing detection and cure rates of MDR-TB cases and significantly strengthened clinical and case management practices at the MDR-TB reference center level. The DMIS implementation has also enriched the quality of information of TB patients, offering a comprehensive database for further research and scientific publications. Annex Figure 3 shows the increased TB notification rate from 2000 to 2011, and Annex Figure 4 demonstrates the improvement in TB outcomes across the years. 98 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 Annex Figure 3: Number of TB notified cases between 2000-2011, CRPHF. Annex Figure 4: TB treatment outcome across 2001-2010.  SPS helped expand national coverage of Brazil’s DR-TB treatment centers by increasing the number of DR-TB treatment centers from 62 to 154.  The Brazilian DR-TB cure rate has increased to 62.2 % (2008), with a marked decrease in mortality and treatment failure rates. Objective 3  The Project supported the national public health lab network to meet international standards and also to implement quality systems according to ISO norms.  CRPHF assumes its national reference role within the national laboratory network, with the implementation of an EQA program for Smear Microscopy in place for 27 Lacens. This has resulted in 12 out of the 27 public laboratories now being regularly supervised for culture and DST.  As a result of these achievements, after an audit of CGLAB/MoH, its status as a national TB reference laboratory was re-accredited in 2009. Additionally, the CRPHF agreed to begin work with SPS to raise the laboratory’s accreditation to ISO norm 17025. USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 99  The project helped establish a network of private and public partnerships with several international organizations. These include FIND, the BMGF, the Union, the Brazilian TB– Network, MoH, CRPHF, Beckton and Dickinson, BioMérieux and several other international laboratories. These partnerships have enabled SPS to introduce new rapid tests for TB using molecular biology from companies such as GeneXpert and HainlifeScience. Achievements  The project helped to increase the number of MDR-TB Reference Centers from 63 in 2004 to 138 in 2012. Sustainability SITE TB is now managed by the health surveillance department at the National School of Public Health/Fiocruz. Technical and managerial steps have been defined to ensure a fully functional transfer and subsequent sustainability. Financial Breakdown by years Annex Figure 5: SPS Funds by Fiscal Year, FY 07 to FY 09. PROJECT TITLE: SPS MULTI-COUNTRY ASSOCIATE AWARD (2010) Cooperative Agreement: OAA-LA-10-00002 Obligation Period: 2010-2011 Total Funding: $1,673,000 Strengthening Pharmaceutical Systems Program, Brazil. TB/HIV Activities: October 2010 to September 2011 Implementation Period: October 2010 to September 2011 Location: Municipalities of the Southern Region: Rio Grande do Sul, Santa Catarina and Paraná states Total Funding: USD 750,000 Context The HIV incidence rate in Brazil is increasing in all regions, mainly in the South Region, where the HIV incidence rate increased from 5.8 in 1997 to 14.1 in 2007. The South Region also showed the highest increase in the mortality rate caused by AIDS in the last years. Additionally, Porto Alegre, 100 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 the city with the biggest population in Rio Grande do Sul State, presents the highest mortality rate by TB in Brazil. Although a strategic plan and national policy for TB/HIV existed at that time, few TB/HIV related activities were implemented. In order to face these challenges, the TB/HIV project was developed by the National TB/AIDS Program in partnerships with the National Tuberculosis Control Program (NTP). The Project facilitated and co-ordinated with all partners the project execution. The project selected priorities cities in Rio Grande do Sul, Santa Catarina and Paraná states to implement TB/HIV proposed activities, according to epidemiological criteria and focusing on SAEs for TB/HIV integration activities. The project had a one-year period of performance, but due to initial delays it was extended until March 2012. Main Objectives 1. Increase TB case finding among PLWHA 2. Implement Isoniazid Preventive Therapy (IPT) 3. Ensure TB infection control measures in SAE 4. Strengthen the Advocacy, Communication and Social Mobilization (ACSM) component in partnerships with the civil society 5. Develop a TB/HIV surveillance system to be used in SAE Findings  A TB infection control guide was developed and disseminated and consultants supported the implementation of infection control activities, including local biosafety.  Biosafety workshops were conducted for all targeted SAEs together with the adaptation of the facilities according to the recommendations.  An educational campaign based on the “3Is” (“Essa dupla não combina”, Annex Figure 6) was developed and educational materials (video, folders, posters, t-shirts, buttons, stickers, educational booklets, and bus outdoor) were created in partnership with PLWHA and community leaders in order to engage civil society and strengthen the advocacy, communication and social mobilization (objective 4). Annex Figure 6: Educational Campain “Essa dupla nao combina”. Management The TB/HIV project was very challenging from the beginning, since it relied on coordination and integration of TB and HIV Programs, which historically were vertically orientated. The project was instrumental in mediating dialogue between the two technical areas at the national level and at the state and municipal levels. USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 101 Achievement  TB/HIV Work Group at MoH was officially created, representing a step forward in the integration process of these two programs. Another successful result of the TB/HIV integrated approach was the inclusion of TB/HIV co-infection in the HIV/AIDS and in the TB national guidelines and the inclusion of TB drugs into HIV/AIDS drug management system (SICLON). Moreover, TB drugs for TB treatment became available at SAE level. So as the TB case findings increased among PLWH and more TB cases are diagnosed, it became possible to treat them at SAE level, avoiding referring the patient who has TB and HIV to be treated in two different facilities.  A TB clinical screening tool was developed and implemented at SAE level to facilitate periodical screening for active TB to all HIV patients. This questionnaire consisted of 5 main questions to increase the suspicion of TB diagnose in HIV patients (Annex Figure 7). Trainings of health professionals were also carried out to increase awareness and to disseminate the use of the new tool for TB clinical screening. Clinical trainings for management TB/HIV patients were also conducted for doctors and nurses. Annex Figure 7: Stamp questionnaire for TB screening at SAE level.  Santa Catarina has an electronic medical record system for all patients, which are also linked with laboratory. The questionnaire for TB screening was developed and included in the electronic medical records. Florianopolis and Itajaí also developed and incorporated in the existing electronic medical record system information about IPT.  At SAE level, nurses were trained for TST application, reading and interpretation. Awareness of SAE professionals on IPT for PLWHA was also raised during the trainings. They were also trained on IPT follow up and adherence. Isoniazid access at all levels was assured and the implementation of isoniazid for TB latent infection (IPT) has been gradually occurring (Objective 2).  The number of PPD and IPT has been increased in most of the selected facilities, especially considering that before 2011 very few PPD and IPT were provided at SAE level. In São Leopoldo, PPDs provided at SAE more than doubled from 2009 to 2012, from 101 to 242. In Porto Alegre, similar trend was also observed (Annex Figure 8). 102 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 Annex Figure 8: Number of PPD, PPD≥5mm and IPT provided at IAPI, Porto Alegre, 2008 to mid-2012. Note: Total number of HIV patients each year was not provided. Limitations  In Rio Grande do Sul state, 10 municipalities were initially selected. However, the project was not implemented in those original municipalities as planned, since 8 of the 10 municipalities selected had no progress in the proposed activities. The main reason was a lack of human resources. Eventually, only Porto Alegre (IAPI and CSVC) and São Leopoldo (Policlinica) municipalities remained in the project. In Paraná, 2 port cities, Itajaí and Paranaguá and the capital city of Santa Catarina State (Florianópolis), cities with high burden for TB and HIV infection, were also selected.  Although Project support was instrumental in promoting dialogue and collaboration between the two technical areas at the national, state and municipal levels, it was very short. There was insufficient time to observe major progress and results.  Lack of political commitment and support by the state constrained the integration of TB and HIV/AIDS programs. In the absence of this support, municipal leadership was the deciding factor for program integration. Financial Breakdown by activities Annex Figure 9: SPS TB/HIV Activity Budget Spend, October 2010 to September 2011. Total: USD 750,000,000 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 103 Sustainability TB/HIV workgroup is continuing through the MoH and formal recommendations for all Brazilian States will be launched to adopt the integration between TB and HIV/AIDS using the same strategy adopted in South states. Moreover, campaign materials will be disseminated to all other States of Brazil and publication of the successful experiences will be conducted. PROJECT TITLE: SYSTEMS FOR IMPROVED ACCESS TO PHARMACEUTICALS & SERVICES (SIAPS) Cooperative Agreement: OAA-A-11-00021 Obligation Period: 2011-2012 Implementation Period: 2011-2016 Location: National Total Funding: $2,100,000 Context Given the planned closure of the USAID/Brazil health program, SIAPS focuses its strategy on technical and financial support in consolidating successful intervention with the possibility of long￾term sustainability. Main Objectives The overall objective is to strengthen TB/DR- TB control allowing MSH and partners’ key activity achievements to become fully sustainable within the Brazilian Health System. The three main objectives to focus on are as follows: 1. Strengthen TB pharmaceutical management and information systems 2. Support TB state programs in strengthening DOTS and community DOTS implementation 3. Strengthen Hélio Fraga National TB Reference Center activities and the Public Health TB Laboratory network Current responsibilities that are still under MSH control are planned to be transferred as early as possible to partners. All trainings and capacity building activities will be conducted on a training of trainers (TOT) basis with a specific focus in training key partners/public employees capable of replicating developed methodologies (including e-learning platforms where possible). Findings The project has recently started and under implementation. For this reason, no final reports are available for this evaluation. 104 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 Financial Breakdown by Activities: Annex Figure 10: SIAPS Activity Budget Spend, October 2011 to September 2012. Limitations Overall MSH projects In general, reporting has not been strictly along the lines of the individual projects but rather according to work streams. Some activities of each project overlap, such as RPM plus activities developing through SPS and continuing with SIAPS. No final reports for RPM plus and SPS projects were available for this evaluation, including financial information by activity. No quarterly report was available for SIAPS project. MSH keeps records for only 5 years. All activity reports and project work plans were reported and approved each year by the USAID Mission in Brazil and sent to USAID Washington. There were some issues to recover these files for this evaluation. B. PAN AMERICAN HEALTH ORGANIZATION (PAHO) PROJECT TITLE: EXPANSION OF DIRECTLY OBSERVED TREATMENT SHORT-COURSE STRATEGY IN BRAZIL Cooperative Agreement: 512-G-00-06-00024-00 Obligation Period: September 29, 2006 –January 2013 Implementation Period: 2006-2013 Funding: $9,000,000 Location: Rio de Janeiro (RJ) state; São Paulo (SP) state; Manaus, Rio Grande do Sul, Federal District, Bahia, and Minas Gerais Local partners: National level (Ministry of Health NTP and DN) and Secretariats of Health of priority States, São Paulo (SP) Rio de Janeiro (RJ) Rio Grande do Sul, Federal District, Bahia, Minas Gerais and municipality of Manuas, Non-Governmental Organizations (NGOs): Transformarte Context Initially, PAHO was selected as a key implementing partner of USAID to provide technical assistance for TB as it is a multilateral agency which would ease entry into the field of TB in Brazil. Total: USD 2,100,000 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 105 PAHO/ Brazil has worked closely with PAHO headquarters in Washington, as well as WHO headquarters in Geneva, to influence change and encourage health authorities at the federal, state and municipal levels to adopt international standards of care to improve case detection and treatment outcomes. To influence policy, PAHO/Brazil worked with federal authorities to encourage adoption of the DOT strategy. As noted earlier, at the inception of USAID/Brazil’s TB program, the country was strongly against adoption of the DOTS strategy, including directly observed therapy and sputum smear microscopy for case detection. Concurrently, states had a great deal of autonomy (and still do) to implement their own health interventions. A multi-pronged approach was selected as the best method for national adoption of DOTS. Although Brazil adopted DOTS as a national strategy in 200X, standard operating procedures (SOPs), clinical manuals and guidelines still needed to be developed and finalized, in accordance with the new policy. In addition, staff had to be trained, supervised and mentored to implement the new procedures, including DOT, patient reporting and registration, and cohort analysis. Objectives The original project, started in 2006, sought to integrate TB control into primary care, including the Community Health Agents Program (PACS) and the Family Health Program (PSF) to ensure effective expansion of the access to diagnosis and treatment. It had three specific aims: 1. To improve the management of TB cases and laboratory capacity for TB diagnosis in the states of Rio de Janeiro and São Paulo to help reach the WHO targets, 2. To mobilize civil society in order to create increased public awareness about TB’s symptoms and treatment while decreasing stigma, 3. To carry out surveillance of drug resistant tuberculosis (DR-TB) and the prevalence of HIV infection in TB patients in states with the highest disease burden. Later, modifications to the grant included additional objectives and over time, funds included support to Minas Gerais and the western district of Manaus in Amazonas state. Findings The PAHO/Brazil TB project has undoubtedly made important contributions to policy, protocol and practice for the integration of DOTS at the primary health care level in Brazil. Site visits and key informant interviews confirm the valuable assistance. The Project provided financial support to contract nurse supervisors and other program staff, rent vehicles for delivery of laboratory samples, convenes meetings, training workshops, conduct on-site supportive supervision and develop media campaigns with IEC materials. Yet, adequate Project documentation is lacking. It is not possible to accurately measure to what extent Project objectives were met. Major Achievements:  PAHO/Brazil, in collaboration with other partners, supported both the 2nd national DRS and the resulting inclusion of Ethambutol as a fourth drug for the first phase of treatment in accordance with international standards.  Recruiting of a task-force formed by 9 professionals, to reinforce the activities of the state staff in the actions of Plan and to implement DOTS in 12 municipalities of Rio de Janeiro state and 3 municipalities of São Paulo State. Besides, a task-force of 9 qualified nurses was created for monitoring the plans implemented for DOTS.  Updating of registration instruments: TB registry, TB symptomatic book, cards for supervised treatment and elaboration of a monitoring and supervision guide. 106 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012  PAHO assistance was essential to support the successful application to the Global Fund Round Planned activities for DR-TB and HIV infection surveillance in the states of São Paulo, Rio de Janeiro, Rio Grande do Sul, Federal District, Bahia, and Minas Gerais included: (1) hiring personnel to collect samples, perform tests and analyses, (2) supervision visits, (3) transportation of samples. Supervisory visits were carried out to the state laboratories (LACENs) that participated in the 2nd Survey of Anti-tuberculosis Drug Resistance in the six states. Local research coordinators were hired to assist with the survey. Transportation of samples was paid for in Bahia and Rio de Janeiro. Although the planned activities were carried out, it is unclear if they met targets or not. Social mobilization for greater TB awareness and stigma reduction was planned to include: (1) street theater presentations in low income com nm munities, (2) programs on community radio and TV, (3) seminars for young people and community leaders. While a 2008 quarterly report mentions planning meetings for these activities, it is unclear if and when they were carried out. Following is more detailed discussion on the major investments in the states of São Paulo and Rio de Janeiro, with high disease burden. São Paulo The Project sought to improve management and laboratory capacity in the state with the planned interventions to strengthen “TB WEB”, the new TB information system, provide management courses for TB program managers, train health care workers, monitor and supervise DOTS, as well as improve lab quality control with increased use of quality sputum smear microscopy and sputum cultures. Although it was confirmed that these activities occurred, it is not possible to determine if the planned number of health care workers were trained, of if fewer or more were trained, due to lack of documentation. Some other planned activities were not documented to have occurred, such as examination of contacts or infection control measures. Major achievements:  Implementation of the active search of TB suspects, in the routine of the basic health units.  Maintenance and expansion of laboratories with quality control of the microscopies, with concordance in the reading of greater of 95%  Increase in the coverage of patient in DOT. In the period from October 2006 to September 2007, 81% of the new smear positive cases were on DOT in comparison with 66.2% of the previous period.  Cure rate close to 80% in 2005 and 2006, among new cases.  Although still high at around 14%, default rates have fallen compared to previous years  TB database (TB WEB) with less inconsistency and improved completeness.  In the district of Itaim Paulista and Carapicuíba positive changes have been documented in this small area. DOTS training, along with sample transportation logistic support and provision of internet access to GAL contributed significantly to improve TB program. Rio de Janeiro In Rio de Janeiro state, planned interventions occurred such as (1) development of DOTS implementation plans for 12 high disease burden priority municipalities, (2) training health professionals in the five elements of the DOTS Strategy, (3) increase the identification of coughers, (4) hiring of 10 nurse supervisors to supervise, monitor, and evaluate the implementation of DOTS in the 12 municipalities with the highest disease burden, (4) increase the USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 107 use of sputum smear microscopy, (5) renting nine vehicles for supervision visits, (6) training in the use of the TB registry, treatment card and coughers registry. The extent to which activities occurred is unclear, as some activities were not specified with targets compared to actual occurrence, such as the number of trainings planned compared to the number that were conducted. In Rio de Janeiro Advocacy, Communication and Social Mobilization activities were conducted in partnership with PAHO/Brazil, Global Fund, NGOs and civil society. Activities included street theater, radio messages, a group of community health agents of the municipality of São João de Meriti performed a RAP on TB and used a ventriloquist to communicate information on TB, as well as a public hearing in the legislative assembly was held. Although target has not been met in all 12 municipalities, improvements in TB control are noted such as the information system, laboratories, logistics and the distribution of drugs. DOTS as a control strategy has been heavily promoted and acknowledged in the municipalities. In particular, DOT has not yet reached 100% of patients in these municipalities, but there has been change in a positive direction. Major achievements:  In 2009, in all but two of the 12 municipalities (Japeri and Quemados), case detection rate was 70% or better.  Provision of DOT improved in all 12 municipalities. Whereas in 2005, only one of the municipalities provided DOT, all 12 municipalities provided DOT in 2010. However, provision ranged from over 95% in four municipalities and less than 20% in another four municipalities.  In 2005, only one municipality had a cure rate 85% or higher, in 2009 that number had risen to four.  From 2005 to 2008, average default in the municipalities steadily decreased from 14.5% to 11.5%. Yet, in 2009, it increased again to 13.1%.  The nurse supervisors hired by PAHO fulfilled an instrumental role in the whole process of supervision and monitoring of municipal actions, supporting and guiding towards improvement. Financing Mission funds were first directed to PAHO/Brazil via the TBCTA agreement (see section on TBCTA) then later a grant agreement was signed between the mission and PAHO. The original agreement was planned for one year in the amount of $1,160,840, with a counterpart contribution of $105,000 from PAHO/Brazil, for a total of $1,265,840. Activities were to be split among the following three areas: 1. DOTS Expansion in Rio de Janeiro state = $331,765 2. DOTS Expansion in São Paulo state =$453,218 3. Surveillance for DR-TB and HIV =$375,857 In later years, additional funding was added and the scope of work (SOW) expanded with additional indicators required for reporting. Total budget of $9,441,000 (with $441,000 contribution from PAHO) for entire award period of September 2006 to September 2013: As of the most recent financial report received, dating 31 March 2011, the majority of funds were spent for DOTS expansion in São Paolo and Rio de Janeiro in accordance with the budget, as depicted in Figure X below: 108 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 Annex Figure 11: PAHO Funding Disbursement, Sept. 2006- Sept. 2011, $5,499,748.55 Total: $5,499,748.55 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 109 Annex Figure 12: Expansion of DOTS: Rio de Janeiro, Funding disbursement, Sept. 2006- Sept. 2011 Annex Figure 13: Expansion of DOTS, São Paulo, Funding Disbursement, Sept. 2006- Sept. 2011. 110 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 Annex Figure 14: Nurse Supervisor Training and DOTS Implement, Funding Disbursement, Sept. 2006- Sept. 2011. Funds were distributed in one of two ways, depending on the project. For the São Paulo State, a Letter of Agreement (LOA) was signed annually. Funds were transferred for the State to an NGO, Paulista Association of Public Management (APGP), which managed the activities according to the annual work plans. For all the other states, including Rio de Janeiro and municipality of Manaus, PAHO/Brazil directly managed the funds. PAHO/Brazil hired all consultants working in the municipalities, rented vehicles to transport samples and project supervision in all municipalities. It managed all logistical details for all activities, such as food, training materials and transport of workshop participants. PAHO/Brazil leveraged funding for trainings by co-sharing the cost with the government in some cases to provide training to even larger number of recipients. Reporting Inputs and outputs were reported with workshops, meetings and supervision visits documented in the limited quarterly and annual reports provided to the evaluation team. Despite USAID’s minimum reporting requirements for PAHO/Brazil to provide technical and financial reports on a quarterly basis; and annual data collection and progress reports once each calendar year, the evaluation team obtained only two quarterly reports for 2007 and one quarterly report received for 2008. Neither the Mission nor PAHO/Brazil was able to provide the full set of work plans and technical quarterly and annual reports. One annual report was provided to the evaluation team for FY07. The annual report did not report on the set of agreed upon indicators (see Objectives) as required by the signed agreement: Work plans provided were not clearly marked. It was not possible to ascertain if the submitted work plans were drafts or finals and if they have been approved by the Mission, and if so by whom or when. Changes to work plans were uncertain and differences unclear. For example, FY08 and FY09 had a discrepancy of U$D 26,500 for the same activity. In some cases, work plans were incorrectly completed with sections requiring a listing of planned deliverables offering a listing of expected objectives. Work plans were provided for FY 08 and FY09 but no annual reports were submitted so it was not possible to determine if deliverables were submitted in a timely basis, if at all. USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 111 Gender was inconsistently reported. One second quarterly report, 2011 for Rio de Janeiro provided breakdown of trainees by gender. Ratio was 1:4, male to female health care workers trained in Rio in this period. The 2007 annual report did not include gender variables. Management PAHO/Brazil was widely acknowledged as a crucial and valued technical partner, with staff possessing the highest technical knowledge and credentials. At the same time, it was criticized by some stakeholders for being overstretched, as staff were tasked with multiple activities. Due to administrative and bureaucratic delays and its own internal procedures and processes, it was reportedly a “difficult” partner at times. Complaints were made regarding delays in transfer of funds to pay for the salary of contracted staff who were paid on an irregular basis. PAHO/Brazil was flexible to change project sites, from one municipality to another in the same state, when operational difficulties of administrative and project management occurred. Based on PAHO’s evaluation that Rio de Janeiro state level supervision was insufficient, and there was variability in resources management, a decision was made to individualize work plans and interact more directly with each of the state’s 12 priority municipalities. Sustainability In 2005, PAHO/ Brazil initiated the DOTS Supervision Project with 15 nurses trained to ensure the implementation, monitoring, and evaluation of the DOTS activities in the priority municipalities. During the three years of the project, the local and subnational NTP management capability was improved in the target areas in order to implement and supervise the DOTS strategy. The success of this intervention encouraged the Ministry of Health of Brazil (NTP) together with the Global Fund project, in 2008, to expand its coverage to the rest of the priority municipalities. Although Mission funding to PAHO/Brazil will soon come to an end, the Agency has been successful in signing multi-year agreements with municipalities and the NTP to continue to provide technical assistance for TB:  NTP: US $ 4 million/year  Rio Grande do Sul US $ 3 million/year  Rio de Janeiro US $ 6 million/year  Manaus US $ 300,000.00/year (pending political decision).  New 5 year agreement with the state of Pernambuco covering neglected diseases, including TB. US $1million.  New PAHO 5 year regional project with funding from USAID/Washington: TB in the Big Cities includes São Paulo. In addition, PAHO acts a contracting mechanism to facilitate the hiring of NTP staff. Of the 38 persons working at the central unit, 34 are contracted via NTP funds to PAHO. Labs were greatly expanded in most municipalities through Project support, although human resources were a continued impediment to long-term success. The labs helped to increase diagnosis, but no data was available about overall effects of the program over a significant period of time. After Project support ended for the lab technicians, they left their positions and workload increased with resultant longer wait times to receive test results. 112 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 C. SOCIEDADE CIVIL BEM-ESTAR FAMILIAR NO BRASIL (BEMFAM) ANTI-TB COALITION (AUGUST 2002 – DECEMBER 2008) Funding: $3,194,556 Location: Six Municipalities in Pernambuco state; Rocinha favela in Rio de Janeiro Context Sociedade Civil Bem-Estar Familiar no Brasil (BEMFAM) is an affiliate of the International Planned Parenthood Federation (IPPF) and the largest reproductive health and family planning NGO in the country. It operates in 15 of the 26 states in Brazil and operates 16 of its own clinics. BEMFAM was considered a cost-effective and sustainable partner since it is a local, indigenous Brazilian organization. As such, it had fewer differentials in salary and benefits, compared to government counterparts. As the main objective was to implement DOTS at the municipal level, an organization was needed that would facilitate this through an established local municipal network. BEMFAM is located throughout the country and had been a social marketer of condoms for the Mission’s HIV program. Although BEMFAM did not have TB expertise it was considered a capacity they could hire. The more important consideration was their local networks nation-wide. In particular, BEMFAM was located in many of the same municipalities with the highest TB burden. Prior to their involvement in TB activities, BEMFAM conducted a feasibility study to determine how their comparative advantage could be successfully leveraged to support the Mission’s TB program. It was determined that BEMFAM could reinforce political commitment at the state and municipal levels. It had available decentralized physical infrastructure and personnel, as well as administrative systems and monitoring and evaluation. BEMFAM was to support the expansion of DOTS in the FHSand CHA Program in the priority municipalities. In August 2002, an existing cooperative agreement was modified to include TB activities, among other efforts. Objectives were:  To contribute to the National TB Program (NTP) to reduce the prevalence of TB in the metropolitan area of Recife, including the municipalities of Cabo de Santo Agostinho, Camaragibe, Jaboatao dos Guararapes, Olinda, Paulista and Recife.  Provide administrative assistance to CDC in the financial contracting of local institutions to implement six research activities to improve TB prevention and control. Specific objectives of the Project were to: 1. Ensure political commitment of diverse partners involved in the project- Ministry of Health, Secretary of Health of the State of Pernambuco, Secretaries of Health of the intervention municipalities, and Pernambuco Society of Pulmonology and BEMFAM— through commitment of human, financial and technical resources for the implementation of DOTS in the health units of the priority municipalities. 2. Support the secretaries of health of the municipalities to guarantee access to diagnosis by smear microscopy, with a particular attention to vulnerable populations. 3. Support the municipal secretaries of health to expand the use of standardized short course chemotherapy, including directly observed treatment. 4. Support the utilization of the registry system to permit the evaluation of the treatment outcome for each patient, as well as analysis of cohorts. USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 113 5. Monitor and evaluate the project, consolidate and disseminate lessons learned and best practices. Overall indicators of the Project were:  % of health facilities that adopt the DOTS strategy  % of TB cases that receive DOT The targets of Project, consistent with the targets of the State TB control program, were:  Sustain TB case detection at 78% or higher  Cure at least 80% of patients with directly observed treatment (DOT);  Treatment abandon rate less than 10%;  Implementation of DOTS in 50% of health facilities;  Integrated TB-HIV services (100% of TB cases are tested for HIV). In addition to the above objectives, USAID/Brazil sought to provide support to CDC to conduct six research activities: 1. Regional and national drug resistance survey; 2. Evaluation of national surveillance for TB, including a study of morbidity and mortality indicators in at least two cities; 3. Evaluation of determinants of treatment default; 4. Survey of the prevalence of HIV among TB patients; 5. Evaluation of lab capabilities to detect TB cases; and, 6. Evaluation of quality and efficiency of TB-related information system CDC did not have an existing mechanism to fund local institutions, which would cause a delay of transfer of funds by 8 to 12 months. To facilitate efficient support to this effort, the MOH, CDC and USAID/Brazil asked BEMFAM to provide administrative assistance to contract the local institutions, given their history of financial contracting and developing technical cooperative agreements with public and private organizations throughout Brazil. BEMFAM managed the funds and distributed the financial support, as indicated by CDC, to local partners. Findings Objective 1 The Project was successful in garnering public and private sector collaboration in support of TB activities in the intervention areas and to move TB higher on the agenda at the State level. The Project was successful in ensuring political commitment of diverse partners. Objectives 2, 3 and 4 IEC campaign was developed and implemented with brochures, posters, T-shirts and webpage. While it is clear that the IEC campaign was conducted with an output of over 1,600 persons trained in calendar year 2004 and audience reached of over 26 million in 2005, it is unclear if stigma or treatment default were reduced or detection of TB suspects increased due to lack of project documentation. 114 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 Figure X: Screen capture of Project Figure Y: Screen capture of MOH campaign campaign In 2004, the project partnered with Transformarte, a local NGO whose objective is to implement strategies for the prevention of sexually transmitted diseases, promote human rights and create health-promoting artistic activities with youth and community members in poor neighborhoods. The project trained 30 peer educators, who worked as group facilitators and health agents in the community. They attended more than four training meetings per month. TB was a major topic in one of the training sessions. It was an innovative project, as it was the first time TB was included in youth health prevention projects. In August 2006, the Mission invited a team of international reviewers to evaluate the Project. The evaluation team concluded that the Project had contributed to the decentralization and expansion of DOTS in the priority municipalities and was projected to achieve 50% DOTS coverage. It was reported to have helped to increase political commitment of the municipal health authorities to DOTS and contributed to improved coordination and collaboration between the state and municipal programs. The Project was attributed to having increased the percentage of registered TB patients whose treatment outcome is evaluated, a reduced default rate, decentralized of lab services, expanded quality control for smear microscopy, and trained many staff in DOTS. Criticism included little progress in detection of TB suspects and TB cases. The use of smear microscopy for both diagnosis and control continued to be low, as well as acceptance of HIV testing by TB patients. Further, presentation and analysis of data and cohort analysis was considered weak at all levels. Reportedly Project performance did improve over time but it is unclear to what extent the recommendations of the 2006 evaluation were adopted and executed Objective 5 The Project did monitor itself and an evaluation was performed, per Objective 5. Monitoring could have been improved by providing annual updates on improvements in baseline data. Lessons learned and best practices were disseminated through the organization’s networks and its staff but the extent to which this was done is unclear due to lack of documentation. Targets FY04 Annual report provided baseline data for selected indicators such as the number of new cases detected (2,632), number of smear microscopy exams performed (26,261) and default rate (14%). Some baseline data were unavailable such as the number of TB suspects detected, number of Family Health Programs with DOT and the percentage of all TB cases under treatment that receive DOT. Case detection rate was reportedly 78% in 2003, while cure was low at 62%, 65% and 63% in 2001, 2002 and 2003, respectively. Subsequent annual reports did not consistently USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 115 report on the progress towards reaching targets. Overall indicators were also not reported in annual reports. Reports failed to provide an explanation on how project activities contributed to reaching objectives and targets. Management BEMFAM worked in a very collaborative manner with municipal and state authorities and included civil society. Although it was acknowledged that this participatory work had its difficulties and delays it conferred legitimacy to the project and promoted buy-in and ownership from stakeholders. In work planning sessions, each partner was given responsibility for particular indicators and could see how their contribution impacted the overall achievement of targets. BEMFAM also worked closely with PAHO and invited them to provide comments and feedback during their work planning. This collaboration helped BEMFAM to improve its technical expertise in TB and the implementation of the project. Funding Information was unavailable. Sustainability The targets of the Project, consistent with the targets of the State TB control program, were: Sustain TB case detection at 78% or higher It is unknown, as the Evaluation Team was unable to obtain these data. Cure at least 80% of patients with directly observed treatment (DOT) Treatment results are not separated by DOTS/NON-DOTS so it is not possible to identity the cure rates for DOTS patients. Treatment default less than 10% Table 1: Percent of smear positive patients that default Municipality 2007 2008 2009 2010 2011 Recife 13.3 13.8 13.1 17.2 12.6 Cabo de Santo Agostinho 21 12.7 6.9 12.8 6.4 Camaragibe 6.7 7.7 5.1 2.6 2.6 Jaboatão dos Guararapes 7.3 11.6 12.8 13.3 7.4 Olinda 6.8 9.0 10.1 11.2 6.8 Paulista N/A N/A N/A N/A N/A Source: NTP Most municipalities have experienced a decline in the default rate, with only Recife experiencing a default rate over 10% in 2011. This target seems to have been sustained after the close of the project. Implementation of DOTS in 50% of health facilities Table 2: Number and Percent of Health Facilities with DOT 2009 2010 2011 Municipality Nº US DOT % Nº US DOT % Nº US DOT % Recife 131 115 87.8 140 123 87.9 140 123 87.86 Cabo 41 38 92.7 41 41 100.0 41 41 100.00 116 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 Camaragibe 43 43 100.0 43 40 93.0 43 40 93.02 Jaboatão* - - - 94 - - 94 32 34.04 Olinda 37 37 100.0 37 37 100.0 37 37 100.00 Paulista 47 37 78.7 50 40 80.0 50 40 80.00 Fonte: Diagtube/SEVS/SES￾PE According to statistics provided by the State TB Coordination, in 2011, all but one of the six priority municipalities where BEMFAM worked had at least 80% of health facilities with DOT. It would seem as though this target was sustained after the close of the Project. It should be noted, however, that based on definition, if even just one patient is under DOT then the health facility can be said to be providing DOT. Table 3: Smear Positive TB cases, by municipality, with DOT, 2007 - 2011 Municipality Year 2007 2008 2009 2010 2011 Recife 55.8 51.0 50.9 49.7 54.7 Cabo de Santo Agostinho 80.7 76.4 67.2 62.8 49.4 Camaragibe 89.6 84.2 76.7 55.9 84.2 Jaboatão dos Guararapes 23.3 24.7 30.8 30.5 46.6 Olinda 44.0 45.3 41.3 41.3 58.5 Paulista 62.0 56.3 71.4 69.3 69.4 Source: SINAN/ SEVS/SES – PE If we look at the percentage of patients that have directly observed treatment, we see that it is much lower, ranging from 49.4% to 84.2% in 2011. Clearly, 50% or more of health facilities with DOTS does not necessarily mean that 50% or more patients are receiving DOT. Integrated TB-HIV services (100% of TB cases are tested for HIV) It is unknown, as the Evaluation Team was unable to obtain these data. The municipalities where the project was conducted over four years ago still reminisce of the “good old days.” Municipal TB authorities interviewed who collaborated with the project fondly discussed how the project helped to facilitate their work by contracting human resources and transportation for pick-up and delivery of lab samples. The August 2006 evaluated predicted a possible risk that the success achieved would not be sustained after Project end, unless actions were taken to ensure State and/or Municipal funding for recurring costs such as incentives, laboratory supplies, transport of sputum specimens, and supervision (including personnel). Overall results were somewhat sustainable. At the UBS visited, DOTS was implemented by the FHS team. The nurses and community health agents interviewed were actively engaged in providing DOT and registering and reporting cases. At the secondary level, policlinica, however, TB treatment was not in accordance with DOTS strategy. In fact, reportedly some physicians preferred to treat TB patients behind a glass so as not to get infected. This municipality expressed their priority to have TB patients treated at the primary care level and to put into place in 2013 a plan for referral and counter-referral to ensure TB patients receive quality care from the Family Health Program (PSF). USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 117 On November 8, 2004, STOP TB Brazil (Parceria Basileira para o Controle da Tuberculose) was founded with the participation of BEMFAM. In fact, BEMFAM participated as a member of the Executive Board during the 2005-2006 biennium. Today, this partnership is a strong advocate for TB control in the country and has 100 members, including public and private organizations. Limitations As the BEMFAM project ended in 2008, work plans, technical and financial reports were of limited availability to the evaluation team. No final Project report was provided to the evaluation team. It is unclear if such a report was written and submitted to the Mission. D. JSI PROJECT TITLE: BCC FOR MORE EFFECTIVE TUBERCULOSIS CONTROL PROGRAM Cooperative Agreement: 512-TO-10-00001 Obligation Period: September 30th, 2010 – September 29th, 2011 Implementation Period: October 1st, 2010 - September 30th, 2011 Location: São Paulo State and high priorities municipalities Total Funding: $ 1,053,000 Local partners: São Paulo State Tuberculosis Control Program, the municipal TB control programs in the selected TB high burden municipalities, the NGO Rede Paulista de Controle Social da Tuberculose and the Health Coordination of Franco da Rocha Penitentiary. Context Behavior Change Communication (BCC) for Effective TB Control, (BCC for TB), was a one-year project held from October 1st, 2010 to September 30, 2011. The USAID-funded contract was signed with John Snow, Inc. (JSI), in collaboration with John Snow/Brazil Consultoria (JSB). The project was developed in São Paulo State, including the city of São Paulo and other selected high TB burden municipalities. The rationale for the intervention was based on the persistence of high default rates in São Paulo State, despite of reduction of incidence and mortality rates by TB reported between 2003 and 2007. Furthermore, DOTS expansion in São Paulo State suffered with barriers to reach particular vulnerable populations and high risk groups, such as prisoners. Objectives The main project goal was to improve TB control through provision of technical assistance for BCC and facilitation of DOTS training and supervision. The objective of the KAPB study (impact evaluation) was to detect differences in levels of KABP between health professionals and TB patients before and after BCC implementation and DOTS training. The project activities involved four main areas: 1. Training and supervision in DOTS (municipalities of Osasco, Guarulhos, Itaquaquecetuba, Mogi das Cruzes, Mauá, Barueri, Carapicuíba, Santo André, São Bernardo do Campo, Taboão Serra, Itapevi, Santos, Diadema, São Paulo - Penha District and Freguesia do O/Brazilândia District – and Franco da Rocha Penitentiary); 2. Development of BCC strategies and providers peer-to-peer interventions for TB control; 3. BCC intervention implementation (Freguesia do O/Brazilândia District, Guarulhos Municipality and Franco da Rocha Penitentiary); 118 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 4. Impact evaluation (Freguesia do O/Brazilândia and Penha District, Guarulhos Municipality, Franco da Rocha Penitentiary, Embu Municipality). The proposed Monitoring and Evaluation (M&E) plan included different methods and data sources: monitoring data from SINAN, tracking project administrative and training records, creating a performance monitoring tool for verifying and tracking selected indicator not present in SINAN, conducting focus group discussions and key informant interviews. Indicators included effectiveness indicators (outcome: TB indicators defined in according with TB program managers) and performance indicators (output: number of training courses and trainees) (M&E plan, 2010). Main findings The project represented a new and strategic investment in an area of intervention that in the previous years suffered with the lack of financial resources and specific methodologies and tools for better working. JSI process of work looked for the participation of different stakeholders and the meetings for the elaboration of the “De peito aberto” campaign had the participation of members from civil society, São Paulo State and Municipality TB program managers, representatives of the Mayor’s communications office, the State Penitentiary System and health providers. This methodology of working gave the possibility to share the ownership of the campaign with those people most impacted by TB disease as well as by public health providers. The training on DOTS implementation and expansion led to an improvement of technical knowledge of participants, measured by the evaluation of pre and post test scores. Training and supervision in DOTS A total of 879 (734 females; 83.5%) health professionals (medical doctors and nurses), lab technicians, administrators, health community agents and penitentiary agents were trained. In particular, 67 health and prison professionals at Franco da Rocha penitentiary (units I, II and III) received DOTS training, a number twice higher the originally planned number of professionals to be trained. Participants to the training courses answered to a pre and post-test and the overall mean increase of 13.5% points between the tests for all the professionals categories trained was observed. The role of JSI on training activities involved the coordination with TB managers of State and municipalities for organization and administration of training (trainer selection, logistics of meeting, preparation of didactic materials). Development of BCC strategies and providers peer-to-peer interventions for TB control Implementation included three phases. The first phase was directed to organize the research and the planning process: data collection on client/market segments, opinions on treatment barriers and opportunities through focus group methodologies. Deliverables and documents were produced (Focus Group Moderator’s Guides and Focus Group Report). The strategy design in the second phase intended to form the basis of communication strategy and campaign development, resulting in the set of deliverables contained in the Marketing Mix Plan document. The third phase was dedicated to the implementation and evaluation of communication materials; the aspects of the materials such as themes, sub-themes, symbols, colors, images, format and graphics were tested in focus groups. New knowledge regarding program strategy innovation, problem-solving and continuous quality improvement was taught to staff from TB program of São Paulo State and the other involved municipalities. BCC intervention implementation JSI provided technical assistance and capacity building to the TB Control Program in São Paulo state in launching the “De peito aberto” campaign between July-September, 2011 in three intervention sites: Guarulhos and Freguesia do Ó Health Care Units and Franco da Rocha Penitentiary clinics. The campaign involved peer to peer communications by Rede Paulista through USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 119 their community mobilizers. “De peito aberto” posters were produced (male and female versions) and allocated in waiting rooms and medical offices of health units. Colorful pamphlets (male and female versions) were also available and were used to inform and educate clinic visitors (Annex Figure 15). Male posters and pamphlets were distributed in the Penitentiary. Short videos spots were developed to support the campaign. Annex Figure 15: Pamphlets on TB produced by the BCC project. The key messages presented in the communication materials were: 3 weeks of coughing is a sign of alert, tuberculosis has cure, 6 months of supervised treatment without stopping, diagnostics and medicines are available free at São Paulo state public health clinics. Rede Paulista, in their peer to peer communications, reached a total of 2,022 clients and community people at risk. These included 49 TB clients who were in the KAPB study, 202 who were other clinic visitors/friends/family of TB clients, and 1,762 other community members. An additional 1,777 community people were reached through ten social mobilization events organized and conducted by Rede Paulista. The activities organized at Franco da Rocha Penitentiary included a TB control-oriented community theatre performance, theatre workshop for prisoners, group discussions and interaction with friends and family visitors on line during visitation day. Between July-September, 2011, approximately 13,000 pamphlets and 150 posters (“De peito aberto” campaign, others) were distributed by JSI and local partners (Secretariat TB Control, Municipality TB Control, Franco da Rocha Penitentiary Health Unit, Rede Paulista). Impact evaluation In order to detect differences in levels of KABP before and after BCC implementation and DOTS training, variables were chosen and pre-tested and post-tested to assess the levels of KABP among four groups: a) TB patients from health care units; b) HCWs from health care units; c) prisons health system´s clients; d) prisons HCWs. JSI discussed with TB managers from São Paulo State and municipality and National TB program which indicators to use in the impact evaluation. A list of indicators was produced with data from the State TB Program intervention and control sites (Municipalities of Guarulhos and Embu and Districts of Freguesia do Ó and Penha) collected from SINAN quarterly reports. The indicators used refer mainly to DOTS treatment, identification of new cases/microscopy, and treatment follow-up. Whenever possible data disaggregated by gender were provided. The availability of indicators from all the municipalities and districts involved was not uniform. A trend for an increase in case detection and treatment completion rates applicable to all the studied areas could not be observed. The default rate to TB treatment was not clearly reported among the selected indicators. A total of 32 patients and 72 health professionals completed the both pre and post questionnaires. TB patients submitted to the intervention presented a significant improvement in the score obtained in the post test KABP questionnaire (i.e. a significant difference between the coefficients in pre and post-tests), compared to the control group in Freguesia do Ó. Older patients presented 120 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 better coefficient between pre and post-tests but gender did not determine a statistically significant difference on the coefficients. Regarding HCWs, DOTS training contributed to a positive impact in the mean KABP coefficient at the two municipal level locations (Guarulhos and Embu). Limitations The real impact of all project activities on the improvement of main indicators of TB program is difficult to evaluate. During the period of project development other implementers (GFATM PAHO) also contributed to training on DOTS (among other activities) in São Paulo State, making it difficult to attribute indicators improvements to a particular intervention. Furthermore, the relatively short duration of the project (one year) limited the ability to assess the impact of project activities on TB program indicators. Health care workers who took part in the project acknowledged the importance of training activities on DOTS and of the usefulness of communication materials and new educational tools on TB provided by JSI project. However, some hurdles were also identified. Local partners complained about communication problems between the parties during the development of project activities, which caused misunderstandings about the duties of each at the different phases of the project implementation. These relationship difficulties represented an impediment for the continuation of project activities of AIDSTAR-ONE/JSI project (see below) in the health units of São Paulo State. Delays for delivering educational campaign materials (posters and pamphlets) for both the activities in the prisons and in the health care units were reported. It was particularly problematic during the campaigns with the prisoners’ families outside the penitentiary. Considering difficulties to change the already set dates for the meetings due to bureaucratic reasons, the volunteers from Rede Paulista had to use the informative material of previous campaigns from NTP (Annex Figure 16) rather than the educational materials specific of the BCC project in some occasions. Annex Figure 16: NTP educational campaigns. USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 121 It was also mentioned some constraints with JSI partnership related to the delivery of funds to financial support (such as transportation and meals) for Rede Paulista volunteers who were conducting the study in Franco da Rocha penitentiary and in the health units of São Paulo State. Some of local partners from TB programs were disappointed with the lack of originality of communication materials produced since they expected something more innovative respect to the campaign material they already receive from the National TB program. Regarding the BCC intervention implementation study some deviations of study protocol were reported. The inclusion criteria for the participation of patients were not fully respected, leading to the inclusion in the study of patients at the end of TB treatment or who were defaulters, minors (without agreement of patients’ parents or tutors), mental disabled or patients from other health units outside the study area. The communication material supposed to be delivered to patients during the interviews arrived only at the end of the study. São Paulo SAP penitentiary review process allowing access for the project interventions was lengthy and complicated and caused some delay to adhere to the initially established schedule of activities implementation. Funding breakdown The Total Estimated Ceiling Price of this Task Order was $1,053,646.00, as follows: Annex Figure 17: BCC project funding breakdown. PROJECT TITLE: SOCIAL TECHNOLOGIES FOR SUSTAINABLE ACCESS TO TUBERCULOSIS CONTROL AND HIV/AIDS INTERVENTIONS IN BRAZIL 2011 - 2012 Cooperative Agreement: GHH-I-01-07-00059 Obligation Period: September 29th, 2011 –September 30th, 2012 Implementation Period: October 1st, 2011 –December 31st, 2012 Location: São Paulo and Rio de Janeiro cities Total Funding: $ 1,110,000 Local partners: National level (Ministry of Health NTP and DN) and Secretariats of Health of priority States, São Paulo (SP) and Rio de Janeiro (RJ) and Non- 122 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 Governmental Organizations (NGOs): EPAH (Espaço de Atenção e Prevenção Humanizada) and Rede Paulista de Controle Social da Tuberculose in São Paulo city and Arco Iris in Rio de Janeiro city. Context “Social Technologies” was a one-year field support activity under the USAID AIDSTAR-One Contract with John Snow, Inc. (JSI), in partnership with John Snow do Brasil, Ltda (JSB). The basis for the project interventions was the evidence that behavioral determinants and stigma/discrimination perception are associated with TB/HIV treatment adherence and access to health services in a country where TB is the most common cause of death among AIDS patients and in which 10% of patients with TB are co-infected with HIV-. The expanded access to TB/HIV interventions is particularly important for the most at risk populations (MARPs), including people living with HIV/AIDS, TB patients, prisoners, men who have sex with men (MSM), lesbian, gay, bisexual, transvestite and transsexual (LGBT) and sex workers. Objectives The general objective of the project was to increase access to prevention, diagnosis and care for TB/HIV co-infected patients providing technical assistance (TA) in social technologies at the National and State levels. The three specific project objectives were: 1. Application of social technologies in building upon and expanding access to TB and HIV interventions for MARPs, with particular focus on prisons program expansion; 2. Capacity building, training, and supervision in social technologies at the National (NTP) and priority State level (Rio de Janeiro) and 3. Continue support in communication campaign implementation and impact evaluation in the State and Municipality of São Paulo based on “De peito aberto” campaign experience initiated with BCC project in 2010-2011. A total of 19 indicators were selected (13 performance/output indicators and 6 impact/outcome indicators) to evaluate the impact of project activities. Community grants competition for NGOs resulted in 27 total applications and three NGOs were selected: one in Rio de Janeiro (Arco Iris) state and two São Paulo state (Rede Paulista and EPAH). Main findings The SocialTech project represented an expansion of the previous BCC project held in São Paulo. It now includes other areas (Rio de Janeiro) and other MARPS besides prisoners. Education campaigns about Tuberculosis was definitely a relevant piece to support the strategy of TB control, increasing access to TB diagnosis and treatment, TB/HIV co-infection for MARPS and reducing the number of defaulters on TB treatment. However, the real impact of these interventions on TB indicators is difficult to measure. One of the reasons is due to the selection of performance indicators to monitor and evaluate the results. A number of educational campaigns during AIDSTAR-ONE project in partnership with NGO EPA were conducted and around 4,000 pamphlets of the “De peito aberto” campaign (Annex Figure 19) was distributed. “Quero fazer” program focused on counseling and testing for HIV. "Tipo assim" project consisted of training on sexuality and prevention for adolescents and young adults between 16 to 24 years, residents in the south area of São Paulo and in other public areas of São Paulo city such as shopping centers, restaurants, bars and night clubs. Aiming to increase the level of TB treatment knowledge, attitudes, and behaviors/practices (KABP) of prisons´ tuberculosis patients and ex-patients, an intervention at Franco da Rocha Penitentiary was developed during 2011-2012. KABP evaluation tool consisted of 41 variables, regarding adherence to TB treatment, stigma, and forms of TB transmission. The strategy was developed in two of the three Franco da Rocha units (P1 and P2), and the third unit was used as a control site (P3). Forty-nine questionnaires were applied at Franco da Rocha units before and USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 123 after a period of 8-9 months from the educational intervention and two models for identification of potential statistical significant differences were used: penitentiary unit over time (Analysis Model 1) and intervention vs. control (Analysis Model 2). In Unit P1, significant increases in TB treatment knowledge, adherence, and social support was observed comparing before and after the intervention. In Unit P2 the impact was lower, but significant differences in KABP coefficients were observed after the intervention when compared to the control unit (Table I). Table I. Results of KABP intervention at Franco da Rocha Penitentiary (2011-2012). Prisons´ Unit (Prisoners affected by tuberculosis) Analysis Model 1 (comparing ex-ante/ex￾post per unit) Analysis Model 2 (comparing ex-ante with control – Unit P3) Analysis Model 2 (comparing ex-post with control – Unit P3) Unit P1 5,5 (valor-p<0,05)* 4,9 (p-value>0,05) 9,9 (p-value<0,001)* Unit P2 1,9 (valor-p>0,05) 7,25 (p-value<0,05)* 9,3 (p-value<0,01)* Unit P3 - 0,15 (valor-p>0,05) JSI was instrumental to promote technical assistance to NTP and State TB staff on how to apply social technologies to improve TB Control. A plan of training was also developed with NTP coordinator and staff during the second quarterly. A partnership with SEAP (Secretaria de Estado de Administração Penitenciária), the State of Rio de Janeiro Penitentiary Administration Health Department and TB program of Rio de Janeiro State was developed to extend the campaign on TB-HIV co-infection ("De Peito Aberto: Juntos podemos prevenir a tuberculose e a AIDS") to the penitentiary complex of Bangu, the largest penitentiary in Rio de Janeiro, housing 28,000 inmates. Limitations Some constraints related to the implementation of project by JSI with local partners were reported, such as sharing information and logistics issues. Delays in funds release and to receive the educational campaigns materials were some of the main issues raised in São Paulo. Due to those delays, the implementation of the project last only one month rather than the 4 months previously planned. In São Paulo and Rio de Janeiro the educational material used in the communication campaign was received from TB State Program and NGOs involved only in the beginning of November 2012. Other constraints raised by local partners were that focus group recommendations were not shared with partners and the written text used in the “De peito aberto” campaign was considered not appropriate to MARPS. Moreover, the type and quality of paper used in the communication campaign (pamphlets) was too expensive and referred to as, “a waste of money”. Due to the lack of baseline data and few results available at the time of this evaluation (especially for Rio de Janeiro), it was not possible to perform a comprehensive full assessment of this project. 124 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 Funding breakdown Annex Figure 18: AIDSTAR-One project funding breakdown. Annex Figure 19: Pamphlet used during the campaign “De Peito Aberto”. Direct Labor 16% Indirect Costs/Overhea d 13% For Other Direct Costs 45% Grants 3% Subcontract 18% Fee 5% AIDSTAR‐One Funding Breakdown Total: US$ 1,053,646.00 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 125 E. CENTER FOR DISEASE CONTROL AND PREVENTION (CDC) Cooperative Agreement: AAG-P-00-99-00006/ GHN-T-00-06-00001 Obligation Period: 2001 – 2007/2006 – 2007 Implementation Period: Unknown Location: National. Total Funding: $2,212,966/$30,000 Context The USAID/MOH/CDC collaboration included the provision of technical assistance in the expansion and improvement of effective diagnosis, treatment and monitoring of TB, with particular emphasis on TB/HIV and MDR-TB. The scenario of TB in Brazil when the project begins shows TB as was one of the main causes of morbidity and mortality in the country, with an estimated 90,000 new cases/year; 6,000 deaths were attributed to TB and the incidence rate was of 50 cases/100,000 inhabitants. Moreover, treatment outcomes were provided for only 12% of smear-positive pulmonary cases notified in 1999. An estimated 80% of smear-positive cases were notified in 2000 but treatment outcomes were available for only 65% of notified cases. Objectives The activities supported by USAID through the collaboration with CDC and Brazilian MOH were: a) evaluation of the National TB Surveillance System (SINAN); b) planning and implementation of a national DRS; c)planning and implementation of a national TB/HIV co-infection survey; d) evaluation of the current epidemiological situation of TB in Brazil; e) national cohort study performed at the State level, using State-level data; f) improving care for patients with TB and HIV co-infection; g) evaluation of the impact of antiretroviral (ARV) treatment on the incidence of TB in Brazil; h) improving quality control of TB laboratories; i) training and facilitating operational research and program management. Main findings Evaluation of the National TB Surveillance System in Brazil (SINAN) The objective was to describe and evaluate the National TB Surveillance System (SINAN) aiming to provide recommendations to its better performance. The evaluation of SINAN showed that the system was able to register only 70% of all TB cases in the country, which means that 30% of TB cases were not notified. Recommendations to improve the performance of SINAN were included in a written report and presented to MOH; adoption of recommendation resulted in the increase of available information regarding treatment results from 33% to 50%. All project activities were terminated in 2003. The evaluation results were presented at the Union TB congress in Paris, in October 2003. The publication of the results as a manuscript in a scientific journal, initially planned for 2004, was not actually performed. Planning and implementation of a national drug resistance survey (DRS) The project aimed to determine the prevalence of primary and secondary drug resistance to anti￾TB drugs in Brazil. The protocol was initiated in 2003 with meetings involving the participation of representatives of Centro de Referência Professor Hélio Fraga (CRPHF - Central Reference Laboratory), LACENs (State Reference Laboratories for TB) and technical consultants from WHO aiming the protocol preparation and training of lab personnel participating in the survey. The study was initially expected to start in March 2004, but due to delays in the receipt of samples for the necessary proficiency testing by the CRPHF, the initiation of the protocol was postponed to August 2004. Deviations from the internationally recommended culture and DST procedures were identified by WHO in 2006. In 2007, WHO consultants conducted a second visit, carrying out a survey in 6 States that notified 55% of TB cases in the country. WHO consultants reported again important deviations from the survey protocol; international 126 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 recommendations on the performance of DST were not being followed and bacteriological procedures were out of quality control. The results were considered unacceptable by WHO for publication as they were not comparable to those obtained in other surveys. In 2009, the NTP introduced ethambutol as the fourth drug in the intensive phase of treatment, based on the preliminary results of DRS which showed an increase of primary resistance to isoniazid from 4.4% to 6.0 %. By the time of preparation of this evaluation the final results of the DRS were not available. Planning and implementation of a National TB/HIV co-infection survey The study of the prevalence of HIV infection in TB patients was conducted concomitantly with the DRS. The aim of the study was to determine the prevalence of HIV infection among TB patients and its association with resistance to anti-TB drugs. TB patients enrolled in the DRS were offered HIV testing. HCW taking part in the study were trained to offer pre- and post-counseling for HIV testing. The inclusion of TB-HIV co-infection surveillance into the DRS was supposed to strength the relationship between the TB and HIV/AIDS National programs. In the last CDC report of September 2006 no information was available regarding the results of this study. Evaluation of the current epidemiological situation of TB in Brazil The data collected using SINAN database and through the cohort study allowed the analysis of the epidemiological scenario of TB in Brazil between 2001 and 2003. The results of this study were presented at the Union World TB Conference in October 2003. National cohort study performed at the State level, using State-level data This study had the objective of analyze the results of TB treatment in a cohort of TB patients from the whole country who initiated TB treatment in 2001. The study was conducted during March-April 2004. At the end of 2004 data collection and data entry were completed. Preliminary data showed a low treatment completion rate of 62% and a high rate of default of treatment (13%). Deficiencies in the register book (Black book) were identified through this study and these findings helped NTP to change the black book accordingly. Evaluation of the impact of antiretroviral (ARV) treatment on the incidence of TB in Brazil A joint TB-HIV work plan was completed for the MOH in 2004 aiming to improve the care for patients with TB and HIV co-infection (activity f). A national retrospective study involving HIV￾seropositive adults between 1995 and 2001 was conducted during March-April 2004. HIV-infected patients were randomly selected from HIV treatment centers chosen by population-proportional￾to-size sampling. A total of 463 HIV-infected patients were enrolled during the study period in 11 Brazilian states. The study showed that the use of HAART by HIV patients was associated with a significant reduction in the risk of TB disease; an 80% reduction in incident TB was observed in HIV-infected patients treated with HAART compared to ART-naıve patients. A manuscript with the study results was prepared and published in a scientific journal in 2007 (Miranda et al., 2007). Improving quality control of TB laboratories The activities implemented through the DRS, such as training of lab technicians from the local and state laboratories involved in the project, hiring of lab technicians to support state laboratories and the supervisory visits of WHO consultants, improved performance of TB laboratories in different parts of the country. Training and facilitating operational research, management, and program management In June 2004, an operational research course for Latin American National TB program managers and staff from National TB programs was held in Brazil. A total of 49 professionals from 19 Latin American countries participated to the course. The course aimed to offer training on basic methods for the implementation of epidemiologic and operational research projects on TB. USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 127 Major achievements Mission funding covered a long period of time (2001-2007) and a broad range of activities.  Training of human resources finalized to the preparation of qualified personnel for the realization of epidemiological and operational research studies through the organization of several training courses and supervision visits.  Reinforcement of evaluation system for monitoring and evaluation of TB cases in the whole country, through the technical support given to the assessment of the national registry of TB cases (SINAN).  Among all the activities undertaken, undoubtedly the one that requested a higher complexity of planning, personnel training, standardization of procedures and financial resources was the DRS. The results of DRS will be of huge importance to define the policy regarding diagnosis and management of resistant TB cases in the country. Limitations The paucity of information available to the evaluation team limited our ability to adequately evaluate the projects conducted by CDC. The data and comments presented herein refer to the information present in the activities reports (four in total) sent to USAID between 2003 and 2006 that contained brief information on the activities held, without specific referral to the objectives or financial reporting. The projects had completed five years prior this external evaluation. The serious technical and administrative problems identified in the conduction of DRS prevented that internationally validated results were available after all these years, despite the large amount of funds invested in the survey. F. TUBERCULOSIS COALITION FOR TECHNICAL ASSISTANCE (TBCTA) Cooperative Agreement: HRN-A-00-00-00018 Funding: $1,852,000 (2000 -2005) Location: Rio de Janeiro; National Context Funding was provided to TBCTA so that partners (IUALTD, PAHO and KNCV) could efficiently get started and the TB program could benefit from their comparative advantage, subject to mid￾term evaluation. As funding was initially small for TB, it was important to be cost-effective. To operate a contract locally a team was needed. Therefore, managerial tasks were shifted to Washington by providing field support to USAID/Washington-based mechanism. In March – April 2001, TBCTA consultant team made a TB assessment visit. The team’s recommendations were translated into a work plan and finalized in July 2002. It was decided to focus the initial Mission assistance on Rio de Janeiro State and the Federal NTP central unit. Objectives  Accelerate implementation and expansion of the DOTS strategy in Brazil.  Develop institutional and technical capacity for TB control at federal, state and municipal levels.  Integrate TB control efforts within primary health care services more effectively; develop or strengthen linkages between the TB and HIV/AIDS programs and multidrug resistance efforts. Findings The majority of TBCTA Mission funds were transferred to PAHO, prior to PAHO signing an agreement directly with the Mission (via the USAID Regional representation in Peru) later in 2006. (See PAHO section for further details). See Figure A. 128 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 A PAHO/Brazil medical officer was contracted by TBCTA in February 2004 for 11 months (subsequently extended for the life of the project) to provide technical assistance for TB. In June 2002, a medical epidemiologist was contracted by IUATLD to coordinate Ministry of Health TB research activities and to provide epidemiological support to the national TB program in Brasilia, including TBCTA activities. In 2003, Rio reportedly had 7.4% DOTS coverage and by 2005, it had increased to almost 20%. Of course, as stated previously, a performance evaluation cannot attribute causality but the support provided by the Project did contribute to this increase, given the technical assistance provided and thousands trained. Project-specific indicators show an increase in completeness of reporting to NTP (65% in 2003 to 90% in 2005) and an increase in the number of microscopy units with at least 1 lab technician trained in AFB microscopy (from 3988 in 2003 to 4509 in 2005). Surveillance of MDR-TB was lacking in 2003 but occurred in 2004 and 2005.Additionally, HIV prevalence among TB patients was reported in 2004 and 2005, while it was unreported in previous years. Main accomplishments:  Provided support to the national survey on TB drug resistance and prevalence of co￾infection (2005-2007).  A joint national TB/HIV plan was developed.  TBCTA assisted the NTP in preparation of the successful fifth round GFATM proposal for DOTS expansion.  Operational research conducted on TB suspect case finding (by sputum-smear examination) at the Rio de Janeiro General Hospital Emergency Unit, TB suspect case finding among people living in state shelters, and training former homeless TB patients to supervise treatment of newly homeless TB patients.  In 2004 alone, trained 4,639 university-level professionals, 1,822 intermediate-level health personnel, and 6,455 community health workers.  Assisted the NTP in the launching of a successful national media TB campaign to increase community education and awareness of TB control and prevention. The campaign aired a total of 8,000 times on television and 7,000 times by radio. Management The Project heavily relied on in-country long-term technical experts with extensive years of experience in Brazil. Overall, Project partners worked collaboratively with each other, NTP, the Mission and USAID/Washington to coordinate activities. There was some friction in relations between some stakeholders during the implementation of the controversial national DRS. Eventually though the resultant preliminary results of the DRS did serve to encourage national adoption of the internationally recommended four-drug treatment regimen. Did the Project reach its objectives? Yes, DOTs was expanded; capacity developed; integration more effective and linkages strengthened.   But due to limited data, including lack of specific targets, it is difficult to measure or quantify to what extent DOTS expansion was accelerated, capacity development was sustained, and integration and linkages strengthened.   USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 129 Funding Figure A: TBCTA funding disbursement by partner, 2001-2005 Note: PMU/ACF: Project management unit/allocable cost factor Funding provided to the IUATLD totaled $408,929, 78% of these funds supported the costs of short-term and in-country technical assistance. The remaining funds were used to implement training courses, including fees, transportation, per diem and materials. Below, Figure B shows that PAHO/WHO used most of the funds for project implementation- training, human resources, IEC campaign, monitoring and evaluation (M&E) meetings, and equipment. Additionally, the Project took steps to reduce overhead costs by switching funds from one partner to another in order to incur savings. Figure B: PAHO/WHO Funds Disbursement, TBCTA, 2001-2005 Sustainability The activities began under this Project were continued with direct funding to PAHO/Brazil and to TBCAP (see section on TBCAP below). Interestingly, many of the challenges first noted with this early Project continue to this day, such as integration of TB and HIV programs at the state and municipal level, high staff turnover, and constraints of a slow and complex bureaucratic system. Limitations As this funding ended over five years ago, detailed information unavailable and memories were limited. 130 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 G. TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TBCAP) Cooperative Agreement: GHS-A-00-05-00019 Funding: $455,730 (October 2005 – September 2007) Location: National Context The Mission continued to provide staffing support through this cooperative agreement to augment NTP technical capacity. Objective To improve the central level NTP and help NTP reach goals of its TB program. Findings This project funded two years of the contract of a CDC medical epidemiologist who was based at the NTP offices and provided expert technical assistance, particularly for the conducting of the national DRS and HIV Prevalence Surveys in 3 additional Brazilian states. The specialist also helped to develop and distribute a software program for data entry, train health care workers for data entry and enroll over 1,000 patients in the survey. Funding As depicted in Figures X and Y, the majority of the funds were spent for the consultant position and its associated costs, including duty travel. FY06 and FY07 funding combine to a total of 430,808, which are 24,922 less than the obligated amount. These remaining funds were reportedly reprogrammed to other TB activities. An additional note: ACF charges were incorporated in the FY06 budget of $17,850 as a footnote citation; it is uncertain how other budget line items were revised to accommodate this charge. Figure X: TBCAP Funds Disbursement, October 2005- September 2006 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 131 Figure Y: TBCAP Funds Disbursement, October 2006- September 2007 The increase in the budget from FY06 to FY07 is accounted for, in part, by consultant relocation costs ($32,000), office supplies ($16,775) and other services such as telephone, electricity, etc. ($17,283). This cooperative agreement funded the consultancy of a technical expertise to provide needed assistance to the NTP to generate scientific evidence for national decision-making. The cooperative agreement fulfilled that purpose. Important scientific knowledge was generated with the support of the consultant. Funding for the research studies was provided to CDC and BEMFAM. (See the sections on CDC and BEMFAM for more details regarding the research conducted). Sustainability This position was funded for about five years. After the departure of the Mission-funded medical epidemiologist, it is unclear who has responsibility at the NTP for these activities. Limitations As this funding over five years ago, detailed information was unavailable and memories limited. H. JHU PROJECT TITLE: EXPANSION OF DOTS FOR TUBERCULOSIS CONTROL IN RIO DE JANEIRO, BRAZIL Cooperative Agreement: Obligation Period: Implementation Period: September 2002 – December 2004 Location: Municipality of Rio de Janeiro (RJ) Total Funding: $ 2,000,000 Local partner: TB program staff of municipality (city) of Rio de Janeiro and Communicable Diseases Coordination, Health Secretariat of Rio de Janeiro City Context The project aimed to strengthen TB control in Rio de Janeiro (RJ) city, a metropolitan area with a high incidence of TB and default of anti-TB treatment, through expansion of DOTS strategy and intensive contacts evaluation. The expansion of DOTS program relied mainly on the detection 132 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 and treatment of TB cases in their own community by community health workers (CHWs) integrated into the Family Health /Community Health Workers Program in the AP-2.1 (programmatic area 2.1) of Rio de Janeiro city. Five “slums” communities were selected as priority areas for CHWs program implementation, the largest of them is Rocinha, with an estimated population of 62,000 inhabitants. CHWs offered treatment in the health care units or at home. Bus tokens and food voucher were given to TB patients treated at health care units. Cultures for MTB and DST were performed in cases of retreatment. HIV counseling and testing was offered to all adults patients with TB. Objectives The specific objectives of the project were: 1. Expand DOT for TB to the AP-2.1 area of Rio de Janeiro city, using Municipal Health Department Health facilities with CHWs and incentives, incentives alone or routine health center DOTS without incentives. 2. Increase the cure rates for TB in the AP-2.1 area, achieving a treatment completion rate of at least 80% in year 1 and 85% in subsequent years. 3. Evaluate the effectiveness of CHWs on promoting adherence to DOTS and increasing treatment completion by comparing TB treatment outcomes among patients receiving CHWs DOT to those receiving health center-based DOT with incentives or without incentives. 4. Evaluate the effectiveness of CHWs for improving case detection through active case finding among contacts of infectious TB cases and other community members. 5. Diagnose at least 70% of TB cases by microscopy and culture in AP-2.1 by implementing the innovative rapid diagnostic system MODS (Microscopic Observation for Detection and Susceptibility). 6. Conduct surveillance for drug resistance among patients with a history of previous treatment who present for renewed therapy of TB using the MODS system. 7. Implement routine HIV testing for all adult TB patients in AP-2.1 and compare rates of HIV testing in the health center using CHWs with health centers not using CHWs. Main findings The implementer together with local partners published several scientific papers whose study period involved the years and the area of intervention (DOTS expansion) covered by mission support. Since the project final report was not available for consultation, the results here presented refer to data collected from these manuscripts. From January 2000 to December 2001 a longitudinal study comparing the treatment success rates between patients treated under DOT (1,190; 12%) with those who received self￾administered therapy (SAT) (8,739; 88%) in the Rio de Janeiro city showed that patients under DOT had higher cure rate (49.3% vs. 9.4%) and lower default (10.2% vs. 15.3%), death (2.5% vs. 3.8%) and transfer out rates (4.9% vs. 7.1%) compared to the SAT group. Another study, published in 2007, estimated the10-year cost of treating a population of 262,000 TB cases in Rio de Janeiro city and found that DOTS was a highly cost-effective intervention. In the same year, the treatment modalities and outcomes of 1,811 TB patients diagnosed between 1 January 2003 and 30 December 2004 in AP 2.1, that includes Rocinha favela, were evaluated. 1,215 (67%) were treated under DOT; among these, 726 (60%) received clinic-based treatment and 489 (40%) community-based treatment. Patients offered community-based treatment were more likely to accept DOT than those offered clinic-based treatment (99% vs. 60%, respectively, p<0.001). Treatment success rates for all TB patients were higher (90.4% vs. 78.2%, p<0.001) and the default rate was lower (4.7% vs. 12.7%, p<0.001) among those treated with community-based DOT compared to clinic-based DOT. This study documented the benefits of a community based model of DOT using CHWs in the Rocinha favela, demonstrating high rates of treatment success in all patients, including retreatment patients. During two years (Nov 2000 to Dec 2004) a community-randomized trial involving urban neighborhoods in the downtown area of Rio de Janeiro city was implemented to compare the DOTS-Ampliado (DOTS-A) to the standard DOTS strategy. DOTS-A added intensive screening of household contacts of active TB cases and provision of treatment to secondary cases and preventive therapy to contacts with latent TB infection (LTBI) to the standard DOTS strategy. USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 133 DOTS-A was associated with a modest reduction in TB incidence and according to the view of implementer and local partners could represent an important strategy for reducing the burden of TB. In the last manuscript published in 2010 regarding the results of USAID supported project, the impact of a door-to-door case-finding campaign for TB, including symptom screening and spot sputum collection, was compared to a case-finding strategy based on the distribution of an educational pamphlet. A door-to-door case-finding campaign was more effective at detecting prevalent cases of TB and influencing people to come for care than leafleting, but no differences were seen in time to treatment start or treatment completion. Major achievements The purposes of this project was to assist TB program of Rio de Janeiro city with the expansion of the DOTS strategy, to assess the impact of the expansion of DOTS on TB control, and to conduct operational research on the use of CHWs to promote TB case detection, supervised therapy and contact evaluation. CHWs were hired by the Rio de Janeiro city with JHU funds, started with 40 then added more than 80. The TB program in Rio de Janeiro was almost exclusively clinic-based at the time the project was conducted and the integration of its activities into the PSF/PACS program was an important strategy through the decentralizing of TB control activities. The project supported by USAID demonstrated through appropriate scientific methodology the validity of innovative interventions for expansion of the DOTS strategy in an urban context of high prevalence of TB and high social complexity, such the favelas of Rio de Janeiro city. It helped to promote the strategy of pushing out the drug lords and gangs and establishing services for the population, such as those in Rocinha. Rocinha was reportedly a model for the TB program in Rio de Janeiro. The intervention continued with funding from the Rio de Janeiro city and is still on going at a higher level than when funded by the Mission. Limitations Full project reports, including financial reports were not obtained, as the project closed eight years ago. It was found that JHU had some activities in São Paulo state. However, it is not possible to know the nature of those interventions as no documentation was provided. Funding break down No information available. 134 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 135 ANNEX G: THE UNIFIED HEALTH SYSTEM (SUS) AND THE NATIONAL TB CONTROL PROGRAM The Unified Health System (Sistema Único de Saúde, or SUS) is currently the main mechanism for public health care delivery in Brazil. Based on the 1988 federal Constitution and regulated by Law 8.080 and Law 8.142 since 1990, the SUS was established with the objective of improving access to health services and quality of care, as well as enhancing the country’s overall health situation. The SUS is tasked with guaranteeing free and complete basic health care to the entire Brazilian population. The objectives of the SUS explicitly state that health is a human right, and requiring the State to help the population by means of promotion, protection, and restoration of health, along with the integration of such assistance into prevention activities. The SUS is financed by government subsidies derived from general taxes, payroll social security tax contributions paid by the formally-employed population. In 2000, a constitutional amendment (EC-29) defined how income should flow among the national, state, and municipal levels and throughout the system. Federal resources consisting of more than 70% of SUS’s total budget revenue are gradually being passed on to states and municipalities as direct transfers from the National Health Fund to state and municipal accounts. The SUS consists of a large network of public health care institutions and services at federal, state and municipal levels; it incorporates health posts, hospitals, university health centers, laboratories, blood banks, as well as research centers. Basic health needs (and simple procedures) are to be met at the local level (i.e. the center in closest proximity to the residence of the user). More specialized or complex procedures are outsourced, as needed, by the means of a referral process to a secondary or tertiary health service delivery point or a hospital. Private sector involvement in the SUS is said to be complementary and provided in the form of contracts and service agreements when the public health care does not have the capacity to provide services for all the population in a particular region. Additionally, private (voluntary) plans, group health plans, self￾managed health plans [employee enrollment], and insurance companies comprise a second system that supplements the public health system. THE NATIONAL TB PROGRAM (NTP) The execution of the NTP includes innovative strategies which expand and strengthen DOTS, with a focus on articulation with other government programs to broaden the control of TB and other co-morbidities, for example HIV/AIDS. In addition, the NTP has decentralized the means of control to the Basic Health program, increasing the access by the general population and the most vulnerable groups or those groups with growing risk of contracting TB – such as street dwellers, prisoners, and indigenous populations. This complements the work with non-governmental organizations or civil society - to strengthen TB control from a societal standpoint and to guarantee sustainability of such TB control actions. Keeping TB on the public health agenda includes updating the ways to promote equality and well￾being, to guarantee access by patients, but also in a broader sense, it has required consolidation with the Unified Health System and its directorates. There is also need to update some aspects of the system, including the organogram and some procedures, redefinition of institutional missions within civil society, and ways of improving case detection. 136 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 From a basic operational standpoint, good TB control is based on the active case finding, early detection, and treatment until cure with the objective of interrupting the chain of transmission and avoiding secondary cases. To optimize the planning and evaluation of TB control, the NTP is organized into the following components and subcomponents: • Planning and administration, including financing • Health Services: this area implements the recommendations of the Ministry of Health to control TB. To do this, its activities are done in agreement with the states and municipalities in order to implement activities that lead to the control of TB. • Prevention • Diagnosis • Epidemiologic surveillance • Monitoring and evaluation • Research • Development of institutions and staff • Communication and Social mobilization • International cooperation, including such for research: the aim is to incorporate research into the regular schedule of health services, thereby improving the quality and access to new technologies. The work with international programs for prevention and control of TB helps to keep as a priority the recommendations for the country and to validate the associated actions. Roles and responsibilities of state and municipal levels basic health units; reference centers. At the state level, their responsibilities for TB control and prevention include such areas as management of all the state activities related to TB; monitoring the epidemiologic indicators as well as those goals established at the municipal level; consolidating and analyzing the data generated by the information system, and seeing that those are disseminated through bulletins or reports, as well as using that information for further planning, monitoring and evaluation; promoting training and capacity building and assuring that this is integrated into other teaching and service opportunities; advocate for quality assurance at the state level; integrating with the State and Regional References labs, as well as with secondary and tertiary referral units; promote development of research; supporting municipal programs; stimulating the participation of civil society. At the municipal level, key actions and responsibilities include such activities as: monitoring the epidemiologic indicators; coordinating active case finding among those with respiratory symptoms, supervising local TB activities; conducting contact tracing; notifying SINAN of TB cases within municipality; assuring that diagnosis and treatment completion are conducting according to guidelines; making sure that anti-TB drugs are available and distributing them to health centers; providing BCG vaccination to newborns; coordinating with the Family Health Services and other entities. The Basic Health units are essential to service delivery of good TB control and prevention activities in their communities. They are required to carry out searches for those who have respiratory symptoms – whether at the unit or in the home. They also are expected to do sputum collection and to request cultures, DST, and species identification when appropriate. As far as treatment, they perform DOT (at the health unit or as a home visit) and monitoring of cases (including defaulters) or to refer them as needed. Their responsibilities also include offering USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 137 HIV testing to all TB patients. They are also expected to provide IPT, to search for adverse effects of anti-TB drugs, vaccinate babies with BCG, and do PPD skin testing for contacts or suspected cases of TB, or to refer those patients upwards in the system. As for information systems, they are to fill out the surveillance forms, and to refer difficult cases, cases with adverse effects of treatment, defaulters, and any cases that appear to be MDR-TB. Reference centers have more complex and higher levels of responsibility, which include coordination with state and municipal levels. Some of their activities encompass further investigation of more challenging cases, including the use of bronchoscopy, biopsy, chest radiography and CT of the chest, and analysis of body fluids to reach a diagnosis. Ideally they should have multi-disciplinary teams that include other health professionals, social workers and mental health staff. For further information: Link: http://portal.saude.gov.br/portal/saude/profissional/visualizar_texto.cfm?idtxt=31101 138 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 139 ANNEX H: SUMMARIZED HIERARCHICAL ORGANOGRAM OF CRPHF AND FIOCRUZ Annex Figure 20: 2004 Hierarchical Organogram representing FIOCRUZ and CRPHF. MoH: Ministry of Health; FIOCRUZ: Fundação Oswaldo Cruz; CRPHF: Centro de Referencia Professor Helio Fraga; SVS: Secretaria de Vigilância em Saúde (Health Surveillance Vigilance Secretariate); NTP: National TB Program. Annex Figure 21: 2008 Onwards Hierarchical Organogram representing FIOCRUZ and CRPHF. MoH: Ministry of Health; FIOCRUZ: Fundação Oswaldo Cruz; CRPHF: Centro de Referencia Professor Helio Fraga; SVS: Secretaria de Vigilância em Saúde (Health Surveillance Vigilance Secretariate); ENSP: Escola Nacional de Saúde Pública (National Public Health Institute); NTP: National TB Program 140 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 141 ANNEX I: OVERALL TB DRUG DISTRIBUTION FLOW Annex Figure 22: First and Second Line TB drug Distribution Flow.mMSH: Management Science for Health; CRPHF: Centro de Referencia Professor Hélio Fraga. 142 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 143 ANNEX J: DISCLOSURE OF ANY CONFLICTS OF INTEREST Mr. Valdir de Souza Pinto had worked as a consultant for Management Sciences for Health (MSH) prior to his participation in the Evaluation Team. To avoid any possible conflict of interest, he did not participate in data collection, analysis or writing of the evaluation of MSH TB projects. NO OTHER CONFLICTS OF INTEREST EXIST FOR MEMBERS OF THE EVALUATION TEAM. 144 USAID/BRAZIL: PERFORMANCE EVALUATION OF THE TUBERCULOSIS PORTFOLIO, 2001-2012 For more information, please visit http://www.ghtechproject.com/resources GH Tech Bridge II Project 1725 Eye Street NW, Suite 300 Washington, DC 20006 Phone: (202) 349-3900 Fax: (202) 349-3915 www.ghtechproject.com