EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) JULY 2009 This publication was produced for review by the United States Agency for International Development. It was prepared through the Global Health Technical Assistance Project. Evaluation of the Tuberculosis Control Assistance Program (TB CAP) DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. This document (Report No. 08-001-191) is available in printed or online versions. Online documents can be located in the GH Tech Web site library at www.ghtechproject.com/ resources.aspx. Documents are also made available through the Development Experience Clearinghouse (www.dec.org). Additional information can be obtained from: The Global Health Technical Assistance Project 1250 Eye St., NW, Suite 1100 Washington, DC 20005 Tel: (202) 521-1900 Fax: (202) 521-1901 info@ghtechproject.com This document was submitted by The QED Group, LLC, with CAMRIS International and Social & Scientific Systems, Inc., to the United States Agency for International Development under USAID Contract No. GHS-I-00-05-00005-00. EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) i ACKNOWLEDGEMENTS The TB CAP evaluation team would like to thank the many generous people who contributed their time and energy to help us complete this challenging, complex, yet fascinating task. Prominent among them we thank Ms. Cheri Vincent, TB CAP Cognizant Technical Officer, and her colleagues at the Bureau for Global Health and Regional Bureaus, U.S. Agency for International Development Dr. Maarten van Cleeff, TB CAP Project Director; Dr. Jeroen van Gorkom, TB CAP Project Deputy Director; Dr. Ersin Topcuoglu, TB CAP Monitoring and Evaluation Officer; Ms. Inge Sasburg, TB CAP Financial Controller; and the many other staff in the TB CAP Project Management Unit for enthusiastically, meticulously, and openly explaining the many facets of how TB CAP works The national TB managers and staff who set aside time from the inspiring work they do to describe their programs, experiences with TB CAP, and suggestions for how best to control TB USAID Mission staff and TB CAP partners who are earnestly supporting TB control efforts, for sharing their opinions about the many challenges and remarkable successes they are witnessing, and for their efficient organization of the country visits The many technical experts and managers from TB organizations worldwide with whom we spoke, for providing their insight into global TB control efforts and international development work Ms. Elsa Berhane, Program Manager for the TB CAP Evaluation at the Global Health Technical Assistance Project (GHTech), for her dedication, hard work, and seemingly endless patience. Finally, we wish to express our admiration for those working at the sharp end, especially the many unrecognized but vital DOTS community volunteers. Quotes from two of these extraordinary people stay in our minds: One woman, herself HIV-infected, when asked why she was providing TB support, said, ―I just feel that I should help others the way others have been helping me.‖ Another community volunteer, asked what one thing could make her work easier and better, told us, ―I wish I could have more education about TB, so I would know more to help my patients.‖ ii EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) iii ACRONYMS AIDS Acquired immune deficiency syndrome APA Annual plan of activities ART Antiretroviral therapy ATS American Thoracic Society CDC US Centers for Disease Control and Prevention CDR Case detection rate CPT Cotrimoxazole preventive therapy CTO Cognizant technical officer DHT District health team DOT Directly observed treatment DOTS The WHO-recommended treatment strategy DRS Anti-tuberculosis drug resistance surveillance EQA External quality assurance FHI Family Health International GDF Global TB Drug Facility Global Fund Global Fund to Fight AIDS, Tuberculosis and Malaria GH Tech Global Health Technical Assistance Project GLC Green Light Committee HIV Human immunodeficiency virus HRD Human resource development IC Infection control IR Intermediate Results ISTC International Standards for Tuberculosis Care JATA Japan Anti-Tuberculosis Association KNCV Royal Netherlands Tuberculosis Foundation M&E Monitoring and evaluation MDR Multidrug resistance MDR TB Multidrug-resistant TB MOU Memorandum of Understanding MSH Management Sciences for Health NGO Nongovernmental organization NTP National Tuberculosis Control Program NTTLCP National Tuberculosis and Leprosy Control Program OR Operations research PEPFAR President's Emergency Plan for AIDS Relief PHC Primary health care PMU Program Management Unit iv EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) PVO Private voluntary organization TA Technical assistance TB Tuberculosis TB CAP Tuberculosis Control Assistance Program TBCTA Tuberculosis Coalition for Technical Assistance TB/HIV HIV infection in TB patients TREAT TB USAID Technology, Research, Education, and Technical Assistance for TB Project The Union International Union Against Tuberculosis and Lung Disease USAID United States Agency for International Development WHO World Health Organization EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) v CONTENTS Acknowledgements .................................................................................................................................. i Acronyms ................................................................................................................................................ iii Executive Summary ................................................................................................................................ vii I. Background.................................................................................................................................... 1 The TBCTA Project (2000–2005) ................................................................................................... 1 The TB CAP Project (2006–2010) .................................................................................................. 2 II. Evaluation Purpose and Methodology ............................................................................................ 3 The Evaluation Process ................................................................................................................. 3 III. Findings and Recommendations .................................................................................................... 5 A. Overarching Findings and Recommendations ............................................................................ 5 B. Strategy and Governance .......................................................................................................... 6 C. Partners and Partner Coordination ............................................................................................. 8 D. Administration and Management ................................................................................................ 8 E. Monitoring and Evaluation ........................................................................................................ 12 F. Knowledge Management .......................................................................................................... 13 G. Planning and Implementation .................................................................................................. 14 H. Results and Impact .................................................................................................................. 20 I. Operations Research ................................................................................................................ 30 IV. Compiled Recommendations........................................................................................................ 31 Overarching Recommendations ................................................................................................... 31 Strategy and Governance ............................................................................................................ 31 Partners and Partner Collaboration .............................................................................................. 32 Operational Structure ................................................................................................................... 32 Operational Management ............................................................................................................. 33 Financial Management ................................................................................................................. 33 Monitoring and Evaluation ............................................................................................................ 33 Responding to Requests for Support ............................................................................................ 34 Assessing Country Needs ............................................................................................................ 34 Selecting Coordinating and Collaboration Partners ....................................................................... 34 Creating Country Workplans ........................................................................................................ 35 Approving Country Workplans ...................................................................................................... 35 Implementing Country Workplans ................................................................................................. 35 Core and Regional Projects .......................................................................................................... 35 Global Results and Impact ........................................................................................................... 36 Impact on National TB Control Programs...................................................................................... 36 vi EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) Impact on Patients and TB epidemiology ...................................................................................... 36 Results and Impact by Intermediate Result (IR1) .......................................................................... 36 Impact by Intermediate Result (IR2) ............................................................................................. 36 Impact by Intermediate Result (IR3) ............................................................................................. 37 Impact by Intermediate Result (IR4) ............................................................................................. 37 Results and Impact by Intermediate Result (IR5) .......................................................................... 37 Cross-cutting Results and Impact ................................................................................................. 38 Operations Research ................................................................................................................... 38 ANNEXES Annex I. Scope of Work ......................................................................................................................... 39 Annex II. TB CAP Desk Audit ................................................................................................................. 53 Annex III. Evaluation Questionnaires ..................................................................................................... 55 Annex IV. Field Visit Itineraries .............................................................................................................. 57 Annex V. Washington Telephone Interview Schedule ............................................................................ 63 Annex VI. Field interviews and Interviews with TB CAP Stakeholders .................................................... 67 Annex VII. Proposed TB Cap Indicators ................................................................................................. 81 Annex VIII. References .......................................................................................................................... 85 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) vii EXECUTIVE SUMMARY In the 1990s the United States Government began to increase its funding for international tuberculosis (TB) control through the United States Agency for International Development (USAID). The USAID TB Coalition for Technical Assistance (TBCTA) project, begun in 2000, demonstrated that US assistance could have a tangible positive impact by channelling USAID support through a partnership of highly experienced TB control organizations to deliver technical assistance (TA) in small-scale projects. The successor TB Control Assistance Program (TB CAP) was intended to expand on this with TA on additional aspects of TB control (such as country-level drug management systems and TB/HIV integration) and greater capacity building in more countries. In 2009 USAID commissioned the Global Health Technical Assistance (GH Tech) Project to conduct a two-month evaluation of TB CAP by a four-person team of independent consultants. The team analyzed past and present TB CAP activities and management through a desk review of TB CAP documents and materials, field visits to four countries, meetings with technical agencies and donors, and interviews with stakeholders in 16 other TB CAP countries, including Mission, TB CAP partner, and national TB program staff. It found that In countries it directly supported, TB CAP has improved capacity to provide better-quality services to control TB and has indirectly strengthened other countries through its contributions to global TB control efforts. There was near-universal praise for the quality of TA provided by TB CAP and its partners, and for many of the tools that TB CAP has helped create, test, and disseminate. There were varying degrees of satisfaction with in-country coordination of TB CAP activities, although the perception of most Missions was that coordination issues were quickly and effectively resolved. National TB control programs (NTPs) and Missions would like to see USAID continue to support TA, are concerned about the impact on their TB control efforts if the support ends, and hope that the range, quality, and convenience of current TB CAP services are maintained. Missions are increasingly using TB CAP to implement their TB programs, and the President’s Emergency Plan for AIDS Relief (PEPFAR) has increased its funding through TB CAP for TB￾related HIV activities. The organizational model of TB CAP (a partnership of cooperating agencies with a program management unit providing administrative and management oversight) is functioning well, although it does not always rigorously adhere to its own policies and procedures.1 TB CAP support would be more efficient and have more impact if it were better aligned and harmonized at the Mission level with the activities of PEPFAR and the Global Fund to Fight AIDS, TB, and Malaria. The impact and management of TB CAP is hampered by the one-year focus of its workplans and lengthy delays in workplan development and approval, compounded by the need to create and approve a new workplan each year. 1 TB CAP Operational Manual, Version January 2009 viii EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) TB CAP is a highly successful project that has had significant impact, is well-managed, and is achieving its goals. Given the progress TB CAP has made in some countries to improve conditions for TB control programs worldwide, the evaluation team strongly recommends that USAID continue and expand its support for it. Although TB CAP has effectively achieved many of its project expectations, USAID has a more ambitious goal: If USAID is ―to contribute significantly to the global reduction of morbidity and mortality associated with tuberculosis,‖2 as it desires, it should take a more long-range and ambitious approach to the TA it provides. Certainly the TB CAP project has demonstrated sound strategic principles that should be even more rigorously embedded in future USAID support for TA, notably the TB CAP emphasis on enhancing the institutional and managerial capacity of national systems of TB control supporting a variety of technical areas, with a focus on ensuring that the fundamental building blocks of an effective TB control program are in place, especially effective drug management and sufficient capacity and coordination of laboratory networks strengthening and integrating directly observed treatment (DOTS) systems between government health facilities and nongovernmental institutions and providers, such as community-based volunteers customizing support to align it with each country’s state of development and national TB control strategy and priorities, policies, and technical guidelines. To realize its overarching goals USAID should shift its focus from short-term, high-yield activities toward longer-term, country-wide activities that convert the experiences of TB CAP for greater national and global impact. This would not require wholesale refashioning of the current TB CAP model, but it would require more closely aligning some of its elements: USAID should encourage its Missions to develop multiyear workplans for TB control, linked to national strategies and plans, and formally support their approval, while providing for annual adjustments and resourcing based on needs, progress made, and availability of funds. Longer-term planning would make country TB control measures more sound and help stimulate investment in technical capacity to meet demand. The future implementing mechanism should include a governing board to deal with issues of governance, program strategy, and budgeting, mandated to develop and implement a strategic agenda for scale-up of support to countries. Medium- and long-term strategic oversight is essential to ensure the best possible support to countries, maximum benefits to the global community, and efficient and responsible use of USAID funding. The administrative arm should be empowered within the sphere of its responsibility to make management decisions—seeking USAID or board-level approval when necessary and operating efficiently and transparently. USAID should also consider improving the TB CAP model, which has already demonstrated its effectiveness, to assure the quality of TB control measures in health facilities, promote timelier financial forecasting and reporting by implementing agencies, and adopt more effective approaches to collecting, analyzing, and sharing experiences. To accomplish overarching USAID goals, this mission should be reset at an even higher level and structured to achieve it. 2 US Agency for International Development: Expanded Response to Tuberculosis, January 2009. EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 1 I. BACKGROUND From the late 18th through the early 20th century, tuberculosis (TB) was the leading cause of death in dozens of countries. In 1993 the profile of TB as a health threat was raised when the World Health Organization (WHO) declared TB a global emergency. The announcement, combined with the launch of the directly observed therapy strategy (DOTS) and international TB program guidelines, led to a scale-up of TB programs and a demand for technical assistance (TA). In 2000 the Dutch Government and the WHO, in collaboration with the Royal Netherlands Tuberculosis Foundation (KNCV), organized the Amsterdam Ministerial Conference on Tuberculosis and Sustainable Development, which produced the Amsterdam Declaration to Stop TB—a defining moment in the restructuring of global efforts to control TB. In it the ministerial delegations of the 20 countries with the highest burden of TB called for action. During the Amsterdam Conference six major international organizations working on TB control formed a coalition to create more cohesive TA for TB in each country: the Tuberculosis Coalition for Technical Assistance (TBCTA). Funding for TB control had become available to the United States Agency for International Development (USAID) in 1998. Partly in response to the Amsterdam Declaration, USAID began to explore options for providing technical expertise through its Global Health Bureau and USAID regional offices and country missions. TBCTA offered its support to USAID in September 2000 and shortly thereafter a cooperative agreement was signed. THE TBCTA PROJECT (2000–2005) TBCTA then established formal operational structures. As prime contractor KNCV made subagreements with coalition partners. The structure consisted of two coordinating bodies: (1) the board of directors, with representation from each TBCTA partner, made major decisions; and (2) the project management unit (PMU), housed at KNCV, was responsible for day-to-day activities. The cooperative agreement covered September 2000–September 2005 and was later extended through December 2006. Originally, the five-year budget was almost US$29 million. Because of the increase in USAID’s overall budget, there has been greater than anticipated demand for TB TA from Missions; at the end of Year 3 the ceiling was increased to US$42.8 million. In April 2004, an external evaluation of the performance, impact, and lessons learned from the TBCTA project concluded that [TBCTA] was an excellent choice to serve as an initial collaborator for USAID’s entry into the TB arena, linking USAID with experienced and well-respected international expertise that is generally not available in U.S. institutions. The new TBCTA for the first time brought the key international TB organizations together into an operational, program-focused structure. TBCTA is now recognized as an innovative Partnership that speaks with a single voice on major TB issues in countries where USAID works. The coalition has successfully collaborated to address international and national issues that have limited the expansion of DOTS and has demonstrated that it is much more than the sum of its individual Partner institutions. USAID health officers We commit ourselves to accelerate action against tuberculosis through…ensuring that sufficient human & financial resources are available on a sustainable basis & expanded to meet the challenges of stopping tuberculosis.. and that the implementation capacity is developed to utilize these resources efficiently & effectively. Amsterdam Declaration to Stop TB 24 March 2000 2 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) value this coordinated approach to TB, and it has been suggested as a model for other health programs (e.g., HIV/AIDS, malaria). TBCTA’s final report was published in 2007. THE TB CAP PROJECT (2006–2010) In 2005 USAID requested applications for a TB Control Assistance Program (TB CAP) to expand upon the work of TBCTA, giving more emphasis to TB/HIV integrated approaches, involving new partners at the country level (especially the private sector), and improving drug management systems. TB CAP was also expected to have an effective monitoring and evaluation (M&E) system to strengthen program planning. It would emphasize building human, institutional, and financial capacity for DOTS implementation that could be sustained past the life of the project. The TB CAP project was awarded to KNCV, which signed a second cooperative agreement with USAID. The TB CAP cooperative agreement covers September 2005–September 2010 and has a total estimated value of US$150 million. KNCV’s six subgrantees are the American Thoracic Society (ATS), Family Health International (FHI), International Union Against Tuberculosis and Lung Disease (The Union), Japan Anti-Tuberculosis Association (JATA), Management Sciences for Health (MSH), and the WHO. The Centers for Disease Control (CDC) is a member of the coalition but funds for this agreement do not flow to or through CDC. A description of KNCV and its TB CAP subgrantee partners is located in Annex 1 (Scope of Work) to this report. TB CAP assistance has two purposes: (1) to improve and expand the capacity of USAID to respond to the global TB epidemic by providing well-coordinated state-of-the-art, context-appropriate, technically sound, and cost-effective consultation and TA to high-prevalence countries and Missions, and to build additional global capacity for providing TA; and (2) to complement and expand global TB control efforts, such as those advocated by the Stop TB Partnership. The ultimate goal is to maximize efforts to accelerate the pace of DOTS expansion to meet global targets, in collaboration with other global TB partners. TB CAP has the following goals in countries where it invests: 90% of public clinics implementing DOTS At least 70% case detection rate At least 85% treatment success rate and/or cure rate 75% of countries meeting MDR TB quality standards as defined by TB CAP 100% of countries effectively coordinating nationwide TB and HIV programs. TB CAP’s five intermediate results (IRs) are IR 1: Increased political commitment for DOTS IR 2: Strengthened and expanded DOTS programs IR 3: Increased public and private DOTS participation and collaboration IR 4: Increased and strengthened TB and HIV/AIDS coordinated activities IR 5: Improved human and institutional capacity. EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 3 II. EVALUATION PURPOSE AND METHODOLOGY In 2009 USAID commissioned the Global Health Technical Assistance (GH Tech) Project to independently evaluate TB CAP’s performance, impact, and lessons learned against the project’s objectives and its contribution to the objectives of the Stop TB Partnership. The evaluation was intended to analyze common factors, patterns for success, and areas for improvement. It was charged with answering these principal questions: What is the evidence that TB CAP has or will have helped accelerate DOTS expansion to reach global targets and build institutional and individual capacity for TB control in targeted countries? Is this the optimal framework for a multifaceted mechanism within the Stop TB Partnership, taking into consideration USAID’s comparative advantages? The results were to be used to provide feedback to TB CAP on project operations and management and recommendations to TB CAP and USAID on doing the work better. The evaluation was also to inform USAID on design of future TB activities. THE EVALUATION PROCESS The four-person consultant team consisted of Dr. Stephen C. Joseph (team leader), Dr. Fabio Luelmo, Mr. Timothy Ryan, and Dr. Stanley Scheyer. One team member first carried out a desk audit of TB CAP documents and materials and gave the report to the team as work began.3 The team was also granted access to the TB CAP E-room of project documents, managed by the KNCV PMU. Two team members attended the meeting of the TB CAP Board of Directors February 18–19, 2009, and pre-evaluation meetings were held with TB CAP Project Managers. During the week of March 10–14, the full team gathered in The Hague, Netherlands, at the PMU/KNCV headquarters for extensive briefings by the PMU and for planning the evaluation. Separate questionnaires (Annex III) were formulated to guide interviews and country visits with the three key referent groups: Mission staff, country coordinating partners, and National Tuberculosis Program (NTP) managers. After those meetings, two team members carried out a field visit in Indonesia, a third visited WHO in Geneva and the headquarters of the Union in Paris for intensive discussions, and the fourth returned to Washington to lay the groundwork for an extensive series of telephone interviews. Two team members then visited Nigeria, Mozambique, and Uganda. Each visit began with 1-2 days of meetings in the capital with USAID and national and TB CAP partner representatives. This was followed by 2-3 days of fieldwork, usually in rural areas at a considerable distance from the capital, to observe the TB control activities of both TB CAP and other organizations. During each visit, health and TB control activities were observed at all levels of the health system: community health workers and DOTS volunteers, DOTS and general activities at primary and secondary health centers, and district and regional hospitals. Informational discussions were held with TB, health, and governance authorities at each level. At DOTS sites the team looked at clinical screening and treatment, laboratory activities and registers, clinical and TB control registers, patient follow-up, supervision of community volunteers, and stockage and distribution of drugs. The final phase of each visit consisted of 1-2 days of meetings and debriefings in the capital and visits to national reference laboratories, training sites, and drug procurement and stockage facilities. There were discussions with PEPFAR personnel on TB/HIV coordination, and with CDC personnel on survey and research activities. 3 The desk audit (Annex II) is available from USAID as a separate document. 4 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) Meanwhile, team members in Washington carried out telephone interviews with USAID staff, TB CAP partners in country, and NTP management in16 of the 20 TB CAP countries proposed by USAID. Individuals who did not return calls were sent a written questionnaire. Telephone and in-person interviews were also conducted with key individuals from partner and USAID staff in Washington, DC.4 4 Itineraries of the field and country visits can be found in Annex IV, a register of those interviewed by telephone and email in Annex V, and a list of contacts made during the evaluation in Annex VI. EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 5 III. FINDINGS AND RECOMMENDATIONS A. OVERARCHING FINDINGS AND RECOMMENDATIONS In its first three years, TB CAP has strengthened capacity of NTPs, local NGOs, and private voluntary organizations (PVOs) to provide better-quality services to control TB in 17 of the 22 countries with the highest burden of TB. There is general recognition that without TB CAP support countries would not have achieved as much progress, particularly in the areas of community DOTS, engaging public and private care providers, building up laboratories, and TB/HIV integrated approaches. Furthermore, support from TB CAP has helped reinforce the capacity of other countries to manage TB programs. TB CAP has had a major positive effect on TB control activities worldwide, which can be at least partly attributed to the expertise of the organizations chosen to provide TA. There is great satisfaction with the technical quality of assistance to NTPs and Missions—an increasing number of Missions are choosing TB CAP to implement TB activities and returning for additional support. The governance structure of TB CAP promotes collaboration among partners, although the respective roles and responsibilities of the TB CAP Board of Directors and the PMU are not consistently applied. The impact and management of TB CAP are handicapped by the one-year timeframe of its workplans and lengthy delays in workplan development and approval. This timeframe may contribute to too short-range a mindset on the part of partners, the board, the PMU, and USAID. Those interviewed would like to see USAID support for TB TA continue. Both NTPs and Missions are concerned about the impact on their TB control efforts if support ends and hope that the range, quality and convenience of current TB CAP services are maintained. TB CAP has elevated the USAID profile within the TB arena and established it as one of the most influential players in TB control worldwide. As a result, USAID is now in an excellent position to contribute significantly to reduction of the morbidity and mortality associated with TB globally. Recommendations USAID should continue its support for TB TA, but with a more long-range and ambitious approach. If USAID chooses to implement a follow-on project to TB CAP, there should be only a minimal gap between the end of the current project and the beginning of the new one. In fact, USAID should plan for some overlap to account for the initial period of planning new project activities that would be a necessary precursor to implementation and thus ensure smooth continuity from one project to the next. The new project should be launched within the original TB CAP timeframe; the period of time covered by an extension is an exceptional opportunity to complete unfinished work. In the current project, TB CAP should not expand the number of countries supported beyond those already identified for APA5 unless the project period is extended with sufficient financial resources. Within existing resources, the focus should be on qualitatively improving current assistance. The USAID implementing mechanism should have a board composed and empowered to focus on issues of governance, program strategy, and budgeting, and given a mandate from USAID to design and implement a strategic agenda for scale-up of country support. The PMU should be empowered to make management decisions and recommendations for board-level decisions. USAID should encourage and provide guidance to Missions to develop multiyear workplans for TB control, linked to national TB control strategies, and formally support the approval of such workplans 6 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) while providing for annual adjustments of activities and resourcing based on needs, progress, and funds. Longer-term planning with the anticipation of funds would make country TB control measures more sound and help stimulate investment in the technical capacity needed to meet current and projected country demand. B. STRATEGY AND GOVERNANCE The TB CAP Board of Directors is responsible for developing policies, strategic planning, and budgeting. It also gives direction to program implementation and monitors its progress and quality. The board resolves issues and determines ways to improve core, regional, and country-funded activities.5 The PMU is responsible for overseeing day-to-day management of all projects. It must facilitate the drafting and approval of plans for TB CAP work, transfer funds to reimburse partners for work performed, and monitor and report on activities and cumulative impact. In practice, however, these roles are not strictly or consistently applied. The work of the board puts too much emphasis on management and not enough time and effort on medium- to long-term strategic planning. Members of the board are well qualified to make strategic program decisions, but are often asked to make day-to-day program management decisions, which would be better left to the PMU. For a variety of reasons (such as competing interests for the short time that board members have together, a lack of prompting and preparation of strategic issues for discussion, or a failure to assert its role as a strategically focused body), this strength is not being used optimally. For instance, the following issues would ideally be addressed by the board: TB CAP support in countries would be more efficient and have more impact if it were better aligned with the activities of the Global Fund to Fight AIDS, TB and Malaria (Global Fund) and with PEPFAR. The deterioration of primary health care services and facilities in a number of countries in recent years makes it more difficult to provide quality TB diagnostic and treatment services at the peripheral service level. Concerns about increased drug resistance and TB/HIV coinfection have led governments and the international community to give unbalanced emphasis to these problems at the expense of strengthening community and primary peripheral health service-based DOT. There are areas where TB CAP needs to increase its awareness and competence, especially advocacy, information management and dissemination, quality assurance, and the best indicators of TB control. Widespread dissemination and uptake of products and experience gained from TB CAP activities requires an effective marketing strategy. Discussions with NTPs and partners and field observations identified a strongly felt need for TB CAP to support field operations research and use locally available data. The USAID TREAT TB project offers a significant mutual opportunity. However, it will be very important that the two projects, within their respective roles, are carefully coordinated to work synergistically. Although the board is responsible6 for monitoring the progress of program implementation and quality, it has no clear mechanism for objectively evaluating and addressing underperformance. 5 TB CAP Operational Manual, Version January 2009 6 ibid EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 7 The board should be responsible for approving all country and core workplans and budgets and determining the coordinating partner for each project, but the current approach allows for ambiguity because it does not always place responsibility for the decision squarely on the board. The indicators currently used in TB CAP are not always the most appropriate for evaluating the impact on TB programs given the duration of project interventions. Recommendations The roles and responsibilities of the board, working groups, and the PMU should be clearly communicated, and the operational practices of each revisited to ensure they are acting on these mandates. Working groups should institute regular meetings, electronically or telephonically when possible, so that they can keep strategies for core and country projects up to date. TB CAP assistance in countries with weak peripheral DOT services should prioritize reinforcing DOT, consistent with the Stop TB Strategy, while also addressing, in a balanced way, the emerging problems of drug resistance, infection control, and TB/HIV coinfection. Although there are many priority needs, TB CAP should focus on ensuring that the fundamental building blocks of an effective TB control program are in place, especially drug management systems and the capacity and coordination of laboratory networks. TB CAP should shift focus from short-term, high-yield projects toward longer-term, country-wide projects that scale up the experiences it has gained and convert them into national improvements and thus greater global impact. TB CAP should intensify its efforts to build the institutional and managerial capacity of national systems of TB control. In its country workplans and strategies, TB CAP should give priority to supporting the national strategy and guidelines of each country, consistent with the state of development and TB control needs. DOTS health facility and community volunteer systems should be strengthened and integrated. More capability in advocacy, information management and dissemination, quality assurance, and best indicators of TB control should be acquired by TB CAP, represented in governing bodies, and built into future program strategy. TB CAP should offer Missions TA to increase their understanding of strategic and programmatic issues in TB control and coordination of TB with other health activities, such as HIV and primary health care. TB CAP should provide TA on operations research as needed to improve TB control at the service delivery level. Furthermore, USAID should ensure that coordination between TREAT TB and TB CAP, starting from Washington and cascading down into countries, is carefully managed and potential areas of overlap are identified in advance rather than addressed when issues arise. The PMU should recommend country as well as core projects for board decisions on coordinating partners. This will be especially important during any transitional period between TB CAP and a follow-on project. Partners being considered should recuse themselves from these board decisions. 8 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) C. PARTNERS AND PARTNER COORDINATION The TB CAP partnership model is unusual for USAID but is working well and allows for flexibility, innovation, strength-in-diversity, and acceptability to countries. Missions and NTPs appreciate the range and flexibility of TB CAP TA. There is a consensus that TB CAP’s current partners represent the right mix of capabilities, although some areas need strengthening. Partners reported that working together in TB CAP has benefited them by increasing their own capacities, amplifying the impact of their work, and providing them with an entry point into regions and countries where they otherwise might not have operated. They also felt that TB CAP had improved cooperation between agencies and reduced competition. TB CAP support has brought these organizations to the same table and catalyzed consensus on standards and approaches so that they are increasingly speaking with one voice. New synergies between partners likely would not exist if the activities if there were no partnership. While the technical quality and relevance of its assistance was highly praised, several individuals reported insufficient coordination of TB CAP activities in some countries, particularly in the first years of the project. One common issue was that the country coordinating partner had been ineffective. This was usually addressed by finding a coordinating partner that was more capable of carrying out the work, but in some cases implementation was delayed. In countries where the TB CAP coordinating partner could have benefitted from senior support from the home office, this is often not recognized or is insufficient. The USAID brand is associated with TB CAP activities, as is the TB CAP brand itself. Still, there is some confusion about what TB CAP is (donor or project), the role of USAID, and whether TB CAP or the partner is implementing the project. This is likely unavoidable given the high profile of the partners implementing TB CAP. In the countries visited, the USAID brand was prominent. Recommendations The board should define clear policies and criteria for the selection of collaborating partners, such as constraints on the number of partners involved in particular types of projects, to ensure optimal composition of project teams. The board should set a minimal acceptable amount of visiting supportive supervision for TB CAP field officers, and partners should maintain that level of support for all programs they coordinate, increasing the amount based on the needs and capacity of field officers. The board should establish base criteria for measuring the performance of coordinating and collaborating partners and set a policy for responding to both underperformance and unwillingness to carry out agreed-upon responsibilities, and the PMU should develop systems for monitoring and regularly reporting on such issues to the board for action. Some partners are less effective than others at coordinating country or core projects. These partners should remain collaborating partners and not be responsible for coordination. D. ADMINISTRATION AND MANAGEMENT The PMU based within KNCV in The Hague facilitates meetings and discussion among TB CAP governing bodies (the board and working groups) and relies on the strategic guidance and decisions of these bodies to select TB CAP activities, balanced by direction received from Missions, NTPs, and the TB CAP Cognizant Technical Officer (CTO) in Washington, D.C. The PMU routinely monitors TB CAP activities and payment to agents for work performed. Clear and accurate details of these activities must be communicated to USAID throughout each year in specific formats, ranging from qualitative information (project achievements) to technical data (performance and EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 9 impact indicators) and financial figures (budget requirements and expenditures). All initial PMU staff were KNCV staff except one (an M&E officer seconded from MSH). All are paid by USAID. As the number, scale, and complexity of TB CAP country activities increased, however, the initial staffing proved insufficient. USAID and TB CAP agreed that the PMU required more technical expertise to ensure that activity workplans were of consistent technical quality and benefited from the experience gained from other TB CAP projects. The PMU has therefore augmented the original staff with several technical experts, each dedicated to a specific area (drug resistance, human resource development, infection control, and TB/HIV coinfection). Operational Structure Although the TB CAP project structure is unusual for USAID, the staffing seems to fit the work required and staff has been increased only when the need was clear and only after discussion with USAID. USAID staff consider staffing and other operational costs to be within acceptable limits and the organization to be responsive and effective. In some cases the board clearly takes the lead on decision-making (such as approval of core projects). In other cases, there was a perception that the PMU did the decision-making (such as allocation of coordinating partners to activities). With its new technical focal points in place, the PMU now seems better equipped to provide strategic technical oversight of TB CAP activities; however, it is not clear that this is in the mandate of the PMU; nor is it clear how this oversight would be implemented, monitored, or adjusted. Recommendations The PMU should exercise complete responsibility for making and executing management decisions, seeking USAID or board advice or approval only when necessary. The PMU should recognize that all governance-related decisions, including selection of projects, allocation of projects to partners. and agreement on TB CAP strategy is the responsibility of the board. The PMU should clearly define the role of its technical focal points and communicate this to all parties. Areas where they could be useful would be to collect and document strategies in technical areas as defined by governing bodies and ensure that they are consistently reflected in actual TB CAP workplans. The PMU should systematically review the general progress of TB CAP regularly and report on it to the board. The PMU should be authorized to define and provide guidance on country project support, in consultation with NTPs and partners. Operational Management KNCV seems in every way to be meeting the terms of its cooperative agreement with USAID, and its coordinating partners seem to be meeting the terms of their subagreements with KNCV. The PMU manager is considered to be technically astute, accountable and responsible, flexible and open to new ideas and constructive criticism, and a nearly ideal political liaison with donors, partners, and country programs. With occasional exceptions partners feel that the PMU is doing a good job of coordinating country, regional, and core projects. Core projects, however, are not always adequately linked to country projects. 10 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) The TB CAP CTO seems to have provided helpful, beneficial, direct oversight and not obstructed progress with unexpected requests. Several interviewees commented that the CTO rapidly intervened upon request and effectively removed obstacles, such as when delays occurred at Missions. Several interviewees had a perception that the line between KNCV and the PMU was occasionally blurred and that the PMU did not always support the interests of all partners equally, although no clear evidence was found to support this. What the PMU does to ensure effective, appropriate workplan development is unclear. Some workplans included activities that seemed unrealistic given the workplan timeframe. There are debilitating delays in the process of drafting and approving country workplans. These delays do not seem to be the result of bottlenecks in the PMU, and the PMU seems to react quickly to address them, but it does not seem to have a system for routinely monitoring the steps in workplan development and approval against expected lead times in order to identify and address such delays before they become critical. Recommendations The PMU should clarify its role in the development of workplans and consistently adhere to the definition. If it is to be responsible for pre-planning workplans before a coordinating partner is selected, this should always be the case, and potential coordinating partners should not be involved. If it is to be responsible for ensuring the technical quality and strategic relevance of workplans, this too should be done consistently, and the criteria should be clearly communicated to partners in advance. If it is to delegate all responsibility for workplan preparation, it should not participate at all and instead should consistently seek partners to conduct this work. The PMU should establish a system for registering potential new core and country projects in a Web￾based system accessible to all partners and set out a standardized process for workplan development, approval, and implementation. It should also take responsibility for updating this system with the dates when events occur, so that it will be possible to quickly and easily identify projects at different stages of development and flag those that may need intervention. Such a process would also facilitate general reporting and, if financial details were included, make financial tracking more accurate and timely. There are several Web-based procurement systems upon which to model such a system. Financial Management Although the financial management of TB CAP was reviewed, a formal detailed audit was considered not to be within the scope of the evaluation. Based on the first two annual independent audits of TB CAP (a third audit is pending), however, it appears that KNCV has fairly represented its work in its official annual financial statement to USAID and its cost-share statement, and is compliant with the financial requirements in the KNCV/USAID cooperative agreements. The auditors noted no matters involving financial control and operation that they considered to be material weaknesses. Furthermore, KNCV seems to have responded to many of the changes suggested by the auditors and indicating a readiness to improve performance even when improvements were not mandatory. The cooperative agreement between USAID and TB CAP has a 17.8% cost share over the course of the entire project. In the first three years of TB CAP, the cost share far exceeded this (approximately 50%). However, the bulk of it is provided by a subset of partners. The PMU maintains a stockpile of money (similar to a vaccine buffer stock) from which it reimburses partners upon request. It then requests USAID/ Washington to replenish the stock. The amount of financing required at any given time, however, is not based on any systematic planning of expected future payments to be made. As a result, the stockpile has at times been higher than the amount planned (US$500,000 minimum). EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 11 PMU monitoring of partner financial expenditures (declared and accrued) is not easy. One critical challenge is delay in implementing TB CAP projects and thus delay in seeking reimbursement. Although partners must develop annual workplans for core projects, the percentage that are fully implemented in the first year is decreasing: 47.6% in APA1 (10 of 21), 45% in APA2 (9 of 20) and 41% in APA3 (12 of 29). A second challenge is the timeliness of partner reporting on actual expenditures. Clearly, there was an initially steep learning curve between partners and the PMU on what and how to report, but partner the reporting has greatly improved. Nonetheless, some partners reportedly take months to inform the PMU of expenses they incur, making it hard for the PMU to accurately forecast the amount of finances it requires in its stockpile. There is some evidence that partners are reporting expenditures (declared and accrued) to the PMU more quickly. For example, 78% of the total budget for APA2 core projects completed in that year was reported in that year, and in APA3 the comparable figure was 82%, thanks to new measures to increase communication and frequent requests for better reporting of expenditures. A third very important challenge is confusion about the definition of accrual. According to the PMU, commitments can only be considered as accrued expenses when the products or services have been delivered. This eliminates a substantial amount relating to work that has been agreed to and which will almost certainly have to be paid. For example, a legal contract for the delivery of future services has equal value, for the purpose of unofficially estimating future financial requirements, as an invoice for services already delivered, yet such figures are not included in reports from partners or in financial forecasting by the PMU. These types of problems do not appear to be the result of unwillingness to participate in the reporting process. Partners consider the TB CAP reporting requirements to be manageable. One common concern, however, is that there is insufficient funding to support the management and reporting requirements. The PMU funds 50% of a project officer within each partner organization, and the percentage does not change even when the number and size of projects increases. After justification was requested from partners for an increase in the funding for project officers, the PMU analyzed the time spent on TB CAP work and concluded that on average ―the project officers are spending 50% of their time on TB CAP tasks. The division is evenly split between supporting workplan development and implementation, and administrative tasks, such as reporting and contracts.‖ It was acknowledged that ―three project officers are spending more than the 50%, two of which well over the limit (about 73.5%),‖ mainly to develop country workplans and/or provide implementation support. Recommendations TB CAP should continue to have its finances independently audited each year and should encourage the auditors to identify recommendations and potential risks and monitor progress on previous recommendations. All partners should be encouraged to have their cost shares audited. The PMU should establish a system for more quickly and accurately forecasting expenditures and future financial requirements. The PMU should more closely monitor project planning to ensure that results planned are realistic and achievable. It should clarify the date when a project should start reporting; currently, it is not clear in annual reports when a project began, and therefore it is not possible to evaluate whether it is on schedule. The PMU should institute policies to enforce quicker reporting of expenditures, such as by not releasing new funds until outstanding expected expenditures have been reported. 12 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) The PMU should adopt a definition of accrual that allows for estimated expenditures to be reported unofficially to allow for better estimating of progress and future financial needs. Financial Reporting USAID reports that day-to-day PMU forecasts of funding requirements in some cases seem unreliable; for example, costly activities were known to have begun but forecasts for the costs are unrealistically low. On the other hand, independent auditors have confirmed that TB CAP is presenting its annual financial statements to USAID fairly, and USAID perceives annual PMU financial reporting as being generally timely. The PMU has been informeding USAID of possible delays in advance, reasonably explained why the delay might occur, and receiving USAID approval to deliver later than expected if necessary. It is less clear if the detailed financial information presented in TB CAP annual reports is reliable assessing progress. Figures often change from year to year without explanation. The number and frequency of changes in TB CAP annual reports suggests that the figures are not definite at the time of reporting and therefore may not be accurate. Furthermore, these are often figures that would be expected not to change (such as a country budget approved two years ago), which raises questions about the soundness of PMU financial management practices. Among many examples, more than half the approved APA2 budgets for 15 countries listed in the year 2 report changed in the year 3 report without explanation, and the APA1 Approved Management Budget changed in each annual progress report (from $1.14 million to $1.05 million to $1.13 million). Since the TB CAP project seems to take longer than anticipated to disburse approved budgets, annual spending is an important indicator for monitoring progress. Actual spending for TB CAP projects in a given year, however, is not possible to determine. For countries, TB CAP progress reports calculate spending by comparing all country budgets (including budgets from previous years) with all country expenditures (including expenditures from previous years). As a result, the spending reported in each annual report reflects not spending against budget for that year but the cumulative average. Not surprisingly, then, the average appears to be improving every year (54% in APA1, 66% in APA2, and 73% in APA3) as more and more budgets and completed expenditures are back-loaded into the formula. Although not ideal, a truer measure of the annual change in spending is to look only at countries in their first year, when they have no previous budgets or expenditures to weight the average. If this is done, spending for new projects has not improved: 54% in APA1 (10 countries), 54% in APA2 (9 countries), and 8.2% in APA3 (2 countries). For core projects in their first year, the trend is in fact regressing: 71.5% in APA1, 66% in APA2 and 58% in APA3. This effect is no doubt heavily influenced by the start dates of projects (many of which start late in a reporting year) and the trend in core projects of programming multiyear efforts within a forced single-year framework. E. MONITORING AND EVALUATION An independent evaluation found that M&E of activities was an area of weakness in the predecessor project, and with the addition of an M&E specialist in TB CAP, this area is perceived as having vastly improved. In particular, the reporting on technical progress in countries and the impact of TB CAP projects appears to be well accounted for. The information (qualitative and, to a lesser extent, quantitative) generated by the PMU is valuable for communicating the value that TB CAP has for countries and the international community. This reporting is considered to be timely and of high quality. M&E of project finances is also much better, although the PMU and partners must continue to improve the speed and reliability of financial reporting and refine financial indicators to ensure they require only a minimum burden to generate. In other respects, however, little has been done to address recommendations related to M&E from that earlier evaluation, particularly monitoring the responsibilities of the partners and evaluating project EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 13 progress against process indicators. According to that earlier evaluation, ―steps to improve TBCTA internal efficiency are quickly needed.‖ Although there are examples of the PMU intervening when partners underperform, there is no system for tracking performance other than financial and no mechanism for addressing underperformance. Some partners decided they were not willing to perform the work they had initially agreed to, and the PMU was able to skilfully substitute another partner. It does not appear that partner work is being systematically evaluated for quality, however, or that there is a strategy for addressing weaknesses. It also does not appear that the PMU has indicators or a system for M&E of its own performance or the process of developing and implementing workplans. The lack of such an M&E system, especially for workplan development, makes it difficult for the PMU to detect and address delays before they become urgent problems or to report on areas where progress is improving. For example, while there is substantial anecdotal evidence that bottlenecks and delays occur in workplan development and detailed analysis of specific projects reveals that these delays are real, because there is no system for capturing this information as it happens, there is no routine analysis and reporting on performance. Reporting on technical performance in countries and among core projects appears to be well accounted for in TB CAP systems and reports, but still remains more quantitative than analytic. 32 Total number of countries that received direct support through the TB CAP mechanism (APA1-APA4). Note this does not include countries that benefited from non-country TB CAP projects, such as core projects that trained staff in regions or developed tools that were tested and used in countries. 28 Total number of countries that were USAID Tier 1 or Tier 2 priority countries (APA1-APA4). Tier 1 = 17; Tier 2 = 11. Exceptions: Botswana, Djibouti, Sudan, and Viet Nam. 25 Total number of countries in which the Missions funded the support (APA1-APA4). 17 Total number of countries that were high-TB￾burden countries (APA1-APA4). Recommendations The PMU should develop, in collaboration with partners, a simple, clear standardized operating procedure for drafting, approving, and implementing a TB CAP workplan (country or core) with indicative times for each action, and gain agreement from all partners to do their best to adhere to thr procedure and agree to be monitored against the schedule. Such a system would be a valuable guide when beginning a new project as well as being necessary for monitoring. Relying on that monitoring instrument, the PMU should create and maintain an up-to-date internal M&E system to track the workplan process against set indicators, both to allow rapid analysis of workplans in process and their status and to determine where schedules are failing. The PMU should coordinate standardized, independent annual evaluations of TB CAP projects to assess, analyze, and report on the quality of TB CAP support provided, its impact, and the ensuing challenges and opportunities. TB CAP reports to USAID should include a summary and indicators of annual PMU and partner performance. USAID should review with the PMU its quarterly/semi-annual reporting and ensure that only essential items are required. F. KNOWLEDGE MANAGEMENT TB CAP has produced many tools in its first three years, some of which have been extremely valuable. 14 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) International Standards for Tuberculosis Care (ISTC) is an example of tools that help countries worldwide. Its function supports the enhancement and expansion of DOTS programs. TB CAP work served as the ideal test bed for validating and refining ISTC. The reputation of the partners in TB CAP facilitated its rapid adaptation, adoption, and introduction by countries. Very quickly, in relative terms, the ISTC has become a global standard and is now part of the WHO M&E indicators collected from all member states. Although ISTC’s success is also due to its unique nature and genesis (it was created by a society of medical practitioners rather than more traditional public health organizations) and a lack of competitive tools, there can be no doubt that it would not have reached the current level of acceptance as quickly (or at all) without TB CAP. The ISTC represents the best that TB CAP has to offer. Unfortunately, the same cannot be said of everything TB CAP partners produce. For a variety of reasons, many of those interviewed claimed that the tools developed by TB CAP (and in the global community at large) far outnumber those actively being used. Nevertheless, TB CAP support has accelerated development of some valuable international tools and their adaptation and adoption. However, guidance for adapting technical guidelines to unique country conditions and stage of program development is at times insufficient. Recommendations TB CAP should leverage the activities conducted to promote the ISTC with application in countries of proven methods of involving private practitioners in management of TB patients according to national guidelines. TB CAP should only produce new tools when it is clear that existing tools cannot be adapted. TB CAP should analyze successful tools and experiences from its projects and establish a process for disseminating them and monitoring how they are used. G. PLANNING AND IMPLEMENTATION The number of USAID Missions buying into TB CAP has increased dramatically, from 10 Missions in APA1 to 22 in APA4, as did the total value of these buy-ins (from US$4.2 million to US$40.7 million; see Figure 1). Figure 1: Total US Dollar Value of USAID Mission Buy-Ins to TB CAP in $Millions USAID gives priority to supporting countries with a heavier burden of TB, among other criteria.7 By APA4 32 countries had received some support from TB CAP (see sidebar). Of these, 17 were TB high-burden countries, and 28 were USAID priority countries. While Missions and countries continue to be well satisfied with the technical quality of the assistance they receive, partners see the requirement to prepare and approve new plans each year as duplicative and inefficient. Furthermore, the one-year 7 USAID’s TB efforts are concentrated in 38 countries to maximize impact and focus financial resources, TA from USAID/Washington and its partners, and staffing; 20 of the countries are designated as Tier One, which gives them higher priority for funding than the 18 countries in Tier Two. EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 15 focus is seen as being inconsistent with the goals of optimizing and linking TB CAP activities to national strategic plans. Responding to Requests for Support TB CAP typically moves into a new country after the Mission, having consulted with USAID/ Washington, makes a request for a TB CAP project. Upon learning of the Mission need, TB CAP selects a partner to coordinate the TB CAP activity—a delicate process affected by such factors as the type and amount of support required and the technical capacity and interests of partners. The selection of a coordinating partner is complicated because the scope and complexity of the work is often great, and there is a need to accommodate the experiences and desires of the USAID Mission and NTP staff. USAID Missions or the NTP may have an established relationship with an existing partner, and request to work with them. If the partner is already involved in other health activities in the country or has an established operation that can be used to support and manage the TB CAP effort, this is often an additional factor to be considered. Recommendations USAID Missions should be better informed about the added value of longer-term investments that strategically strengthen the institutional capacity of national TB control programs. TB CAP should offer, upon request, TA to Missions to increase their understanding of strategic and programmatic issues in TB control and coordination of TB with other health activities, such as HIV and primary health care. Assessing Country Needs In preparation for a country project, PMU staff often visit the country to assess needs, interview national stakeholders, and draft a preliminary plan for support before officially selecting a partner to coordinate the work. Although the PMU advises partners of a forthcoming assessment mission, some partners report they were unaware that TB CAP was in the country. Such assessments also do not appear to follow a standardized format or produce a formal record of the analysis that is shared with all partners. In addition to PMU staff, assessments may sometimes involve other partners, particularly when there is a previous relationship between a partner and the country. Recommendations The Mission, the PMU, and the NTP should jointly undertake a standardized assessment for every potential new country before coordinating and collaborating partners are selected. TB CAP should periodically reevaluate the support it is providing to a country to ensure that what is being done remains effective in light of new needs or changed national policies. Selecting Coordinating and Collaborating Partners Based on the outcomes of the visit to the country and taking into account these many practical considerations, the PMU identifies a suggested coordinating partner. At that time, the PMU circulates this information to all partners and requests volunteers to serve as collaborating partners, asking that each explain the role they would play in the project. The PMU then establishes a suggested composition for the coordinating partner and subpartners and circulates final information to all. 16 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) According to data from the PMU, the three organizations that have coordinated the most core project workplans in the first three years are WHO (28), the PMU (12), and the Union and KNCV (9 each). The two organizations that have coordinated the most annual country workplans in the first three years were KNCV (13) and the Union (8) (see Table 1). Thus the number of country workplans coordinated by each partner was relatively stable in each of the first three years of TB CAP. The proportion of the total TB CAP country workplan budget, however, has significantly decreased each year for FHI (from 36% in APA1 to 10% in APA3) and the Union (from 31% to 11%), has significantly increased for KNCV (from 23% to 44%), and remained relatively stable (+/–10%) for other partners. Several partners stated that although this process has not led to serious oversights or grave problems, it is not always clear why a coordinating partner was chosen, and in a few cases partners felt their skills were overlooked or another partner was chosen to coordinate TB CAP work in a country where they had more to offer. The evaluation team heard of no instance where a partner requested to be a collaborator and was turned down. Some partners felt there were instances when there were too many collaborating partners, and it was unclear what each was bringing to the project. There was some concern expressed about a partner agreeing to collaborate and failing to carry out agreed activities on time. Recommendations The PMU should provide information and a recommendation to the board about assignment of coordinating partners for country and core projects. The decision on selection should rest with the board. The board should establish a clear, transparent procedure for selecting coordinating and collaborating partners. All partners should be clear about the criteria used for selection. The major criteria for selection of a coordinating partner should be its quality and capacity. The board should adopt a conflict of interest statement, and members with a conflict should recuse themselves from selection decisions. Balancing allocation of work among partners while ensuring the highest quality of support would be advantageous to the harmony of TB CAP. A partnership is at its best when all contributors are carrying an appropriate load. The PMU should seek to identify when issues related to allocation of projects arise, determine the cause, and present the situation to the board for resolution. The PMU should only coordinate country projects when all other measures have first been attempted and the success of the TB CAP project is at risk. Creating Country Workplans Once selected, the coordinating partner must work with any collaborating partners to draw up a detailed annual workplan and budget. The approach to planning varies, however. In some cases, collaborating partners are heavily consulted on all aspects of the workplan; in others the partners are asked only for input on their section. Table 1. Distribution of TB CAP Workplans, APA1 to APA3 Core Workplans Country Workplans ATS 7 0 FHI 4 6 JATA 4 2 KNCV 9 13 MSH 5 6 PMU 12 2 UNION 9 8 WHO 28 5 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 17 In general, however, it is expected that planning will be done in cooperation with the NTP and its approval gained before workplan and budget are submitted for approval to first the PMU, then the Mission, and then the USAID CTO. In most countries the TB CAP coordinating partner works closely with the NTP, and the workplan reflects NTP priorities. Some Missions reported they were heavily involved in drafting the workplan, others that they received only a final draft. A few NTPs reported that they saw a proposed workplan only after it was nearly complete. In some countries the Mission or NTP asked TB CAP for short-term, limited TA so the planning was limited to the specific activity requested. Regardless of the variances, however, these aberrations were almost always based on the desires of the participants. None of those interviewed complained that they were excluded from a part of the process in which they wanted to participate or that the change in process directly resulted in poor implementation. Some did refer to inconveniences or inefficiencies resulting from their not being involved earlier and expressed a desire to be included in the future. However, it seems likely that the greatest negative impact of these variances is the lack of clarity on how the process should be performed, and a lack of evidence for consensus on the single most effective approach (regardless of individual wishes) to develop a workplan. Sometimes changes in Mission or NTP priorities require that TB CAP fill in gaps by adjusting the activities in the workplan. TB CAP is seen by NTPs as a source of flexible financing that can plug gaps to keep the program going. Recommendations In countries where TB CAP assistance is requested for multiple years, initial workplans should be approved for the entire term, with annual adjustments to correspond as closely as possible to sequential needs identified in national strategic TB plans and based an the availability of funds. For multiyear workplans, a realistic and longer timeframe should be allowed for planning and approval of the initial plan. Shorter, more streamlined planning and approval processes should apply to annual updates. The PMU should determine an acceptable lead time for workplan development, identify projects that far exceed that time, and analyze the causes of delays. The PMU should then identify strategies to reduce delays, such as the use of tools like workplan templates to facilitate faster start-up or closer oversight and management of the work planning process by PMU staff. Missions should allocate annual TB CAP funds based as closely as possible on annual workplan adjustments and budget requirements. There is a need for defining and providing guidance on logical, sequential, incremental steps NTPs should take, depending on how developed their program is. Approving Country Workplans Some consider approval by the USAID Mission and USAID/Washington for procurement to be a problem that causes delays. This problem is seen as having improving over time, with the USAID CTO moving decisions along. A common concern of partners is that approval of workplans is delaying implementation. Often the time to develop or approve a workplan seems excessive. Based on a random sample of 10 workplans, four took more than four months to develop a draft, and in four instances Mission approval took two or more months. USAID CTO approval seems faster, with only two plans requiring more than 40 days (average 18 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) two weeks). There is a solid consensus that the TB CAP approval process is not simple enough. It often delays the start of work beyond the start of the fiscal year. A significant problem is the many levels of approval involved, and the delays that can occur at each. Lack of a standardized system for recording milestones in workplan development and approval makes it hard to identify where bottlenecks most often occur. However, it does appear from the sampling of workplans that long delays often occur at the national level due to political or other complexities that cannot be prepared for, such as changes in the government, which in one country delayed NTP approval by three months. In other cases, getting the Memorandum of Understanding between TB CAP and the country signed was cited as causing delays (in one case, almost 6 months) in TB CAP start-up. Other long delays in workplan approval seem to be caused by a lack of involvement by key reviewers and approvers in preparation of workplans. For example, in one country, after several weeks of preparation and agreement by the NTP and the TB CAP partner, the Mission, seeing the workplan for the first time, raised issues that required the NTP and partner to revisit it. Delays also result from a lack of information about planning and budgeting requirements, or lack of use of information available, within the Mission. In some cases, workplans approved by the Mission were challenged by the CTO and thus needed to be clarified or adjusted by the TB CAP partner, delaying project approval. Greater Mission understanding of what matters for CTO approval and areas where justification is required (for example, the number of backstopping days allowed or the reasons for choosing an international rather than a national consultant) would decrease these types of problems. If possible, the draft work plan should be seen concurrently by the Mission and the TB CAP CTO. Recommendations The process of planning projects should be standardized and performed consistently, and approval of workplans should be streamlined. The PMU should, wherever possible, use an existing USAID/country MOU when establishing its TB CAP project. If multiyear projects are introduced, TB CAP and USAID should consider how this can simplify the annual approval process. Implementing Country Workplans In some countries, TB CAP activities are out of phase with national priorities for TB control. In some instances the activity or the geographic location requested is based on Mission priorities or input from technical agencies, which may or may not match the needs of the national program. TB CAP has facilitated significant integration of TB into general health services, such as through improvements of diagnostic and drug management services that support other health needs and linking hospitals with primary health facilities. However, the deterioration of primary care services and facilities in a number of countries in recent years makes it more difficult to provide quality diagnostic and treatment services at the peripheral service level. There has been some progress in involving the private sector in TB control activities as a result of TB CAP interventions. The dissemination of the ISTC has played a major part in this success. The coordination of planning between TB CAP, PEPFAR, CDC, and other country partners is seen as improving, but it varies greatly from country to country. EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 19 Figure 2: Total US Dollar Value of PEPFAR Funding Through TB CAP PEPFAR contributions to TB CAP are increasing each year (see Figure 2), however, they are mainly directed to HIV testing of TB patients rather than TB diagnosis and treatment in persons with HIV. The increased occurrence and concern about multi-drug-resistant TB (MDR-TB) has resulted in some countries placing unbalanced emphasis on these problems at the expense of strengthening community and peripheral health service￾based DOT. Without effective DOT at these levels, drug resistance will be increased. Recommendations Joint planning with PEPFAR, CDC, and other TB partners needs to be better in a number of countries. Examples of how it is being effectively carried out should be shared. TB CAP should continue to offer NTPs a flexible means for financing emergency needs essential for the functioning of the national program. Core and Regional Projects TB CAP core funding can be used to fund strategic core projects. All core projects are linked to an IR and its strategic approaches and should ideally benefit several countries. TB CAP core projects are seen by partners and USAID as addressing priority areas in global TB control. The board has not continuously updated the strategic plan or developed an agenda to prioritize core projects. Because the clientele is global, technical policies and guidelines are offered in a ―one size fits all‖ model. Countries and their priorities vary considerably and require TA to adapt these guidelines to their own conditions while adhering to key principles. In some cases national guidelines did not reflect international recommendations. Also, TB CAP tools may have been formally adopted but not incorporated into national policy or plans. Many of the NTPs interviewed were not aware of the TB CAP product or tools. This varies by tool, with the ISTC being highly regarded in many countries. Partners see the planning and approval of core projects as time-consuming and inefficient. The one-year planning and approval process for products that require several years to develop, test, approve, and implement causes problems for them. In some cases, it results in numerous small, fragmented projects, each requiring annual submission, review, and approval. In other cases, partners choose to compress multiyear workplans into unrealistic, shorter workplans, with the expectation that uncompleted activities can be carried over into later years. Recommendations The board and working groups should maintain an up-to-date strategic program that prioritizes the development of core projects. Core projects requiring multiyear funding should be proposed and approved as a single project with annual adjustments. 20 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) Whether national policies, guidelines, and practices are consistent with international principles should be periodically checked. TB CAP needs more capacity and methods for disseminating information about available tools and promote their utilization. H. RESULTS AND IMPACT Global Results and Impact TB CAP has facilitated TA by TBCTA member organizations and promoted coordination among the partners. It has also helped build the capacity of the PMU and partners to plan and monitor TA activities and to provide information responding to USAID requirements. Building human capacity for TB CAP country projects has benefited not only those countries but also built technical capacity to advise other countries. The experience acquired in special areas (such as community DOTS) can influence global policy. TB CAP and its constituent bodies supported the development of guidelines and training materials on the main components of TB control that serve as models for national norms. Examples are the ISTC, the guidelines for TB control in prisons, and the training modules for health facilities and laboratory consultants. WHO is the main source of policy and technical guidance for governments, the Union is a source for NGOs/PVOs as well as for governments, and ATS is a respected source for academic and medical organizations. The tools developed are useful to improve national capacity to implement programs. Its materials are used by many countries in addition to those directly supported by TB CAP, and staff development activities increase global and regional capacity. Locally appropriate strategies were tested in pilot areas and contributed experience for development of tools, which are disseminated and explained through training courses, workshops, and publications. Their effect on national policy is most noticeable in those areas currently under rapid change, such as management of MDR and the laboratory network. Field observation noted the institutional response to the ISTC and that country TB control norms are modelled on WHO technical guidelines. Involvement of several institutions is a relatively recent development in global TB control; TB CAP both benefits from and promotes it. As the single entry point for TA requests by Missions and NTPs TB CAP is stronger that the sum of its parts. TB CAP has supported development of new international human resources to advise national programs by providing technical and managerial training workshops and field experience, and in the more difficult area of advising in conformity with national mores. Adaptation of global recommendations to national and local levels is essential. Staff indicated that technical support is best provided by consultants with technical knowledge plus field experience, preferably in the national language. TA from TB CAP complements financial assistance to countries from the Global Fund and other donors, including PEPFAR; the technical and financial assistance through CDC; and drug provision and management support from the Global Drug Facility (GDF) and the Green Light Committee (GLC). Acccording to Global Fund and NTP staff, TB CAP consultants have successfully contributed to preparation, revision, and monitoring of Global Fund national proposals and grants. Interviews evidenced that TB CAP is well aligned with the DOTS and Stop TB strategies and that technical areas are well addressed and incorporate new developments into global recommendations as required. However, there are numerous examples of successful interventions developed with TB CAP support that are not well known to NTPs and collaborating institutions (PEPFAR, NGOs/PVOs) in other countries. In some instances the TB CAP in-country technical capacity and sources of information are underused by other institutions, such as the Global Fund Secretariat and focal points, PEPFAR, and the USAID country missions. EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 21 TB CAP projects appear to have both direct and indirect benefits for country health systems generally. For example, projects on creating and improving laboratory capacity (both human and infrastructure) to diagnose TB support the work of the laboratories in diagnosing other diseases. There is a similar general benefit from TB CAP work in infection control. Less directly, but equally important, the skills that staff in the national health system need for more effective TB control are also relevant to other health areas. For example, training in TB drug management and quality assurance can be immediately applied to management of other drugs. The same warehouses that are used to store TB drugs are frequently used to store other drugs. The practices warehouse staff learn, such as first-expiring/first-out policies, are equally important for other drugs. Recommendations Successful experiences should be systematically collected, analyzed, and discussed by TB CAP field officers and NTP staff, and used for capacity building, for example, through group visits to selected sites. The use of TA capacity and the availability of TB CAP in countries should be promoted to the Secretariat of the Global Fund, country focal points, and other partners that fund TB control. The staff of USAID Missions could also use TB CAP consultants to update their knowledge of TB control and discuss strategies appropriate to each country. Impact on National TB Control Programs TB CAP provides TA to NTPs following the DOTS strategy. Although the project is not directly responsible for national policy or NTP implementation, the selection of priorities for TA, quality of program delivery, and epidemiological indicators are indirect measures of the relevance of the TA provided to a particular country. Personal and telephone interviews with NTP staff found that governments were satisfied with the technical quality of TA and with the system of having one focal organization that can channel requests and obtain assistance in a variety of areas from a variety of providers. In addition to supporting country human resource development, local staff contracted by TB CAP are a valuable resource for future national programs. Field observation showed significant variation in TB CAP support to countries related to the technical focus, the area of the health system receiving support, and the approaches to cooperation with national entities. In some countries TB CAP (in particular the coordinating partner) is at the center of the NTP; in others it provides support through consultants as requested or manages pilot areas aiming to expand successful experiences. In one country the NTP is not making use of the network of primary health facilities to improve patient access to DOT and has concentrated efforts only on community DOTS; although TB CAP works closely with the NTP, it does not seem to be influencing this national policy. A major TA need is political commitment (IR 1), demonstrated by government financial support to a strong NTP, program guidelines compatible with international recommendations, and firm managerial/logistical capacity at the NTP central level (IR 2). In one country visited, TB drugs at all levels were about to expire. Another country had sufficient drugs but some facilities with rapid increase in patient load could not keep adequate reserves of drugs and other supplies. Lack of NTP funds and capacity to move drugs through customs are common problems. TB CAP and GDF have helped countries to solve supply problems, but strengthening NTPs with logistics staffing seems urgent. TB CAP can do this directly by mobilizing other resources, such as the Global Fund, or by advocating for the national government to assign additional staff to the NTP—the latter is the more sustainable option but takes more time. 22 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) Field observation showed a substantial increase in commitment from state and district authorities in recent years, both in funding and in the form of transport for supervisors (motorcycles, bicycles); in all cases this commitment was promoted by NTP supervisors and coordinators and by TB CAP. However, there are still major shortages of resources, especially human resources, in countries with a high prevalence of HIV, which are losing health facility staff as the general patient workload increases. Also, in some countries regular funding from the national government is not available for small but crucial expenses, such as funds to get the drugs through customs or for distribution. As a result TB drugs may be in port for long periods while there are stock-outs at health facilities and treatment interruption for patients. Drug management is a key element of TB control that no IR specifically addresses. Tools for drug management were used for training, but logistics is still a weak area for NTPs. Observation showed in one country inadequate storage of TB drugs that could be corrected with minor investment of national funds; TB CAP had identified the problem and suggested solutions that were apparently not acted upon. Governments tend to absorb TB CAP staff into routine program activities, as if they were government employees. TB CAP has done extremely well in laboratory systems (IR 2), but there is still need for TA in this area. Support to build up national and intermediate laboratories is often backed by Global Fund monies and driven by the need to diagnose and manage MDR, with TB CAP providing TA. Support for external quality assurance (EQA) of microscopy has led to its extension to the district level in areas where TB CAP has been working, and the governments can see the benefits of proper diagnosis. However, EQA of smear microscopy is not always accompanied by direct supervision to improve the organization of work and maintenance/repair of microscopes. One of the responsibilities of a national TB laboratory is to organize a system to train staff in maintenance and small repairs; and to keep a few reserve microscopes that can be used by peripheral facilities while theirs are undergoing repair. In spite of the training workshops supported by TB CAP, there is still a gap in the pool of international laboratory consultants, particularly for organization of microscopy networks. The same is true for consultants in other technical areas and in program organization. There is not yet enough experienced staff to cope with current and future demand for assistance. TB CAP has supported countries preparing proposals for Global Fund support. NTP staff stated that TB CAP assistance has been very valuable to help overcome problems created by interruption of Global Fund support. TB CAP has done quite well in community involvement and participation. In some areas, most TB activities, including administration, are done by members of the community. TB CAP is filling gaps by working in provinces that are not otherwise covered. Field observation showed that TB CAP has involved local NGOs and PVOs, with variable coverage and success. Results were excellent where existing social structures (community leaders and activists or a local or national organization) were mobilized. They were less good and probably less efffective where such organizations do not exist or have limited coverage; organization of local community volunteers for few patients in dispersed populations does not seem cost-effective unless they are backed by local health facility staff. One limiting factor is the availability of transport for health facility or supervisory staff to reach community volunteers. Often community members and local organizations are willing to collaborate but do not get enough guidance on what to do or how to use their capacity. State and district authorities in countries visited recognized the value of the TB CAP contribution and requested further information and direct training to keep abreast of the development of NGOs. EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 23 Recommendations Future support should center on building NTP capacity so that it can take over control of project planning, identify gaps, decide on TA needs, and make the TB program sustainable. Analysis of the national program to establish priorities for action within the DOTS and Stop TB strategies should be strengthened. Although all components require TA, some basic areas should receive special attention (e.g., guidelines consistent with international recommendations, regular availability of first-line drugs, and good-quality microscopy) before expansion of, or in concert with, case detection or MDR management. A future TB CAP project should address the issue of drug supply and management systems and laboratory systems quality, in an IR. Laboratory training and technical advice should cover direct supervision and organization of a system for microscope maintenance and repairs. Staffing support by TB CAP should be limited-term, with clear terms of reference, to be replaced by government staff where possible. Training workshops should be complemented by increased coaching of participants in joint missions with experienced laboratory or program consultants. Impact on Patients and TB Epidemiology On Patients The main interventions of TB control are to diagnose cases early (mainly the most infectious sputum￾smear-positive), to treat them appropriately until cure, and to prevent drug resistance. Often patients with symptoms suggestive of TB first seek care from a wide array of non-NTP health care providers, such as clinics operated by formal and informal private practitioners and public, private, voluntary, and corporate institutions (e.g., general and specialty public hospitals; NGOs; faith-based organizations; prison, military, and railway health services; and health-insurance organizations). Non-NTP providers offer major opportunities to further TB control, they are often trusted by the community, and by involving them, NTPs can decrease diagnostic delay and concurrent transmission while increasing case detection. Failure to involve all who provide care for TB-symptomatic patients hampers case detection, delays diagnosis, causes improper diagnosis and inappropriate and incomplete treatment, increases drug resistance, and places a large unnecessary financial burden on patients.Increased public and private sector DOTS participation and collaboration (IR3) is therefore crucial to facilitating patient access to quality TB services. TB CAP support has been vital in helping countries initiate and scale up the engagement of all care providers. TA was provided through TB CAP tools, workshops, and meetings to guide countries in the development of strategies to implement PPM. With HIV infection a major cause of disease and death, particularly in sub-Saharan Africa, patient management includes offering HIV testing, and providing cotrimoxazole preventive therapy (CPT) to those infected and antiretroviral therapy (ART) according to patient need. Data are systematically collected, analyzed, and reported to WHO and disseminated through annual TB Reports. TB CAP has helped countries to expand the networks of quality-assured TB microscopy and of health facilities and community supporters that provide DOT, increasing effective access of patients to TB care. The impact on the case detection rate has been inconsistent, and it is too early to measure impact on outcomes, even in the first TB CAP countries—data will be available only after another two to three years. TB CAP has developed tools for drug management and provided training, but the national capacity 24 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) to ensure a regular supply of drugs to patients is still inadequate in many countries. NTP staffing is often insufficient and drug supply depends heavily on external funding. Management of TB/HIV (IR 4) is vital to patient care in areas with high HIV prevalence. It includes the capacity to offer HIV testing (ideally 100% in high HIV areas) and the use of CPT and ART (see below). TB CAP provides TA to national laboratories in preparation for drug resistance surveys (DRS) and MDR management. Identification and treatment of drug-resistant TB is important to patient care, but not as important as the organization of good treatment and a regular supply of first-line drugs to prevent further MDR-TB. MDR treatment can only be successful if there is a good system to ensure patient access to DOT and regular drugs, particularly as the treatment is less tolerated and much longer. On TB Epidemiology Evaluating the epidemiological impact of TB CAP interventions is not easy—in some cases it is not possible. Attributing changes exclusively to TB CAP technical support would be illogical, particularly when there is also external support through the Global Fund, Global Drug Facility (GDF) and other donors including other USAID TB-related funding, and when governments implement the control program with national and local resources. In some cases TB CAP supports part of a country (a limited number of states or districts), allowing for comparison with nonintervention areas. However, the impact is always slow, there is a lag before data become available, and progress often cannot be evidenced until the project is over. In addition, provinces or districts are chosen on special criteria (better or worse off than the rest) and thus usually not comparable. Recommendations Future support should target the national program and the TB problem as a whole, measure impact with the same data as the NTP, and complement this with evaluation of activities. Many TA activities impact the whole country (such as technical guidelines, central laboratory and network, and human resource development, improvement of central level capacity for management and logistics, ACSM). Impact by Intermediate Result IR 1: Increased Political Commitment for DOTS Deterioration of primary health care services in many countries is increasing the difficulty of providing quality TB diagnosis and treatment services at the periphery. This is particularly noticeable in countries with a high prevalence of HIV infection, where some peripheral health facilities were not offering services while others with limited staffing were overwhelmed by the patient caseload. Many countries continue to depend almost exclusively on external funding for TB control (including TB CAP) and to commit inadequate amounts of their own resources. Government commitment to maintain or increase financial support, which is a condition for some donors, such as the Global Fund and GDF, is often not maintained in part because it would duplicate funding. Grants do not require budget lines that would facilitate government funding after the grant term, although budget lines indicate the willingness of a governments to sustain the program. Observation and interviews in the countries visited evidenced increased political commitment to TB control, particularly at the state and district levels, through new national budget lines for TB activities and state and local financial support and transport for supervisors. However, national commitment is still weak in some countries, with the NTP heavily dependent of external support from the Global Fund and other donors, particularly for drugs and infrastructure. Recommendations Future TB CAP country projects should include a strong component of advocacy to governments (national, state/provincial, and district) to increase political commitment and program sustainability. EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 25 Such advocacy should not be limited to ministries of health but should also extend to, for example, ministries of finance and planning. IR 2: Strengthened and Expanded DOTS Programs One of the indicators of political commitment and a key element of TB control is a strong NTP unit with managerial capacity and resources. TB CAP has trained NTP central staff and, as observed in the countries visited, works closely with them, often providing staffing to collaborate in NTP unit activities, which has increased the capacity of national programs. However, in some cases TB CAP staff can be absorbed in routine activities as if they were government employees, which can interfere with their role as advisors. In some countries periodic visits by experienced staff of the coordinating or collaborating partners were requested to strengthen the advisory role of TB CAP. Integration of TB services in primary health facilities is the main strategy to improve patient access to diagnosis and treatment. Expansion to community and nongovernment facilities further improves access. In all countries visited available health facilities were not used efficiently for systematic detection; and in at least one country were not used to offer DOT; treatment was provided by community staff and volunteers, often without technical back-up. In addition to the laboratory component and the quality of chest X-ray diagnosis, diagnostic capacity can be improved by simple methods for detecting infectious TB in health facility outpatients, particularly those who present with other than respiratory symptoms. This activity was not routine in the health facilities the evaluation team visited. MDR is a major threat in some countries, particularly those where the population has had financial capacity to seek care and buy TB drugs in the private sector, and where the NTPs had higher default and failure rates. MDR increases failure and death during TB treatment, particularly in HIV-infected persons. The addition of major external funding is both an opportunity to manage MDR and a major risk of increasing resistance if second-line drugs are not used with strict DOT and in appropriate combinations. MDR represents less that 6% of global TB incidence, however, and is currently concentrated in only a few countries. MDR management can require up to two years of treatment with less effective and more toxic drugs, and the drug regimen is as much as 100 times more expensive than first-line drugs. The current international recommendation is strict DOT six days per week for the whole period, with injections the first six months. The TB program must be prepared to offer patients easy access to regular second-line drugs. Bacterial culture is necessary for diagnosis and control of treatment, and DST at least to first-line drugs is required even with standard regimens. TB CAP has provided assistance to prepare for MDR management by supporting drug resistance surveillance studies, national reference laboratories, human resource development, national guidelines, and proposals to the Global Fund and the GLC. Current TB CAP indicators for MDR (policy, laboratory, and surveys) are rapidly superseded by the development of MDR control policies and expansion of access to reserve drugs in most countries; future projects should also focus on patient management. The importance of infection control has become more widely recognized in recent years, and TB CAP supports training courses for consultants and national staff. The emphasis so far has been on infection control in the laboratory and in high-HIV-burden countries. There is little information available about simple methods to reduce exposure of staff and general outpatients to potential sources of TB infection. Recommendations TB CAP activities should include periodic assessment of the appropriateness of the national strategy by external consultants and national partners. 26 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) Periodic visits by experienced staff of coordinating or collaborating partners should be used to reinforce the advisory role of TB CAP country staff. Detection of suspected cases attending general health facilities should be promoted as a method to increase case detection and improve TB infection control, and used as program indicator. Simple studies can provide the denominator for health facility level, and standard operating procedures for case detection at the health facility level should be developed. The indicators for MDR activities in future projects should be expanded to patient management. Possible indicators are number of MDR patients treated and proportion of cultures negative at six months (equivalent to sputum conversion for first-line drugs). TA should be balanced to include strategies to manage TB/HIV and MDR but give priority to supporting NTP managerial capacity to organize basic interventions to cure TB in HIV-infected patients and prevent further development of drug resistance. Investment of funds and staff time in MDR should be balanced with investments in basic control: treating MDR patients is of little benefit to the community if more drug-resistant cases are created by inadequate treatment of new patients. IR 3: Increased public and private DOTS participation and collaboration The TB CAP-supported PPM-subgroup meeting organized in Kenya was a key launch pad for PPM globally.The meeting brought together stakeholders, outlined the strategic way forward for PPM implementation, and facilitated detailed planning of country-level PPM activities. TB CAP also helped develop the National Situation Assessment Tool, which has been instrumental in helping countries decide if and how they could pursue PPM, and TB CAP–trained consultants helped countries initiate PPM. For instance, in TB CAP’s second year consultants used the NSA tool for situation assessments in 10 countries in the African and Eastern Mediterranean regions. There is very little documentation on the linkages, lessons, and best practices of engaging a diversity of providers, such as informal providers, social security organizations, businesses, and hospitals, but TB CAP is helping build a solid evidence base through site visits. Its documentation has been useful in formulating practical guidance to help countries employ different strategies to engage each type of provider. TB CAP support facilitated the development, packaging, piloting, training for and dissemination of the International Standards for Tuberculosis Care as well as the engagement of health professional associations in PPM. Field observation discerned very good examples of private hospital involvement in TB control and of coordination between academic hospitals and the public health network. The ISTC has also been useful for mobilizing academic clinical leadership, large hospitals, and medical associations in TB control. Private practitioners are not much involved. Their importance varies with health-seeking behaviour and access to public health services, so the priority will vary from country to country. There were clear widespread functional gaps between peripheral health facility and community-based DOT services. Country systems of drug procurement and especially distribution and the organization and service quality of laboratory services are two areas often seen to be deficient. Recommendations Successful experiences in the public-public and public-private mix supported by TB CAP should be replicated and used to train national staff and consultants. EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 27 TB CAP should test strategies to involve private practitioners in TB diagnosis according to national guidelines and in referral for treatment, with rapid expansion of the more effective strategies in TB CAP-supported countries. Guidelines developed with TB CAP support should be disseminated to involve different types of providers. IR 4: Increased and strengthened TB and HIV/AIDS coordinated activities The expansion of HIV in TB patients in sub-Saharan Africa continues, although HIV prevalence is stabilizing in some countries. In one 250-bed hospital new TB cases went from 633 in 1996 to 1,313 in 2007, and an increase of the proportion of patients infected with HIV among those cases went from 37% to 76%. The increase in TB incidence overloads health facilities, reduces health staff through death and departure, and increases patient deaths during treatment. The rapid increase in caseload stresses the logistic capacity of NTPs to provide drugs and other supplies. Figure 3. Coverage of HIV Testing of TB Patients in a Sub-Saharan Country (2006–08) Coordination between the national TB and HIV programs (IR 4) is difficult to measure; the outcome of better patient management is a more objective indicator. The proportion of TB cases covered by HIV testing and CPT has increased significantly in recent years. Field observation showed very rapid progress in these indicators, with most of the TB cases being tested, receiving CPT, and progressively having access to ART. However, there are still no relevant data for several countries; and information on TB screening in HIV patients is almost nil. TB programs seem to be much more active in HIV screening than HIV programs are in TB screening and management (see Figures 3 and 4), even though HIV patients are less than 15% of the global TB burden and the risk of TB for those who are HIV-infected is greater than 40%. Better coordination of PEPFAR and USAID assistance for TB/HIV at Mission level, particularly in sub-Saharan Africa, might stimulate better coordination of the national programs. Recommendations TB CAP should improve coordination with PEPFAR, and advocate for PEPFAR support to prevention and management of opportunistic diseases in the HIV-infected, especially TB. TB CAP should identify and share examples of where TB/HIV have been effectively integrated within health services. 28 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) Figure 4. TB patients Tested for HIV in APA1 Countries (Percent) IR 5: Improved human and institutional capacity TA to build national capacity (IR 5) was identified as an enduring priority during field visits and interviews. It is a major activity of TB CAP, whose achievements include the support of regional centers in Indonesia and Nigeria that can provide high-quality training and train trainers. TB CAP has helped draft and disseminate training materials, facilitate their use, and increase the pool of international consultants in all fields of TB control. NTP managers noted that local staff on contract to TB CAP to provide support to the NTP and to NGOs can later be used by the national program; and they will have valuable experience in TB control. The supply of experienced international TB consultants and qualified NTP staff is still inadequate to meet current and projected demand. There is a need for permanent programs to train new staff and provide refresher training, even though training alone does not solve the problem of inadequate or insufficient staffing—a problem for primary health care generally, particularly in high HIV incidence countries. Recommendations Successful experiences should be used to train international consultants and selected NTP staff through intercountry visits. Description, analysis, and dissemination of those experiences should be increased and methods developed to facilitate access to the information. Tool development and training should be complemented by TA to NTPs on systems for continuing education (starting with identifying training needs and staff) and by advocacy to ensure sufficient staffing for TB program management and patient care. The supply of international consultants and qualified NTP staff should be expanded by exposing them to successful experiences within and outside their countries. TB CAP should develop a strategy to replicate regional training centers in additional locations. Cross-cutting Results and Impact The indicators currently used in TB CAP are not always the most appropriate to evaluate its impact given the short term of the project interventions. A list of proposed TB CAP indicators is included as Annex VII. A few of them are discussed here. Proportion of TB patients tested for HIV 0 10 20 30 40 50 60 70 80 2006 2007 2008(Q 1-3) Year Proportion Target achieved EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 29 Coverage: The proportion of health facilities providing DOTS is a useful indicator; however, it should specify public and private facilities separately. Diagnosis: The case detection rate (CDR) is a useful indicator at the national and global levels, and countries commit to reaching a specific target (>70% CDR). However, WHO estimates of TB incidence by country vary from year to year for the same country, limiting the value of CDR as a monitoring tool. The indicator denominator is approximate, and the indicator is incorrectly used for provinces or districts. The proportion of cases confirmed by smear microscopy is a useful measure of the quality of and access to medical diagnosis for other forms of TB and for children. The proportion of cases in children is an indicator of diagnostic capacity and coverage of registration. A more useful indicator of case detection at all levels is the absolute number of suspected cases examined by smear microscopy and sputum smear positivity. The number of patients examined can be compared with the adult general outpatient load in health facilities and with the state or national population. Data for this indicator are available from laboratory registers but are not normally used for monitoring: few facilities visited had analyzed the data and shown the information in graphic form in the laboratory and TB treatment room walls. Treatment: The outcomes of treatment and the advanced predictor smear conversion at two to three months indicate the quality of treatment. Good programs have over 80% success rates (the global target is 85%) and less than 5% default/transfer rates. Deaths during treatment indicate late diagnosis or high HIV prevalence; high failure rates under strict DOT may indicate poor-quality drugs, an irregular supply of drugs, or high MDR prevalence. TB/HIV: TB and HIV coordination is difficult to evaluate without subjective interpretation; there was progress from 1.3 to 2.4 in year 3, out of a maximum of 4 points. Patient-oriented indicators seem more useful: the proportion of TB patients with HIV testing and with CPT has increased from 1.7 to 2.7 out of a maximum of 4 points. Case fatality of HIV-positive TB patients (better called deaths during TB treatment) seems a useful indicator of better case management and use of CPT and ARV. HIV prevalence in TB cases is important but not vulnerable to short-term TB interventions. Countries have been rapidly implementing the recording and reporting system for TB/HIV. Laboratory: Population per microscopy unit is useful as a guide for planning but not as an indicator. The optimum is one per 50,000–100,000 inhabitants for light microscopy; it assumes only one reader per microscope, so common targets cannot be established for different countries. The coverage of microscopy units with EQA is a good indicator (and has increased), but the quality of microscopy accepted (5% error) is too high (low predictive value). Even with this high limit the baseline was only 4% of satisfactory laboratories in 2005; the proportion increased to 64% in 2007 and 94% in 2008, which is major progress. Guidelines: The production, dissemination, and acceptance of the ISTC and its recent updating have had major impact, particularly in academia and medical associations. It probably has also influenced large hospitals, facilitating hospital coordination with primary health care (PHC) facilities and PPM. A large proportion of persons with symptoms consult private practitioners; involving them in TB diagnosis and treatment (including registration and follow up) requires additional strategies and different indicators. The global PPM guidelines have been rolled out in TB CAP countries through workshops and with the help of TB CAP-trained PPM consultants.TB CAP is also facilitating the development of guidelines for different types of providers, such as hospitals, social security organizations, businesses, and informal providers. With TB CAP support, a document is also being developed that advises countries implementing PPM initiatives on how they can continuously measure the contribution of different types of health care providers to TB control. There is sufficient experience in large-burden countries and there are global guidelines on the involvement of private practitioners to test strategies in pilot areas for rapid expansion in TB CAP￾supported countries. 30 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) The Compendium of Indicators is often recommended for measuring program interventions and impact, but the indicators are not prioritized and there is insufficient guidance on when they are valuable. There are no criteria and methodology for evaluating the quality of a TB control program as a whole. Although WHO and the Union have published methods for organizing program assessments, the assessments rest on individual experience and there are no criteria for which areas are critical and what levels of performance are acceptable. Recommendations The indicators for NTPs should be targeted to specific interventions and critical processes and objective (not dependent on estimates). The Compendium of Indicators needs to be updated with priorities and guidance on the pros and cons of each indicator, and when to use them. TB CAP should study the feasibility and usefulness of developing standard criteria to evaluate the quality of TB control programs. I. OPERATIONS RESEARCH Contrary to impressions held by a number of partners and country programs, USAID does allow TA for operations research (OR) at the country level through TB CAP. This is a weak area in almost every country. Most studies can be done with local resources based on available or easily collected data, but staff need some training and guidance. In most cases OR to improve service delivery and program effectiveness is very similar to monitoring, and OR would improve self-evaluation at the peripheral level. However, staff capacity to plan OR and to draw conclusions for action is very limited. TB CAP support should incorporate preparation of simple protocols, recommendation on areas for field OR, training of staff in pilot projects, and dissemination of the procedures and results. Examples of simple protocols would be the proportion of adult outpatients with persistent cough to compare with routine detection, patient and provider diagnostic delay, reasons for default, and the proportion of patients really taking DOT. USAID support to the TREAT TB project through the Union represents an opportunity to build national capacity, but has some risks. The resources of TREAT TB can support useful studies and also NTP capacity building for use of the results not only for research but also for analysis of routine data and monitoring. However, competition between research and program activities for the limited human resources at the national level is a risk. Recommendations TB CAP should develop and test simple protocols for operations research at the service level, and train staff in using them and analyzing the results for action. Where PHC facilities are not available, develop and test alternative ways to increase access of the population to quality TB diagnosis and treatment. Coordinate with other projects (such as the USAID TREAT TB project) to ensure that in-country activities are complementary. EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 31 IV. COMPILED RECOMMENDATIONS OVERARCHING RECOMMENDATIONS USAID should continue its support for TB TA but should take a more long-range and ambitious approach to it. If USAID chooses to implement a follow-on project to TB CAP, there should be only a minimal gap between the end of the current and the beginning of the new one. In fact, USAID should plan for some overlap to account for the initial period of planning new project activities that would be a necessary precursor to implementation and thus ensure smooth continuity from one project to the next. The new project should be launched during the original TB CAP timeframe; the period of time covered by an extension is an exceptional opportunity to complete unfinished work. In the current project, TB CAP should not expand the number of countries supported beyond those already identified in APA5 unless the project period is extended with sufficient additional financial resources. Within resources, the focus should be on qualitatively improving current assistance. The USAID implementing mechanism should have a board that is composed and enabled to actively focus on issues of governance, program strategy, and budgeting, and given a mandate from USAID to develop and implement a strategic agenda for scale-up of support to countries. The PMU should be empowered to make and execute management decisions and to make recommendations for board￾level decisions. USAID should encourage and provide guidance to its Missions to draft multiyear workplans for TB control, linked to national TB control strategies, and formally support the approval of such workplans while providing for annual adjustments of activities and resourcing based on needs, progress, and funds. Longer-term planning with the anticipation of funds would make country TB control measures more sound and help stimulate investment in the technical capacity needed to meet current and projected country demand. STRATEGY AND GOVERNANCE The roles and responsibilities of the board, working groups, and the PMU should be clearly communicated, and the operational practices of each revisited to ensure they are acting on these mandates. Working groups should institute regular meetings, electronically or telephonically when possible, so that they can keep strategies for core and country projects up to date. TB CAP assistance in countries with weak peripheral DOT services should prioritize reinforcing DOT, consistent with the Stop TB Strategy, while also addressing, in a balanced way, the emerging problems of drug resistance, infection control, and TB/HIV coinfection. Although there are many priority needs, TB CAP should focus on ensuring that the fundamental building blocks of an effective TB control program are in place, especially drug management systems and the capacity and coordination of laboratory networks. TB CAP should shift focus from short-term, high-yield projects toward longer-term, country-wide projects that scale up the experiences it has gained and convert them into national improvements and thus greater global impact. 32 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) TB CAP should intensify its efforts to build the institutional and managerial capacity of national systems of TB control. In its country workplans and strategies, TB CAP should give priority to supporting the national strategy and guidelines of each country, consistent with the state of development and TB control needs. DOT systems at health facility and community volunteer levels should be strengthened and integrated. More capability in advocacy, information management and dissemination, quality assurance, and best indicators of TB control should be acquired by TB CAP, represented in governing bodies, and built into future program strategy. TB CAP should offer Missions TA to increase their understanding of strategic and programmatic issues in TB control and coordination of TB with other health activities, such as HIV and primary health care. TB CAP should provide TA on operations research to improve TB control at the service delivery level. Furthermore, USAID should ensure that coordination between TREAT TB and TB CAP, starting from Washington and cascading down into countries, is carefully managed and potential areas of overlap are identified in advance rather than addressed when issues arise. The PMU should recommend country as well as core projects for board decisions on coordinating partners. This will be especially important during any transitional period between TB CAP and a follow-on project. Partners being considered should recuse themselves from these board decisions. PARTNERS AND PARTNER COLLABORATION The board should define clear policies for selection of collaborating partners, such as the criteria for choosing partners and constraints on the number of partners involved in particular types of projects, to ensure optimal composition of project teams. The board should set a minimal acceptable amount of visiting supportive supervision for TB CAP field officers, and partners should maintain that level of support for all programs they coordinate, increasing the amount as the needs and capacity of field officers demand. The board should establish base criteria for measuring the performance of coordinating and collaborating partners and set a policy for responding to both underperformance and unwillingness to carry out agreed-upon responsibilities, and the PMU should design systems for monitoring and regularly reporting on such issues to the board for action. Some partners are less effective than others at coordinating country or core projects. These partners should remain collaborating partners and not be responsible for coordination. OPERATIONAL STRUCTURE The PMU should exercise complete responsibility for making and executing management decisions, seeking USAID or board-level advice or approval only when necessary. The PMU should recognize that all governance-related decisions, including the selection of projects, allocation of projects to partners and development and agreement on TB CAP strategy is the responsibility of the board. The PMU should clearly define the role of its technical focal points and communicate this to all parties. Areas where they could be useful would be to collect and document strategies in technical EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 33 areas as defined by governing bodies and to ensure their consistent reflection in actual TB CAP workplans. The PMU should be authorized to define and provide guidance on country project support, in consultation with NTPs and Partners. OPERATIONAL MANAGEMENT The PMU should clarify its role in the development of workplans and consistently adhere to the definition. If it is to be responsible for pre-planning workplans before a coordinating partner is selected, this should always be the case, and potential coordinating partners should not be involved. If it is to be responsible for ensuring the technical quality and strategic relevance of workplans, this too should be done consistently, and the criteria should be clearly communicated to partners in advance. If it is to delegate all responsibility for workplan preparation, it should not participate at all and instead should consistently seek partners to conduct this work. The PMU should establish a system for registering potential new core and country projects in a Web￾based system accessible to all partners and set out a standardized process for workplan development, approval, and implementation. It should also take responsibility for updating this system with the dates when events occur, so that it will be possible to quickly and easily identify projects at different stages of development and flag those that may need intervention. Such a process would also facilitate general reporting and, if financial details were included, make financial tracking more accurate and timely. There are several Web-based procurement systems upon which to model such a system. FINANCIAL MANAGEMENT TB CAP should continue to have its finances independently audited each year and should encourage the auditors to identify recommendations and potential risks and monitor progress on previous recommendations. All partners should be encouraged to have their cost shares audited. The PMU should establish a system for more quickly and accurately forecasting expenditures and future financial requirements. The PMU should more closely monitor project planning to ensure that results planned are realistic and achievable. It should clarify the date when a project should start reporting; currently, it is not clear in annual reports when a project began, and therefore it is not possible to evaluate whether it is on schedule. The PMU should institute policies to enforce quicker reporting of expenditures, such as by not releasing new funds until outstanding expected expenditures have been reported. The PMU should adopt a definition of accrual that allows for expected expenditures to be reported unofficially to allow for better estimating of progress and future financial needs. MONITORING AND EVALUATION The PMU should develop, in collaboration with partners, a simple, clear standardized operating procedure for planning, approving, and implementing a TB CAP workplan (country or core) with indicative times for each action, and gain agreement from all partners to do their best to adhere to the procedure and agree to be monitored against the schedule. Such a system would be a valuable guide when beginning a new project as well as being necessary for monitoring. 34 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) Relying on that monitoring instrument, the PMU should create and maintain an up-to-date internal M&E system to track the workplan process against set indicators, both to allow rapid analysis of workplans in process and their status and to determine where schedules are failing. The PMU should coordinate standardized, independent annual evaluations of TB CAP projects to assess, analyze, and report on the quality of TB CAP support provided, its impact, and the ensuing challenges and opportunities. TB CAP reports to USAID should include a summary and indicators of annual PMU and partner performance. USAID should review with the PMU its quarterly/semi-annual reporting and ensure that only essential items are required. Knowledge Management TB CAP should leverage the activities conducted to promote the ISTC with application in countries of proven methods of involving private practitioners in management of TB patients according to national guidelines. TB CAP should only produce new tools when it is clear that existing tools cannot be adapted. TB CAP should analyze successful tools and experiences from its projects and establish a process for disseminating them and monitoring how they are used. RESPONDING TO REQUESTS FOR SUPPORT USAID Missions should be better informed about the added value of longer-term investments that strategically strengthen the institutional capacity of national TB control programs. TB CAP should offer, upon request, TA to Missions to increase their understanding of strategic and programmatic issues in TB control and coordination of TB with other health activities, such as HIV and primary health care. ASSESSING COUNTRY NEEDS The Mission, the PMU, and the NTP should jointly undertake a standardized pre-assessment for every new country before coordinating and collaborating partners are selected. TB CAP should periodically re-evaluate the support it is providing to a country to ensure that what is being done remains effective in light of new needs or changed national policies. SELECTING COORDINATING AND COLLABORATION PARTNERS The PMU should provide information and a recommendation to the board about assignment of coordinating partners for country and core projects. The decision on selection should rest with the board. The board should establish a clear, transparent procedure for selecting coordinating and collaborating partners. All partners should be clear about the criteria used for selection. The major criteria for selection of a coordinating partner should be its quality and capacity. The board should adopt a conflict of interest statement, and members with a conflict should recuse themselves from selection decisions. Balancing allocation of work among partners while ensuring the highest quality of support would be advantageous to the harmony of TB CAP. A partnership is at its best when all contributors are EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 35 carrying an appropriate load. The PMU should seek to identify when issues related to allocation of projects arise, determine the cause, and present the situation to the board for resolution. The PMU should only coordinate country projects when all other measures have first been attempted and the success of the TB CAP project is at risk. CREATING COUNTRY WORKPLANS In countries where TB CAP assistance is requested for multiple years, initial workplans should be approved for the entire term, with annual adjustments to correspond as closely as possible to sequential needs identified in national strategic TB plans and based an the availability of funds. For multiyear workplans, a realistic and longer timeframe should be allowed for planning and approval of the initial plan. Shorter, more streamlined planning and approval processes should apply to annual updates. The PMU should determine an acceptable lead time for workplan development, identify projects that far exceed that time, and analyze the causes of delays. The PMU should then identify strategies to reduce delays, such as the use of tools like workplan templates to facilitate faster start-up or closer oversight and management of the work planning process by PMU staff. Missions should allocate annual TB CAP funds based as closely as possible on annual workplan adjustments and budget requirements. There is a need for defining and providing guidance on logical, sequential steps NTPs should take, depending on how developed their program is. APPROVING COUNTRY WORKPLANS The process of planning projects should be further standardized and performed consistently, and approval of workplans should be streamlined. The PMU should, wherever possible, use an existing USAID/country MOU when establishing a TB CAP project. If multiyear projects are introduced, TB CAP and USAID should consider how this can simplify the annual approval process. IMPLEMENTING COUNTRY WORKPLANS Joint planning with PEPFAR, CDC, and other TB Partners needs to be better in a number of countries. Examples of how it is effectively being carried out should be shared. TB CAP should continue to provide NTPs a flexible means for financing emergency needs essential for the functioning of the national program. CORE AND REGIONAL PROJECTS The board and working groups should maintain an up-to-date strategic program that prioritizes the development of core projects. Core projects requiring multiyear funding should be proposed and approved as a single project with annual adjustments. Whether national policies, guidelines, and practices are consistent with international principles should be periodically checked. 36 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) TB CAP needs more capacity and methods for disseminating information about available tools and promoting their use. GLOBAL RESULTS AND IMPACT Successful experiences should be systematically collected, analyzed, and discussed by TB CAP field officers and NTP staff, and used for capacity building, for example, through group visits to selected sites. The use of TA capacity and the availability of TB CAP in countries should be further promoted to the Secretariat of the Global Fund, country focal points, and other partners that fund TB control. The staff of USAID country missions could also use TB CAP consultants to update their knowledge of TB control and discuss strategies appropriate to each country. IMPACT ON NATIONAL TB CONTROL PROGRAMS Future support should center on building NTP capacity so that it can take over control of project planning, identify gaps, decide on TA needs, and make the TB program sustainable. Analysis of the national program to establish priorities for action within the DOTS and Stop TB strategies should be strengthened. Although all components require TA, some basic areas should receive special attention (e.g., guidelines consistent with international recommendations, regular availability of first-line drugs, and good-quality microscopy) before expansion of, or in concert with, case detection or MDR management. A future TB CAP project should address the issues of drug supply and management systems and laboratory systems quality in an IR. Laboratory training and technical advice should cover direct supervision and organization of a system for microscope maintenance and repairs. Staffing support by TB CAP should be limited-term, with clear terms of reference, to be replaced by government staff where possible. Training workshops should be complemented by increased coaching of participants in joint missions with experienced laboratory or program consultants. IMPACT ON PATIENTS AND TB EPIDEMIOLOGY Future support should target the national program and the TB problem as a whole, measure impact with the same data as the NTP, and complement this with evaluation of activities. Many TA activities impact the whole country (such as technical guidelines, central laboratory and network, and human resource development). RESULTS AND IMPACT BY INTERMEDIATE RESULT (IR1) Future TB CAP country projects should include a strong component of advocacy to governments (national, state/provincial, and district) to increase political commitment and program sustainability. Such advocacy should not be limited to ministries of health but should also extend to, for example, ministries of finance and planning. IMPACT BY INTERMEDIATE RESULT (IR2) TB CAP activities should include periodic assessment of the appropriateness of the national strategy by external consultants and the national partners. EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 37 Periodic visits by experienced staff of the coordinating or collaborating partners should be used to strengthen the advisory role of TB CAP country staff. Periodic assessment of the appropriateness of the national strategy by external consultants and national partners should be included in TB CAP activities. Detection of suspected cases attending general health facilities should be promoted as a method to increase case detection and improve TB infection control, and used as a program indicator. Simple studies can provide the denominator at the health facility level. The indicators for MDR activities in future projects should be expanded to patient management. Possible indicators are number of MDR patients treated and proportion of culture negative at six months (equivalent to sputum conversion for first-line drugs). TA should be balanced to include strategies to manage TB/HIV and MDR but give priority to support the NTP managerial capacity to organize basic interventions to cure TB in HIV-infected patients and prevent further development of drug resistance. Investment of funds and staff time in MDR should be balanced with investments in the basic control strategy: treating MDR patients is of little benefit to the community if more drug resistant cases are created by inadequate treatment of new patients. IMPACT BY INTERMEDIATE RESULT (IR3) Successful experiences in the public-public and public-private mix supported by TB CAP should be replicated and used to train national staff and consultants. TB CAP should test strategies to involve private practitioners in TB diagnosis according to national guidelines and in referral for treatment, with rapid expansion of the more effective strategies in TB CAP-supported countries. Guidelines developed with TB CAP support should be disseminated to involve different types of providers. IMPACT BY INTERMEDIATE RESULT (IR4) TB CAP should improve coordination with PEPFAR, and advocate for PEPFAR focus on support to countries to prevent and manage opportunistic diseases in the HIV-infected, especially TB. TB CAP should identify and share examples of practices where TB/HIV has been effectively integrated within health services. The TB CAP Board should develop a strategy to mobilize PEPFAR resources and efforts in prevention, diagnosis and treatment of opportunistic diseases in HIV/AIDS patients, of which the most important is tuberculosis. RESULTS AND IMPACT BY INTERMEDIATE RESULT (IR5) Successful experiences should be used to train international consultants and selected NTP staff through intercountry visits. Description, analysis, and dissemination of those experiences should be increased and methods developed to facilitate access to the information. Tool development and training should be complemented by TA to NTPs to develop systems for continuing education (including identification of training needs and staff) and by advocacy to ensure sufficient staffing for TB program management and patient care. The supply of international consultants and qualified NTP staff should be expanded by exposing them to successful in-country experiences within and outside their countries. 38 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) TB CAP should develop a strategy to replicate the model of existing regional training centers in additional locations. CROSS-CUTTING RESULTS AND IMPACT The indicators for NTPs should be more targeted to specific interventions and critical processes and not dependent on estimates. The Compendium of Indicators needs to be updated with priorities and guidance on the merits and problems of each indicator, and when to use them. TB CAP should study the feasibility and usefulness of developing standard criteria to evaluate the quality of TB control programs. OPERATIONS RESEARCH TB CAP should test simple protocols for operations research at the service level, and train staff in using them and analyzing the results for action. Where PHC facilities are not available, develop and test alternative ways to increase access of the population to quality TB diagnosis and treatment. Coordinate with other projects (such as the USAID TREAT TB project) to ensure that in-country activities are complementary. EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 39 ANNEX I. SCOPE OF WORK I. IDENTIFICATION OF THE TECHNICAL ASSISTANCE General Description: The U.S. Agency for International Development, Global Health, Office of Health, Infectious Diseases and Nutrition (USAID/GH/HIDN) seeks the services of GH Tech to review the performance, impact, and lessons learned of the Tuberculosis Control Assistance Program (TB CAP), a USAID-sponsored project formed to control tuberculosis (TB) globally. II. BACKGROUND TB is an ancient disease that continues to plague millions of people each year despite a curable treatment for the disease. In 2006, the World Health Organization (WHO) estimates there were 9.2 million new cases and 1.7 million deaths from TB. After a decline in TB through the late 1980s, the global TB burden has grown alongside the HIV epidemic, and HIV-infected patients are at greatly increased risk for developing TB. In 2006, an estimated 709,000 of the new TB cases were in HIV-infected patients, and in some African countries more than half of new TB cases were in HIV-infected patients. TB imposes a heavy economic burden in both the developed and developing world. Families coping with TB lose 20% to 30% of their annual income to the disease. Overall, TB costs the global economy around $12 billion annually. There have been some hopeful signs of improving TB control in recent years. Although the number of people developing TB continues to rise each year along with the global population, the estimated rate of infection per capita has been fairly stable since 2003. Serious challenges remain, however, including the emergence of multi-drug-resistant TB (MDR-TB) and extensively drug-resistant TB (XDR-TB). These strains have grown resistant to the most powerful anti-TB drugs due mainly to the improper use of antibiotics, poor adherence to TB treatment, and poor quality drugs. Strategy for Tuberculosis Control Global TB control is guided by the Stop TB Partnership’s Second Global Plan and the World Health Organization’s (WHO) Stop TB Strategy. In line with the Millennium goals, the Second Global Plan aims to halve TB prevalence and deaths by 2015 relative to 1990 levels. The Stop TB Strategy identifies the six main components required to achieve these targets, including the provision of high-quality DOTS expansion and enhancement; address TB/HIV, MDR-TB, and other challenges; contribute to health system strengthening; engage all care providers; empower people with TB and communities; and enable and promote research. DOTS is at the heart of the Stop TB Strategy. For countries to successfully implement DOTS they must demonstrate political commitment to fund and implement an effective national TB control program. DOTS also requires an integrated network of capable laboratories, surveillance through a standardized recording and reporting system, and a robust logistics system that ensures a secure supply of drugs. In addition, implementation of DOTS requires improved access to primary care services that are affordable, equitable, committed, and well-organized. Education and training are essential elements to ensure the availability of human resources. The WHO established the Green Light Committee to help control and prevent MDR-TB through access to quality-assured second-line anti-TB drugs and prevention of the development of resistance to anti-TB drugs by assuring the appropriate use of these drugs. The Tuberculosis Control Assistance Program TB CAP is a five-year $150 million ceiling cooperative agreement competed by USAID. USAID/GH/HIDN manages TB CAP in close coordination and collaboration with the USAID Regional Bureaus and USAID Missions providing funding. The USAID-designated Cognizant Technical Officer 40 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) (CTO) resides in GH/HIDN in Washington, DC. Along with the Contracting Agreement Officer, the CTO is ultimately responsible for ensuring that the project strategic approach, timeliness of products, financial management and cost control, quality of products, and reported results are according to the overall cooperative agreement and approved work plans. TB CAP is executed by a consortium of seven organizations, which collectively refer to themselves as the Tuberculosis Coalition for Technical Assistance (TBCTA).8 The prime organization is the KNCV Tuberculosis Foundation (KNCV) and the six sub-grantees are the American Thoracic Society (ATS), Family Health International (FHI), International Union Against Tuberculosis and Lung Disease (the Union), Japan Anti-Tuberculosis Association (JATA), Management Sciences for Health (MSH), and the World Health Organization (WHO). The Centers for Disease Control (CDC) is a coordinating member of the Coalition but funds for this agreement do not flow to or through CDC. The organizations that form the partnership are described below: 1. KNCV, based in the Netherlands, is a medical development organization that focuses on combating TB globally. Its mission is to promote effective and efficient TB control in both a national and an international context. It has been working in TB control for over 100 years and in the 1980s began contributing to the development and implementation of effective TB control programs in low-income countries. It also hosts the International Tuberculosis Surveillance Centre (ITSC) and the Tuberculosis Surveillance Research Unit (TSRU); contributes to international policy development especially within its collaboration with WHO and the the Union; and collaborates with numerous international organizations. It is the prime organization for this cooperative agreement and responsible for administrative management. 2. ATS, with headquarters in New York City, is a non-profit educational and scientific society and is the major medical professional and scientific organization with an interest in TB. The ATS has considerable experience in collaborating with the CDC and Infectious Disease Society of America in developing guidelines for TB prevention and control used both in the United States and in other parts of the world. The ATS publishes a highly respected and widely read scientific journal. In addition, the Society conducts an annual international conference that attracts 15,000-16,000 attendees and is a major forum for the presentation of new information about TB and its control as well as providing an arena for TB training and education. A broad range of specific technical expertise is represented in the organization. 3. FHI is a nongovernmental, nonprofit organization dedicated to improving lives and increasing knowledge worldwide through research, education, and services. FHI has implemented programs in 66 countries and has worked on HIV/AIDS in the developing world since 1986. FHI advocates for greater coordination of HIV and TB programs and provides technical expertise in increasing TB patient access to HIV prevention and voluntary counseling and testing (VCT), integrating TB preventive therapy into services for HIV positive patients and VCT, and intensified case finding of TB cases among partners of HIV-infected TB patients. 4. The Union, an NGO headquartered in Paris, is comprised of constituent, organizational, and individual members. There are Union regional organizations in North America, Latin America, Europe, Africa, Asia, and the Middle East. The Union disseminates information on TB, lung disease, and tobacco, coordinates and assists the work of its members throughout the world, and maintains close links with WHO, other UN agencies, and government and nongovernment organizations in the 8 TBCTA was created in 2000 when a group of international TB partners came together to develop an unsolicited proposal to USAID. The original partners were the American Lung Association (ALA), ATS, The Union, KNCV, WHO and CDC. More information can be found at www.tbcta.org. EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 41 health and development sector. The Union’s expertise and its activities are focused on the areas of technical assistance, education, and research. 5. JATA, established in 1939, has expanded its mission from TB control in Japan to fighting TB around the world. JATA houses the Research Institute of Tuberculosis (RIT), which is a WHO collaborating center. JATA’s range of activities include public awareness raising, community mobilization, fundraising, clinical services, surveillance and research, reference for mycobacterial disease, human resource development, and international collaboration. 6. MSH is a nongovernmental, nonprofit organization that specializes in managing large multidisciplinary technical assistance programs in developing countries. MSH has experience supporting national TB programs in planning, cost analysis and financing, human resources, integrating packages of services, and monitoring and evaluation. 7. WHO, headquartered in Geneva, has a presence in virtually all developing countries as well as having responsibility for defining the international health policy for TB control. It provides technical assistance and advice to countries on policy formulation, project planning and implementation, and monitoring and evaluation of TB control activities. WHO maintains global surveillance and monitoring of TB incidence, drug resistance, and the status of control programs. 8. The CDC is an agency of the U.S. Department of Health and Human Services based in Atlanta, Georgia. CDC's mission is to promote health and quality of life by preventing and controlling disease, injury, and disability. CDC seeks to accomplish its mission by working with partners throughout the nation and world to monitor health, detect and investigate health problems, conduct research to enhance prevention, develop and advocate sound public health policies, implement prevention strategies, promote healthy behaviors, foster safe and healthful environments, and provide leadership and training. CDC has developed and sustained many vital partnerships with public and private entities that improve service to the American people. The purpose of TB CAP assistance is two-fold. First, it improves and expands the capacity of USAID to respond to the Global TB epidemic by providing well-coordinated state-of-the-art, context-appropriate, technically sound, and cost-effective consultation and technical assistance to high-incidence countries and USAID Missions. Second, it complements and expands existing global TB control efforts, such as those advocated by the Stop TB Partnership. The ultimate goal is to work in collaboration with other global TB partners and maximize existing efforts of Coalition partners to accelerate the pace of DOTS expansion to meet the global targets mentioned above. A further goal is to build additional global capacity for the provision of technical assistance. All of the TB CAP organizations are active members of the Stop TB Partnership. The TB CAP has two managerial levels that enable the partners to operate efficiently and effectively. The levels include the following: A Board of Directors that is responsible for overall policy, planning, quality assurance, and decision￾making with equal representation from all partner organizations; and A Program Management Unit (PMU) that is responsible for day-to-day project management of the administrative and technical work, and which is based at KNCV in The Hague. TB CAP programmatic objectives are to: Accelerate implementation and expansion of the DOTS strategy with emphasis on the high-burden countries as well as other countries strategically important to USAID. Develop institutional and technical capacity for TB control in target countries. 42 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) – Integrate TB control efforts within primary health care services more effectively; – Develop or strengthen linkages between TB control programs and HIV/AIDS prevention and treatment programs, and MDR-TB prevention and control efforts; – Disseminate the objectives of the TBCTA to USAID missions and the countries selected; and – Expand the capacity for providing high quality technical assistance worldwide. Within TBCTA a special group, Task Force Training (TFT), had been established to address the human resource development issues. Under TB CAP, TFT was broadened and changed to have a different structure to include a forum for HRD beyond the project. The task force was turned into a working group on HRD and a broader TB HRD platform network and meeting was created to provide input from the countries and partners with experience in developing human resources across the health sector and beyond. An online forum is available to maintain the network. In addition, two institutions in developing countries were competitively chosen to build regional institutional capacity. Currently, TB CAP has received field support from 31 USAID countries: Afghanistan, Bangladesh, Botswana, Brazil, Cambodia, 5 Central Asian Republics (CAR), Djibouti, Dominican Republic, DR Congo, Ethiopia, Ghana, Indonesia, Kenya, Malawi, Mexico, Mozambique, Namibia, Nigeria, Pakistan, Philippines, Russia, Sudan, South Africa, Uganda, Vietnam, Zambia, and Zimbabwe. Each country focuses on different gaps in the TB control program that need strengthening to reach the global targets. In addition, TB CAP has received field support from the Africa Bureau, Latin American and Caribbean Bureau, Europe and Eurasia Bureau, East Africa Regional Mission, and Guatemala Central American Program (G-CAP). Lastly, there are funds from the GH Bureau in Washington for core-funded activities. III. PURPOSE The purpose of the evaluation is to comprehensively review the performance, impact, and lessons learned of the TB CAP project collectively, in the different technical intermediate results (IRs), and in individual countries against the goals and objectives set for the project as well as its contribution to the goals and objectives of the Stop TB Partnership in a collaborative, efficient, and cost-effective manner. The evaluation should break down and analyze common factors or patterns for success or areas for improvement in the specific objective outlined below. The team will look retrospectively at the TB CAP cooperative agreement activities since inception and answer the principal questions: What is the evidence that TB CAP has made or will have made accelerations on the pace of DOTS expansion toward reaching the global targets and building institutional and individual capacity for TB control in targeted countries? In addition, is this the optimal framework and setup for a multifaceted mechanism with an array of partners within the Stop TB Partnership, taking into consideration USAID’s comparative advantages? The results will be used to provide feedback to TB CAP on project operations and management and provide recommendations to TB CAP and USAID on improving implementation. In addition, the evaluation will inform USAID in the future design of TB activities. IV. SPECIFIC OBJECTIVES AND ILLUSTRATIVE EVALUATION QUESTIONS Overarching Issues What are the overall lessons learned and best practices from the Coalition? Is the sum of the Coalition greater than the individual parts? What specific synergies have developed because of the Coalition? Has the Coalition been advantageous to USAID, each of the partners individually, and the broader TB community? Is TB CAP contributing to the coordination of partners and global leadership at different levels? EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 43 Has the Coalition taken full advantage of the specific capabilities of each of the partners? Do the Coalition partners meet the needs of USAID and the global demand for TB control? What needs are not being met? Has the capacity for technical assistance expanded and the quality improved? Is the Coalition too broad or narrow in partners and focus? Has USAID facilitated the work of TB CAP? How and to what extent have partners’ capacity been strengthened? Administrative Management Evaluate to what extent TB CAP has met the management functions outlined in the cooperative agreement, including planning, allocation of funds, and coordination of subagreements. Evaluate the efficiency and cost-effectiveness of the administrative structure and management of TB CAP, including the PMU and individual TB CAP partners. Evaluate to what extent the PMU and the individual TB CAP partners have been effective in addressing and resolving problems related to performance, quality of services, and human resources. Evaluate to what extent TB CAP supported the administrative components of the country work plans. Evaluate to what extent the new PEPFAR/OGAC funding stream has affected the management, administration, and reporting of TB CAP. What has been the impact of the knowledge management component of TB CAP (internal and external) and the use of electronic media on the project? How has TB CAP’s monitoring and evaluation plan effectively captured and informed program results? Is it effective, sufficient, and timely for measuring progress of the global, regional, and country level activities and for meeting USAID information/reporting needs? How has USAID’s oversight and management at different levels aided or hindered TB CAP’s strategic planning, management, and accomplishment of results? Identify the important strengths and weaknesses of the operational management of TB CAP. Evaluate TB CAP’s approach and efficiency in setting up in-country presences. What are the strengths and weaknesses of the different country management and implementation approaches of the TB CAP partners? What are the strengths and weaknesses of the different core project management and implementation approaches of the TB CAP partners? What are the strengths and weaknesses of the multipartner approach for core, regional, and country level activities? Has each partner filled a specific gap? To what extent has the coordinating and collaborating partner approach been efficient and effective? Technical Evaluate to what extent TB CAP has met the technical and programmatic objectives described in the technical documents, work plans, and cooperative agreement. 44 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) How has TB CAP met the intermediate results described in the technical documents, work plans, and cooperative agreement? How has TB CAP met the TB technical needs of USAID Global and Missions? Evaluate to what extent TB CAP has been innovative and creative in its approach to controlling TB. How did TB CAP improve and expand USAID and partners’ capacity to respond to the TB epidemic? Evaluate the product/outputs of the TB CAP combined efforts to contribute to controlling TB globally. Evaluate the product/outputs of the TB CAP efforts at the country level, including the reach and impact. What is the perceived impact (value added) TB CAP has on stakeholders working in TB control? Evaluate to what extent TB CAP has built institutional or individual capacity regionally or at the country level. How has TB CAP contributed to the integration of TB control within primary health care services? How has TB CAP contributed to the implementation of the Stop TB Strategy and the Global Plan to Stop TB (2006–15)? Specifically, evaluate the quality of TB CAP’s input into the following components: – Pursuing high-quality DOTS expansion and enhancement: focusing in case detection, standardized treatment, an effective drug supply and management system, and an M&E system – Laboratory capacity strengthening – Management of MDR and XDR-TB – Collaborative activities to address TB and HIV – Health systems strengthening – Engaging all care providers – Infection control Evaluate to what extent TB CAP has contributed to the availability of high quality technical assistance worldwide. Coordination and Collaboration How has TB CAP collaborated with other USAID-funded cooperating agencies, stakeholders, and other organizations and groups working in the field of TB control (specifically PEPFAR, Global Fund, TBTEAM, and GLI)? What role has TB CAP played in assisting countries with leveraging additional resources for program development and implementing including Global Fund, PEPFAR, etc? How has TB CAP contributed to the success of Global Fund TB grants in the country? How effective has TB CAP been in leveraging GF resources? How has TB CAP contributed to PEPFAR TB/HIV objectives? EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 45 What has been the role of the CDC in the Coalition? Has the coordination been maximized? Are there limitations in coordinating? Evaluate to what extent TB CAP has played a role in the global response to TB control. What has TB CAP contributed to the development of global TB policies and strategies? Evaluate to what extent tools developed by TB CAP have been disseminated and used beyond the Coalition partners. How does TB CAP relate to and work with the Stop TB Partnership? What role, if any, has TB CAP played in coordinating activities at the country level, working with the NTP and with other technical agencies? If they are serving in this role, are there some lessons learned or best practices in the area of partner coordination? Does TB CAP present an added advantage and appeal to the National TB Control Program (NTP) in comparison to other partners working on TB control? What has TB CAP contributed to ensure sustainability of TA interventions? V. METHODOLOGY The evaluation will gather information through reports and data review, TB CAP staff and local partner interviews, focus group discussions, and interviews with consumers, such as MOH personnel and local service providers and recipients, interviews with USAID Mission and Washington staff, and country field visits. The team should develop data collection and evaluation tools for ensuring consistency of information for different target groups and field visits. Sources of information and methods used should be described in the final report. Desk Review A member of the evaluation team will carry out a desk review of materials generated by the project as preparation for the full team background reading and prior to the Team Planning Meeting (TPM). This task will include review of various sources of information, such as quarterly reports, trip reports, annual reports and other relevant documents to be provided by the PMU. This desk review will help to organize the materials for external evaluation team analysis and review of progress to date. It will allow the team to quickly digest the wealth of information and maximize their time. The analyzed information and summaries as well as all the documents for conducting this desk review will be provided to the team. The team will have full access to the TB CAP eRoom where all the pertinent documents are kept and some communication is held. In addition: USAID/Washington will provide the team with historical program documents before the TPM. These documents will include the technical proposal, cooperative agreement, yearly work plans, financial documents, progress reports and any other relevant materials. The TB CAP partners will be a good source of information for additional documents on project activities. These documents will be made available upon request in advance for the desk review task and/or the team will have access to e-Room. GH Tech and the evaluation team will be responsible for collecting and reviewing any other relevant documents throughout the evaluation. 46 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) Team Planning Meeting A team planning meeting will be held in The Hague before the evaluation begins. The meeting is essential for the following reasons: Enabling USAID personnel, such as the CTO, TBTEAM, and others to present the team with the purpose and agenda of the assignment; Agreeing upon an approach for working with USAID and partners; Developing a common understanding of the individual team members’ responsibilities and roles; Establishing a team atmosphere for working together; Reviewing and clarifying logistical and administrative procedures for the assignment; Developing a common understanding of the assignment’s objectives and outcomes; Developing a preliminary draft outline of the team’s report; and Sharing information on individual working styles and agreeing on a procedure for resolving differences of opinion both within the team and with the client. Brief USAID Country Missions (USAID/W) and TB CAP country staff (GH Tech) prior to start of the assignment; and team will contact Missions on arrival and before departure. Ongoing discussions with USAID on the changes, adaptations, and developments of the evaluation. Interviews and Consultation Meetings 1. USAID/W (CTO, TB team leader, Division Chief, AO and others) 2. USAID Regional Bureaus 3. USAID Missions providing field support 4. PMU 5. Each Partner Headquarters Office (including CDC) 6. OGAC 7. Observe one board meeting (optional) 8. IR members 9. Stop TB Partnership Members Field/Country Visits* USAID Missions Other US government agencies working in TB Senior governmental health officials National Tuberculosis program manager and staff Other local TB institutions partnering with TB CAP EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 47 TB CAP employees in the field International organizations implementing TB activities Health care workers and managers at project field sites Clients Health facilities Laboratories *The criteria for choosing the countries to visit are the following: – Significant USAID field support; – Epidemiological importance on TB and TB/HIV; – Implementation of a variety of partners in coordinating and collaborating roles; – Scope of technical areas and level of funding; and – Geographic diversity. In-country Field Visits Debriefings At the end of each country visit, the team will report orally on the top findings to USAID Missions. It is anticipated that this session will help the team to clarify any questions or issues that came up during the field visits. Debriefings (Washington, DC) At the end of field visits, the team will report orally (and with slides) on initial top-line findings to both USAID and TBCTA Members. It is anticipated that these sessions will help the team to clarify any questions or issues that came up during the field visits and to solicit support for any gaps in information. Once the report is in the final draft format, the team will provide a debriefing in which the findings will be presented to USAID/W and TB CAP for comments. VI. TEAM COMPOSITION AND QUALIFICATIONS Senior Public Health Specialist: A public health expert with extensive experience with USAID project design, implementation, and evaluation. The person should have an excellent understanding of USAID operational, management, and technical approaches. Public Health Specialist: Two experts in international public health with expertise in monitoring, evaluation, and strategic planning. In addition, a good understanding of developing coalitions, coordinating partners, and developing partnerships. S/he should have experience with the Stop TB Strategy and its approaches. A good understanding of human resource and institutional development is desired. Senior TB Specialist: Expert in the field of international tuberculosis control with excellent understanding of the global strategy and its implementation. S/he should have many years of experience monitoring and evaluating various TB programs throughout the world. The expert should not be directly affiliated with any of the partners in the TB CAP coalition. An understanding of USAID is desired but not necessary. 48 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) Other Skills: The following general development skills should be covered by one of the three experts: health systems strengthening, advocacy and policy, Global Fund operations, scale-up, global leadership, sustainability, program quality assurance, surveillance, laboratory systems, drug management, community mobilization, and poverty alleviation. The Team Leader could be either of the Senior Public Health Specialists, preferably the one with more leadership experience. The level of effort for each consultant is estimated and outlined in Table 1, Tentative Implementation Schedule. VII. PERIOD OF PERFORMANCE The assignment will be conducted from o/a end January 2009 through May 15, 2009 (TBD, based on consultant availability). The following table is the projected schedule for implementing the methodology. It may be refined as a result of the TPM that will take place at the beginning of this assignment. Table 1: Tentative Implementation Schedule Task Team Leader/ Senior Public Health Specialist Public Health Specialist #1 TB Specialist Public Health Specialist #2 Desk review of relevant documents 1.00 10 Meeting with USAID and assignment planning in DC (includes a day of travel). 5.00 2.00 0 0 TBCTA Board Meeting to be held in Washington, DC February 18-19; team leader will attend/observe 2.00 0 0 0 Review of documents/background reading 4.00 4.00 4.00 4.00 International Travel to Europe, Asia, Africa & U.S. Travel to The Hague, Netherlands 2.00 1.00 1.00 2.00 Team Planning Meeting (TPM) in The Hague: finalize evaluation tools/guidelines, revise timeline, and finalize team interviews & trips 2.00 2.00 2.00 2.00 Continue preparation work: develop interview questions and analysis guidelines 1.00 1.00 1.00 1.00 Data collection/site visit in The Hague 2.00 1.00 1.00 2.00 Data collection/site visits in Geneva & Paris 4.00 - - 4.00 Travel to the US from Europe - 1.00 - 1.00 Conduct country level site visits in the ANE Region (1st country), including travel to/from country - 8.00 8.00 - Conduct USAID/W and other local key informant interviews (DC partners), including phone interviews - 7.00 - 7.00 Conduct country level site visits in Africa (1st country), including travel to & from country 8.00 - 8.00 - EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 49 Task Team Leader/ Senior Public Health Specialist Public Health Specialist #1 TB Specialist Public Health Specialist #2 Conduct country level site visits in Africa (2nd country), including travel to & from country 7.00 - 7.00 - Conduct country level site visits in Africa (3rd country), including travel to & from country 8.00 - 8.00 - Data analysis/ preliminary report preparation 8.00 8.00 Report preparation: discussion, analysis of data and preparation of Draft #1 executive summary &full report to USAID/W and TBCTA (Team reforms in the U.S.) 9.00 9.00 9.00 9.00 Debrief USAID/W and TBCTA members (separately) 1.00 1.00 1.00 1.00 Team departs Washington, DC/returns to residence 1.00 1.00 1.00 2.00 Review of Draft #1 by USAID/W and TBCTA (5 days) 0 0 0 0 Consultant team revises Draft #1 based on comments (becomes final version) 5.00 3.00 3.00 3.00 Feedback to final version/approval by USAID (5 days) 0 0 0 0 Team finalizes report: Completes revision and finalizes report substance 2 1 1 1 GH Tech edits/formats final version of the Executive Summary and Full Report (30 days) 0 0 0 0 Submit Executive Summary and Full Report to USAID/W (one hard copy; diskette in Word and .pdf files, the latter for Web site use if chosen by USAID/W 0 0 0 0 Assignment close-Out: USAID approval of deliverables and completion of GH Tech evaluation form 0 0 0 0 Total Consultant Days 64 50 55 57 *A six-day work week is authorized when working in the field (out of the US or resident country). VIII. DELIVERABLES AND CLOSE OUT Assignment Deliverables 1. Team Planning Meeting: All team members shall participate in the two-day Team Planning Meeting at the GH Tech Project in The Hague at the beginning of the project. The purpose will be to a. Review the goals and objectives of the assignment. 50 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) b. Develop a work plan, which will include, as needed, revision of the implementation schedule and initial development of the interview guides. – Review the background of the TB CAP Project and its current status. – Identify clients involved in the task, develop a common understanding of their relationships and interests, and agree on an approach to working with clients. – Clarify team members’ roles and responsibilities. – Establish a team atmosphere, share individual working styles, and agree on procedures for resolving differences of opinion. – Review and develop final evaluation questions. – Review and finalize the assignment timeline and share with USAID. – Develop data collection methods, instruments, tools, and guidelines. – Review and clarify any logistical and administrative procedures for the assignment. – Develop a preliminary draft outline of the team’s report. c. Assign drafting responsibilities for the final report. 2. Interviews and Site Visits: Team members will conduct interviews and site visits in accordance with the schedule developed prior to the TPM. 3. Draft Executive Summary and Full Report and Briefing with USAID/W and the TB CAP members: A detailed evaluation report with a summary of findings, including results and indicators for the project, feedback on performance and implementation, and recommendations for improved implementation and results. The evaluation team is required to provide the following report and briefing deliverables: a. A draft report will be provided to USAID/W and TB CAP no later than May 1, 2009. The report should not exceed 40 pages with an executive summary of no more than 5 pages. b. A debriefing with USAID and the partners to present the findings and received comments will occur before submitting the draft report. 4. Final Executive Summary and Full Report: After comments have been received from USAID and TB CAP on the first draft, the final Executive Summary and Full Report entitled ―Evaluation Report of the Tuberculosis Control Assistance Program (TB CAP)—Executive Summary (May 2009)‖ and ―Evaluation Report of the Tuberculosis Control Assistance Program (TB CAP)—Full Report (May 2009)‖ will incorporate the team responses to USAID and partner comments and suggestions. The draft final report should be completed within 5 days after USAID provides its feedback. After the final but unedited draft report has been reviewed by USAID, GH Tech will have the documents edited and formatted, and will provide the final report to USAID/HIDN for distribution (5 hard copies and a CD ROM). It will take approximately 30 days for GH Tech to edit/format and print the final document. B. Assignment Closeout Acceptance of the final products and completion of the GH Tech evaluation form: USAID/Washington will give the GH Tech Project written acceptance of the final version of the Executive Summary and the Full Report, and it will complete the GH Tech evaluation form, which will be provided to USAID/Washington by the GH Tech Project. EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 51 IX. RELATIONSHIPS AND RESPONSIBILITIES USAID: USAID will provide technical direction and assist in notifying the USAID Missions to help plan the trips. GH Tech: GH Tech will be responsible for travel logistics, including obtaining country clearances for GH tech consultants. GH Tech will work with the consultants to produce a report. Consultants will be responsible for coordinating and facilitating evaluation-related field trips, interviews, and meetings as needed. GH Tech will provide all technical and logistical support as well as submit report to the Development Clearing House. X. FUNDING This assignment will be funded by Global Health CSH funds earmarked for tuberculosis. 52 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 53 ANNEX II. TB CAP DESK AUDIT The desk audit report provided as a separate document, and available from USAID upon request. 54 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 55 ANNEX III. EVALUATION QUESTIONNAIRES QUESTIONNAIRE 1: USAID MISSION STAFF 1. Why did you decide to put money into TB in the country? 2. How did you decide in which TB areas to invest? 3. Why did you choose TB CAP to do this (versus an individual institution)? 4. How satisfied were you (generally) with the technical assistance they provided? with management of activities? with the quality of technical assistance? with responsiveness to Mission needs? with relationships among partners, such as TB CAP partners, other TB and/or NGO partners, PEPFAR/CDC, NTP, donors (Global Fund, GDF, bi- / multilateral donors)? 5. What in your views were major achievements, challenges, strengths, weaknesses? 6. What would your Mission like to see from future TB support? 7. Any other comments or information you would like to share? QUESTIONNAIRE 2: NATIONAL TB PROGRAM STAFF 1. What support did TB CAP provide the NTP? 2. How did you decide what TB CAP would do for you? How was that decision made? 3. How satisfied were you (generally) with the technical assistance they provided? with management of activities? with the quality of technical assistance? with responsiveness to NTP needs? with relationships among partners, such as TB CAP partners, other TB and/or NGO partners, PEPFAR/CDC, NTP, donors (Global Fund, GDF, bi- / multi-lateral donors)? 4. What in your views were major achievements, challenges, strengths, weaknesses? 5. What would your NTP like to see from future TB support from USAID? 6. Any other comments or information you would like to share? QUESTIONNAIRE 3: TB CAP PARTNERS 1. How do you support the NTP? 2. For TB CAP, what specific activities did you provide? 3. What in your views were your major achievements, challenges, strengths, and weaknesses? 4. What would your institution like to see from future TB support from USAID? 5. Any other comments or information you would like to share? 6. How do you feel that TB CAP technical assistance meets the priority needs of TB control in the country? Please comment on management of activities quality of technical assistance relationship with NTP relationships among partners, such as TB CAP partners, other TB and/or NGO partners, PEPFAR/CDC, NTP, donors (Global Fund, GDF, bi- / multilateral donors) 56 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 57 ANNEX IV. FIELD VISIT ITINERARIES During field visits the team members were accompanied by TB CAP; by national, state, and district NTP staff; and for some visits by staff from local partner organizations and from the USAID Mission. MARCH 10-13, THE HAGUE, THE NETHERLANDS, AT KNCV Meetings and briefing with the TB CAP PMU Review and discussion of evaluation methodology, schedule, and instruments Discussion of desk survey findings Administration and logistics (E. Berhane by phone) March 14. Departure to Indonesia, Geneva, and Washington, D.C. MARCH 16, GENEVA, SWITZERLAND, AT WHO Discussions and briefing by WHO Stop TB Department staff involved in TB CAP PPM, drug resistance, human and institutional capacity, political commitment, laboratory strengthening, TBTEAM, TB and HIV/TB coordinated activities Meeting with TB CAP board member and project officer MARCH 17, PARIS, FRANCE, AT THE UNION Meetings and briefing with the Union director and staff involved in TB CAP MARCH 16-21, INDONESIA March 16, Jakarta Briefing with USAID Attended TB CAP international workshop on infection control Briefing with KNCV/TB CAP Briefing with WHO Briefing with FHI March 17, Jakarta Briefing with Global Fund Briefing with CCM Visit to microbiology laboratory, U. of Indonesia Visit to Persahabatan Hospital Visit to PPTI (NGO) Travel to Surabaya and Semarang March 18, Surabaya Province, East Java Visit and briefing at the Provincial Health Office Visit to the Dr. Sutomo General Hospital Visit to a health center (Puskesmas A) Visit to the provincial laboratory 58 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) March 19, Surabaya Visit to Puskesmas B and health centers Visit to the provincial TB drug warehouse Departure for Kakarta March 18, Semarang Province, Central Java Visit to Provincial Health Office and briefing Visit to district warehouse in Semarang Visit to lung clinic BP4, Semarang Travel to Solo March 19, Semarang Visit to Puskesmas Health Center Visit to Muwardi Hospital Travel to Yogya Visit to Gadjah Mada University Regional Training Centre (TBCTA) Depart for Jakarta March 20, Jakarta Visit to Indonesian Medical Association (IMA) Debriefing USAID Briefing/debriefing NTP Debriefing KNCV/TB CAP March 21, Departure to Uganda and Washington, DC MARCH 23-27, UGANDA March 23, Kampala Meeting with TB CAP Officers, Union offices, Wandegeya Meeting with PEPFAR coordinator Meeting with IHAA (US government partner supported by TB CAP), Nakasero Visit Kawempe Home Care Initiative (local partner of TB CAP) Travel to Nakasongola District (district with limited intervention) Meet District Health Team (DHT) Nakasongola Visit a hospital, Health Center IV and community Travel to Hoima district (TB CAP–supported district) March 25, Hoima, Uganda Meeting with DHT Hoima Visit one hospital, Health Center IV, and community in Hoima district Travel to Masindi district (TB CAP–supported district) Meeting with the DHT Masindi district EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 59 Visit a hospital, Health Center IV, and community in Masindi district Travel back to Kampala March 26, Kampala Courtesy call, USAID Mission Director Meeting with NTLP program manager and officers, Wandegeya Meeting with CDC / visit central laboratory Meeting with WHO Departure for consultants to Entebbe airport, travel to Addis Ababa and Abuja MARCH 29-APRIL 3, NIGERIA March 29, Abuja Meeting with TB CAP, discussion of schedule Meeting with NTBLCP, briefing March 30, Abuja Meeting with USAID/CDC Meeting with WHO Meeting with FHI Meeting with MSGH Meeting with CHAN / ARFH (principal recipient, Global Fund) Travel to Lagos Team 1 March 31, Team 1 State Ministry of Health. Courtesy call on Commissioner/PS/DPH & CO Visit to Lagos University Teaching Hospital (LASUTH) Visit to Mainland Hospital, Yaba Visit to Lagos General Hospital, Lagos Island March 31, Team 2 Meeting with ILEP partners in Abuja Travel to Zaria (road) Visit Zaria National TBL Training Centre Travel to Kaduna April 1, Team 1 Travel to Enugu (air) and Abakiliki, Ebonyi state (ground) Courtesy call on Commissioner/PS/DPH & CO, Abakiliki Visit to Martha Hospital, Afikpo Visit to Presbyterian Hospital, Uburu Return to Enugu 60 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) April 1, Team 2 Travel to Kachia LGA Courtesy call on the Chairman of Kachia LGA Visit to DOTS centers and Kachia community for CTBC activities Travel back to Kaduna April 2, Team 1 Travel to Abakiliki, Ebonyi Visit to Mile 4 Hospital Visit FMC, Abakiliki Travel to Enugu (ground) and Abuja (air, via Lagos) April 2, Team 2 Depart Kaduna for Abuja Meetings with FCT DPH and FCT STBLCP at PH department, Area 11 Visit DOTS center, Bwari General Hospital Visit DOTS center, Garki PHC Visit DOTS center, Maitama DH Group meeting at TB CAP office April 3, Abuja Meeting TB CAP Debriefing with HMH Debriefing with WHO Debriefing with USAID April 4, departure to Mozambique via Addis Ababa APRIL 6-11, MOZAMBIQUE April 6, Maputo Meeting and briefing by FHI/TB CAP Meeting with USAID Mission Courtesy visit to the National Director of Health, Ministry of Health Meeting with the head of the NTP Visit to the National Reference TB laboratory Courtesy visit to the WHO representative April 7, Maputo Review reports, training materials, and documents Travel to Beira (air) April 8, Beira Courtesy visit to Provincial Director of Health, Sofala EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 61 Travel to Nhamatanda district Meeting with ADPP (community organization) Visit to Lamego Health Center Travel to Dondo District Visit to TB and TB/HIV programs in Dondo, including CB DOTS projects Travel to Beira Visit Beira Central Laboratory Travel to Maputo April 9, Gaza Travel to Gaza Courtesy visit to the Provincial Director of Health Visit CB DOTS projects in Chibuto Travel to Chowke Visit TB and TB/HIV staff and patients at Carmelo Hospital Review CB DOTS expansion in two districts of Gaza Return to Maputo April 10, Maputo Meeting with CDC Debriefing with USAID Debriefing with FHI/TB CAP Departure for Washington, DC 62 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 63 ANNEX V. WASHINGTON TELEPHONE INTERVIEW SCHEDULE Date Interviewees Tuesday, March 24, 2009 Peg Marshall (USAID/LAC Bureau) Wednesday, March 25, 2009 Catherine Chiphazi (USAID/Malawi) Thursday, March 26, 2009 Connie Davis (USAID/ESA) Ya Diul Mukadi (FHI) Cheri Vincent (USAID/GH TB CAP CTO) Friday, March 27, 2009 Nobukatsu Ishikawa (JATA) Pedro Suarez (MSH) Sunday, March 29, 2009 Mohammad Khakerah Rashidi, TB CAP Country Team Lead/Afghanistan (MSH) Seita Akihiro (WHO Dijbouti) Monday, March 30, 2009 Mohammad Shapor Ikram, (USAID/Kabul) Ruth Tembo and Peter Halpert (USAID Zimbabwe) Valentina Anisimova (KNCV Botswana) Getachew Wendimagegn (NTP Ethiopia) Tuesday, March 31, 2009 Victor Ombeka and Amos Kutwa (KNCV Kenya) Ezra Shimeles (KNCV Ethiopia) Christy Hanson (USAID/GH) Susan Bacheller (USAID/GH) Wednesday, April 1, 2009 George Sinyangwe (USAID/ZAMBIA) I.D. Rusen (Union) Emmanuel Kiangala, Sandrine Ruppol (Union DRC) Ridha Djebeniani, (WHO Djibouti) 64 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) Date Interviewees Thursday, April 2, 2009 Nazrul Islam (NTP Bangladesh) Agnes Gebhard and Katja Lumelova (KNCV Vietnam) Eugene McCray (CDC) Paula Fujiwara (Union) Friday, April 3, 2009 Farai Mayhunga (NTP Namibia) Joconiah Chirenda (NTP Botswana) Monday, April 6, 2009 Stella Goings (Africa Bureau) Tuesday, April 7, 2009 Phil Hopewell (ATS) Wednesday, April 8, 2009 Renee Liebenberg, Peter Gichangi, and Sangita Patel (USAID Namibia) Thursday, April 9, 2009 Katja Lumelova and Peter Gondrie (KNCV) Chifundo Kachiza (MSH Malawi) Cheri Vincent (TB CAP CTO) Amy Bloom (USAID/TB Team) Ellen Lynch (USAID/Vietnam) Friday, April 10, 2009 Fran Du Melle (American Thoracic Society) Irene Koek, Chief, Infectious Diseases Division, Bureau for Global Health Monday, May 4, 2009 William Coggin, HIV/TB Coorinator, OCGAS Email Responses 4/2/2009 Robert R. Makombe (CDC Zambia) 4/5/2009 Sukumar Sarker (USAID/Bangladesh) 4/5/2009 Rhehab Chimzizi (MSH Ghana) 4/7/2009 Molly Lindner (USAID/Mexico) EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 65 Date Interviewees 4/8/2009 Mansour Ahmed (USAID/Dijbouti) 4/4/2009 Xerses Sidhwa (USAID/Ethiopia) 4/18/09 Charles Sandy (NTP Zimbabwe) 66 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 67 ANNEX VI. FIELD INTERVIEWS AND INTERVIEWS WITH TB CAP STAKEHOLDERS Country Organization Name Position France Union Nils Billo Director General France Union Monica Andrfa Artaux Project Officer France Union Gilles Etienne Program Coordinator Indonesia FHI Indonesia Chawalit Natpratan Deputy Director for Technical Support Indonesia FHI Indonesia James Johnson n/a Indonesia Gadjah Mada University Adi Utarini Professor Indonesia Gadjah Mada University Haripurnomo Kushadiwijaya Head of Public Health Department Indonesia Gadjah Mada University Laksono Trisnantoro Professor Indonesia Gadjah Mada University Yodi Mahendradhata Professor Indonesia Global Fund CCM Sudijanto Kamso Professor of population and biostatistics, University of Indonesia, and CCM member Indonesia Global Fund CCM Tine Tombokan Vice-chair, CCM Indonesia Indonesia Global Fund PMU Cecep Slamet Budiono Coordinator Indonesia IDI (Indonesia Medical Association) Fachmi Idris President, IDI Indonesia IDI (Indonesia Medical Association) Pandu Riono Member, IDI Indonesia Indonesia Ministry of Health Iwan Muljono Director General, Communicable Diseases Indonesia KNCV Ratna Ekasari Technical Officer, HDL Indonesia KNCV Tony Priliono Technical Officer, HDL Indonesia KNCV Indonesia Anette Nainggolan Finance Controller Indonesia KNCV Indonesia Atik Setiowati Office Manager Indonesia KNCV Indonesia Hari Basuki TB CAP Technical Officer 68 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) Country Organization Name Position Indonesia KNCV Indonesia Jan Voskens Senior TB Consultant, National TB Officer TBCAP Indonesia KNCV Indonesia John Sugiharto National Technical Officer Indonesia Ministry of Health Carmelia Basri Head of Immunization and former NTP Manager Indonesia NTP Indonesia Jane Soepardi NTP Manager Indonesia NTP Indonesia Nadia Wiweko Head of Section Standardization Indonesia Persahabatan Hospital Arifin Nawas Pulmonologist Indonesia Persahabatan Hospital Clement Mayakori Director, Medical Services Indonesia Persahabatan Hospital Erlina Burhan Pulmonologist Indonesia PPTI H. Soediono Secretary General Indonesia PPTI Mariani Reksoprodjo Executive Director Indonesia PPTI Is Soetadi Treasurer Indonesia PPTI Raisis Arifin Panigoro Vice-Chairlady Indonesia PPTI Rati Siswono Yudohusodo Chairlady Indonesia Puskesmas (HC) Jagir Peni Tjahgati Head Indonesia Semerang Lung Clinic (BP4) Elham VCT Coordinator Indonesia Semerang Lung Clinic (BP4) Nurukyan Head of Lung Clinic Indonesia Semerang Muwardi Hospital Reviono Pulmonologist Indonesia Semerang Muwardi Hospital Trilastiti Hospital Vice-Director Indonesia Semerang Provincial Drug Warehouse Wahyu Indah Widowat Warehouse Manager Indonesia Semerang Provincial Health Office Hartanto Section Head, Provincial Health Office EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 69 Country Organization Name Position Indonesia Semerang Provincial Health Office Budi Nugroho Provincial TB officer, monitoring and evaluation Indonesia Semerang Provincial Health Office Retno Rindaswati Provincial TB officer, PPO, and Global Fund Indonesia Semerang Provincial Health Office Pri Dewi Kristini Provincial TB officer Indonesia Semerang Puskesmas Purwodiningratan Ida Nurnani Head, Puskesmas Indonesia Semerang Puskesmas Purwodiningratan Titiek Kadarsik Chief of Communicable Diseases Indonesia Surabaya City Health Office Esty Martiana Rachmie Director, Municipal Health Office Indonesia Surabaya City Health Office Meliana TB CAP Technical Officer Indonesia Surabaya Provincial Head Office, East Java Ansarul Fahrudda Section Head, Communicable Disease Control Indonesia Surabaya Provincial Head Office, East Java Budi Rahayu Head, Communicable Disease Control and Health Problem Division Indonesia Surabaya Provincial Head Office, East Java Setya Budiono Provincial Deputy Supervisor for TB Indonesia Surabaya Provincial Head Office, East Java Waras Provincial Deputy Supervisor for TB Indonesia Surabaya Provincial Head Office, East Java Sudarman TB Logistic Coordinator/Global Fund (National Office) Indonesia Surabaya Provincial Laboratory Endriana Soeryt Sp.Pk Microbiology Installation Head Indonesia Surabaya Provincial Laboratory Eveline Irawan Clinical Laboratory Section Head Indonesia Sutomo Hospital Setiawan TB CAP Technical officer, HDL Indonesia Sutomo Hospital Sisi Indriyani TB CAP Technical officer, HDL Indonesia Sutomo Hospital Sudarsono Head, Tuberculosis Department 70 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) Country Organization Name Position Indonesia University of Indonesia Agus Sjahrurachman Head, Microbiology Laboratory; Professor in Clinical Microbiology Indonesia University of Indonesia Retno Kadarsihs Clinical Microbiologist Indonesia USAID Charles Oliver Chief, Health and Infectious Diseases Section, Office of Basic Human Services Indonesia USAID Lisa Baldwin Senior HIV/AIDS Technical Advisor Indonesia USAID Ratna Kurniawati Public Health Advisor, HIV/AIDS and Infectious Diseases Program Specialist Indonesia WHO Bingherdeni Consultant Indonesia WHO Indonesia Firdosi Mehta Medical Officer for Tuberculosis Mozambique (Gaza Province) Chokwe District ArthurMacuacua District TB Supervisor Mozambique (Gaza Province) Chokwe District Maria Elisa Verdu Sister, Carmelo TB & HIV Hospital Mozambique (Gaza Province) Provincial Directorate of Health Carla S. Batista Palege Provincial Chief Physician Mozambique (Gaza Province) Provincial Directorate of Health Eunice Jorge Coordinator TB/HIV/Malaria Mozambique (Gaza Province) Provincial Directorate of Health Luis N. Mapanzeul Provincial TB Supervisor Mozambique (Maputo) CDC Hanifa Raman TB/HIV & OI specialist Mozambique (Maputo) CDC Jessina Massa n/a Mozambique (Maputo) CDC Lisa J. Nelson Mozambique Country Director Mozambique (Maputo) FHI Jorge Blanco Country Director Mozambique (Maputo) Ministry of Health Mouzinho A. O. SaÍde National Director, Health Promotion and Disease Control Mozambique (Maputo) Ministry of Health Paula Samo Gudo NTP Manager Mozambique (Maputo) National TB Reference Lab Elizabete Coelho Director EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 71 Country Organization Name Position Mozambique (Maputo) National TB Reference Lab Salomao Maungato Deputy Director Mozambique (Maputo) TB CAP Algy Cassamo Advisor, TB/HIV Mozambique (Maputo) TB CAP Benedito Toalha Advisor TB/HIV (FHI) Mozambique (Maputo) TB CAP Dario Sacur Field Project Director (FHI) Mozambique (Maputo) TB CAP Grace Cahenya Advisor TB Laboratory (MSH) Mozambique (Maputo) TB CAP Grace Kaifenta Laboratory Consultant Mozambique (Maputo) USAID Alfredo Mac Arthur Advisor, Infectious Diseases Mozambique (Maputo) USAID Jeri Dible Health, Population and Nutrition Officer Mozambique (Sofala Province) Dondo District Catarina Mabule Director of Health Mozambique (Sofala Province) Dondo District Lavos Domingos Coordinator AJUPSIC (Homecare) Mozambique (Sofala Province) Secretary of Health Juvenaldo Amo's Advisor, TB/HIV/malaria Mozambique (Sofala Province) Secretary of Health Marina Karagiane Provincial Director of Health Nigeria (Abuja Federal District) Abuja Bwari General Hospital Babade Chairman Nigeria (Abuja Federal District) Abuja Bwari General Hospital Fatiz Tanko DOTS FP Nigeria (Abuja Federal District) Abuja Bwari General Hospital Mr. Ekpeyong Laboratory FP Nigeria (Abuja Federal District) Abuja Bwari General Hospital Mr. Laminge FCT TBL Supervisor Nigeria (Abuja Federal District) Abuja Bwari General Hospital Mohamed Zakwoi DOTS FP, Garki PHC Nigeria (Abuja Federal District) Abuja Bwari General Hospital Betty Ndak Chief Nursing Officer 72 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) Country Organization Name Position Nigeria (Abuja Federal District) Abuja Bwari General Hospital Rabi Ibrahim AMAC TB Supervisor Nigeria (Abuja Federal District) Abuja Bwari General Hospital Saddiq TCT TB Control Officer Nigeria (Abuja) FHI Bridget O'Connor Deputy Chief of Party Nigeria (Abuja) FHI Christoph Hamelmann Country Director Nigeria (Abuja) FHI Mohammed Ibrahim Director, Medical Services Nigeria (Abuja) FHI Mohammed Liman Sr. TB/HIV Advisor Nigeria (Abuja) FHI Monye Chinedu Sr. Program Officer Program Management Nigeria (Abuja) FHI Robert Chiegil Director, Program Management Nigeria (Abuja) FHI Sunday Olarewaju n/a Nigeria (Abuja) Global Fund Christine Whalen TB CAP External Consultant Nigeria (Abuja) ILEP Charles Nwafor Medical Advisor GLRA Nigeria (Abuja) ILEP Moses Onoh Medical Advisor, TLMN Nigeria (Abuja) ILEP Mr Elijah Uzim Accountant, NLR Nigeria (Abuja) ILEP Mr Gehardt Oehler Administrator, GLRA Nigeria (Abuja) ILEP Mr Klaus Gilden Country Representative, GLRA Nigeria (Abuja) ILEP Sunday Odoh Country Representative, TLMN Nigeria (Abuja) ILEP Tahir Dahiru Medical Advisor, NLR Nigeria (Abuja) MSH Don Harbick Country Director Nigeria (Abuja) NTBLCP, Ministry of Health Gabriel Akang Deputy National Coordinator Nigeria (Abuja) NTBLCP, Ministry of Health Kenneth Adagba Program staff Nigeria (Abuja) NTBLCP, Ministry of Health Mansur Kabir National Program Manager EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 73 Country Organization Name Position Nigeria (Abuja) NTBLCP, Ministry of Health Mr. Ojika Laboratory Focal Point Nigeria (Abuja) NTBLCP, Ministry of Health Nkem Chukwueme Monitoring and evaluation Nigeria (Abuja) TB CAP Emmy van der Grinten National Coordinator (KNCV) Nigeria (Abuja) TB CAP Enang Oyama Sr, Technical Advisor, Systems strengthening Nigeria (Abuja) TB CAP Johan Verhoef Senior Administrative and Finance Officer Nigeria (Abuja) TB CAP Jumoki Onazi Monitoring and Evaluation Nigeria (Abuja) USAID Abu Ugbede Pharmaceutical Logistics Nigeria (Abuja) USAID Chistina Lausi Information Manager Specialist Nigeria (Abuja) USAID Christina Chappell Team Leader, HIV/AIDS-TB Nigeria (Abuja) USAID Elina Sverdlova PEPFAR Logistics Advisor Nigeria (Abuja) USAID Olse Ogony Assistant Program Manager TB/HIV Nigeria (Abuja) USAID Sianna E. Tackett Deputy Team Leader, HIV/AIDS-TB Nigeria (Abuja) USAID Temitayo Odusote Program Manager, TB/HIV Nigeria (Abuja) WHO Amos Omoniyi NPO TB/HIV-TB/HIV Advisor TB CAP Nigeria (Abuja) WHO Ayodele Awe NPO Tuberculosis Nigeria (Abuja) WHO Peter Eriki WHO Country Representative Nigeria (Ebonyi State) Federal Medical Center - Federal Hospital A. K. Onyebuchi Head of Clinical Services Nigeria (Ebonyi State) Federal Medical Center - Federal Hospital Attai Ekaette Principal Nursing Officer Nigeria (Ebonyi State) Federal Medical Center - Federal Hospital P. Ezeonu Medical Director Nigeria (Ebonyi State) Mather Misericordiae Hospital, Afipko Theresa Chirinma Aicpelu Deputy Director of Nursing Services 74 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) Country Organization Name Position Nigeria (Ebonyi State) Mile Four Hospital (Mission Hospital) Eghuba Sylvester O. Medical Superintendent Nigeria (Ebonyi State) Mile Four Hospital (Mission Hospital) Sir L. I. Otuh TB Supervisor Nigeria (Ebonyi State) Presbiterian Hospital, Uburo (Mission Hospital) Reverend Ibe O. Uka Hospital Secretary Nigeria (Ebonyi State) State Health Management Board Chijioke Osakwe WHO NPO TB, South East Zone Nigeria (Ebonyi State) State Health Management Board Samuel Nwhite DSTBLCO Nigeria (Kaduna State) Abuja Abdul Razaq Dikko Community HAST Officer Kano Nigeria (Kaduna State) Abuja Kaleem Phido SBS Officer Nigeria (Kaduna State) Kachia Alhaji Sabiu Suleiman Traditional Leader, Kachia Nigeria (Kaduna State) Kachia Hon. John Saidu Chairman, Kachia LGA Nigeria (Kaduna State) Kachia Mohamed Bello F. Haji Community HAST Officer, Kachia Nigeria (Kaduna State) Kachia Sunday Tukurah LGA TBL Supervisor, Kachia Nigeria (Kaduna State) Kachia - Crossing PHC Peter Laboratory Scientist Nigeria (Kaduna State) Kachia - Crossing PHC Rosemarie Gabriel IC Nigeria (Kaduna State) Kachia - Sabon Sarki PHC Hajiya Aisha Chairman, FOWMAN (Federation of Muslim Women of Nigeria) Nigeria (Kaduna State) Kachia - Sabon Sarki PHC Iliya Dodo Laboratory scientist Nigeria (Kaduna State) Kachia - Sabon Sarki PHC Zaka Makadi Chairman, Community Volunteers Nigeria (Kaduna State) Kachia - Sabon Sarki PHC Veronica Bature IC Nigeria (Kaduna State) National TB and Leprosy Centre, Zaria Clement Clinical Services EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 75 Country Organization Name Position Nigeria (Kaduna State) National TB and Leprosy Centre, Zaria Gidado Mustapha Training Director Nigeria (Kaduna State) National TB and Leprosy Centre, Zaria Kehinde Jimoh Information Management Nigeria (Kaduna State) National TB and Leprosy Centre, Zaria Mr. Rafael Accountant Nigeria (Kaduna State) State Health Office Julius B. Gajere State TB Control Officer Nigeria (Kaduna State) State Health Office Usman Gwarzo Zonal Manager Nigeria (Kaduna State) State Health Office Yusuf Mohamed Senior Medical Advisor Nigeria (Lagos) General Hospital, Lagos Island Lawal Oatunji Sakiru MO Chest Clinic Nigeria (Lagos) General Hospital, Lagos Island Wale Ajala Director, Laboratory Nigeria (Lagos) Lagos State Health Ministry Femi Taiwo Director, Disease Control Nigeria (Lagos) Lagos State Health Ministry I.G. Akinrinbole State TB and Leprosy Supervisor Nigeria (Lagos) Lagos State Health Ministry Mra L.O. Ojikuth State TB Laboratory Focal Person Nigeria (Lagos) Lagos University Teaching Hospital (LASUTH) M.O Bacolm TBLS Nigeria (Lagos) Lagos University Teaching Hospital (LASUTH) Mrs A.O. Oluwateyo CNO, DOTS Centre Nigeria (Lagos) Lagos University Teaching Hospital (LASUTH) Ojomo Abiola TB Lab Nigeria (Lagos) Lagos University Teaching Hospital (LASUTH) Yetunede Kuyinu DOTS Centre 76 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) Country Organization Name Position Nigeria (Lagos) Mainland Hospital, Yaba Olufemi Onanuga CMD Nigeria (Lagos) WHO Lagos Daniel Olusoji WHO NPO TB Nigeria (Lagos) WHO Lagos Elom Emeka MLS, NTBLCP Switzerland WHO Leopold Blanc Coordinator TBS, STB Switzerland WHO M. Yesudian Public-private mix, TB, /STB Switzerland WHO Malgorzata Grzemska TBS, STB Switzerland WHO Marcos Espinal Executive Secretary, Stop TB Partnership Switzerland WHO Mukund Uplekar Public-private mix, TBS, STB Switzerland WHO A. Brands TBS, STB Switzerland WHO A. Pantoja STB Switzerland WHO C. Gunnemberg STB Switzerland WHO Diana Weil Deputry Director, STB Switzerland WHO H. Getaun TB/HIV, STB Switzerland WHO Jacob Creswell Human Res. Development, TBS, STB Switzerland WHO Karin Bergstrom Human Res. Development, TBS, STB Switzerland WHO Karin Weyer Laboratory strengthening, STB Switzerland WHO Katherine Floyd STB Switzerland WHO Mario Raviglione Director, Stop TB Department Switzerland WHO Mateo Zignol Drug Resistance, STB Switzerland WHO Pierre-Yves Norval TBS, STB The Netherlands KNCV Jeroem van Gorkom Deputy Director, TB CAP The Netherlands KNCV Maarten van Cleeff Director, TB CAP The Netherlands KNCV Peter Gondrie Director, KNCV The Netherlands KNCV Ersin Topcuoglu Monitoring and Evaluation EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 77 Country Organization Name Position The Netherlands KNCV Ineke Huitema Human Resource Development The Netherlands KNCV Max Weis Infection Control The Netherlands KNCV Sharat Verma MDR The Netherlands KNCV Yared Kebede HIV Uganda (Hoima District) District Health Office Albert M. Balijura Health Educator, CBHC Coordinator Uganda (Hoima District) District Health Office Alex Batwaula Regional Liason Medical Officer, TB CAP Uganda (Hoima District) District Health Office Bommy Timica TB Focal Person Uganda (Hoima District) District Health Office C.C. Asiimus E. District Health Inspector Uganda (Hoima District) District Health Office Moses Asunbusa District TB/HIV Supervisor Uganda (Kampala) CDC Rose Odeke CDC TB/HIV Technical Advisor Uganda (Kampala) Central TB Laboratory Bagaya Bernara Laboratory Project Officer, TB CAP Uganda (Kampala) International HIV/AIDS Alliance (IHAA) Milly Katana Chief of Party/Project Director Uganda (Kampala) JSI (USAID/DELIVER Project) Paschal Nicholas Mujasi Deputy Chief of Party Uganda (Kampala) National TB/Leprosy Programme, Ministry of Health Francis Adatu NTLP Manager Uganda (Kampala) National TB/Leprosy Programme, Ministry of Health Stephen Ekaru-L'ago Logistics Officer, NTLP Uganda (Kampala) PEPFAR Michael A. Strong Coordinator Uganda (Kampala) PEPFAR Reuben Haylet Deputy Coordinator Uganda (Kampala) TB CAP (Union) Achiles Kata Technical Advisor Uganda (Kampala) TB CAP (Union) Anna Nakanwagi-Mukwaya Chief of Party, TB CAP Uganda (Kampala) TB CAP (Union) Caroline Egaddu Monitoring and Evaluation Officer Uganda (Kampala) TB CAP (Union) Fiona Nakalema K. Administrative Assistant 78 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) Country Organization Name Position Uganda (Kampala) TB CAP (Union) Geoffrey Wamala Finance and Operations Officer Uganda (Kampala) TB CAP (Union) Martin Ruhweza Deputy Chief of Party Uganda (Kampala) TB CAP (Union) Rhoda Namobiru Finance Assistant Uganda (Kampala) TB CAP (Union) Rose Okot Chono Technical Officer, Partner Support Uganda (Kampala) USAID Uganda David Eckerson Mission Director Uganda (Kampala) USAID Uganda Rachel Cintron Health Team Leader, HIV/AIDS & Education Office Uganda (Kampala) WHO Abel Nkolo National Professional Officer, TB/HIV Uganda (Kampala) WHO Imdw Joseph National Professional Officer, TB Uganda (Masindi District) District Health Office John Turyagarwkayo District Health Officer (DHO) Uganda (Masindi District) Health Centre IV Isingoma Siras District TB/Leprosy Supervisor Uganda (Masindi District) The Union/TB CAP Alex Batwaula Regional Liaison Officer, Western Region Uganda (Masindi District) Masindi Hospital George Bategamya Principal Medical Officer and Superintendent Uganda (Masindi District) Masindi Hospital K. Williams Health Educator/Comm. Mobilization TB CAP Uganda (Masindi District) Masindi Hospital Peter Tigambirna Focal Person, TB CAP Uganda (Nakasangola District) District Health Office Ssekitto Gerala District Health Officer Uganda (Nakasangola District) Health Center IV Godfrey Kasibante In charge, HSD Uganda (Nakasangola District) Health Center IV Henry Sendikaduna District TB and Leprosy Supervisor (DTLS) Uganda (Nakasangola District) Military Hospital Captain Oten Director Uganda (Nakasangola District) Military Hospital Pande Sfeonley Medical Officer, TB/HIV Uganda (Nakasangola District) Military Hospital TBK Igaga Chief Medical Officer EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 79 Country Organization Name Position United States ATS Fran Du Melle TB CAP Liason Officer United States ATS Phil Hopewell TB CAP Board Member United States CDC Eugene McCray TB CAP Board Member United States FHI Diul Mukadi Director, Care and Treatment Division; TB CAP Technical Officer United States FHI Julia Masterson Director, Public Health Programs; TB CAP Program Officer United States JATA Nobukatsu Ishikawa TB CAP Board Member United States MSH Catherine Mundy Principal Program Associate for Laboratory Services United States MSH Diana R. Silimperi Vice President of the Center for Health Services United States MSH Pedro Suarez TB CAP Board Member United States MSH Claire E. Moodie Senior Technical Officer/TB CAP Project Officer United States MSH Maria Pia Sanchez Principal Technical Advisor for Emerging Infectious Diseases United States Union I.D. Rusen TB CAP Board Member United States Union Paula Fujiwara Director, HIV Department United States USAID (Africa Bureau) Stella Goings TB Coordinator and Principal Advisor, HIV/AIDS United States USAID (East and Southern Africa Bureau) Connie Davis Senior Technical Advisor, Infectious Diseases United States USAID (GHB) Amy Bloom Senior Technical Advisor United States USAID (GHB) Cheri Vincent TB CAP Cognizant Technical Officer United States USAID (GHB) Christy Hanson TB Research Advisor United States USAID (GHB) Irene Koek Chief, Infectious Disease Division United States USAID (GHB) Susan Bacheller TB Team Leader United States USAID (Latin American and Caribbean Bureau) Peg Marshall Regional Advisor for MCH and ID 80 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 81 ANNEX VII. PROPOSED TB CAP INDICATORS Note: Priority indicates the suggested importance of the indictor for project monitoring. Area Indicator Notes Priority Political commitment Has a national strategic plan 2 Guidelines consistent with international recommendations (ISTC, WHO) Including DOT at health facility and community levels 1 NTP unit with budget line items for NTP activities/drugs Minimum is NTP staffing with funding for essential activities (planning, supervision, training, logistics). Optimum is with government funding for regular supply of TB drugs. 2 % of government funding of budget, including loans Influenced by grants. Complement to TB budget line. 3 Per capita investment in TB program Excluding general health staff salaries. Amount is less important than utilization. Minimum $0.10, some countries over $10, depending on policy, with little relation to efficacy. 3 Diagnosis Coverage (% of population) Difficult to measure, poor definition of DOTS (one facility [per district]), most countries at 100%. TB diagnosis may cover more than general health access (patients travel) but the result is inadequate access to treatment. 3 CDR (case detection) of new S+ Global and national target, national commitment. The denominator is an uncertain estimate that varies frequently. The indicator is incorrectly used at district and local levels, which may have different prevalence. 2 Population per microscopy laboratory (number of labs/population) Optimal 50,000/150,000 depending on population concentration (higher in cities, lower in dispersed populations or those difficult to access), technique (light, fluorescence), and staffing. Complemented by average number of smears per reader (with light microscopy acceptable load from 2-20 smears per day). 3 Number of slides per light microscopy reader Optimum average 2-20 per reader per day. Used once a year for planning expansion or staffing. 2 % of microscopy units with regular EQA Either rereading an ad hoc proportion of smears or lot sampling. Complemented by % of labs with acceptable results. Definition of “acceptable” is unclear, and should be based on predictive value. 1 82 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) Number of suspected cases detected and examined by microscopy (trend). Data from TB microscopy registers In facilities it can be compared with adult outpatient load (expected 3%, the real % can be studied easily); nationally with the population (average no. of health visits/ person/ year is fairly constant). The trend measures expansion of case detection. Gives weight to CDR. 1 Positivity of suspects (trend) Complementary, indicates efficiency of microscopy and TB prevalence. Should be <10% and >1% (at less than 1%, the predictive value is low); >10% in specialized facilities or high self-referral 2 Proportion of pulmonary smear– positive TB out of total cases and pulmonary cases Rough indicator of the capacity to diagnose and report children and EP, and quality of clinical diagnosis of pulmonary TB. Expected is about 50% of total cases and 2/3 of pulmonary cases. Low proportion of pulmonary cases suggests X-ray over-diagnosis 3 # labs with culture and population per culture laboratory Optimum 1 per province or 1 per million inhabitants. Need more culture labs when implementing MDR treatment 3 # with DST One good one per country is enough, in large countries 1 per province. Need more when implementing MDR treatment 3 Treatment Treatment success rate (treatment outcomes) Strong, reliable. Calculated on a cohort of new pulmonary smear– positive cases is the main indicator of program performance. Complemented by % cured (most cases should be cured, with smear controls in second phase of treatment); failure rate (high due to MDR and poor drug quality); deaths during treatment (high due to HIV and late diagnosis); and default rate (optimum <5%). Sputum conversion at 2-3 months is an early predictor, a useful indicator for new areas or short-term (< 5 year) projects. 1 Drugs Drug stockouts: at central level at peripheral level Need better definition of duration and if the stockout resulted in interruption of drugs to patients. Indicates poor government commitment (funding), planning (reserve stocks for 1 year), and management (distribution) 3 M&E Registration consistent with WHO recommended system (patient cards; TB register, lab register and quarterly reports at district level) Registration on paper promotes local analysis; electronic systems facilitate consolidation and transmission of data. The analysis of data and graphic trends is a useful indicator of staff capacity for self￾monitoring 3 Quarterly analysis and reporting at district level Use of data at district and peripheral levels for self-monitoring is an important element to be included in training and supervision. 3 MDR DRS conducted One study useful for decisions on MDR management; periodic studies to show trends and impact of treatment (positive or negative) 3 Treatment policy, guidelines, and training materials Availability and consistency with international recommendations 1 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 83 Treatment of MDR with GLC GLC reduces cost and also ensures rational use of drugs 3 % MDR in new TB cases Impact/quality of previous TB control. Must be studied according to standard (IUATLD) protocols by quality assured laboratories 3 Number of MDR patients treated Indicates expansion of access to MDR diagnosis and treatment 2 Number of MDR patients that converted culture at 6 months Equivalent to smear conversion in first-line drugs, early predictor of treatment success 3 PPM Piloting or scaling up PPM interventions Difficult to define unit of measure 3 Number of private providers collaborating with the NTP There is a problem of definition: Is one MD equivalent to one private hospital or NGO? 3 % of total notified TB cases diagnosed by PPM partners Very difficult to collect routinely, special studies are a better option. 3 ISCT promoted and endorsed by professional organizations Minimum 3 TB/HIV Policy re testing for HIV in TB patients Minimum 3 Surveillance of HIV in TB patients Minimum (available data from surveys or surveillance system) 3 % TB tested for HIV The usefulness and target depends on HIV prevalence 1 % TB HIV+ on CPT Useful in countries with high HIV prevalence 2 % TB HIV+ on ART Depends on national criteria (based on CD4 count or not) 3 HRD Plan for HRD in place Useful but insufficient. Permanent training program required for new staff and retraining; including system to identify training needs and staff. 2 Job descriptions up to date Not well defined 3 % health facilities with staff trained in last 3 years Annual survey, not well defined 3 HRD focal point and funds at NTP Also indicates government commitment. 1 Number of staff trained, by type Useful for planning and monitoring 3 TB mortality Deaths due to TB, from death certificates Shows rapidly the impact of interventions (diagnosis and treatment). Epidemiological indicator, not very influenced by program, useful for short-term projects; data coverage is usually partial so not very good to compare countries but good for trends. 1 84 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) 85 ANNEX VIII. REFERENCES In addition to interviews, the evaluation team referred to a significant number of electronic and print resources to inform the evaluation, including (but not limited to): All TB CAP published Annual Reports (APA1, APA2, and APA3) All TB CAP published Semi-Annual Reports (APA1, APA2, and APA3) Independent Auditor Reports of TB CAP (2005-2006, 2006-2007) TBCTA Independent Evaluation Report TBCTA Final Report TB CAP Cooperative Agreement and KNCV subagreements with partners Every approved annual country workplan and budget for each TB CAP country supported between 2005 and 2009 Every available monitoring spreadsheet for each TB CAP country supported between 2005 and 2009 TB CAP Annual Workplans (APA1, APA2, and APA3) US Agency for International Development: Expanded Response to Tuberculosis, January 2009. The Stop TB Strategy The Global Plan to Stop TB, 2006–2015 Compendium of Indicators for Monitoring and Evaluation of National Tuberculosis Programs International Standards of Tuberculosis Care TREAT TB informational flyers Course curriculum and materials from Zaria and Gadjah Mada regional training centers National TB control strategies and plans, policies, guidelines, and other published elements of TB control in all four field sites visited TB CAP EXACT Financial System TB CAP Financial Status Reports (SF269) for APA1, APA2, and APA3 TB CAP RFA M-OAA-GH-HSR-05-1015 86 EVALUATION OF THE TUBERCULOSIS CONTROL ASSISTANCE PROGRAM (TB CAP) For more information, please visit http://www.ghtechproject.com/resources.aspx Global Health Technical Assistance Project 1250 Eye St., NW, Suite 1100 Washington, DC 20005 Tel: (202) 521-1900 Fax: (202) 521-1901 www.ghtechproject.com