EVALUATION OF TUBERCULOSIS PROGRAM IN INDIA WHO REPORT APRIL 2011 This publication was produced for review by the United States Agency for International Development. It was prepared by Aime De Muynck, David Berger, Tim Clary, Selva Kumar, Gani Perla, and Rajeswari Ramachandran, through Social Impact. EVALUATION OF TUBERCULOSIS PROGRAM IN INDIA WHO REPORT DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Evaluation of Tuberculosis Program in India: WHO Report 3 This document is available in printed or online versions. Documents are also made available through the Development Experience Clearing House (www.dec.org). Additional information can be obtained from: Social Impact, Inc. 2300 Clarendon Boulevard Arlington, VA, 22201 Tel: (703) 465-1884 Fax: (703) 465-1888 info@socialimpact.com This document was submitted by Social Impact, Inc., to the United States Agency for International Development under USAID Evaluation Services, Contract No. RAN-I-00-09-00019. Evaluation of Tuberculosis Program in India: WHO Report 4 ACKNOWLEDGMENTS The evaluation team would like to thank USAID/India, WHO, CTD, the STOs and their staff, the WHO consultants, and the various partner organizations whose shared time, experiences and insights were critical for the team to understand the scope and achievements of WHO’s support to RNTCP activities in India, the challenges and constraints faced by WHO, and the efforts undertaken to overcome them. Evaluation of Tuberculosis Program in India: WHO Report 5 ACRONYMS ACSM Advocacy, Communication, And Social Mobilization AFB Acid Fast Bacillus AIC Air Borne Infection Control AIDS Acquired immune deficiency syndrome AP Andhra Pradesh (India) ART Anti-Retroviral Therapy ASM American Society of Microbiology BPHRC Blue Peter Public Health and Research Centre BSL3 Biosafety Level 3 (India) CBO Community-Based Organization CDC Centers for Disease Control C-DOTS Community-Based DOTS C&DST Culture & Drug Susceptibility Testing CIDA Canadian International Cooperation Agency CMC Christian Medical Centre CTD Central TB Division (India) DCP Drug Control Officer DDG Deputy Director General DFID Department for International Development DMC Designated Microscopy Centre DOTs Directly Observed Therapy, Short Course DQA Data Quality Assurance DST Drug Susceptibility Testing DTC District Tuberculosis Center DTO District Tuberculosis Officer EQA External Quality Assurance FIND Foundation for Innovative New Diagnostics FY Fiscal Year GFATM Global Fund to Fight AIDS, Tuberculosis and Malaria GLC Green Light Committee GoI Government of India HIV Human Immunodeficiency Virus HR Human Resources IC Infection Control IEC Information, Education, and Communication IMA Indian Medical Association IQC Indefinite Quantity Contract IRL Intermediate Reference Laboratory JMM Joint Monitoring Mission LED Light Emitting Diode LPA Line Probe Assay LRS Institute of TB and Respiratory Diseases (in Delhi) LF Lab Technician M&E Monitoring And Evaluation MDR-TB Multidrug-Resistant TB MOH Ministry of Health MSH Management Sciences for Health NAAT Nucleic Acid Amplification Test NGO Nongovernmental Organization Evaluation of Tuberculosis Program in India: WHO Report 6 NPO National Program Officer NRL National Reference Laboratory NTI National Tuberculosis Institute, Bangalore OPD Outpatient Department OR Operational Research PF Provident Fund PIH Partners in Health PMDT Programmatic Management of Drug-Resistant TB PMP Performance Monitoring Plan PP Private Practitioner PPM Public-private mix PR Principal Recipient QA Quality Assurance QC Quality Control RNTCP Revised National Tuberculosis Control Programme (India) SAMS Strategic Alliance Management Services SEARO WHO Regional Office for South-East Asia SLD Second-Line Drug SNRL Supranational Reference Laboratory SOP Standard Operating Procedures SLDST Second-Line Drug Susceptibility Testing SR Sub-Recipient STC State Tuberculosis Centre STDC State Training and Demonstration Centre STO State Tuberculosis Officer TA Technical Assistance TASC2 Technical Assistance and Support Contract II TB TEAM TB Technical Assistance Mechanism TB Tuberculosis TDS Tax Deduction at Source ToR Terms of Reference TOT Training of Trainers TRC Tuberculosis Research Centre, Chennai TU Tuberculosis Unit TWG Technical Working Group USAID United States Agency for International Development WHO World Health Organization WHO-WR World Health Organization Representative WHO-SEARO World Health Organization Regional Office for South-East Asia XDR-TB Extensively drug-resistant TB Evaluation of Tuberculosis Program in India: WHO Report 7 TABLE OF CONTENTS EXECUTIVE SUMMARY.............................................................................................................................................................. 8 I. INTRODUCTION................................................................................................................................................................11 II. PURPOSE OF THE EVALUATION...........................................................................................................................12 III. RESEARCH DESIGN AND EVALUATION METHODOLOGY.....................................................................12 METHODS..................................................................................................................................................................................12 Team Planning Meeting (TPM) ............................................................................................................................................12 Site Visits and Interviews.......................................................................................................................................................13 The Evaluation Framework...................................................................................................................................................13 IV. FINDINGS.........................................................................................................................................................................14 PROCESS FINDINGS ABOUT THE MANAGEMENT AND IMPLEMENTATION OF THE PROGRAM .14 Monitoring and Evaluation....................................................................................................................................................15 OUTCOME AND RESULTS..................................................................................................................................................16 Findings About the Project’s Achievements and Consequences, Conclusions and Activity-Specific Recommendations...................................................................................................................................................................16 Support of Operational Research (OR) through TRC......................................................................................................19 Engage the Private Sector in TB Control............................................................................................................................22 WHO Consultants’ Network.................................................................................................................................................24 A Case Study: WHO consultants’ involvement in poorly performing districts..............................................................................33 V. GENERAL CONCLUSIONS AND RECOMMENDATIONS.............................................................................38 APPENDIX 1: SCOPE OF WORK...........................................................................................................................................40 APPENDIX 3: PERSONS CONTACTED...............................................................................................................................58 APPENDIX 4: LIST OF REFERENCES..................................................................................................................................65 APPENDIX 5: TEAM MEMBERS.............................................................................................................................................67 APPENDIX 6: PPM PARTICIPANTS IN THE VARIOUS STATES................................................................................68 MAPS Map 1: Districts selected for the review .........................................................................................................31 FIGURES Figure 1: Quarterly TB suspects’ examination rate by district, UP, 2009-10 .................................... 32 Figure 2: Quarterly Case notification rates of S+ cases by district, UP, 2009-10 ............................ 32 TABLES Table 1: Determining factors of RNTCP performance in 4 districts of east Uttar Pradesh, 201036 Evaluation of Tuberculosis Program in India: WHO Report 8 EXECUTIVE SUMMARY All of India has been covered with directly observed therapy, short course (DOTS), services—the internationally recommended strategy for tuberculosis (TB) control—since March 2006; targets of seventy percent case detection and eighty-five percent cure rate of new smear positive (NSP) cases have already been met for several years. Nevertheless, there is no proof that the Revised National Tuberculosis Control Program (RNTCP) has had an epidemiological impact; furthermore, India is facing a growing problem of multi-drug resistant tuberculosis (MDR-TB). In order to adequately cope with this MDR-TB burden, RNTCP needs to expand the network of accredited laboratories and involve the private practitioners (PPS) more effectively. Given that a significant fraction of the Indian population, including the poorest segments, consult private practitioners, an epidemiological impact can be reached only if the private sector becomes fully involved in TB diagnosis and care. It is in this challenging context that USAID/India is supporting RNTCP through the services of the World Health Organization (WHO). The main objectives of the partnership are to (1) strengthen the laboratory network for mycobacterial culture, Drug Sensibility Testing (DST) and the introduction of line probe assay; (2) intensify the involvement of private health care providers/organizations; (3) support operational research; and (4) provide technical support to all RNTCP activities, via the RNTCP consultant network. USAID/India contracted Social Impact, Inc. (SI), to carry out an in-depth evaluation of the support provided to RNTCP by the World Health Organization. The objectives of the evaluation were to determine the impact of WHO support, relative to stated objectives and achievements, and to make suitable recommendations for future direction and priorities. The evaluation team conducted an initial planning meeting with USAID/India, prepared an overall framework that was reviewed and approved by USAID/India (see Appendix A), and carried out a series of field visits and in-depth interviews in Andhra Pradesh, Uttar Pradesh, Maharashtra, Gujarat and Rajasthan. The team made the following conclusions and recommendations, in reference to each of the key activities: 1. Implementation arrangements. While WHO’s management of the USAID-funded TB project has been sufficient to deliver the desired results, the project is now entering a new, critical phase. Not only are the headquarters of WHO/India office severely understaffed, but the initiation of RNTCP3 and the quintupling of proposed resources will put heavy burden on the WHO/India office; it is difficult to envision how the WHO office could continue to function effectively if current staffing levels are maintained. The hiring of additional, well￾qualified, senior-level staff with good managerial capabilities is of utmost and urgent importance. Though USAID/India may not have sufficient leverage with WHO/India and WHO/Geneva to ensure that this hiring takes place quickly, it may, in conjunction with USAID/Washington (and, perhaps, other donors), apply sufficient pressure on WHO in order that a significant number of the needed staff soon enter the recruitment process. USAID must make it clear to WHO that future funding decisions will be based on WHO’s implementation of these decisions. 2. Monitoring and evaluation. Efforts to strengthen the public sector’s M&E capacity through WHO must be put into context; M&E capacity varies greatly by state and, to date, M&E as a discipline has been subsumed into the other aspects of TB prevention and control efforts. In states that have sufficient M&E capacity for routine monitoring, the role of the WHO consultant has evolved into higher-level epidemiological functions, yet the corollary training for these functions is lacking. Likewise, there have been relatively few opportunities for the WHO consultants and their counterparts to meet and discuss TB M&E as a discipline in and of itself. This needs to change. Finally, M&E at the central level (Central TB Division, or CTD) has been driven primarily by WHO/India, as presently there is very limited capacity within CTD. A combination of actions, including WHO hiring sufficient M&E staff for its headquarters and then strengthening the M&E capacity of CTD, along with instituting a more organized and formal M&E training program for both its consultants and their public sector counterparts, should ensure that progress continues in strengthening the TB M&E system. Evaluation of Tuberculosis Program in India: WHO Report 9 3. The laboratory network. USAID support allows WHO to provide technical assistance to build capacity at RNTCP national reference laboratories (NRLs)and intermediate reference laboratories (IRLs) in order to (1) perform culture and drug-susceptibility testing (DST) services; (2) develop and initiate a large scale demonstration and field evaluation of Nucleic Acid Amplification Testing (NAAT) for early and improved TB case detection; (3) create two additional NRLs with capacity for second-line DST (SLDST); and (4) create a RNTCP Laboratory Task Force and support its functioning. These activities have taken place as planned and were carried out adequately, with two exceptions: the national laboratory task force has not yet materialized due to administrative obstacles related to the use of funds by WHO, and the money for the NAAT field evaluations has not been spent yet for administrative and logistical reasons, although everything is ready to start soon.. USAID should continue to support WHO’s execution of the NAAT field trials, in order to allow RNTCP to incorporate this highly promising test into a diagnostic strategy in field conditions. Better coordination between the engagement of WHO and PATH is desirable in order to optimize the efficiency of their joint efforts in lab strengthening, especially given that both are financially supported by USAID. Build-up of strong laboratory leadership and technical staff might be considered for extra support; USAID should insist that the government of India (GoI) create the conditions necessary to attract and keep valuable technical staff, and support the enrollment of the private laboratories in external quality assurance (EQA) programs and in networking with the intermediate reference laboratories (IRLs) and national reference laboratories (NRLs). 4. Engaging non-public providers. Pilot projects have demonstrated that public-private mix (PPM) is feasible in TB control in India and has a proven potential to increase RNTCP performance. Field observations made during this review have identified that scale-up is constrained by a series of factors related both to the PPs themselves and to the public sector authorities. The conditions underlying the potential success of PPM include a clear understanding and delineation of the roles, responsibilities, and accountabilities of both public and private sector actors in a true spirit of partnership, equity, risk sharing, and transparency. Given that the involvement of PPM in RNTP3 will need to increase substantially, the following actions are recommended: (1) carry out a study to ensure RNTCP3 is conducting activities based upon evidence-based assessment and (2) carry out action researches to optimize effective implementation of RNTCP3’s novel approaches. Efforts should be made to ensure the partnership with PPs are mutually beneficial, which requires exploring what a “win-win” situation would look like for the several groups of PPs; their monetary and non-monetary expectations with respect to their active participation in PPM; the amount of time they are willing to spend on PPM; and the maximal administrative burden they are willing to accept. Via the consultants’ network, WHO should indicate any special needs with regards to the transition from RNTCP2 to RNTCP3 at the state and district levels, and state what support the private sector could offer to bridge those gaps. 5. Support Operational Research through TRC. The USAID funds utilized to support TB operational research (OR) in India have provided some of the best returns on investment. Not only are the results used to provide better prevention, care, treatment and support (and thus save lives), but they have also led to cost-savings within the TB program. This is highly commendable, and most of the credit belongs to the Tuberculosis Research Centre (TRC) in Chennai, which has received guidance from WHO through USAID support. However, much of the progress in moving the TB OR agenda forward could be lost in the upcoming years. Currently, TRC is facing a serious staffing shortage. A large cohort of senior-level experts and managers are retiring, with few mid-level managers to replace them. In addition, TRC is understaffed, in general. Additional challenges that need to be addressed include: (1) determining TRC’s future role in OR, both in India and internationally; (2) remedying the lack of collaborating partners in India; (3) ensuring that its research methods remain up-to-date; and (4) possible cash flow and cash management issues. 6. WHO consultants’ network. The principal factor of DOTS’ successful implementation in India has been—and still is—the involvement of WHO-contracted local consultants, who provide technical support at CTD, state and district levels. Indeed, eighty-six field-level consultants are assigned to specific states and work closely with district and state TB officers, enhancing the capacity of district health systems for supervision, Evaluation of Tuberculosis Program in India: WHO Report 10 monitoring and evaluation; assisting in data management and electronic transmission of the quarterly surveillance data to the national level; and improving record-keeping and monitoring the consistency and accuracy of the quarterly cohort data. Additionally, they provide feedback on RNTCP performance to the national level. Half of the financial support for this network is provided by USAID and half by the United Kingdom Department for International Development (DFID). Since September 2008, the consultants depend administratively on RNTCP only, while the management of the network is outsourced. This change of label is of great concern to the consultants, diminishes the attractiveness of the position, and increases the turnover. The network has contributed significantly to the good performance of RNTCP, and provided an element of stability and continuity within a context of frequently-changing GoI staff at the state level. This network is of low cost, but it produces tremendous benefits for TB control at the state, district and even national levels in India. It is recommended that USAID continue to support this network up to the end of the launching period of RNTCP3 (2017), and, if possible, take over full financial support once DFID withdraws its support in December 2011. In order to increase the sustainability of RNTCP performance, a concrete plan to operationalize the transfer of their knowledge and skills to the homologues in the RNTCP system should be prepared by WHO, CTD and the state tuberculosis officers (STOs), detailing precise benchmarks for the transfer. In addition to the eighty-six state-level consultants, ten other consultants provide technical support to the CTD, with thematic responsibilities in the domains of PPM, HIV/TB, HR, advocacy, communication and social mobilization (ACSM), etc. They are one of the pillars of CTD’s good performance, but they have no real counterparts, which handicaps an eventual exit strategy. Their present job satisfaction is found to be very high, although their career development opportunities are minimal, and the present salary is less attractive than it has been in the past. This network provides excellent value for money. As CTD is moving quickly to take up the RNTCP3 challenges in 2012, the consultants’ role will become more crucial than it is today, because there is yet no model in the country to reach the ambitious RNTCP3 targets. A case study carried out in East Uttar Pradesh state has learned that the WHO consultants play an important role in correctly performing districts as well as in poorly performing ones: in the latter, the consultants’ support is crucial to maintaining a minimum level of case finding and case holding services. GENERAL CONCLUSIONS AND RECOMMENDATIONS The evaluation recognizes the crucial role that WHO plays in the implementation of RNTCP2. It is a key observation that, without the heavy involvement of WHO, RNTCP would not have been able to reach the targets of universal coverage, of seventy percent case detection and eighty-five percent cure of NSP cases, in such a short time and in so many states and districts. The question of “value for money” invested by USAID/India in WHO support to RNTCP can be answered, without any doubt, positively for all the domains that were reviewed. The role that the WHO consultants play at state and CTD levels is of capital importance for the good performance of TB control in India. In those states and districts where RNTCP performance is still sub-optimal, the involvement of the WHO consultants prevents decreases in performance; in states and districts where the performance reaches the case detection and cure targets, their involvement allows exploration of methods to increase the quality of the RNTCP implementation and strengthen evidence-based program management. The evaluation team recommends continuing and, if possible, increasing the level of funding of WHO, and especially taking over the full support of the WHO consultants network once DFID terminates its financial support in December 2011. However, guarantees should be obtained for the transfer of skills to homologues within the system, and benchmarks should be established for monitoring this transfer. Support to the WHO-consultants network should continue until the end of the launching of the RNTCP3 program (2017). Evaluation of Tuberculosis Program in India: WHO Report 11 I. INTRODUCTION A review mission of USAID support provided to the Revised National TB Control Program (RNTCP) through WHO took place from February 2, 2011 to March 30, 2011. The team, comprising of six experts,1 visited Dr. L.S. Chauhan, acting Deputy Director General of the Central TB Division, CTD; and Dr. P. Dewan, WHO-WR international TB expert. The team also interviewed a series of key staff working in partner organizations (see list in Annex 4) and carried out field visits in various states, including Andhra Pradesh, Uttar Pradesh, Rajasthan, Gujarat, and Maharashtra. The main content of this WHO review was discussed with USAID technical staff on March 16, 2011, and with the USAID Front Office on March 18, 2011. WHO provides technical support to the RNTCP in the areas of planning, capacity building, monitoring, surveillance, evaluation, drug distribution, electronic connectivity, Public-Private Mix (PPM), and operational research.2 This support to RNTCP started in 1993 and increased when RNTCP extended Directly Observed Therapy—Short course (DOTS) coverage from 1998 onwards. India has faced several challenges in implementing RNTCP, including (1) sub￾optimal functioning of the general health services, which has a negative impact on case finding indicators and on the start of treatment; and (2) the huge role of the private sector in health care delivery and provision of outpatient services (OPD). Unfortunately, the services provided by the private sector are still of inconsistent quality. Unequal socio-economic development in the states and districts influences access to health services and adherence to treatment. Additionally, differentials exist in the attention that individual states give to health (as health is a state matter), affecting the filling of staff positions, the transfer rate of staff, etc.3 In spite of these challenges, RNTCP has realized the remarkable achievements of reaching universal coverage by March 2006, and of achieving the case detection target of seventy percent and the treatment success target of eighty￾five percent. Nevertheless, there is as yet no objective evidence of an epidemiological impact on the incidence of TB. This implies that, in order to influence the incidence, extra measures need to be taken, such as (1) involving the private sector right from the TB suspect’s first contact with the health system; (2) improving the sensitivity of the diagnostic tests (to avoid labeling TB cases as “non cases” when in fact they are the consequence of a “false negative test only”); (3) improving the early detection of cases through active contact tracing; and (4) continuously monitoring the trends of TB suspects examined, cases found, and cases treated/cured through modern surveillance techniques. To face these challenges, India is now ready to embark on a major new initiative to provide universal access to TB care, decrease treatment delay, increase treatment outcomes, address on a large scale the problem of multi-drug resistance, and involve the private sector in a full-fledged and comprehensive manner. This ambitious plan, referred to as RNTCP3, will require early and complete detection of TB, a significant increase of human resources, an expansion of laboratory and treatment centers, a major involvement of private providers, and innovative strategies to deliver care to vulnerable populations.4 1 A list of the team members and their domains of expertise is included in Annex 1. 2 Sahu, S. & L. S. Chauhan. “The role of WHO in the successful implementation and expansion of the DOTS program in India.” Agarwal, S. P. and L. S. Chauhan. “Tuberculosis control in India,” Elsevier, New Delhi, 2005: pp. 187–190. 3 Khatri GR & Frieden TR. Controlling tuberculosis in India. NEJM 2002; 347: 1420-1425. 4 “TB India 2010, RNTCP Status Report,” Central TB Division, Ministry of Health and Family Welfare, March 2010 Evaluation of Tuberculosis Program in India: WHO Report 12 II. PURPOSE OF THE EVALUATION USAID/India contracted Social Impact, Inc. to carry out an in-depth and thorough evaluation of the support provided by WHO to RNTCP using the financial and technical support delivered by USAID. The objectives of the review are to: • Determine the impact of WHO relative to stated objectives and achievements; • Make suitable recommendations for the future direction and priorities of the WHO support to RNTCP. The original statement of work (SOW) called for a comprehensive evaluation of WHO support, considering the appropriateness of the project activities in achieving the objectives, the level of impact, cost-effectiveness, and future directions. Based on an in-country meeting with USAID/India, the focus of the evaluation shifted to address the issue of value for money and answering the question of whether, and to what extent, the resources provided by USAID to WHO to support RNTCP activities have been well spent. Consequently, the focus of the evaluation became much more qualitative and descriptive, rather than a study focusing on the level of targets reached, project performance with respect to effectiveness, efficiency and equity, and impact achieved. The review team examined project trends, investigated the “pull and push” factors that determined the level of achievements, and assessed the quality of implementation. The assessment explored the main successes and lessons learned from the present WHO support provided to RNTCP2 and focused especially on the level of preparedness of the TB services at national, state, district and lower levels to take on the huge challenge of RNTCP3 conceptualization, implementation and monitoring. III. RESEARCH DESIGN AND EVALUATION METHODOLOGY The evaluation team conducted an initial planning meeting with USAID/India and then prepared an overall framework that was reviewed and approved by USAID/India (see Annex 2). The evaluators used a range of methods and approaches for collecting and analyzing the information required to assess the project against the evaluation objectives. METHODS • Desk review of documents (see list in Annex 3) • Attendance of 2 workshops held on Feb 17–18 in Delhi on the following topics: DOTS+ and RNTCP3 • Adaptation of the SOW (a framework was developed and approved by USAID/India; see Annex 2) • Interviews of key informants (see full list in Annex 4) • Validation of the received information • Field visits: Gujarat, Andhra Pradesh (AP), Uttar Pradesh (UP), Rajasthan and Maharashtra • Inventory of activities • Views of authorities (DDG, DTOs, STOs, ex-STOs, etc.) Team Planning Meeting (TPM) Evaluation of Tuberculosis Program in India: WHO Report 13 An initial teleconference facilitated by the team leader was held before the evaluation began. This presented an opportunity for the mission members to present the purpose, expectations, and agenda of the assignment. In addition, the TPM also: • Clarified team members’ roles and responsibilities • Established the timeline, shared experiences and thoughts on the evaluation methodology • Allowed for the exchange of ideas about the data collection tools and guidelines Site Visits and Interviews • A thorough review of the various projects was carried out through site visits and interviews of project staff, and key informants. • Interviewees included key members from all stakeholder groups, including RNTCP, WHO, other donors, partners in TB control, and beneficiaries. • An interview questionnaire was prepared, presented to USAID/India for comments, and updated where necessary; written approval by USAID/India was obtained. • Various site visits were carried out focusing especially on the assessment of specific activities of RNTCP supported by WHO. • The team evaluated the state and district level periodic reports to take stock of the performance indicators. • With respect to the review of the performance of the state WHO consultants, interviews were carried out with the Deputy Director General (DDG), four STOs and twenty consultants. In addition, in-depth interviews were conducted with fourteen newly-hired consultants, twelve consultants with more on-the-job experience, and four former consultants. The in-depth interviews were based on a specially developed questionnaire and on group discussions. The plausibility of the findings was reviewed by the WHO international consultant, and the findings were compared with those of a recent MBA thesis authored by Dr. Lal.5 With respect to the performance of the WHO consultants working at the CTD level, the review is based on interviews of the DDG, and four STOs, and also in-depth interviews of nine CTD consultants. The in￾depth interviews were based on an ad hoc questionnaire and group discussions. The Evaluation Framework The framework of this evaluation focused on a set of questions about the program’s management and operations, along with a set of specific technical questions pertaining to the level of achievement of the WHO-supported program objectives. Details are given in Annex 2. In addition, questions were posed with an aim towards identifying contributing factors and barriers that determine the level of achievement of the objectives, and probing whether WHO occupies any specific niche for providing technical assistance in India. Based on the evaluation, the team prepared a set of recommendations for each of the activities. 5 Dr Lal has been employed as WHO consultant in Kerala; afterwards, he joined WHO as PPM expert, and is presently employed by the Global Fund Headquarters in Geneva. Evaluation of Tuberculosis Program in India: WHO Report 14 IV. FINDINGS PROCESS FINDINGS ABOUT THE MANAGEMENT AND IMPLEMENTATION OF THE PROGRAM Implementation Arrangements, Management and Program Monitoring USAID/India’s support to the World Health Organization has been channeled through a USAID/Washington￾managed grant. Though this implementation arrangement allows the WHO great flexibility in being responsive to changing CTD and USAID priorities, and reduces the management burden on USAID, it does have some drawbacks. First, it reduces the direct connection between USAID/India and WHO’s office in India, which has made the decision-making and information-sharing processes between the two organizations and other stakeholders less transparent. The grant structure also makes determining WHO’s contribution to TB prevention and control efforts in India difficult to ascertain, due to the lack of a formal, high-level monitoring structure. There has never been a Performance Management Plan associated with WHO’s activities. Finally, the grant structure provides little leverage to the donor when critical issues arise regarding WHO’s management of its activities. The most critical issues that the WHO in India faces in managing the USAID-funded TB support is that currently, it is severely understaffed. A single, CDC-seconded medical officer, Dr. Puneet Dewan, supervises more than ninety consultants in addition to preparing donor reports, organizing workshops, and performing other necessary activities to ensure program success. This is not sustainable. It is of the utmost urgency that the WHO hires additional (national and possibly international) senior-level staff to support its efforts. While it is recognized that the WHO hiring process is often slow and arduous, this current situation is unacceptable. Unfortunately, USAID/India has little influence in these matters; thus, USAID/Washington must exert pressure on WHO/India and WHO/Geneva to rectify this matter. One additional issue which should be examined is that WHO’s most important partner for OR, the Tuberculosis Research Centre (TRC) in Chennai, has noted that the transfer of funds from WHO during the last few years has often been inconsistent and delayed. This has led to having to halt and/or discontinue implementation of research projects. Whether this cash flow issue is internal to TRC (see Operations Research section) or is the responsibility of WHO needs to be investigated and quickly rectified. This is especially crucial given the planned scale-up of funding as part of RNTCP3. Core Recommendations WHO/India, in conjunction with USAID/Washington and USAID/India, must develop an organizational chart, along with a timeline and associated actions, to ensure sufficient senior-level staff for WHO/India headquarters is hired by or before the end of 2011. Of particular importance, WHO/India must have at least one professional manager dedicated to providing technical and mentoring support to the WHO consultants. WHO/Geneva should be clearly informed that both performance and future funding decisions by USAID will be made based on its ability to move forward with this hiring. Additional Recommendations 1. While it is not necessary for WHO to develop a formal Performance Management Plan (PMP), a longer￾term framework for monitoring and evaluation should be designed to provide a better understanding of WHO’s output contributions to TB control and prevention efforts in India under USAID support. Evaluation of Tuberculosis Program in India: WHO Report 15 This framework should specifically include output-level baselines demonstrating links to outcome level targets, indicators and benchmarks for WHO’s draw-down plan for its consultants, and specific management performance indicators (i.e. WHO’s ability to sufficiently staff its India headquarters). 2. USAID/India may want to explore the possibility of moving toward a bi-lateral cooperative agreement with WHO, although this may not be possible. Missions in other countries in some instances have been able to negotiate these types of agreements with WHO’s regional office. Furthermore, USAID/India may consider placing the support to WHO under a larger health-systems-strengthening project, delegating the authority over the implementation arrangements to the primary implementing partners. 3. USAID/India and WHO/India need to establish a more formal feedback mechanism for all documents (work plans, quarterly reports, activity reports, etc.) submitted by WHO/India to USAID/W. Currently, USAID/W is providing guidance and support to WHO/India, but USAID/India needs involvement that is more intensive. Monitoring and Evaluation It must be recognized at the outset that the capacity of the public sector to conduct TB M&E varies greatly by state. Some states are still struggling with basic TB reporting, while others have not only mastered the required reporting, but have also strengthened their capacity such that their ability to conduct M&E even exceeds that of the central level. It is within these varying capacities that the WHO consultants must define their roles. In some states, WHO consultants’ primary focus is on basic data collection, collation, analysis and interpretation, while in other states that role has evolved into a quality-check function, and in still other states the public sector is driving the M&E agenda while the WHO consultants focus on special initiatives, operations research, and higher-level analysis. Finally, it must also be recognized that in some states with particularly well-developed M&E systems, it is the public-sector personnel who have the stronger M&E skills in comparison to their WHO counterparts. Yet, despite all of this developing capacity, the process for skills transfer and skills building remains informal and unstructured for both WHO consultants and public-sector employees. Though there have been previous M&E trainings, such as the “Managing Information for Action (MIFA)” courses, there has been insufficient follow-up and refresher trainings to further strengthen these acquired skills. Indeed, in the states visited by the team, there was no observed formal and continuous program for strengthening M&E skills in terms of training or mentoring. The whole process was conducted on-the-job, leaving little time for reflection or long-term planning for sustainable M&E capacity building. In fact, M&E as a discipline has apparently been ignored and relegated to a supporting function within other TB technical areas and efforts. This should cease; M&E must be recognized as a specific technical skill area requiring both initial and refresher trainings. Regardless of these challenges, the data generated is being utilized for decision-making. At the district and state levels, there are numerous examples, including the tracking and analysis of trends in case detection and rates and their correlation with accessibility, and equipment status and its impact on indicators. Likewise, at the central level, generated data and subsequent reports are utilized, though primarily for providing feedback to states on their progress. Further, it must be acknowledged that at the central level it is mainly WHO that drives the M&E process, primarily because of a lack of capacity within CTD. Additional Recommendations 1. WHO needs to hire its own senior-level M&E specialist within its Delhi headquarters and ensure that the CTD has someone (either a public-sector employee or WHO consultant) with sufficient M&E skills to act as a focal point. Once in place, they can then begin to strengthen the M&E skills of CTD staff Evaluation of Tuberculosis Program in India: WHO Report 16 such that the routine reporting, recording, monitoring, and evaluation functions increasingly are managed by CTD staff. WHO consultant support can thus be gradually phased out in this area. 2. WHO headquarters, in conjunction with its state-based consultants and state-level M&E counterparts, needs to develop a formal skills transfer mechanism, including a work plan with specific activities, a timeline with milestones, and supporting instructive activities. While it is recognized that there may be a turnover of staff within the public sector, by developing the supporting documents (including SOPs, M&E Plans, and DQA guidelines) M&E skills will instead be institutionalized rather than residing solely within individuals. 3. WHO’s consultants, likewise, need additional training if they are to assume higher-level M&E functions. While many of the consultants adequately manage these additional responsibilities, there will, in the near future, be a point where the cohort of WHO consultants performing higher-level functions will be sufficient and the demand for higher-level skills great enough such that formalized trainings and a training schedule will be needed. 4. WHO needs to initiate national-level workshops and other forums in which TB M&E is treated as its own specialized technical discipline. These forums would not only allow training for multiple stakeholders at all levels, but would allow also sharing of best practices, innovations, and lessons learned. WHO and CTD may also consider allowing both WHO consultants and public sector employees in one state to participate in the periodic site reviews of other states. OUTCOME AND RESULTS Findings About the Project’s Achievements and Consequences, Conclusions and Activity-Specific Recommendations The objective of WHO’s support is to provide technical assistance (TA) to RNTCP via the following: • Strengthening the Laboratory network for mycobacterial culture, Drug Susceptibility Testing (DST) and introduction of line probe assay (LPA) • Strengthening the involvement in RNTCP of private health care providers, focusing on medical colleges and professional medical societies • Supporting collaborative research activities with the TRC on epidemiological impact assessment, drug resistant TB and HIV-associated TB • Technical support to all RNTCP activities, via the RNTCP consultant network6 Strengthen the Laboratory Network The expansion phase of the DOTS program in India has focused, until 2006, on geographic coverage, including the establishment of microscopy services for TB diagnosis at a decentralized level. However, progress toward the elimination of tuberculosis has remained elusive so far, despite intensified implementation of the DOTS strategy. This 6 This network was originally called, “WHO consultants’ network.” Today, some of the CTD consultants are still “WHO consultants” in the true sense of the concept, while all the others are “RNTCP consultants” At any event, both groups carry out the same tasks. To avoid confusion throughout this report we prefer to call them “WHO consultants” and not “RNTCP consultants.” Evaluation of Tuberculosis Program in India: WHO Report 17 sub-optimal performance is partly the result of insufficient access to advanced diagnostic tests.7 Indeed, for decades little effort has been made to improve techniques for diagnosing tuberculosis; consequently, tuberculosis tests are antiquated and inadequate.8 Fortunately, there has been a recent, unprecedented level of interest, funding support, and focus on the development of new and improved diagnostics such as the nucleic-acid-amplification test (NAAT), of which WHO has endorsed widespread use. The expanded role of laboratories involves the use of more advanced techniques and requires building critical laboratory capacity. USAID/India has provided technical assistance to TB-control efforts via WHO to strengthen the laboratory network for mycobacterial culture, and DST has introduced LPAs and prepared field testing of NAAT.9 The latter test may allow a relatively unskilled health care worker to diagnose tuberculosis and detect resistance to a key anti-TB drug within ninety minutes. Findings Additional laboratory specialists—such as an international laboratory focal point based at WHO/SEARO with seventy percent of work-time allocated for India-related activities—have been in place since February 2009, in order to provide in-country assistance in strengthening laboratory capacity for quality assurance, culture, and DST, and in further expanding laboratory services for the rapidly-expanding MDR-TB treatment program. The national plan for scale-up of the RNTCP laboratory capacity to perform culture and DST was developed in 2009. Laboratory-related technical guidelines for culture and DST were developed and updated, and were revised by CTD in August 2010. WHO Supranational Reference Laboratory (SNRL) staff undertakes frequent field visits to the state-level laboratories in order to supervise, monitor and provide technical assistance to the NRLs. The proposition in FY 2010 that an international laboratory expert from a WHO SNRL should undertake two assignments to India under USAID support did materialize; indeed, external assistance for SLDST capacity building came from Ms. Dorris Hillmann of the Borstel laboratory, who provided TA to TRC and NTI during two separate visits in 2009. Afterwards, another visit from a European microbiologist was felt to be no longer necessary because WHO preferred to send NRL staff to TRC for Second-line DST testing, with the consequence that, at present, all NRLs are trained and accredited for SLDST. By mid-2010, two additional NRLs10 had been established for DST testing in NTI and LRS. These NRLs are of the “stand-alone” type, implying that they are rather small, have limited staff and service support, and can barely share services such as support staff, equipment and supplies—nor do they have a sufficient critical mass of laboratory peers who can share technical expertise. As per the recommendation of the Joint Monitoring Mission (JMM) 2009, establishing a national laboratory task force to accelerate the process of accreditation of culture & drugs susceptibility testing (C&DST) labs across the country has become a priority. This task force is to be based at the National TB Institute, Bangalore—one of the RNTCP national reference laboratories. Key activities will include laboratory design consultation, bio-safety design and assessment of compliance, and training of laboratory staff, as well as on-site mentoring through the accreditation process. This task force was to be in place by June 2010, but this has not materialized due to administrative issues related to the use of 7 Wallis, R., M. Pai, D. Mennes, et al. “Biomarkers and Diagnostics for Tuberculosis: Progress, Needs and Translation Into Practice,” The Lancet, 2010; 375: 1920-1933. 8 Small, P. M., M. Pai. “Tuberculosis Diagnosis–Time for a Game Change,” NEJM, 2010; 363: 1070–71. 9 This is very important, because quality-assured culture and drug-susceptibility testing at the state level will also be a prerequisite for the future availability of treatment of MDR/TB. 10 The four NRLS are TRC, NTI, LRS and JALMA Institute, Agra. They work closely wit the IRLs, monitor and supervise the IRLs’ activities, and undertake periodic training for the IRL staff in EQA, culture, and DST activities. Evaluation of Tuberculosis Program in India: WHO Report 18 funds by WHO and insufficient technical manpower at WHO/India office. This review examined how this delay is affecting the lab strengthening effort. Operationally, WHO managed to achieve the national goals of laboratory accreditation through greater involvement of NRLs, including standardization of training across both key training sites; through TA from Dr. Ranjani (the international laboratory focal point based at WHO/SEARO) directly to IRLs; TA from WHO-RNTCP medical consultants in the field to overcome some of the implementation issues; TA from FIND, including detailed standardized lab assessments, upgrading of facilities, and development of LPA; and also through support from PATH for minor upgrading in a few labs. Because of these joint measures, the handicap imposed by the delay in setting up the lab task force was overcome; RNTCP achieved its targets for laboratory accreditation. In fact, as of June 2011 there will be twenty-seven accredited labs (eighteen government, nine other sector), including ten accredited for LPA and following informed WHO authorities. CTD expects to be on target for the national lab scale-up plan. The protocol for the field evaluation and demonstration of NAAT has been developed; the large-scale field testing was planned to be initiated by the end of 2010. However, nothing had happened at the time of this review. According to high-level WHO authorities, this delay is due to (1) the lab committee’s lateness in granting national approval for the project (in February 2011), which was required to move the project,11 and (2) the lack of senior technical manpower at WHO/India office. The field evaluation at eighteen sub-district level settings will be ready for implementation from mid-2011 onwards.12 Conclusions Although it is virtually impossible to assess the efficiency of USAID support to lab strengthening through direct or indirect economic indicators, this exhaustive review allows the evaluation team to declare that the financial support has been spent in a very effective way with respect to time and cost. With respect to time, important progress has been made in the period reviewed. With respect to the amount of money spent on the various supported activities, especially with respect to the development of the national plan to scale-up the RNTCP lab capacity, and the technical assistance, the progress has been considerable although the money spent was minimal and even some of the allotted money will only be spent now. The choice of the intermediaries for support, such as the focal points in CTD, SEARO, support to TRC, etc., seems to have been a very judicious one for a number of reasons—a major one being that these intermediaries fall under the leadership umbrella of either CTD or WHO. According to the available literature,13 NAAT has high specificity and positive predictive value, but policy development requires that this new tool be field tested before it can be implemented broadly in real-world situations in India. The planned field trials will be of utmost importance. One laboratory focal point in the CTD does not seem to be sufficient for the multitude of tasks to be carried out, and may cause a delay in establishing the remaining IRLs. The question about the overall relevance of the USAID support to lab strengthening of RNTCP is difficult to answer if one looks only at isolated supported activities. On the contrary, one has to look at the broader picture of total support to lab strengthening, of which USAID support is only one component. This global picture allows the evaluators to confirm that RNTCP has significantly strengthened lab support to the program. 11 It seems that there was some reluctance on the part of the NRLs to use an untried technology. 12 This delay has allowed a substantial refinement in the field testing plan, including expansion of the sites through cost-sharing with Bill and Melinda Gates Foundation and other partners using the same protocol. 13 Pai, M., J. Minion, K. Steingart, A. Ramsay. “New and Improved Diagnostics: Evidence, Policy, Practice and Impact,” Curr Opin Pulm Med, 2010; 16: 271–284. Evaluation of Tuberculosis Program in India: WHO Report 19 Core Recommendations 1. USAID support for the expertise at SEARO and CTD should be prolonged and may be increased, so as to allow the creation of a functional, national laboratory task force to support the process of accreditation of C&DST. 2. USAID should allow WHO to spend the remaining allocated sum, within the agreed-upon lab strengthening framework, on the other key lab activity—the gene-expert demonstration—because there is budget inflation due to cost escalations relative to original projections. Additional Recommendations 1. USAID should support the building up of strong laboratory leadership and technical staff through funding of advanced training and regular exposure to international experiences in high-level laboratories. 2. USAID is encouraged to support financially the enrollment of private laboratories in EQA programs and their networking with the IRLs and NRLs. Support of Operational Research (OR) through TRC USAID’s support to TB OR through WHO technical assistance can be considered a major success; there are numerous examples of how the results of the OR have impacted lives in India, either directly or indirectly. For example, it was USAID-funded OR which led to a reduction in the number of samples needed for diagnosis, leading to increased lab efficiencies and, thus, providing labs the opportunity to diagnose more TB suspects. This led then to more persons initiating treatment and, therefore, better treatment outcomes (the greater the number and the earlier patients initiate treatment, the more likely the chances of increased numbers of patients successfully treated). Likewise, the OR done on infection control policies can potentially reduce nosocomial (facility-acquired) TB transmission in hospital and lab staff as well as patients, thereby reducing the number of new TB cases. Other examples of USAID￾supported OR’s influence on policies, protocols and practices include: • Optimizing Public-Private partnerships • Setting standards for External Quality Assurance • Conducting prevalence studies • Providing guidance on TB/HIV policies • Revising the reporting and recording system • Guiding implementation of community-based DOTS • Examining why patients default from treatment • Establishing policies for the introduction of rapid diagnostics These are just a few of the examples of a research agenda that is relevant, expanding, and utilized by international experts. It should be noted that initially, TRC’s OR focused solely on the clinical aspects of TB, but during the last decade has increasingly moved into areas such as socio-behavioral characteristics, economic impacts, and cost-benefit analysis. It can no longer be considered a Model DOTS project, as originally named under USAID support, and must change its name to reflect this expanded role. The current structure for USAID’s funded TB OR begins with funds given to WHO, which then supports TRC to conduct the actual research. This is not to say that WHO has a minimal role; indeed, its role is quite substantial. WHO provides strategic support to TRC and the OR agenda in general by: (1) organizing instructive workshops (for example, on how to write and submit proposals for international support) and coordination workshops (annual OR Evaluation of Tuberculosis Program in India: WHO Report 20 agenda setting meetings); (2) screening and reviewing TRC’s proposals; (3) mentoring and providing technical assistance to TRC on cutting-edge methodologies; (4) facilitating the OR collaborative process with medical colleges and other institutions through its WHO Consultants Network; and (5) providing forums, within which results can be analyzed, interpreted, and disseminated. Despite all of these successes, TRC is currently at a crossroads that could determine if it will continue to achieve similar results and successes as previously. It faces a number of challenges it must address—some critical—to ensure the successful future implementation of the OR agenda. These include: (1) resolving staffing issues; (2) determining its future role in OR, both in India and internationally; (3) remedying a lack of collaborating partners in India; (4) shoring up weak coordination of research centers in India; (5) ensuring that its research designs and methodologies remain up￾to-date; and (6) facing possible cash flow and cash management issues. Most critical of all of TRC’s present challenges is staffing. For the past several years, TRC has been understaffed and a large cohort of senior managers, researchers and scientists will be retiring shortly. Few of the mid-level staff can replace them; instead, previous retirees are brought back to TRC on short-term contracts to continue providing senior-level guidance. Given these impending retirements, there will be a large loss of not only technical knowledge and expertise, but also of mentors for junior-level staff. The institution and its ability to conduct high quality OR potentially may suffer. While WHO and USAID are constrained in how much leverage can be brought to bear on the government of India’s bureaucratic process, some attempt must be made to rectify this situation. If these staffing and other issues can be resolved, then TRC will need to address is its future role, both within India and internationally. Historically, most TB OR has been centered within TRC and other institutions, such as the National Tuberculosis Institute (NTI, Bangalore) and, more recently, the L. R. S. Institute of Tuberculosis and Respiratory Diseases, New Delhi (LRS). Recent challenges at NTI have reduced its OR role, and LRS only recently has become fully engaged in OR. However, there is a plethora of field-based research being done by other partners— such as PATH and the Union—which could provide a great variety of experiential data that TRC could analyze. Currently, almost all of TRC’s OR is focused on one district within the state of Tamil Nadu which, because of its specific socioeconomic, demographic, and epidemiological characteristics, limits TRC’s ability to generalize results. Given the large investments that have been made in TRC, this seems an opportune time for TRC to move into a mentoring function. In other words, it needs to position itself to assist fledgling TB research institutions and other partners within India, and help them develop their own OR capacity. Furthermore, TRC needs to start establishing itself as a true, international research institution. Currently, while it often collaborates with other international organizations, it is most often in the position of a junior partner in these efforts and/or provider of field data. This, in the long-term, is detrimental for TRC. USAID can support this process by ensuring in the short-term that TRC staff has adequate opportunities (and funding) to attend and participate in both state-of-the-art conferences and trainings, such that TRC’s experiences are presented to global audiences and submitted to international peer review. Long-term, USAID can require that funded partners give TRC, as much as feasible, the technical lead and conduct periodic, external reviews of this process to guarantee that there is a true transfer of skills and promotion of TRC as a leading agency. However, if TRC is to move into a more internationally renowned position, it must ensure that the quality of both its research and its underlying methods remains state-of-the-art. While TRC is producing reliable results, often its research uses epidemiological and other methods that are no longer the most current. Improvements in study design and advanced statistical methods now allow for quicker studies and smaller sample sizes (and subsequently, less expensive and of shorter duration), which nevertheless provide results of equal (or even higher) validity. TRC, with WHO in a mentoring and reviewer role, needs to update its efforts in these areas. Finally, as mentioned previously in reviewing the WHO’s management of the overall TB control efforts, there may be an issue with the flow of funds to TRC or within it. During the site visit to TRC, it was noted by several researchers that tranches of funds were at times delayed for long periods of time. Consequently, research that was underway had to be suspended or halted. These types of managerial delays can invalidate results and lead to greater overall costs. The WHO must investigate this issue immediately and identify suitable remedies. Evaluation of Tuberculosis Program in India: WHO Report 21 Core Recommendations 1. WHO must increase its oversight role with regards to TRC, focusing especially on issues of staffing, cash management and cash flow, and improvement of research methodologies. While it has been recognized previously that WHO and USAID exert limited influence on TRC’s ability to hire people, both organizations working together can bring pressure on the government of India to move forward with the additional hires. Furthermore, in the process of recruitment WHO can play an important function in ensuring that potential candidates are of the highest caliber by participating in the screening and interview process. WHO must also work immediately with TRC to address its cash flow issues and prepare a brief report for USAID on the fundamental issues and how they will be resolved. Finally, in its ongoing role of mentor, WHO must review and then improve TRC’s epidemiological methods through workshops, advanced seminars, distance learning, or another effective modality. All of these recommendations are, of course, predicated on WHO having sufficient staff (see WHO Management section). 2. Given current human resource constraints within WHO, if WHO is unable to provide the increased oversight role, per Recommendation 1, USAID should either request that WHO, or itself directly, provide TRC with technical assistance in the areas of financial supervision, human resources development, and general management practices. This can be achieved by either providing intermittent technical assistance (via a WHO partner or, perhaps, a pre-existing USAID management consulting partner) or by issuing a separate agreement to establish a partnership program with a similar organization that not only has the technical skills but, more importantly strong management practices.14 WHO/India and USAID should explore the possibility of creating a broader forum of research institutes in India for conducting TB OR. These centers could then develop and carry out multi-center and multi￾cluster research in the various ecological niches of the country. This approach might have a number of advantages, including: (1) creating a stronger expertise base; (2) reducing variance between geographical areas and, therefore, reducing needed sample sizes; (3) improving the ability to generalize the findings; and, (4) increasing national ownership of research results. Additional Recommendations 1. WHO and TRC must collaborate in developing a long-term strategy for TRC for its future function within India and internationally. This strategy must address how TRC will build its own capacity first, such that it can move into a mentoring role within India and then, finally, as a true international partner. In addition, it must identify which organizations it considers to be the most promising potential partners, with a specific focus on utilizing data and results from areas outside of Tamil Nadu. 2. USAID, WHO, and TRC must rename the USAID-funded work to reflect its current agenda. Model DOTS is no longer appropriate. 14 For example, in Latvia the WHO Collaborating Centre for Research and Training in Management of MDR-TB first was partnered with CDC to develop its technical skills; once the skills had reached a high-level of proficiency, the center was then partnered with a university hospital (University of Arkansas for Medical Sciences) to focus on developing stronger management practices, including business, marketing, and research plans. Evaluation of Tuberculosis Program in India: WHO Report 22 Engage the Private Sector in TB Control The private sector in India is vast and heterogeneous and remains the primary source of health care for the majority of households in both urban and rural areas. It is essential that all actors of the private sector become effectively engaged in TB control, especially now that RNTCP3 intends to provide universal access to TB care and decrease pre-treatment delay. The total pre-treatment delay consists of two components, patient delay and health system delay, and might last almost one year (between 23 to 321 days).15 Long health system delays were observed when private providers were consulted first.16 The RNTCP has made several attempts so far to engage the private health sector in general and private practitioners in TB control in particular.17 Vast numbers of private providers (PPs) across the different states are affiliated with the Indian Medical Association (IMA, see Annex 5) under the support of GFATM. Medical colleges with 289 hospitals are providing TB services. Presently, there are 2,500 NGOs and over 30,000 PPs involved in TB care. However, these impressive numbers comprise only a minuscule fraction of the large private sector in the country, and their precise contribution to TB control as yet is not fully known. A study carried out two decades ago in Mumbai revealed the extent of inappropriate tuberculosis management practices among private practitioners18 and stimulated national efforts to educate and engage the practitioners in TB control. A 1998 follow-up study19 showed that the average patient generally did not take the drugs as prescribed by their doctors and the treatment completion rates were rather poor. A recent study carried out in the same area noted that, unfortunately, little seems to have changed over time.20 Although several pilot projects have demonstrated that PPM is feasible in TB control and that it has the proven potential to increase RNTCP performance, field observations made during this review mission have identified that up￾scaling is constrained by a series of factors, including the following: • Current system of supervision is unpalatable to PPs. RNTCP relies heavily on intense supervision to guarantee the quality of diagnostic and treatment practices. A special cadre has been created, called the senior treatment supervisor (STS) and the senior TB lab supervisor (STLS); both are non-doctors, and their supervision is not acceptable to PPs. • Private sector apprehension related to financial viability, due to reported delays in payment of remunerations/incentives to physicians participating in PPM • PPs remain reluctant to sign MoUs with the GoI, as is evidenced by the participation of an important number of PPs in PPM that have not signed an MoU • Limited use by the PPs of the “special sputum collection and transportation centers” created by RNTCP • PPs feel that the required recording and reporting of DOTS activities provide no additional value to their work • PPs’ contributions to referrals are poorly documented • Currently, various STOs or DTOs do not report private sector performance 15 Bawankule, S., Q. Zahiruddin, A. Gaidhane, N. Khatib. “Delay in DOTS in new pulmonary TB patient from rural area of Wardha district, India,” Online Journal of Health and Allied Sciences, 2010; 9 (1) : 1–7. 16 Rajeswari R., V. Chandrasekaran, M. Suhadev, et al. “Factors Associated With Patient and Health System Delays in the Diagnosis of Tuberculosis in South India,” IJTLD, 2002; 6: 789-795. 17 Central TB Division. “Directorate General of Health Services,” Ministry of Health and Family Welfare, 2008. Available at . 18 Uplekar, M. W., D. S. Shepard. “Treatment of Tuberculosis by Private General Practitioners in India,” Tubercle, 1991; 72: 294–290. 19 Uplekar M., S. Juvejkar, S. Rangan, P. Nunn. “Tuberculosis Patients and Practitioners in Private Clinics in India,” IJTLD 1998; 2: 324-329; 20 Udwadia Z. F., L. M. Pinto, M. W. Uplekar. “Tuberculosis Management by Private Practitioners in Mumbai, India: Has Anything Changed in Two Decades?” PLoS ONE 2010; 5(8): e12023. doi:10.1371/journal.pone.0012023. Evaluation of Tuberculosis Program in India: WHO Report 23 • It was observed that some constraints are the result of the attitude of the public sector itself, corroborating the findings reported by the JMM21 – Engagement of the private care providers does not seem to be a high priority 2009 mission: – The available staff at all levels have little capacity to undertake PPM activities seriously – The state and district expenditures on PPM are very low – The name of the referring private provider is rarely recorded and feedback is seldom sent – Anti-TB drugs are widely available through private pharmacies without prescription. Based on these observations, it might be concluded that the key conditions that underlie the potential success of PPM hinge upon the clear understanding and delineation of the roles, responsibilities and accountabilities of both public and private sector actors in a true spirit of partnership, equity, risk-sharing and transparency. An effective intervention package should therefore include the following provider-side components:22 • Improved referral and information system that uses simple, practical tools • Adequate monitoring and supervision of the PPs • Provision of free anti-TB drugs to patients treated in the private sector • A strong commitment on the part of RNTCP to supporting, supervising and evaluating PPM projects • Use of NGOs and/or medical associations as intermediaries The WHO has supported the PPM initiative from its inception in 2003 in fourteen major Indian cities, and has adopted a systems approach for the phased scale-up of PPM activities that incorporates local mapping of all care providers in a systematic manner, prioritizes the providers most likely to contribute to TB case notification and treatment, agrees on the provider-specific task mix, establishes a surveillance system to measure the contribution of the different providers, and implements and evaluates the results.23 WHO also contributes to PPM through its consultants’ network. The consultants guide the PPM implementation to the extent of their capacity, and their efforts have contributed significantly to the growing success of PPM. However, many challenges and barriers remain. Recommendations Given that PPM involvement in RNTP3 will need to be much bigger and more intensive than in RNTCP2, those challenges have to be met,24 and to do so, the following actions are recommended: Core recommendations 1. Efforts should be made to make the partnership with PPs mutually beneficial,25 21 Joint Monitoring Mission RBTCP, India; 15-28 April 2009. SEA–TB 321. which requires exploring what a “win-win” situation would look like for the several groups of PPs. 22 Lönnroth, K., M. Uplekar, V. K. Arora, et al. “Public-Private Mix for DOTS Implementation: What Makes It Work?” Bull. WHO, 2004, 82: 580–586. 23 Lal, S. S., S. Sahu, F. Wares, et al. “Intensified Scale-Up of Public-Private Mix: A Systems Approach to Tuberculosis Care and Control in India,” IJTLD 2011; 15: 97–104. 24 Bhatia, V. “Enhancing Private Sector Contribution to TB Care in India,” Jan 2010. GFATM, p. 24. 25 Unfortunately, this review mission offered insufficient opportunity to hold extensive in-depth interviews and focus groups with representatives of the different types of private practitioners operating in the various geographical regions and socio-cultural networks of the country. These would have allowed us to explore what a “win-win” situation would look like for each of these groups, what monetary and non￾monetary expectations they might have with respect to their active participation in PPM, how much time they would be willing to spend on PPM, the maximal administrative burden of reporting and quality control measures they would be willing to accept, and what the terms of an acceptable MoU would look like. Evaluation of Tuberculosis Program in India: WHO Report 24 2. Via the consultants’ network, WHO should indicate any special needs at the state and district levels for the transition from RNTCP2 to RNTCP3, and state what support the private sector could offer to bridge those gaps. 3. WHO and CTD should develop an effective system to supervise PPs who participate in PPM that is guided by a concern for continuous education and motivation, and is acceptable to the PPs. Action-research initiatives are needed to develop and optimize such a supervision system. 4. There is a need to adapt the current surveillance system so that it is able to capture data on referral practices by PPs at state level; and include the performance of PPs in TB control as an integral part of every quarterly report at district and state level; 5. USAID should continue and even increase its financial support to RNTCP via WHO, in order to allow CTD to successfully implement RNTCP3. Additional recommendations 1. Organize specialized PPM training for the WHO consultants. Every state requires an approach customized to its resources and specific challenges. 2. WHO should provide TA to the STOs to map the private practitioners in all states, not only focusing on their number and localization, but also on their strengths and core competences. 3. WHO should support, financially or otherwise, local initiatives, because it has been shown that local initiatives are key to the success of PPM initiatives. 4. The IMA should continue its pilot project to find the best way for PPs to refer their clients. WHO has supported a pilot project on sputum collection and transport in Lucknow. This pilot should be continued and expanded to other states. Private labs should be developed as an alternative to public sector microcopy centers in order to serve PPs who remain reluctant to refer their patients to government centers. WHO Consultants’ Network Many factors have contributed to the successful implementation of DOTS in India. One important factor has been the use of WHO-contracted local consultants to provide technical support at CTD, state and district levels.26 Eighty￾seven field-level consultants work closely with the district and state TB officers; an additional ten consultants provide technical support to the Central TB Division. This network of advisors, based on experiences gained from the polio eradication program in India, started in August 1999. In the beginning, it was sponsored by WHO and the Canadian International Development Agency (CIDA); once CIDA withdrew its support, USAID and DFID stepped in to guarantee continuity. DFID will withdraw its financial support to the network in December 2011. After a decade of field experience, the consultants continue to serve an important role, linking state and district staffs with national and international guidelines and supporting good TB practice, while at the national level, consultants 26Khatri, G. R., and T. Frieden. “Controlling Tuberculosis in India,” NEJM 2002; 347: 1420–1425. Evaluation of Tuberculosis Program in India: WHO Report 25 play a lead role in carrying out the mission and core functions of CTD. Given important differences in their ToRs, the state level consultants and the CTD level consultants are discussed separately below. State Level Consultants Findings At present there are eighty-seven consultants working at state level; some concentrate on basic DOTS implementation in a series of adjacent districts, while others focus more on specific themes. • Administrative dependence: Prior to September 2008, WHO consultants, although never serving as actual WHO staff, served under the umbrella of, and were directly hired by, WHO. This association with WHO stimulated their performance, provided access to senior decision-making and also gave their recommendations authority and credibility. Since Sept 2008, the administrative hiring and management of the consultant network has been outsourced to a firm called Strategic Alliance (SAMS). This was the consequence of WHO-HQ’s concern about the possible legal liability of engaging hundreds of public health professionals in India under a special scheme, called “SSA” employees. • Presently, the WHO consultants are project-specific employees of SAMS, which is responsible for compliance with all statutory obligations with respect to provident fund (PF) and tax deduction at source (TDS); specifically, SAMS is responsible for leave, salary and benefits administration, payroll and claims processing, activity reporting, triangulation of claims, district visit monitoring, and technical and financial assurance review. SAMS also provides centralized travel management and arranges biannual meetings and conferences. However, WHO continues to provide day-to-day technical direction and guidance to the network. • Recruitment is based on strict eligibility criteria (MBBS and field experience of one to two years, at least). Age does not seem to be a selection criterion. Short-listing happens through a series of consecutive interviews, leading to the decisive technical interview by experts from the office of the WHO-Representative (WHO-WR)/Delhi. Longer field experience and public health training are not selection criteria; however, both increase the chances of selection. This procedure has been maintained from the start of the network and has maintained its objectivity and transparency over time. • Profile:27 However, of the consultants who joined recently, more have postgraduate training in public health than the previous batch. Indeed, of the forty-two consultants who were in service in Sept 2008, eighty percent had postgraduate training in public health, while of those who have joined since Sept 2008, ninety-one percent earned a postgraduate degree. The current age of recent consultants is thirty-one; thirty-two for those that joined since August 2008, and thirty-six for those already employed in August 2008. Taking age as a proxy for experience, this suggests that the consultants have less professional expertise than previously. This is proven by their prior work experience, which decreased from seventy to fifty months. Becoming a WHO consultant seems to attract mainly males: as of Sept 2008, eighty-three percent of the consultants were males, and ninety-three percent of the new recruits are males. 27 The sole data available are those collected by SAMS; their earliest data available are those of the consultants taken over by SAMS when the management was outsourced in August 2008. If the data of age and experience, at the year of joining the network had been available, then the eventual shift in the age and experience profile would have been easy to determine through a cohort analysis. Evaluation of Tuberculosis Program in India: WHO Report 26 • Preparation for the job: There is an initial induction training on technical and operational aspects of RNTCP, including how to carry out supportive supervision and provide adequate feedback; this inductive training also aims to enhance skills related to different aspects of management, use of computer data, data analysis and interpretation, writing and presentation skills. The training lasts three weeks, and is preceded or followed by a “shadowing” period in the state of employment (depending on the availability of a senior consultant); however the shadowing did not always occur, because some consultants were asked to start work immediately. Observation in the field has shown that the short duration of “induction training” does not allow trainees to cover in-depth the whole spectrum of skills they need, and that several consultants have insufficient skills in the domain of evidence-based program management, finance and budget management, ACSM, PPM and/or OR. • ToRs: Written ToRs existed from the start of the network. In the initial years, the ToRs were limited to core functions only; they became more elaborate afterwards and even included the concept of reporting of activities to the STO, which created a conflict of authority in some states. Reporting to the STO was removed again some years ago, and presently the ToRs are restricted to the components of the Stop TB strategy. At present, the ToRs are rather general in nature and broad in scope, allowing the individual consultants to put quite a lot of creativity into their implementation. • Job implementation: The consultants are assigned to specific states, and work closely with state and district TB officers. At the launching of RNTCP in a given state, the consultants assisted in the preparation of the action plans and commencement of the DOTS services. Once the basic DOTS services had been implemented, the consultants then enhanced the capacity of the district health system for supervision, monitoring and evaluation. Presently, they assist in data management and electronic transmission of quarterly surveillance data to the national level, improve record keeping and monitor the consistency and accuracy of the quarterly cohort data, and support supervision, monitoring and evaluation of the DOTS implementation at district and state levels. They provide feedback on RNTCP performance to the national level.28 The job duties in the field generally follow the ToRs, although occasional deviations towards other tasks that routinely correspond to STC staff were reported by some consultants. Many STOs rely heavily upon those consultants and delegate various responsibilities to them; for example, some WHO consultants mentioned that they are heavily involved in routine supervision activity itself, rather than providing support to routine supervisors. Recently, some consultants tasked with providing thematic support to PPM, ACSM, OR, or HIV-TB programs. The interviewed DDG, STOs and DTOs expressed full satisfaction with the high level of engagement of the WHO consultants in RNTCP and the quality and seriousness of their work. Without their involvement, RNTCP would not have reached the high level of performance it has now. • Relationship with state WHO consultants: Apparently, there are few links between the CTD consultants and those working at state level. • Supervision & performance appraisal: Various modalities have existed over time. In Dr. Tom Frieden’s era, there was very close supervision with personal visits and regular telephone contacts. Afterwards, the supervision became limited to the monthly report and the end-of-assignment report with regular feedback, the nature and frequency of which depended on the available WHO staff. In recent months, routine technical 28 Sahu, S. and L. S. Chauhan. “Role of the World Health Organisation in the Successful Implementation and Expansion of the DOTS Program in India.” In: Agarwal, S. P. and L.S. Chauhan. “Tuberculosis Control in India,” Elsevier, New Delhi, 2005: 187–190. Evaluation of Tuberculosis Program in India: WHO Report 27 supervision seems to have become less regular, due to the fact that the position of National Program Officer (NPO) of WHO is not yet filled. In AP and some other states, the consultants organize a monthly meeting that serves as an opportunity for peer review of their activities and also for updates on administrative and technical issues. • Job satisfaction: The perceived level of job satisfaction appears to be high. Employment as a “WHO consultant” is felt to be conducive to professional development, personal growth under the technical guidance of high-level international WHO experts, and development of managerial and diplomatic skills. The level of job satisfaction seems to be influenced by the umbrella of the “WHO” brand. The recent change of the title, from “WHO consultant” to “RNTCP consultant” seems to be a serious concern affecting perceived authority and credibility. It not only provokes uncertainty about their status and future, but has a negative impact on their level of job satisfaction, and could have a negative impact on job stability and retention. The decreased age at recruitment and the big turn over could be an indication of a diminishing of the job’s appeal. • Transfer of knowledge & skills to STC and DTC staff: The obligation for such transfer is not part of their ToRs, and consequently there are no transfer targets, and no guidelines or benchmarks. However, the consultants are aware of the need to transfer their knowledge and skills to someone in the health system. Some spontaneously mentioned having the opportunity to transfer skills to the STO as well as the medical officer (MO) state TB cell; others refer to transfer of skills to selected staff from the state, such as trainers from the medical colleges, etc. • Career development: Guidance or efforts towards professional development from the National WHO office were at some times sporadic or even non-existent. Until recently, little or no career development concerns were shown by the National WHO office, as mentioned by several interviewees. According to one interviewee, “No efforts were made for career or professional development or at least to provide guidance to the consultants. In spite of our requests, no steps were taken to provide link ups with international organizations or universities. Nor were there any career opportunities with TB programme above the level of consultant. Hence the interest would stagnate after a period of time.” All interviewees agreed that they expect their employer to be attentive towards their individual career/personal development and to show that concern through granting personal development initiatives, such as allowing them to assist during professional meetings, encouraging and guiding them to publish their experience, etc. • Remuneration: Until recently, the salary and other benefits were substantially higher than what they would have received as GoI staff, but the present lack of a competitive salary scale is reported to be a serious concern, which could jeopardize their sustainability on the job. Conclusions • Contribution to RNTCP performance: This network has contributed significantly to RNTCP’s positive performance. It proved to be an element of stability and continuity within the context of a frequently￾changing cadre at the state level; guaranteed the quality of program implementation; facilitated introduction of novel strategies; and allowed maximum adaptability of the new strategies to the realities on the ground. In those states where the conditions were suboptimal, the presence of WHO consultants has allowed a reasonable level of performance; in those states with excellent RNTCP performance, the WHO consultants provide TA, facilitate the training of district personnel, contribute to improving data quality, strengthen Evaluation of Tuberculosis Program in India: WHO Report 28 evidence-based program management, and introduce novel methods and strategies. If the consultants would had not been there, the case finding and case holding targets would have been achieved at a much slower rate. • Role in supervision of RNTCP: On-site supervision by specialized staff from outside the program (WHO consultants) has been—and still is—a pillar of RNTCP’s success. Although consultants in some states are involved in field supervision, routine engagement in field supervision is not part of their ToRs; they are supposed to give only support and TA to the supervisory cadre of the district (DTO, STS, STLS). This task is essential to maintaining the quality of the supervision and follow-up on the corrective actions taken. • Administrative cadre: The recent change in title (from being “WHO consultant” to “RNTCP consultant”) is a crucial issue and has important implications for the performance, perceived authority, and credibility of the consultants. It may affect their performance, and even the global performance of RNTCP2, and certainly of RNTCP3. • Comparative advantage: Most of the interviewees are of the opinion that the WHO consultants’ network has a definite impact on quality—especially in the areas of data management, training, and continued capacity building. Without the network, routine surveillance reports would have been less reliable. The urgency behind and motivation for pushing the program forward and ensuring its success would have been substantially weaker without these consultants. RNTCP performance in various districts certainly would have been much lower if the consultants had not been there. DOTS Plus program implementation definitely would have been hindered if the consultants had not provided critical support. Historical data have shown that preparatory activities for DOTS implementation was fifty percent faster in districts that were supported by a WHO consultant than in those districts that lacked such support.29 • Value for money: The financial investment in this network is low and the routine expenses are limited. However, the cost-benefits for the program are tremendous. Although this evaluation did not receive exact financial data to calculate cost-benefit ratios, and to estimate the number of TB deaths averted through the involvement of the WHO consultants and the number of secondary cases avoided, it can be asserted that the cost-benefit ratio is very high and is probably one of the areas with the highest return for money invested. – Where states suffer from poor and/or unstable administration, the WHO consultants guarantee stability and quality in RNTCP implementation. – In states with good STC management, the consultants are still quite important, because they provide technical assistance, assist the DTOs to function correctly, facilitate training, play an important part in guaranteeing the quality of routine surveillance data, contribute to advanced analysis of those data, and introduce novel strategies (such as 2/230 – At the national level, WHO consultants have an important task in helping to develop new strategies and adjusting guidelines to field realities. ). • Exit Strategy: Providing external support to RNTCP for more than ten years raises questions about its rationale and sustainability. Stationing eighty-seven WHO consultants at state level has to end some day, but the key questions are when, and how? Various withdrawal options are possible: 1. Sudden withdrawal: Analysis of the field situation in various states has shown that RNTCP is not ready for a sudden and definitive withdrawal of the network; this would negatively affect the performance of 29 Khatri, G. R. and T. Frieden. “Controlling Tuberculosis in India,” NEJM, 2002; 347: 1420–1425. 30 This policy refers to the reduction of the screening duration of cough from three weeks to two weeks, and the reduction of the number of direct sputum examinations for diagnostic purposes from three to two (thus previously the “3/3 policy” was adhered to and presently it is the “2/2 policy”). Evaluation of Tuberculosis Program in India: WHO Report 29 RNTCP, given the enormous challenges that are likely to be presented by RNTCP3 conceptualization and implementation. 2. Gradual withdrawal: Any withdrawal has to be gradual; this means that the number of the consultants in each state has to decrease gradually, as the given state prepares to take over the consultants’ functions. At present, a gradual withdrawal has already taken place in states, like Kerala,31 3. Several other exit strategies could be imaged, such incorporating the WHO consultants into the STC structure, or into the state level administration, or redesigning their location, or keeping a minimal pool of state-level consultants attached to either WHO alone, or to a consortium of donors. Details are given in the footnote. but this withdrawal took place in the context of RNTCP2, which is less complex than RNTCP3. Although this option looks appealing, it implies that the total mass of RNTCP expertise risks a decrease; the timing of such a decrease is very unfortunate, since the launch of RNTCP3 will create more demand for high-level creative thinking and piloting novel schemes. 32 Recommendations • Future directions until 2017: Although the RNTCP should become rapidly self-sufficient with respect to RNTCP2 implementation of “basic DOTS,” the WHO consultants’ network at state level should continue, at least until the end of the launching of RNTCP3 in 2017. Additionally, an increase in the number of consultants should be considered in order to: – Assist RNTCP in solving the problem of the ninety-seven underperforming districts – Initiate RNTCP3, focusing especially on ensuring universal coverage, contact tracing, decrease of treatment delay, and involvement of private practitioners – Optimize evidence-based program management – Reinforce the MDR/DOTS Plus implementation The evaluation team recommends that USAID continues to support the WHO consultants’ network, and should consider taking over the total cost once DFID has withdrawn its support after Dec 2011. • Future directions beyond 2017: A reduction of the number of consultants might be considered after 2017; however, the withdrawal of the WHO consultants’ network should be gradual and its intensity dictated by the level of RNTCP performance at state level. 31 In 2006 there were four WHO consultants In Kerala; today there is only one, but the Basic DOTS implementation indicators remain good. 32 The option of incorporating the consultants into the TB control program of the state, to which they are presently attached, makes them administratively dependent on the State TB control officer (STO). Their salary will be paid by GoI. This option would maintain the indispensable expertise at state level, but make the consultants subject to the administrative rules and habits that are prevalent in the GoI sphere. Various comparative advantages of the present system risk getting lost. A second option would relocate the consultants at zonal or even national level, and make them administratively no longer dependent from the State TB program but from the central level TB control program (CTD). Their salary would be paid by GoI. In this scheme, seniority and expertise could be leveled up, producing better career development perspectives. However, some comparative advantages of the present system risk also getting lost. A third option would keep them at state level, but attach them to the higher-level administration. Their salary would be paid by GoI. This option would maintain the expertise at state level, increase the pool of technical knowledge, and facilitate decentralization of the RNTCP, but make the consultants subject to the administrative rules and habits that are prevalent in the GoI sphere. A fourth option would maintain a reduced number of state-level consultants attached to either WHO alone, or to a consortium of international donors. Their location could be zonal or national. Their contract should reflect this special dependence and be attractive financially and career development-wise. This alternative suffers from the lack of long-term sustainability, but could give the necessary high-level support to RNTCP3. The latter will create enormous demands for the leadership of RNTCP, and needs very high level expertise, especially because there is no model yet for the conceptualization or implementation of RNTCP3. Evaluation of Tuberculosis Program in India: WHO Report 30 In order to increase the sustainability of RNTCP3 after 2017, a concrete plan to institutionalize the transfer of knowledge and skills should be developed by WHO, CTD and the STOs; this plan should contain defined areas, such as data and program management, implementation practices, drug supply, capacity building, etc.,33 and precise benchmarks for each of the states. Most consultants interviewed are in favor of a gradual handing over of tasks to state-level authorities, after due theoretical and practical training. Once such a cadre has been trained, the consultants should then become engaged in new tasks, in line with the priorities decided by CTD. The responsibility for the remaining WHO consultants is an important question to be considered by the next Joint Monitoring Mission in 2012. • Modalities of WHO consultants’ involvement in RNTCP3: 1. The TA to RNTCP3 implementation will require a very solid level of expertise in disease and program management, which implies that major efforts should be undertaken to retain experienced consultants. Measures such as restoring the WHO umbrella and creating optimal working conditions are very much encouraged. 2. The present diversification of their role in support to districts and thematic support should be maintained. 3. Relocation of the consultants towards the STC headquarters could be considered, as it allows easier exchange of experiences and peer review of personal performance. 4. To optimize these very valuable resources, consider extending coverage area to include adjacent states (or assign consultants “zonal coverage”). 5. The line of command should remain national, but the reporting-information flow towards the STOs should be reinforced. 6. More emphasis should be put on building consultant capacity, especially in financial management, OR, advocacy, PPM, MDR/EDR management and evidence-based program management. 7. Strengthen supportive supervision of the network, accompanied by detailed feedback. 8. Strengthen the M&E of the network. 9. Necessary attention should be given to the career development of the individual consultants. The possibility of moving into CTD-level WHO consultancy positions should be promoted, but this should happen solely on merit. 10. The creation of a mid-level career structure for the WHO consultants could take care of some of these career-development concerns, and also could partially solve the problem of the decreasing age of the newly-selected consultants, as each senior consultant could coach one or more of the junior consultants. • Focus of RNTCP3: For RNTCP3 to be successful, it needs a broader focus than RNTCP2, as one consultant mentioned: “RNTCP has had a very focused and limited approach, but it can only be successful when the primary health care system is fully functional. Unless the focus broadens, limiting activities to only TB, out of the range of services which the PHC would provide, would limit its success.” WHO consultants therefore need to broaden their focus from a purely “vertical disease approach” to a fully integrated and horizontal approach. 33 All areas related to the STOP TB Strategy Evaluation of Tuberculosis Program in India: WHO Report 31 CTD-Level Consultants Findings At present there are ten WHO consultants working at the Central TB Division (CTD). They are only engaged in thematic domains, such as ACSM, HR, PPM, M&E, etc. • Recruitment: The procedure to recruit and hire consultants remains objective and transparent. There is an initial call for candidates through various means. Those who respond to the selection criteria are short-listed; further short-listing occurs through phone interviews, followed by technical interviews with CTD-WHO staff. No evidence was found to support rumors that recruitment has become less stringent than in the past. • Administrative dependence: Prior to September 2008, WHO consultants, although never serving as actual WHO staff, operated under the umbrella of the WHO and were directly hired by the WHO office. This association with WHO facilitated their performance by providing access to senior decision-making and gave their recommendations in the field both an authority and credibility. Today, only half the CTD consultants are still engaged by WHO, although recently some were selected and are administratively managed through a firm called “Strategic Alliance” This fundamental difference in administrative designation is a source of frustration for the “unlucky ones” who are not employed under the auspices of WHO. • ToRs and the daily work: The ToRs are very succinct. The distribution of daily activities is not driven primarily by the ToRs, but by day-to-day requirements. • Focus of the work: thematic. • Job security: Their contracts are short term (only months instead of years), and some are uncertain that their contract will be extended. This creates extra stress, of course. • Work culture: The consultants feel the pressure to work long hours (sixty to seventy hours per week), remaining on the spot until the boss leaves, and they rarely take their full amount of leave days. • Supervision of CTD consultants: The consultants report to the DDG on a day-to-day basis, and are supervised by the CMO, to whom the CTD consultant is attached. Presently, there is no regular supervision by WHO, due to absence of a NPO. • Performance appraisal: Carried out by the NPO (or his replacement) on a day-to-day basis, but no formal professional assessment occurs, and consequently there is no possibility to obtain reward for excellent professional achievement. • Job satisfaction: is very high, due to the feeling they have of serving a higher-level purpose. • Remuneration: There is a feeling of being underpaid for the level of expertise that is expected from them. • Accountability: All WHO consultants expressed their feeling that they are held accountable for the technical assistance they provide. • Career development concerns: The contract does not contain any capacity-building option. There is no “employer-employee” relationship; no professional growth opportunities are provided, nor is there an opportunity for advanced training in a chosen thematic area. The interviewees gave several practical examples Evaluation of Tuberculosis Program in India: WHO Report 32 of situations in which the consultant had to take personal leave to attend a conference at which their accepted paper was to be presented. There is no opportunity for exposure to WHO experience, neither at regional nor global levels. There is also limited opportunity to publish professional experiences. Conclusions • Contribution to RNTCP performance: This network of CTD consultants significantly contributes to the RNTCP’s positive performance. The evaluation team received the impression that CTD consultants are employed more as specialized staff of CTD, and less as consultants. The thin line between employment in routine activities or as technical advisor to the technical staff of CTD seems to have been blurred in more than one instance. • Administrative cadre: The dual administrative cadre creates frustration and could affect the smooth working relationship between the various members of the CTD consultants’ network • Comparative advantage: DOTS Plus program conceptualization, planning and monitoring certainly would have been hindered if the consultants had not provided critical support to the CTD. • Value for money: The same comments can be made here as for the state WHO consultants: “The financial investment in this network is very low and the routine expenses are limited. However, the cost-benefits for the program are tremendous.” Even without entering into economical return calculations, it can be asserted that the cost-benefit ratio is certainly very high and is probably one of the areas with the highest return for money invested. • Exit strategy: Providing external support to RNTCP for over ten years raises questions about its rationale and sustainability. Stationing ten WHO consultants at CTD level has to end some day, but the key questions are when, and how? Various withdrawal strategies are possible: – Sudden withdrawal: In-depth interviews with CTD consultants and with CTD & WHO authorities have revealed that RNTCP is not ready for a sudden and definitive withdrawal of the entire CTD consultants network, for the same reason as the state-level consultants, namely, that such brisk withdrawal will have a huge, negative impact on RNTCP performance, especially given the enormous challenges likely to be presented by RNTCP3 conceptualization and implementation. – Another alternative is to incorporate them administratively in the CTD staff, but this implies that the label of “WHO consultant” will disappear, and the attractiveness of the job could seriously decrease. – A third alternative would be to create a GoI and donor consortium, in which the GoI bears the financial expenses (basic salary, etc.) of the CTD consultants network, and WHO, and/or eventually other international organizations, retain the technical mentoring of their activities. The consultants should receive career development stimuli, such as a better financial package, increased exposure to international events, and be allowed to bear a title which reflects their special status in the CTD and their affiliation with WHO/other international organizations. Recommendations • Further support until 2017: It is strongly suggested that USAID/India continue to support the CTD consultants network until 2017. USAID/India should insist that the concerns which have been raised in this report be taken care of and request a phasing-out plan, with concrete benchmarks, for taking over the responsibilities that belongs to the GoI cadre by the CTD staff. Evaluation of Tuberculosis Program in India: WHO Report 33 • Further support beyond 2017: A GOI-donor consortium should be created to discuss modalities for further support. • Contract duration: Given the great challenges to be faced when launching RNTCP3, job security until the end of the launching period in 2017 is highly advisable. If a contract of five years’ duration administratively is not feasible, then a renewable, two-year contract should be considered. • Supervision and mentoring: More time should be spent in mentoring and supervising the individual consultants. However, given that the NPO position remains unfilled; this is virtually impossible. In practice, it is urgent that the WHO fills the open NPO position. • Performance appraisal: Appraisals should be structured along the same lines as the classic, HR￾management techniques used. A positive evaluation should create opportunities for professional growth. • Remuneration: Has to become competitive in order to retain the most valuable experts for longer periods on the job. • Career development opportunities: WHO should change its attitude in this domain, from one of neglect to a more proactive approach. In practice, WHO should keep a record of its consultants’ career development expectations and should match those expectations with upcoming opportunities in the fields of capacity building, international meetings, exchange of experience in international meetings, etc. A Case Study: WHO consultants’ involvement in poorly performing districts Background In 2010, CTD made the important policy decision to move from its current program performance objective, to detect at least seventy percent of new smear sputum cases (NSP) to universal access for TB control.34 The new initiative, known as RNTCP3, intends to provide universal access to timely and quality diagnosis and adequate treatment for all TB patients in the community, including vulnerable and marginalized people. The intermediate target is to detect ninety percent of all TB cases and successfully treat ninety percent of them by 2015. RNTCP3 implementation will begin in 2012. This strategy is based on the global consensus that the prior objectives, of seventy percent case detection and eighty-five percent, treatment success are insufficient to achieve adequate reduction of the disease burden and do not produce an epidemiological impact on TB.35 In order to achieve these new program objectives, district and sub-district health programs, the front-line service providers for TB care, must provide high-quality services. Study Aims 34 The indirect indicator to measure “universal access to care” is the NSP detection target of at least ninety percent. 35 A TB-control program has an epidemiological impact if the incidence of new cases decreases significantly, and if it is proven that the decrease is attributable to the proper functioning of that-TB control program Evaluation of Tuberculosis Program in India: WHO Report 34 To analyze the factors that are associated with the difference in RNTCP2 performance in “high” and “low” performing36 districts, to determine what role the WHO consultants presently play in those districts, and what role they are might play in both types of districts in the context of RNTCP3. Study Area A case study was conducted in four districts of Eastern Uttar Pradesh. These districts were chosen based upon their previous history of lower RNTCP performance than districts in Western UP. Eastern UP generally is considered to be of a lesser socio-economic status and also less developed. Specifically, two districts with sub-optimal performance (Varanasi and Chandauli districts) and two higher performing districts (Kaushambi and Sant Ravidas Districts) were selected purposively. Chandauli district is adjacent to the state of Bihar, one of the poorest and weakest performing states in India. Districts Visited Varanasi Chandauli Sant Ravidas Nagar Kaushambi Map 1: Districts selected for the review 36 The distinction is based upon performance in case detection and cure-rate targets: “poorly performing districts” did not hit those targets in the last two years. Evaluation of Tuberculosis Program in India: WHO Report 35 Figure 1 shows the very low TB-suspects examination rate in Chandauli district; the rates continued to decrease in recent quarters, while rates were quasi-stable and reached approximately identical levels in the other districts. 50 75 100 125 150 175 200 225 250 1Qu09 2Qu09 3Qu09 4Qu09 1Qu10 2Qu10 3Qu10 4Qu10 Rate per 100,000 pop Kaushambi SRD Nagar Chandauli Varanasi Figure 1: Quarterly TB suspects’ examination rate by district, UP, 2009-10 Figure 2 shows a low case-notification rate in Chandauli and Varanasi districts, with a very steep decline in recent quarters. These graphs show a low performance in both districts, with a recent, decreasing trend; the decrease in SRD Nagar district was less pronounced, while there was an increasing trend in Kausambi district. 30 40 50 60 70 80 90 100 110 1Qu09 2Qu09 3Qu09 4Qu09 1Qu10 2Qu10 3Qu10 4Qu10 rate per 100,000 pop Kaushambi SRD Nagar Chandauli Varanasi Figure 2: Quarterly Case notification rates of S+ cases by district, UP, 2009-10 Evaluation of Tuberculosis Program in India: WHO Report 36 Methodology Study Design: A cross-sectional study. Key personnel interviewed: Health providers (DM, STO, DTO, MOTC, STS, STLS, LT), WHO consultants, TB patients and DOT providers. Data collection: Initial desk review of the program data for the last two years was carried out on key program parameters, such as the number of chest symptomatics examined, number of smear positives detected, proportion of initial defaulters, treatment outcome of registered patients, and number of retreatment cases registered. Field visits were carried out in each of these four districts, and the following issues were verified: contact tracing in place, External Quality Assurance (EQA) in place, initiatives for community DOT, role of contractuals, involvement of private practitioners, involvement of medical colleges, ACSM practices, smear conversion rates, training activities, type of DOT providers used, and the involvement of NGOs. In-depth interviews of key personnel, including the DTOs, CMOs and district collectors, were carried out in each of these four districts, focusing on issues such as political commitment, utilization of finance, staff positions and vacancies, training and the quality of training, the role of health staff in supportive supervision and feedback, and conducting periodic program reviews. Data analysis: Indicators were compared between poorly-performing and well-performing districts. The contributions made by the WHO consultants was also compared between these two types of districts Results The salient findings are detailed in Table 1. Determining factors Varanasi Chandauli SRD Nagar Kaushambi TB suspects examination rate 154 99 174 183 NSP notification rate 13 16 13 14 % of initial defaulters 9 6 0 0 Periodic program reviews No No Yes Yes DMCs as per population norms No No Yes Yes Staff positions filled No No yes yes Staff trained