USAID/RDMA: MID-TERM EVALUATION OF THE “GREATER MEKONG SUB-REGION MULTIDRUG RESISTANT TUBERCULOSIS PREVENTION AND MANAGEMENT” PROGRAM (CAP-TB) ประเทศไทย 中国 Figure 1 March 2014 This publication was produced for review by the United States Agency for International Development. It was prepared by Dr. Aimé De Muynck, Carina Stover, and Dr. Amy Bloom. 1 MID-TERM EVALUATION OF THE “GREATER MEKONG SUB-REGION MULTIDRUG RESISTANT TUBERCULOSIS PREVENTION AND MANAGEMENT” PROGRAM (CAP-TB) Cooperative Agreement No. AID-486-A-12-00002 DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Documents are made available through the Development Experience Clearinghouse (http://dec.usaid.gov). 2 ACKNOWLEDGEMENTS The evaluation team would like to acknowledge and thank all the fine effort that went into the CAP￾TB program to date and the outstanding dedication displayed by all involved during the conduction of our mid-term evaluation from February through April 2014. The USAID/RDMA and USAID/Burma officials provided crucial assistance to the evaluation team with documentation, names and contacts, and insights on the CAP-TB project, for which the team is very grateful. Special acknowledgements go to Jittinee Khienvichit, Marisa Sanguankwamdee and Netima Lohpattananon of USAID/RDMA, whose support during the team’s time in country was most appreciated, but also before and after our time in country. The evaluation team would like to express their sincere thanks to the USAID teams in Thailand and Burma for their support and encouragement during this mission; to the FHI 360 regional and country teams for their technical and logistical support; to the various NTP offices for their interest in this mission and their advice and suggestions; to the CDC official in Bangkok who shared her views on the implementation of the CAP-TB project, and to the WHO officials in Thailand, Burma, and China who explained their views on the role played by the CAP-TB project in MDR-TB control in their respective countries. A special thank also goes to Max Millington for his editing skills. The evaluation team wishes to acknowledge the commitment of the FHI 360 teams of the regional office in Bangkok; Rayong province, Thailand; Rangoon and Mandalay, Burma; and Kunming, China, for allowing their regional and country teams to prepare the agenda and necessary documentation for this mission, facilitate the visits to the field stations, and arrange contact with the implementing partners. The evaluation team recognizes the important contribution of the translators in Rayong, Thailand; in Yangon and Mandalay, Burma; and in Kunming, China. Without their active involvement, the field visits and interviews would have been far less informative. 3 ACRONYMS AIDS Acquired immunodeficiency syndrome APRO FHI 360 Asia Pacific Regional Office BCC Behaviour Change and Communication BTB Bureau of Tuberculosis (in Thailand) CAP-TB Greater Mekong Sub-Region Multidrug resistant tuberculosis prevention and management project CATII Category II TB patient CDC Centers of Disease Control CHW Community Health Worker DALY Disability-adjusted life years DM Diabetes mellitus DOT Directly Observed Treatment DOTS Directly Observed Treatment, Short course DQA Data Quality Assessment DST Drug-Susceptibility Testing EQA External Quality Assessment FGD Focus Group Discussion FHI360 Family Health International 360 GFATM Global Fund to Fight AIDS, TB, and Malaria GMS Greater Mekong Subregion GP General Practitioner (medical) HIV Human Immunodeficiency Virus HR Human Resources IA Implementing Agent IC Infection Control IEC Information, Education, Communication IUATLD International Union against Tuberculosis and Lung Disease JMM Joint Monitoring Mission KAP Knowledge, Attitudes and Practise LED Light emitting diode (fluorescent microscopy) LSHTM London School of Hygiene and Tropical Medicine MBCA Myanmar Business Coalition on AIDS MDR-TB Multi-drug resistant tuberculosis MHAA Myanmar Health Assistants Association MMA Myanmar Medical Association MoU Memorandum of Understanding NCCM National Catholic Commission on Migration NCTB National Control of TB (China) NGO Non-Governmental Organization NTP National Tuberculosis Programme OPD Outpatient department PGK Pyi Gyi Khin (PGK - Women’s General Development Enterprise & Co￾Operative Limited) PHO Rayong Provincial Health Office PLHIV People living with the Human Immunodeficiency Virus PMDT Programmatic Management of Drug-resistant Tuberculosis PMP Project Management Plan PPP Public-private partnerships QQ “QQ” Chat Platform in China RDMA (USAID) Regional Development Mission for Asia SLD Second Line Drugs SOW Scope of work TAD Treatment after default 4 TAF Treatment after failure TB Tuberculosis TCC Tuberculosis Clinical Center TSG Technical Strategic Group UN United Nations USAID United States Agency for International Development WFP World Food Program WHO World Health Organization XDR-TB Extensively drug-resistant tuberculosis YAT...... Yunnan Anti-Tuberculosis Association 5 Contents 1 EXECUTIVE SUMMARY .................................................................................................................... 7 1.1 MAIN BODY OF REPORT...........................................................................................................11 1.2 Evaluation purpose and Evaluation questions.........................................................................11 1.3 Project background..................................................................................................................11 1.4 Evaluation Methodology..........................................................................................................12 1.5 Response to critical Questions (Thailand, Burma, and China).................................................14 1.5.1 BURMA ..........................................................................................................................14 1.5.1.1 Burma: Key findings..................................................................................................14 1.5.1.2 Burma: Recommendations .......................................................................................25 1.5.2 CHINA............................................................................................................................26 1.5.2.1 China: Key Findings...................................................................................................26 1.5.2.2 China: Recommendations.........................................................................................35 1.5.3 THAILAND......................................................................................................................37 1.5.3.1 Thailand: Key Findings...............................................................................................37 1.5.3.2 Thailand: Recommendations ....................................................................................44 2 OVERALL PROJECT CONCLUSIONS AND LESSONS LEARNED.........................................................45 3 OVERALL RECOMMENDATIONS....................................................................................................47 3.1 To USAID/RDMA ......................................................................................................................47 3.2 To FHI360 regional office.........................................................................................................47 Tables TABLE 1. COMPARATIVE TABLE OF AVERAGE COST OF PROVIDING A FULL SLD TREATMENT BY THE IAS (MMA, PGK AND MHAA) IN COMPARISON WITH THE NATIONAL AVERAGE IN BURMA. .....................................................................................................................................................................21 TABLE 2: ESTIMATED COST OF FULL MDR-TB TREATMENT, YUNNAN PROVINCE, 2013. .....................32 TABLE 3: “RE-ON-PRE” SCREENING CRITERIA OF TB/MDR-TB SUSPECTS, AND MALE/FEMALE RATIO. .....................................................................................................................................................................38 TABLE 4: COMPARATIVE COST ESTIMATION FOR A FULL MDR-TB TREATMENT, YUNNAN AND RAYONG PROVINCES, 2014.....................................................................................................................42 Figures FIGURE 1. CAP-TB BURMA’S “COVER YOUR COUGH” CAMPAIGN. ..........................................................19 FIGURE 2. PROPORTIONAL DISTRIBUTION OF THE EXPENSES BY THE 3 COUNTRY OFFICES, 2013...21 Annexes A. Scope of Work B. Analytical Framework C. Evaluation Calendar and Persons Met D. Interview Questions E. References and Source of Information and Data used F. Gender Distribution of TB Patients Covered G. Budget Charts H. Comments on LSHTM CAP-TB research protocols I. Disclosure of any conflicts of interest 6 EXECUTIVE SUMMARY A midterm evaluation of the Greater Mekong Sub-Region Multidrug resistant tuberculosis prevention and management project (CAP-TB) funded by USAID, was undertaken in Thailand, Burma, and China in February-March 2014. Evaluation purpose and evaluation questions The core objectives of this midterm evaluation were to: • Assess the project’s performance and progress toward intended results; • Provide insights and feedback to each of the partners and stakeholders with respect to both program strengths and areas where technical, administrative, and management efforts could be improved; and • Provide evidence and models for adapting future programs and projects, including improving USAID/RDMA designs, strategies, and policies. This midterm evaluation focused on answering the following 5 key questions posed by USAID/RDMA: 1. To what extent and how has CAP-TB strengthened MDR-TB management? 2. To what extent and how has the CAP-TB community-based approach contributed to TB and MDR-TB prevention, diagnosis and treatment? 3. Are the management arrangements optimal for achieving project objectives efficiently and effectively? 4. To what extent and how has the strategic information generated by the project and Knowledge Gateway information been used? 5. What measures/mechanisms have been put in place to achieve sustainability and which still need to be addressed? Project background The Control and Prevention of Tuberculosis “CAP-TB” Project was awarded on October 14, 2011 and will end on October 13, 2016. Its overarching aim is to reduce the incidence of and mortality due to MDR-TB. To achieve this, FHI360 and its implementation partners focus on the following objectives: • Develop, implement, and evaluate local models for MDR-TB prevention and management that can be scaled up; • Increase detection of MDR-TB cases and treatment successes in areas where the model MDR-TB program is implemented; and • Build local systems and capacity to sustain these models beyond CAP-TB’s active presence. A key priority for CAP-TB is to develop capacity among local organizations in each of the three countries. The project is being implemented in Rayong Province, Thailand; Yunnan Province, China; Rangoon and Mandalay, Burma. FHI360 is the prime cooperating agency and leads a consortium of Partners in each focus country. Evaluation design, methods and limitations The evaluation was conducted by a team of consultants consisting of two international senior TB specialists and a senior international public health specialist. Two members of the team met with the FHI360 regional office staff at the beginning of the assignment and also after the compilation of the field work. The information gathered through literature review, meetings with the implementing teams, field visits, in-depth interviews, rapid reviews of health facility records, and use of strategic information by Project partners was synthesized by the team; findings were arranged by themes and sub-themes under the five critical questions, and conclusions and recommendations formulated. The following operational challenges were met by the evaluation team: IAs’ lack of fluency in English was encountered in each focus country, but was largely overcome through expert translators, and support by FHI360 staff; furthermore lack of relevant quantitative performance data in each focus 7 country, especially with respect to outcome and determinants, made it cumbersome to attribute the role of the project in assessing the effectiveness and efficiency of the performance. Key Findings Burma The CAP-TB partners include the following 4 organizations: Myanmar Medical Association (MMA); Pyi Gyi Khin (PGK); Myanmar Health Assistants Association (MHAA) and Myanmar Business Coalition on AIDS (MBCA). The areas covered are concentrated in Rangoon and Mandalay. Management: CAP-TB is successful in strengthening MDR-TB program management, but much less in data management. CAP-TB’s partners are actively involved in referring TB suspects and contact tracing activities. The vulnerable members (PLHIV, elderly, migrants and DM patients) of the target population are screened, although with limited success. The treatment quality was scaled up, through training of the staff, strengthening community-based Directly Observed Therapy (DOT) and providing a package of support services. Linkages with community health workers are well developed and highly functional. CAP-TB has provided 1 Xpert machine and a start-up supply of Xpert cartridges. CAP-TB collaborates with all implementers in TB/MDR-TB prevention and control in the covered areas. While the Burma FHI360 and USAID/Burma officers seem ready to take on a more autonomous role, the pros and cons (including funding and contractual arrangements) need to be analysed prior to initiating any change. Community-based approach for Project Performance: CAP-TB has successfully piloted the country’s community supporters. DOT is adapted to the local situation and designed to meet individual patients’ medical and personal needs. DOT providers are trained and mentored by CAP￾TB staff; their intervention promotes good treatment adherence. The working relationship with the implementing partners and collaborating hospitals is very effective. CAP-TB has been able to elicit significant community involvement in MDR-TB prevention and control, through outreach of Implementing Agents (IAs) to patients, use of educational tools, and involvement of cured MDR-TB patients and local celebrities in the health Information, Education and Communications (IEC) activities. CAP-TB project gives full attention to infection control (IC). There is scarce information on the key factors of the project performance, and of the effects of the community approach. Project effectiveness and efficiency: Project staff has great management skills and high motivation to provide TA and capacity building, and establish effective relationships with NTP, WHO and other authorities, local NGOs and with the four in-country IAs. The management of CAP-TB in Burma is optimally efficient. The data gathering and reporting system works well in Burma, but routine data are insufficiently used as evidence-for-action. Strategic information: CAP-TB gathers strategic information through routine data and operational research results. The Knowledge Gateway is central to the dissemination of the information, but there is little evidence of its use and of its perceived usefulness by the field staffs. There are important gaps in strategic information/data for strategic decision making, mainly due to lack of stratified case finding and case holding parameters and of their determinants such as gender. CAP-TB has organized several data management trainings of IAs and performed regular assessment checks of data quality; however their ability to interpret the routine information is still basic. There are problems with data confidentiality due to lack of project-specific record storage. Sustainability: Ensuring financial and technical sustainability of the supported programs remains a challenge. Nevertheless, there are encouraging signs towards sustainability, such as integration of project activities with existing national TB priorities and health systems, capacity of the communities to support project activities, cost-effectiveness, active interface of the collaborating hospitals with the communities, and a functional Public Private Partnership (PPM). 8 China Management: CAP-TB is very successful in strengthening MDR-TB program management. However, reinforcing data management was less successful. CAP-TB has provided 1 GeneXpert machine to Yunnan CDC, as well as cartridges. The referral linkages for patients at risk for MDR￾TB are successfully strengthened; the target has been largely exceeded due to a higher than expected commitment from TB staff at the community and district level, and from private clinics, and through introduction of a referral slip, to trace non-respondents. Contact screening is an effective activity of TB control. Highly vulnerable members of the target population are screened, although with small success. There is a very high initial default rate, which has no received sufficient attention so far. The quality of the treatment has been scaled up, through addressing major concerns of the patients; improve treatment adherence and providing a package of support services. CAP-TB is strongly involved in promoting collaboration with all IAs in the covered areas. In Yunnan province, the present rapport between country and regional FHI360 office staff is suitable to all involved parties, and should best be maintained in the present state, because a too early transition might be detrimental to the country program. Community-based approach for Project Performance: CAP-TB has demonstrated great effectiveness and creativity while organizing training sessions for community outreach workers. Correct implementation of DOT contributes to improve treatment adherence rates; however, some hospitalized patients do not receive DOT after their discharge, and the default rate is high. There is a superb relationship with the IAs and hospitals. The community approach is well appreciated and supported by the provincial authorities and the partners. CAP-TB has been able to elicit a highly significant community involvement. CAP-TB’s IEC work is outstanding from a technical and pedagogical point of view; the internet-based TB QQ network, called “57 Zone”, is an excellent example of involving affected individuals to manage their disease with the aid of experts, and providing support and networking in a safe and non-judgmental setting. Contact tracing and IC are two effective prevention activities. The project has not given sufficient attention to measure the key factors of project implementation and the effects of the community approach. Project effectiveness and efficiency: The actions undertaken are effective, and efficient. CAP￾TB China program has an overall solid M&E system, particularly in the establishment of M&E focal persons, clear roles and responsibilities of partners and documentation tools; nevertheless there are still some deficiencies in the data management resulting in little use of the routine data by the IAs for decision making. Strategic information: CAP-TB project has focused on the development and distribution of strategic information, as well as on its incorporation into MDR-TB prevention, treatment, and follow-up strategies. Social media is the cornerstone of CAP-TB’s efforts to spread relevant knowledge on TB. In Yunnan province, the same gaps in strategic information have been observed as in Burma; however more use is made of the strategic information, than in any other focus country. Sustainability: National and Yunnan provincial authorities recognize CAP-TB’s MDR-TB control approach as being very innovative for the country and the province. Ensuring financial and technical sustainability of the CAP-TB supported MDR-TB prevention and control program remains a challenge, although there are encouraging signs towards sustainability. The latter may be enhanced by more accurately determining the TB burden, as current assumptions seem to underestimate the TB burden, leading to low funding levels as compared to those of other diseases. Thailand Management: CAP-TB is successful in developing human capacity and providing the necessary supplies & commodities. The management of the MDR-TB cases is carried out correctly, as suggested by the good interim results. There is a high performance of MDR-TB case finding activities. 9 CAP-TB is actively involved in contact tracing activities, although the number of cases detected remains small. CAP-TB contributed to the national strategic plan calling for increasing active testing of at risk populations, although unequal attention was given to the at-risk groups. The initial default rate was high, but the performance of the MDR-TB treatment and the follow-up were good. CAP￾TB works closely with community health workers, and has taken a series of capacity building initiatives for them. CAP-TB is heavily involved in promoting collaboration with all IAs in the province. Given that there is a lack of clarity about the specificity of CAP-TB’s niche, there are differences between the implementers in conceptualizing the target public of MDR/TB control in Rayong province. Community-based approach for Project Performance: CAP-TB has provided an effective training and mentoring of the community DOT providers; a cordial and constructive relationship has been established with the health providers and collaborating hospitals. The key factors of project implementation and the effects of the community approach on patient adherence have not been measured. Project effectiveness and efficiency: The routine implementation of the capacity building initiatives is very good; the planning done by the FHI360 BKK office is performance-informed, and the TA and capacity building inputs are decided in function of the needs. The cost of a full treatment course for an MDR-TB patient in Rayong province is 68% higher than the national estimate; but the reasons for this excessive cost have not yet been examined. The data gathering and reporting system works well in Rayong province, but the data management is not optimal, resulting in little utilization of the evidence for programming. An effective and cordial relationship has been established with NTP, WHO and other authorities, and the local partners. Strategic information: The project partners in Rayong province use various mechanisms to obtain the latest strategic information, however the Knowledge Gateway is used hardly, for reasons of difficult linguistic and internet access. Sustainability: Ensuring financial and technical sustainability of the supported programs remains a challenge. Creating a dialogue with beneficiaries, authorities and partners, on exit strategy alternatives could be of great help to promote the sustainability of the CAP-TB approach, but has not been started yet. Overall project Conclusions and Recommendations CAP-TB represents an appropriate regional focus on prevention and control of MDR-TB. The project is appropriately focused on the highly vulnerable strata of the populations, although their screening lacks effectiveness. The comprehensive approach was adapted in each country according to local needs. CAP-TB has been successful in strengthening MDR-TB program management in all focus countries, but much less in data management. CAP-TB has successfully piloted community supporters, and demonstrated effectiveness and creativity while organizing their training. The DOT provided is appropriate to the local situation and designed to meet individual patients’ medical and personal needs. The actions undertaken are effective in all focus countries and also efficient in Burma and China, but much less in Rayong province. CAP-TB project has focused on the development and distribution of strategic information, as well as on its incorporation into MDR-TB prevention, treatment, and follow-up strategies. The information produced by the specially developed Knowledge Gateway is used sparsely. CAP-TB’s MDR-TB control approach is considered as being very innovative in all focus countries. Ensuring financial and technical sustainability of the CAP-TB supported MDR-TB prevention and control program remains a challenge, although there are encouraging signs towards sustainability, especially in Burma and China. The mechanisms to enhance sustainability differ by country; one approach in Yunnan province might be to accurately determine the TB burden, as 10 current assumptions seem to underestimate the TB burden, leading to low funding levels as compared to those of other diseases. Implementation studies should be supported to decrease the prevalence of the determining factors of the incidence and outcome of the MDR-TB cases. Evidence-informed management trainings & post-course follow-up should be organized for all staff involved in project implementation. The project managers should involve the private sector more in MDR-TB prevention and control; hereto a “win-win” strategy should be developed through field research and the findings should be incorporated in the up-scaled PPM strategy. Sharing of ideas and experiences among PHI360 staff and implementers across the 3 focus countries should be improved. The project managers should enter into constructive dialogue with the beneficiaries, partners and authorities in order to develop a country-specific exit strategy and promoting alternatives to guarantee the sustainability of the CAP-TB model. 1 MAIN BODY OF REPORT 1.1 EVALUATION PURPOSE AND EVALUATION QUESTIONS The purpose of the evaluation was to analyze the CAP-TB project performance to date and suggest recommendations to help the project meet its intended purpose, and to analyze the value-added by the regional project to knowledge and skill enhancement, and organizational capacity building of local partner institutions. This midterm evaluation focused on answering the following 5 key questions posed by USAID/RDMA: 1. Project management: To what extent and how has CAP-TB strengthened MDR-TB management? 2. Community-based approach for Project Performance: To what extent and how has the CAP-TB community based approach contributed to TB and MDR-TB prevention, diagnosis and treatment? 3. Project effectiveness and efficiency: Are the management arrangements optimal for achieving project objectives efficiently and effectively? 4. Strategic information: To what extent and how has the Strategic Information generated by the project and Knowledge Gateway information been used? 5. Sustainability: What measures/mechanisms have been put in place to achieve sustainability and which still need to be addressed? 1.2 PROJECT BACKGROUND The Control and Prevention of Tuberculosis “CAP-TB” Project was awarded on October 14, 2011 and will end on October 13, 2016. CAP-TB was planned for a budget of $10 million over this 5 year period. Slow start-up has meant that implementation has not been taking place for much of the first half of the Project life. The Project goal is to contribute to reducing incidence and mortality due to multidrug resistant tuberculosis (MDR-TB) across the Greater Mekong Sub-region (GMS). The Project is carried out in three MDR-TB high burden countries in the region: Burma, China and Thailand. Project activities focus on: (1) developing, implementing and evaluating local models for MDR-TB prevention and control that can be scaled up, (2) increasing detection of MDR-TB1 cases and treatment success in areas where model MDR-TB is implemented, and (3) building local systems and capacity to sustain these models beyond CAP-TB. The spread of MDR-TB threatens to undermine recent gains in controlling drug-susceptible TB, as any TB patient can develop MDR-TB; similarly MDR-TB can become resistant to second-line drugs (SLDs), leading to extensively drug￾1 MDR-TB strains are TB strains that are resistant to the 2 main first-line TB drugs (INH and Rifampicine) 11 resistant TB (XDR-TB2), making thus detection, treatment and prevention of MDR-TB cases even more of an imperative. FHI360 is the prime cooperating agency and leads a consortium of partners. In Yunnan, China, CAP-TB’s partners include the International Union Against Tuberculosis and Lung Disease (IUATLD) and the Yunnan Anti-TB Association (YATA). In Rayong, Thailand, CAP-TB’s partners include the National Catholic Commission on Migration (NCCM) and Rayong Provincial Health Office (PHO). In Rangoon and Mandalay, Burma, CAP-TB’s partners include the Burma Medical Association (MMA), Pyi Gyi Khin (PGK), Myanmar Business Coalition for AIDS (MBCA), and Myanmar Health Assistants Association (MHAA). CAP-TB also collaborates with the National Tuberculosis Programs (NTP) and Center for Disease Control of each country, the U.S. Centers for Disease Control, the World Health Organization (WHO), and some International Non￾governmental Organizations (NGOs), private sector partners working on MDR-TB control, and other partners working on TB, MDR-TB, and HIV control. The project is being implemented in Rayong Province, Thailand; Yunnan Province, China; Rangoon and Mandalay, Burma. As significant national and international efforts are already in progress in each focus country to improve diagnosis and management of MDR-TB and to prevent new cases of MDR-TB, CAP-TB activities try to align with and support ongoing efforts while tailoring project activities to the needs and priorities of the individual countries3. CAP-TB anticipates reaching the following achievements at the end of the implementation: - Improved management of MDR-TB prevention and control - Strengthened capacity among local partners - Stronger health information systems - Greater laboratory and management capacity - Equitable access to prevention, treatment and care for MDR-TB patients - Improved health-seeking behavior and linkages between national partners and health institutions - Strengthened inter-country learning and collaboration in the GMS - Greater advocacy and communication around MDR-TB 1.3 EVALUATION METHODOLOGY This mid-term evaluation of the USAID CAP-TB program was carried out in February-March 2014 at the request of USAID/RDMA. Apart from the literature review and report preparation, five weeks were spent doing field visits and interviews in the three focus countries. Team Composition The evaluation team was composed of three senior international health consultants: Dr. Aimé De Muynck, Carina Stover, and Dr. Amy Bloom (who participated for two weeks in Burma and China). Evaluation Methodology: Two members of the team met with the FHI360 regional office staff at the beginning of the assignment and also after the compilation of the field work in the three focus countries. The information gathered was synthesized by the team and determined themes and sub-themes under the five critical questions, and conclusions and recommendations were formulated. At the end of the field work, the evaluators conducted debriefing sessions with both USAID/RDMA and FHI360 Regional and country offices. The Scope of Work (SOW) is found in Annex A. The Analytical Framework for the evaluation approach was developed by the team leader and is found in Annex B, and the Evaluation 2 XDR-TB is defined by WHO as a TB strain that has developed resistance to INH and Rifampicine, as well as to any member of the quinolone family and at least one of the following second-line anti-TB injectable drugs: kanamycin, capreomycin, or amikacin. 3 CAP-TB Project. Country Assessments (Burma, China, Thailand). July 2012. 12 Calendar and Persons Met is found in Annex C4. The following list outlines the processes used by the team to approach the mid-evaluation: • Background literature review, including a review of indicators from Performance Monitoring Plan (PMP), partner reports, and the results of Organizational Development Assessments • Team planning meetings to outline the report and evaluation methodology and to define team responsibilities • Meetings with USAID/RDMA and USAID/Burma project management teams • Meetings with FHI360’s CAP-TB teams and IAs in each country • Interviews with NTP, WHO, IUATLD and other critical partners in each country. Rapid appraisal methods (such as key informant interviews, focus group discussion, community interviews, direct observations and mapping) were used with country implementing partners and sub-partners, community health volunteers, and health facilities involved in the project. • Field visits to clinical sites • Extensive discussions held with community health volunteers and beneficiaries (TB and MDR-TB patients) • Rapid reviews of health facility records and other available clinic-based data • An informal assessment was done to assess gender equality and women’s empowerment programming priorities and achievements to date • Project partner use of strategic information was examined • Findings and recommendations presentation to USAID/RDMA and FHI360 • Drafting of the report for review by USAID/RDMA • Finalization of report utilizing USAID/RMDA inputs Evaluation limitations The following operational challenges were met by the evaluation team: IAs’ lack of fluency of in English was encountered in each focus country, but was largely overcome through expert translators, and support by FHI360 staff; furthermore lack of relevant quantitative performance data in each focus country, especially with respect to outcome5 and determinants, made it cumbersome to attribute the role of the project in assessing the effectiveness and efficiency of the performance. Data stratified by person (age, gender, migration status) and place (particularly distance between the residence and clinic providing MDR-TB treatment) characteristics were lacking. To understand how the project is reaching its goal of MDR-TB prevention, routine data on the factors that lead to MDR￾TB6 would have been useful (e.g. data describing side effects’ profile, practice of DOT7, defaulting, etc.). During the preparation of this review, the evaluators proposed to carry out a series of Focus Group Discussions (FGD), which would have been most helpful to gather subjective perspectives from key stakeholders and to provide in-depth information on perceptions, insights, attitudes, experiences, and beliefs. However, the evaluators were forced to obtain approval by an ethical committee in each focus country. This was nearly impossible to arrange in such a short notice; consequently this idea had to be dropped, and only a single ‘mini FGD with MDR-TB patients’ was held in Burma. 4 Because of respect for the patients and care takers who want to keep their names out of the public reach due to the stigma associated with TB, especially MDR-TB, we have not included these individual names. 5 Number of MDR-TB cases receiving second line drugs (SLDs) that were culture negative culture at month 6; number who died or defaulted by month 6, and who were found not to have MDR-TB ( i.e., “false positives’’). Furthermore, data on treatment delay. 6 As documented in a published study in the medical journal ‘Thorax 2012; 67: 632-638’, under the title ‘ Factors contributing to the high prevalence of MDR-TB: a study from China’ by Liang L, WU Q, Gao L, et al 7 DOT helps patients to take their drugs regularly and complete their treatment, thus achieving cure and preventing the development of drug resistance. DOT is given by a DOT-provider, belonging to a health facility, or the family (called ‘FAM-DOT’) or the community (called ‘COM-DOT’) 13 Most of the available data lacked any stratification by age, gender, residence, etc., and information on risk factors was generally absent. Therefore, assessment of outcome trends and key determinants of project effectiveness was not possible. As a consequence, the team had to rely mainly on qualitative information and on impressions based on crude data. Generalizability and transferability of the findings Generalizability can be defined as the extension of the findings and conclusions from the sampled population to the population at large; thus they are assumed to be true in a larger group than those being studied. Transferability is the ability to transfer the findings to situations with similar parameters, populations and characteristics. In the concrete context of this midterm review, the generalizability of the findings is debatable, because the original sampling was a purposive8 one, and most of the potential confounders9 are lacking, therefore the findings can at most be “transferable” to the country in which the findings were obtained. 1.4 RESPONSE TO CRITICAL QUESTIONS (THAILAND, BURMA, AND CHINA) Key findings are presented for each of the five questions (see SOW SOL-486-13-000045) for each focus country and the corresponding recommendations made. The overall conclusions and recommendations follow the country sections. Annotations, such as (a), (b) etc. refer to the sub￾questions in the SOW. 1.4.1 BURMA 1.4.1.1 Burma: Key findings The CAP-TB partners include the following 4 organisations and portfolios: Name Acronym Portfolio Myanmar Medical Association MMA Training for GPs in 18 townships Conduct DOTS to MDR-TB patients Manage minor adverse effects Refer cases to medical facilities using community supporters Pyi Gyi Khin PGK Provide “living support package” to MDR-TB patients Conduct community outreach, finding and referring TB suspects Carry out home-based care, including contact investigation and IC in 7 townships in Rangoon Myanmar Health Assistants Association MHAA Provide “living support package” to MDR-TB patients Conduct community outreach, finding and referring TB suspects Carry out home-based care, including contact investigation and IC in 2 townships in Rangoon and 7 townships in Mandalay Myanmar Business Coalition on AIDS MBCA Provide screening for all employees; Advocate for workplace places around TB Conduct community outreach, finding and referring TB suspects in Monywa industrial zone 8 A purposive sampling is a non-representative subset of some larger population, and is constructed to serve a very specific purpose. Because they do not truly represent a larger population (here of a country), no valid inferences about the larger group from which they are drawn. 9 A Confounder is a background factor for which the groups differ AND the background factor itself influences the outcome. One way to control for such confounder is adjusting for this factor in the analysis. The basic condition is, of course, that information is available on those background factor(s), which rarely happened in the CAP-TB project. 14 (a) Human Capacity and Supplies/Commodities – CAP-TB has been successful in strengthening the MDR-TB program management, but much less in data management. The targets for training in active case finding methods were greatly exceeded (221%); training in programmatic management of MDR-TB was successfully organised, and the targets were met (103%). The field visits allowed noticing that the trainings produced not only knowledge, but also skills and motivation to implement the lessons learned. Various trainings were organised for staff from CAP-TB partner organisations, for TB champions of MBCA, private practitioners, township medical officers, and local drug sellers. Support was given to the training-of-trainers on programmatic management of MDR-TB organized by NTP. FHI360 regional office provides support to the FHI360 country team through frequent digital communications, and backstopping of the IAs in Rangoon and Mandalay by means of monthly supervision, information dissemination, training, and logistical support. The targets of training in data management/HMIS for IAs were barely met (achievement of 30% only), because one-day trainings are insufficient to create adequate awareness for data quality and motivation to use data in program implementation. Due to time constraints, the reasons for these data management deficiencies10 could not be explored during this review. CAP-TB has procured 1 Xpert11 machine and 2200 cartridges12 in August 2013 for the Lower Myanmar TB Center, potentially enabling patients to be initiated on treatment quickly, although culture and DST results are still indispensable in Burma for an Xpert+ person to be put on SLD treatment. (b) Case Management and Referrals for MDR-TB – MDR-TB cases are managed effectively, as suggested by the good interim results (low initial default rates; low default rates, few missed treatment doses; high smear conversion rates), although no hard data are available to prove those results. Through CAP-TB’s involvement the service gaps observed during the baseline assessment carried out in 201213, were largely overcome through the creation of effective linkages across health services. CAP-TB organized in a very constructive way four local NGO’s (non-governmental organizations) to provide patient support, and to refer TB suspects, as could be observed by the evaluators during the various meetings and the field visits. The work of CAP-TB in organizing these IAs is nothing less 10 Such as content and duration of the training, supervision as continuous education, critical mass of trainees per centre, etc.) 11 The GeneXpert MTB/RIF test (Xpert) is a cartridge-based new molecular test for TB which diagnoses TB by detecting the presence of TB bacteria, as well as testing for resistance to the drug Rifampicin. Rifampicin-resistant TB is usually MDR-TB and always requires second-line drug therapy, so this has immediate treatment implications. Xpert can detect TB, including MDR-TB on the basis of a sputum sample, in less than 2 hours, potentially reducing the time to diagnose and treat TB. The Xpert MTB/RIF test exhibits high sensitivity and specificity for detecting pulmonary TB disease, and it has similar accuracy as culture. In December 2010, the World Health Organization (WHO) endorsed the Xpert MTB/RIF for use in TB endemic countries and declared it a major milestone for global TB diagnosis. This test has the potential to revolutionize the diagnosis of TB, as the Xpert MTB/RIF test is feasible for use in peripheral labs and clinics by unskilled personnel. 12 The cornerstone of the GeneXpert testing process is a self-contained, single use cartridge. Sample extraction, amplification and detection are all carried out within this self-contained "laboratory in a cartridge". The shelf-life = 18 months. These cartridges can be provided in the free market, although at an expensive price; presently initiatives are being taken by big donors such as USAID, to help the countries buy those cartridges in bulk format, lowering thus the prices. 13 Gaps in capacity building, diagnosis and treatment were observed at the occasion of the Country assessments (Burma, China and Thailand) carried out in 2012. CAP-TB Project, USAID, July 2012. Qu1: To what extent and how has CAP-TB strengthened MDR-TB management? 15 than ground breaking14, and has proven that community-based MDR-TB management is feasible and can contribute to improving case detection, access to diagnosis and treatment, and survival. CAP-TB successfully strengthened referrals for MDR-TB suspects15 as one component of outreach and advocacy activities. A standardized referral form has been created to coordinate the referral and back-referral, and to document the referrals. Of the 197 TB suspects referred in 2013, 94% effectively consulted a diagnostic service. (c) Diagnosis, Treatment and Follow-up – c.1. Diagnosis The project objective of increasing detection of MDR-TB cases in areas where the model MDR-TB program is implemented, has not been met: in FY13 only 18% of the expected number of MDR-TB cases have been diagnosed due to late installation of the Xpert machine, as a consequence of difficulties to obtain the importation permit. The average total delay is shortened (estimated by Dr Thandar Lwin to be of the order of 6 months, while in the past it was 12 months), although no exact data are available (neither on their total delay, nor on the delay caused by the patient, the access to care and /or the health system). Given the present lack of SLDs16, the treatment delay for new MDR-TB patients is increasing still, without CAP-TB being able to solve this problem. CAP-TB is actively involved in contact tracing activities, although the number of cases detected thus far remains small; this low detection is partly due to the fact that about 50% of MDR-TB patients reside in remote and quasi inaccessible areas. The evaluators could not obtain data on the total number of possible contacts, and it is thus impossible to assess the effectiveness of this important activity. Several instances were mentioned of the occurrence of 2 cases in a household, sometimes simultaneous, but mostly subsequent in time; proving thus that intra-household transmission is occurring in Burma. The project aims to improve case finding of highly vulnerable members (PLHIV, elderly, migrants and DM patients) of the target population. CAP-TB’s focus on those groups is excellent, because mathematical modelling has shown that active case finding is more effective than passive case finding to control the MDR-TB epidemic17. However no data could be obtained on the case finding performance in PLHIVs and migrants, and the available data on elderly and diabetics show an insufficient performance. The notification rates in elderly male and female NSP TB patients are low in Burma; although the prevalence rates are the highest in the 65+ stratum18. Only 2% of the CAP￾TB supported MDR-TB cases are in the age group 65+, suggesting problems for the elderly to access screening. In Mandalay, 6 out of the 73 MDR-TB patients presently in treatment have diabetes mellitus II (DM) (= 8%); their monthly follow-up includes also glycaemia, proving that their follow￾up happens in a comprehensive way. 14 Based on the international experience of the evaluators, such cordial, constructive and supportive working together of the 4 IAs is rarely seen at this intensity and scale in other endemic countries. Although the working relationships of FHI360 country offices and regional office with the partners and stakeholders are also cordial in China and Thailand, in Burma there is a supplementary element, namely the collective awareness of the 4 IAs of their role in MDR-TB prevention and control, and of their motivation to collaborate with the other IAs and the FHI360 country office. The lesson to be learnt from this Burma experience is that IAs can be more than only implementers, but also be ‘ambassadors’ of MDR-TB control. 15 These are: Treatment failure patients, treatment defaulters, patients who remain smear positive at two months, close contacts of known MDR-TB patients, and PLHIVs, migrants, elderly and diabetics. 16 The baseline assessment carried out by FHI360 in 2012 had observed “an alarming shortage of SLDs (due to international shortages and delayed deliveries). The shortage of SLDs is today even more alarming than in 2012, as stated by Dr Thandar Lwin on 17 Feb 2014. However, CAP-TB is not responsible for SLD procurement. 17 Dye C & Williams B. Criteria for the control of tuberculosis. PNAS 2000; 97: 8180-8185. 18 As shown by the National TB Prevalence Survey, carried out in 2009-2010 16 The reason why this figure of 8% is substantially lower than the national figure of 12% (as informed by Dr Thandar Lwin) could not be explored during the review. The data on the number of MDR￾TB/DM patients in Rangoon is not available to the evaluators. CAP-TB has included children in the vulnerable population groups, because they are also at risk for TB and MDR-TB. The relevance of that inclusion is debatable, since the burden of childhood TB is unknown in Burma, as the recent nation-wide disease prevalence survey was limited to people above 15 years of age, and contact tracing of index adult cases and identifying possible index cases for TB in children is not systematically done in Burma. c.2. Treatment The performance of the MDR-TB treatment is good, as measured by adherence indicators, rather than cure, as no patients are long enough on treatment to be cured or to have successfully completed their treatment. Qualitative information gathered in the field suggests that the treatment adherence has increased since the start of CAP-TB. c.3. Follow-up CAP-TB has scaled up the quality of the MDR-TB treatment through providing additional staff for counselling and strengthening community-based DOT, and also through a package of support services. Many patients reported that it would be impossible to adhere to treatment without that package, as it is their sole income during the intensive phase of MDR-TB treatment. Regrettably, the nutritional support is scheduled to stop in March 2014. Following information provided by Dr Thandar Lwin, World Food Program (WFP) might take over that support. Given the complexity of the distribution mechanism, Dr Thandar Lwin might request CAP-TB to guarantee continuity and functionality of the system. The CAP-TB project implements DOT in a very holistic way, taking into account patients’ medical and personal needs, as was observed in the field. Basically a trust relationship is being built between each MDR patient and the corresponding DOT provider, which certainly affects the treatment adherence very positively and results in few missed doses and low default rates (only 2%). (d) Linkages with Community Health Workers – Those linkages are well developed and functional, and CAP-TB works very closely and effectively together with the community health workers. (e) Replicability – CAP-TB project presents a series of originalities, such as strengthening the local expertise, providing community-based and nutritional support to the patients, putting focus on IC and creating & fomenting PPM linkages. Following the national NTP director these initiatives have to be continued and if possible replicated, after the ending of the CAP-TB project. (f) Collaboration and Coordination – CAP-TB collaborates with all implementers in TB/MDR￾TB prevention and control. The question has to be raised of the readiness of the field officers in Rangoon to take on more program management responsibilities, limiting the role of FHI360 regional office to backstopping, advising, and assessing. The answer to this question is related to the size and experience of the staff. While the Burma FHI360 and USAID/Burma officers seem ready to take on a more autonomous role, the pros and cons (including funding and contractual arrangements) need to be directly examined prior to initiating any change. 17 (a) Capacity strengthened at the community level –CAP-TB has successfully piloted the country’s community supporters, which is critical to meet the shortage in human resources. They are capacitated and intensively used to intensify MDR-TB screening, referral, contact tracing and treatment adherence. (b) DOT - is appropriate to the local situation and designed to meet individual patients’ medical and personal needs. DOT community providers are trained and mentored by CAP-TB staff, and provided with tools such as a home visit form to guide them and to ensure quality services. A relationship of mutual trust has been built which motivates patients to comply with their treatment, and produces excellent adherence, few missed doses19 and low default rates (2%), although the SLD treatment is very toxic. The support provided by the DOT providers is assessed monthly by the IAs, and the quality is found to be good. The evaluation team witnessed multiple community DOT provider-patient interactions and confirmed that patients rely on their DOT providers for medication as well as support. (c) Relationship with Health Providers – The working relationship with the implementing partners and collaborating hospitals is very cordial and constructive, as was observed in the field. CAP-TB developed a home visit form as a tool to guide the community supporters to deliver quality care of services. The community approach is well understood by the IAs and the staff of the project; however, little documentation was found with respect to the key factors of the functioning of the community approach; similarly little documentation exists of its potential effect on patient adherence, patient satisfaction, and the outcome of patient treatment. (d) Community Mobilization – CAP-TB has been able to elicit a significant community involvement in MDR-TB prevention and control, especially through the outreach to patients, the educational methods used, the involvement of cured MDR-TB patients and of local celebrities in the health Information, Education and Communications (IEC) activities. CAP-TB’s IEC work is very good; in particular, the manuals, training materials and posters are outstanding from a pedagogical and technical point of view. In 2013 CAP-TB reached 16,269 beneficiaries in 17 townships and used 20,745 IEC materials; however no data are available on the impact of those IEC messages on the prevention, diagnosis and treatment of MDR-TB patients; the sole indications of effectiveness are indirect ones, such as the high referral figure; the high satisfaction of the patients with the package of services provided (information provided by MHAA staff), and the low initial default rate (4% in Mayangone township in 2013), the latter is also suggestive for a good community support of the MDR-TB patients. The involvement of local celebrities, such as Sai Sai Khem Len, in photo-shoots or other opportunities in Rangoon and Mandalay, to promote the “Cover Your Cough” campaign is a demonstration of the effective community mobilization efforts by the FHI 360 country office. 19 This is a rough estimate provided by the IAs, but no objective monitoring data could be gathered. Qu 2: To what extent and how has the CAP-TB community-based approach contributed to TB and MDR-TB prevention, diagnosis and treatment? 18 Figure 1: CAP-TB Burma’s “Cover your Cough” campaign. While a challenge, many opportunities such as targeting women with health education messages and services in places where they frequent (such as market places and clothing factories) have been missed. Targeting messages and services to the needs of women has unfortunately not been highlighted enough in the CAP-TB program design; it is not clear how many female TB suspects20 have missed screening for that reason. (e) Follow-Up during Treatment - The CAP-TB project correctly focuses on preventing the transmission of resistant TB strains through early case finding and opportune treatment. Shortening the delay before starting SLD treatment could be an effective preventive approach. CAP-TB has introduced GeneXpert testing to speed up the case detection; however, the official national policy still obliges to carry out culture confirmation and DST before starting SLD treatment, thus losing the prevention advantage of this diagnostic tool21. The effectiveness of this prevention is also hampered by the lack of SLDs22. Consequently the prevention of the transmission of resistant bacilli is insufficient, but the possibility of the CAP-TB project to intervene is limited, given that both issues are outside the project’s influence. Fortunately the CAP-TB project gives full attention to IC, which has an impact on the transmission of resistant bacilli. In 2013, 262 MDR-TB patients from 13 covered townships were evaluated using check lists, and educated about household infection measures; 98% met quality IC standards. However, cough hygiene and IC at home, particularly for MDR-TB patients prior to treatment and on waiting lists for treatment initiation still pose a challenge, because many patients reside in small, congested houses. 20 The distribution by age and sex of the TB suspects would have been very informative to assess the case detection by age and sex, unfortunately this was not available in the project areas. 21 It is still subject of debate if an Xpert+ result might suffice to start SLD treatment, as is the case in Thailand. 22 Presently there are numerous MDR-TB patients on the waiting list due to a lack of second line drugs (SLDs). Since the end of Feb 2014 the MDR-TB screening algorithm has been opened to new cases; previously it was restricted to the CATII patients. This decision will increase the number of diagnosed MDR-TB cases and will create an even greater issue for the authorities as they are trying to procure SLDs for all in need. It is clear that the optimal outcome would be that all individuals requiring SLDs would be able to acquire them, but if that is not possible, the government might eventually feel obliged to take desperate measures such as slowing down MDR-TB diagnostic activities or using treatments they know are less appropriate for patients on the waiting list. FHI360 and its implementing partners are aware of this dilemma, but have been unable so far to contribute to any solution given their budgetary constraints. 19 Although the CAP-TB project focuses in principle on MDR-TB prevention, it is not certain that the project gives sufficient attention to the treatment performance of the CATI and especially the CATII patients, in order to effectively prevent the development of MDR-TB. Problem areas found were: the lack of monitoring of the occurrence of major treatment side effects and the actions undertaken by the DOT providers, the low detection rate of contacts screened, and the insufficient attention to the IC measures by those who are on MDR-TB treatment waiting lists23. (f) Adequacy of Documentation of the Key Factors & Effects of the Community approach – The CAP-TB project has got unique experiences in Burma such active involvement of the 4 IAs, and community-based achievements, but also important challenges such as the long delay before treating the diagnosed MDR-TB patients due to lack of SLDs. These facts are documented in great detail in the annual performance reports; however the evaluators have not found hard data on the key factors of the successes and weaknesses of the project implementation, nor of the effects of the community approach on patient adherence, delay, regularity of treatment intake, intensity of IC measures, etc. The evaluators were informed by the field staff in Burma that they are not fully aware of the strengths and lessons learned from the CAP-TB projects in the other focus countries, nor are the IAs in the other focus countries aware of the successes and challenges met in Burma. Furthermore, 3 of the 4 IAs are carrying out the identical activity of “Conducting community outreach, finding and referring TB suspects”. Each IA has its own style to implement these activities; creating thus a kind of natural experiment. A comparative monitoring of the performance of each of these activities could have been possible, in order to optimise the performance; unfortunately neither the FHI360 country office nor the regional FHI360 offices have carried out any comparative monitoring or operational research on these topics. (a) Management of the technical assistance and capacity building inputs – The FHI360 country team has shown evidence of great management skills and high motivation and is very effective in providing TA and capacity building, as evidenced by the degree of satisfaction expressed by the field staff of the IAs and the targets reached (or estimated24): the number of individuals trained in TB-case finding activities has largely surpassed the target (achievement 221%), while the number of individuals trained in programmatic management of MDR-TB has reached the target (achievement of 103%); and 100% of the local organizations were provided with TA for strengthening community-based approaches for PMDT. With respect to supporting the diagnosis, the targets were not reached as no individual was trained in laboratory diagnosis techniques, and no laboratory was provided with TA for the roll-out of new diagnostics; this was due a management decision taken given a change in needs and a delay in arrival of GeneXpert (arrived only in August 2013). The country assessment by FHI360, the annual performance reports and the annual work-plans show that the planning for the next year is performance-informed, and that the TA and capacity building inputs are decided in function of the needs. 23 This concern is very important, as the lack of second-line drugs may result in very long waits, during which time MDR-TB patients continue to transmit the bacilli to those with whom they have contact. 24 Given that no single MDR-TB patient has finished yet the full treatment schedule, and thus no treatment outcome indicators can be determined, this analysis is called “intention to treat analysis”. Such analysis is based on the initial treatment assignment and not on the treatment eventually received. This technique is frequently applied in experimental research. Qu 3: Are the management arrangements optimal1 for achieving project objectives efficiently and effectively? 20 The management of CAP-TB in Burma is optimally efficient, as the treatment cost is about equal to the national average, and is nearly identical between the various IAs (see Table 1), and the project expenditures focus principally on the primary processes and less on secondary processes25 (Figure 2); 54% of the total cost is attributable to drugs for TB treatment and side effects, while the rest is spent on infection control, health education, support to the patient, etc. (see details in Table 1). Table 1. Comparative table of average cost of providing a full SLD treatment by the IAs (MMA, PGK and MHAA) in comparison with the national average in Burma. Figure 2. Proportional distribution of the expenses by the 3 country offices, 2013. (b) Communications with other implementing partners, key stakeholders, and RDMA – FHI360 is well aware of the planned and realized activities of the other implementers and key stakeholders in the country, and of their MDR-TB control performances, and adapts its own schedule and agenda to their activities. The IAs declared to receive sufficient information on issues related to M&E and coordination through the meetings organized by FHI360. The relationship with USAID/RDMA has become more complex since the CAP-TB grant was awarded; indeed in July 2012, USAID established its mission in Burma after a 23-year hiatus. With the advent of the US-Burma partnership for Democracy, Peace and Prosperity, the practical day-to￾day focus of control of USAID activities in-country shifted from USAID/RDMA in Bangkok to the 25 See WG7 Theme: Effectiveness and efficiency – Leadership and management issues. www.eua.be/typo3conf/ext/bzb_securelink/pushFile.php 21 local Mission, and USAID/Burma is gradually taking more ownership of its portfolio. In the light of this changing picture, USAID/RMDA has not yet clearly defined the lines of communications and the roles of responsibility and oversight of FHI360 Bangkok and the FHI360 country office, especially now that USAID/Burma is adequately staffed and eager to directly manage CAP-TB activities in Rangoon and Mandalay. (c) Monitoring, data collection and management, data quality assurance, reporting design and processes, and use of data – The data gathering and reporting system works well in Burma: fieldworkers collect and compile data, report to field officers, and get regular feedback. Given that this data management system unites all essential elements needed to inform programs, it could function satisfactorily; however the data management at field level is less optimal: indeed data are sometimes flawed, and analyses to determine trends are not performed or are not correctly interpreted at the level of the IAs. In several places the evaluators noticed a lack of a reliable denominator. Although several 1 day trainings had been imparted on Data Quality Assessment (DQA) and data coordination and communication, the IAs do not seem able to analyse and interpret data adequately. Furthermore the use of good quality data as source for evidence-for￾action does not seem to be appreciated sufficiently by the field teams. The evaluators did not find evidence that the regional FHI360 office and the national FHI360 office stress enough this issue of evidence-informed action taking. (d) Sufficiency of level of engagement and types of relationships established between lead organizations and in-country implementing partners with government counterparts - FHI360 country office has been able to establish an effective relationship with NTP, WHO and other authorities and with the four in-country IAs. The relationship between the 4 IAs and the partners and stakeholders is remarkably cordial, as was observed by the evaluators and corroborated by the IAs during in-depth interviews. The day-to-day functioning is going on very smoothly without critical incidences that could impact on the efficiency or the effectiveness of the project. (e) Facilities and NGOs engaged in implementing the model – Local NGOs and other collaborators such as employers of large factories, are engaged in implementing the CAP-TB model. In January 2013 an organizational capacity assessment was carried out by an FHI360 APRO consultant; 14 members from the private sector NGOs participated in the assessment. Following info provided by the IAs, the cooperation with the private sector workers is going on smoothly. (f) Annual planning process – RDMA and FHI360 regional and country offices have adequately adhered to the USAID planning process. (a) Types of common users, use of information from Knowledge Gateway – CAP-TB gathers strategic information generated through routine data and operational research studies. The lessons learned are then regularly disseminated to clinicians, public health practitioners, and government officials. The CAP-TB Knowledge Gateway is central to the dissemination of the strategic information. This Knowledge Gateway is a very interesting approach; however, we found little evidence of its use or of its perceived usefulness, although the field staffs want to receive regularly updated information on TB control and management. The evaluation team is not convinced of the cost-effectiveness or usefulness of the Knowledge Gateway for this purpose. (b) Limitations (i.e., accessibility/friendliness) in the Knowledge Gateway from the users' perspectives - All field staff in the focus countries, inclusive Burma, reported Qu 4: To what extent and how has the strategic information generated by the project and Knowledge Gateway information been used? 22 difficulties in using the Knowledge Gateway system, mainly due to linguistic constraints, and limited access to internet in various places. (c) Gaps in strategic information/data for strategic decision making - In Burma, the main gaps in strategic information/data for strategic decision making are lack of: - stratified data on case finding following time, place and person characteristics - data on patient delay (total delay, and delay stratified following patient, access and health service) - data on equity (see the disparities in the gender ratio of the at risk persons) - crude26 and stratified27 data on case holding, such as default rates, and profile of the defaulters - monitoring of the effectiveness of contract tracing - data on the cost-effectiveness of the MDR-TB treatment - comparative data on the performance of other implementers of MDR-TB prevention and control in other provinces, to avoid spending efforts on developing and trying-out approaches that have already shown their degree of being implementable. In this way CAP-TB could gain time and also effectiveness, efficiency and equity of the MDR-TB prevention and control while adapting measures that have been proven operational elsewhere, or modifying less effective ones, or introducing novel approaches. The CAP-TB annual performance report mentions and intense efforts to review the existing monitoring and data reporting system, but the above mentioned gaps are not adequately addressed. (d) Design, use and limitations of key strategic information, including gender assessment, in programming for results – A community survey on consumer demands and health seeking/purchasing behavior related to diagnosis, treatment, and drug management was undertaken, providing a sound basis for adapting and improving IEC messages. However, the degree in which this information and especially the one provided by the routine surveillance data, is used to steer the planning and implementation of the activities is not obvious. Surely at the level of the IAs, strategic decision making is insufficiently based on data; furthermore lack of stratified data impediments gender assessment of case finding and case holding activities. (e) Use of strategic information by project partners – To reinforce the use of the routine data to improve programming, the project has organized several data management training sessions. However, despite these trainings and the regular data quality assessment checks performed at multiple levels, the IAs’ ability to interpret the routine information correctly is still basic, as was observed during the presentations by each of the IAs, and during the field visits. Furthermore, there is also a concern about data confidentiality due to lack of project-specific record storage, as mentioned in CAP-TB’s annual performance report. However this issue was not mentioned at all to the evaluators, which might imply that the IAs are not aware of it. Given the stigma attached to MDR-TB this is a serious concern that merits to be tackled urgently. Ensuring financial and technical sustainability of supported programs remains a challenge, especially now that USAID financial support to CAP-TB is decreasing. This decrease is a threat to meeting the CAP-TB PMP targets, but can also be seen as an opportunity for the local and national governments 26 Crude data are presented in an unanalyzed form; not adjusted to allow for related circumstances or data. 27 Stratification is defined as the process of partitioning data into distinct or non-overlapping groups Qu 5: What measures/mechanisms have been put in place to achieve sustainability and which still need to be addressed? 23 to (at least partially) fund the MDR-TB control and prevention in the covered project areas and for the project to optimise the cost-effectiveness of its interventions. To achieve sustainability, several aspects of the MDR-TB control have to be addressed, namely the performance of MDR-TB infection control, the service delivery system, the workforce, the information systems, and the use of routine data. This routine data would help steer program management, financial security, and the leadership/governance of the MDR-TB control. (a) Level of integration of project activities with existing national TB priorities and health systems – CAP-TB project concepts and strategies have been taken into consideration in the development of the national MDR-TB guidelines, which is a very positive and encouraging sign towards sustainability. Following Dr Thandar Lwin, the strong points of the CAP-TB project are its community-based approach, PPM linkage, and its capacity to organise the local NGO’s; these approaches are fully in line with national priorities. Providing patient support is an excellent initiative taken by CAP-TB, which serves as a model of holistic care. This initiative will be implemented by the Global Fund from April 2014 onwards, but only for the new batch of MDR-TB patients. The project activities of MDR-TB prevention and control are well integrated in the public as well as private basic health system and hospitals, as could be observed in the field. (b) Measures to help ensure country adoption or adaption of the approaches used in implementing the CAP-TB model – The capacity of the communities to support project activities is a positive factor for the sustainability of the CAP-TB approach. CAP-TB has empowered the communities through capacity building and strengthening the skills of the community leaders and community volunteers, and though organizing information sessions addressed to the whole community. The broad participation base of the community, improves its chances for long-term sustainability28. The limitations of the use of the information spread by this Knowledge Gateway has been discussed above; and are such that this initiative risks to end once the CAP-TB project stops, unless it would receive support by the government or another local organization and would be completely transferred to Burma. The continuous supply of quality guaranteed drugs, laboratory equipment and supplies is an important factor for sustainability. The problems with the SLD supply have been mentioned above. (c) CAP-TB project system to acquire data necessary for collecting a cost-effective analysis of implementation of the model - The costing has been detailed in Qu3(a), however the evaluators have no knowledge of any cost optimising initiatives being taken. (d) Ability of local partners to leverage other partnerships and resources as a result of CAP-TB to implement approaches used in the MDR-TB model – CAP-TB has created mechanisms for the hospitals to actively interface with the communities; this is an important step towards making the community outlook of CAP-TB sustainable. Creating a dialogue with beneficiaries, partners and governmental authorities on exit strategic alternatives could be of great help to promote the sustainability of the CAP-TB approach; the evaluators explored if such dialogue has been started already, but the answers were evasive; consequently the evaluators do not know if such dialogue has started at all. 28 Apina FG. What communities need from global and national stakeholders to ensure engagement of community groups. Communication at the 15th core Group meeting of the TB/HIV Working Group; 3)4 Nov 2009, Geneva. 24 CAP-TB’s involvement with the private sector is another approach to sustainability. MMA is a key Public-Private Partner (PPP), where presently 237 private-sector partners join hands with NTP, thanks to USAID support; this number is 228% higher than the original target. 1.4.1.2 Burma: Recommendations 1. The data management at the level of the IAs shows important deficiencies at the level of the quality of the data, the analysis and interpretation, the feedback and the use for program management. Although several 1-day trainings have already been organised by FHI360, more training is needed: the target groups should be the implementing partners and also the supervisors; the duration should be at least 1 week; the course should be focused on evidence￾informed program management instead of being focused only on data management, the supervisors should be actively involved in the training as facilitators, there should be a post￾training implementation plan, and also a monitoring assessment of its implementation. 2. Given that presently no data are available on the occurrence and causes of treatment delay of MDR-TB patients, it is recommended to systematically register the delay and the eventual causes, and search for an approach to decrease it. 3. More attention should be given to the different groups of highly vulnerable people: - PLHIV: monitor the case finding and case holding performance in PLHIV - Migrants: register the residence of the MDR-TB suspects, and especially monitor those residing outside the township, as they have a non-negligible probability to default - Elderly: the causes of the low notification rates should be explored - DM: the causes of the low co-morbidity figures in the project should be explored 4. Given the low number of MDR-TB cases found through contact tracing, more attention should be given to this activity, especially in the households of MDR-TB cases residing outside the township. 5. Presently there is no systematic registration of the missed treatment doses. It is recommended to instruct the DOT providers to register any missed dose and the reasons. 6. Register the major side-effects and also the actions taken, and regularly carry out medical audits of these events and of the actions taken. 7. Several IAs carry out similar activities, although no systematic comparative follow-up is being done in order to optimise the project implementation. Hereto it is recommended to take advantage of this opportunity of such ‘natural experiment‘ and carry out implementation researches. 8. Monitor the impact of the educational efforts, and carry out cost-benefit analyses aiming to optimize the educational efforts, messages and financial inputs. 9. To optimize the prevention activities of CAP-TB, more attention should be given to the performance of the first-line treatment, especially of CATII cases. 10. Study the key determinants of the community approach. 11. Give special attention to the IC of MDR-TB patients on the waiting list. 12. Take care of the confidentiality of the patients’ data (study the flow of the data, look at eventual gaps in confidentiality, breach those gaps and train the staff to scrupulously respect the confidentiality of the patients). 13. Inform the implementing partners about the phasing out and try to define an optimal phasing-out strategy in agreement with the partners. 25 1.4.2 CHINA 1.4.2.1 China: Key Findings (a) Human Capacity and Supplies/Commodities - CAP-TB has been very successful in strengthening MDR-TB program management, but less in reinforcing data management. Various trainings were organized for private sector partners, pharmacists, health service center staff of the XiShan district, community outreach workers, and also patients and their families. Several topics were covered such as IC, new diagnostic tools, MDR-TB case detection and management, TB/HIV diagnosis and treatment, and TB counselling. CAP-TB project has displayed a lot of creativity in updating the capacity of such a broad range of potential trainees. The capacity building follows an exemplary pedagogical sequential approach, namely: needs assessment, training inspired by that assessment, follow-up, refresher courses and continuous training through supervision, and also a possibility to consult the trainers via skype, in case of difficulties in the implementation. The targets for training people in active case finding methods related to TB were greatly exceeded (336%); the overachievement is due to limited initial target setting for Fy13 and consequent supplementary engagement of the Women’s federation community workers, the Blue Sky peer counsellors and No3 Hospital staff; this extra engagement reflects the increasing involvement of the CAP-TB staff in Yunnan province. Training in programmatic management of MDR-TB was successfully organized, and the target was also largely exceeded (220%). The CAP-TB office in Yunnan invited IUATLD, through its office in Beijing, to provide expert MDR-TB management trainings, organize regular case review sessions, and contact experts via Skype for clinical consultations of severe MDR-TB cases. Attention has been given to minimize duplication of efforts when similar training activities were conducted by other organizations, such as IUATLD, in the same catchment area. Interviews with field staff learned that the trainees are very much satisfied with the content and approach of the trainings, and felt to be well prepared for their involvement in MDR-TB management. Training in data management exceeded also its target (290%), although the performance still shows various gaps (see comments in chapter Qu3 (c)). CAP-TB has provided 1 GeneXpert machine to Yunnan CDC, as well as cartridges, to enable patients to be initiated on treatment quickly; however, in China confirmation by culture and DST are still required29 before MDR-TB treatment can be started, leading to treatment delays (however, this delay is outside the intervention possibility of the CAP-TB project). CAP-TB provided a limited number of cartridges during the initial phase; however, further cartridge procurement has been devolved to the country, which has taken up this responsibility. CAP-TB provided also 1 LED fluorescent microscope30 to Yunnan TCC. 29 This is a pure policy decision which is identical to the one taken by NTP/Burma, but different to the one taken by NTP/Thailand. 30 Light-emitting diodes (LED) have been developed to offer the benefits of fluorescence microscopy without the associated costs. In 2009, the evidence for the efficacy of LED microscopy was assessed by the World Health Organization, on the basis of standards appropriate for evaluating both the accuracy and the effect of new TB diagnostics on patients and public health. Smear microscopy remains the mainstay for TB diagnosis in disease endemic developing countries. However, replacing conventional light microscopy (LM) with fluorescence microscopy (FM) would immediately improve TB case detection, speed up sputum examination, and reduce workload, because FM detects 10% more TB cases than LM and requires only 25% of the time taken to read a Ziehl Neelsen (ZN) stained smear. Qu1: To what extent and how has CAP-TB strengthened MDR-TB management? 26 (b) Case Management and Referrals for MDR-TB – The quality of care for MDR-TB patients is very good. The target of the number of MDR-TB cases diagnosed was almost achieved (86%). CAP-TB successfully strengthened referral for patients at risk for MDR-TB31: the achievement rate was 132%, and the effectiveness 94%. This success is due to a higher than expected commitment from TB staff at the community and district level, and from private clinics. A contributing factor for this success might be the referral slip, through which those who do not present at the referral site within one week are traced. The evaluators could observe in the field that this referral slip is effectively being used. (c) Diagnosis, Treatment and Follow-up – c.1. Diagnosis The project objective of increasing detection of MDR-TB cases in areas where the model MDR-TB program is implemented has been nearly met (86% of diagnostic target met). This success was due to the increased referrals made. The diagnosis is strengthened through capacity building of the medical and laboratory staff in new diagnostic tools. There is an important gender imbalance in the case detection (male to female ratio = 4), which has not received the necessary attention by CAP￾TB. CAP-TB project focuses on preventing the person-to-person transmission of TB strains which are already resistant to the first-line drugs, through early identification and opportune treatment of the MDR-TB patients. Shortening the delay before starting SLD treatment is an effective preventive approach, although no data could be obtained on the total delay, neither on the determinants of the delay of MDR-TB cases. That delay is probably long, as the total delay for non-resistant TB cases reaches an average of 60 days. Contact screening is a very intense and effective activity of TB control in Yunnan province; in 2013 not less than 99.97% of the contacts were screened, of which 334 had symptoms of TB, 45 TB were diagnosed (14 smear positive, and 31 smear negative). The project has put special efforts to screen the highly vulnerable members (PLHIV, elderly, migrants and DM patients) of the target population, although with small success. - PLHIV32: CAP-TB and the IAs put great attention to this co-morbidity, as the 93% of the HIV/AIDS patients were screened for TB symptoms, and 0.51% diagnosed. - The case notification rate among the elderly in Yunnan province is very low in the very old age groups. CAP-TB is aware of this problem33, and has the intention to shift the screening focus to the clinic setting where the elderly are seen as outpatients. This shift can only be effective, if the elderly consult Out-Patient Departments (OPDs) in sufficiently big numbers. An easy way to examine this could be through analyzing the OPD registers and/or the TB lab registers. Another approach could be to organize focus group discussions with male and female elderly people to explore the reasons 31 Treatment failure patients (TAF), treatment defaulters (TAD), patients who remain smear positive at two months, close contacts of known MDR-TB patients, and PLHIVs. 32 In Yunnan province 12,043 HIV/AIDS cases were reported between Jan and Dec 2011, which accounts for about 1/6th of the total number for whole China, and shows a 14% increase since 2010. 92.6% were screened in 2013, of which 0.5% were found TB positive 33 The project had attempted to increase case detection in elderly in Yunnan through community efforts focused on areas where the elderly congregate in the city; in 2013 the CAP-TB project had focused on training four outreach workers who spent most of their time in the communities, conducting outreach in pharmacies and private clinics. These efforts were very time-consuming, but did not result in an increase in TB suspects or TB cases. Therefore, CAP-TB shifted its focus for 2014 to hire project staff who are based in Xi Shan District CDC, and who utilize the same strategy that was developed during 2013 (tested by the outreach workers), but who were specifically focused on working hand-in-hand with Xi Shan CDC. 27 of the low detection. Anyhow, before launching a different screening strategy, the reasons for the failing present strategy have to be thoroughly examined, what has not happened so far. - A very promising initiative of bidirectional screening of diabetes mellitusII (DM) patients has been started 6 months ago in 10 community centers, enabling those centers to screen DM patients for TB and TB patients for DM. Interim data show low DM rates in some community centers, although a high number of co-morbid cases were expected34. CAP-TB could not explain the reasons for that finding, which merits further investigation. Given the increased risk of failure, relapse, and death among diabetics with TB35, this bidirectional initiative has great potential to contribute to a better control of the MDR-TB problem. - Few Migrants/mobile population are diagnosed in the project area. It is a big challenge to reach them, particularly the males as they work during the day and the illegal migrants which they face additional problems in gaining access to health care services. To increase the case finding among migrants, CAP-TB developed and is piloting a program based on the “buddy” concepts. The evaluators fully support the piloting of this concept. c.2. Treatment 43% of the diagnosed MDR-TB cases did not start treatment; this initial default rate is very high, and has also not received sufficient consideration by FHI360 country office and the implementing partners. Following YATA, the reasons for this high initial default are: geographical or temporal limited accessibility of the treatment centers, fear of stigma, preference for Chinese medicine, and discouragement due to long delay before the start of the treatment. This very high initial default rate of MDR-TB cases is in complete contrast with the very low initial default rate of the non-resistant TB cases (<0.5%). This discordance has not received the necessary attention by CAP-TB. The CAP-TB project has scaled up the quality of the treatment, through training the staff to address major concerns of patients, improve treatment adherence and increase the quality of life of the patients by means of a package of support services. The medical staff has given the possibility to consult their teachers through Skype in case of difficult clinical problems. Thanks to CAP-TB, a change in mentality and behavior among the TB doctors took place: they use only standardized treatment schedule, do no more abuse fluoroquinolones, their capacity in clinical management is enhanced, they give more attention to side effects of first and specially second line drugs, and regularly review the treated MDR-TB cases with colleagues of other hospitals. The evaluators could confirm this most remarkable change of mentality and behavior. CAP-TB provides to 28 MDR-TB patients a monthly ‘living support package’ that consists of nutritional support, transportation support, home IC measures, and adherence and adverse effects counselling. That nutritional support is for many MDR-TB patients their only effective income during the intensive phase of MDR-TB treatment, and they could not continue the treatment course without that support. c.3. Follow-up CAP-TB’s partners implement DOT correctly, although there is a problem with some hospitalized patients and HIV/TB patients who receive only unsupervised treatment after their discharge. This might be the cause of the high default rate (= 33%) for reasons of serious side effects and subsequent loss of confidence in classical MDR-TB treatment. 34 Singla R, Khan N. Does diabetes predispose to the development of multidrug-resistant tuberculosis? Chest. 2003;123(1): 308-309. 35 Kang YA, Kim SY, Yo KW, et al. Impact of diabetes on treatment outcomes and long-term survival in multidrug-resistant tuberculosis. Respiration. 2013;86(6):472-478 28 The case fatality rate is very low (= 2%), which suggests either exceptionally good care or insufficient follow-up36. The initial results of the treatment performance seem good, although it is too early to assess the final outcomes, as none of the individuals has finished their treatment yet. The number of treatment doses missed is low, which is a good indicator of the effectiveness of the DOT in increasing treatment adherence; however, this indicator could not be objectively verified. (d) Linkages with Community Health Workers – These linkages are very well developed and highly functional, because CAP-TB’s Community outreach team provides trainings and refresher trainings for outreach. (e) Collaboration and Coordination – CAP-TB is strongly promotes collaboration with all implementers of MDR/TB prevention and control in the covered areas. CAP-TB country office submits quarterly reports to the regional FHI360 team for review, analysis, and feedback. There is regular contact via skype and e-mail between the regional and country FHI360 offices; these contacts are felt by the country office as very supportive. In Yunnan, the present rapport between country and regional FHI360 office staff is very satisfactory to all involved parties, as learned through in-depth interviews, and should best be maintained in the present state, because a too early transition might be detrimental to the country program. (a) Capacity strengthened at the community level - CAP-TB has demonstrated great effectiveness and creativity in organizing training sessions for community outreach workers, assessing their residual knowledge, developing a home visit form and a community outreach flip-chart. The community outreach workers conduct outreach education and community events, promote uptake of TB services and visit TB patients in their homes. An estimated 5,636 people were reached with TB prevention, diagnosis and treatment messages and 9, 636 pieces of anti-TB IEC materials were distributed. 64 TB patients were visited to support treatment adherence and check on treatment adverse effects. A home visit form was developed to deliver quality services. A community outreach flip-chart tool was developed to spread information on basic knowledge, risk assessment evaluation, TB diagnosis and treatment and CAP-TB services. MDR-TB patients are contacted through home visits and/or follow-up phone calls to assess the quality of the support received, in order to incorporate the obtained information in further strategic up-scaling. The CAP-TB project is regularly carrying out enquiries of the target public concerning their knowledge about TB. Through in-depth interviews of the target public, the evaluators could verify the capacity strengthening activities and their good impact. (b) DOT – Although DOT is correctly implemented in the project area, some hospitalized patients do not receive DOT after their discharge, which is certainly a major problem, as it is proven that universal DOT for tuberculosis is associated with a decrease in the acquisition and transmission of resistant tuberculosis37. 36 It could be that a number of defaulters had died, and are thus misclassified 37 Moonen PK, Quitugua TN, Pogoda JM, et al. Does directly observed therapy (DOT) reduce drug resistant tuberculosis? BMC Public Health 2011;11: 19. Qu 2: To what extent and how has the CAP-TB community-based approach contributed to TB and MDR-TB prevention, diagnosis and treatment? 29 It is not known how many of the MDR-TB defaulters patients have received DOT, so assessing the effectiveness of DOT is impossible. Furthermore CAP-TB is not directly responsible for implementing DOT, and can thus not be blamed for those instances where patients receive unsupervised treatment; however, a closer involvement of CAP-TB in patient follow-up could certainly prevent or at least diminish the occurrence of such instances of unsupervised treatment. (c) Relationship with Health Providers – There is a superb relationship with the IAs and hospitals, as could be observed in the field. CAP-TBs’ community approach is well accepted by the provincial authorities, as declared by the provincial deputy-director of TB, and by the staff of the project. (d) Community Mobilization - CAP-TB has been able to promote a highly significant community involvement in MDR-TB control and prevention efforts. • CAP-TB’s Information, Education and Communications (IEC) work is most impressive; in particular the manuals, training materials and posters, which are outstanding from a technical and pedagogical point of view. CAP-TB cannot reach everyone in the community through one-to￾one and small group outreach activities, therefore ‘edutainment’ events are organized, enacting scenarios on key tuberculosis issues. These edutainment initiatives are an illustration of the very creative mind-set of CAP-TB staff in Yunnan. • CAP-TB supports an MDR-TB patient to develop an internet-based TB QQ38 network, called the “57 Zone”, which is remarkably successful in sharing information with fellow patients, strengthening cohesion among patients and providing peer support. This is an excellent example of involving affected individuals to help themselves manage their disease with the aid of experts, and providing support and networking in a non-judgmental setting. Such network is very innovative for China, as was recognized by the vice-director of NCTB in charge of MDR-TB management, who would like it to be expanded throughout China. (e) Follow-Up during Treatment - The CAP-TB project focuses on preventing transmission of resistant TB strains through early case finding and opportune treatment. Contact tracing is one of the prevention activities; in 2013 19,547 close contacts of new smear positive TB patients were confirmed in Yunnan; 99.97% were screened, of whom 334 were found to be clinical TB suspects and 13% diagnosed as TB case. However, the number of detected cases among contacts of MDR-TB cases is not known. CAP-TB gives full attention to IC: outreach and community workers visit TB and MDR-TB patients in their homes to assess IC practices, and to educate MDR-TB patients about household IC measures. Significant attention is given to teaching IC to the staff, outreach workers, patients and their families. Yunnan TCC holds regular monthly patient and family meetings to discuss household IC. FHI360 developed an IC tool for the household level to guide TB staff to assess IC through key questions and provide advice accordingly. Assessment of IC practices will be carried out every 3-6 months at every MDR-TB household, starting in 2014. Observations made in the health centers and hospitals, suggest that the IC practices are good, although cough hygiene and IC practices at home and in public places still pose a challenge (in Yunnan only 29% of the MDR-TB patients on treatment wear masks), particularly for MDR-TB patients prior to treatment and for those on the waiting list. IC assessment is also being carried out in health facilities supported by CAP-TB. IUATLD, CAP-TB’s partner, identified IC risks at Yunnan Province’s TB Clinical Centre, and the provincial government promptly implemented the recommendations to decrease the IC risks. 38 Tencent QQ, popularly known as QQ, is an instant messaging software service developed by Tencent Holdings Limited. QQ offers a variety of services, including online social games, music, shopping, microblogging, and group and voice chat. It is widely used in China: as of 20 March 2013, there were 798 million active QQ accounts, with a peak of 176 million simultaneous online QQ users. 30 (f) Adequacy of Documentation of the Key Factors & Effects of the Community approach - CAP-TB project in China has unique experiences such the impressive IEC and the active involvement of the partners. These experiences are documented in great detail in the annual performance report; however the evaluators did not see objective data on the key factors of the project implementation, neither of the effects of the community approach on patient adherence, delay, regularity of treatment intake, initial default, etc. The evaluators were informed by the field staffs in Kunming that they are not aware of the strengths and lessons learned from the other focus countries, neither are the IAs in the other focus countries aware of the successes and challenges encountered in Yunnan. (a) Management of the technical assistance and capacity building inputs - The CAP-TB country team has great management skills and high motivation, to reach the project objectives. The effectiveness of the actions undertaken is very good, and nearly all activities surpassed largely the target. However, there are still some issues of concern, namely the gender disparity in case finding, a very high initial default rate and a high default rate. Apparently little attention has been given so far to these issues. In such circumstances, the management arrangements to reach the project objectives effectively can be labelled as ‘very good’ but not as ‘optimal’. Treatment estimates show that the IAs of CAP-TB project produce great quality services for an average treatment cost of 4981$ per patient. This treatment cost is 23% lower than the national average cost ($6,50039). A project aiming to prevent and control MDR-TB can be labeled as “efficient” if and only if its expenditures are focused on the realization of its main objectives. Such is certainly the case in Yunnan where 67% of the costs are due to treatment, as shown by table 2, and the management arrangements are optimal for achieving project objectives efficiently. The CAP-TB project has not provided evidence that the cost could still decrease while maintaining good implementation quality. Given that the USAID financial support is decreasing, such an exercise in trying to decrease the cost while maintaining the quality might become soon necessary. 39 Figure provided by the sub-director of NTP Qu3: Are the management arrangements optimal for achieving project objectives efficiently and effectively? 31 Type of service Unit cost (in US $) in Yunnan Package of services for MDR-TB patients Diagnostic 763 Examination before getting in-patient 69 Treatment (in-patient) 522 Second line drugs (domestic) 2,143 Injection (out-patient) 290 Treatment for side- effects 653 Check-up examinations (OPD) 541 Transportation - Food - Estimated total cost for one MDR-TB patient (24 months) $4,981 Table 2: Estimated cost of full MDR-TB treatment, Yunnan province, 2013. (b) Communications with other implementing partners, key stakeholders, and RDMA - In-depth interviews with the implementing partners learned that they are very much satisfied with the frequency, amount and variety of communications they receive from CAP-TB. CAP-TB is fully aware of the planned and realized activities of the other implementers and key stakeholders in the project area, and adapts its own schedule and agenda to their activities if needed.. (c) Monitoring, data collection and management, data quality assurance, reporting design and processes, and use of data – Following the FHI360 regional staff “CAP-TB China program has an overall solid M&E system, particularly in the establishment of M&E focal persons, clear roles and responsibilities of different partners and documentation tools in both English and Chinese”. The evaluators agree with this statement; indeed in the project area the data management system unites all essential elements needed to inform programs, and they observed regular collection and compilation of the data. Nevertheless, the following deficiencies could be noticed in the field: flaws in some data, near-absence of trend analysis, instances of incorrect interpretation of the data; furthermore the field staff did not always seem to grasp the relevance of the routine data for daily management of the project, resulting in little use for decision making. (d) Sufficiency of level of engagement and types of relationships established between lead organizations and in-country implementing partners with government counterparts - The CAP-TB project staff has been able to develop and achieve an outstanding interface with all partners, and has a particularly exemplary relationship with the local clinical units, the provincial government, NTP, WHO and IUATLD. The day-to-day functioning is going on very smoothly without critical incidences that could impact on the efficiency or the effectiveness of the project. (e) Facilities and NGOs engaged in implementing the model – The project staff involves NGOs, private practitioners and other collaborators, in implementing the model. The training sessions are opened to other NGOs, which partly explains some of the over-achievements of the targets. 32 (f) Annual planning process – RDMA and FHI360 regional and country offices have adequately adhered with the USAID planning process. (a) Types of common users, use of information from Knowledge Gateway – In Yunnan province, the CAP-TB project has focused on the development and distribution of strategic information, as well as on its incorporation into MDR-TB prevention, treatment, and follow-up strategies. Social media has become the cornerstone of CAP-TB’s efforts to spread relevant knowledge on TB. However, because Facebook, YouTube, Blogger and Twitter are blocked in China, CAP-TB had to develop other avenues to spread out its messages; recently the educational platform was expanded to WECHAT public platform, which seems to be a quite popular medium among Chinese mobile users, and allows CAP-TB to reach an even wider audience. When an important TB development is distributed by the Knowledge Gateway, it is being translated immediately by the Kunming FHI360 country office and reposted to the “57 Zone” and Weibo websites. (b) Limitations (i.e., accessibility/friendliness) in the Knowledge Gateway from the users' perspectives - Difficulties were reported in using the Knowledge Gateway system, due to linguistic constraints; therefore the project developed alternative systems. The evaluators noticed field staff’s timid enthusiasm for the Knowledge gateway, for reasons of language, delay in transmission of the information and felt lack of comparative advantage. (c) Gaps in strategic information/data for strategic decision making - In Yunnan, the main gaps in strategic information/data for strategic decision making are: - stratified data on case finding and case holding indicators - data on delay (total delay, and delay stratified following patient, access and health service) - data on gender, as risk factor for case screening, treatment adherence and outcome - data on determinants of access to care, and treatment adherence - data on the active case finding in the vulnerable population groups. (d) Design, use and limitations of key strategic information, including gender assessment, in programming for results - CAP-TB builds effective linkages within the health services, creating an “enabling” environment to support MDR-TB control and prevention. Through CAP-TB’s involvement, the quality of care for MDR-TB patients has improved substantially, as was acknowledged by the staff of the hospitals visited, and by the TB Control Director of Yunnan province. However, issues remain at the level of the implementing partners in the analysis of routine data to evaluate and improve implementation in the CAP-TB supported areas and in the use of the routine data. In China the male to female ratio of MDR-TB cases is 2 to 140; however in Yunnan province it is 4 to 1. No authority or IA mentioned this remarkable finding, and apparently the reasons for this problematic differential are not being explored yet. (e) Use of strategic information by project partners – Yunnan province is the place where more use is made of the strategic information, than in the other focus countries. 40 Shenije Tang, Shouong Tan, Lan Yao, et al. Risk Factors for Poor Treatment Outcomes in Patients with MDR￾TB and XDR-TB in China: Retrospective Multi-Center Investigation. PLoSOne 2013; DOI: 10.1371/journal.pone.0082943 Qu 4: To what extent and how has the strategic information generated by the project and Knowledge Gateway information been used? 33 To reinforce the use of the routine data, the project has organized various activities such as external review of MDR-TB management, assisting to the TB Working Group and review workshops, monthly TA for supervision, and strengthen Data Quality Assurance and data analysis to Yunnan CDC and Xishan CDC, and identifying information gaps and proposing solutions. Despite these trainings and regular data quality assessment checks performed at multiple levels, IAs’ ability to interpret the routine information correctly is still basic as observed during the field visits. (a) Level of integration of project activities with existing national TB priorities and health systems - The national and Yunnan provincial authorities recognize CAP-TB’s MDR-TB control approach as being very innovative for the country and the province41. The very well developed interface between the CAP-TB project and the Yunnan provincial authorities at all levels, and between CAP-TB project, IUATLD office in Beijing and NTP office in Bejing have created a strong basis for trust and collaboration, supporting future expansion and sustainability of this project. The CAP-TB model has a good potential for sustainability given government support and buy-in (literally, as some authorities have already started to financially support some of the outreach workers, such as the IC structural modifications in No 3 hospital). Ensuring financial and technical sustainability of the CAP-TB supported MDR-TB prevention and control program remains a challenge, especially now that USAID’s financial support for CAP-TB is decreasing. This decrease is a threat for the sustainability, but can also be seen as an opportunity for the local and national governments to (at least partially) fund the MDR-TB control and prevention in the covered project areas and for the CAP-TB project to optimize the cost-effectiveness of its interventions. In Yunnan province the question of the sustainability has not been addressed sufficiently; this may be, in part, the result of a foreshortening of the project; sustainability is usually addressed once at least partial outcomes have been determined. Sustainability may be enhanced by more accurately determining the TB burden, as current assumptions seem to underestimate the TB burden, leading to low funding levels as compared to those of other diseases. (b) Measures to help ensure country adoption or adaption of the approaches used in implementing the CAP-TB model – CAP-TB project has engaged the local communities very successfully. The renovation of the building of Yunnan Province’s TB Clinical Centre by the authorities, once IUATLD had identified IC risks and had suggested modifications, is an example of such a success story. The continuous supply of quality-guaranteed SLDs and laboratory supplies is important for the sustainability; in Yunnan no gaps were observed in the supply chain. CAP-TB project has added very valuable elements to the work force in Yunnan province, and trained 810 people working at different levels of the system. It should be regrettable if these well trained staff members should soon have to leave the MDR-TB work force. The reviewers are not 41 Statements made in Beijing by the vice-director of the Chinese CDC and in Kunming by the deputy director of the Yunnan provincial health department Qu 5: What measures/mechanisms have been put in place to achieve sustainability and which still need to be addressed? 34 aware of concrete steps that have been taken or that are planned to develop an exit strategy42 with the local/national governments or the private sector. (c) CAP-TB project system to acquire data necessary for collecting a cost-effective analysis of implementation of the model – The costing has been detailed in Qu3(a), however the evaluators have no knowledge of any cost optimizing initiative being taken. (d) Ability of local partners to leverage other partnerships and resources as a results of CAP-TB to implement approaches used in the MDR-TB model – CAP-TB has created mechanisms for the hospitals to actively interface with the communities, increasing hospital’s outreach and creating trust within the community, and making it more likely that individuals will access services at a facility with trained personnel and quality medications in the future; this is an important step towards making the community orientation of the CAP-TB methodology sustainable. CAP-TB has empowered the communities through capacity building and skills’ strengthening of the community leaders and volunteers, and though organizing information sessions addressed to the whole community. This capacity of the communities to support the project activities during the project duration is a positive factor for the sustainability of the CAP-TB approach. The strong emphasis on digital communications by the CAP-TB project in Yunnan has been received with enthusiasm by the implementing partners and the authorities; however the project has not yet produced hard data to demonstrate their short and long-term impact, in order to incorporate it in MDR-TB control in a sustainable manner. Engaging the private sector in an effort to ensure quality diagnostic and treatment of susceptible TB patients could be an effective way to prevent MDR-TB cases (though the data to support this hypothesis is incomplete), and is another approach to sustainability. 1.4.2.2 China: Recommendations 1. As no routine data on the incidence, screening and treatment adherence of the PLHIV are available yet, more attention should be given to monitor the detection, treatment and follow-up of this vulnerable population. 2. There is a low notification of the elderly. The reasons should be explored and addressed urgently. 3. The bidirectional screening trial of DM and TB has ended in March 2014; this activity should continue, but the focus should be more that of an implementation research, exploring the reasons why the performance was apparently insufficient in certain centers. 4. Migrants/mobile populations are at high risk of acquiring TB and MDR-TB, of late access to diagnosis and care. This population group need more careful monitoring. 5. The delay before treatment of MDR-TB cases is not known yet; the delay should be studied, monitored carefully and addressed if found excessive. 6. There is a preoccupying underrepresentation of female MDR-TB patients. The reasons need to be urgently examined, and the findings translated into appropriate actions. 7. There is a very high initial default rate. The causes need to be explored through operational research and relevant actions undertaken to increase the acceptance of the MDR-TB treatment. 8. Given the high default rate and low case fatality rate, there is a suspicion of errors in the ascertainment of the treatment outcome. Therefore a better ascertainment is necessary. 9. In several instances, it was mentioned that there is a lack of monitoring of the MDR-TB subsequent on their discharge from the hospital. Therefore there should be a close follow￾up of all MDR-TB cases after their discharge from the hospital. 42 The evaluators raised this question in several opportunities, but no concrete answer could be obtained. 35 10. The CAP-TB project emphasizes counselling very much, but the impact of counselling is not monitored or assessed. Operational research of the impact of counselling is necessary. 11. Although the attention to IEC messages and material is exemplary, their effectiveness is not known. Here also operational research is indicated. 12. It was observed that only 29% of the MDR-TB cases wear masks at home and in public places, and the reasons of this low proportion are not known. It should be explored why those patients do not wear masks during the intensive phase. 13. The occurrence of major side effects is not being monitored, neither of the actions taken. A systematic monitoring of the occurrence of major side effects is necessary. 14. No comparative data are available of the effectiveness and efficiency of MDR-TB prevention and control activities by other implementers. It is recommended that the CAP-TB project obtains such data, and carry out comparative analyses. 15. Evidence-informed MDR-TB control management training is necessary, in order to increase the quality and use of the data. 16. The burden of TB disease has to be determined through setting up operational research, based on follow-up of all treated TB cases and verbal autopsy of the deceased TB/HIV patients. 36 1.4.3 THAILAND 1.4.3.1 Thailand: Key Findings CAP-TB covers the community of PHE since 2012, and those of BANKHAI, KLAENG and MABTAPUD since FY13, using the same strategy as in PHE. CAP-TB’s IAs include the following 2 organizations: the Rayong Provincial Health Office (PHO) and the National Catholic Commission on Migration (NCCM). (a) Human Capacity and Supplies/Commodities – CAP-TB has been successful in developing human capacity and providing the necessary supplies & commodities. The targets for programmatic management of MDR-TB were exceeded (149%), and the training in data management was close to its target (86%). CAP-TB has provided 1 GeneXpert machine as well as enough cartridges to cover 1 year of activities, potentially enabling patients to be initiated on treatment quickly, because a positive Xpert result is considered sufficient for initiating MDR-TB treatment in Thailand. The usefulness of Xpert testing is presently diminished through a cartridge stock-out after the first year of implementation, because provincial authorities had not initiated the procurement process in a timely manner. Given that replenishing the stock does not belong to the agreed tasks of CAP-TB, strictly speaking CAP-TB is not to be blamed for this stock-out. FHI360 regional office is backstopping the CAP-TB implementers in Rayong province by means of monthly supervision, information dissemination, training, and logistical support. CAP-TB has provided supplementary staff to the Provincial Health Office (PHO) and the provincial hospital. (b) Case Management and Referrals for MDR-TB – The MDR-TB cases are managed correctly, as shown by the good interim treatment results (low default rates, rare treatment refusal, and high smear conversion rates), although the team could not provide quantitative data. Home visits are regularly carried out by a multi-disciplinary team of 5 health professionals, which visit each MDR-TB patient 3-4 times during the intensive phase of the treatment and once during the continuous phase. Following the PHO staff, this system seems to work well, although its impact and cost/time effectiveness are not documented yet. The information gathered during the home visits by the multi-disciplinary team is entered into the patient record, but it is not clear if and how this information is used to improve patient management. CAP-TB strengthened referral linkages for patients at risk for MDR-TB as a component of outreach and advocacy activities. A standardized referral form has been created to coordinate the referral and back-referral. In total 879 individuals were referred to TB and MDR-TB related services (achievement rate of 176%; and a male to female ratio of 2.6), of which 496 (=56%) accepted the referral, which is a rather low acceptance figure. It has to be noted that these referrals were made both for diagnosis as well as side effects (the relative proportions are not known). (c) Diagnosis, Treatment and Follow-up c.1. Diagnosis There is a high performance of MDR-TB case finding activities (180% of the target)43. No data on the treatment delay of the MDR-TB patients of the CAP-TB project are available, which is an important shortcoming. 43 However, when annualizing the figure and taking into account the cases coming from other provinces, then the adjusted figure (105%) is lower, but still higher than the target. Qu1: To what extent and how has CAP-TB strengthened MDR-TB management? 37 CAP-TB is actively involved in contact tracing activities, although the number of cases detected thus far is small. The evaluators were not given exact data on the total number of possible contacts, and it is thus impossible to assess the effectiveness of this important activity. Although the CAP-TB funded activities contributed to the national strategic plan calling for increasing active testing of at risk populations44, there are various problem areas, such as the unequal attention to the at-risk groups, e.g. the probability of the elderly to be screened is 15 times lower than that of the DM patients; a gender disparity in the screened high risk groups (see table 3), and no results of contact tracing activities. The gender disparity in screening is apparently unnoticed by PHO, FHI360 country office and FHI360 regional office probably due to insufficient analysis of the data. Criteria Potential TB patients Screened through Xpert RE = retreatment cases Treatment after default (TAD) Treatment after failure (TAF) 72 suspects (64 m; 8 fem) Ratio m/fem = 8 ON TB patients who have persistently positive culture after 2 or 3 months on treatment 51 suspects (36 m, 15 fem) Ratio m/fem = 2.4 PRE High risk persons: PLHIV, elderly, migrants and DM patients 130 suspects (101 m, 31 fem) Ratio m/gem = 3.3 Table 3: “Re-on-pre” screening criteria of TB/MDR-TB suspects, and Male/Female ratio. The project is less successful in addressing the needs of highly vulnerable members (PLHIV, elderly, migrants, DM patients and inmates) of the target population: - PLHIV: CAP-TB project focuses on TB/HIV patients, because Rayong’s prevalence for the TB/HIV co-infection is among the highest in the country. The number of co-infected patients (MDR-TB/HIV) and the quality of the MDR-TB/HIV care are unknown to the evaluators; nevertheless up-scaling is necessary and was agreed upon during the recent TB/HIV conference held in Rayong province on 20 March, 2014. - Elderly: Based on the TB notification rates among male and female elderly in Thailand45, eight of the 41 MDR-TB cases detected in Rayong should be elderly; however, the number in Rayong province is four times lower, as reported in the FHI360 2013 annual report. - Migrants: There is insufficient screening and case finding in this population group. From October 2011 to March 2013, 12 of the 286 MDR-TB cases reported to the NTP in the whole country were non-Thais; if we accept a similar proportion in Rayong province, then at least 2 migrant MDR-TB cases should have been diagnosed. However, no single migrant MDR-TB case has been diagnosed in Rayong province. - DM46: No hard data on DM-MDR/TB co-morbidity could be obtained in Rayong province, although their number should not be negligible; indeed, the risk to develop MDR-TB is 6 times higher in diabetics than in non-diabetics in Thailand 47. - Inmates: The case finding in the central prison of Rayong province is insufficient. 19% of the TB cases in Thailand’ prisons are MDR-TB48. Extrapolating those data to Rayong province, 8 MDR￾44 There numbers are: PLHIV (12,560), elderly (67,987, of which 0.2% were screened), migrants (14,770, of which 1.5% were screened), DM patients (17,779, of which=3% were screened) and inmates (6,211). The proportion of screened PLHIVs and inmates could not be obtained. 45 Source: Bureau of TB, Dept. of Diseases Control, Ministry of Public Health, Thailand, 2013 46 Baker MA, Harries AD, Jeon CY, et al. The impact of diabetes on tuberculosis treatment outcomes: A systematic review. BMC Medicine 2011; 9-81. 47 Bashar M, Alcabes P, Rom W.N., et al. Increased Incidence of Multidrug-Resistant Tuberculosis in Diabetic Patients on the Bellevue Chest Service, 1987 to 1997. Chest. 2001;120(5):1514-1519. doi:10.1378/chest.120.5.1514 48 Jittimanee SX, Ngamtrairai N, White MC, et al. A prevalence survey for smear-positive tuberculosis in Thai prisons. IJTLD 2007;11(5):556-61. 38 TB cases might be expected, although only 3 MDR-TB cases were diagnosed in Rayong central prison. c.2. Treatment In Rayong province 4 MDR-TB cases refused treatment, bringing the initial default49 rate at 10%50. The evaluators were not informed of any measures being envisaged to study the determinants of this initial default rate. The performance of the MDR-TB treatment is good, as shown by the adherence indicators that have increased over time51, through support for counselling and strengthening community-based DOT, and the provision for transport of the patients52. c.3 Follow-up The follow-up is good, especially thanks to the involvement of the NCCM and other DOT providers. Missing treatment doses is probably exceptional, given that there were only 2 defaulters (= defaulter rate of 5%, which is rather low). Four MDR-TB patients passed away during treatment, bringing the case fatality rate on 10%, which is a reasonable figure also. (d) Linkages with Community Health Workers – These linkages are well developed and functional. CAP-TB has set-up a call centre in the provincial hospital and provided 2 extra staff, to provide hot-line service to facilitate information on how to manage treatment complications. (e) Replicability – Following the National NTP director, CAP-TB has developed a good model for MDR-TB prevention and care, which he hoped could be expanded to other areas in the country. (f) Collaboration and Coordination – CAP-TB is heavily involved in promoting collaboration with all implementers in TB prevention and control in the province. The Advisory Committee and Working Group meets monthly in Rayong and is well attended by provincial practitioners, as well as the FHI360 coordinator. This meeting is creating an opportunity to update participants’ knowledge with respect to relevant topics, and to share up-to-date information related to MDR-TB management. (a) Capacity strengthened at the community level – The trainings and refresher trainings by NCCM are effective, as shown by the high referral figures for screening. However, the capacity building efforts to migrant health volunteers were less effective, when taking as indicator the poor screening and case finding results among migrants. Training on TB/MDR-TB, screening and referral was provided by NCCM also to PLHIV to enable them to screen for potential TB suspects among their peers; however given that the number of 49 An initial defaulter is someone who has been diagnosed as a TB or MDR-TB case, but refused to start the treatment. The rate is being determined by dividing the number of cases who do not start treatment (= numerator), by the number of diagnosed cases (= denominator). 50 In India, country with the highest number of TB cases, the average initial default rate = 4%, although there are areas with higher rates. 51 Oral information provided by the medical staff form Rayong PHO and Rayong hospital 52 200 Bath for transportation per visit. Qu 2: To what extent and how has the CAP-TB community-based approach contributed to TB and MDR-TB prevention, diagnosis and treatment? 39 PLHIVs referred is not available, the extent to which this activity is successful could not be determined. (b) DOT - The training and mentoring of the community DOT providers by CAP-TB staff is effective, as shown by the high treatment adherence (taking the few missed doses and low default rate, as indicators), and the high patient satisfaction (as could be observed in the field). NCCM’s53 DOT provision is both appropriate to the local situation and designed to meet individual patient medical and personal needs, as reported by the PHO staff, who supervises them regularly a.o. at the occasion of home visits by the multidisciplinary team. Five MDR-TB patients are followed￾up by Rayong PHO; the remaining MDR-TB patients are being followed–up by the Rayong hospital staff; no data could be obtained on the quality of their follow-up. Given the imminent end of the involvement of NCCM (by March 2014), CAP-TB intends to train village health volunteer leaders to take over this responsibility to manage the patient care; however, no concrete plan for the take-over has been shown to the evaluators. There is a problem of accessibility of the patients for DOT, as several patients live up to 1 km away from their DOT provider; furthermore some of the DOT providers are up to 60 km away from the Rayong hospital, which implies that they need to be well trained in order to be able to work autonomously and effectively with only a minimal supervision. Insufficient use is being made of the modern digital means to stay into close contact with those DOT providers and monitor their activities in an easily accessible and cost-effective way. (c) Relationship with Health Providers – There is a very constructive relationship with the health providers and collaborating hospitals, as was observed in the field. (d) Community Mobilization - Thanks to NCCM active involvement with local organizations, an MoU could be signed between the CAP-TB partners and the local authorities54, to form a Community TB Committee; the latter was created in September 2012 in the PHE community, and increased the commitment of the local authorities for TB/MDR-TB prevention and control. NCCM joined the Ban Khai District health Office and the Ban Khai Hospital in the “Stop illnesses the Buddhist way” Project; however no hard data on the effectiveness of this project could be obtained. An outbreak of dengue fever (April to September 2013) limited the community outreach activities of the village health volunteers. Given that these health volunteer are not disease-specific, seasonal variation in their availability will certainly happen more in the future, and should be taken into consideration by the PHO when planning their involvement in the TB and MDR-TB control strategy. In 2013 CAP-TB reached 4,775 beneficiaries through community radio; however no data are available on its impact; the only proofs of effectiveness are indirect ones, namely the referral figure, and the initial default rate. (e) Follow-Up during Treatment – The individual patient treatment records are very much detailed, very carefully filled in and kept up-to-date; however, there is a lack of compilation and analysis of the data, so that the occurrence of major side effects and the actions undertaken to effectively combat them, and the results of the repeat follow-up sputum cultures, are unknown. 53 Presently NCCM staff conducts home visits to 10 TB patients and 5 MDR-TB patients on a daily basis 54 They directly supervise the village health volunteers. 40 Although a series of IC activities have taken place, only 27% of the households put those measures in practice55; this low proportion might be due to the fact that most patients live in a rental room/house, which does not allow proper ventilation. (f) Adequacy of Documentation of the Key Factors & Effects of the Community approach – Although the facts are documented in great detail in the annual performance reports, the evaluators have not seen data on the key factors of the project implementation, neither on the effects of the community approach on patient adherence, delay, regularity of treatment intake, intensity of IC measures, etc. (a) Management of the technical assistance and capacity building inputs - The capacity building initiatives are very effective, as shown by the number of staff trained in case finding and in programmatic management of MDR-TB (achievements of respectively 221% and 103%), while 100% of the local organizations were provided with TA for strengthening community-based approaches for PMDT. In 2013 no training in laboratory diagnosis techniques was provided, and no TA was given for the roll-out of new diagnostics, given the late arrival of the GeneXpert machine (arrived only in august 2013). The analysis of the country assessment by FHI360, the annual performance reports and the annual work-plans shows that the planning done by the FHI360 BKK office is performance-informed, and that the TA and capacity building inputs are decided in function of the needs. The cost of a full treatment course for an MDR-TB patient in Rayong province is 68% higher than the national estimate (see table N° 4)56. Taking this cost as indicator for the efficiency of the project, it may be concluded that the CAP-TB project in Rayong province is the least cost-effective of all 3 focus countries. Apparently the reason for this very high treatment cost in Rayong province has not been explored yet, and no cost decreasing initiatives have been considered. 55 Figure based on the FHI360 annual report 56 Following data provided by the Thai national director of NTP the average cost for MDR-TB treatment is estimated to be $7,000 Qu3: Are the management arrangements optimal for achieving project objectives efficiently and effectively? 41 Type of service Unit cost (in US$) in Yunnan Unit cost (in US$) in Rayong Package of services for MDR￾TB patients Diagnostics 763 1,075 Examination before getting in-patient 69 25 Treatment (in-patient) 522 10,500 Second line drug (domestic) 2143 Injection (out-patient) 290 Treatment for side- effects 653 Regular examinations (OPD) 541 Transportation 110 Food 25 Total cost per one MDR-TB patient (24 months) 4981 $1,173,557 Table 4: Comparative cost estimation for a full MDR-TB treatment, Yunnan and Rayong provinces, 2014. (b) Communications with other implementing partners, key stakeholders, and RDMA - FHI360 is well aware of the planned and realized activities of the other implementers and key stakeholders in the project area, and of their MDR-TB control performances. However, the evaluators found in the CAP-TB project planning and programming, no single reference to experiences from other MDR-TB control implementers. (c) Monitoring, data collection and management, data quality assurance, reporting design and processes, and use of data - The reporting system works well in Rayong province, as regular site visits (every two weeks) and strategic information-related monitoring of routine reporting are carried out by the country program manager. Data examined in Rayong hospital and PHO appeared sometimes flawed, there are inconsistencies between the various sources in the denominator used to calculate the rates, time and geographical trends are not explored sufficiently, and field staff did not always seem to understand the relevance of these data for daily management of the project, resulting in little utilization, although 86% of the IA staff got training in collection, analysis and use of data, and on-site mentoring on data quality assurance is happening regularly. (d) Sufficiency of level of engagement and types of relationships established between lead organizations and in-country implementing partners with government counterparts – The CAP-TB project staff has established an effective and cordial relationship with NTP, WHO and other authorities and with the Rayong PHO, the Rayong hospital and NCCM, as was noticed by the evaluators during in-depth interviews. (e) Facilities and NGOs engaged in implementing the model – PHO, the Rayong hospital and NCCM are fully engaged in implementing the model. 57 Data provided by FHI360 office in Bangkok 42 (f) Annual planning process – RDMA and CAP-TB adhere adequately to the USAID planning process. (a)Types of common users, use of information from Knowledge Gateway – The project CAP-TB implementers in Rayong province make little or no use of the strategic information generated by the project and by the Knowledge Gateway. (b) Limitations (i.e., accessibility/friendliness) in the Knowledge Gateway from the users' perspectives – The staff in Rayong mentioned annual meetings and occasional trainings as their main source of information. The director of the NTP in Thailand mentioned NTP’s own website as his main source of information; thus the Knowledge Gateway is even at national level of limited relevance. (c) Gaps in strategic information/data for strategic decision making – In Rayong province, the main gaps in strategic information/data for strategic decision making are similar to the ones observed in Burma (see IICA: Burma: Qu4 (c)), and the strategic information and data are little used for strategic decision making. (d) Design, use and limitations of key strategic information, including gender assessment, in programming for results – The annual performance report shows various efforts to create an evidence-informed base for programming MDR-TB prevention and control activities. However, as long as the data mentioned in the previous chapter are not available, the CAP-TB program will not be able to optimally program its case finding and case holding activities. (e) Use of strategic information by project partners – The project partners in Rayong province mentioned using various mechanisms to obtain the latest strategic information, such as consulting various websites, and taking advantage of technical trainings and meetings. However, none mentioned the CAP-TB sites, such as the Knowledge Gateway. (a) Level of integration of project activities with existing national TB priorities and health systems – Ensuring financial and technical sustainability in Rayong province remains a challenge. The director of NTP declared to recognize the value and importance of CAP-TB’s project strategic approach and to be interested to support CAP-TB’s Project extension to other partners in Thailand. However, no concrete plan was explained to the evaluators, although such extension might be financially feasible, given that Thai authorities are inclined to use national funds to sponsor health activities58. (b) Measures to help ensure country adoption or adaption of the approaches used in implementing the CAP-TB model – CAP-TB has empowered the communities through capacity 58 As reported by the 2013 JMM review Qu 4: To what extent and how has the strategic information generated by the project and Knowledge Gateway information been used? Qu 5: What measures/mechanisms have been put in place to achieve sustainability and which still need to be addressed? 43 building and skills’ strengthening of the community leaders and community volunteers, and though organizing information sessions addressed to the whole community. The continuous supply of quality guaranteed drugs and laboratory equipment is an important factor for achieving sustainability. The problems with the procurement of GeneXpert cartridges in Rayong have been mentioned before. Although CAP-TB had no possibility to intervene in this issue, a planning support to timely order the cartridges could have strengthened the MDR-TB control substantially and could have contributed to its sustainability. The CAP-TB project has added some key members to the MDR-TB work force in Thailand, and trained some 90 staff. It should be regrettable if these well trained staff members should leave the MDR-TB work force. The evaluators are not aware of any steps that have been taken or that are planned to tackle this issue in Thailand. (c) CAP-TB project system to acquire data necessary for collecting a cost-effective analysis of implementation of the model - Present surveillance system provides most of the essential data to steer project performance; however, given that the system follows the national scheme, it does not allow assessing the performance, nor provide it enough data on the determinants of the program performance. A KAP survey was carried out by the Bureau of Tuberculosis (BTB), while the Mahidol University executed a study on community empowerment to strengthen the quality of TB control in marginalized populations. CAP-TB itself did not carry out any operational research in the Rayong project area. This is regrettable, because operational research in domains such as DOT, supervision of the MDR treatment, delay and reporting of surveillance data, could provide a clear picture of the added value of the CAP-TB project in comparison with the other implementers in Thailand, and contribute to its sustainability.The costing has been detailed in Qu3(a), however the evaluators have no knowledge of any cost-effectiveness analysis being done by the project, neither of cost optimizing initiatives being taken in the light of the imminent decrease of the USAID funding. (d) Ability of local partners to leverage other partnerships and resources as a results of CAP-TB to implement approaches used in the MDR-TB model – Presently 2 private hospitals are being engaged with Rayong PHO in MDR-TB care. Increased involvement with the private sector in an effort to ensure quality diagnostic and treatment of susceptible TB patients could be an effective way to prevent MDR-TB cases (though the data to support this hypothesis is incomplete) and to promote sustainability. Creating a dialogue with beneficiaries, health authorities, community representatives and other partners, as well as governmental authorities, on exit strategic alternatives could be of great help to promote sustainability. The evaluation team noticed that the eventuality of a transition-out is not yet being touched upon in Thailand, except for NCCM ending its engagement in March 2014. That ending could have critical implication for the project as it will probably affect the life circumstances of the MDR-TB patients that are presently monitored by NCCM. 1.4.3.2 Thailand: Recommendations 1. Given the low screening performance of the at–risk groups (PLHIVs, elderly, migrants, and diabetics), the reasons should be studied, and adequate measures taken to improve the screening performance. 2. Given the lower than expected number of inmates with MDR-TB, more efforts should be directed towards case finding in the prisons in Rayong province. 3. Given the gender disparity in the ‘re-on-pre’ screening results, more attention should be given to gender and other equity criteria in the case finding. 4. Attention should be given to the monitoring of any missed treatment dose, the actions taken and their impact. 44 5. More attention should be given to contract tracing of exposed members of the household or working/living environment of MDR-TB cases. 6. Given the higher than national average cost of the MDR-TB treatment in Rayong province, the reasons of this excessive cost should be explored, and cost reducing efforts should be undertaken. 7. Given the deficiencies in data management at provincial level (see chapter Qu4(c)), an adequate training course in evidence-informed program management should be organized, followed by an intensive and supportive supervision. 8. Given the present lack of a specific niche of the CAP-TB project, FHI360 together with USAID, the national authorities and provincial partners should decide on its specific niche, and prioritize its strategic approach and activities in the light of that chosen option. The recent orientation towards PLHIV, decided in March 2014 could be one of the eventual options. 2 OVERALL PROJECT CONCLUSIONS AND LESSONS LEARNED Project management: To what extent and how has CAP-TB strengthened MDR-TB management? CAP-TB has been very successful in strengthening the MDR-TB program management in each of focus country, but much less in data management. The MDR-TB cases are being managed effectively, and the service gaps observed during the baseline assessment carried out in 2012 have largely been filled through the creation of effective linkages across health services. As a consequence the referral target of MDR-TB suspects have been met in China and Thailand, and most suspects responded to the referral. Concrete data on the average delay and the performance of the contact tracing lack in all countries. CAP-TB project aims to improve case finding of highly vulnerable members of the target population, such as PLHIVs, elderly, migrants and diabetics, however, this intention is not being materialized in an significant increase of their case finding performance. The treatment performance in all countries is encouraging, although no patient is long enough on treatment to determine treatment outcome rates. CAP-TB has scaled-up the quality of the MDR-TB treatment through providing additional staff for counselling and strengthening community-based DOT and a package of support services. DOT is provided in a very holistic way, and is certainly one of pillars of the great treatment adherence in all focus countries. The linkages with the Community Health Workers are very well developed and highly functional. CAP-TB collaborates with all implementers of TB and MDR-TB prevention and control in the covered areas, although its approach is apparently not inspired by them. Community-based approach for Project Performance: To what extent and how has the CAP-TB community based approach contributed to TB and MDR-TB prevention, diagnosis and treatment? In all focus countries CAP-TB has successfully piloted community supporters, to meet the shortage in human resources and to contribute to more effective screening, and case holding. CAP-TB has trained and mentored DOT community providers to meet patients’ medical and personal needs. There is a very cordial and constructive working relationship with the IAs and the collaborating hospitals. CAP-TB has been able to elicit significant community involvement in MDR-TB prevention and control, especially through the outreach activities, the educational methods used , the involvement of cured TB patients (in Yunnan) and of local celebrities (in Burma). However, many opportunities to target women with special health education massages and screening possibilities have been missed, which might explain the low screening rates of females in many places (especially in Rayong). In all focus countries CAP-TB focuses on preventing the transmission of resistant TB strains, through providing a rapid diagnostic tool (GeneXpert test), and giving full attention to IC measures; although cough hygiene and IC control at home and in public places still pose a challenge in all countries. 45 Although CAP-TB has got unique experiences and successes in each of the focus countries, their key determinants have not been sufficiently explored. Project effectiveness and efficiency: Are the management arrangements optimal for achieving project objectives efficiently and effectively? CAP-TB country teams have great management skills and high motivation to reach the project objectives. The effectiveness of the actions undertaken is very good, and nearly all activities surpassed the target. However, there are still some issues of concern, namely the very high initial default rate and default rate (in Yunnan province), the gender disparity in case findings (in Rayong province), and the treatment delay (in Burma and recently also in Rayong province). The management of CAP-TB in Burma and Yunnan is optimally efficient, as the project expenditures focus principally on the primary processes and less on secondary processes, although the same assertion cannot be made for Rayong province. The data gathering and reporting system works well; however the data management at field level is less optimal in all countries, resulting in limited use of routine recording and reporting information for decision making. In all focus countries the FHI360 country offices have been able to establish effective, smooth and very functional relationships with NTP, WHO, and other authorities, the private practitioners, local NGOs and the IAs. Strategic information: To what extent and how has the Strategic Information generated by the project and Knowledge Gateway information been used? CAP-TB has focused in each focus country on developing and distributing strategic information, as well as on its incorporation into MDR-TB prevention, treatment, and follow-up strategies. CAP￾TB’s Knowledge Gateway is central to the dissemination of the strategic information; however, difficulties were reported by the field staff in all focus countries while accessing and using the Knowledge Gateway system, and in Yunnan the FHI360 country office developed alternative systems. Although CAP-TB has organized training sessions in analysis, interpretation and use of data and the ascertainment of their quality, and carries out regular data quality assessment checks, IAs’ ability to interpret the routine data is still basic in all 3 countries. Comparable gaps were found in the data available for strategic decision making in all 3 countries, namely lack of stratified data on the cost￾effectiveness and performance of screening, case finding and case holding, and also of comparative data on the performance of other MDR-TB control implementers in the same country. Furthermore there is also a concern with data confidentiality due to lack of project-specific record storage; apparently insufficient attention is given to this issue. Sustainability: What measures/mechanisms have been put in place to achieve sustainability and which still need to be addressed? CAP-TB has shown a great ability in providing assistance in organisational capacity building in each of the 3 focus countries, and to help ensure country adoption of the approaches used in implementing the CAP-TB model. Direct partnership with national governments varies according to the country, but in all countries there is evidence that CAP-TB’s approach is highly appreciated by the NTP authorities; in Burma CAP-TB’s approach has inspired the national MDR-TB guidelines. There is high potential for CAP-TB to influence national and provincial policies in MDR-TB prevention and control through creating close working relationships with implementing agents, private practitioners, hospitals and local communities. 46 There are operational challenges in each focus country related to decreasing in USAID financial support, supply of SLD’s (in Burma), stock-outs of cartridges (in Rayong), and capacities of the local organisations. These challenges are being met adequately by each of the FHI360 national teams. Creating a dialogue with beneficiaries, partners and governmental authorities on exit strategic alternatives could be of great help to promote the sustainability of the CAP-TB approach. However, dialogue has not been started yet in any of the focus countries. 3 OVERALL RECOMMENDATIONS 3.1 TO USAID/RDMA 1. Project management: Given that the MDR-TB epidemic might be on the raise in each focus country59, CAP-TB and partners need to continuously monitor the overall epidemic situation and the epidemiological trends in order to optimize the response of the MDR-TB prevention and control program. RDMA and USAID/Burma should clearly define lines of communications and also the roles of responsibility and oversight of FHI360 Bangkok and the FHI360 country office, especially now that USAID/Burma is adequately staffed and eager to directly manage CAP-TB activities in Burma. 2. Project performance: As country contexts vary widely for engaging with the national and provincial governments, USAID should encourage and support implementation studies in each focus country aiming to document the key determinants and effects of CAP-TB’ community approach, in order to optimize MDR-TB prevention and control performance. 3. Project effectiveness and efficiency: USAID should encourage and support operational research studies to address the high initial default and default rates (in Yunnan province), gender disparity in case finding (in Rayong province), and treatment delay (in Burma), and support the organization of evidence-informed program management trainings of all staff involved in project implementation and of a post-course follow-up. 4. Strategic information: The burden of TB disease in the areas covered by the project is not sufficiently known; there should be technical assistance and financial support to determine the burden of TB disease in those areas, and especially in Yunnan province. 5. Sustainability: USAID/RDMA should enter in dialogue with the beneficiaries, the partners and the governmental authorities of each focus country about the modalities of the phasing-out and the possible approaches towards sustainability; and develop country-specific phase-out plans jointly with the beneficiaries, the partners and the governments. 3.2 TO FHI360 REGIONAL OFFICE 1. Project management: Provide TA to the focus countries to improve their ability to forecast supply need of GeneXpert cartridges, to negotiate the price and to plan for timely procurement. More autonomy could be granted to the Burma FHI360 country office, modifying the role of the FHI360 regional office into one of backstopping, advising and assessing. 59 In Burma due to the continuous transmission by the high number of MDR-TB cases on the waiting list; in Thailand by the long waiting list due to present lack of Xpert cartridges, and in Yunnan province due to the high number of initial defaulters 47 2. Project performance: Although CAP-TB has got unique experiences and successes in each of the focus countries, the key factors of the successes are not sufficiently known, therefore the FHI360 regional office should take the initiative to start exploring them through quantitative and qualitative methods, involving the project implementers in the design, implementation, analysis and interpretation of those operational studies. 3. Project effectiveness and efficiency: In view of the decreasing financial support by USAID, consider exploring the cost of the various service delivery models, assessing which parts of the interventions could be more cost-effective while maintaining the quality, in order to provide information for sustainability. Examine the reasons of the low notification rate of vulnerable population groups (PLHIVs, elderly, migrants and DM patients), and use the findings as evidence to optimise the screening and case finding efforts. 4. Strategic information: Promote the exchange of experiences and lessons learned between the focus countries. Evaluate the “Knowledge Gateway” support to determine its relevance, effectiveness and efficiency. Refocus efforts to strengthen internet-based knowledge sharing via local governments (NTP), WHO and other relevant organizations. 5. Sustainability: With the aim of promoting evidence-informed sustainability, the capacity building of the country offices and of the IAs should be strengthened, especially through trainings in evidence-informed program management of the staff and their supervisors. Examine the involvement of the private sector from a “win-win” strategic outlook through operational research, and incorporate the findings in the PPM strategy. 48 ANNEX A: CAP-TB Mid-term Evaluation Scope of Work SOL-486-13-000045 Statement of Work for Mid-Term Performance Evaluation of CAP-TB Project 1. PROJECT TO BE EVALUATED Cooperative Agreement No. AID-486-A-12-00002 Titled “Greater Mekong Sub-region Multidrug Resistant Tuberculosis Prevention and Management” Implementing Agency: FHI 360 Total Estimated Cost: $10 million for 5 years Cooperative Agreement contract under Regional Development Mission Asia The effective date of this Cooperative Agreement is October 14, 2011 through October 13, 2016. Sub-partners: Burma: Myanmar Medical Association (MMA), Myanmar Business Coalition for AIDS, Myanmar Health Assistants Association and Pyi Gyi Khin China: Anti-TB Association of Yunnan and The Union Thailand: Rayong Provincial Health Office and Anti-TB Association of Thailand and National Catholic Commission on Migration; 2. PROJECT BACKGROUND 2.1 Context Mycobacterium tuberculosis strains that are resistant to anti-TB drugs are of increasing concern across the globe and particularly, in the Asia Region. There were an estimated 650,000 cases of multidrug resistant tuberculosis (MDR-TB) in 2010. It is significantly more challenging to treat MDR-TB than drug-susceptible TB because the duration of treatment is much longer and requires expensive second-line drugs (SLDs) that are associated with a greater incidence of adverse reactions. The spread of MDR-TB is heighted when treatment is inappropriately or incompletely administered. Similarly, MDR-TB can become resistant to SLDs, leading to extensively drug-resistant TB (XDR￾TB), for which there is no treatment. Despite the high costs, comprehensive treatment programs have demonstrated the efficacy of MDR-TB treatment and mathematical models have suggested the therapy can be cost-effective in resource-poor settings. CAP-TB focuses on three countries in the Greater Mekong Sub-region (GMS) with high TB (and MDR-TB) burden including Burma, China, and Thailand. China has the highest TB burden and the second largest epidemic of MDR-TB globally after India, accounting for over 20 percent of global MDR-TB cases. Burma and Thailand are both ranked by the World Health Organization (WHO) as high TB and high MDR-TB burden countries. Indeed, WHO estimates that 4.2 percent of new TB cases and 10 percent of retreatment cases in Burma, and 1.7 percent of new TB cases and 35 percent of retreatment cases in Thailand are cases of drug￾resistant TB. 2.2 Plan USAID RDMA is providing funding to FHI 360 to implement the Greater Mekong Sub-region Multidrug Resistant Tuberculosis Prevention and Management or “CAP-TB” Project. The CAP-TB Project goal is to contribute to reducing incidence of and mortality related to multidrug resistant tuberculosis (MDR-TB) across the Greater Mekong Sub-region (GMS). The CAP-TB Project targets three MDR-TB high burden countries in the region: Burma, China, and Thailand. Project activities focus on developing, implementing and evaluating implementation of elements of the model for MDR-TB prevention and management that can be scaled up, increasing detection of MDR-TB cases and treatment success in areas where model MDR-TB programs are implemented, and building local systems and capacity to sustain implementation of the model or elements of it beyond CAP-TB. CAP-TB focuses on high-risk focus groups, which include people living with HIV, migrant and mobile populations, and diabetic individuals. CAP-TB’s level of involvement in the various elements of the 49 model varies by country and by project site. (The model guiding implementation is illustrated below.) In addition, CAP-TB emphasizes the sharing of information and best practices across these countries and the Greater Mekong. CAP-TB plan emphasizes the delivery of high quality technical assistance (TA) to facilities and health providers who are part of the model (government agencies, public/private sector companies, academic institutions, technical organizations, civil society groups, and other community-based TB programs) and in strengthening the capacity of local implementing partners to receive and manage direct USAID funding. General program priorities are to build upon existing efforts to prevent and better manage MDR-TB in each target country. The following achievements are anticipated at the end of implementation: improved MDR-TB management; strengthened capacity among local partners; stronger health information systems (HIS); greater laboratory and management capacity; equitable access to prevention, treatment and care for MDR-TB; improved health-seeking behavior and linkages; strengthened inter-country learning and collaboration in the GMS; and greater advocacy and communication around MDR-TB. Under this, the implementation follows the CAP-TB model and plan approaches. As significant national and international efforts are already in progress to improve diagnosis and management of MDR-TB and to prevent new cases in each target country, the CAP-TB local implementation of the model places an emphasis on support for on-going efforts and tailors project activities to the needs and priorities of individual countries. 2.3 Project Implementation The CAP-TB Project began on October 14, 2011 and will end on October 13, 2016. The Project has planned for a budget of $10 million over this 5 year period. FHI 360 is the prime cooperating agency. FHI 360 leads a consortium of sub-partners. In China, CAP-TB’s sub-partners include the International Union Against Tuberculosis and Lung Disease (the Union), Yunnan CDC and Yunnan Anti-TB Association in China. In Thailand, CAP-TB’s sub-partners include the National Catholic Commission on Migration (NCCM) and Rayong Provincial Health Office (PHO. In Burma, CAP-TB’s sub-partners include the Myanmar Medical Association (MMA), Pyi Gyi Khin (PGK - Women’s General Development Enterprise & Co-Operative Limited); Myanmar Business Coalition for AIDS, and Myanmar Health Assistants Association. CAP-TB also collaborates with the National Tuberculosis Programs, other private sector partners, the WHO, the U.S. Centers for Disease Control and other International Non-governmental Organization partners working on MDR-TB 50 control. The project is being implemented in Rayong Province, Thailand; Yunnan and Guangxi Provinces, China; and Rangoon and Mandalay Provinces, Burma, as seen below: 2.4 CAP-TB Results Framework The project Overall Strategic Objective (SO), Intermediate Results (IRs) and Outputs are listed below. SO: To improve the prevention and management of MDR-TB in Burma, Thailand and China IR1: Strengthened MDR-TB prevention 1.1 Mobilized communities to advocate for and use TB services 1.2 Scaled-up implementation of TB infection control in health facilities IR2: Strengthened MDR-TB management 2.1 Ensured capacity, availability, and quality of laboratory testing to support the diagnosis and monitoring of TB patients, including the rapid diagnosis of MDR-TB 2.2 Strengthened case-finding and referrals for MDR-TB 2.3 Strengthened human resource capacity for MDR-TB management 2.4 Scaled-up quality treatment and community approached for PMDT IR3: Improved strategic information for MDR-TB 3.1 Strengthened capacity of TB programs to collected, use, and analyze data for management 3.2 Increased TB research activities IR4: Strengthened enabling environment for MDR-TB 4.1 Improved capacity of National Tuberculosis Programs (NTPs) to develop, finance, and implement national TB control strategies in line with global strategies 4.2 Strengthened partnerships for quality TB care, including private sector 3. EVALUATION TYPE, PURPOSE AND KEY QUESTIONS This mid-term evaluation will be a performance evaluation as defined in the USAID Evaluation Policy (see Appendix 1). The main purpose is to assess the project performance and its progress towards intended results. The evaluation will provide insights and important feedback to each of the partners and stakeholders to understand both the strengths and areas where technical, administrative and management efforts could be improved. It will also provide evidence and learning for adapting future programs and projects, including improving USAID/RDMA designs, strategies and policies. 51 3.1 Purpose of the Evaluation This mid-term evaluation is being undertaken to analyze the CAP-TB project performance to date and obtain recommendations on improvements needed for the project to meet its intended purpose. In addition, the evaluation will analyze the value-added by the regional project to knowledge and skill enhancement and organizational capacity building of local partner institutions. The evaluation conclusions and recommendations are to be used by RDMA’s OPH office and the implementing partner to strengthen project implementation. RDMA will also use the evaluation to inform future programming of regional projects. The evaluation report shall be provided to USAID/RDMA for electronic distribution to implementing partners at various levels and key stakeholders in each country. RDMA OPH will be provided with hard copies and an electronic copy of the report. The dissemination strategy will include an electronic copy of the Executive Summary together with the full report on the CAP-TB website by FHI 360. A copy of the full report will also be submitted via RDMA OPH and Program/Project Development Office (PDO) to USAID’s Development Experience Clearing House (DEC.) 3.2 Evaluation Questions and Methodology Five key evaluation questions are set for this task. Relative priority is indicated in percentage terms at the end of each. 1) Project performance: To what extent and how has CAP-TB strengthened MDR-TB management? (relative priority: 20%?) Data gathering and analysis should include but not limited to the following: • Human capacity and supplies/commodities; • Case management and referrals for MDR TB; • Diagnosis, treatment and follow up; • Linkages with community health volunteers. Methodology: (1) Use data from Performance Monitoring Plan and indicators, partner reports and results of Organizational Development Assessments. (2) Interviews and/or other rapid appraisal methods should be used with country implementing partners and sub-partners, community health volunteers, health facility involved in the project and a sample of those attending the training. (3) Focus group discussion with community health volunteers and beneficiaries (MDR-TB patients) and review of health facility records and data. (4) Analyze by country and respondent categories (when appropriate, disaggregated by gender) as well as across countries. 2) Project performance: To what extent and how has the CAP-TB community based approach contributed to TB and MDR-TB prevention, diagnosis and treatment? (30%) Data gathering and analysis should include but is not limited to the following: • Capacity strengthened at the community level; • Referrals and DOT; • Relationship with health providers; • Community mobilization; • Adequacy of documentation of the key factors and effects of community approach. Methodology: (1) Use data from Performance Monitoring Plan and other project documents, training reports, partner reports and results of client satisfaction surveys. (2) Interviews and/or other rapid appraisal methods should be used with community health volunteers, country implementing partners and a sample of those attending the training and health facilities involved in 52 implementing the project. (3) Focus group discussion with community beneficiaries and community health volunteers. (4) Analyze by country and respondent (when appropriate, disaggregated by gender) categories as well as across countries. 3) Program management: Are the management arrangements optimal for achieving project objectives efficiently and effectively? (20%) Data gathering and analysis should include but not limited to the following: • Management of the technical assistance and capacity building inputs; • Communications with other implementing partners, key stakeholders and RDMA; • Monitoring, data collection and management, data quality assurance, reporting design and processes and use of data; • Sufficiency of level of engagement and types of relationships established between the lead organization and in-country implementing partners with government counterparts, facilities and NGOs engaged in implementing the model; • Annual planning processes. Methodology: (1) Use of project documents. Interviews with RDMA, key lead partner staff, country implementing partners, provincial government authorities, NGO partners and health facilities involved in implementing the project. (2) Interviews to seek evidence of the relationships and level of engagement, and level of satisfaction and suggestions for improvements. (3) Analyze by country and respondent categories as well as across countries. 4) Strategic information: To what extent and how has the Strategic Information generated by the project and Knowledge Gateway information been used? (15%) Analysis should include: • Types of common users, use of information from Knowledge Gateway; • Limitations (e.g. accessibility and friendliness) in the Knowledge Gateway from users’ perspectives. • Gaps in strategic information/data for strategic decision making; • Design, use, and limitations of key strategic information, including gender assessment, in programming for results i.e. analysis on gender equality and women’s empowerment programming priorities and achievements to date; • Use of strategic information by project partners. • Recommendations on priorities for effectively addressing MDR TB prevention and management among the most vulnerable populations in the sub-region, incorporating findings from the CAP-TB project and other stakeholders. Methodology: (1) Review strategic information available data. (2) Interview persons in each country (regional government, implementing partner managers and staff, NGOs and facilities engaged in project activities. (3) Determine which knowledge management activities they have been engaged in, what information found the most helpful, and use of the information and opinions on the Knowledge Gateway. (4) Summarize project data on number of persons a) attending annual meetings and b) using the web-based Knowledge Gateway site. 5) Sustainability: What measures/mechanisms have been put in place to achieve sustainability and which still need to be addressed? (15%) Analysis should include: 53 • Level of integration of project activities with existing national TB priorities and health systems; • Measures to help ensure country adoption or adaption of the approaches used in implementing the CAP-TB model; • A project system to acquire data necessary for conducting a cost-effectiveness analysis of implementation of the model in each country; • Ability of local partners to leverage other partnerships and resources as a result of CAP-TB to implement approaches used in the MDR-TB model. Methodology: (1) Use existing data from project documents. (2) Interviews and/or other rapid appraisal methods should be used with lead partner, government counterparts at national and/or sub-national levels as well as with NGO partners, health facility managers and project lead staff involved in implementing the CAP-TB Model. (3) Review the information concerning unit costs/expenditures and the other key elements of the system, analyze adequacy of the guidance and instructions for data entry (4) Review national TB plans and strategic plans or operational plans and identify areas where the model can better contribute to the achievements of the national objectives. 4. ANALYTICAL FRAMEWORK The overall analytical framework should include analysis on the relevance of the project design and interventions to the current TB and MDR-TB situations and provide recommendations on what modifications that may need to be made. In addition, the Evaluation Team shall include a strategic assessment and analysis to generate ideas and recommendations on future priorities and directions for USAID/RDMA based on findings from the CAP-TB project performance evaluation and additional input from the other stakeholders. 5. GENDER CONSIDERATIONS The Evaluation Team should identify and address relevant gender inequalities and women’s empowerment opportunities and challenges within the project’s areas of work. Recommendations should outline the most significant gender opportunities and challenges that need to be considered during activity implementation and monitoring. Describe how both women and men were engaged in and affected by the work undertaken, disaggregate by age or other dimensions as appropriate. The desk review should include a specific gender analysis relating to project implementation. Where applicable, the data in the Evaluation should be disaggregated by gender. The final report should include a section on gender analysis. 6. TECHNICAL TEAM AND WORK PHASES 6.1 Team composition and leadership: This SOW is for two international consultants: 1) Evaluation Specialist as the Team Leader; and 2) TB Expert. Both will have expertise in conducting program evaluations, understanding of MDR-TB or TB control interventions and field experience working with community and public health service delivery programs in South-East Asia. The Evaluation Team may be supported by a Senior TB Adviser from USAID Headquarters and/or from RDMA where possible. Each Team member will have writing responsibilities for drafts and finalization of the report. Translation support services will be recruited by FHI360 in all countries. The Evaluation Team Leader will be responsible for the overall coordination within the broader team, assisting them to identify essential information sources, detailing individual responsibilities, and planning the overall team schedule in consultation with the CAP-TB Project Lead/Chief of Party and RDMA. The Evaluation Team will present to RDMA the findings, consolidate feedback and submit to RDMA’s Office of Public Health the final report. 54 The Team Leader will lead and manage team members and ensure the quality and timeliness of the deliverables described under Section 6. As Team Leader, the consultant will undertake; • Preparatory work: Work with the RDMA staff, before the evaluation team members assemble, to refine a plan of action for information gathering, including document review, key informant interviews and site selection. • Management of field work: Lead the field work process and maintain communications with RDMA. • Report writing: Quality assurance of the final report. Provide an annotated outline of the final evaluation report, discuss with team members, and assign writing responsibilities; ensure timely and quality team contributions. 7. DELIVERABLES The Evaluation team leader will be responsible for coordinating and managing the drafting of deliverables, consolidating the individual contributions, and submitting the drafts and final report. Each evaluation team member will be responsible for contributing to the deliverables and drafting relevant sections of the documents based on his/her expertise and the tasks assigned by the Team Leader. The required deliverables as a joint output for the Evaluation Team are listed below: 1) Evaluation Design, Tools and Work Plan Two weeks prior to the beginning of the field work and based on their review of the project documents, resources, sub-partners and project sites, the Evaluation Team Leader will submit to RDMA OPH a draft work plan for the evaluation, including an analytical framework. In addition, it will submit to the evaluation COR RDMA OPH rigorous and appropriate methodology which includes but is not limited to the following for each of the evaluation questions. • Sub-questions that lead to answering the larger evaluation question. • Data sources (what existing data and sources to obtain new information.) In instances in which community client input is provided, the Team shall propose a feasible plan for sampling. • Data collection methods (guided by, but not limited to those suggested above) • Plan for analyzing a) quantitative and b) qualitative information. The plan should be based on obtaining country-level analysis and, as applicable, aggregation of data across the countries. • Proposed data collection instruments. The Evaluation Team, led by its Team Leader, will revise these based on RDMA comments. The work plan from the evaluation COR is required before the Evaluation Team commences field work. 2) Outline of the Evaluation Report An annotated outline of the evaluation report, including sub-sections of the main body of the report, shall be submitted to RDMA for approval by the end of two weeks of field work. 3) Debriefings Debriefings on findings, preliminary conclusions and recommendations will be provided to USAID/Burma prior to country departure. In this country the team will begin its field work by a meeting with a USAID representative who will brief the team. At the conclusion of the field work, RDMA OPH will be debriefed on the main evaluation findings, and preliminary conclusions, recommendations and lessons learned. The Evaluation Team will also present a mission-wide debriefing with a focus on findings and recommendations. Guidance on the structure of the presentation can be found in Appendix 2. The oral debrief will be accompanied by a written document or power point presentation, with electronic copies provided to the evaluation COR. 4) Draft Evaluation Report 55 Following the required structure for final reports and addressing comments from debriefings, a draft of the evaluation report will be submitted to RDMA OPH within 10 working days of receipt of written feedback from the RDMA debriefing. The draft evaluation report should also include feedback received from the various debriefings. RDMA will provide written feedback from the debrief to guide the formation of the Evaluation Report. 5) Revised Draft Evaluation Report A revised draft evaluation report will be submitted within 4 working days of receiving written comments from RDMA OPH. The revision will incorporate all feedback provided by RDMA OPH reviewing team, the RDMA Monitoring and Evaluation Working Group, and RDMA Program Office. The report should conform to USAID Evaluation Policy “Criteria to Ensure the Quality of The Evaluation Report (see Appendix 1). 6) Final Evaluation Report The Final Evaluation Report will be submitted within 3 working days of receiving comments from RDMA on the Revised Draft. The full report must not exceed 30 pages, excluding appendices. The structure of the final report should be: • Executive Summary of the Evaluation, no more than 3 – 5 pages, that concisely states evaluation purpose, methodologies, key findings, and conclusions, recommendations, and lessons learned • Acronyms • Table of Contents • Main body of the report: introduction, background and methodology along with a statement related to methodological limitations; findings/conclusions/recommendations on each evaluation question and in general/overall Project conclusions, recommendations and lessons learned and highlight lessons learned • References and List of Persons Contacted • Appendices: at a minimum the appendices will include: Evaluation SOW, Final evaluation design and work plan, any statements of differences, all data collection tools, and any other sources of information. The Team Leader shall ensure that the final report meets USAID required standards for evaluation reports (See ADS 203.3.1.8) This includes but is not limited to specifying that the Contractor is expected to put a high quality photo representative of the project evaluation on the front cover, with a brief caption on the inside front cover explaining the photo with photographer credit. Permission is required from those in photo/place of photo and photographer to use in a public document. It is imperative that proper ethical procedures be observed in using photos of persons. For additional guidance on preparing an Evaluation Report, please see the USAID Evaluation How￾To Note found at the link: here. 7) Electronic Handover of Data and Records The Evaluation team will hand over to the evaluation COR any data and records collected by the Evaluation Team (e.g. interview transcripts or summaries) in an electronic file in easily readable format agreed upon with RDMA. The data should be organized and fully documented for use by those not fully familiar with the project or evaluation. USAID will retain ownership of all datasets. 8) Submission of the RDMA Approved Report to DEC In order for RDMA to submit the report to the Development Experience Clearing house (dec.usaid.gov), the Contractor/Team Leader must provide an electronic copy of final evaluation report within one month of the conclusion of the evaluation. The RDMA Program Development Office will be responsible for uploading the final version of the Evaluation Report to the DEC. 8. PERIOD OF PERFORMANCE Time period: The Evaluation will consider the entire period of project performance: October 14, 2011 – December 31, 2013. 56 Payment: First payment of 20% will be made upon submission of the work plan. The final payment will be made upon submission of the approved final report. The overall period of performance of this entire consultant services for the Evaluation Team Leader is expected to require approximately 42 working days over an elapsed 12-week period, and approximately 38 working days over an elapsed 12-week period for the TB Expert. An illustrative schedule and time requirement is as follows (exclude Sundays as free time); Description Team Leader TB Expert Period (No. of days) (No. of days) Preparatory phase includes planning meeting (in person or teleconference) with RDMA and reading background documents and first deliverables and revised deliverables. 6 5 Week 1/2 Meeting with RDMA, FHI360 and CDC and flight to China. 2 2 Week 2 Field work in China (includes data analysis and initial draft report for China. Travel to Burma. 8 8 Week 2/3 Field work in Burma (includes briefing & debriefing mission in Burma, data analysis and initial report for Burma section and travel to Thailand. 8 8 Week 4 Field work in Thailand, data analysis and initial report for Thailand section 7 7 Week 5 Data analysis and report writing 4 4 Week 6 Debriefing with USAID/RDMA (OPH and mission-wide presentation) 1 1 Week 7 Submission of 1st draft that incorporates feedback from the debriefing 1 1 Week 7 Respond to USAID Burma and RDMA comments & finalize the report 4 2 Week 10 Submission of final report by February 2013 1 0 Week 12 Total 42 38 9. RELATIONSHIPS AND RESPONSIBILITIES 3.3 CONSULTANT TEAM LEADER The Team Leader will coordinate and manage the evaluation team and will undertake the following specific responsibilities throughout the assignment: • Plan and facilitate assessment-related Team Planning Meetings with the other international consultant and USAID evaluation team member(s). • Be the primary point of contact with FHI360 and RDMA. The team leader is solely responsible for ensuring the quality and timeliness of deliverables for USAID, and will coordinate and manage the Evaluation team and will undertake the responsibilities described above. FHI 360 As the lead for CAP-TB, FHI 360 will be responsible for the following. 57 • For each of the three countries, provide for each geographic area a list of names with titles of key partners, facilities, trainees, and CBO communities engaged in the project to RDMA to transmit to the Team Leader no later than three weeks prior to commencement of the evaluation. • Provide an electronic copy of all country-level IR data, disaggregated by sub-partner, and if available, its catchment areas (sites), to RDMA no later than four weeks prior to commencement of the evaluation. • Provide to RDMA electronic copies of all semi-annual and annual report, M&E plan, the latest FY implementation plan, special studies, and other documents on CAP-TB, including the gender assessment report if available. • Send letters to key partners and sub-partners about the upcoming evaluation. • Following guidance from the Team Leader, FHI 360 country program staff member set up appointments with the key stakeholders and sub-partners to be visited. • Provide translation support and in-country transportation support for the Evaluation team. 3.4 RDMA OPH Prior to contracting with Evaluation Team members, RDMA OPH will respond to any queries about the SOW and/or the assignment at large. In addition, to avoid conflicts of interest (COI) or the appearance of a COI, RDMA OPH will review previous employers listed on the CVs for proposed consultants and obtain additional information regarding potential COI with the project contractors or NGOs evaluated/assessed and information regarding their affiliates RDMA OPH will designate the appropriate staff person to serve as the point of contact and a source of technical information about the project activities. • RDMA OPH shall serve as the point of contact between FHI 360 and the Team Leader prior to the beginning of the field work stage and after completion of all field work. • RDMA OPH shall ensure that all documents, files and lists mentioned above are obtained from FHI 360 and transmitted to all members of the evaluation team in a timely manner. • RDMA will provide the evaluation team with a list of key stakeholders in each country. • To ensure that the field portion of the evaluation begins as scheduled, RDMA OPH shall provide the Team Leader comments on the first deliverable within four working days of receipt of the document. RDMA OPH, USAID/Burma shall assist the Team Leader with the following: • Provide guidance on recommended secure hotels and methods of in-country travel (i.e., car rental companies and other means of transportation) and if necessary, identify a person to assist with logistics (i.e., visa letters of invitation etc.). • Provide timely review of draft/final reports and approval of the deliverables. 3.5 REQUIRED QUALIFICATIONS AND APPLICATION PROCESS A consultant bidding on the evaluation should submit a written statement of interest and a CV. He/she should specify clearly the position being applied for. Both Team Leader and the TB Specialist should submit a succinct description of the proposed methodology, evaluation design and approach. The proposal should not exceed a maximum of 7 pages. The Team Leader will have the primary responsibility as the Point of Contact between the team and the USAID missions (both in RDMA and USAID/Burma). Evaluation team members will have different key role and responsibilities as follows; 1. Team Leader: 58 The Evaluation Team Leader will have the primary responsibility as the Point of Contact between the team and the USAID Missions (both in RDMA and Burma Mission). The Team Leader is also responsible for the overall management and coordination of the team, including detailing individual responsibilities and tracking performance, and ensuring the delivery of high-quality and timely deliverables to USAID. As Team Leader, the consultant will: • Work with the RDMA staff and the team members to finalize evaluation methodology and to refine a plan of action for information gathering, including document review, key informant interviews, as described in Section 6: Deliverables. • Work with evaluation team to draft and finalize questionnaires for key information interviews and focus groups. • Finalize the team’s overall schedule in consultation with the CAP-TB Project Lead/Chief of Party, as described in Section 6: Deliverables. • Provide an annotated outline of final evaluation report, discuss with team members, and assign writing responsibilities; ensure timely and quality team contributions of the deliverables described under section E. • Present the debriefing with RDMA’s Office of Public Health and other USAID staff, consolidate all draft sections from team members, and finalize the report for RDMA. • Manage the performance of and ensure that deliverables are met for all team members. • Act as the primary point of contact with USAID and with other key stakeholders, and as the lead communicator when presenting and debriefing on aspects of the evaluation findings. . • Assume responsibility for the quality and timeliness of all deliverables submitted to USAID throughout the evaluation. • Plan and facilitate team meetings and briefings with USAID. The qualifications of the Team Leader should include: • A graduate or doctorate degree in Public Health, Evaluation, or related field; • Minimum of 10 years of experience in a relevant field related to TB, MDR-TB, public health and infectious diseases programming; • Understanding of TB and MDR-TB transmission, and prevention, treatment and care strategies; • Field experience working with TB control programs; • Experience working in South East Asia; • Experience managing or participating in infectious diseases focused evaluations; • Excellent analytical, writing, and presentation skills; • Experience managing teams, including logistics, planning, and budget management; • Experience in undertaking operational research. 2. TB Specialist The TB Specialist will assist the Team Leader to draft relevant assigned sections of the Evaluation Report, assist Team Leader to draft and address comments from USAID on initial and revised draft reports, as well as to prepare presentations for briefing/debriefing with USAID/RDMA and USAID country offices. The TB Specialist should have comprehensive experience in working on TB control prevention, MDR-TB diagnosis, treatment and follow-up. He/she should be familiar with issues connected with strategic information in relation to planning, programming and policy making. It is particularly important that he/she has extensive experience in operational research. Desired qualifications and skills include; • Post-graduate qualification e.g. PhD/Masters in Public Health or related fields; • Extensive experience in MDR-TB control interventions; 59 • Field experience working with community and public health service delivery programs in Southeast Asia; • Excellent analytical skills focusing on sustainability and feasibility within a non-project context Excellent writing skills in English; • Experience conducting evaluations, preferably with USAID. 3.6 SUPPORTING DOCUMENT FOR PREPARATION WORK Necessary supporting document will be supplied to the evaluation team prior to arrival to RDMA. APPENDIX 1 CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT • The evaluation report should represent a thoughtful, well-researched and well organized effort to objectively evaluate what worked in the project, what did not and why. • Evaluation reports shall address all evaluation questions included in the scope of work. • The evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline need to be agreed upon in writing by the technical officer. • Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex in the final report. • Evaluation findings will assess outcomes and impact on males and females. • Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). • Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions. • Findings should be specific, concise and supported by strong quantitative or qualitative evidence. • Sources of information need to be properly identified and listed in an annex. • Recommendations need to be supported by a specific set of findings. • Recommendations should be action-oriented, practical and specific, with defined responsibility for the action. APPENDIX 2 GUIDE TO USAID/RDMA EVALUATION PRESENTATIONS This document is intended to serve as a guide to evaluation teams in their presentations to the USAID’s Regional Development Mission for Asia (USAID/RDMA). Once an evaluation has been completed, the team leader should convene a debriefing meeting with team members, including the appropriate USAID/RDMA staff to agree to the terms of the draft final report and formal presentation to the Mission. The evaluation team should debrief the technical team on items to be presented, including but not limited to: evaluation findings, results, prioritization of results, concerns and recommendations. The evaluation team should schedule this debrief no later than one day before the presentation. Below is a suggested format to evaluation presentations. Presentations should include, but are not limited to: PRESENTATION TOPIC TIME (estimated) 1) Introduction of Evaluation Team 5 2) Project Context & Overview • Background • Environment (i.e. context, including challenges) • Project Objectives 5 60 • Project Main Activities • Stage of Project upon Evaluation 3) Purpose/Focus of the Evaluation • Purpose of Evaluation • Evaluation Methodology • Integration of USAID Evaluation Policy Standards 10 4) Evaluation Findings • Summary of input from evaluation respondents • Project Impacts • Implementation Concerns 10 5) Overall Conclusions 5 6) Recommendations to USAID/RDMA, including recommended prioritization in light of overall programming context 10 TOTAL 45 minutes + up to 30 minutes of Q&A/discussion General Tips and Considerations: 1. Presentation should be clear and straight forward. Long text and reading off slides can lose the audience’s attention. Providing the appropriate USAID/RDMA staff a draft of the presentation slides at least one day in advance for comment is strongly recommended. 2. The presentation should focus on the evaluation findings and conclusions and specifically link findings to decision-making steps. 3. Consider distributing project and evaluation background to meeting participants prior to presentation 61 ANNEX B: Analytical Framework for the Mid-Term Evaluation of CAP-TB SOL-486-13-000045 Author: Aimé De Muynck, MD, PhD, team leader Address: 37 Hof ter schrieklaan B2600 Berchem Belgium Tel 00 32 3 2396178 E-mail: aimedemuynck@gmail.com Description of proposed methodology, evaluation design and approach 1. Proposed methodology As no evaluation can look at everything, priority areas had to be determined and 5 key evaluation questions have been stated in the document “Statement of Work for Mid-term evaluation of CAP￾TB Project (SQL-486-13-000045)”. The methodology has to allow producing relevant and accurate answers to the 5 key questions that the evaluation has to answer. For each key evaluation question, additional detailed sub-questions have been developed, based on the project document. These additional questions took into consideration quality and relevance criteria, such as: - Their ability to be answered sufficiently well enough to inform understanding and support future action by the primary intended users. - Their ability to be answered in a timely manner and at reasonable cost. - Data exist to provide relevant answers of the questions are quantitative in nature. - The answers should not be predetermined by the phrasing of the question (= not being a ‘leading question’). Methods Activities Source of data Qu 1: To what extend and how has the CAP-TB strengthened MDR-TB management? Use existing data Start with analyzing the base-line data with respect to diagnosis, treatment and follow-up Project partners’ reports Search for trends of the performance Id. Judge the quality of the data; if inconsistencies and/or errors are found in the data, then data cleaning will be suggested as needed in the evaluation report. Via triangulation Determine the potential impact of the training of project staff on MDR-management practices, through before – after comparisons Supervision reports Carry out a crude analysis, and also a simple stratified analysis by country, gender and relevant other categories such as partner organizations, and then a pooled analysis to obtain global estimates Apply Rapid appraisal In-depth interviews of key informants at country implementing partner/sub-partner level Take a purposive sample 62 methods Focus group (FGD) and/or nominal group discussions of community health volunteers and beneficiaries (MDR patients and other potential beneficiaries). To increase the in formativeness of these discussions, they should best be held in gender and linguistic homogeneous groups when possible. This means that we need a FGD moderator in each of the countries who is fluent in English, the local language and in FGD techniques. Furthermore the discussions need to be audio￾recorded, typed out60 and translated into English. Then we can complete the analysis to get the main info out of the FGD and make a synthesis of the FDGs in each country. In each country 4 groups of volunteers (1 group of community health volunteers, 1 group of vulnerable populations such as MSM or sex workers, 1 group with male MDR patients, and 1 with female MDR patients). Optimally there should be at least 8 participants per group. Quick review (supplemented with discussion with facility workers) of health facility records61 in search of ‘critical incidents’, that reflect managerial interventions, such as health system delay, interventions when the MDR patient suffers from side effects of the treatment, timely treatment monitoring, etc. Health facility records Review of supervision reports62 to assess the effectiveness of the patient and data management, and also to check the appropriateness of the feedback, and the follow-up on earlier proposed interventions Supervision reports Qu 2: To what extend and how has the CAP-TB community based approach contributed to TB and MDR-TB prevention, diagnosis and treatment? With community based approach we understand the involvement of the local communities in the prevention, care and follow-up of the MDR-TB cases, such as through involving community health volunteers, participatory action research and through a sustained collaboration among the family members of the MDR-TB patients, the health professionals, the public health officials, the lab technicians and the pharmaceutical suppliers. DOT volunteers have a comprehensive role in reporting symptomatic family contacts, adverse medication side effects, TB-related complications, psychosocial stressors that threaten to affect patient’s adherence, and psycho-social counselling for the patient during the treatment. These DOT-volunteers need both initial and continuous training to strengthen the local capacity to address the health and social problems that plague those communities in which MDR-TB is endemic. The first part of the question (“to what extend”) will be answered through determining the dose￾60 Typing out the transcripts is preferable, although it is not essential if the translator can give a trustworthy brief of the information obtained during these FGDs 61 Given that the local language is usually being used to fill in the health records, this prosed review will require involvement of local staff. 62 If they are being made in English, otherwise they will have to be reviewed by local collaborators 63 response relationship between the amount and coverage of CAP-TB community based activities and the input, throughput, output and outcome indicators of TB and MDR-TB prevention, diagnosis and treatment. The second part of the question ‘how,’ will be explored through the use of rapid appraisal methods Use existing data Analyze the base-line data as a starting point for the pre-post comparison of the contribution of the CAP-TB community towards prevention, diagnosis and treatment performance Partner reports Review the monitoring data, in order to know their sufficiency to answer the raised questions, and to judge their quality. If the data are found to be sufficient in quantity and quality, then environmental epidemiological methods will be applied to explore dose￾response relationships and determine eventual thresholds. Id. Apply Rapid appraisal methods In-depth interviews of key informants at implementing partner/sub-partner level; these being community health volunteers, country implementing partners and implementers of the project Sampled through a purposive sampling scheme Focus group and/or nominal group discussions with community health volunteers and community beneficiaries The same persons who participate in the FGD mentioned in Qu 1 Development of spider diagrams Based on the available data at community level, items, such as capacity strengthening, referrals, DOT, relationship with health providers, and community mobilization, will be scored through applying a nominal group technique, and a spider diagram developed per country. This exercise might be done through involving a group of people who are acquainted with the usual of the TB control program and the CAP￾TB project The selection of these participants might be done in agreement with the national team Apply case￾control method At country level the global community scores will be ranked; the lowest quartile will be considered as “cases of low performance” and the communities of the highest quartile will be used as controls. A list of potential determinants will be developed through a causal analysis approach and their contribution searched through a case￾control method at community level. The odds ratios will allow to determine the contribution of the various potential determinants to the project performance, and also the attributable fractions The data will be obtained through interview of community leaders Qu 3: Are the management arrangements optimal for achieving project objectives efficiently and effectively? 64 Surveillance data management techniques Review the process of data recording63 , compilation, summarization, analysis, interpretation, and reporting The most efficient way to obtain these data is through the supervision reports Review the data quality64 assurance procedures with respect to error prevention, checking and data cleaning Id. as above. If time allows, a checking of the records and reports at the various levels will be undertaken Check the Data management supervision practices, inclusive the feedback reporting Supervision reports Apply Rapid appraisal methods In-depth interviews of key informants, such as government officers, government counterparts, NGOs and any other key agencies engaged in implementing the MDR-TB control program Through purposive sampling65 Review the capacity building efforts Review the frequency and intensity of the capacity building efforts (training programs and others) Training reports Review the monitoring of the post-training implementation efforts (such as continuous training during the supervisions, and refresher courses) Training reports and interview of trainees through field visits Qu 4: To what extent and how has the Strategic information generated by the project and knowledge gateway information been used? To get the right answers to these important questions (To what extent/ how) the methods that will be used rely on the evidence based program management approach. We will explore each of the 3 project countries to determine how actionable knowledge is being derived from the statistical information obtained from the routine data and/or from any available data, and if the actionable knowledge is being generated in an adequate way. Given time constraints this activity will be carried out on purposive sample basis. Next we will explore if the decisions are based on the generated actionable knowledge (=evidence-based). And finally we will check if the decisions are being translated into actions Analysis of the strategic decision making processes, based on the available routine data Review of the available information (quantity, quality, accessibility, appropriateness) as a potential basis for decision making at the various levels in the decision making process Surveillance reports Review the use of feedback mechanisms as capacity building opportunities Supervision reports, Evidence of feedback (through registers and/or interview) Apply rapid assessment methods Analysis of the appropriateness (in scope and time) of the actions taken In-depth interviews of decision makers 63 Review the process of data recording compilation, summarization, analysis, interpretation, and reporting 64 Here also there might be language barriers; the intervention of local staff is necessary 65 Here also there might be language barriers; then intervention of local staff should be indispensable as interviewer or as translator 65 Analysis of the (optimal) use of the strategic information by project partners, detection of eventual bottlenecks in the use of information In-depth interviews of decision makers66 Description of the profile of the users of the available information and Knowledge Gateway sources Interview with a purposive sample of the users Deriving propositions to improve the effective use of information Based on pooling of the obtained information Review of the decision making autonomy at the various levels in the decision making process of the CAP-TB project In-depth interviews of decision makers Review the annual planning processes Check the level of evidence being used as basis for the planning processes (at annual or other frequency Annual plans Qu 5: What measures/mechanisms have been put in place to achieve sustainability and which still need to be addressed? The assumption has to be made that an effective DOTS program should be in place before a MDR-TB treatment program could become sustainable, because a poor TB control program would generate continuously more MDR-cases than the TB control program could handle. Furthermore the strategic decision between a centralized, inpatient treatment program vs. an integrated, home-based treatment program is determined by the MDR prevalence and the bed capacity of the area. Use existing data Determine the total cost & expenditures, the unit cost per cured MDR case, and compare with unit cost in non-project areas (if the latter data exist and could be easily available to the experts; otherwise time constraints will not allow to make this interesting comparison) Project documents NTP documents Use rapid appraisal methods In-depth interview of key informants concerning the level of integration of project activities, the availability and use of efficiency and equity indicators, the acceptability of the project by the users, the authorities of the project areas and the national authorities Interview of purposive sample Focus groups and/or nominal group discussions with people involved in strategic decision making in MDR-TB control in each of the 3 countries, aiming to explore the generalizability of the approaches used in implementing the CAP-TB model, and also their ability and intention to leverage other partnerships and resources as a result of CAP-TB to implement approaches used in the MDR-TB project. Also, to determine the extent to which the three countries are benefitting from or taking advantage of a regional approach. Mixed public of decision makers involved in CAP￾TB areas and in areas which are not yet covered by the project Review existing plans Review 5-year plans, and other national/regional plans to discover intentions to implement or extend MDR-TB programs in non-project areas 5-year plans of the 3 countries 66 This information could also be obtained through FGDs, although there will be a serious time constraint; therefore the use of in-depth interviews is preferred 66 2. Proposed evaluation design 2.a/ Analytical framework This refers to the inputs, process, outputs and outcomes of the CAP-TB project in each of the 3 participating countries. It contains a series of checklists consisting of key question sand sub-questions to ask to the main actors that intervene in the MDR-TB project cycle. The research questions are focused on determining the level of successes, challenges, risks and mitigation strategies for the overall CAP-TB implementation and in each of the 3 countries. They also seek to establish any lessons learned to steer the remaining period of the project and to be an example for the non-project areas. Question Sub-questions 1. To what extend and how has the CAP-TB strengthened MDR-TB management? Is CAP-TB management focused towards a treatment partnership with the patients? Is there a continuous focus on patients’ concerns and priorities? Is the follow-up proactive? What action is undertaken if the patient misses an appointment? Are treatment side effects being monitored closely and addressed properly? How many MDR contacts are/have been followed-up? How many laboratories roll out diagnostics? How many staff members are trained in programmatic management of MDR-TB? 2. To what extend and how has the CAP-TB community based approach contributed to TB and MDR-TB prevention, diagnosis and treatment? Are lay providers (‘expert patients’, volunteers, peer educators) involved in patient management? Are action research activities undertaken and if yes, are they participatory? Is there coordination of care at community level? Is there a functional partnership of the MDR-TB control program with the local communities? Is there a continuous capacity building effort of the community health workers? 4. To what extend and how has the Strategic information generated by the project and knowledge gateway information been used? Are the staff at all levels capable of deriving evidence out of the routine surveillance data? Is the evidence being used for decision making? Are the actions taken based on the evidence derived from the surveillance data? Are the actions taken timely (What is the average delay between the evidence derived and the actions taken)? Are the actions taken addressing in function of the attributable fraction of the determinants of the difference between what is observed and what is expected? Do the authorities at peripheral level have enough decision making authority, so that they can use the information obtained through the surveillance data? 5. What measures/mechanisms have been put in place to achieve sustainability and which still need to be addressed? Is there sufficient bed capacity for the intensive phase? What are the challenges faced by the CAP-TB project to retain the MDR-TB patients after the intensive phase? What are the challenges to monitor and trace MDR-TB patients after their intensive phase? Is there an equilibrated integrated, home-based MDR-TB treatment 67 program? If yes, have all determinants of the performance of this home￾based program got sufficient attention? Is there a local commitment to financially support the MDR-TB control program once the major donors have withdrawn? 2. b/ Techniques and methods intended to be used to answer the questions The evaluation will make use of existing data that will be summarized by appropriate statistical methods to describe the progress of the MDR-TB problem in each of the 3 implementation countries as well as at a the regional coordinational level. The description will use the classical time￾place-person stratified approach. If it will be feasible to obtain performance estimators in exposed (=with CAP-TB exposure) and un-exposed areas (neighboring areas that are not yet covered by the CAP-TB project), then use will be made of rate and risk ratios. If indicated pooling will be used to obtain global estimates. Stratified analysis and logistic regression techniques will be used. Rapid assessment methods, such as participant observation, key informant interviewing, focus group discussions and nominal group techniques67 will be used during this evaluation. A key problem here is the level of mastering of English by the key informants68. It is therefore possible that some in￾depth interviews, focus group discussions and nominal group discussions need to be done in the local language of the concerned countries. If that would be the case, then the questions will have to be translated by someone who is well acquainted with the local language, the English language, and also with the local culture. The interviewing will have to be done by interviewers who are well acquainted with this type of qualitative research, and they should be supervised by an expert who is well acquainted with the local language, with English and with the subject matter. Furthermore the discussions need to be audio-recorded, typed out and translated into English. Given the serious time constraints of this mid-term evaluation in 3 countries, it could very well be that the external evaluators lack the time to carry out those FGDs and/or nominal group discussions as is desirable from a scientific point of view. 2.c/ Sampling plan Given the time constraints, it is probably not feasible to proceed to develop a listing of the study units (be it communities, health facilities, MDR patients, DOT providers, community health workers, PPM practitioners) as basis for a random sampling or a systematic sampling. Eventually a quota sampling could be a reasonable alternative to obtain representative findings, although that requires a very good knowledge of the study units and their geographical repartition. Therefore a purposive sampling scheme is opted for. 2.d/ Data collection instruments - for the quantitative data with respect to the MDR patients, the communities, and the case control subjects: data capturing sheet that will be introduced into excel format - for the qualitative data: the most common sources of qualitative data collection we intend to use are interviews, direct observation and reviews of documents. With respect to the interviews, most of the questions will be of an open-ended, conversational format. A highly structured format will probably be less applicable, given the fact finding nature of this evaluation, and the fact that various 67 Life history interviewing is an interesting narrative method that uses a form of individual interview directed to documenting the respondent's life, or an aspect of it that has developed over the life course. it could generate interesting empirical evidence of the critical incidence in the decision making of MDR-TB patients with respect to diagnosis, treatment and treatment adherence. Time constraints will be too much important for the evaluators to use this technique. 68 This consultant has a substantial experience in carrying out consultancies and teaching post-graduate courses to TB officers in Thailand and Myanmar, and had to face the problem of their limited mastering of the English language. Even during the post-graduate training courses interpreters had to be called in to made the message fully understood. 68 questions might have to be repeated in various formats, in order to get an unbiased answer69. A lot of attention will also have to be given also to non-verbal signals. 2.e/ Outputs The choice of the output indicators takes into consideration how the evaluation might contribute to improving the program implementation and, if necessary, the program itself; and also how the findings might contribute to the primary intended users making major evidence-based decisions about the program. These end-products refer to the Strategic Objective (SO): ‘To improve the prevention and management of MDR-TB in Burma, Thailand and China ‘’, with the following intermediate results (IR) and outputs. The list of indicators has taken all the indicators of the USAID Results Framework into consideration Outputs Indicators IR1: Strengthened MDR-TB prevention Mobilized communities to advocate for and use TB services N° of communities that use TB services out of total number of target communities Newly registered TB patients through USAID supported sites N° of patients registered Individuals reached with TB prevention and treatment messages N° of individuals reached Scaled-up implementation of TB infection control in health facilities N° of facilities where risk assessment was carried out , prior to MDR-TB control implementation, out of the total N° of health facilities with CAP-TB implementation N° of healthcare workers educated about cough and spitting etiquette, out of total N° of healthcare workers N° of MDR-TB patients educated about cough and spitting etiquette, out of total N° of MDR-TB patients N° of vulnerable populations getting adequate information and access to MDR services, out of total vulnerable population. Existence of standard Operating Procedures manual to address Tb-IC control N° of health facilities with a coordinating body for TB-IC surveillance of TB disease among health workers, out of total N° of health facilities N° of facilities with TB-IC ACSM materials, out of total N° of health facilities N° of facilities where these ACSM materials are used, out of total N° of health facilities N° of health facilities with a designated monitor of the Tb-IC prevention/control measures IR2: Strengthened MDR-TB management Ensured capacity, availability, and quality of laboratory testing to support the diagnosis and monitoring of TB Mean distance between residence of MDR-TB cases and nearest laboratory that is capable of diagnosing MDR-TB N° of laboratories that are well-equipped and have bio-safety cabinets, out of total laboratories that examine sample of MDR-TB 69 The evaluators are aware of the eventuality that some interviewees might be more concerned to give the type of answers that they think might please the evaluators, rather than to express their true meaning and feelings. 69 patients, including the rapid diagnosis of MDR-TB suspects N° of lab technicians working in those laboratories that receive appropriate MDR-TB training N° of such laboratories that practice internal quality control N° of such laboratories that are subject to external quality control Strengthened case-finding and referrals for MDR-TB N° of USAID supported facilities with strengthened MDR referral systems N° of MDR-TB suspects with DST, out of total N° of MDR-TB suspects N° of MDR proven cases that are referred for treatment, out of total N° of diagnosed MDR cases Health Services Delay of MDR-TB cases Strengthened human resource capacity for MDR-TB management N° of health staff involved in MDR-TB control that received appropriate training, out of total N° of health staff involved in TB control Scaled-up quality treatment and community approaches for PMDT N° of MDR-TB centers that reach vulnerable populations with quality services IR3: Improved strategic information for MDR-TB Strengthened capacity of TB programs to collected, analyze and use data for management N° of TB workers trained in data management, out of total N° who are involved with TB related data management Increased TB research activities N° of research projects related to MDR-TB N° of research projects that led to a publication or a scientific communication IR4: Strengthened enabling environment for MDR-TB Improved capacity of National Tuberculosis Programs (NTPs) to develop, finance, and implement national TB control strategies in line with global strategies N° of NTP staff that has received TB control training, out of total N° of staff involved in NTP control N° of national strategies developed and/or revised with CAP-TB support Strengthened partnerships for quality TB care, including private sector Existence of a coordinating body between NTP, antipoverty initiatives and health system strengthening at national and peripheral level Existence of an established partnership between NTP and the private medical practitioners N° of private practitioners who refer MDR-TB patients to the NTP, out of total N° of private practitioners who see MDR-TB suspects IR5: Strengthened TB/HIV Collaboration HIV testing of TB suspects % of MDR-TB patients tested for HIV through the CAP-TB project TB testing in HIV+ patients % of HIV+ patients for MDR-TB in HIV care or treatment settings with USAID support 2.f/ Plan of analysis. 2.f.1. With respect to the quantitative data: the analysis will in first instance be carried out following descriptive epidemiological principles and techniques, focusing on the crude data in first instance and also on the stratified data by time, place and person criteria. 70 The statistical analysis will be carried out through EDA (Exploratory Data Analysis70) techniques. Trends will be checked through comparing the performance data over time with the baseline data. The correct assessment of the impact of the project requires a comparison group. Therefore comparable data from neighboring areas, without exposure to the CAP-TB project, will be searched and explored as ‘unexposed areas’. If that is possible, then rate ratios can be determined and attributable fractions might be determined. 2.f.2. With respect to the qualitative data: As they require a very thorough mastering of the language of the interviewees/participants, and the mastering of the English language might be borderline in some of the countries71, in each of the 3 countries the intervention of people acquainted with rapid appraisal techniques, TB and the local language might be necessary. The recordings of the focus group/nominal group discussions and the in-depth interviews will have to be transcribed and translated72. Hereto we will need one or two translators in each country as we will need them to be unbiased, meaning it can’t be someone on the project asking the questions73. The qualitative information will be carefully explored, and subsequently organized into similar categories, e.g., concerns, suggestions, strengths, weaknesses, similar experiences, program inputs, recommendations, outputs, outcome indicators, etc. The categories or themes will be labeled, e.g., concerns, suggestions, etc., and an attempt will be made to identify patterns, or associations and causal relationships in the themes. Interfaces between the qualitative and qualitative findings will be established. 3. Evaluation approach The introduction of the CAP-TB project in the 3 countries might be considered as a natural experiment. Consequently the causal interpretation of the findings will need to be done prudently, and control of confounders and effect modifiers and search for alternative explanations of the measured progress will have to be a constant concern. The findings will be explored from the point of view of “searching for best practices”. Once a project area is being found that merits the label of “best practice”, then the characteristics of the project implementation in that area will be explored; a comparative analysis in the other project areas will be undertaken, to learn about the vulnerability for change in those areas. As the results of an evaluation are context- and people-dependent, the primary intended users of the findings will be involved as early, and their suggestions taken into consideration along the duration of the evaluation process. An effort will be undertaken to triangulate as much as possible the evaluation findings with published operational research findings carried out in the 3 project areas and to determine the value of a regional approach to exchanging best practices and lessons learned. ANNEX C: Calendar and Persons Met Time Activities Venues Contact # January o/a 15 ( for 6 days) 70 This is a very powerful distribution-free technique developed by Prof J. Tukey, Princeton university, USA 71 The team leader has the experience of teaching postgraduate courses to TB control officers in Thailand and Myanmar; the constant presence of a translator was indispensable, for reason of insufficient mastering of the English language by the participants. 72 As mentioned above: Typing out the transcripts is preferable, although it is not essential if the translator can give a trustworthy brief of the information obtained during these FGDs 73 We will verify if this could be done through the help of the USA embassy or through the USAID office. 71 Preparatory phase includes planning meeting (in person or teleconference) with RDMA and USAID HQ, reading background documents and first deliverables and revised deliverables. Tuesday, 4 February 2014 22.10 PM Carina Stover Arrive Bangkok with OZ 741 at 22.10 Plaza Athenee Hotel (conf # 326170) Wednesday, 5 February 2014 06.20 AM Aime De Muynck arrives Bangkok with TG 935 at 06.20 Plaza Athenee Hotel (conf # 326181) 14.00-16.30 Meeting with M&E Team; Jittinee & Marisa and AOR Majuro Room (Room # 2634) Thursday, 6 February 2014 09.00-10.00 Meeting with Health Advisor, Dr. Aye Aye Thwin Aye Aye's Office on the 25th floor 10.00-12.00 Conf. call with Elisa Adelman, previous COR for CAP-TB Almaty call # +7-727- 2507612 (ext.6358) 12.00-13.30 Lunch 13.30-14.00 Check-in with USAID Office Director, Chris Barrett (Confirmed) Chris's Office 15.00-16.00 Meeting with Sara Whitehead, USCDC (Confirmed) Majuro Room (Room # 2634) Friday, 7 February 2014 8.00-14.00 Meeting with FHI360 FHI360's Office PM Meeting with Dr. Chawetsatn (Confirmed) Bureau of Tuberculosis Saturday, 8 February 2014 All day Planning meeting of the Evaluation Team, synthesizing notes Sunday, 9 February 2014 3.30 pm Leave for Rayong (from Plaza Athenee) 6.00 pm Check in at Phavina Serviced Residence Khun Siriporn (Nok): 081- 173-4176 Evening Free Monday, 10 February 2014 8.50 am Leave for Rayong Provincial Health Office 9.00 am – 12.00 pm Meet with Rayong Provincial Health Office Ms Phantira Jitman 086- 344-6067 12.00-12.45 pm Lunch 12.45 pm Leave for Rayong Hospital 1.00-4.00 pm Meet with Rayong Hospital Phuwanet Meeting Room 5th Floor, Rayong Hospital Ms Chanthima Polsak 092- 783-0132 Evening Free Tuesday, 11 February 2014 72 8.20 am Check out Mr Sompong Sansri 086- 234-5513 8.30 am Leave for Ban Phe Sub-district 9.00 am – 12.00 pm Meet with NCCM Ban Phe Sub￾district Health Promotion Hospital 12.00-1.00 pm Lunch at Ko Hong Afternoon Leave for Bangkok Wednesday, 12 February 2014 Time TBD LSHTM meeting with London School at USAID Office USAID (Majuro Room # 2634) 15.00-16.30 Meeting with Dr. Mukta Sharma, WHO Thailand (Confirmed) USAID (Majuro Room # 2634) 17.00 Depart for Suvarnabhumi Airport to catch flight to Rangoon PG 705 (2015-21.30) Thursday, 13 February 2014 10:30 AM In-brief with USAID/Burma US Embassy 1:00 PM Meeting with FHI 360 Team FHI 360 office Friday, 14 February 2014 9:00 AM Meeting with CAP-TB Burma IAs, MMA, PGK, MBCA, MHAA FHI 360 office 2:00 PM Meeting with WHO WHO office Saturday, 15 February 2014 Data analysis and report drafting for Burma section Sunday, 16 February 2014 Free time Monday, 17 February 2014 1:00 PM Daytrip to Nay Pyi Taw to meet National TB Programme manager Disease Control Compound, NPT Tuesday, 18 February 2014 9:00 AM Meeting with Regional Officer, Lower Myanmar TB Lower Myanmar TB, Aung San 10:30 AM Visit to MHAA Office MHAA Office 2:00 PM Visit to Mayangone township health center and meeting with Township Medical Officer Mayangone THC Wednesday, 19 February 2014 9:00 AM Visit to South Okkalapa township health center and meeting with Township Medical Officer South Okkalapa THC 10:00 AM Meeting with MDR TB Patients South Okkalapa THC 12:00 PM Meeting with MMA CAP-TB GP MMA 2:00 PM Travel to Mandalay. Yangon Domestic airport Night Stop Golden City Hotel Thursday, 20 February 2014 8:30 AM Meeting with MBCA field staff Golden City Hotel 11:00 AM Meeting with MHAA field staff MHAA Office, Mandalay 73 Night Stop Night stop Golden City Hotel Friday, 21 February 2014 7:00 AM Travel to Yangon. Mandalay International airport 1:00 PM De-brief with USAID/Burma US Embassy 5:30 PM Travel to Bangkok by PG 704 (18.45-20.40) Yangon International Airport Saturday, 22 February 2014 14:05-15:30 KMG Travel to Kunming and hotel check-in Aime and Amy : TG612 10:55-14:05 New Era Hotel Ge Ya 13577192602 16:30-17:00 Evaluation team internal meeting with FHI360 China for schedule review New Era Hotel meeting room Sunday, 23 February 2014 Free time Ge Ya 13577192602 Monday, 24 February 2014 09:00-12:00 Meeting with FHI 360 CAP-TB team and YATA/Yunnan CDC Staff - YATA gives an overall introduction on: 1) TB/MDR-TB in Yunnan; 2) government response FHI KM office meeting room Ge Ya 13577192602 14:05-15:30 KMG Travel to Kunming and hotel check-in Carina: TG612 10:55-14:05 New Era Hotel 13:00 – 17:30 Meeting with FHI 360 CAP-TB team and YATA/Yunnan CDC Staff - FHI 360 present CAP-TB overview - Q&A/open discussion FHI KM office meeting room Tuesday, 25 February 2014 8:30-9:00 Travel to Xi Shan CDC 08:30-10:50 Meeting with Xi Shan CDC - Mr. XIA gives a background introduction on TB in Xi Shan District /online database) - Ms. LI Yan and Ms. ZHANG Qian introduces the CAP￾TB project work by Xi Shan CDC - Q&A/open discussion Xishan CDC Mr. Xia Xuejun 13629602995 10:30-12:00 Visit Xing Yuan community for DM/TB screening - meeting with community staff who carry out the DM/TB screening - meeting with DM/TB patient DM/TB screening site Mr. Xia Xuejun 13629602995 12:00-13:00 Lunch 13:00-13:30 Travel to Chuang Fang Community 13:00-15:00 Visit Chuang Fang Community for TB/MDR-TB follow up - Meeting with TB staff from Fu Hai Community Health Center and Chuang Fan Community Health Station - Meeting with TB patients from the community Chuang Fang Community Mr. Xia Xuejun 13629602995 15:00-17:00 Travel to Yunnan Bureau of Health and meet with Ms. Ou Yang, YN Bureau of Health Wednesday, 26 February 2014 8:30-9:00 Travel to Yunnan TB Clinical Center (TCC) 9:00-12:00 Visit Yunnan TB Clinical Center (TCC) for TB/MDR-TB treatment and care Yunnan TB Clinical Center Ms. Guo Ya’nan 74 - Ms. GUO Yanan introduces the background information of TCC and an overview of the CAP-TB work at TCC - TCC doctor, nurse and peer counselor introduces CAP￾TB activities at TCC - Visit the TCC facility - Meeting with group of TB/MDR –TB patients and family members - Q&A/open discussion 13888649793 12:00-13:30 Lunch 13:30-14:00 Travel Kunming No.3 hospital 14:00-17:30 Visit No. 3 hospital for TB/MDR-TB treatment and care Focus on grants and OCD (PM) - Site visit of outpatient clinic and introduction of out￾patient and in-patient enrollment procedures in Kunming No.3 Hospital - Group education session for TB/MDR-TB patients and family members (optional) 14:00-15:00) - Introduction of CAP-TB project progress in Kunming No.3 Hospital - Meet with TB doctors - Meeting with TB/MDR-TB patients Kunming No.3 Hospital Dr. Li Mingwu 13816888842 Thursday, 27 February 2014 8:30-9:30 Travel to Yunnan Care Center 9:30-11:00 Meeting with doctors from Yunnan Care Center Yunnan Care Center Dr. Li Xia 13099423809 11:00-12:00 Travel back to the downtown 12:00-13:30 Lunch 13:30-15:00 Internal meeting for the evaluation team New Era Hotel 15:00-18:00 Feedback meeting with FHI360 CAP-TB China team New Era Hotel Night Data analysis and report drafting Friday, 28 February 2014 Morning Data analysis and report drafting New Era Hotel 16:40-19:55  BKG Travel to Beijing CA4173 16:40-19:40 Westin Hotel Saturday, 1 March 2014 10:00-12:00 Meeting with director Chen Mingting from NCTB Dr. Lin Yan 13910122674  US Flight back to US Sunday, 2 March 2014 Free time Monday, 3 March 2014 09:00-10:30 Meeting with WHO Fabio IUATLD office Dr. Lin Yan 13910122674 10:30-12:00 Meeting with director Lin from Union China office Union office Dr. Lin Yan 13910122674 17:05  BKK Flight back to BKK Tuesday, 4 March 2014 PM Travel from Beijing to Bangkok with TG 615 at 17.05 Wednesday, 5 March 2014 Report writing and preparing a presentation for briefing, meeting with FHI360 for more information if needed Majuro Room (Room # 2634) Thursday, 6 March 2014 75 Report writing and preparing a presentation for briefing, meeting with FHI360 for more information if needed Majuro Room (Room # 2634) Friday, 7 March 2014 9:00AM Debriefing for FHI360 FHI360 Office 11.00AM Debriefing at RDMA (entire Mission) Dili room (Room # 2515) PM Evalution Team meeting and planning for finalization of report Monday, 8 March 2014 00.30 AM Aime De Muynck departs Bangkok with TG 934 at 00.30 01.20 AM Carina Stover departs Bangkok with OZ 744 at 01.20 ANNEX C: Persons Interviewed Name Position FHI360 IN BRIEF .Anh Innes CAP-TB COP Jackie McPherson Deputy Regional Director, FHI 360 Asia Pacific Regional Office Eric Stephan CAP-TB Knowledge Management Advisor Sutinee Charoenying APRO Prog. Manager, Program Management Unit Siritharin Chareonsiri APRO Program Officer, Program Management Unit Hatairat Jirajariyavech CAP-TB Finance Officer Aaron Schubert HIV/AIDS and TB Team Lead Marisa Sanguankwamdee Program Management Specialist Jittinee Khienvichit Monitoring and Evaluation Specialist PHO - Rayong Miss Patchara Jampathong SSF Project Coordinator Ms. Jongdee Indrasub Provincial TB Coordinator Miss Phantira Jitman CAP-TB Project Coordinator Rayong Hospital Mrs. Rungtiva Burathep TB Clinic Nurse Dr. Bralee Suntiwut Pulmonologist Mrs. Jaruwan Iamsaard Nurse Miss Piyapat Phongprasert Medical Technologist Miss Wassana Utaisawung Public Health Officer, CAP-TB Project Coordinator Mrs. Sunsanee Yuwapat Infection Control Nurse Miss Praweena Kurewong Radiographer Miss Jantima Polsak Public Health Officer, CAP-TB Project Coordinator NCCM Bangkok Office Mr. Aumnaj Srivichai Finance Officer Mr. Pipat Traichan Program Coordinator Rayong Office Mr. Sompong Sansri Field Manager 76 Ms. Nongnut Noisin Field Officer Ms. Sangchan Prangchan Field Officer Ms. Piyarat Suebsaen Field Officer BTB Dr. Chawetsan Namwat Director, BTB Dr. Sriprapa Nateniyom Medical Doctor, BTB BURMA USAID/Burma Thullan T. Dinh Health Tech Advisor USAID/Burma William Slater Director, Health Officer Dr. Hloo Aung Cho Aye Myat Zan Admin Assistant FHI360 CAP-TB OFFICE YANGOON Dr. Zow Ni Thaung Program Officer Dr. Soe Htut Sung Sr. M&E Officer Dr. Phyo Wai Tun Program Officer Dr. Khin Zarci Aye Country Program Manager Dr. Ag Myirnt Than Acting Program Coordinator Aung Khim Program Coordinator Thant Zarv M & E Officer, MHAA Dr. Nae Nae Oo Coordinator Sith Itotu Agry Project Officer Dr. Khim Sue Win Project Manager Dr. Kyah Tinn Sun Project Officer Naung Naung Project Officer D. Marg San Program Manager Myanmar South Okalpa Clinic Dr. Aye Aye Moe TMO, South Okalpa Dr. Ilcaine Scoe Win Project Manager MMA CAP-TB Project Dr. Saw Nwet Nwet Myit Team Leader, TB, NTP Dr. Kyann Linn Tun Project Officer, MMA CAP-TB T/M – Daw Tin Mar Win TB Coordinator Dew Hla Hla Win MMA CAP-TB U Ang Min Soe O.R.W (Pyi Gyi Khin) Dr. Sutur Project Officer Private General Practitioners - Yangon Dr. Khaing May Ag GP De Illun Than Nwe HS TB Coordinator MBCA Dr. Kar An Area Coordinator Ma Thwei Thwei Myint ORW Leader Dr. Saw Nwet Mipit NTP Team Leader MHAA 77 Ba Maung Township Community Facilitator Moe Kalyar Moe Admin and Finance Nway Aye Mya Htun Township Community Facilitator CHINA Name Department/Title Feb.24,2014 Monday FHI360 meeting room Mr. ZHA Shun Deputy Director of YN CDC and Board Chairman of YATA Ms. XU Lin Secretary-general of Yunnan Anti-Tuberculosis Association Mr. CHEN Jing’Ou TB Program Officer, Yunnan Anti-Tuberculosis Association Ms. YANG Huijuan TB Program Officer, Yunnan Anti-Tuberculosis Association Ms. PAN Yuying TB Program Officer, Yunnan Anti-Tuberculosis Association Ms. LI Ling CAP-TB China Program Manager, FHI 360 Ms. XU Zhixiang CAP-TB China Program Officer, FHI 360 Ms. ZHAO Xinru CAP-TB China Program Officer, FHI 360 Ms. GE Ya CAP-TB China Finance and Admin Officer, FHI 360 Feb.25,2014 Tuesday Xishan CDC Mr. XIA Xuejing Head of Chronic Diseases Division, Xi Shan CDC Ms. LI Yan CAP-TB project officer/Community Outreach Worker, Chronic Diseases Division, Xi Shan CDC Ms. ZHANG Qian CAP-TB project officer/Community Outreach Worker, Chronic Diseases Division, Xi Shan CDC Mr. CHEN Jing’Ou TB Program Officer, Yunnan Anti-Tuberculosis Association Ms. YANG Huijuan TB Program Officer, Yunnan Anti-Tuberculosis Association Chuanfang Community Dr. HAN Jianjun, Head/TB staff, Chuan Fang community health station Dr. SONG Ruowen TB staff, Fu Hai Community Health Center Mr. PI Dongxing Beneficiary/TB patient, Chuan Fang community Mr. XIA Xuejing Head of Chronic Diseases Division, Xi Shan CDC Ms. LI Yan CAP-TB project officer/Community Outreach Worker, Chronic Diseases Division, Xi Shan CDC Ms. ZHANG Qian CAP-TB project officer/Community Outreach Worker, Chronic Diseases Division, Xi Shan CDC Mr. CHEN Jing’Ou, TB Program Officer, Yunnan Anti-Tuberculosis Association Ms. YANG Huijuan TB Program Officer, Yunnan Anti-Tuberculosis Association Yunnan Health Bureau Director OU’YANG Lin Deputy Director - Disease Control Bureau of Yunnan Health Bureau Ms. XU Lin Secretary-general of Yunnan Anti-Tuberculosis 78 Association Mr. CHEN Jing’Ou TB Program Officer, Yunnan Anti-Tuberculosis Association Ms. YANG Huijuan TB Program Officer, Yunnan Anti-Tuberculosis Association Ms. PAN Yuying TB Program Officer, Yunnan Anti-Tuberculosis Association Ms. LI Ling CAP-TB China Program Manager, FHI 360 Feb.26,2014 Wednesday Yunnan TCC Ms. GUO Ya’nan Deputy Director - General Management Office, TCC Dr. MAO Xiaoyun Deputy Director - Clinical Division, TCC Dr. YIN Kunfu Deputy Director - TB Laboratory, TCC Ms. NIE Guiying Chief Nurse, Clinical Division, TCC Ms. YU Zhonghui Program Officer/TB counsellor, General Management Office, TCC Ms. ZHANG Wen Nurse/TB Counselor, Clinical Division, TCC Mr.HUANG Zhengdong Peer Counsellor, 57 Zone Mr. WANG Linsheng Peer Counsellor, 57 Zone LU Hongguang, LIU Yongbo, YAO Xinzhong and He La Ma Ta TB patients Mr. GOU Songxiang Family member of TB patient Mr. CHEN Jing’Ou TB Program Officer, Yunnan Anti-Tuberculosis Association Ms. YANG Huijuan TB Program Officer, Yunnan Anti-Tuberculosis Association Ms. PAN Yuying TB Program Officer, Yunnan Anti-Tuberculosis Association No.3 Hospital Dr. LI Mingwu Director of No.2 TB Division, Kunming No.3 Hospital Dr. LAI Minghong vice director of No.2 TB Division, Kunming No. 3 Hospital Dr. LIU Cai TB doctor, Kunming No.3 Hospital Dr. LIU Yongli TB doctor, Kunming No.3 Hospital Dr. LIU Sha TB doctor, Kunming No.3 Hospital Dr. ZHANG Kaiyi TB doctor, Kunming No.3 Hospital Dr. CHENG Chonhui TB doctor, Kunming No.3 Hospital Dr. WANG Lin TB doctor, Kunming No.3 Hospital Dr. GAO Wenjun TB doctor, Kunming No.3 Hospital Dr. WAN Rong TB doctor, Kunming No.3 Hospital Ms. MA Meng CAP-TB project liaison person\Nurse supervisor, Kunming No.3 Hospital Ms. YANG Qiuyan Nurse\TB counselor, Kunming No.3 Hospital Ms. LUO Wenjuan Nurse\TB counselor, Kunming No.3 Hospital Ms.LI Mengyan Nurse\TB counselor, Kunming No.3 Hospital Ms. DU Chaomei Nurse\TB counselor, Kunming No.3 Hospital Ms. LI Cuicui Nurse\TB counselor, Kunming No.3 Hospital Feb.27,2014, Thursday Yunnan Care Center Mr. YANG Xingping Director of No.2 Infections Division, Yu nnan AIDS Care Center 79 Ms. LI Xia Director of No.1 Infections Division, Yunnan AIDS Care Cen ter Ms. LI Ling CAP-TB China Program Manager, FHI 360 USAID/RDMA DEBRIEF Michael Yates Director’s Office, USAID/RDMA Carrie Ann Thompson Director’s Office, USAID/RDMA Aaron Schubert Office of Public Health, USAID/RDMA Aye Aye Thwin Office of Public Health, USAID/RDMA Christopher Barrett Office of Public Health, USAID/RDMA Jittinee Khienvichit Office of Public Health, USAID/RDMA Marisa Sanguankwamdee Office of Public Health, USAID/RDMA Thitima Klasnimi Office of Public Health, USAID/RDMA Jedsada Taweekan Program Development Office, USAID/RDMA Katherine Younker Program Development Office, USAID/RDMA Nitasmai Ransaeva Program Development Office, USAID/RDMA Selam Kebrom Program Development Office, USAID/RDMA Shirley Hoffmann Program Development Office, USAID/RDMA Suzanne Polak Program Development Office, USAID/RDMA Caroline Bertolin Regional Office of Procurement, USAID/RDMA Praveena Virasingh Regional Office of Procurement, USAID/RDMA Alfred Nakatsuma Regional Environment Office, USAID/RDMA Daniel Whyner Regional Environment Office, USAID/RDMA ANNEX D: Questions for Site Visits Location/Country _____________________________________________ 1 What is the average patient delay of the MDR/TB patients ? Do you know the average delay due to the patients, lack of access to diagnostic and/or treatment centers, and health system delay? 2. How many supervision visits did your centre receive the last 3 months? If there have been supervision visits: who was the supervisor? If there have been supervision visits: did the supervisor look into the data management system? If there have been supervision visits: was there a feedback given? Were the feedback instructions implemented? Was the effect of the implementation monitored? 3 Number of TB suspects examined (by direct microscopy) during the last 12 months (you find that in the 80 lab register) How many of those suspects examined were MIGRANTS? (Even if there is no systematic monitoring, you can find that through counting the names of the migrants) 4 How many of the TB cases diagnosed did NOT start treatment (= these are the initial defaulters) (you can find that by comparing the number of the diagnosed patients , [see lab register] and the treatment register ) 5. Who is responsible for the quarterly reporting of the Tb data? 6. To whom are the quarterly reports being send? 7. Is anyone at your centre responsible for analysing the TB data? 8. How do you check the quality of the data? If you suspect that there might be errors in your data, what do you do? 9. Has anyone been trained in data management at your centre? 10. Is the quarterly report being discussed during your staff meetings? If yes, what aspects are being discussed? 11 List for each MDR -TB patient the DOT provider? Indicate for each DOT provider if he/she had been trained? Indicate how each DOT provider was trained? Are the DOT providers being supervised ? If yes, by whom? Do you have a spot map of the residence of the MDR -TB patients? Do you have a spot map of the DOT providers? 81 12 Is there a systematic registry of the side effects of the second line drug treatment? 13 Is there a systematic registry of the side effects of the first line drug treatment? 14 How many of your MDR/TB patients have missed an appointment during the last month? What does your staff do when an MDR/TB patients misses an appointment? 15 How does your staff do contact tracing of CATI TB patients? How does your staff do contact tracing of CATII TB patients? How does your staff do contact tracing of MDR/ TB patients? 16 How far do each of the MDR/TB patients live from your centre? 17 How far does each MDR/TB patient live from the nearest laboratory? 18 How many of your MDR/TB patients were referred by private practitioners? 19 Do you have the notification rates of the NSP (New Smear Positive TB cases) patients, stratified by age and sex? If not, this is the way how to proceed/ - You have the distribution of the NSP cases through your quarterly report - You can have the age and sex distribution of your population through the census data. If they are not available, you can request those from the national NTP office, or you can google them (because most of the time, they are accessible through internet). If the census data are not recent, that does not matter so much; please use whatever you can find In this way you have the numerator data (from your report), and the denominator data (from the census), and you calculate the rates per 100,000 pop Next you plot those data in a line chart; this means you have a line chart with 2 lines: 1 for the males, and 1 for the females 20 How often are you visited by USAID/RDMA or Washington? Communications and supervision? 21 How often do you visit with other countries with CAP-TB or other MDR program? 22 How do you stay up to date on TB-MDR? Locally and elsewhere? 23 Do you have a MDR model plan? If so, how was it developed? If not,why not? 24 Are you documenting lessons learned? If so, can you share them with us? 82 25 How have you adjusted your budget based on budget cuts? What support have you gotten to this? 26 Likewise, how have you adjusted your performance indicators? 27 If you have not done the budget or performance adjustments, how do you plan to get this done and how do you think it will turn out? For instance, will you continue to expand? Will there be staffing cuts or hour reductions? 28 Overall, how is your program going and what are successes and challenges? 29 What other organizations or funds do you depend on and how? (Global Fund and SLDs) 30 What do you think of community-based participation and how much are you supporting this? 31 What do you think of de-centralisation of MDR care and mgt.? 32 Any thoughts on incentivizing health volunteers? 33 What do you think of the role of doctors and other health workers? How do you help them with this? How do you try to modify current practises? (supply training, modify work responsibilities, etc) 34 Do you have anything for capacity building tracking? 35 Do you work at all with the private sector? If so, how? 36 Do you see any turnover of responsibility (technical or financial) to the local government or NGOs? If so, what? 37 Do you have any cross border issues? 38 Is there a stigma associated with MDR and TB that could be addressed? 39 Gender issues you are addressing? 50 How do you define vulnerable populations and how are you targeting them? 51 Have you noted any drug supply, stock outs, expiration, and cost issues? 52 What is happening with types of diagnosis and the machines/supplies? 53 Patient tracking systems exist or working? 54 How have you impacted on identifying potentially infected persons, diagnosis, and successful treatment? 55 What training have you and other staff received? What do you need? 56 Do you work at all with the private sector? 57 Observe stocks in facilities 58 Observe compliance with adequate and quality sanitation 59 Question patient satisfaction in facilities 60 Question how patients were diagnosed and what they were told about transmission (to other family members for instance) treatment and compliance, etc 83 ANNEX D: References and Source of Information and Data used WHO & The Union. Collaborative Framework for care and Control of Tuberculososis and Diabetes WHO. Policy Framework for implementing New Tuberculosis Diagnostics, March 2010; pgs. 21. Nishikiori N. Strategy for TB g-high risk and vulnerable populations. Conference given to 7th National TB Programme and Laboratory Managers’ Meeting; 12-15 Sept. 2011, Manila, The Philippines. Brust J., Sarita Shah N, Scott M., et al. Integrated, home-based treatment for MDR-TB and HIV in rural South Africa: an alternate model of Care. IJTLD 2012; 16: 998-1004. Shin S, Furin J, Bayona J, Community-based treatment of multidrug-resistant tuberculosis in Lima, Peru: 7 years of experience. Soc. Sci & Med. 2004; 59: 1529-1539. Zhou Ch, Chu J.C., Liu J, et al. Adherence to Tuberculosis treatment among migrant pulmonary Tubefculsosis patients in Shandong, China : a quantitative survey study. PLoS ONE 2012; 7(12):e52334.doi:10.1371/journal.pone.0052334. Corbett E.L., Watt C.J., Walker N., Maher D., Williams B.G. , Raviglione M.C., Dye C. , The growing burden of tuberculosis: global trends and interactions with the HIV epidemic. Arch Intern Med. 2003; 163(9): 1009-21. Goldhaber-Fiebert JD, Jeon CY, Cohen T, et al. Diabetes mellitus and tuberculosis in countries with high tuberculosis burdens: individual risks and social determinants. Int J Epidemiol 2011; 40 (2): 417- 428. Basu S., Stuckler D., Bitton A., Glantz S.A. Projected effects of tobacco smoking on worldwide tuberculosis control: mathematical modelling analysis. BMJ, 2011; 343 (oct04 1): d5506 DOI: 10.1136/bmj.d5506 Barroso EC., Mota RMS, Santos RO, et al. Risk factors for acquired MDR-TB. J Pneumol 2003; 29: 89-97. Pazarlı P., Dildar Yetiş D., Yazıcıoğlu Moçin O., Karagöz T.; The Effect of Smoking on Treatment Outcome of Multidrug­Resistant Tuberculosis. Turk Toraks Derg 2013; 14: 93­7. 84