USAID/INDIA REACH – THE ACCOUNTABILITY LEADERSHIP BY LOCAL COMMUNITIES FOR INCLUSIVE, ENABLING SERVICES (ALLIES) PROJECT MID-TERM PERFORMANCE EVALUATION Final Evaluation Report Collaborating, Learning & Adapting in India Mechanism (CLAIM) IDIQ # 72038619D00001 Task Order # 72038620F00003 June 15, 2023 DISCLAIMER: The views of the expressed in this document do not necessarily reflect the views of the United States Agency for International Development or the United States Government ii | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov ACKNOWLEDGEMENT The evaluation team wishes to thank Chandan Samal, Subrato Mondal, Mohd. Tariq, Anita Sharma, and Amrita Goswami of the United States Agency for International Development (USAID)/India their guidance and support. We are very thankful for the time and contribution of all those interviewed, especially the Tuberculosis Champions (TBCs) and other national, state, and district officials who shared their valuable time and experience with us to inform this evaluation. We, particularly, want to acknowledge the contribution and support of the ALLIES (Accountability Leadership by Local Communities for Inclusive, Enabling Services) team, which has been a close partner throughout this evaluation. We would like to thank each member of the ALLIES team for providing documents, facilitating site visits, and providing detailed information on project activities. Finally, the team wishes to thank the people with tuberculosis (PwTB) who graciously gave us time and shared their experiences and personal stories. This allowed the evaluation team to fully understand how the project influenced their lives and how it can strive to do more. We would like to specifically acknowledge the following team members of the ALLIES team who supported this midterm performance evaluation: NPMU Mr. Subrat Mohanty Dr. Sripriya Pandurangan, Mr. Sumit Kumar Chhattisgarh Team Mr. Fidius Kerketta Mr. Ravi Ranjan Ms. Sarla Singhania Jharkhand Team Ms. Anima Kisku Mr. Vikash Kumar Choudhary Mr. Rahul Shekhar Odisha Team Mr. Laxmidhar Singh Mr. Susant Mishra Mr. Tapan Kumar Pradhan Dr. Jyoti Ranjan Mohanty Tamil Nadu Team Mr. K. Balasubramanian Mr. P. Rajeswaran Mr. R. Karthikeyan USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | iii ABBREVIATIONS ADR Adverse Drug Reaction AETBCS Achieving Excellence in TB Care and Services AKAM Azadi ka Amrit Mahotsav ALLIES Accountability Leadership by Local Communities for Inclusive, Enabling Services ANM Auxiliary Nurse Midwife ASHA Accredited Social Health Activist AWP Annual Work Plan BAP Block Action Plan C2A Call-to-Action Project CAF Community Accountability Framework CAFR Community Accountability Framework Report CBO Community-Based Organization CDCS Country Development Cooperation Strategy CHO Community Health Officer CHPC Comprehensive Primary Health Care CINI Child in Need Institute CLM Community Led Monitoring CMO Chief Medical Officer CP Continuation Phase (for Anti-TB Treatment) CSO Civil Society Organization CSR Corporate Social Responsibility CTD Central Tuberculosis Division DBT Direct Benefit Transfer DOTS Directly Observed Treatment, Short-course DPC District Program Coordinator DPPMC District Public Private Mix Coordinator DR-TB Drug-Resistant Tuberculosis DS District Strategist DS-TB Drug Susceptible Tuberculosis DTO District Tuberculosis Officer EE Empowerment Evaluation ELM Employer-Led Model EPTB Extrapulmonary Tuberculosis ER Elected Representatives ET Evaluation Team FAST Find Access Support Treat FICCI Federation of Indian Chambers of Commerce & Industry FY Fiscal Year GIP Gender Integration Plan GLRA German Leprosy and TB Relief Association GOI Government of India HIV Human Immunodeficiency Virus HR Human Resources HWC Health and Wellness Centre IDI In-Depth Interview IEC Information Education Communication iv | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov IP Implementing Partner IP Intensive Phase (for Anti-TB Treatment) IWD International Women's Day JHPIEGO Johns Hopkins Program for International Education in Gynecology and Obstetrics KHPT Karnataka Health Promotion Trust KII Key Informant Interview KTSN Kalinga Tuberculosis Survivors Network LOI Letter of Intent LON Local Organization Network LOP Life of Project LPA Line Probe Assay LT Laboratory Technician M&E Monitoring and Evaluation MDR-TB Multiple Drug Resistant Tuberculosis MEL Monitoring, Evaluation, and Learning MLA Member of Legislative Assembly MLC Member of the Legislative Council MO Medical Officer MOHFW Ministry of Health and Family Welfare MOU Memorandum of Understanding NHM National Health Mission NPY Nikshay Poshan Yojana NTEP National Tuberculosis Elimination Program OPD Out Patient Department PDCA Plan-Do-Check-Act PRI Panchayati Raj Institution PwTB People with Tuberculosis QoC Quality of Care QoS Quality of Services QPR Quarterly Progress Report REACH Resource Group for Education and Advocacy for Community Health SBCC Social and Behavior Change Communication SC Sub Center SLN Survivor Led Network SNC Sub National Certification SOP Standard Operating Procedure STDC State Tuberculosis Training and Demonstration Centre STLS Senior Tuberculosis Laboratory Supervisor STO State Tuberculosis Officer STS Senior Treatment Supervisor TAT Turn Around Time TB Tuberculosis TBC Tuberculosis Champion TBF Tuberculosis Forum TBMCGF TB Mukt Chhattisgarh Foundation TBS TB Survivor TEJ Tuberculosis Elimination Jharkhand TPT Tuberculosis Preventive Treatment USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | v TU Tuberculosis Unit UDST Universal Drug Susceptibility Testing UNAIDS Joint United Nations Program on HIV/AIDS USAID United States Agency for International Development USG United States Government WHO World Health Organization WTB World TB Day vi | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov TABLE OF CONTENT ACKNOWLEDGEMENT II ABBREVIATIONS III TABLE OF CONTENT VI EXECUTIVE SUMMARY 1 PROJECT BACKGROUND 1 EVALUATION PURPOSE AND LEARNING QUESTIONS 1 METHODOLOGY 2 FINDINGS AND CONCLUSIONS 2 RECOMMENDATIONS 7 INTRODUCTION 9 PROJECT BACKGROUND 9 EVALUATION PURPOSE 10 LEARNING QUESTIONS 10 METHODOLOGY 12 EVALUATION DESIGN / METHODS 12 DISSEMINATE AND REPORT 14 LIMITATIONS OF THE EVALUATION 15 FINDINGS AND CONCLUSIONS 16 TBC MODEL FINDINGS 26 ACTIVITIES TO BE DONE BY TB CHAMPIONS 26 EMPOWERMENT EVALUATION 38 ALLIES GOAL AND DESIGN 43 ADDITIONAL FEEDBACK FROM PERSON WITH TUBERCULOSIS (PWTB) 47 RECOMMENDATIONS 51 RECOMMENDATIONS FOR REACH 51 RECOMMENDATIONS FOR USAID (FOR USAID EYES ONLY) 53 ANNEX 1: LIST OF STAKEHOLDERS INTERVIEWED 55 ANNEX 2: EVALUATION TOOLS 57 ANNEX 3: EVALUATION MATRIX 69 ANNEX 4: FINDINGS ABOUT TBCS FROM ALLIES DISTRICTS 73 ANNEX 5: NON-ALLIES PROJECT DISTRICT TBC MODEL FINDINGS 76 ANNEX 6: CAF TOOL 78 ANNEX 7: SURVIVOR LED NETWORKS’ MEMBERS TRAININGS 90 ANNEX 8: DISCREPANCY IN TARGETS 91 ANNEX 9: ALLIES LOP TARGET ANALYSIS 92 ANNEX 10: SUGGESTIONS - ALLIES TBC’S ROLES 96 USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 1 EXECUTIVE SUMMARY Panagora conducted a mid-term performance evaluation of the USAID/India-funded Accountability Leadership by Local Communities for Inclusive, Enabling Services (ALLIES). Implemented by the Resource Group for Education and Advocacy for Community Health (REACH), ALLIES is a four￾year activity from September 26, 2019, through September 25, 2023. This report presents the findings and conclusions of the midterm performance evaluation and makes recommendations on how to improve its interventions to ensure that ALLIES meets its stated goals. PROJECT BACKGROUND ALLIES activity is being implemented with the goal “To enable environments for TB elimination by leveraging community action as an ally to build a culture of accountability through a rights￾responsibilities-based approach.” REACH piloted a model of TB champions (TBCs) under an earlier USAID-supported C2A project. The Government of India (GOI) has accepted the TBC model for scale-up across the country. ALLIES takes the C2A work forward in 9 districts in Odisha, Jharkhand, and Chhattisgarh and added six districts in Tamil Nadu and has added an innovation called the Community Accountability Framework (CAF), a tool that tries to identify and bridge the gaps in quality of services (QoS) and quality of care (QoC) for TB patients. The TBCs are trained to implement the CAF through the administration of a survey to a small number of people with TB (PwTB) in their area. The CAF tool is not intended as a survey to measure program effectiveness, but rather to provide immediate feedback from the community to the health system about the gaps, and a plan for addressing the same is developed. The ALLIES activity aims to enable environments for TB elimination by supporting community action to build a culture of accountability using CAF as a tool. The objectives of the activity are: 1. To create powerful advocates as key change agents or community enablers who can undertake strategic advocacy for enabling environments at multiple levels to shape rights￾respectful, gender and age-responsive TB services. 2. To establish community-owned mechanisms to monitor the quality of TB care and services, and give feedback to the program for timely responses, helping institute accountability and strengthening community empowerment. 3. To generate local solutions and resources in response to identified needs, such as counseling for behavior change, nutrition, local support groups, social services, etc. 4. To promote discourse on enabling the policy, regulatory and financial environments to support TB elimination and reduce TB-related stigma/discrimination at state and national levels. EVALUATION PURPOSE AND LEARNING QUESTIONS This mid-term evaluation of ALLIES aims to assess the activity’s performance and progress in achieving its objectives to date, identify areas for improvement, and suggest adjustments to existing program interventions. The specific Learning Questions are: 1. To what extent has the ALLIES activity made progress toward its aim ‘To enable environments for TB elimination by leveraging community action to build a culture of accountability’? 2 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov 2. How have the intervention activities and approaches integrated gender across program implementation? What evidence exists to substantiate the reduction of gender gaps? 3. To what extent the Community Accountability Framework (CAF) is implemented and what is the effectiveness? How has the Empowerment Evaluation (EE) approach contributed to the implementation of the CAF process? 4. How effective the new district and state-level TB survivors-led networks are in facilitating the rights-based approach to TB, and what is their role in advocacy promotion? 5. What are the overall accomplishments, challenges, and learnings out of the activity implementation so far? There are also some detailed sub-LQs defined by USAID. These are presented and discussed in the body. This Executive Summary focuses on the main LQs. In the course of this evaluation, USAID/India added learning questions directed toward districts where the GOI is implementing the TBC model: a. What is the progress towards deployment and operationalization of TB Champions (TBCs) by the state government in the study district/s? How effectively are the TBCs in the selected GOI districts supporting the improvement of TB healthcare services and facilitating support to PwTBs to access quality care? b. What role has ALLIES played and its contribution towards the provision of guidance, policy advocacy, capacity building, tools, etc. towards effective deployment and operationalization of TBCs in the GOI districts? c. What are the overall accomplishments, challenges, and learnings in the process? What data are being used by the state government to monitor the implementation and effectiveness of the TBC approach? METHODOLOGY The assessment used a mixed-method approach for data collection and analysis. The team reviewed the project documents and used a participatory approach to gather information from USAID/India, the Implementing Partner (IP), NTEP officials at the National, State, and District levels, other stakeholders, and the community. ALLIES MIS data and PwTB feedback data were reviewed in detail. These data were verified, triangulated, and analyzed to develop a comprehensive and cohesive account of the achievements and challenges and recommend actions for the future. We reviewed the data collected by ALLIES through the CAF instrument and concluded that it was not appropriate to use as a measure of program effectiveness in improving health system QoC and QoS. FINDINGS AND CONCLUSIONS The main objectives of this mid-term performance evaluation were to assess the activity’s progress in achieving its objectives to date and identify areas for improvements, suggested adjustments to existing program interventions, and recommended priorities for future implementation. As per USAID Evaluation Policy, the evaluation was principally aimed to serve learning purposes by identifying the challenges incurred by the ALLIES activity to date and formulating appropriate recommendations for corrective actions and effective implementation during the remaining years of the activity. In this Executive Summary, for each learning question (LQ), we start with a summary of ALLIES’ main achievements and follow with detailed discussions of the challenges. In the main body of the report the achievements and challenges are discussed together within the overall findings narrative for each LQ. USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 3 LQ 1 To what extent has the ALLIES activity made progress toward its aim ‘To enable environments for TB elimination by leveraging community action to build a culture of accountability’? Achievements to Date • ALLIES has built on the TBC model piloted in the C2A project and trained 654 TBSs to TBCs with Tamil Nadu contributing over 300 to this achievement. The project has further gone ahead in strengthening this cadre by equipping them with additional skills in leadership, communications, digital tools, human rights and advocacy, counselling and support for livelihoods and income-generation. • The ALLIES strategy has also brought in elected representatives (ERs), private sector employers, and journalists. It has reached out to about 500 ERs to sensitive them on TB issues, over double of the overall target. ALLIES has increasingly focused the work with ERs on local governments, as they provide opportunities for tangible changes, for example, ensuring that PwTB in their constituencies receive GOI social services and subsidies. • In the activities with the private sector, under the rubric of the Employer-Led Model (ELM) which was piloted by REACH during the C2A project, ALLIES engaged them in TB awareness, health education, and service delivery. ALLIES reached out to almost 200 companies, requesting them to sign Letters of Intent to support National Tuberculosis Elimination Program (NTEP) activities. • ALLIES has leveraged REACH’s existing Media Fellowships and Awards program, which was established in 2008. ALLIES has supported 44 journalists across the country and built their capacity to report ethically on TB. This effort led to 93 stories in local languages on various issues related to TB. Challenges • While ALLIES had made considerable efforts to sensitize a large number of ERs the program has not been able to strategically garner their support to improve quality of care (QoC) or quality of services (QoS) for PwTBs. The ER strategy success varies by state depending on the core functional strength of the ER system, that works well in Tamil Nadu but less so in Jharkhand. The focus on leveraging national or state social programs schemes (e.g., livelihood, pension, scholarships, food and housing, untied funds, welfare schemes for tribal and backward classes) is large ad hoc and lacks a strategic focus. • While ALLIES sensitized about 200 corporations, less than 30% committed to support TB elimination and further only 8% conducted any activities. The ALLIES’ ELM strategy has not leveraged the CSR pool of funds that many of the corporates bring to the mix. • There is no strategic targeting of corporations by the type of establishment/ business such as foundries, cement manufacturing, spinning that particularly compromise lung health and predispose to TB. • The ALLIES activity has not targeted the informal sector that employs a larger proportion of unskilled/ migrant workers. This is a huge gap as the unorganized sector does not provide any medical coverage/social security to employees, state schemes may not cover migrants and they live and work in highly vulnerable conditions. • The media fellowship is a useful program to build capacity on reporting around important health, socioeconomic and human rights aspects of TB. There seems to be some loss in follow up with not all media fellows delivering at least three stories that are required to successfully complete the fellowship. There is anecdotal evidence that these stories have led to changes in TB programs. 4 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov LQ 2 How have the intervention activities and approaches integrated gender across program implementation? What evidence exists to substantiate the reduction of gender gaps? Achievements to Date • ALLIES took the lead in supporting the Central TB Division in rolling out ‘The National Framework for a Gender Responsive Approach to TB.’ The activity supported the design of training materials for master trainers and conducted training workshops for Master Trainers from 33 states and Union Territories. • Gender issues have been integrated in key training curricula developed by REACH: “Achieving Excellence in TB Care and Services (AETBCS) for providers”; “CAF for TBCs”; “Right Based Approach on TB & Health for TBSs and TBCs”; and “TB Survivor to TB Champion”. Challenges • While the ALLIES Theory of Change (TOC) mentions gender sensitive and gender responsive TB services, no specific activity or output in the project is aimed at achieving it. The original ALLIES program design is limited to disaggregating gender data post-facto. • Some of the critical aspects of the accountability dimension of ALLIES do not incorporate gender. For example, the CAF tool, feedback to system, and engagement with other ALLIES do not factor in any specific gender related issue. • The ET does not have access to any ALLIES monitoring or survey data to comment on reduction in gender gaps. • The master trainers’ training conducted at the national level sets the ball rolling on implementation of the ‘The National Framework for a Gender Responsive Approach to TB’, however, it needs follow-up by ALLIES to maintain the momentum at least in the project states. LQ 3 To what extent the Community Accountability Framework (CAF) is implemented and what is the effectiveness? How has the Empowerment Evaluation (EE) approach contributed to the implementation of the CAF process? Achievements To Date • The TBC model piloted by REACH in the previous USAID funded activity has been adopted by the GOI and is being scaled up by states with training of TBCs supported by partners working in the TB space. Through ALLIES, REACH continues to support the GOI´s scale up efforts. • The most important new strategy developed under ALLIES has been the CAF approach which is layered on the TBC model developed in the previous USAID funded activity. • The CAF toolbox was adapted in consultation with multiple stakeholders to enhance the accountability, coverage and effectiveness of TB programs and generate demand by strengthening community confidence in TB services1. ALLIES has reached out to 26,939 PwTB to get feedback on TB services in 4 states, and further worked on this feedback to plan and improve TB services across 137 TUs in 15 districts. 1 CAF IN ACTION- Operationalizing a Community Accountability Framework in India, REACH USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 5 Challenges • While ALLIES planned to include many other allied organizations in the use of CAF, such as the ER and SLNs, currently the process is exclusively the responsibility of the TBCs. There seems to be a weakening of the ‘community’ part in the Community Accountability Framework. While the TBCs do come from the same communities affected with TB, they work as an integral part of the NTEP team taking instructions from the STS. The process has moved towards being a PwTB feedback survey. The feedback process at the district and state level is informal and opportunistic. The tool as it is being used does not empower communities. • The TBCs are envisioned to be community representatives and champions who can advocate for the rights of PwTB to access quality service, be treated with respect and not be discriminated against either within the family, community, or workplace. However, with respect to accountability and the use of the CAF tool, in the scaled-up version of the model the TBCs are appointed as the lowest ranking team member in the TU, working in subordination to and on instructions of the STS. While the GOI’s TBS to TBC curriculum mentions advocacy as a function, it is unclear how the lowest rung member of the NTEP team can effectively advocate for better services. • The selection of PwTBs for application of the CAF tool seems to vary considerably by TU/states, so the results have to be interpreted with caution. In TUs where the PwTBs selected are the ones that are hard to reach, it serves the purpose of follow up well for NTEP, but does skew the CAF results. There is an inherent bias in the process as the person collecting the data is also responsible for working on the gaps. More importantly, given that it does not have a robust sample design, it cannot be used to measure program effectiveness in improving the health system’s QoC and QoS for PwTB. • The CAF process is time-consuming and takes about 70% of the TBCs’ time. This is more so due to multiple points of data recording and entry (KOBO and ALLIES app). Even though the ALLIES app was aimed at making the process paperless and swift, even with the fully functional version of ALLIES app there will be steps of manual analysis of the CAF data and development of the BAP. The process does not seem to be time-efficient. • In the absence of service data from Nikshay the ET cannot comment on CAF’s effectiveness in improving QoC and QoS. There has also been a strong government push towards effective implementation of NTEP due to the goal of TB elimination by 2025. A comparative analysis of the ALLIES and non-ALLIES districts would have provided more evidence but could not be done. LQ 4: How effective the new district and state-level TB survivors-led networks are in facilitating the rights-based approach to TB, and what is their role in advocacy promotion? Achievements to Date • Building on the previous USAID funded program, C2A, ALLIES has supported capacity building of Survivor Led Networks in all project states. The membership of the SLNs has grown during the project with Tamil Nadu having almost 1400 members. • The SLNs have been a key player in representing the interests of PwTB through state and district level TB Forums. A step forward in the ALLIES project has been the creation of district level networks that undertake advocacy at the district level. These networks have been leading celebration of important days so as to bring attention to reducing stigma for PwTB. 6 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov • ALLIES has supported the SLNs to showcase their collective strength e.g., 46 TBCs from the Delhi network participated in the ‘National Conference on Women Winning Against TB’, presided over by the Vice President of India, and Cabinet Ministers. ALLIES has also supported two cross learning visits of the SLN Leadership to other states to share experiences. • ALLIES has been supporting the SLNs with different livelihood trainings that help PwTB find economic opportunities. ALLIES has supported the SLN in Chhattisgarh in starting a unit for manufacture of products such as cleaning products, soaps and slippers with support from a corporate. The SLN is developing linkages with bulk users such as hospitals and hotels for sale of these products. • ALLIES partnered with Touched by TB in August 2020 to create a directory of TB networks and Civil Society Organizations (CSOs) and strengthen the SLNs in the North-east (Assam, Sikkim, Meghalaya, and Nagaland) and Delhi. The Touched by TB Network was awarded the ‘On-Ground Heroes’ award for 2022 by the Apollo Tyres Foundation. Challenges • Networks are yet to mature as institutions that can manage and sustain independent of ALLIES support. District level networks are at 'starting-up' stage with sporadic activities • Institutional strengthening of the SLNs needs more work in terms of role clarity, governance and management process, work planning, monitoring, reporting, etc. LQ 5 What are the overall accomplishments, challenges, and learnings out of the activity implementation so far? Achievements to Date • The ET has interpreted this LQ as a summatory question. Building on the previous USAID activity, C2A, and in ALLIES, REACH has created a cadre of TB champions who are empowered, committed to serving and making sure that the lives of people suffering from tuberculosis can be made better. They have been empowered through numerous capacity building efforts to help them to be strong voices for all PwTBs in their sphere of influence. • With USAID funding, REACH has engaged organizational allies to prioritize TB as a cause of health, economic and social problems. ALLIES has engaged elected representatives, employers, media and survivor networks to create awareness of the need to address the disease and the role each of these stakeholders can play. The Project team along with the TBCs celebrated all important occasions such as the WTD, IWD, AKAM, Gandhi Jayanti and others. • The evaluation team’s review of the periodic progress reports shows at mid-term that ALLIES continues to make progress in most of the interventions. Most of the MEL plan output indicators reflect this progress. • ALLIES has become the “go to” team for the states to provide TBS to TBC training in districts beyond USAID funded areas. The team has supported the project states in the statewide TBC trainings. The effort has included building capacity of other partners such as JHPIEGO, KHPT and The UNION as trainers for countrywide scale up of TBC strategy. The ALLIES team has also supported the training in some non-project states such as Haryana for TBC training. • ALLIES organized the TB Champion Conclave on 23rd March, 2023 in Varanasi. The conclave centered around the evolving role of a TB Champion, voices from the community and strategies to strengthen community engagement. Over 50 TB champions from across USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 7 the country participated and shared their stories of empowerment on the eve of World TB Day. They also shared their contributions in the fight to end TB as well as the gaps and challenges that must be addressed to end TB in India by 2025. The Varanasi Statement was released to capture the essence of this community mobilization and commitment. • ALLIES took the lead in developing the strategy document; “To End Stigma and Discrimination Associated with TB” in partnership with other partners working on tuberculosis elimination. The strategy has been endorsed by the government of India and serves as the guidance for the country. The ALLIES project developed the Workplace Policy on TB for the state of Jharkhand after extensive deliberations with the Departments of Health and Industries. The Policy has been notified by the Jharkhand Cabinet making Jharkhand the first state to have such a policy across the country. Challenges • Engaging and empowering communities can be elusive, and this reflects in the ALLIES design. ALLIES sews together diverse stakeholders through multiple activities to move towards the larger goal in the project geographies. While the project Theory of Change (ToC) mentions some concrete activities and outputs, it is unclear on how these activities and outputs would move ALLIES towards tangible outcomes. • Considering the ToC does not have clearly defined outcomes, ALLIES is not tracking any outcome indicators. Most of the 16 indicators in the MEL Plan and detailed in the Performance Indicators Reference Sheet are output indicators. The few outcome indicators are intermediate, rather than final outcomes. In that the CAF tool does not have a robust sampling method, the CAF data are not appropriate for measuring outcomes. • Thematic Expertise on social and behavior change (SBC) is lacking in the ALLIES team. Being primarily a community engagement project, the absence of SBCC/IEC expert in the team is noticeable. This is reflected in the inadequacy of communication materials and absence of SBCC strategy for improving health seeking behavior and stigma reduction. • CAF is a good initial effort and has potential as a concept, but it needs to be further refined to provide some feedback to the health system. It is a time-consuming process, the sampling process varies by geography, monthly data collection seems unnecessary, data sharing and follow up mechanism is informal. • While the GOI has adopted the TBC model, the current CAF approach is likely not able to be layered on the scaled-up TBC model. The value of CAF comes from it being independent of the health system and yet it is led by TBCs who are an integral part of the GOI’s health bureaucracy. RECOMMENDATIONS As mentioned above, the Recommendations put forward by the Evaluation Team are in function of the main challenges identified in the Findings and Conclusions section of the report. RECOMMENDATIONS FOR REACH 1. Consolidate the TBC Model on the principles of “Build Operate Transfer”. It is crucial that the GOI and the state governments have complete toolkits to effectively implement the TBC model at scale. 2. Strategize on transitioning TBCs to the new Health and Wellness Centers (HWCs). REACH should play a lead role in this transition with a careful eye on developing the tools and processes for TBCs, NTEP and also the HWC ecosystem to make this change. 8 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov 3. Making CAF more effective by reducing questions and frequency of administration. The questions should be tested for validity, reliability, and consistency. The processes for providing feedback and bridging gaps should be systematized and documented. Data entry and analysis should be streamlined for prompt action 4. The ALLIES team should strategize on how the learnings from the CAF implementation in ALLIES be transformed to a scalable model. As the TBCs are constrained in pushing for accountability within the GOI system, we recommend that the district level survivor’s networks can be the community voice to assess quality of care and provide feedback to the health system. 5. ALLIES should develop a clear Organization Development Plan with the SLNs to build them into technically and financially strong compliant institutions that can fund raise from the government and donors to design advocacy programs for PwTB needs. 6. Leverage Employers to nurture skilling and income generation activities. The Corporate Social Responsibility Funds can be tapped into for the same as well. The Corporates could also support through employee volunteering to convert PwTBs trained in certain skills to develop into microentrepreneurs either individually or as collectives. 7. Describe what ‘rights- respectful’, ‘gender’ and ‘age responsive’ TB services should look like and how to get there. Demonstrate a model that makes services ‘rights- respectful’, ‘gender’ and ‘age responsive’ and is simple enough to be scaled. RECOMMENDATIONS FOR USAID 1. Take the lead in restructuring ALLIES to make it more strategic, methodical and evidence– based. There needs to be a particular focus on better data use for monitoring and tracking. 2. Health Office should strategize on how to support GOI policy of transitioning the TBCs to HWCs. There is a need to consider which partners/ grants would be best suited to play this role in conjunction with RAECH. 3. Define a cohesive funding strategy for providing technical assistance to the government to scale up the TBC model to be effective in achieving the goal of TB elimination. 4. The Employer Led Model can be taken up a notch by leveraging private sector partnerships across the USAID/India Mission, working with the Private Sector Engagement (PSE) team in the Mission 5. Work with CTD, MOHFW, GOI to regularly analyze NIKSHAY data for in-depth insights for improving the NTEP implementation. The ET understands that this is a difficult task but suggests that exploring viable mechanisms to do this would be a big step forward. 6. The Achieving Excellence in TB Care and Services (AETBCS) curriculum and training should be considered for providers under other USAID grants working with service providers across various technical components of NHM. 7. Finally, given that the CAF data are not suitable for measuring program outcomes, USAID should collaborate with the GOI to conduct a quantitative analysis of TB time series service data and to measure the contribution of the ALLIES’ approaches and tools improving QoC and QoS for PwTB, for example using a matched sample of ALLIES TBC and GOI TBC districts. USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 9 INTRODUCTION USAID/India has contracted Panagora Group to provide monitoring, evaluation, and learning (MEL) services to generate learning inputs for the implementation of the Mission’s Country Development Cooperation Strategy (CDCS). Under this contract, Panagora conducted a mid-term performance evaluation of a USAID/India funded tuberculosis (TB) activity, Accountability Leadership by Local communities for Inclusive, Enabling Services (ALLIES), implemented by the Resource Group for Education and Advocacy for Community Health (REACH). ALLIES is a four-year activity that runs from September 26, 2019, through September 25, 2023. The ALLIES activity is being implemented in 15 districts of Chhattisgarh, Jharkhand, Odisha, and Tamil Nadu (TN), training TB Champions who work with the National TB Elimination Program (NTEP) to implement a Community Accountability Framework (CAF), to understand and improve the Quality of Care (QoC) and Quality of Services (QoS) offered to people with TB. Initially, ALLIES partnered with Child in Need Institute (CINI) in Odisha and Jharkhand and with German Leprosy and TB Relief Association (GLRA) in Chhattisgarh but had to drop them due to changes in Indian statutes regarding sub-grants of donor funds. ALLIES also has a non-financial collaboration with Touched by TB for working in five states (Assam, Sikkim, Meghalaya, Nagaland, Delhi) to strengthen TB Survivor Led Networks (SLNs). This report presents the findings and conclusions of the midterm performance evaluation and makes recommendations on how to improve its interventions to ensure that ALLIES meets its stated goals. PROJECT BACKGROUND Most multilateral and bilateral agencies are committed to and are funding TB elimination globally. World Health Organization’s (WHO) End TB Strategy aims to reduce three-fourths of all TB deaths and reduce the incidence by half by 2025 (from the 2015 baseline). USAID’s Global TB Strategy 2023-2030 commits to provide high-quality TB technical and development assistance through programs founded on diversity, equity, and inclusion principles, and implemented in partnership with affected individuals and communities. The TB Local Organizations Network (LON) is a key component of the USAID Global Accelerator to End TB and partners with local organizations to implement locally generated solutions to improve TB diagnosis, treatment, and prevention. The ALLIES activity is aligned with USAID-s Global TB Strategy principles, and builds on its predecessor, the USAID/India funded TB Call-to-Action (C2A) (2016-2019) that was implemented in six states (126 districts) to strengthen a community response and advocate for increased financial, intellectual, and other resources for TB. ALLIES activity is being implemented with the goal “To enable environments for TB elimination by leveraging community action as an ally to build a culture of accountability through a rights-responsibilities-based approach.” ALLIES takes the C2A work forward in 9 districts in Odisha, Jharkhand, and Chhattisgarh and added 6 districts in Tamil Nadu. C2A piloted a model of TB champions (TBCs) by working with TB survivors (TBS) who undergo training and volunteer to support People with Tuberculosis (PwTB), reduce stigma, raise awareness, provide feedback to the health system, and advocate for TB-affected communities. While providing feedback to the health system was part of the role, no systematic mechanism was created. Experiences from the C2A and other projects such as Tuberculosis Health Action Learning Initiative (THALI) led to the Government of India (GOI) accepting the TBC model for scale-up across the country. The Central TB Division (CTD) issued the “From TB Survivors to TB Champions: A training curriculum” and requested all states to deploy two TBCs in each block. 10 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov The ALLIES activity aimed to take the C2A work forward by layering on a pilot to create an accountability mechanism for identifying and bridging gaps in QoS and QoC. The TBCs are trained to implement CAF wherein they complete a tool with a selected number of PwTBs in their area. Feedback is provided to the health system about the gaps and a plan for addressing the same is developed. The ALLIES activity aims to enable environments for TB elimination by supporting community action to build a culture of accountability using CAF as a tool. The objectives of the activity are: 1. To create powerful advocates as key change agents or community enablers who can undertake strategic advocacy for enabling environments at multiple levels to shape rights￾respectful, gender and age-responsive TB services. 2. To establish community-owned mechanisms to monitor quality of TB care and services, and give feedback to the program for timely responses, helping institute accountability and strengthening community empowerment. 3. To generate local solutions and resources in response to identified needs, such as counseling for behavior change, nutrition, local support groups, social services etc. 4. To promote discourse on enabling the policy, regulatory and financial environments to support TB elimination and reduce TB-related stigma/discrimination at state and national levels. EVALUATION PURPOSE This mid-term evaluation of ALLIES aims to assess the activity’s performance and progress in achieving its objectives to date, identify areas for improvement, and suggest adjustments to existing program interventions. The evaluation aims to serve learning purposes by identifying challenges faced by the activity and formulating appropriate recommendations for corrective actions and effective implementation during the remaining duration of the ALLIES activity and for the design and implementation of other, future USAID/India activities. The specific objectives of the midterm evaluation are: a. Assess progress towards achieving the project's aim and objectives. b. Assess the validity of the project's strategic approaches and results framework. c. Assess program performance by Implementing Partner (IP). d. Identify lessons learned. e. Recommend actions to improve performance, strategy, and future design. LEARNING QUESTIONS Through the findings derived from data collected over the course of this mid-term performance evaluation, the evaluation team (ET) will seek to provide conclusions and recommendations related to the following Learning Questions (LQs) and sub-questions formulated by USAID/India: 1. To what extent has the ALLIES activity made progress toward its aim ‘To enable environments for TB elimination by leveraging community action to build a culture of accountability’? • To what extent the activity has created the key ALLIES who can undertake strategic advocacy for enabling environments at multiple levels to shape rights-respectful, gender and age-responsive TB services? USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 11 • To what extent ALLIES has established community-owned mechanisms to monitor quality of TB care and services, and give feedback to the program for timely responses, helping institute accountability and strengthening community empowerment? • To what extent has the activity successfully promoted discourse on enabling the policy, regulatory and financial environments to support TB elimination and reduce TB-related stigma/discrimination at state and national levels. 2. How have the intervention activities and approaches integrated gender across program implementation? What evidence exists to substantiate the reduction of gender gaps? 3. To what extent the Community Accountability Framework (CAF) is implemented and what is the effectiveness? How has the Empowerment Evaluation (EE) approach contributed to the implementation of the CAF process? 4. How effective the new district and state-level TB survivors-led networks are in facilitating the rights-based approach to TB, and what is their role in advocacy promotion? 5. What are the overall accomplishments, challenges, and learnings out of the activity implementation so far? In the course of this evaluation, USAID/India added several questions directed towards districts where the GOI is implementing the TBC model: a. What is the progress towards deployment and operationalization of TB Champions (TBCs) by the state government in the study district/s? How effectively are the TBCs in the selected GOI districts supporting the improvement of TB healthcare services and facilitating support to PwTBs to access quality care? b. What role has ALLIES played and its contribution towards the provision of guidance, policy advocacy, capacity building, tools, etc. towards effective deployment and operationalization of TBCs in the GOI districts? c. What are the overall accomplishments, challenges, and learnings in the process? What data are being used by the state government to monitor the implementation and effectiveness of the TBC approach? 12 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov METHODOLOGY EVALUATION DESIGN / METHODS The assessment used a mixed-method approach using both quantitative and qualitative methods for data collection and analysis. The team used a participatory approach to gather information from USAID/India, the Implementing Partner (IP), NTEP officials at the National, State, and District levels, other stakeholders, and the community. These data were verified, triangulated, and analyzed to ensure that the findings, conclusions, and recommendations are based on an accurate understanding of the ALLIES activity. ALLIES midterm performance evaluation was conducted in four phases: (1) Document Review, (2) Sampling and Research Tool Development, (3) Assessment and Analysis and (4) Sharing Findings and Report Writing. PHASE 1: DOCUMENT REVIEW Phase 1 of the evaluation consisted of a desk review of key project documents provided by USAID/India and the REACH, which included: ALLIES Cooperative Agreements, Monitoring Evaluation and Learning Plans (MELs), Performance Indicator Plan, Annual Work Plans (AWP); Quarterly Progress Reports, Training Reports and Curriculum, Technical and Operational Guideline, Assessment Study Reports, Case Studies, Technical Reference documents, Information Education Communication (IEC) materials, etc. The desk review formed the basis for the evaluation approach and lines of inquiry which were explored and clarified further in meetings with USAID/India and ALLIES. PHASE 2: SAMPLING AND RESEARCH TOOL DEVELOPMENT Phase 2 entailed the development and finalization of the sampling plan and research tools. Sampling Plan The ET designed a sampling plan for districts across all four ALLIES states. One district was selected from each of the four states based on the distribution of the urban, rural, and tribal geographies of ALLIES implementation areas. The sample includes one urban, two rural and one tribal district. Table 1 provides the break-up of number of respondents across the four project states. The ET had selected Anugul district in Odisha which was later changed to Mayurbhanj at the request of the ALLIES team. ET was informed that the project is lagging in activities in Anugul due to turnover in NTEP staff and Mayurbhanj district is performing better with all interventions in place. The sample district was changed to Mayurbhanj with USAID/ India concurrence. USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 13 Table 1: Stakeholders Interviewed State District TB Unit (TU) TBC PwTB Other National & State level Respondents National State District/TU Chhattisgarh Raipur 2 9 8 14 6 11 Jharkhand Gumla 2 5 7 7 15 Odisha Mayurbhanj 2 5 8 9 12 Tamil Nadu Coimbatore 2 4 8 9 8 Total 4 8 23 31 14 31 46 The ET interviewed a total of 145 respondents; 130 in the ALLIES districts and 15 in non-ALLIES districts where the government is scaling up the TBC model. USAID requested the ET to include additional evaluation questions during the course of the evaluation. This expanded data collection to districts where the state governments are scaling up the TBC model. While all four states were planned, only three states were covered as respondents in Tamil Nadu could not be contacted. The table below presents the district and mode of data collection for non-project districts. Research Tools Key Informant Interviews (KIIs): The list of key respondents was finalized in consultation with the USAID/ India and ALLIES teams. The list of respondents is available in Annex 1. ET developed different KII tools for different respondent categories keeping in view of their role and level of engagement with the project. The tools were finalized in consultation with USAID/India. The KII tools are available in Annex 2. The KII tools covered the following areas of inquiry: • Understanding of project design, key interventions and how well they are responding to the priorities at state, district, and community levels • Progress, achievements, and challenges for the project interventions • Understanding of projects strategy by allies, their role, contribution, project inputs for capacity building, technical reference material, job-aids, achievements, and key gaps and challenges • Understanding of CAF pilot, process of its deployment, feedback for health service improvement, key gaps, suggestions for its improvement • Overall, project’s learning, achievements, key gaps, and challenges State District Interview Mode Chhattisgarh Dhamtari face-to-face Jharkhand Sahibganj virtual Odisha Sundargarh virtual 14 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov In-Depth Interview (IDI) with People with Tuberculosis (PwTB): Client perception/ feedback is considered an important component contributing to measurement of health outcomes and quality of services and care. ET conducted semi-structured interviews with PwTBs to understand their perception regarding access, adequacy, affordability and quality of care and services. The interviews also attempted to gauge their perception on social stigma, discrimination, access to socio-economic entitlements, etc. PHASE 3: ASSESSMENT AND ANALYSIS Phase 3 entailed the analysis of both qualitative and quantitative data collection (secondary data), and its analysis. Qualitative data analysis: KIIs with USAID/ India stakeholders, REACH ALLIES team, CTD, and 4 project state and district NTEP officials, health care providers and other technical partners; interviews with and observations of TB Champions; semi-structured interviews with PwTBs in selected TB Unit (TU) catchment area; and the in-depth interviews with NTEP officials and TB Champions in non-project areas. Quantitative Data Collection: ALLIES shared project MIS data with ET. The data included targets and achievements on the key performance indicators (Life of Project (LOP), annual and quarterly), coverage data of TUs and TBCs in the project, CAF output data (from April 2020 to January 2023), data from various trainings the project conducted, coverage data of SLNs, activities with Elected Representatives (ERs) and activities with Employer Led Model (ELM). ET also planned to analyze GOI’s service data to gauge the change in performance of health services in terms of quality of care, access to health services, adherence rate and treatment outcome and trends in gender gaps. However, the data was not provided to ET. Data Analysis: ET translated and transcribed the interviews (from Hindi, Odia and Tamil) to English. The ET organized the data by relevant evaluation questions and subsequent areas of inquiry (Refer Annex 3 for Evaluation Matrix) from the KII transcripts using MS Excel. A triangulation analysis was then conducted across the analysis from the KIIs, secondary data gleaned during the desk review and project’s quantitative data. The triangulation supported ET to develop a comprehensive and cohesive account of the achievements and challenges and recommend actions for the future. Confidentiality The ET informed and sought verbal agreement with all respondents regarding confidentiality of their personal information and responses via informed consent at the beginning of the interview. Following Agency policy and Panagora Group’s internal research policy, each participant’s identity, personal information, responses, etc. have not been disclosed to anyone outside of the research team. All audio recordings were analyzed by the ET to respect respondent confidentiality and will be erased after the report's approval. DISSEMINATE AND REPORT Preliminary Findings Presentation and Feedback Three consultative presentations with USAID/India and one with REACH shared high level findings and sought feedback (e.g. availability of documented TBC selection criteria and listing of their roles, USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 15 availability of some of the Technical Reference Materials for TBCs, etc.) to clarify ET’s preliminary findings and inform drafting of the evaluation report. Report Development and Finalization Based on USAID feedback, ET drafted a detailed evaluation report and submitted it to Panagora Group for feedback and further input. The USAID/India Program Office reviewed a first draft and requested changes. The Executive Summary was rewritten to ensure that ALLIES achievements were clearly placed alongside the challenges identified by the evaluation. LIMITATIONS OF THE EVALUATION The evaluation methodology had some potential biases and limitations that have implications for the types of findings and conclusions that can be drawn from this mid-term evaluation. These include: • Positive response (‘halo’) bias: Probing questions sometimes result in positive response bias i.e., the tendency of respondents to subjectively focus on positive outcomes. The teams mitigated this bias by probing for both successes and challenges to develop the most holistic picture possible relative to the evaluation questions. • Selection bias: Selection bias is an inherent risk when implementers help to facilitate contact with members of some stakeholder groups. The team worked closely with USAID and ALLIES staff to organize KIIs. ET requested the ALLIES team to share a complete list of all stakeholders prior to starting data collection to mitigate the risk of selection bias. Subsequently, the team identified individuals from this list to contact for interviews. • Subjective measurements: Qualitative approaches can result in performance analysis being dependent on the professional opinions and experience of the evaluation team which may result in findings, conclusions, and recommendations derived from their subjective interpretations. The team mitigated this bias through systematic triangulation of findings across stakeholder groups and methods and drew evidence-based conclusions and recommendations based on the data rather than on their professional experiences. 16 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov FINDINGS AND CONCLUSIONS The main purpose of this mid-term performance evaluation was to assess the ALLIES activity’s progress in achieving its objectives to date and identify areas for improvements, suggested adjustments to existing program interventions, and recommended priorities for future implementation. While it identifies areas of substantive progress, as per USAID Evaluation Policy, we emphasize that the evaluation was principally aimed to serve learning purposes by identifying the challenges incurred by the ALLIES activity to date and formulating appropriate recommendations for corrective actions and effective implementation during the remaining years of the activity. For each LQ section below, we have endeavored to ensure that the findings and conclusions include both ALLIES’ main achievements and the challenges identified in this mid-term evaluation. The recommendations that follow are in function of these findings and conclusions. LQ 1 To what extent has the ALLIES activity made progress toward its aim ‘To enable environments for TB elimination by leveraging community action to build a culture of accountability’? The ALLIES activity’s definition of allies is broad and includes all stakeholders. The ET assessed specific interventions with i) TBCs, ii) SLNs, iii) Elected Representatives (ERs), iv) Employers and v) Journalists. Other allies include affected communities and health workers. Work with the allies in the project districts is a continuation of the C2A project legacy. In the following sections we respond to each of the sub LQs. In the conclusions we go back to the overarching LQ. Sub LQ 1.1 To what extent the project has created the key ALLIES who can undertake strategic advocacy for enabling environments at multiple levels to shape rights￾respectful, gender and age-responsive TB services? Findings • ALLIES has built on the TBC model piloted in the C2A project and trained 654 TBSs to TBCs with Tamil Nadu contributing over 300 to this achievement. The project has further gone ahead in strengthening this cadre by equipping them with additional skills in leadership, communications, digital tools, human rights and advocacy, counselling and support for livelihoods and income-generation. • The ALLIES strategy has also brought in elected representatives (ERs), private sector employers, and journalists. It has reached out to about 500 ERs to sensitive them on TB issues, over double of the overall target. ALLIES has increasingly focused the work with ERs on local governments, as they provide opportunities for tangible changes, for example, ensuring that PwTB in their constituencies receive GOI social services and subsidies. • In the activities with the private sector, under the rubric of the Employer-Led Model (ELM) which was piloted by REACH during the C2A project, ALLIES engaged them in TB awareness, health education, and service delivery. ALLIES reached out to almost 200 companies, requesting them to sign Letters of Intent to support National Tuberculosis Elimination Program (NTEP) activities. • ALLIES has leveraged REACH’s existing Media Fellowships and Awards program, which was established in 2008. ALLIES has supported 44 journalists across the country and built their capacity to report ethically on TB. This effort led to 93 stories in local languages on various issues related to TB. USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 17 Other Allies TB Champions. TB Champions (TBCs) are volunteers, trained and paid by ALLIES who support the health system and connect PwTB with services. Their responsibilities include supporting PwTB, taking feedback from the PwTBs, following-up to address gaps identified and conducting community meetings for awareness and engagement. TBCs are the heart of ALLIES, have become an effective bridge between the PwTBs and the health system and were universally acknowledged by all NTEP respondents. Detailed findings on TBCs are presented under LQ 3 as the role of TBCs and use of CAF are intricately linked. Overall, the evaluation finds that ALLIES has continued to effectively support the TBC model to integrate the Champions into the GOI’s TB policies and services. The integration of TBCs into GOI systems creates opportunities and challenges: opportunities because the TBCs have direct communication opportunities for engagement with the NTEP officials in the TUs. The role of the TBCs is discussed in detail in Section 3; other detailed findings on the TBC model are also presented in Annex 4. Survivor Led Networks (SLNs). The SLNs are collectives of TBSs that work as community structures recognized by NTEP to facilitate welfare of PwTB, TBSs and TBSs. C2A created state level SLNs in the three project states. ALLIES created the state SLN in Tamil Nadu and is working with all four to strengthen them into sustainable effective institutions. ALLIES is also establishing SLNs in the project districts, but the activities were reported to be sporadic and included celebrating special days. ALLIES takes the lead, and not the network, in organizing these events. The network members participate in the District TB Forum meetings, but these have not been regular. The network members are highly motivated individuals and reported to the ET that they are being supported by ALLIES for capacity building, livelihood provision, and state TB-forum participation. Refer to findings in EQ 4 for more detailed findings on SLNs. Elected Representatives (ERs). In that ERs are not covered in other LQs, here we provide detailed findings on ALLIES’ approach to working with ERs. Initially, the project targeted national, state and district-level ERs: Members of Legislative Assembly (MLAs), Members of Legislative Councils (MLCs), Ward Members and Panchayati Raj Institutions (PRI) Member Post-COVIDVID pandemic (Year 2, Quarter 4), ALLIES adopted a more localized approach, focusing more on the PRI members as it was harder to engage the higher-level ERs2. Since the PRI represent smaller geographies, they were more accessible to engage PwTB-relatedtivities. Table 1: ER- LOP Target and Achievement Performance Indicator LOP Target as per LOP MEL Plan Achievement* (2020) AWP 1 AWP 2 AWP 3 AWP 4 Number of ER newly engaged by the ALLIES project 12 100 12 15 207 477 * LOP targets have been changing annually, so ET cannot comment on the numbers (Source: ALLIES MIS and Plan Documents (MEL, AWPs) 2 QPR-Year 2, Q4 18 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov Table 2: ERs Sensitized and Engaged State Sensitized (No.) Engaged (%) Key Activities and their Count (#) Chhattisgarh 98 37 (38%) ▪ PWTB nutrition support (#13) ▪ Wall Paintings/Writings (#22) ▪ Banner/ Jingle (#2) Jharkhand 110 5 (4%) ▪ Community Meeting on TB awareness (#1) ▪ Involved TB in their agenda of the Panchayat level meeting (#1) ▪ Supported PWTB in getting PM Aawas Yojna (#1) ▪ Supported PWTB in getting the medicine with the coordination of TU (#1) ▪ TB awareness meeting with PRIs members (#1) Odisha 437 45 (10%) ▪ Involvement of TBCs for meetings (#6) ▪ Nutrition Support (#15) ▪ World TB Celebration Involvement (#15) ▪ Wall Paintings (#2) ▪ Miscellaneous activities (#7) Tamil Nadu 704 410 (58%) ▪ Gram Sabha Meeting on TB awareness (#89) ▪ PRI Meeting (#303) ▪ Nutrition Support (#18) Total 1,349 497 (37%) (Source: ALLIES MIS data) ALLIES sensitized ERs through individual and group consultations. In Jharkhand, ALLIES sought a letter from the Health Minister to appeal to the ERs to engage with the project. Review of data from the progress reports and interactions with the PRI found that ALLIES’ engagement strategies have focused on PRI meetings (~64 percent), Gram Sabha meetings on TB awareness (~19 percent), nutrition support to PwTB (~9 percent) and wall paintings for awareness raising (~5 percent). Some ERs have taken further initiatives on their own, e.g., the MLA of Bijatala, Mayurbhanj in Odisha committed to make his constituency TB free by 2023. ALLIES supported NTEP in development of a vision document and a micro plan in collaboration with other government departments. A Block Task Force has been formed and one meeting has been held but progress on any other action points was not shared by the district respondents. TB does not seem to emerge as a top priority for the ERs from the interviews. Few ERs are engaged in specific activities to assist PwTB in their constituencies. An ER from Tamil Nadu shared ‘I have more than 350 villages here, but overall, only 3-4 TB patients….”. Most of the ERs are not aware of the issues associated with TB or the concept of rights-respectful, gender and age-responsive health services. There are instances of ERs committed to the cause of TB elimination and take pride in their area having no PwTB. An ER in Chhattisgarh stated, “We discuss TB in panchayat meetings and have USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 19 pledged to eliminate TB.” A limited number of ERs have supported PwTBs with food baskets as part of the Nikshay Mitra Yojana. Employers. The Employer-Led Model (ELM) was piloted by REACH during the C2A project, with a goal to engaging them in awareness, health education, and service delivery. USAID/India established the Corporate TB Pledge (CTP) Secretariat at The UNION in April 2019, under iDEFEAT TB Project for corporates to use their resources to combat TB. The Operational Guidelines for ELM were launched by the GOI in Sep 2019. ALLIES continued the ELM work that involves sensitizing and engaging corporates. The ALLIES team informed that consultative meetings are organized in collaboration with NTEP at the state and district level (including non-project districts) to sensitize corporates and usually attended by the Corporate Social Responsibility (CSR) leads of the companies. The team then follows up with the Employers to sign the Letter of Intent (LOI) of working on TB with the District government and sign up to CTP online. For example, in Year 3, Quarter 2, ALLIES held a sensitization session in Jharsuguda district, Odisha in which 10 companies participated. The ET found it difficult to interpret the performance metrics for the ELM intervention in view of the changing indicators and targets. ALLIES AWP1 submitted in Nov 2019 had three indicators, MEL Plan submitted in Feb 2020 had two different indicators, one of which was subsequently dropped. The Performance Indicator Reference Sheet defines only one indicator ‘Number of new companies that have joined the ELM’ as a corporate/industry/PSU who undertake activities related to TB care and prevention and report to the state/district TB cell in specific format mentioned following the signing of the Letter of Intent (LOI)/Memorandum of Understanding (MOU). The reasons for the change are not described. As per the definition, ALLIES has significantly underachieved its ELM target based on the data provided by the REACH team. Finally, in reviewing the ELM approach and activities, the evaluation team notes that ALLIES has not leveraged the CSR pool of funds that many of the corporates bring to the mix. 20 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov Table 3: ELM Target vs Achievement Indicator Target LOP as per MEL Achievement (2020) AWP 1 AWP 2 AWP 3 AWP 4 Number of new companies that have joined the employer-led model 80 -- 40 40 40 5* Number of workplaces implementing ELM activities -- 30% increase over baseline -- -- -- 16 No. of industries reporting at least once a quarter to State / District TB cells -- 192 -- -- -- 5* Number of industries sensitized 192 -- -- -- 198 (Source: ALLES Plan Documents and MIS.) (*1 in Tamil Nadu, 1 in Chhattisgarh, and 3 in Odisha) Table 4: ELM Reported Activities State Employers Sensitized LOI Signed Both LOI and CTP Signed Workplace Policy Undertake Activities Report to State TB Cell/ District TB Cell # % # % # % Chhattisgarh 36 14 39% 13 36% 4 11% 5 1 Jharkhand 56 36 63% 9 16% 0 0 2 0 Odisha 22 4 18% 2 9% 3 14% 3 3 Tamil Nadu 84 5 6% 2 2% 0 0 6 1 Total 198 58 29% 26 13% 7 4% 16 (8%) 5 (2.5%) (Source: ALLIES MIS data) A total of 198 companies were sensitized of which 29 percent have signed LOI with NTEP (lowest in Odisha and Tamil Nadu). Of all the sensitized employers less than 8 percent reported to have done some activities for TB and further less than 3 percent are reporting to NTEP. The ET found that employers are not aware of the ELM guidelines or of the specific activities to be conducted. One employer in Jharkhand said “We once did a TB screening camp for employees as suggested by the project staff. Overall, we are not aware of the project objectives and its activities, we have our own health program for communities, and the project needs to approach us to collaborate.” Corporates that run in-house hospitals reported organizing screening camps for employees and nearby communities. Once a case is identified they report to NTEP and refer to the public sector facility for USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 21 treatment. ALLIES has provided some wall charts with clinical protocols to be displayed at some of these hospitals. It seems from the interviews that these are activities that the corporates have been doing irrespective of their engagement with ALLIES. Journalists. ALLIES has engaged journalists through Media Fellowships and Awards since 2008. USAID/India has been supporting the fellowship program for TB since 2016, first under the C2A and now under ALLIES. The fellowship is a well-instituted program by REACH and has been commended for using the platform for creating awareness and advocacy. Table 5: Media Fellows- Target vs Achievements Indicator LOP Target as per Achievement MEL (2020) AWP 1 AWP 2 AWP 3 AWP 4 Number of Fellowships awarded -- 48 -- -- -- 44 Number of journalists given awards -- 16 -- -- -- -- Number of TB-specific stories by fellows 144 -- 144 180 180 93 Number of state-level media round tables organized 8 8 8 8 12 9 (Source: ALLES Plan Documents (MEL and AWPs) and MIS Data) ALLIES is on track to achieve its stated targets on the number of Fellowships awarded and media round tables held. However, the target for the number of TB-specific articles or news items is yet to be achieved. The media fellows shared deep appreciation for the fellowship and said that it provides useful insights about issues related to TB. This intervention goes beyond the project states. The project does not have any data on the readership of the stories. Table 6: Some Examples of spin-off effects of stories by ALLIES Media Fellows (from ALLIES reports) Stories State Responses Socio-economic status of TB patients Tamil Nadu Three private hospitals expressed their willingness to set up Find Access Support Treat (FAST) Centers for TB Non-availability of food basket to TB Patients Chhattisgarh Health department took cognizance and acted High-risk zone for TB in Chhattisgarh Chhattisgarh State Health Minister committed to improve TB services Low rate of disbursement of Nikshay Poshan Yojana (NPY) Bihar Improvement in the disbursement of NPY funds (Source: ALLIES Reports) While it is not possible to comment on the impact of stories by media fellows, it seems they do highlight local issues that get noticed and trigger action. Conclusions • ALLIES made considerable efforts to sensitize a large number of ERs but has not been able to strategically garner their support to improve QoC or QoS for PwTBs. The ER strategy success varies by state depending on the core functional strength of the ER system, that 22 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov works well in Tamil Nadu but not so much in Jharkhand. ER have the authority to leverage many national or state social security schemes (e.g., livelihood, pension, scholarships, food and housing, untied funds, welfare schemes for tribal and backward classes) to improve quality of life for TB affected communities that has not been tapped into. • ALLIES sensitized about 200 corporates, however less than 30% committed to support TB elimination and further only 8% conducted any activities. The ALLIES’ ELM strategy has not had success in using the CSR pool of funds that many of the corporates bring to the mix. • There is no strategic targeting of corporates by the type of establishment/ business such as foundries, cement manufacturing, spinning that particularly compromise lung health and predispose to TB. • The ALLIES activity has not targeted the informal sector that employs a larger proportion of unskilled/ migrant workers. This is a huge gap as the unorganized sector does not provide any medical coverage/social security to employees, state schemes may not cover migrants and they live and work in highly vulnerable conditions. • The media fellowship is a useful program to build capacity on reporting around important health, socioeconomic and human rights aspects of TB. There seems to be some loss in follow up with not all media fellows delivering at least three stories that are required to successfully complete the fellowship. There is anecdotal evidence that these stories have led to changes in TB programs. Sub LQ 1.2 To what extent the project has established community-owned mechanisms to monitor quality of TB care and services, and give feedback to the program for timely responses, helping institute accountability and strengthening community empowerment? Findings regarding community-owned mechanism to monitor quality of TB care and services are discussed in detail under LQ 3. CAF is the centerpiece of ALLIES and merits a detailed discussion. Sub LQ 1.3 To what extent has the project successfully promoted discourse on enabling the policy, regulatory and financial environments to support TB elimination and reduce TB-related stigma/discrimination at state and national levels. Findings ALLIES led the development of the ‘Strategy to End Stigma and Discrimination Associated with TB’ for CTD in collaboration with other partners working on TB. ALLIES supported the Departments of Health and Industries in Jharkhand to develop a ‘Workplace Policy for TB.’ The policy was approved by the state cabinet and can now be implemented for better protection of PwTB. Under the policy all industries are required to arrange for the treatment of employees under an employer-led model for TB and its related comorbidities, including occupational lung diseases. The policy also requires employers to ensure a safe and healthy working environment for employees so that they do not suffer from such diseases and arrange for periodic testing of all employees for TB and other related diseases. Advocacy by the ALLIES team led to the state government of Chhattisgarh establishing a stipend in the amount of Rs.200 to all PwTB in addition to the Rs.500 being provided under the NPY. This is a fully funded program supported by the Chief Minister. Considering that a large proportion of the PwTB are malnourished as a cause and effect of the disease and treatment, an amount of Rs. 700 per month helps families provide nutritious food to PwTB. Even this relatively small amount has been welcomed by all working in TB space. USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 23 NTEP has created the national, state and district TB Forums to advise on strategies for engaging communities, increasing community participation, review progress of NGO related activities and facilitate community financing to sustain PwTB support services. While the TB Forum meetings have been irregular3, KII respondents in all states reported that the SLN members attend the TB Forum meetings when they are held. Interviews with TBCs revealed that they feel empowered to speak up and raise issues in these meetings. The GOI issued guidelines for the TBC strategy to be scaled up in all states. Key Informants in the state shared that the ALLIES team supported the Chhattisgarh State TB Cell in preparation of the TBC (TB Mitaans as called in Chhattisgarh) Guidelines with the roles, selection criteria, training, and reporting for the TBCs. Interactions with the REACH team and the TBCs revealed some instances of TBCs and members of SLNs advocating for timely diagnostics services in their districts and blocks. However, the ET observed that there is an absence of a documented strategy to identify an issue, create an evidence￾based case and advocate for policy change (e.g., for priority issues like livelihoods, specific support provision for vulnerable communities/ individuals, etc.). Conclusions • The IP has successfully worked with the National, State and District NTEP officials and made significant progress in advocating for PwTB friendly policies and has started the process for implementation of some of these policies and guidelines such as inclusion of SLN members and TBCs in the TB Forum meetings. LQ2 How have the intervention activities and approaches integrated gender across program implementation? What evidence exists to substantiate the reduction of gender gaps? Findings ALLIES took the lead in supporting the Central TB Division in rolling out ‘The National Framework for a Gender Responsive Approach to TB.’ The activity supported the design of training materials for master trainers and conducted five zonal training workshops for Master Trainers. According to the project MIS, the training was attended by representatives from 33 states and Union Territories across the country. A national level key informant told the ET that now the states with master trainers are developing plans to cascade down the training. However, no such plans were shared with the ET by the ALLIES team. The ET reviewed the material developed by ALLIES for the different trainings. Gender issues have been integrated into key training curricula as detailed below. # Training Curriculum Description 1 Achieving Excellence in TB Care and Services (AETBCS) for providers Session on Gender-related barriers to realizing human rights 2 CAF for TBCs Gender based issues discussed while providing training. 3 ALLIES TB Forum Assessment June 2022 24 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov # Training Curriculum Description 3 Right Based Approach on TB & Health for TBSs and TBCs Discussion on women’s issues and vulnerability to TB, Women’s vulnerability in healthcare settings 4 TB Survivor to TB Champion Mentions of gender specific barriers to care cascade, Gender sensitivity and family support (Source: ALLIES Training Curriculum) ALLIES monitoring data shows that 53 percent of TBCs in REACH districts across the four states are women; Odisha is an outlier, with only 23 percent women TBCs. According to the key informants in the state, Odisha government requires the applicants for TBC positions to have a two￾wheeler driving license and this has been a constraint for women to apply. The ET assessed the progress of ALLIES against the plan submitted at the beginning of the project as a contractual requirement by USAID. The findings below are based on ALLIES’ QPRs, and MIS data and triangulated with data from KIIs with the ALLIES team members. • The GIP committed that at least 30 percent of community members will be women. While no community trainings were done, 51percent of the TBS to TBC trainees were women. • Gender issues have been integrated into key training curriculum as detailed above. • As committed in the GIP project staff has been trained in gender sensitivity. • Providers have been trained using the AETBCS curriculum that integrates gender. • The GIP envisaged including gender as a key aspect in QoC and QoS, which has not been done. • The CAF tool, gap identification, feedback to system, and engagement with other ALLIES does not factor in any specific gender related issues. • There were plans to engage community influencers to address gender norms with respect to TB, but no progress on this was shared by any respondent. • The GIPs mentions pilots to address gender specific challenges, but no such pilot has been described in the AWPs or in the KIIs. • The GIP talks about advocating for gender responsive services but there was no data available to this effect. • Training on gender responsive services is mentioned in the GIP, however, was only done as a component of AETBCS and not addressed separately. • ALLIES fore fronted the gender issues in stigma at the national policy level through the ‘Strategy to End Stigma and Discrimination Associated with TB’. No evidence/data/ report on stigma to include gender and social differences was shared with the ET. Conclusions • While the ALLIES Theory of Change (TOC) mentions gender sensitive and gender responsive TB services, no specific activity or output in the project is aimed at achieving it. The original ALLIES program design is limited to disaggregating gender data post-facto. • Some of the critical aspects of the accountability dimension of ALLIES do not incorporate gender. For example, the CAF tool, feedback to system, and engagement with other ALLIES do not factor in any specific gender related issue. • The ET does not have access to any ALLIES monitoring or survey data to comment on reduction in gender gaps. USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 25 • The master trainers’ training conducted at the national level sets the ball rolling on implementation of the ‘The National Framework for a Gender Responsive Approach to TB’, It may need to be followed up by ALLIES to maintain the momentum at least in the project states. LQ 3 To what extent the Community Accountability Framework (CAF) is implemented and what is the effectiveness? How has the Empowerment Evaluation (EE) approach contributed to the implementation of the CAF process? Review of project documents shows that ALLIES works with a diverse group of ALLIES including the SLNs, ERs, Employers, journalists, TB affected communities, health workers etc. KIIs with REACH, USAID and the government highlight that the TBCs are the heart of the project. TBCs are TB Survivors who have volunteered to serve TB affected communities and are trained and paid by ALLIES. The TBCs support the health system and connect PwTB with services. The findings on TBCs and CAF are discussed together as they are intricately linked as explained later in this section. But the lens applied to assessing TBCs and CAF is different. The ET has developed Figure 1 to show the evolution of the TBC model based on review of government and REACH documents and interactions with stakeholders. The TBC model was piloted by REACH in the C2A project and has been adopted for scale up the GOI with a directive to the states to appoint two TBCs per TU. The directive was then modified to have two TBCs at each Health and Wellness Centre as the hub of Comprehensive Primary Health Care (CPHC). The original SOW for the evaluation of ALLIES did not include assessment of the TBC model in its own right, but how it contributes to community empowerment for health services accountability. During initial discussions with USAID/India LQs were expanded to include assessing the scale up of the TBC model by the government and the potential for support to the same by USAID/India. Please see the methods section for details. All findings regarding the TBCs are to be read from the lens of addressing questions related to the scale up of the TBC model by the government. ALLIES piloted a Community Accountability Framework (CAF) layered on top of the TBC model with the goal of improving quality of care and services and making the health services accountable to TB affected communities. Review of project documents including the Performance Indicators Reference Sheet highlights that the early vision was to engage a range of allies in CAF, however the current pilot on ground relies mostly on the TBCs using CAF, a tool that generates empirical evidence for deficiencies in services against a set of defined criteria. The TBC and the ALLIES team then work with NTEP to resolve these deficiencies. The ET has assessed the CAF pilot in terms of Figure 1: Evolution of TBC Model 26 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov its effectiveness in improving QoC and QoS in the project and goes on to describe findings that could have implications for the future for CAF including the possibility (or not) for scale up. TBC Model Findings This section discusses findings from the TBC model in the ALLIES and government scale up districts in three ALLIES states. Tamil Nadu has not yet taken a decision about scaling up the TBC model. In addition to the findings presented here, Annexes 4 and 5 contain more details on the roles of TBCs in the ALLIES districts and in the GOI districts visited. The ET observed that not all three states have made the same progress. Odisha scaled up TBCs even before ALLIES could start on ground, including in the ALLIES own project districts; So, ALLIES is using TBCs appointed by the state NTEP for its activities. Chhattisgarh has also progressed well and has already done multiple rounds of trainings for the scale up of the TBC model. Jharkhand has been slow to scale up and while the TBCs were appointed in some districts, the first training for the TBCs happened at the end of January 2023. The ET requested data on the numbers of TBCs appointed and trained from the CTD, state government as well as the ALLIES team, but was not successful. Discussions with NTEP in the scale up districts revealed that ALLIES helped Chhattisgarh develop guidelines for selection, training, and deployment of TBCs in scale up districts. This was appreciated by the state level key informants. In Jharkhand and Odisha, district level Key Informants shared that there is no clarity about selection, role, and payments to the TBCs from the government. This section discusses the findings on TBCs under different themes important for effective scale up. They are referred to as ALLIES TBCs and Scale up TBCs to make a distinction (if any) in the findings between those working with ALLIES and those deployed by the government. A. Activities to be Done by TB Champions The ET reviewed various government and REACH documents to understand how the role of the TBCs has evolved456. For details of roles defined by GOI and ALLIES, please see Annex 10. While the discussion here captures the range of activities, the actual performance depends on what incentives are provided by the state. I. Provide emotional and social support to PwTB: The ET’s interactions with the TBCs, both ALLIES and Scale up, show that they are highly committed to serving the PwTBs and “don’t want others to suffer the way they suffered and will help them in any way possible”. The ALLIES TBCs select 8 PwTBs from the new notifications each month and meet with them to administer CAF. As part of the process, they also counsel them and their families on treatment adherence, nutritional support, cough hygiene, contact screening, Tuberculosis Preventive Treatment (TPT) etc. In addition, they visit any PwTBs assigned by the Senior Treatment Supervisor (STS) and the ones they are aware of in the vicinity of new CAF PwTB. They also receive calls from PwTBs wherein they resolve problems or else inform the STS for further action. ET observations of the process show that they can empathize 4 Guidance Document on Community Engagement under National Tuberculosis Elimination Program, CTD, MOHFW, GOI 5 ALLIES Performance Indicator Reference Sheet 6 ALLIES Operational Guidelines for the rollout of Community Engagement Plan A Typical TBC  A TB Survivor who volunteers  Trained in TBS to TBC curriculum  Performance Based incentive  Resident of the block /HWC area  Link between health system and TB affected communities USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 27 with the target population by virtue of their own experience and the skills built by the ALLIES project. The scale up TBCs visit the PwTBs assigned by the STS and counsel them. The ALLIES TBCs’ follow-up of PwTBs is informal and is done if the PwTB calls the TBC; if the TBC is on the route for visiting a new PwTB or specifically assigned for follow-up by the STS for some reason. There are few instances of self-motivated TBCs who take initiative for follow-up PwTBs beyond the targeted eight. The ET observed that the management of the CAF tool, in that is structured, targeted and time demanding, has become the key priority for the ALLIES TBCs. Both ALLIES and Scale up TBCs, make efforts to bring back the “lost to follow-up” PwTBs back into the fold of the program. Scale up TBCs in Chhattisgarh have and are assigned an incentive specifically for doing so. The NTEP respondents universally appreciate the role played by TBCs in supporting Nikshay Poshan Yojana (NPY). NPY is a Conditional Cash Transfer Scheme which transfers an amount of Rs. 500 per month to the PwTB’s account for supplementing their nutrition as long as they continue treatment. Both types of TBCs help collect the required documents from the PwTB for the NPY bank account linkage (AADHAR and copy of the bank account passbook) and submit them to the STS. In case the PwTB do not have a bank account, the ALLIES TBCs also support them in opening an account by helping with the paperwork. Another important role taken up by the ALLIES TBCs has been in promoting the Nikshay Mitra Yojana where people adopt PwTBs and commit to provide a monthly food basket to the PwTB for six months. Some IDIs revealed that in case of very poor PwTBs the TBCs themselves have contributed to supporting the family. 2. Raise awareness of TB among their communities and reduce stigma in the community Respondents in the TUs and community shared that both ALLIES and Scale up TBCs conduct 5 community meetings per month. The ALLIES TBCs meet with PRIs, visit schools and colleges, and organize celebration of special days to create awareness and also reduce stigma against the disease and PwTBs. 3. Carry out Advocacy The ALLIES TBCs advocate with ERs, mainly at the PRI level, to commit to making their area TB-free and supporting PwTB for food basket and stigma reduction. The ALLIES TBCs also shared that they use the feedback from CAF to advocate for better health care for the PwTBs. TBCs also participate in TB Forum meetings when they are held. Scale up TBCs did not mention meeting with ERs or any other advocacy activities. 4. Provide real-time feedback to the health system This activity is part of the TBC role in the GOI’s ‘TBS to TBC Curriculum’. ALLIES piloted CAF as a systematic way of providing feedback to the health system and addressing gaps found and is discussed in the subsequent sections on this LQ. Conversations with TBCs and ALLIES team members make it clear that CAF is the most important activity for the TBCs in ALLIES districts and consumes about 70 percent of their time. While they continue to do other work, the CAF process 28 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov is what they are accountable for in terms of numbers and timelines. As per the TBC Review Meeting Checklist ALLIES team reviews focuses primarily on CAF7. Interactions with the Scale up TBCs revealed that they do not have a systematic tool or mechanism to take feedback on PwTB perception of quality of services. They do, however, try to resolve what they can by talking to the PWTBs and/or raising it to the level of the STS. These issues include non￾compliance, access to medicines, side effects, non-payment of NPY etc. B. Overall support to NTEP as part of the TU team The TBCs in both ALLIES and Scale up areas support NTEP in Active Case Finding and referral of presumptive cases. It is noteworthy that in interactions with the ALLIES team they feel that they do not have a direct role in increasing notification and their intervention starts after the case has been registered in the TU. TBCs in both areas promote sputum testing at 2 months of treatment and TPT for family members. They may on occasion carry the sputum sample from the PwTB to the lab. This attracts a payment of Rs. 25 as an incentive to whoever does this including the ASHAs. Some ALLIES TBCs are also employed as sputum transporters. This is a budgeted part time role under NTEP to transport sputum samples from peripheral facilities that do not have lab facilities to the central labs. This helps PwTB by not having to travel long distances. Another role that both types of TBCs play is counseling PwTB and their families at the TUs. The STS in both the project and non-project areas shared that this is a very big contribution as the TBCs can spend more time counseling than they themselves can due to clinical, supply chain and reporting tasks. The STS also shared that TBCs are good counsellors as they can empathize; and share their own struggles and how they have overcome them. C. Selection and onboarding of TBCs The selection process followed for the ALLIES TBCs and Scale up TBCs is very different. The team learnt about the selection process during discussions with the ALLIES team. The ALLIES District Strategists (DS) gets the list of PwTBs who have completed treatment from the STS, discusses with the STS and then makes calls to them to assess their aptitude, willingness, and availability. Shortlisted TBSs are invited for training in TBS to TBC curriculum. ALLIES team members observe the trainees and further shortlist for a two month community internship. On successful completion of the internship TBCs are appointed by the project at the TUs. The ET observed that deployment of TBC per TU does not vary by its population catchment size, its geographical spread, and/or TB notification load. The TBCs then participate in CAF and other trainings offered by ALLIES such as leadership, communication, rights-based approach, gender, use of mobile application, and livelihoods (Refer Annex 4, Table 18). in a staggered manner. The TBCs shared that they found the communication training to be very effective and it has helped them become confident in working with communities and raising issues with senior officials in TB Forums. The NTEP District Program Coordinator (DPC) takes the lead in selection of Scale up TBCs. In Odisha the ET learnt that the positions were advertised, in Chhattisgarh TBSs were approached through the STS and in others the information spread by word of mouth. Interviews with district Jharkhand NTEP staff revealed that they are not clear on how to select the TBCs. Later the ET learnt from a key informant the CTD criteria for selection of Scale up TBCs communicated to the states are 1. Individuals above 14 years of age, who have successfully completed the TB treatment and 2. Willingness to function as TB Champions by sharing their personal experiences with TB 7 LON ALLIES Implementation Technical and Operational Guidelines USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 29 patients, community in their area, etc. and 3. Willing to talk and support TB patients during their treatment period. The ET learned from the IDIs in non-project districts that the Scale up TBCs are trained on the job after they join and attend the more formal TBS to TBC curriculum trainings as and when they are organized. The ET learned in Chhattisgarh that there is significant dropout from TBCs selected to attending the training and then working after the training. The data for the same was not available. Partners like The UNION, JHPIEGO, and ALLIES have been supporting the training of Scale up TBCs. In KIIs with partners the ET learnt that ALLIES has worked with them to build their capacity as Master Trainers. Discussions with partners show that the Scale up TBCs trainings do not include communication, leadership, rights-based approach, and other such training. D. Job aids and communication material for TBCs The ALLIES TBCs were observed to have a flip book for awareness generation meetings and some small sized pamphlets for counselling and distribution to TB affected communities. The TBCs explained that this is the only material that they have received. This “Information Education Communication” (IEC) material is in local languages. The ET did not observe other materials on issues like diet (when-what), follow-up protocol, ADR, DR-TB, management of comorbidities, gender, stigma in the workplace, etc. The ALLIES team shared a Communication Skills and TB Information Booklet for TBCs with the ET during the later part of the evaluation, but these were not observed to be available with the TBCs in any of the visits in ALLIES districts. The Scale up TBCs said that they do not have any communication material or job-aids. E. Supervision of TBCs The ALLIES DS guides and mentors the TBCs on a day-to-day basis. In Coimbatore (Tamil Nadu), the DS deputes one TBC as a mentor to a small group of four to five TBCs on a monthly rotation basis. The TBCs reported this to be helpful as it provides a platform to discuss common concerns and cross-learning. The STS also directs and guides the ALLIES TBCs for PwTB follow ups, NPY documentation and other tasks. The Scale up TBCs work in subordination to the STS of the TU where they are stationed. TBCs in Dhamtari were appreciative of the guidance provided by the STS and being accessible any time to answer questions. Key informants in Jharkhand, however, reported that the STS is unclear on the deliverables by the TBCs and hence unable to supervise effectively. F. Remuneration of the TBCs The ALLIES TBCs are paid Rs. 6,000 per month for part-time engagement and Rs. 1,000 for travel. Most of them have other sources of income. Some of the women feel that amount is commensurate with the time they spend and treat this an addition to the family income. Performance-based incentives have been budgeted and approved by the state governments. The ET learned from the key informant interviews that there are variations in the amounts that different states have planned, hence the variation in tasks that the TBCs perform. The Table below provides details of performance-based incentives in different states for Scale up TBCs. 30 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov Table 2: TBC Incentives Jharkhand Chhattisgarh Odisha Rs. 1000 per month (fixed) Rs. 7500 per month, which includes: - ▪ Rs. 400 per community meeting (max 8/month) ▪ Rs. 250 per PwTB follow-up (max 15) ▪ Rs. 250 Travel allowance per month ▪ Rs. 150 per Defaulters resuming treatment (max 2) Rs. 3500 incentives will be paid on deliverable Follow-up of 20 PwTB every month. Each PwTB to be followed up 4 times during treatment. Source: Interviews with TBCs, district NTEP officials In Dhamtari, Chhattisgarh, the ET saw the form that is filled out by the Scale up TBCs for release of incentives with details of each PwTB visited. The STS approves and forwards this form to the accounts department for payment. Delays in the release of payments were reported in all states, with some not having received a single payment since they started. More details on the scale up of the TBC model in the three states studied by the ET are provided in Annex 5. Community Accountability Framework This section describes the evaluation findings for the CAF pilot in terms of its effectiveness in improving QoC and QoS in the project and goes on to describe findings that could have implications for the future for CAF including the possibility (or not) for scale up. With the TBC model has been accepted for scale up across the country, REACH moved on to pilot CAF in the ALLIES activity. CAF is an adaptation from the Community-led-monitoring (CLM) mechanism used by United Nations Program on HIV/AIDS’s (UNAID) Joint HIV Program. The CAF toolbox was adapted in consultation with multiple stakeholders to enhance the accountability, coverage and effectiveness of TB programs and generate demand by strengthening community confidence in TB services8. After the initial testing and refinements, CAF is now a series of steps that starts with a selection of 8 PwTBs by each TBC each month and administration of a 50-item questionnaire. This tool measures compliance with NTEP TB management protocols, service deficiencies and social issues that the PwTB face. The data collected through this tool is then shared with NTEP by the TBC, who then works with them collaboratively to address the gaps. Interviews with the ALLIES team members across all states reveal that CAF is a priority activity for the TBCs and the ALLIES team with strict monthly schedule for collecting feedback, data entry and analysis and action. Also, it is the most systematically planned activity, which connects TBCs directly to the PwTB, their family and health staff in a structured manner. A. CAF Process 8 CAF IN ACTION- Operationalizing a Community Accountability Framework in India, REACH USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 31 The CAF training curriculum was developed in 20209 and the first phase of CAF was launched in Tamil Nadu in December 202010. ALLIES intended to train a minimum of 120-150 community members, including the allies as per CAF curriculum11. However, the TBCs are the only allies who are trained in CAF and no training was reported for other allies by any of the ALLIES team members. Table 8: TBC Trained and Using CAF CAF Tool: The tool consists of about 50 questions under different categories (e.g., quality of service, quality of care, stigma, access to services, etc.) to identify issues in services and at the community level. The tool factors in types and stages of TB to some extent, but does not capture the diversity of PwTBs in terms of co-morbidities and vulnerabilities to be truly representative of the target population (e.g., co￾morbidities, vulnerable communities- children, women, single women, elderly, disability, remote communities, etc.). The tool is paper based and also app based (KOBO, ALLIES). For more details on the CAF tool see Annex 6. Sample Section and Tool Administration: A sample of 8 PwTB is selected by each TBC each month to administer the tool. The sample size, however, remains constant and does not vary with location type (urban, rural, tribal), population spread, load of notification, number PwTB under treatment, health facility level (DH, Block, etc.). Key Informants provided different criteria for selection of the CAF sample: States TBC Trained in CAF TBC Using CAF Jharkhand 44 184 Chhattisgarh 51 (70%) Odisha 60 Tamil Nadu 109 Total 264 Source: ALLIES MIS Data • PwTBs only in the Continuation Phase (CP) of treatment are selected. • Half the sample must be women • TBCs are provided with the list of cases notified in the month and they select the sample. • STS selects the sample. • STS assigns cases that come from hard-to-reach locations. • Only on treatment PwTBs are selected In KIIs with TBCs, the ET found that CAF takes about 70 percent of the TBC’s time. The paper￾based tool is administered to PwTBs, the data is entered online in two apps, KOBO and ALLIES app. Key informants told the ET that once the ALLIES app is fully up and running, KOBO will not be used. CAF administration provides an opportunity to counsel the PwTB and the family on key aspects like cough hygiene, diet, drug adherence, NPY scheme, etc. Also, it helps TBC to address any specific issues like access to health facility, non-availability of proper diet, submission of NPY documents etc. Data Analysis and Identification of Gaps: The data entered by the TBCs is downloaded by the state and national ALLIES team and analyzed. The gaps identified during CAF data analysis are manually entered in the database. The MIS system does not have the ability to flag a trend to facilitate managers to act. The data is analyzed manually to show what number of the total PwTBs interviewed have a response that needs action e.g., It took more than 3 days to start treatment after diagnosis, NPY transfer not received, Cough hygiene counseling not done at the facility etc. 9 CAF IN ACTION- Operationalizing a Community Accountability Framework in India, REACH 10 Year 2, Quarterly Progress Report for Q2 11 REACH-ALLIES MEL Plan, AWP 2, AWP 2 and AWP 4 32 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov Subsequently, a Block Action Plan (BAP) is developed that requires actions to be taken for each gap identified. The ET observed that the different states have different formats for developing a BAP. One state has a printed format while others use notebooks with one page for each PwTB. Some use one notebook for each month. Feedback Mechanism: The data and the plan are discussed with the STS and other NTEP team members at the TU in a monthly meeting attended by the DS and sometimes the district NTEP officials. The NTEP team takes lead in addressing the supply side issues and the TBCs work on the PwTB and community issues such as stigma, alcoholism, and nutritional support. While the ALLIES Implementation Technical and Operational Guidelines mention District Action Plans that aggregate the Block Action Plans, these are not operational. The ET learned in KIIs with ALLIES team members the DS shares feedback with the district NTEP officials on an informal basis and no written report is submitted. Similarly, the feedback is informal and opportunistic at the state level. The ALLIES team members said that the CAF process was initially seen to be threatening by NTEP but over time the TBCs and ALLIES team has built a rapport with NTEP and they are more receptive to feedback. In interaction with the ET, district officials appreciated CAF as a tool to meet with PwTBs and understand supply side deficiencies. It was suggested in some state level KII (Jharkhand, Odisha) that the IP should regularly share their report so that the state can support the project better. B. CAF Data Findings Table 9: CAF Administrations (No. of PwTBs) As we discussed in the section on methodology and elsewhere in this report, the CAF data do not allow for a measure of effectiveness of ALLIES’ interventions in improving QoC and QoS for PwTB. They are not collected using a scientific sampling technique, nor is there a counterfactual with which to compare them, i.e., a rigorous sample of PwTB in matched non￾ALLIES districts. Nor was it possible to access GOI service data. The issue is important, because in the progress reports, ALLIES sometimes suggests that the CAF data can be used to measure changes in health system QoC and QoS for PwTBs across the districts. Here are examples from the Progress Report for Y3Q3: States CAF Administration (April 2021 to Jan 202322 months) F F% M M% O Total Tamil Nadu 3,648 36% 6,513 64% 1 10,162 Chhattisgarh 2,129 38% 3,446 62% 1 5,576 Odisha 1,764 33% 3,532 67% 2 5,298 Jharkhand 2,006 34% 3,894 66% 3 5,903 Total 9,547 35% 17,385 65% 7 26,939* *97% DSTB, 86% on treatment. Source: ALLIES MIS Data • “the proportion of person with TB who were initiated on treatment within 3 days of getting a diagnosis has gone up from 74.2% to 94.7% in the last 15 months in the catchment areas of the TB Units implementing CAF”. • “the decline in proportion of person with TB in all implementing areas of the ALLIES project over 15 months of implementation has been slightly over 4 percentage points”. USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 33 These statements are not accurate. The change in the initiation of treatment or the variation in the number of PwTBs refers only to those PwTBs that were interviewed by TBCs for the CAF tool. These cannot be projected onto “the catchment areas of the TB Units” or “all implementing areas of the ALLIES project”. The ET analyzed the CAF data in those areas of ‘Quality of Care’ and ‘Quality of Services’ that have informed the TBCs’ accountability initiatives (ALLIES indicators for QoS and QoC detailed in Table below). Some of the CAF indicators show positive changes, but for the reasons discussed above, they cannot be generalized to reach conclusions about the effectiveness of ALLIES in improving services of TB Units for all PwTBs in the target districts. Data from the first (Apr 2021) and last (Jan 2023) quarters was excluded as the number of records for first quarter was small (104) and only one month data was available for the last quarter. Data from 20 PwTBs was excluded due to incomplete records. A total of 25,377 CAF records were used for the analysis discussed below. 34 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov Quality of Care (QoC) Indicators Metric Indicator Quality of Care Timely, accurate diagnostics 1. Average turnaround time for UDST 2. Average turnaround time for LPA Timely, appropriate treatment and counseling, and treatment support 3. Average time from diagnosis to treatment initiation for DS-TB patients 4. Average time from diagnosis to treatment initiation for DR-TB patients 5. Average adherence score of TB patients 6. Proportion of TB patients paid NPY benefit 7. Proportion of districts that have conducted district TB forum meetings Not being measured The project has seven indicators for QoC of which ALLIES is measuring four. Findings: High starting point, No service data to validate, Possible contribution to improvement by CAF Figure 3: TAT: Diagnosis to Treatment of DS-TB (0-7 days) 94% 95% 96% 97% 98% 99% 100% JH OD CHG TN Jul-Sep 2021 Oct-dec 2021 Jan-Mar 2022 Apr-Jun 2022 Jul-Sep 2022 Oct-Dec 2022 Figure 2: Turnaround Time (TAT)- Diagnosis to Treatment for DR-TB (0-7 days) Findings: High starting point, No service data to validate, Possible contribution to improvement by CAF USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 35 Figure 4: Regular Drug Intake (Adherence) 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% JH OD CHG TN Jul-Sep 2021 Oct-dec 2021 Jan-Mar 2022 Apr-Jun 2022 Jul-Sep 2022 Oct-Dec 2022 Findings: High starting point, No service data to validate, Possible contribution to improvement by CAF figure 5: Received NPY Amount in last 30 days 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% JH OD CHG TN Jul-Sep 2021 Oct-dec 2021 Jan-Mar 2022 Apr-Jun 2022 Jul-Sep 2022 Oct-Dec 2022 Findings: NPY disbursement is an issue across the states. Raipur, Chhattisgarh had 24% disbursement till the fifth quarter and improved to 48% in quarter six. The ET learnt from state level key informants that it is due to problems with the payment portal and due to the cash flow for this payment head. Quality of Services (QoS) Indicators 36 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov Metric Indicator Quality of Services Convenient care setting 1. No or minimal waiting time at clinic 2. Adequate, comprehensive signage to understand service location Dignified, empathetic care 3. Respectful, compassionate provider behavior 4. No experience of stigma Affordable care 5. No out-of-pocket spend at TB clinics 6. No bribes 7. No, or minimal spending on travel ALLIES has defined seven indicators to assess quality of services. Select indicators are discussed below. Again, we caution that these indicator data cannot be extrapolated to all PwTBs receiving services from TB units, owing to the lack of a robust sampling methodology and the absence of a counterfactual. Findings on wait times: At the state level, the starting point of the indicators is high and shows improvement over time. Feedback from CAF may have contributed. This indicator shows notable improvement in Balod, Chhattisgarh (51 percent to 71 percent), Bhubaneswar, Odisha (49 percent to 82 percent) and Krishnagar, Tamil Nadu (51 percent to 97 percent). The data cannot be generalized to all PwTB receiving services from TB Units. 0% 20% 40% 60% 80% 100% JH OD CHG TN Jul-Sep 2021 Oct-dec 2021 Jan-Mar 2022 Apr-Jun 2022 Jul-Sep 2022 Oct-Dec 2022 Figure 6: Wait time 30 minutes or less USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 37 Findings on TU staff respect and confidentiality: The starting point for this was above 85% for all states, except for Tamil Nadu (72 percent). Health staff was trained in the AETBCS curriculum (focused on communication skills, patent centric delivery, etc.) starting March 2021. The ET cannot comment on the contribution of the training to improvements based on CAF data. Findings on Stigma: Three ALLIES indicators assess this: i) ‘PwTB shared TB status with friends/ community members’, ii) ‘friends/ community treat kindly’ and; iii) ‘caring and accepting family’ (Refer Figures 8 and 9). Stigma does not seem to be very high in all four states from the CAF data. The data cannot be generalized to all PwTB in the ALLIES districts Sharing of TB status by PwTB with to friends/ community (Self-perceived stigma) is comparatively low in Tamil Nadu, which had steadily improved. Cuddalore (27 percent to 73 percent), Krishnagiri (51 percent to 79 percent) and Vellore (46 percent to 69 percent) in Tamil Nadu show significant improvements. However, we do not know whether these data are representative of all PwTBs in ALLIES districts. Figure 9: Caring and Accepting Family 86% 88% 90% 92% 94% 96% 98% 100% JH OD CHG TN Jul-Sep 2021 Oct-dec 2021 Jan-Mar 2022 Apr-Jun 2022 Jul-Sep 2022 Oct-Dec 2022 0% 20% 40% 60% 80% 100% JH OD CHG TN Jul-Sep 2021 Oct-dec 2021 Jan-Mar 2022 Apr-Jun 2022 Jul-Sep 2022 Oct-Dec 2022 Figure 7: Respectful and Compassionate behavior of Service Provider 0% 20% 40% 60% 80% 100% JH OD CHG TN Jul-Sep 2021 Oct-dec 2021 Jan-Mar 2022 Apr-Jun 2022 Jul-Sep 2022 Oct-Dec 2022 Figure 8: TB Status Shared with Friends/ Community Figure 10: PwTB who spent nothing out-of-pocket during treatment 38 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov Findings on affordable care: The graph shows that more PwTBs over time said that they are not spending any money after starting treatment. These data cannot be extrapolated to all PwTB in ALLIES districts. In interactions with most PwTB, the ET heard that they went to private facilities first, some over a year before they were diagnosed with TB. One PwTB that the team interacted with said that her husband insisted on going to private facilities as he did not trust the government system. The lady was admitted twice in private hospitals for tests and treatment and the family ended up borrowing Rs. 250,000 for the same. Most PwTB once their diagnosis is confirmed move from private to public sector facilities as the drugs are free. Another interesting observation was that an elderly PwTB who fainted once was then diagnosed to have some clots in his brain. He shared that to begin with he thought it was because of TB and went for a checkup quickly and was then referred to the state capital for further investigations. 0% 20% 40% 60% 80% 100% JH OD CHG TN Jul-Sep 2021 Oct-dec 2021 Jan-Mar 2022 Apr-Jun 2022 Jul-Sep 2022 Oct-Dec 2022 Empowerment Evaluation Three rounds of capacity building sessions have been conducted virtually with Dr. David M. Fetterman. The ET heard from many respondents that the sessions being in English, staff from some states had difficulty comprehending. Colleagues who understood were translating in real time during the sessions especially for state teams. It was further reported that once the training is completed, ALLIES’ senior team plans to translate the training into a more understandable format for TBCs and build their capacity. Once the teams are skills are built, it will be applied more widely. LQ 3 Conclusions on TBC / CAF • ALLIES’s most important focus is CAF layered on the TBC model developed in the previous USAID funded activity. In the process some of the core functions of the TBCs, most importantly the follow-up of PwTB, have taken a back seat. Apart from the CAF data with contacts with 8 PwTB per TBC each month, there is no data available to comment on the follow up with other PwTBs or even subsequent follow-ups with the 8 CAF PwTBs. • The TBC model piloted by REACH has been adopted by the GOI and is being scaled up by states with trainings of TBCs being supported by partners working in the TB space. The selection process for TBCs in ALLIES is a three-step intense process with the best matched candidates finally being appointed. The process in the scaled-up version is more perfunctory and could make a difference in performance. Also, considering the states do not have the right toolkit including job-aids for the TBCs and communication material for the PwTB, there is a risk that the effectiveness of the model may be compromised as it is scaled up. An important aspect is that different states are budgeting (and hence implementing) it differently so the incremental results due to the TBCs vary. • The biggest value of the TBCs comes from them having suffered the disease themselves and hence the empathy that they have for PwTB. Their motivation and commitment to serve needs to be nurtured. Delays in payments to the TBCs due to reasons that tend to happen in a large government system could be a dampener and may demotivate them in the long USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 39 run. Due to the specific characteristics such as being a TBS, having time to volunteer and being able to travel to the field attrition for this cadre would be difficult to manage. • The TBCs are envisioned to be community representatives and champions who can advocate for rights of PwTB to access quality service, be treated with respect and not be discriminated against either within the family, community, or workplace. However, in the scaled-up version of the model the TBCs are appointed as additional team members in the TU, working in subordination to and on instructions of the STS. While the government TBS to TBC curriculum mentions advocacy as a function, it is unclear how the lowest rung member of the NTEP team can effectively advocate for better services. • The selection of PwTB for application of the CAF tool seems to vary considerably by TU/states, so the results have to be interpreted with caution. In TUs where the PwTBs selected are the ones that are hard to reach, it serves the purpose of follow up well for NTEP but does skew the CAF results. There is an inherent bias in the process as the person collecting the data is also responsible for working on the gaps. • The CAF process is time-consuming and takes about 70% of the TBCs’ time. This is more so due to multiple points of data recording and entry (KOBO and ALLIES app). Even though the ALLIES app was aimed at making the process paperless and swift, even with the fully functional version of ALLIES app there will be steps of manual analysis of the CAF data and development of the BAP. The process does not seem to be time-efficient. • While ALLIES planned to include many ALLIES in CAF such as the ER and SLNs, currently the process is fully led by TBCs. There seems to be a weakening of the ‘community’ part in the Community Accountability Framework. While the TBCs do come from the same communities affected with TB, they work as an integral part of the NTEP team taking instructions from the STS. The process has moved towards being a PwTB feedback survey. The feedback process at the district and state level is informal and opportunistic. The tool as it is being used does not empower communities. • CAF was essentially envisaged to highlight key issues in QoC and QoS for systemic review and improvement by the GOI health bureaucracy, but somewhere has morphed into individual PwTBs feedback and redressal mechanism. Also, this is the only formal mechanism of ALLIES TBCs meeting PwTBs. • The indicators being assessed for QoC are a mixed bag. The first six indicators for quality of care can be more easily analyzed from the service data in the Nikshay portal. ‘Proportion of districts that have conducted district TB forum meetings’ is a proximal process indicator and does not belong in the QoC category. • In the absence of service data from Nikshay, or a robust sample of PwTBs in ALLIES and control groups in matching districts, the ET cannot comment on CAF’s effectiveness in improving QoC and QoS. There could be some contribution from the ALLIES activities, but given that it was not collected through scientific sampling and in the absence of a control group, these data cannot be used to measure ALLIES effectiveness. A statistically robust comparative analysis of the ALLIES and non-ALLIES districts could be done with GOI service data, which was not provided. LQ 4: How effective the new district and state level TB survivors led networks are in facilitating the rights-based approach to TB, and what is their role in advocacy promotion? Key Findings: 40 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov SLN Status: ALLIES envisioned TB SLNs as community structures recognized by NTEP for facilitating welfare of PwTBs and TBSs. The targets for the District SNLs have evolved over time, and ALLIES appears to be close to meeting the latest target. Similarly, for the State level SLNs Table 10: ALLIES Performance Indicators for SLNs Performance Indicator LOP Target as per LOP Achievement** MEL Plan AWP 1 AWP 2 AWP 3 AWP 4 Number of New District TB SLNs established 24 -- 12 15 35 34 Number of new State level TB Survivor led Networks established -- -- 5 6 6 4+3* *SLNs created in Sikkim, Assam and Delhi by Touched by TB (TBT) ** Based on Y1, 2 and 3 QPRs LOP Targets have been changing annually, so ET cannot comment on the numbers achieved The SLNs created in the three C2A states received continued support by ALLIES. A new SLN was created in Tamil Nadu. The SLNs in Chhattisgarh and Jharkhand are registered legal entities allowing them fund raise. USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 41 Table 11: Survivor Networks Registration and Membership Status (project states) State State Level Network Year Formed Registration Status # of Members Presence in # of Districts Chhattisgarh TB Mukt Chhattisgarh Foundation (TBMCGF) 2019 Registered 600 28 Jharkhand TB Elimination Jharkhand (TEJ) 2018 Registered 601 19 Odisha Kalinga TB Survivors Network (KTSN) 2018 Not Registered 759 24 Tamil Nadu TB Free Tamil Nadu Survivors Led Network 2022 Not Registered 1,343 6 Total 3,303 77 (Source: ALLIES MIS Data) Governance The ET reviewed the documents of incorporation of the registered networks and observed that SLNs have a governance structure in place with a President, Secretary and Treasurer. The membership criteria for the networks and the rules of conduct for the SLN are clearly laid out. These positions in the unregistered networks are honorary positions. ALLIES has appointed a Network Coordinator as a contractual employee for the four project states who serves as the liaison between ALLIES and the SLN leadership. Interviews with the Network Coordinators and the SLN leadership revealed that the members are highly motivated and have a strong sense of commitment to help and support PwTB. Some of the leaders have touching stories of facing stigma and discrimination themselves, that has strengthened their resolve to work for communities with TB. The district SLNs function as extensions of the state networks and are not independent institutions. The ET noted in its interviews with District SLN Coordinators that they are not clear about their role. Membership and Participation Members of the networks are enrolled by word of mouth and by enlisting TBSs who are trained in the TBS to TBC curriculum. The CAF tool also has a question at the end that asks about the respondents’ willingness to join the network after completion of treatment. KII with REACH team and the SLN members revealed that SLN meetings are held online as well in person. Online meetings have very limited attendance due to access to mobile technology and competing engagements. Some district members are able to attend in person if they are paid travel expenses by the network or the ALLIES project. Network members also communicate via WhatsApp groups. 42 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov Network Strengthening Discussions with the REACH team and Network Coordinators show that for each training ALLIES organizes, some members from the state are nominated. Data on the number of network members trained by the type of training by state was not available to the ET. The training menu includes leadership, communication, counselling, livelihoods etc. Details of the training are presented in Annex 7. The training was reported to be useful during KIIs with network members (counseling in particular). The network members during KII shared their key concern fund raising for the network and livelihoods for TBSs. The priority area of concern for the networks is provision of livelihood and income for TBSs. While many TBSs and TBCs have been undergone livelihood trainings, ET observed that most of these trainings are not being utilized for income generation, for lack of appropriate value chain linkages. However, the ET observed, in Chhattisgarh, the SLN has successfully established a soap, phenyl and toiletries making enterprise with support from a corporate and is exploring branding/ marketing. ALLIES has supported the SLNs to showcase their collective strength e.g., 46 TBCs from the Delhi network participated in the ‘National Conference on Women Winning Against TB’, presided over by the Vice President of India, and Cabinet Ministers. ALLIES has also supported two cross learning visits of the SLN Leadership to other states to share experiences. The SLNs have been participating in celebration of special days such as World TB Day (WTD), International Women’s Day (IWD), Independence Day, Gandhi Jayanti etc. to create awareness around TB and reduction of stigma. SLNs support the Nikshay Mitra effort (observed in Jharkhand and Odisha) and have also collected money to provide financial support to PwTBs (observed in Jharkhand, Tamil Nadu). The network members interact with the NTEP officials and participate in State and District TB Forum meetings, although the meetings are for not regular. State NTEP officials shared that they recognize the state networks and are willing to collaborate with them. They, however, were vary of the “way issues are raised” in TB Forums. The district chapters are new and have not yet been recognized by district NTEP officials. Touched by TB: ALLIES partnered with Touched by TB in August 2020 to create a directory of TB networks and Civil Society Organizations (CSOs) and strengthen the SLNs in the North-east (Assam, Sikkim, Meghalaya, and Nagaland) and Delhi. The Touched by TB Network was awarded the ‘On-Ground Heroes’ award for 2022 by the Apollo Tyres Foundation. LQ 4 Conclusions: ALLIES Network Strengthening Approach  Adding new members to District Chapters  Building organizational and leadership skills  Increase visibility in national and international for a  Integrate into NTEP community structures. (Source: ALLIES Operation Guidelines for Community Engagement) • The project is on track to achieve its targets for the formation of networks. However, the ET observed that three of the four SLNs were already in place from the C2A project. • Networks are yet to mature as institutions that can manage and be sustainable independent of ALLIES support. District level networks are at a 'start-up' stage with sporadic activities. • Stigma is a key issue the networks intend to address; however, no specific activity has been planned for the same. • The network needs to be empowered to collaborate with NTEP for systemic changes, rather than being confrontational. • SLNs have prioritized livelihood provision and needs to broaden their agenda. USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 43 • Livelihood generation activities need to factor in entrepreneurial skill development, understanding of value chain, financial and marketing linkages. • Institutional strengthening of the SLNs needs more work in terms of role clarity, governance and management process, work planning, monitoring, reporting, etc. LQ 5: What are the overall accomplishments, challenges, and learnings out of the project implementation? The ET has approached Learning Question 5 as a summatory question to discuss the overall findings from ALLIES design, implementation, and MERL (Monitoring, Evaluation, Research and Learning). The discussion highlights implications for ALLIES, the government scale up of the TBC model and the possibilities for CAF. Allies Goal and Design ALLIES activity’s goal is futuristic and challenging and REACH’s effort to take on the challenge is commendable. Engaging and empowering communities can be elusive, and this reflects in the ALLIES design. The activity sews together diverse stakeholders through multiple activities to move towards the larger goal in the project geographies. While the project Theory of Change (ToC), shown in graphic form below, mentions some concrete activities and outputs, it is unclear on how these activities and outputs would move ALLIES towards tangible outcomes and ultimately closer to the goal. The ToC does not define very tangible outcomes and / or does not have SMART (specific, measurable, achievable, relevant, and time￾bound) indicators for the same. Nor does the ToC factor in key outcome areas such as improved QoC & QoS, advocacy, stigma reduction, change in community awareness, health seeking behavior, livelihood/ income generation, CAF performance, network performance, etc. USAID/INDIA CLAIM: ALLIES MIDTERM EVALUATION 40 Project Theory of Change with ET comments highlighted CAF is not Community Owned The structures are not community owned Assumption held to be true Assumption held to be true This is an input statement This is an output statement This is an output statement This is an output statement and same as Output 1 This is an output statement No clear output & outcome to address the objectiv e Looking at the Theory of Change in relation to the REACH MEL Plan, the evaluation identifies considerable weaknesses in proposed outcome measures. The first EOP outcome statement, O1 referring to the creation of the TBC cadre, just repeats an output. The O2 statement does mention an outcome, in terms of improved QoC and QoC, but the relevant indicator in the MEL Plan 44 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov measures the number of people benefited, which says nothing about improvement. The third statement OP3 refers to reduction of stigma, but there is no MEL Plan indicator to measure reduction of stigma other than an indicator that measures laws and other norms passed, which is at best an intermediate result. Whether these changes will reduce stigma is an open question. Hence, REACH has no means of measuring the major ALLIES’ outcomes. The updated MEL Plan offers no substantive improvement on this issue. As per Performance Indicators Reference Sheet the Indicators 1, 4, 5, 6, and 16 have to be reported disaggregated by gender in the quarterly reports. While the project has the data for most of these indicators disaggregated by the required categories, it has not been providing this disaggregation in the quarterly progress reports (QPRs). Data use and visualization has been weak. ET’s review of the QPRs reveals that there are discrepancies in the targets set for the next quarter and the targets used for reporting in the next QPR. While these seem to be errors in calculations, they are frequent. Another trend noticed is that if no target was set for an indicator for a particular quarter, and the activity was completed, a commensurate number is shown as a target and reported against. While the reason for these observations could not be deciphered, Annex 8 illustrates this point using an analysis of QPRs for Year 3 of the project. The Year 3 annual targets for the Performance Indicators have further variations in the LON ALLIES Implementation Technical and Financial Operational Guidelines. Thematic Expertise on SBC is lacking in the ALLIES team. Being primarily a community engagement project, absence of SBCC/IEC expert in the team is noticeable. This is reflected in the inadequacy of communication materials and absence of overall SBCC strategy for improving health seeking behavior and reduction of stigma. Slow progress on Special Interventions. While ALLIES has taken on Special Interventions as pilot projects in select geographies, the team was not provided with any guiding strategy document, implementation or measurement plan, or outcome indicators for the same. The ET does not have adequate information to comment on the design and the implementation of the special intervention pilots started in the last quarter of 2022, so is in very early stages. Special intervention pilots are an important piece with ramifications for equitable reach to vulnerable populations, but lack of clear interventions and outcome measures is problematic in defining success and potential scale up plans. Implications for the future for CAF including the possibility for scale up. CAF is a good effort and has potential as a concept that can be further refined to provide feedback to the health system. There are many aspects of the tool and the process that need further work. It is a time￾consuming process, the sampling process varies by geography, monthly data collection seems unnecessary, data sharing and follow up mechanisms is informal. Most importantly, while the NTEP has adopted the TBC model, the current CAF model is not compatible to be layered on the scaled-up TBC model. The value of CAF comes from it being independent of the health system and yet led by TBCs who are an integral part of the NTEP team. From the field data collection, the ET believes that CAF mainstreaming cannot happen if the TBCs are working in subordination to the STS who approves the TBC incentives. Once the TBCs are managed and paid for by NTEP, any independent data collection by TBCs to obtain feedback on quality of services will face considerable resistance from their superior officers. Even if mechanically the process is continued, it would be difficult, if not impossible, for TBCs to convey genuine feedback in a hierarchical government structure. USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 45 ALLIES faced systemic challenges in creating rights respectful, gender and age responsive TB services. Health services (including TB services) are designed for treating diseases. It is only for the last two decades or so that patients and communities are seen not only as passive recipients but active participants in healthcare. ALLIES has taken an important step towards making TB services client centred. The biggest challenge for the activity has been creating platforms where supply side actors in the health system (particularly the public system) dialogue with communities to understand their expectations and get feedback on services provided. The evaluation shows that there was resistance to the TBCs and CAF initially as they were perceived to be adversarial and drew some resistance. Over time, the ALLIES team has successfully overcome this challenge and created a space where the feedback from CAF is shared with the NTEP team systematically at the TU level and informally at the district and state level. This remains an issue to be reckoned with as we think about the possibilities for scaling up CAF in the absence of an actor like REACH, who presses for community accountability. Left to their own devices, GOI health bureaucracies will resist accountability. Involving community side actors like the ER, Employers and media requires the priority of addressing TB as a health, social and economic menace to filter out to the top of the many issues that these players are trying to grapple with. In the case of Employers, the biggest challenge has been that while there is a conceptual commitment to contribute to TB elimination, translation to tangible actions is uncommon. Similarly for the ER, this is one of the many pressing social issues that demand their attention as they work towards the welfare of their constituents. Absence of funding for livelihood/ income generation for TBSs was quoted as one of the key issues by the Survivor networks in all project states. The ET notes the absence of an exit strategy. There is a brief mention of the exit strategy in the Year 4 workplan, it makes a general reference to transferring the best practices to the NTEP and the Survivor Led Networks supporting the best practices beyond the project. There are no details on how this would actually happen considering the networks themselves are at different stages of evolution technically and legally. In particular, there is no mention of how community accountability will work when the TBCs are integrated within the GOI bureaucracy. Guidance documents for the project are limited. The table below shows that availability of guidance documents for each of the ALLIES that the project has created. While some parts of the strategy are scattered across different documents, there is no single, overarching document that will be useful to the project team and in communication and adoption of the models by the government. Table 3: Status of Availability of Guidance Documents # Particulars Strategy Training Material Reference Material 1 TBC No Yes Yes (Not available in the field) 2 ER No No One (Directed to MPs rather than lower level ERs in project areas) 3 Employers Yes No No 4 Survivor Led Networks No No No 5 Journalists Yes No Yes 46 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov Despite these challenges ALLIES has some important achievements. Some of the achievements are listed here. TBCs can be an important nationwide cadre to contribute to TB elimination. The project has created a cadre of TB champions who are empowered, committed to serving and making sure that the lives of people suffering from tuberculosis can be made better. They have been empowered with a multitude of capacity building efforts that help them to be strong voices for all PwTBs in their remit of influence. ALLIES has engaged allies to bring TB to the CenterStage of health, economic and social problems: The ALLIES project has engaged elected representatives, employers, media and survivor networks to create awareness on the need to address the disease and the role each of these stakeholders can play. The Project team along with the TBCs celebrated all important occasions such as the WTD, IWD, AKAM, Gandhi Jayanti and others. Allies has become a “Go To” Partner for TBC Scale Up. The REACH ALLIES Team has become the go to team for the states to provide TBS to TBC training in districts beyond project interventions. The team has supported the project states in the statewide TBC trainings. The effort has included building capacity of other partners such as JHPIEGO, KHPT and The UNION as trainers for countrywide scale up of TBC strategy. The ALLIES team has also supported the training in some non-project states such as Haryana for TBC training. ALLIES policy influence has been significant. ALLIES took the lead in developing the strategy document; “To End Stigma and Discrimination Associated with TB” in partnership with other partners working on tuberculosis elimination. The strategy has been endorsed by the government of India and serves as the guidance for the country. The ALLIES project developed the Workplace Policy on TB for the state of Jharkhand after extensive deliberations with the Departments of Health and Industries. The Policy has been notified by the Jharkhand Cabinet making Jharkhand the first state to have such a policy across the country. Achieving Excellence in TB Care and Services (AETBCS) training for health staff has been reported to be very useful. The training helped in improving attentive care and communication skills of the service provider. There is evidence that all cadres of health care providers found it unique and made them more sensitive to PWTB needs. LQ 5 Conclusions The ALLIES activity has been successful in creating awareness and bringing TB elimination into the policy agenda at the state and local levels. The contribution from varied ALLIES in the project states sets them on track to TB elimination. The project has contributed to scaling up the TBC strategy across the country and can play an increasingly important role in making it successful. TBCs zeal to serve has been well channeled to improved TB services at the facility and community level. The activity is well on track with reference to MEL Target plan (682 TBS trained, 35 district level survivor network, 264 TBC trained in CAF, etc.). The ET found that while ALLIES has made significant progress in supporting scale up by the GOI, there are issues with the CAF approach layered on top of the TBC model. The TBC is now integrated into the GOI health administration, albeit at the lowest rung of the ladder. This could call Elected Representatives Engaged  Panchayat Heads  Urban Ward Councilors  Members of Legislative Assemblies  Members of Parliament  State Health Minister USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 47 into question the community accountability function of the TBCs, whose appointment and approval of incentives is in the hands of their NTEP counterparts. ALLIES has not addressed this important issue, for example by giving a greater role for the other allies, such as the SLNs. There are also issues with the utility of the CAF tool, which needs review and improvement. As it stands, it is time consuming, involving duplication of data entry, and monthly reporting, which may be an excessive burden. The evaluation also finds that there are some gaps in using standard project management practices, such as a robust Theory of Change to monitor outcomes and move towards its goal; and robust performance data for project oversight and course corrections. The ALLIES activity has not been consistent in formally documenting its strategies and plans, leading to differences in implementation across the project states. There is also a dearth of quantitative evidence on outcomes to say to what extent the strategy has been successful in each state, explain how different approaches in each state contribute to those outcomes and derive lessons for further scale-up. The communication/reference material for the ALLIES activity for some audiences like the media fellows is very rich but lacking for key ALLIES such as the Elected Representatives and the Employers. The absence of an overall SBCC strategy for improving health seeking behavior and stigma reduction is noteworthy in a TB program. The absence of a clear, well documented exit strategy leads to the risk of ALLIES’ interventions not sustained post project. Additional Feedback From Person With Tuberculosis (PwTB) Findings The ET interacted with 31 PwTBs from the selected districts in the project states. PwTBs are the ultimate and most important stakeholders of the program, hence it is crucial to understand their perception of quality of care, access to health services, support provided by TBCs, and issues related to stigma and livelihoods. The analysis is presented under these themes. Table 4: Characteristics of PwTBs Interviewed State Gender Age (In years) Education Status Occupation M F >=50 <=50 No formal education Primar y Secondary Highe r Forma l Non￾Formal Jharkhand 4 3 2 5 4 1 2 1 6 Chhattisgarh 4 4 1 7 2 2 1 3 0 8 Odisha 4 4 5 3 4 4 0 0 1 7 Tamil Nadu 4 4 4 4 0 0 1 7 5 3 Total 16 15 12 19 10 7 2 12 7 24 Percent 52% 48% 39% 61% 32% 23% 6% 39% 23% 77% (Source: PwTB Interviews) 48 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov Table 5: Type of PwTB and Duration of the Anti-TB Treatment State Type of PwTB Duration Of Anti-TB Treatment DS-TB DR-TB Intensive Phase Continuation Phase 0-2 months 3-4 months 5-6 months Jharkhand 7 0 3 3 1 Chhattisgarh 7 1 3 0 5 Odisha 8 0 3 5 0 Tamil Nadu 6 2 2 3 3 Total 28 3 11 11 9 Percent 90% 10% 35% 35% 29% (Source: PwTB Interviews) Insofar as the issue of Quality of Care, this theme was assessed in line with the QoC indicators defined by the project. PwTBs' health-seeking behavior varied depending on whether they lived in tribal, rural, or urban areas. In both tribal districts (Jharkhand & Odisha) PwTBs preferred government facilities over private care, while it was the other way round in Tamil Nadu and Chhattisgarh. In many instances, it was observed that PwTBs visited both private and government facilities multiple times for a concrete diagnosis. Correct diagnosis in the private sector, created distrust on government facilities among them. One PwTB From Tamil Nadu says “I had a breathing issue and went to the GH and ESI Hospital, had tests two times. The result came Negative both times. But I have lost 31 kg weight. Then I went for a check￾up in PSG hospital (Private), they have confirmed TB.” It was found that all respondents and community members were aware about signs and symptoms of TB. However, PwTBs did not have clarity on what tests needed to be done for TB diagnosis, which needs to be conveyed during their first meeting with NTEP staff. A few of the respondents revealed that though they got diagnosed from private facilities but preferred government hospital for treatment as the services are free. For All PwTBs, the duration between diagnosis and treatment initiation was within standard reference time. Due to Government staff promptitude, PWTBs received their test reports within standard reference time. NTEP staff provided the reports in presence of ASHA, so that treatment adherence can be ensured by her. Most of the cases PwTB appreciated ASHAs support during treatment period. One PwTB from Jharkhand said “Only she has given me the medicines and told me to take medicines on time. If needed, she said she will take me for check-up.” In difficult terrains, PwTBs received medicines from TBCs. The government has introduced TPT in a few states. But a significant number of PwTBs reported that their family members were neither tested nor received any medicines from the Government. In Odisha, one PwTB shared that ASHA informed them that testing will be done, and medicines will be provided but nothing happened. PwTBs from Odisha quoted “ASHA asked how many members in your USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 49 family. I said 13 members. She said all your members will get tested for TB, but she didn’t come back.” In Chhattisgarh 2 PwTBs received TPT service and completed the treatment duration. There seems to be a need to focus the attention of health staff on preventive treatment and contact tracing. The majority of the PwTBs were aware of the NPY scheme and appreciated TBCs efforts to assist them in getting the NPY payment. The TBCs helped in addressing operational issues such as opening bank accounts, submitting documents to the bank, submitting documents to the STS and following up with the STS if there are delays in payment. Few PwTBs shared that they were not sure whether they had received NPY or their pension. Some PwTBs flagged that they were not receiving old age pension due to paperwork issues and not resolved yet by anyone, but all are getting rations under PDS. The GOI initiated the Nikshay Mitra and most of the PwTBs from Odisha and Jharkhand have been adopted by Nikshay Mitras. It was evident from respondents’ feedback that TBCs were playing a significant role in linking them with Nikshay Mitra and due to their active involvement PwTBs were aware of who their Nikshay Mitra was. One PwTB from Jharkhand shared “The nutritious food which I received; I got it from “DC saheb”. But they mentioned that food basket received by them was not enough for a month.” Access to healthcare services is still a major issue. PwTBs have to visit health facilities multiple times for tests, reports and sometimes for medicines, which leads to expense on travel. Respondents specially from hard-to-reach areas (Odisha, Jharkhand) reported the desire to avail government transport facilities which is not happening now. It was observed that Respondents were not aware of the Government provision of Rs.750/- travel benefits. Only one PwTB in Odisha mentioned it. TBCs can inform communities during their interaction with PwTBs regarding the provision. Along with travel expenses, in Chhattisgarh and Tamil Nadu, the cost incurred by PwTBs in private facilities ranged from Rupees 8,000-700,000, which led to significant financial burden on their family. Most respondents reported not having to wait for long in the facilities. Two PWTBs reported that they had waited for 2-3 hours for testing. However, locating the facilities was not difficult and tests get done within standard reference time. None of the PwTBs reported any kind of misbehavior or ill treatment from health staff in any of the states. They further shared, health staff were very cooperative, counselled them on TB and treatment adherence. One PwTB in Jharkhand said “They gave medicines for one month and I received calls from the hospital when 2 days of medicines was left with me and call us to the hospital “aap ka do din ka dawa bacha hai”. In Tamil Nadu, all PwTB shared they had been visited by both STS and TBCs and received follow up calls regarding their health or any issues faced during treatment period. Overall, the evaluation found that PwTBs had considerable appreciation for treatment provided in government facilities. At this stage NTEP Staff, ASHA and TBCs coordination was playing a key role in streamlining the process. However, in case of Jharkhand, two PwTBs reported that they had given money for testing in Government facilities and in Odisha, ASHA asked money for her services provided to PwTB. It is clear that TBCs involvement shaped the perception of PwTB regarding TB as seen in Jharkhand and Tamil Nādu. PwTBs recognized the effort of TBCs and sought support from them for their difficulties. One PwTB from Tamil Nadu said “TBCs are giving good mental support… They are encouraging so much. He told me that “If you faced any issue like that 24/7 you can call me”. In some 50 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov instances, PwTBs of Odisha could not recognize TBCs which indicates the need for more frequent contact. It was observed that TBCs paid multiple visits to the PwTBs with vulnerabilities. TB-related stigma or discrimination was not observed in tribal or rural districts. One PwTB from Jharkhand shared that his employer helped him to access TB services and provided income opportunities for his wife. He said “no issue if you are diagnosed with TB. Go to the Govt. health facilities and take medicines, it’s free. He did not discriminate and told me to inform if I need any help. However, in Tamil Nadu and Chhattisgarh, PwTBs had not shared their TB status with the community due to self-stigma due to the fear of losing their job and eviction from their rented house. This was evident in the case of one PwTB from Tamil Nadu “I could not call you to home since I am staying in a rented house. If house owner knew about my TB status, he will tell me to vacate my home.” Overall, all PwTBs were well supported by their family members. On livelihoods, the ET found that, except in Tamil Nādu none of the employers in the formal or informal sector forced any PwTB to quit their jobs. However, in many cases, due to their physical weakness they were unable to perform their jobs. PwTBs who were working in the informal sector had their family members restrict them from working due to labor-intensive work. In Tamil Nadu, no PwTBs lost their livelihood, as most of them were from the formal sector and joined back work after completing the Intensive Phase of treatment. One PwTB from Tamil Nadu quoted “November 14th I got a TB positive test. So, Nov 14th to Dec 13th I have taken leave. Dec 14th I again joined work. But I will always wear a mask.” There were few instances where employers pressured PwTBs to leave their job, which is a cause for concern. But Many PwTBs shared the desire for of work opportunities which are less labor intensive and can be done at home. Conclusions • There are variations in health-seeking behavior between states. • TBCs played a key role in linking patients to Nikshay Mitra and other Government social schemes. • TB-related self-stigma was observed more in urban and semi-urban areas. • Effective counselling by TBCs could shape the perceptions of PwTBs. USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 51 RECOMMENDATIONS RECOMMENDATIONS FOR REACH The recommendations for REACH are aimed at two things. First, to make some course corrections to enhance the efficiency and effectiveness of the ALLIES current implementation. This would broadly mean stepping back and reflecting on what has been achieved and planning strategically for the remaining duration of ALLIES. The second set of recommendations are targeted at taking the lead in supporting the Central TB Division and State TB Cells (of the ALLIES states) in effectively scaling up, monitoring, and documenting the TBC model. An important part of this effort will be to ensure that critical processes in the model are not lost as the government scales it up. 1. Consolidate the TBC Model on the principles of “Build Operate Transfer”. The TBC model has been accepted for scale up by the GOI. It is crucial that the GOI and the state governments have complete toolkits to effectively implement it at scale. REACH could play a crucial role in supporting the government by providing a complete package of resources that will be needed for the scale. This could include: • Training Resource Package • Communication Material and guides • Monitoring tools, SOPs and skills to monitor • Talent Development, Management and Retention Tools • Management tools for CTD, states and districts • Financial guidelines and prototypes • Cross learning platforms and forums 2. Transitioning TBCs to HWCs REACH has piloted the TBC model and best understands its nuances. However, with the government’s decision to transition TBCs to the HWCs, the model will need to be adapted to work as a more integral part of Comprehensive Primary Health Care (CPHC). REACH should play a lead role in this transition with a careful eye to developing the tools and processes for TBCs, NTEP and also the HWC ecosystem to make this change. It will be important to develop a phased strategy for TBCs to move from the TU to the HWC as the hub of their activity and as part of the CPHC. The GOI has issued a directive to place two TBCs at each HWC. There are many aspects of this transition that need to be detailed such as flow of data to the TBCs, supervisory relationships, workload (hence the linked incentive) etc. As a case in point an average HWC-SC will cover about 10,000 population which translates to a monthly notification of 1- 2 per month (at an annual average of 200 per 100,000) and a total number on treatment at about 12 per HWC. These numbers will vary by geography, prevalence of MDR TB and other factors. This implies the role of the TBCs could include more frequent follow ups and hence a need to evolve their skills sets for counseling and addressing problems like side effects, non-compliance to treatment and supporting management of comorbidities. 3. Making CAF more effective CAF as currently implemented by the TBCs is time consuming and intense. To move towards a model that is scale up ready, it needs to be reconfigured. Some areas for consideration are listed below: 52 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov • The questions can be reduced and focused on indicators that are not available from the NIKSHAY portal and ones where the gaps can be addressed within the TU or escalated to the district/state for action. • The frequency could be reduced to ensure that there is adequate time for the gaps to be addressed. • The questions should be tested for validity, reliability, and consistency. • The processes for providing feedback and bridging gaps should be systematized and documented. • Data entry and analysis should be streamlined for prompt action. The current CAF model revolves around the TBC’s leadership. In the scaled-up version of the model, TBCs will work in subordination to the STS and/or the CHO. Within the government hierarchical structures, the TBCs will not be able to seek unbiased PwTB feedback and provide the same to the NTEP team. The ALLIES team should strategize on how the learnings from the CAF implementation in ALLIES be transformed to a scalable model. We recommend that the district level survivor’s networks can be the community voice to assess quality of care and provide feedback to the health system. The networks can serve as equal partners to the health system in addressing issues on the community side such as stigma, lack of means of employment, nutrition support etc. 4. Strengthen Survivor Networks ALLIES should develop a clear Organization Development Plan with the SLNs to build them into technically and financially strong compliant institutions that can fund raise from the government and donors to design advocacy programs for PwTB needs. ALLIES can handhold them to grow into sustainable institutions to contribute to the goal of tuberculosis elimination. While not fully analogous there are lessons to learnt from other similar efforts such as the HIV/AIDS networks. 5. Leverage Employers to nurture skilling and income generation activities The ALLIES project should consider working with employers in the project geographies to support skilling and employment based on the needs of these industries and / or markets. The Corporate Social Responsibility Funds can be tapped into for the same as well. The Corporates could also support through employee volunteering to convert PwTBs trained in certain skills to develop into microentrepreneurs either individually or as collectives. Each employer has a CSR vision document or strategy. ALLIES should consider a more intense approach where the team researches the Corporate, understands it areas of support under CSR and then work with the Corporate to devise a simple plan that they can support for PwTB and SLNs. This could be in the form of medical resources, funding for livelihoods, skilling, employment in the industry, communication campaigns etc. In the long term the SLNs can take lead with the Employers to take this agenda forward. REACH could support the registered networks to the 80G certification that allow the SLNs to legally access CSR funds. 6. Describe what ‘rights- respectful’, ‘gender’ and ‘age responsive’ TB services should look like and how to get there. The ALLIES Project started with a very bold Objective 1 that envisions transforming service delivery for PwTB. Demonstration of any model that makes services ‘rights- respectful’, ‘gender’ and ‘age responsive’ and is simple enough to be scaled can be an infinite contribution to health care service delivery beyond tuberculosis as well. While ALLIES has ventured into implementing special USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 53 interventions in select districts that include addressing vulnerabilities, elderly and disabilities in specific districts, the contours of the model are still unclear. The ET recommends piloting a simple intervention package to create some standards for ‘rights￾respectful’, ‘gender’ and ‘age responsive’ services. The pilot will be served well by being context specific, at least for states and preferably districts as the constructs need to be sensitive to the socio-cultural norms more than the medical norms. There should be a clear focus on systematic measurement and documentation of activities completed using analogues of the PDCA cycle that does not seem to be happening in the current special intervention pilots. RECOMMENDATIONS FOR USAID (FOR USAID EYES ONLY) The recommendations for USAID are aimed at two things. First, to work with REACH to ensure that the ALLIES investment yields maximum return by helping the country progress towards the goal of TB elimination by 2025 including sharing lessons with the global TB community. The second set of recommendations are targeted at USAID taking continued leadership in ensuring that the TBC model, developed with USAID investments, can be successfully and effectively scaled up. With the GOI commitment to scale up TBC model, USAID has unlocked a huge potential that it can harness through its future funding decisions. 1. Take the lead in restructuring ALLIES to make it more strategic, methodical and evidence – based. There needs to be a particular focus on better data use for monitoring and tracking. 2. Strategize with the Mission Health Office on how to support GOI policy of transitioning the TBCs to HWCs. There would be a need to consider which partners/ grants would be best suited to play this role in conjunction with REACH. 3. Define a cohesive funding strategy for technical assistance to the government to scale up the TBC model. The model should learn from the ALLIES experience to be effective in achieving the goal of TB elimination. Some of the aspects that need careful consideration and discussion are: • Transition to the HWCs and coordination with TUs • Role clarity vis-à-vis other frontline health workers • Monitoring systems for performance • Toolkit for all stakeholders • Incentives structures • Career progression and Expansion of role as TB moves to elimination 4. Take up the Employer Led Model a notch by: • Leveraging private sector partnerships across the USAID Mission, working with the Private Sector Engagement (PSE) team in the Mission to strategize on leveraging CSR funds. • Taking the role beyond public health towards rehabilitation and support to TB Survivor Led Networks. Considering there is significant political will behind TB elimination, concerted push from Corporates and / or their collectives like the FICCI and Chambers of Commerce and Industries will not only push the agenda to another level but also strengthen efforts such as the Corporate TB Pledge. 5. Make better use of NIKSHAY data. Work with CTD, MOHFW, GOI to regularly analyze NIKSHAY data for in-depth insights for improving the NTEP implementation. The ET understands that this is a difficult ask but suggests that exploring viable mechanisms to do this would be big step forward. One such option would be to work with offline downloaded data as there are sensitivities with accessing the live portals. 54 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov 6. Support expanded Achieving Excellence in TB Care and Services (AETBCS) training for health staff across various components of NHM. This curriculum and training have been widely appreciated by the service providers and can be considered for providers under other USAID grants working with service providers. 7. Finally, given that the CAF data are not suitable for measuring outcomes, USAID should collaborate with the GOI to conduct a quantitative analysis of TB time series service data and to measure the contribution of the ALLIES’ approaches and tools, for example using a matched sample of ALLIES TBC and GOI TBC districts. USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 55 ANNEX 1: LIST OF STAKEHOLDERS INTERVIEWED Stakeholders’ positions in different levels No. of KIIs conducted National level ▪ Joint Director, National Consultant ACSM, National Team Lead ▪ Technical Specialist-Tuberculosis ▪ Project Management Specialist ▪ Chief Infectious Disease Division 06 ▪ Sr. Advisor and Project Lead ALLIES ▪ Deputy Project Lead ▪ National Operations Coordinator ▪ National Operations Coordinator ▪ Monitoring & Evaluation Lead ▪ Knowledge & Management Specialist ▪ National Operations Coordinator (Special Intervention) ▪ National Coordinator 08 State level Jharkhand Odisha Chhattisgarh Tamil Nadu ▪ STO, State Training and Demonstration Center (STDC) Director, Consultant, SOL ▪ M& E Lead, Network Coordinator ▪ ELM, Media Fellow 07 09 06 09 District Level Chief Medical Officer (CMO), DTO, DEO, DS, DPC, District Network President 04 05 02 05 TU level STS, Senior Tuberculosis Laboratory Supervisor (STLS), ER, BMO Lab Technician, TBC 09 10 12 07 Community Level PWTBs 07 08 08 08 Non-Project Districts and Respondents District Level Jharkhand Odisha Chhattisgarh DPC, DPPMC 2 1 2 TU level STS, TBC 5 1 4 Total no. of Interviewed 145 56 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov Non-Project District Respondents Category State District Respondents Chhattisgarh Dhamtari DPC, DPPMC, STS, TBC (3) Jharkhand Sahibganj DPC, DPPMC, STS (4), TBC (1) Odisha Sundargarh DPPMC, STS USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 57 ANNEX 2: EVALUATION TOOLS Kii Guide – USAID/Reach Leadership / Staff Interview Date: Interviewer: Respondent Name: Designation: Duration of Charge in the position: Contact Details: Has the KI affirmed Informed Consent? Y____ N____ Consent for audio recording Y____ N____ 1. Background for REACH staff/ Leadership 1.1. What are your key roles and responsibilities? 1.2. In your opinion what are key areas of concern in your country/state/ district that impedes improved health outcomes pertaining to TB? 1.3. In your opinion how the ALLIES Project has overall contributed to improve the health outcomes pertaining to TB? (Probe for priority issues concerning policy, health systems, access, quality and adequacy of health services, community awareness, social stigma/ discrimination) 1.4. What challenges need to be addressed for the activity to achieve the planned results (if any)? 1.5. How can activity implementation be accelerated (if needed)? 1.6. What are some of the missed opportunities that can strengthen the program further? 2. How did the project develop its strategy, objectives and activities? 3. What is the potential for scaling up the TBC model? What are the lessons learnt about scaling up the TBC model? What are the processes that will need to be scaled up for effectiveness of the model to be seen at scale? What are the areas where the national / state government may need support from DPs and Partners for effective scale up? 4. According to you, what have been some of the most important achievements of the project? 5. According to you, what have been some of the biggest challenges for the project? 6. What efforts has the project made to integrate gender sensitivity and reduce gaps? In your opinion to what extent have these efforts been successful? 7. What are the most important lessons from the project that can inform other TB investments either in the government or NGO sector? 8. What efforts does the project need to make to ensure sustainability of processes and outcomes after the project is completed? 9. What efforts does the project need to make to ensure scalability of successful interventions? What knowledge products/ toolkits can the project develop to support scale up? * REACH staff will be probed in detail about their roles and details of that responsibility i.e., M&E, communication, KM, Documentation, Operations, partner management etc. 58 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov Kii Guide – Government Official (Central/ State/ District) Interview Date: Interviewer: Respondent Name: Designation: Duration of Charge in the position: Respondent Organization/ Dept.: State & District: Contact Details: Has the KI affirmed Informed Consent? Y____ N____ Consent for audio recording Y____ N____ 1. Background 1.1. What are your key roles and responsibilities ______________ (stakeholder organization)? 1.2. In your opinion what are key areas of concern in your country/state/ district that impedes improved health outcomes pertaining to TB? 1.3. In your opinion how ALLIES Project has overall contributed to improve the health outcomes pertaining to TB? (Probe for priority issues concerning policy, health systems, access, quality and adequacy of health services, community awareness, social stigma/ discrimination) 1.4. What have been three- four main achievements of the project? 2. LQ1: Progress towards establishing ‘enabling environment’ by leveraging community actions to build a culture of accountability for TB elimination. A. To what extent the project has created the key ALLIES who can undertake strategic advocacy for enabling environments at multiple levels to shape rights￾respectful, gender and age-responsive TB services? 2.1. As you are aware, ALLIES project’s mandate is to facilitate advocacy for rights-respectful, gender and age responsive TB services. In your opinion what are the key issues when it comes to ‘rights-respectful’, ‘gender’ and ‘age’ in delivering TB services in your state/ district? Are there any other issues? 2.2. What are the different interventions/ approaches ALLIES project has undertaken towards creating institutional structures for advocating rights-respectful, gender and age responsive TB services? (List down interventions/ approaches and probe for Survivors-led network in state and district, CAF, TB Champions, Private Sector, AETBCS training, any other) 2.3. For each of the above-mentioned interventions ask the following question separately: - 2.3.1.Mechanism of your/ your department collaboration with interventions? 2.3.2. In your opinion if the approach effective in addressing the key issues (as-mentioned above)? Is there a difference in health outcome between project and non-project USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 59 location in your state/ district? Is there evidence (from Nikshay or other data source) to support improved health outcomes? (Ask if data can be shared?) 2.3.3.Key achievement towards the advocacy efforts? 2.3.4.Main challenges which have impacted the efforts of advocacy? 2.3.5.How do you think that these interventions can be made more effective? 2.3.6.Will your department expand, continue, reduce, or discontinue the interventions (especially in case of TB Champions) once the project has ended? If no, why not (probe for issues- policy, financial, human resource, structural, etc.)? If yes, how? (Probe for support/ resource required). Will the department adhere to the program after it has ended? If yes, how? (Probe for support/resource required). If no, why not(probe for issue-Policy, financial, human resources, structural, etc.) B. To what extent the project has established community-owned mechanisms to monitor quality of TB care and services, and give feedback to the program for timely responses, helping institute accountability and strengthening community empowerment? 2.4. What community owned mechanisms have been established in your state/ district by the projects? (Probe for knowledge and understanding of CAF). How many TUs in your State/ District/ Block being covered by community monitoring mechanism? (List the numbers/ district/ blocks/ TUs). 2.5. In your opinion is the mechanism helpful in providing feedback to improve quality and care under TB services? If yes, ask following question: 2.5.1.Mechanism for provision of timely feedback to improve TB health services? (Probe for regular report, meetings, consultation, or any other means) 2.5.2.Any notable areas of community demand, care, and services improvement? (Probe for availability of data to substantiate improved services) 2.6. Would you like to highlight any lessons or challenges pertaining to the intervention? 2.7. Any suggestions in your opinion for the project to be able to provide more timely feedback mechanisms for Improving the intervention? C. To what extent has the project successfully promoted discourse on enabling the policy, regulatory and financial environments to support TB elimination and reduce TB-related stigma/discrimination at state and national levels. 2.8. What activities has the project undertaken to generate a discourse on policy, regulatory and financial environments to reduce TB-related stigma/discrimination? (Probe for each area separately) 2.9. Can you highlight key achievements of the projects about their policy discourse? 2.10.Have there been any policy, regulatory and / or financial changes in NTEP or the health system at large to which the project contributed? (List out the changes) 3. LQ 2: How have the intervention activities and approaches integrated gender across program implementation? What evidence exists to substantiate the reduction of gender gaps? 3.1. Were the intervention activities and approaches designed and implemented appropriately to address gender differences/gaps? 3.2. What evidence exists to substantiate the reduction of gender gaps? 3.3. Did the project focus on Other Vulnerable groups (probe: -Transgender, Migrant population, elderly, person with disabilities)? if yes, how it has been done? If no, how can we address this issue? 4. LQ 3: To what extent the Community Accountability Framework (CAF) is implemented and what is the effectiveness? How has the Empowerment Evaluation (EE) approach contributed to the implementation of the CAF process? 60 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov 4.1. In how many TUs across state/ district CAF has been Implemented? In what capacity your department is involved in its implementation? (Probe if CAFR reports and Block Level Action Plans are shared at any level) 4.2. In your opinion how did the intervention support TB services in the state/ district/ TUs? (Probe for improved service delivery, improved service uptake)? Can you share any data supporting improved service delivery and uptake? 4.3. Do you think Empowerment Evaluation (EE) approach contributed to the CAF implementation? It yes, how? (Probe for any instances of increased accountability of HF/TU staff improved quality of services). If no, why? 4.4. How do you think that the CAF implementation can be made more effective? What are the key challenges? Are there any learnings you would like to share? 5. LQ 4: How effective the new district and state level TB survivors led networks are in facilitating the rights-based approach to TB, and what is their role in advocacy promotion? 5.1. Are/ you aware of TB Survivor Led network established by the project in the state and district? How many are there in state/ districts? Are they registered? Through what mechanism they collaborate with you/ your department? (At central level probe for their collaboration with National TB Forum) 5.2. Are you aware if these networks collaborate across other states/ districts? Can you highlight any cross learnings from other states/ districts the network has advocated/ facilitated? 5.3. Can you highlight some of the key accomplishment in terms of advocating for TB-Patient rights and improved service delivery? 5.4. Any key lessons or learnings? In your opinion how can these networks be more effective in creating synergy with the government and other stakeholders towards addressing policies and improved TB services? 6. LQ 5: What are the overall accomplishments, challenges, and learnings out of the project implementation so far? 6.1. In your opinion what are overall main accomplishments of the project so far? 6.2. What challenges need to be addressed for the activity to achieve the planned results (if any)? 6.3. How can activity implementation be accelerated (if needed)? 6.4. What are some of the missed opportunities that can strengthen the program further? 6.5. What lessons have been learned so far? 6.6. What lessons can be used to improve implementation? USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 61 Kii Guide – Lead of “Survivor Led Network” Interview Date: Interviewer: Respondent Name: Respondent Organization: Respondent Job Title: 1. Do you think survivor led network-building is important? a. Yes or no? b. In either case, please explain in words 2. What is the process followed building of survivor led network-building? a. Steps for the recognition/identification of the group/potential members b. Identification of objective of the network and achievement of goals c. Identification of leadership d. Designing the work-plan and e. Monitoring mechanism (minutes of meetings, monthly and quarterly reports etc.) f. Legal registration of the network g. Financial management process h. Options of funding i. Sustainability plan 3. How did the network support PwTB on the implementation of Community-owned Accountability Framework (CAF) in the community? 4. Did the network ensure the gender-responsive support to PwTB? If no, what are your plan to support? If yes then steps, significant achievements. a. Steps taken to solve gender disparity in the community b. Significant achievements – i. Inclusion of genders and sexual orientation of the community ii. Customized gender-responsive approach as TB affects diverse genders in different ways at different stages of TB 5. Did the network support PwTB in terms of psychosocial support? If no, what are the challenges, if yes, probe- (a. Emotional support - Demonstrate empathy, caring, or concern to bolster the person’s self-esteem and confidence (b. Instrumental support - Share knowledge and information (c. Affiliation support - Provide assistance to help PwTB accomplish tasks (e.g., make daily plans, provide referrals, enrollment in program, etc.) 6. Did the network support PwTB in terms of stigma mitigation? If no, what are the challenges? If yes probe￾a. Steps taken to reduce TB-related stigma at various levels including at families, neighbors, workplaces, and communities b. Steps taken to handle various age-old myths and misconceptions related to TB c. Support for the storytelling to PwTB by speaking up boldly and freely about how TB impacted their lives without fear of judgement or stigma 7. How did the network support PwTB in terms of advocacy with multiple stakeholders including government officials, policy makers/bureaucrats, private sector, health officials, elected representatives and the media on various issues related to TB? a. Feedback mechanism to the multiple stakeholders to support PwTB b. Follow-up on the feedback with the multiple stakeholders with the timelines c. Participation in District/State/National TB Forum 62 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov 8. How did the network provide social security to PwTB? a. Provide social support for access to schemes and benefits b. Prevent or reduce the loss of wages c. Support for the employment/livelihood 9. How did the network support PwTB in terms of community mobilization and awareness? a. Sensitize communities about the basics of TB b. Connecting PwTB with Health facilities to seek TB services c. Connecting PwTB with Health providers to connect for provision of TB services 10. How did the network support the capacity building of PwTB? a. Support for the CAF training for TB survivors b. Support for the AETBCS training for health providers 11. Did the network support the Employer Led Model (ELM) sensitization and engagement in your state/district? If yes, how? Probe for any instances 12. Did the network engage the Elected Representatives to support PwTB? If yes, how did they engage, If not, why? 13. How did the network support the innovations for PwTB in your state/district through CAF? 14. What was the support received from the Tuberculosis Unit, District, State and National level officials of health systems to the network? a. PwTB to be treated with respect and dignity, including the delivery of services without stigma, prejudice, or discrimination by health providers and authorities, if any b. Support for the public health actions – screening, diagnosis, treatment, follow-up, rehabilitation, and direct-benefit-transfer (Nikshay Poshan Yojana) c. Participation in District/State/National TB Forum 15. Did you collaborate/exchange ideas with the other TB Champions/Survivors led network for support PwTB? a. Regular mechanism in-built in the system for the cross learning between the networks 16. What are the major challenges faced by the network? 17. What were the steps taken by the network to overcome the challenges? 18. What would you suggest to TB Champions/Survivors of other geographies decide to formulate the network? USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 63 Kii Guide – Tb Champions Interview Date: Interviewer: TB Champion Name: Category: Working with ALLIES: Community Internship: Non-Project TBC: State and District: TU and Block Name: TB Champion Since (months/ years): Contact Details: Has the KI affirmed Informed Consent? Y____ N____ 1. Background 1.1. Since when you are a TB Champions associated with ALLIES project (do not ask association with project in case the non-project TBC)? Can you provide details for the following? Responsibility of what population Responsibility of how many families How many PwTB in your area Responsibilities of how many PwTB in the community 1.2. What are the key issues regarding TB in your areas? Issues pertaining to Community (probe for stigma/ social discrimination, awareness, vulnerable population, gender-based issues, etc.)? Issues pertaining to Health care and services (wait time, OOPE, distance, behavior of providers non￾availability of diagnostics, medicine shortage, access issues, quality of care, HR shortage, etc.,)? 2. Roles and Responsibilities 2.1. What are your key roles and responsibilities as a TB Champion? 2.2. Why did you become a TBC? What motivates you to be a TBC? 2.3. What are the main: 2.3.1.Challenges in performing your role towards ‘Awareness Generation,’ ‘facilitating stigma reduction’, ‘carrying out advocacy’, ‘providing emotional and psychosocial support’ and ‘providing feedback to health system’? (Probe for each of the role) 2.3.2.Accomplishments towards ‘Awareness Generation,’ ‘facilitating stigma reduction’, ‘carrying out advocacy’, ‘providing emotional and psychosocial support’ and ‘providing feedback to health system’? (Probe for each of the role) 2.3.3.What support did you receive from ALLIES Team/ TU/ PHC/HWC/District team in performing your role? (Omit ALLIES if not a project area TBC) 64 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov 2.4. Community Monitoring: 2.4.1. Is Community monitoring i.e., CAF being implemented in your area? If yes, since when and what role do you play in the whole process? 2.4.2.How regularly Block Action plan is being prepared? In your opinion has the block action plan facilitated improvements in improving TB services and care? Can you tell some main accomplishments? 2.4.3.What are the key challenges in implementing community monitoring systems i.e., CAF? 2.5. Are you associated with State and/or District Survivor led network? How does the network support you in improving QoC and QoS for PwTB? How can they improve the support? 2.6. Is there any coordination mechanism with other health care providers and facilities (e.g., PHC, HWC, ASHAs, ANMs, AWW, VHSNC, RKS, etc.)? If yes, how do you coordinate/ collaborate? Any challenges you face while working with them? 3. Trainings 3.1. What trainings have you received (from ALLIES Project, Government)? (List the trainings separately) 3.2. Do you think that the trainings you received prepared you enough for the role of TBC? 3.3. What additional trainings you think can help you do the work better? 3.4. Any feedback or input for improving the trainings? (Probe for pédagogie, content, mode, etc.) 4. Supportive Supervision and Job-Aid 4.1. Who supervises your work on a routine basis? How regularly does supervisory visit happen? 4.2. Do you think the supervision you receive is helpful? If no, what do you think should be included in supervision? 4.3. Have you received any job-aids to help perform your work? (Probe for checklist, IEC materials, mobile based tool, etc.) 4.4. Do you think the job aid you received are helpful? If yes, which one is more helpful and why? If no, what else you think should be added? 5. Incentive/ Stipend 5.1. How much stipend do you receive for your work? 5.2. Do you regularly receive your payment/ stipend (quarterly/ monthly)? When did you last receive it? 5.3. Does it motivate you to keep doing your work as a TBC? Why/ Why not? 5.4. Is there anything else you would like to share with us? USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 65 Kii Guide – Medical Officer – Tuberculosis Control (Mo – Tc) Interview Date: Interviewer: Respondent Name: Respondent Organization: Respondent Job Title: 1. Are you aware of the Community owned Accountability Framework (CAF) implemented by the TB Champions through ALLIES project in your Tuberculosis Unit (TU)? 2. How were you able to contribute to the implementation of CAF by the TB Champions through ALLIES project? a. Your participation in meetings of TB Champions/Survivors at block/TU/district/state etc.? b. Implementation of feedback given by TB Champions/Survivors in monthly/quarterly meeting c. Any other process that you were able to support the TB Champions/Survivors 3. If yes, what do you think has went well in terms of implementation of CAF by the TB Champions through ALLIES project? a. Gender-based support b. Community-based support c. Right-based support d. Innovations e. Curriculum development – include any more topics f. Trainings – frequency, participants, duration, refresher training g. Quality of services h. Quality of care i. Advocacy 4. Did the CAF implementation by the TB Champions through ALLIES project has led to improvement, if yes, probe for any, of the following indicators: a. Average turnaround time of molecular testing – GeneXpert/Truenat b. Average turnaround time of First-Line Line Probe Assay (LPA) and Second-Line Line Probe Assay (LPA) c. Average turnaround time of Liquid Culture – Drug Susceptibility Testing d. Counselling of Drug-Sensitive and Drug-Resistant PwTB and their families for stigma reduction e. Screening for the clinical symptoms f. Specimen Collection and Transportation g. Diagnosis of TB – pulmonary and extrapulmonary h. Treatment initiation of the Drug-Sensitive and Drug-Resistant PwTB i. Treatment Success Rate j. Follow-ups k. Decline in mortality rate l. TB notifications m. Nikshay Poshan Yojana 5. What would you suggest in terms of improvement of CAF implementation by the TB Champions through ALLIES project? 66 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov In-Depth Interview (IDI) Guide – PwTB FEEDBACK Interview Date: Interviewer: Respondent Name: Male/Female: Duration of Treatment: Duration of Charge in the position: Respondent Organization/ Dept.: State & District: Has the Respondent affirmed Informed Consent? Y____ N____ Consent for audio recording Y____ N____ 1. Background 1.1. How do you think are the quality of services from the health care facility that you visit? 1.2. Can you tell me about your experience of taking services from this facility? When did you first go there? What problem were you facing? 1.3. What happened the first time you went to the facility? How long did you have to wait for? What tests were done? How many days did it take for you to get the reports? Were all the reports given in one go or you had to go on multiple days? 1.4. After how many days of the report was the diagnosis given? After how days of diagnosis was the treatment started? Have you had any difficulties in your treatment? What? How far do you have to travel to get your medicines? 1.5. Have you continued your treatment regularly or have there been breaks? Why did you discontinue in between? 1.6. Who do you usually go to if you have any questions/problems? Are your problems usually resolved? 1.7. Have you ever met a TB Champion? How do they help you? 2. Nikshay Poshan Yojana (NPY) 2.1. Do you know of the NPY? Are you enrolled under it? Are you getting your payments regularly? 2.2. Did you have any difficulty in enrolling for the NPY? What difficulties did you face? How was it resolved? 3. Out of Pocket Expenditure (OOPE) 3.1. Do you spend any money for your treatment? (On what, how much per month)? 4. Stigma and discrimination 4.1 How do think is the behavior of health care workers (in the facility/ when they visit your home) 4.2 Describe how well did the TU staff ensure your confidentiality and privacy? 4.3 Have you shared your status of TB with your friends /community members? How respectful are friends/community attitudes towards you? (Ask only those who reply Yes to G1) How accepting and caring are your family members towards you? (Ask everyone, read aloud) 4.4 If you are working for an employer, how does he/she treat you, after you informed her or him about being treated for TB? 5. Is there anything else you would like to share with us? USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 67 Kii Guide – Private Sector Employers Interview Date: Interviewer: Person Name: Name of the Company: State and District: Contact Details: Has the KI affirmed Informed Consent? Y____ N____ Consent for audio recording Y____ N____ 1. How long has your company been involved in CSR activities? What all do you support as part of your CSR efforts. What activities do you support in health as part of your CSR efforts? 2. In your opinion what are key issues regarding TB in name of the district/state? (Probe for issue at community level- vulnerable population, awareness, social discrimination, lack of access to socio￾economic services, at health service delivery level- availability, access, quality, etc. and at policy level) 3. Do you know about the REACH ALLIES project? Have you been engaged with them? If yes, in what way? 4. Have you conducted any activities to support TB elimination? If yes, what are they? (Probe for camps, drives, campaigns, support to TB patients etc.) 5. Have you made any changes in your workplace to support PwTB? (HR policy, health services, screening efforts, monetary support, extra leave, communication campaigns, stigma reduction efforts) 6. Have you made any commitments towards supporting TB elimination? Have you signed any LOI, taken the TB Pledge? What prompted you to sign this commitment? What made you choose TB as a health problem to support as compared to the other health problems? 7. What are your plans for the next five years to support TB elimination? 8. Anything else that you would like to share with us? 68 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov Kii Guide – Elected Representatives (Mla/ Councillor/ Pri Member) Interview Date: Interviewer: ER Name: State and District: Block/City/ TU Name: Contact Details: Has the KI affirmed Informed Consent? Y____ N____ 1. In your opinion what are key issues regarding TB in your constituency / gram panchayat? (Probe for issue at community level- vulnerable population, awareness, social discrimination, lack of access to socio-economic services, at health service delivery level- availability, access, quality, etc. and at policy level) 2. Are you aware of the different interventions been undertaken by ALLIES project in your constituency/gram panchayat? If yes, what do you think are their key accomplishments as a contribution towards improving TB health care services? 3. In what specific areas/ activities you were involved by ALLIES project? What were your contributions (probe for contribution of their funds, any policy change, participation in event, visit to health facilities, etc.)? 4. In your opinion what are the key challenges towards improving TB related scenario in your constituency/ gram panchayat? How can ALLIES project plan and implement their interventions/ activities better to address these? How better you can contribute? 5. Any other suggestions? USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 69 ANNEX 3: EVALUATION MATRIX Evaluation Matrix for REACH ALLIES Midterm Performance Evaluation # Evaluation Question Illustrative Areas of Enquiry Data Source & Collection Method Key Respondents (for primary data collection) Data Analysis Method 1 To what extent has the ALLIES project made progress toward its aim ‘To enable environments for TB elimination by leveraging community action to build a culture of accountability’? ▪ Design of the interventions around advocates been able to achieve desired objective contributing towards larger goal of TB elimination 1.1 To what extent the project has created the key ALLIES who can undertake strategic advocacy for enabling environments at multiple levels to shape rights￾respectful, gender and age￾responsive TB services? ▪ Progress and achievements under different community structures established by the project, viz., SLNs, Touched by TB initiative, TB Champion model, media engagement, etc.? ▪ Design, structure, operationalization, linkages with NTF and other relevant programs, capacities, gaps, and key lessons from SLNs ▪ Design, strategy and mechanism of engagement and effectiveness of TB Champions ▪ Strategy, interventions, challenges, and opportunities towards retention of TB Champions beyond LoP ▪ Extent of Local solutions/ resource mobilization/ innovations and their achievement towards improving TB services, effective linkages with the schemes under NTEP ▪ Contribution to ALLIES or NTEP by private sector￾LOI signed, TB pledge taken, adoption of workplace policy and key activities against TB action plan ▪ TB action plan and reporting by private sector ▪ Key lessons and success stories ▪ ALLIES project progress reports ▪ ALLIES M&E data ▪ ALLIES reports on key lessons ▪ TB Forum Assessment Report ▪ Private sector M&E data and reports ▪ KII with key stakeholders (state, district, TB Champions, network members, etc.) ▪ MoHFW (CTD) ▪ WHO and other relevant Technical Agencies ▪ State TB Cell ▪ State WHO Consultants ▪ State level TB SLN ▪ District Chief Medical Officer ▪ District TB Cell ▪ District Chapter of TB SLN ▪ Touched By TB ▪ TB Champions ▪ TU (MO) ▪ Private Sector Organizations ▪ Comparison of achievement to targets ▪ Analysis of coherence between mandate, vision, strategy and design and core functionality ▪ Analysis of key activities, and performance on indicators ▪ Assessment of contribution to TB services in the state in alignment with priorities and view of key stakeholders ▪ Assessing interest and commitment to sustained support to the cause of TB (for Private Sector Organization) 70 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov # Evaluation Question Illustrative Areas of Enquiry Data Source & Collection Method Key Respondents (for primary data collection) Data Analysis Method 1.2 To what extent the project has established community￾owned mechanisms to monitor quality of TB care and services, and give feedback to the program for timely responses, helping institute accountability and strengthening community empowerment? ▪ Main accomplishments of CAF pilot ▪ Improved TB care and services (improved facilities, timely services, improved footfall, improved adherence, and cure rates, etc.) ▪ Evidences of effective community monitoring resulting in strategic activities undertaken to address specific issues highlighted in periodic CAFR ▪ Linkages with relevant government programs and functionaries. ▪ Key gap areas and opportunities improve and scale up the intervention ▪ ALLIES project progress reports ▪ ALLIES M&E data ▪ CAF Reports ▪ Performance and/or service data of TUs ▪ KII with key stakeholders ▪ MoHFW (CTD) ▪ State TB Cell ▪ NHM ▪ State WHO Consultants ▪ State level TB SLN ▪ District Chief Medical Officer ▪ District TB Cell ▪ District Chapter of TB SLN ▪ TB Champions ▪ TU (MO) ▪ PwTB Interview ▪ ERs ▪ Comparison of achievement to targets ▪ Analysis of improvement in TB Care Services during the life of the project ▪ Assess the use of M&E data programmatic decisions ▪ View of Key stakeholder ▪ Feedback from community/ PwTB ▪ Listing challenges, gaps and opportunities 1.3 To what extent has the project successfully promoted discourse on enabling the policy, regulatory and financial environments to support TB elimination and reduce TB￾related stigma/ discrimination at state and national levels. ▪ Improved stigma reduction, improved TB notification and improved TB services (healthcare and socio-economic support services) ▪ Policy advocacy on high priority areas at district, state and national level. ▪ Policy changes leading to changes in implementation planning and resource allocations (financial, Human Resources (HR), etc.) at state and district level. ▪ ALLIES project progress reports ▪ ALLIES M&E data ▪ Government reports, state PIPs ▪ Government policy documents ▪ KII with key stakeholders ▪ MoHFW (CTD) ▪ WHO and other relevant Technical Agencies ▪ State TB Cell ▪ NHM ▪ State WHO Consultants ▪ State level TB SLN ▪ District Chief Medical Officer ▪ District TB Cell ▪ District Chapter of TB SLN ▪ Touched by TB ▪ TB Champions ▪ ER ▪ TU (MO) ▪ PwTB ▪ Assess appropriateness of policy advocacy based on priorities highlighted by various key stakeholders ▪ Analysis of outcome of interventions (e.g., improved notification, improve socio￾economic services) ▪ Assessing outcomes related to changes in government policy documents, guidelines, resource allocation, etc. ▪ Views and feedback from key stakeholders 2 How have the intervention activities and approaches integrated gender across program implementation? What evidence exists to ▪ Identification of gender gaps in the project (across states/ districts) ▪ Approaches and interventions addressing gender gaps and their key achievements ▪ Existing gaps and opportunities to address them ▪ ALLIES project progress reports focusing on gender integration in project interventions ▪ ALLIES M&E data (data by gender) ▪ MoHFW (CTD) ▪ State TB Cell ▪ NHM ▪ State level TB SLN ▪ District Chief Medical Officer ▪ District TB Cell ▪ District Chapter of TB SLN ▪ Assess appropriateness of each intervention based on outcome and views of various stakeholders USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 71 # Evaluation Question Illustrative Areas of Enquiry Data Source & Collection Method Key Respondents (for primary data collection) Data Analysis Method substantiate the reduction of gender gaps? ▪ KII with key stakeholders on gender gaps ▪ TB Champions 3 To what extent the Community Accountability Framework (CAF) is implemented and what is the effectiveness? How has the Empowerment Evaluation (EE) approach contributed to the implementation of the CAF process ▪ Coverage in terms of State, district, block/ TUs covered under CAF implementation ▪ Effectiveness of CAF as a tool supporting TB Champions in efficiently performing their role ▪ Improved coordination and collaboration between TB Champions, TUs and other relevant functionaries ▪ Improved TB care and services (improved notification, healthcare facilities, timely services, improved footfall, improved adherence, and cure rates, etc.) ▪ ALLIES project progress reports ▪ ALLIES M&E data ▪ CAF reports ▪ Government reports, and Tb Services data ▪ KII with key stakeholders ▪ State TB Cell ▪ District Chief Medical Officer ▪ District TB Cell ▪ District Chapter of TB SLN ▪ TB Champions ▪ ER ▪ TU (MO) ▪ Analysis of outcome of interventions (e.g., improved notification, timely services, improved footfall, improved adherence, and cure rates, etc.) ▪ Assessing appropriateness of CAF in aiding desired outcomes in improving TB Services. ▪ Views and feedback from the stakeholder ▪ Triangulation of findings in relation to activity’s objectives and outcomes 4 How effective the new district and state level TB survivors led networks are in facilitating the rights-based approach to TB, and what is their role in advocacy promotion? ▪ Adequate participation, regular documented meetings, regular meetings with key government functionaries (multi-sectoral), linkages with CSOs, other organizations working on TB in the state, district. ▪ Regular cross sharing of experiences, learning and lessons ▪ policy advocacy on key priority areas, evidence of changes in implementation planning and resources allocations in state PIPs. ▪ Improved QoC to PwTB. ▪ ALLIES project progress reports ▪ ALLIES M&E data ▪ District and state network documents, reports, meeting minutes ▪ Government reports, and Tb Services data ▪ Government reports, state PIPs ▪ KII with key stakeholders ▪ MoHFW (CTD) ▪ State TB Cell ▪ State WHO Consultant ▪ NHM ▪ District Chief Medical Officer ▪ District TB Cell ▪ District Chapter of TB SLN ▪ State TB Survivor network ▪ TB Champions ▪ ER ▪ Comparison of achievement against objectives ▪ Analyzing policy outcomes across diverse section of community ▪ Listing cross-learning and key lessons ▪ Triangulation of findings 5 What are the overall accomplishments, challenges, and learnings out of the ▪ Key accomplishments, learning and lessons from project ▪ Application and sharing of lessons by the project ▪ Lessons for project improvement and future programming ▪ ALLIES document on lessons and learning ▪ KIIs with key stakeholders All key stakeholders ▪ Triangulation of findings in relation to activity’s objectives and outcomes 72 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov # Evaluation Question Illustrative Areas of Enquiry Data Source & Collection Method Key Respondents (for primary data collection) Data Analysis Method project implementation so far? USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 73 ANNEX 4: FINDINGS ABOUT TBCS FROM ALLIES DISTRICTS Detailed findings from ALLIES Implemented TBC model in the project states are as follows: i. Selection and Induction of TBCs: The IP has developed ‘Operational Guidelines on Community Engagement’ with defined criteria for TBC selection (See text box below), but the ET did not find it was available at the state and district level. In practice, the key selection criteria across the project states are ‘Willingness to work’ as a TBC and the rest varies by state (refer to Table below). On average every Tuberculosis Unit (TU) in the project location has 1.4 (1 to 2) TBCs. The deployment of the number of TBCs at a TU does not factor in the population covered, its spread, and/or TUs average TB notification load. The ET did not see any documented TBC selection methodology available with the project. IP’s Eligibility Criteria of TB Survivor Selection To be nominated as TB Champion a person must meet the following criteria: ● Must be a TB survivor ● Must have completed his/her treatment ● Should be able to read or write regional language ● Knowledge of basic English will be an added advantage ● Must be willing to be involved in outreach activities and dedicate some time for the same ● Willing to travel within panchayat and block For selection to be ensured: ● Gender representation (20 percent of women of total participants) ● DR TB representation (minimum of 2) ● From a vulnerable population or hard to reach areas ● Ensure representation of TB Survivors from TUs which has been selected/identified for Community Accountability Framework implementation Table 16: Variation in TBC Selection Criteria State Criteria Variations Jharkhand Age criteria – between 25 yrs. to 45 yrs. Chhattisgarh Age criteria – 18+ yrs. Odisha Inducted NTEP-TBCs for project Others TBCs - Willingness to travel to remote location and possess a two-wheeler driver’s license Tamil Nādu TBS were first trained (TBS to TBC) and then selected based on their interest to become TBCs 74 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov The ALLIES activity targeted 360 TBS to become TBCs using the NTEP curriculum12. The activity has trained a total of 957 TBS to volunteer as TBC, of which 654 have been trained for project intervention locations and rest for non-project districts (for State NTEP). Only about 32 percent of trained TBCs are now actively working for the project. The dropout rate was reported to be insignificant by the IP team. Table 17: TBS Trained and TBCs Deputed States TBS Trained TBC Deputed Jharkhand 82 45 Chhattisgarh 124 37 Odisha 124 48 Tamil Nadu 324 77 Total 654 207 (31.6%) Source: REACH-ALLIES MIS Data ii. Roles and Responsibilities of TBCs: TBCs are part-time volunteers, working for 15 days a month. ALLIES lists eight areas of support which includes emotional/ Psycho-social support, treatment adherence support, contact tracing support, support to link with social schemes, information on nutrition, sputum collection and transport and support to PwTB lost to follow-up13. The following activities are performed by TBCs on a monthly basis to fulfil the expected role: • Follow-up of PwTBs: There are two ways TBCs follow up PwTBs: formally using the CAF tool and informally. The first mode includes a follow-up of purposively sampled eight PwTB using a structured tool. The tool intends to assess gaps in service delivery. However, the opportunity is utilized to counsel PwTB and the family for psychosocial support, cough hygiene, diet, stigma, and drug adherence-related issues. The informal mode is when either the PwTB or ASHA/ other health worker reaches out or STS directs TBC to address an issue or in some cases the TBCs are personally motivated to follow up a PwTB. The support provision is need-based. • Community Meetings: A TBC is mandated to conduct 4-5 community meetings, viz., meetings with elected representatives (mostly PRI and Ward Members), school, colleges, gram sabha, mahila mandals, etc. in a month. The meetings are used as a platform to create general awareness about TB, related stigma, the importance of drug adherence, the need for community support, etc. TBCs use pamphlets and flip-book provided by the project to conduct these meetings. ET observed that these are one-time activities and are not planned based on an identified priority. 12 As per Year-1, Annual Work Plan 13 Performance Indicator Reference Sheet, ALLIES Project USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 75 • Other Responsibilities: Other responsibilities included supporting project and NTEP staff in celebration of important National and International days (e.g., International Woman days, International TB Day, etc.), support in facilitating food packet under Nikshay Mitra scheme for PwTBs, participation in SNC survey and any other task directed by STS in the TU. Dual role of TBC in Odisha: The cadre exists since before ALLIES project. The ALLIES has inducted state NTEP TBCs in the project. Thus, TBCs are full-time and work for both govt. and project. As per state NTEP, the TBCs have a monthly target of sixteen PwTB follow-up, and two awareness generation meetings. Additionally, they are expected to identify presumptive TB cases (20 per month) through door-to-door identification. TBCs also facilitate ADR case referrals. The TBC incentives are based on the targeted activities. However, it was observed that there is an overlap of targets between NTEP and ALLIES which NTEP staff are not aware of. iii. Training: Several training events have been conducted for TBCs (refer to Table 18). The training has been found useful by TBCs especially ‘counselling skills. There is a felt need for refresher training and technical and reference material which can be referred to for knowledge recall. ET observed several instances/ initiatives in Coimbatore (Tamil Nadu), that were found extremely useful by the TBCs (refer to the text box below). • A TBC in Coimbatore said that a Government Doctor on his own initiative has provided training on HIV-TB and Diabetes-TB to the TBCs. This has been very helpful to TBC while counselling the PwTBs. • There is a monthly learning virtual call on topics like basics of TB, comorbidity, nutrition, alcohol de￾addiction, role of TBCs, advocacy etc. The sessions have been reported to be highly useful by the TBCs. iv. Coordination with other services providers: There is a need-based coordination between TBCs and other health workers, e.g., ASHA/ Lady Health Visitor and ANM. TBCs in Odisha reported participating in Sector Meetings along with STS, ASHAs and ANMs. However, since the task (sputum collection, identification of PwTB, DOTS Provision, etc.) is incentive-based, some instances of conflict with ASHAs were also reported Jharkhand and Chhattisgarh). Overall, the mechanism of coordination is not formalized by the project and is need-based. v. Supervision and Reporting: TBCs are supervised by the project’s district lead, i.e., DS, who guides and mentors them on a day-to-day basis. DS coordinates with district NTEP team and facilitates planning, implementation of activities, and addressing gaps. TBCs are also informally supervised by STS especially regarding PwTB follow-up activities. As per need, they conduct joint visits for follow-up and community meetings. A monthly review meeting is conducted to plan for the subsequent month, which is led by DS and includes the participation of the NTEP team. For reporting purposes, the TBC maintains daily diary format for meetings and activities they perform. For CAF-related work, findings on data entry and reports are presented under CAF sub-section. vi. Materials: ET observed IEC pamphlets in Hindi, Odia and Tamil languages being used by TBC for counselling. Pamphlets cover issues like stigma, hygiene, sign and symptoms of TB, etc. A Flip book in different language was also observed with the TBCs. No other materials observed on important issues like diet (when-what), follow-up protocol, ADR, DR-TB, management of comorbidities, gender, Stigma at workplace, etc. Materials on communication skills were shared with ET. However, the ET did not observe it being used at TU/ TBC level Table 18: Trainings Provided to TBCs Trainings TBC Trained Leadership 98 Communication Skill 189 Digital 184 Human rights and advocacy 714 Counselling Skills 118 Livelihoods 227 76 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov ANNEX 5: NON-ALLIES PROJECT DISTRICT TBC MODEL FINDINGS Areas District-wise Brief Description Sahibganj (Jharkhand) Dhamtari (Chhattisgarh) Sundargarh (Odisha) TBC Selection Criteria No standard documented criteria observed Criteria set as per the state government guidelines made in 2020 with REACH support. ▪ Must be willing to work as a ‘TB Mitaan’ and travel within the allocated blocks for at least 6-7 days in a month ▪ Must have been successfully completed the TB treatment ▪ Must be above 18 years of age ▪ Must be a resident of the block from where activities will be pursued ▪ Should be able to read and write local language Standard Selection criteria ▪ Age within 18-50 yrs. ▪ Min. qualification￾Matriculation ▪ Residing in respective block ▪ Fluent in local language ▪ Willingness to travel ▪ Driving license of 2- wheeler TBS to TBC Training in the district Jan 2023 On-job orientation provided by STS and other district NTEP officials March 2022 Roles and Responsibility ▪ District does not have any directions from the state. ▪ TBCs work as needed and support the STS and CHO ▪ State has developed guidelines and broadly defines TBC’s role ▪ The guideline is not available at district level. The district largely follows payment criteria to define TBCs role. The role includes: - Follow-up of 15 PwTB per month. A PwTB to be ▪ No guidance available at the district level ▪ The role reported to be follow-up of 20 PwTB every month ▪ A PWTB to be followed up 4 times during their treatment period. USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 77 Areas District-wise Brief Description Sahibganj (Jharkhand) Dhamtari (Chhattisgarh) Sundargarh (Odisha) followed up twice in a quarter. - No guidance available to prioritize the selection of these PwTBs - Community Meetings for sensitization - Defaulter counselling for lost-to-follow-up PwTB and their families Stipend Rs. 1000 per month (fixed) Rs. 7500 per month, which includes: - ▪ Rs. 400 per PRI meeting (max 8/month) ▪ Rs. 250 per PwTB follow-up ▪ Rs. 250/- Travelling allowance per month Rs. 3500 deliverable based incentive Incentives other than stipend Provisions for: ▪ Treatment Supporter ▪ Sputum Collection and Transportation Provisions for: ▪ Treatment Supporter ▪ Sputum Collection & Transportation Provision for ▪ Treatment Supporter ▪ Sputum Collection and Transportation IEC Materials No job aid or any communication materials available with the TBCs No job aid or any communication materials available with the TBCs No job aid or any communication materials provided. Monitoring and Supervision STS supervise the TBCs, No guidelines available STS supervises TBC daily/ need basis and has meeting once in a month; List of PwTB covered by TBCs is checked by STS & given to account section for release of stipend STS unclear on the deliverables hence unable to supervise 78 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov ANNEX 6: CAF TOOL USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 79 80 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 81 82 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 83 84 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 85 86 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 87 88 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 89 90 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov ANNEX 7: SURVIVOR LED NETWORKS’ MEMBERS TRAININGS # Trainings for SLN Members Key Training Content 1 Leadership and organizational development Book-keeping/accounting, fund-raising 2 Communication Skilling Communication with stakeholders e.g., providers, Products for communication like quarterly TBS’s newsletter 3 Counselling skills for TBCs To provide psychosocial support and support for treatment adherence 4 Livelihood generation Making slippers, LED bulb, soap, floor disinfectant, mops, candle, badi (dried lentil dumplings). Mushroom cultivation USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 91 ANNEX 8: DISCREPANCY IN TARGETS Y3 Y3 Target as per Y3Q4 QPR Target for Y3Q4 Difference from proposed in Q3 Achieve ments of the quarter Checking Cumulative in Y3Q4 Cumulativ e Achievem ents (yearly) Target of Y3Q3 Difference from proposed in Q2 Achieve ments of the quarter Cumulat ive Achieve ments (yearly) Checkin g Cumulat ive in Y3Q3 Target for the next quarter Target (yearly) Target for Y3Q2 Difference from proposed in Q1 Achiev ements of the quarter Checking cumulative inY3Q2 Cumul ative Achiev ements (Yearly The target for the next quarter Additio nal Inform ation Target (yearly) Target of the quarter Achiev ements of the quarter Cumul ative Achiev ements (Yearly The target for the next quarter Number of TB Survivors trained on Rights-based approach to TB and Health curriculum 443 104 18 114 435 435 111 2 38 321 321 122 443 113 37 192 283 283 113 This quarter 443 113 91 91 150 Number of New District level TB Survivor led Network Chapters formed 6 - - 3 20 20 2 - 9 17 17 - 6 2 -2 1 8 8 2 The district 6 2 7 7 - Number of new State level TB Survivor led Networks formed 5 1 2 - 2 2 1 1 - 2 2 3 5 2 -2 - 2 2 2 5 - 2 2 - Number of opportunities facilitated for TB Champions 44 11 -3 13 49 49 11 - 14 36 36 8 44 11 1 15 22 22 11 The opportu 44 11 7 7 12 Number of health care workers that successfully complete an in-service training program within the reporting period with 327 91 23 175 388 388 82 10 135 213 213 114 367 92 33 78 78 78 92 AETBCS 367 92 - - 125 Number of TB Champions that complete CAF training 28 - 16 35 47 47 7 -7 12 12 12 16 28 - - - - - - We had complet 28 28 - - - Number of Community Accountability Framework Reports generated in a quarter 60 15 1 15 52 37 15 - 16 37 33 16 60 15 1 21 37 17 15 The CAFRS 60 15 16 16 16 Number of Health Facilities offering TB services who are initiated on the Community Accountability Framework 60 15 29 75 91 16 15 - 29 16 29 44 60 15 1 - 16 15 60 15 16 16 16 Number of Block Action Plans Generated in a year 1,410 360 -10 420 1,536 1,536 353 7 393 1,116 1,116 350 1,410 360 - 389 723 723 360 The project 1,410 330 334 334 360 Number of state level media round tables organized 4 4 -3 2 9 9 1 -1 5 7 7 1 8 4 - 2 2 2 - This quarter 8 X - - 4 Number of TB specific stories by fellows 90 45 -40 - 48 48 23 -23 - 48 48 5 90 45 - 38 48 48 - 15 media 90 X 10 10 45 Number of new companies that have joined the employer led model in TB Care. 27 9 11 29 31 31 7 2 1 2 2 20 27 9 - 1 1 1 9 ACC cement 27 X - - 9 Number of Elected representatives newly engaged by the ALLIES project 10 5 -5 153 338 338 3 -3 88 185 184 - 10 5 9 52 96 96 - Details of 10 X 44 44 14 Number of innovations supported through USG assistance 4 2 2 - - - 1 -1 - - - 4 4 2 -2 - - - - 4 X - - - Number of laws, policies, regulations, or guidelines, developed /implemented/adapted with USG support 2 1 1 - - - 1 - - - - 2 2 - - - - - 1 2 - - - - Number of people benefited either through services (screening, counselling, referral & providing services) or 21,600 1,800 3,600 34,563 101,521 101,521 5,400 -3,600 17,021 66,958 66,958 5,400 21,600 1,800 - 45,000 49,937 49,937 1,800 These numbers 21,600 1,800 4,937 4,937 1,800 Y3Q4 Y3Q3 Y3Q2 Y3Q1 Indicators as per PMP 92 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov ANNEX 9: ALLIES LOP TARGET ANALYSIS ALLIES has set LOP targets against defined PMP indicators to measure and monitor its performance throughout the life cycle of the project. The indicators and its LOP targets are defined in two of the approved MEL plans (2020 and Year 4) and in projects yearly AWP. The table below presents the indicators and LOP targets against each across MEL plans and AWP document followed by an observation on target variations and other issues. ET has not considered AWP for Year 1 (Y1) in considering the first year as project initiation and setting up phase. # PMP Indicators Baseline LOP Targets AWP (Y4) & MEL Y4 MEL (2020) AWP (Y2) AWP (Y3) 1 Number of TB Survivors trained on Rights￾based approach to TB and Health curriculum 0 925 500 500 625 2 Number of New District level TB Survivor led Network Chapters formed 0 35 24 12 15 3 Number of new State level TB Survivor led Networks formed 0 6 -- 5 6 4 Number of opportunities facilitated for TB Champions 0 125 48 48 60 5 Number of health care workers that successfully complete an in-service training program within the reporting period with United States Government (USG) support 0 725 500 500 625 6 No of TBS Trained to TBC using NTEP curriculum 0 -- -- 360 -- 7 Number of TB Champions that complete CAF training 0 150 120 120 150 8 Number of Community Accountability Framework Reports generated in a quarter 0 228 216 216 72 9 Number of Health Facilities offering TB services who are initiated on the Community Accountability Framework 0 60 0 48 72 10 Number of Block Action Plans Generated in a year 0 2130 -- -- 2130 11 Number of state level media round tables organized 0 12 8 8 8 USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 93 # PMP Indicators Baseline LOP Targets AWP (Y4) & MEL Y4 MEL (2020) AWP (Y2) AWP (Y3) 12 Number of TB specific stories by fellows 0 180 144 144 180 13 Number of new companies that have joined the employer led model in TB Care. 0 40 80 40 40 14 Number of people sensitized on TB through Employee Led Model Advocacy -- 40,000 -- -- -- 15 Number of Elected representatives newly engaged by the ALLIES project 0 207 12 12 15 16 Number of innovations supported through USG assistance 0 5 4 4 5 17 Number of laws, policies, regulations, or guidelines, developed /implemented/adapted with USG support 0 5 4 4 5 18 Number of people benefited either through services (screening, counselling, referral & providing services) or communication activities 0 1,28,714 20% of the total pop of the blocks assigned to TUs 95,805 30,000 Baseline Values: The baseline values of all the indicators are zero as per project’s MEL Plan which implies the interventions are new in the project location. However, REACH implemented C2A project (2016-2020) in nine of the ALLIES project districts across the four states and many of its interventions were a continuation. For example, in case of PMP indicator ‘Number of new State level TB SLNs formed’, three of the four networks in ALLIES were already in place since C2A project. Similarly, REACH has implemented TBC Model, ER sensitization and Engagement, Employers Engagement, Media Engagement, etc. since from the C2A project. For ER and Employers, the indicator statement implies ‘new engagement’, but continuation of previous engagement should have also been considered. Variation in targets: The LOP targets have varied for same indicators across all the project’s plan documents. For some of the indicators the change is significant from the most recent target plan i.e., MEL Y4. 94 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov PMP Indicator % change from MEL 2020 % change from AWP Y3 Number of TB Survivors trained on Rights-based approach to TB and Health curriculum 85% 48% Number of New District level TB Survivor led Network Chapters formed 46% 133% Number of opportunities facilitated for TB Champions 160% 108% Number of Elected representatives newly engaged by the ALLIES project 1625% 1280% Number of Community Accountability Framework Reports generated in a quarter 6% 217% Number of TB Survivors trained on Rights-based approach to TB and Health curriculum – The MEL Y4 Change Log explains the revision of target as previous LOP Target was achieved by Y3. However, no clarification is reported regarding the basis of significant increase of the LOP target. ‘Number of New District level TB Survivor led Network Chapters formed’, - the LOP target in Y2 was 12 as the project was being implemented in 12 districts across four states. The target was changes to 15 in the Y3 as three mode districts were added in the project in Tamil Nadu. The Change Log in MEL Y4 explains inclusion of network chapters at Touched By TB in other states, however, as per the SOW of Touched by TB intervention, 10 district networks are to be formed, however, the target of 35 by Y4 is still not clear. ‘Number of opportunities facilitated for TB Champions’ - The MEL Y4 Change Log explains the revision of target as previous LOP Target was achieved by Y3. However, no clarification is reported regarding the basis of significant increase of the LOP target. ‘Number of Elected representatives newly engaged by the ALLIES project’ – there is a significant increase in the LOP targets by Y4 (the LOP target for Y2 and Y3 is 12 and 15 which is increased to 207 in Y4). As per the Change Log, the reason for increase is due to inclusion of three new districts in Tamil Nadu. The basis of this increase is not reported. ‘Number of people sensitized on TB through Employee Led Model Advocacy’- it’s unclear that how the target was set to reaching 40,000 people and per change log of ALLIES Y4 MEL Plan the QPR Y1Q4 the project discontinued reporting on this indicator from Y1Q4, after a strategic meeting with USAID on the ALLIES way forward in ELM. The reason for restructuring and change in the ELM strategy is not clarified. ‘Number of Community Accountability Framework Reports generated in a quarter’ – The Change Log (MEL Y4), explains CAFR to be a quarterly activity instead of monthly. However, the basis of significant increase of target is unclear. Other LOP target Variation: ‘Number of people benefited either through services (screening, counselling, referral & providing services) or communication activities’- As per the MEL 2020 plan the LOP target is defined as 20% of the total population of the blocks assigned to TU, later in Y3, the target was reduced to 95,805 and further in Y3, significantly reduced to 30,000. No clarity on reduction is explained in any plan document. The final target mentioned in Y4 is at 1,28,715. The Change Log (MEL Y4) provides explanation, as number of people benefited to be the 8 PwTB reached and 2 of their family members USAID.gov USAID ALLIES Project Mid-Term Performance Evaluation | 95 reached by individual TBC during CAF only. The definition of ‘people benefited’ is limited to CAF which is unclear, as TBC role is envisaged to be beyond CAF. Considering the project coverage is more than 150 TUs across four states (and as norm TU has a catchment area of about 1,00,000 population). Thus, the LOP target is way below 1% of the population. 96 | USAID ALLIES Project Mid-Term Performance Evaluation USAID.gov ANNEX 10: SUGGESTIONS - ALLIES TBC’S ROLES Considering TBCs are a part-time cadre who serve as peer-support group to the PwTBs and are also the bridge between the health system and community, the following table broadly proposes the activities a TBC should be expected to perform. The TBC should coordinate and collaborate with other health workers STS, CHO, ANM, ASHA, AWW, etc. in execution of the activities. The ALLIES activity should strategize on mechanisms for this collaboration. Core Activities Extended Activities Special Role 1. Presumptive TB identification and ensuring test and treat initiation 2. Psych-social support to PwTB (this may need to be done followed up periodically) 3. Counselling of PwTB on various aspects of TB (ADR, nutrition and diet plan, comorbidities, hygiene, treatment adherence, follow-up protocol, emergency referral, etc.) – this need to follow up periodically 4. Family Counselling on care giving, providing moral and emotional support, nutrition and diet, adherence, ADR, contact screening, etc.)- this may need to be followed up periodically 5. Identification of lost-to-follow-up cases and support to re-initiate treatment 6. Community awareness on TB, addressing myth and misconceptions, support to PwTB in the area 7. Addressing stigma and discrimination in the family, community and work places through counselling and other communication activities. 8. Regular feedback to health system on the issues and challenges 9. Support PwTB and family accessing national and state social welfare schemes – NPY, pensions, livelihood, etc. 10. Community feedback to sensitize health worker on patient centric service provision (for improved QoC) 11. Engage local leaders, religious leaders, youth groups, SHGs, teachers, PRIs etc. to support and advocate for PwTB. 12. Active participation in District-TB forum 1. Sputum collection and transportation 2. Treatment supporter 3. Food distribution to PwTBs and their families 4. Participation in active case finding drives 5. Linking PwTBs with Survivor Network District Chapter for additional support 1. Periodic CAF survey for gap identification and feedback to health system 2. Participation in periodic celebration of important national and international days 3. Active participation in state/ district Survivor Network and support improving TB services through advocacy and other activities.