ERADICATING TUBERCULOSIS ACTIVITY (ETB) END OF PROJECT PROCESS EVALUATION Final Report Contract Number: Submission Date: 72067419D00007/72061120N00001 February 14, 2023 Submitted By: Margaret Lada, Chief of Party SoCha LLC Subdivision 694/ Stand 100, Ibex Hill Rd. Lusaka, Zambia Tel: +260 96 7984375 This publication was produced at the request of the United States Agency for International Development (USAID). It was prepared independently by the USAID/Zambia Monitoring, Evaluation and Learning Platform, implemented by SoCha, LLC. The contents are the responsibility of SoCha, LLC., and do not necessarily reflect the views of USAID or the U.S. Government 1 EXECUTIVE SUMMARY This report presents the main findings, conclusions, and recommendations emerging from an end-of￾project process evaluation of USAID/Zambia’s Eradicate Tuberculosis (ETB) activity. The evaluation was implemented by the USAID/Zambia Monitoring, Evaluation, and Learning Platform (Z-MELP) from August to November 2022. The evaluation addressed four questions: 1) What lessons learned and/or best practices should be taken from ETB to inform the implementation of other (existing) TB-focused activities and the design of new activities focused on TB eradication? 2) Were the ETB staffing and partnership structures optimal for achieving project objectives? 3) To what extent are ETB methodologies, interventions, and management setting the stage for future sustainability of project outputs and outcomes? 4) What institutional capacity has ETB built within the MOH and other stakeholders (private sector, Zambian research institutions) to generate sustainability for the processes and outputs achieved? METHODS This end-of-project process evaluation of ETB involved the collection and analysis of both primary and secondary data. The latter were sourced from an extensive document review, while the former were collected through three qualitative data collection methods: 1) Key Informant Interviews (KIIs) (41 KIIs in total) with individuals who had specialized activity or contextual knowledge that was essential to answering the evaluation questions. 2) Focus Group Discussions (FGDs) (6 FGDs, 22 participants) with CBVs (TB Treatment Supporters) who were the focus of ETB’s community-level efforts to improve TB case identification and support for TB patients. 3) A limited amount of semi-structured observation was undertaken at six district-level health volume facilities to observe the quality of ETB’s capacity strengthening support to laboratory technicians, TB focal points, and district TB coordinators. The evaluation team agreed with ETB and USAID/Zambia to focus the evaluation on three of the six provinces where ETB was implemented. Within each province, three districts were purposively chosen as locations for data collection. Within each district, the facilities to visit for observation, the facility staff to be interviewed, and the CBVs to be included in FGDs were determined purposively, based on discussions with ETB, USAID/Zambia, and those national-level MOH/NTLP contacts the evaluation had engaged prior to finalizing its district data collection plans. At each facility visited, the evaluation prioritized KIIs with TB focal points and their immediate clinical support staff. EVALUATION QUESTION #1: WHAT LESSONS LEARNED AND/OR BEST PRACTICES SHOULD BE TAKEN FROM ETB TO INFORM THE IMPLEMENTATION OF OTHER (EXISTING) TB-FOCUSED ACTIVITIES AND THE DESIGN OF NEW ACTIVITIES FOCUSED ON TB ERADICATION? The evaluation identified great value in several the seemingly simple, yet vital, reforms that ETB facilitated. A number of these should be seized upon as best practices for ongoing and future USAID/Zambia TB programming: • The importance of adopting a decentralized zonal-focused model of capacity strengthening for district-level TB clinicians. 2 • A holistic approach to TB case identification, in which site activation places a responsibility on all health care workers in a facility – and all facility departments – to screen patients for TB. • The improved use of simple information communications technologies (SMS, WhatsApp) to improve the flow of information between clinicians at facilities (often provincial) where patients are diagnosed and facilities (district-level) where many patients will seek subsequent care. • The value of opportunistically “piggybacking” on existing HIV partners’ sample courier systems to ensure the timely delivery of sputum samples for quicker TB case identification. The document review and KIIs identified these four reforms as being among the most significant contributors to improved TB case identification (including among children) and patient care. Some of these reforms, such as site activation, are already becoming institutionalized within the NTLP’s new NSP covering the period 2022 to 2026. Others will require ongoing efforts, including through new USAID TB programming, to institutionalize and sustain. Also important to consider for future activities is the important role that ETB played as a convening force within the wider TB ecosystem. ETB provided considerable support to the NTLP in convening both the Technical Working Group (TWG) on TB and in fostering the creation of provincial TB elimination committees. These groups, in turn, have emerged as significant forums for dialogue on how best to identify and remove roadblocks to TB elimination. In the absence of ETB playing this convening role, it cannot be taken for granted that these groups will survive. Ongoing and future USAID/Zambia TB programs may need to identify their own potential entry points to providing this same convening assistance. Finally, ETB’s facilitation of virtual “situation rooms” – an unintended outcome of COVID-19 – to foster discussion of TB data trends and associated action points, is highly regarded by the NTLP and by USAID/non-USAID implementors of TB activities. The “situation rooms” seem to have unblocked a significant barrier to data-driven decision-making around TB by allowing for real-time reflections on available evidence from multiple sources. These innovations should continue and become institutionalized by the NTLP. Where lessons learned provide opportunities for reflection and improvement, the following conclusions emerged from the evaluation: • Partnerships cannot be established ad hoc but must be planned for from the outset of activity implementation. This is true of private sector partnerships, which ETB did not foster to the extent intended. However, it is also true of partnerships with the GRZ, where ETB’s efforts to engage the Parliamentary Caucus on TB, while by no means ineffective, have produced few tangible gains. • Any activity that requires the collection and reporting of data by health facilities must ensure that these facilities have the computers (or tablets) and connectivity required to play this role. ETB’s notable efforts to capacitate personnel at district-level facilities to collect quality data were somewhat undermined by the activity’s failure to ensure that facilities were equipped with the computers/tablets and internet connectivity required to submit these data in a timely manner. Moving forward, any TB-focused activity that relies on facilities to provide most data need to consider a cost-effective way to ensure that facilities have the digital infrastructure required to realistically play this role. This could include training facilities on using smart phones and relevant apps for data collection, as well as ensuring that provision exists for offline reporting. 3 EVALUATION QUESTION #2: WERE ETB STAFFING AND PARTNERSHIP STRUCTURES OPTIMAL FOR ACHIEVING PROJECT OBJECTIVES? ETB’s staffing structure was broadly, though not entirely, fit for purpose. The activity experienced significant turnover in its senior leadership – having in place two Chiefs of Party and three Deputy Chiefs of Party over the course of implementation. This created challenges in terms of the continuity of technical leadership and the generation of institutional knowledge, though the evaluation did not discern a negative impact on activity performance caused by this turnover. The partnership between PATH and its consortium partners was deemed to be strong, with the embedding of partner organization staff within the PATH office seen as helping to ensure effective awareness and collaboration.1 ETB represented a new approach by a USAID/Zambia activity to embed field technical staff directly with the MOH, primarily in PHOs. This engagement with the MOH was viewed as a success, not least by the Ministry itself, which valued day-to-day exposure to ETB activities and the opportunities for learning and capacitation that this provided. For ETB, a close working relationship with the MOH was productive yet challenging. Important relationships were established and MOH technical capacity built. However, near constant turnover in MOH staff (including facility in-charges and provincial and district-level TB coordinators) created problems with technical continuity and placed a burden on ETB staff to provide almost non-stop orientations and repeated technical trainings to new or relocated MOH personnel. It is to ETB’s credit that the activity took steps to address this need by standardizing as much of this orientation/training content as possible so as to ensure consistency and reduce the time demand placed on staff to devise orientation/training materials. Where ETB’s staffing structure was inadequate, it was for the following reasons: • Sufficient human resources were not allocated to the district-level. ETB placed a limited compliment of staff (clinician, M&E staff, driver) at PHOs, but these staff could not provide timely technical assistance or data collection to all districts given the expansive geography they had to cover. • ETB staff working with district TB coordinators and other facility staff did not always have the competencies required to provide needed capacity strengthening support. When ETB staff lacked skillsets that were fundamentally different from the facility staff they were engaging with, they were unable to troubleshoot and often replicated coordination functions already being played by the facility staff instead of providing clear technical leadership on activity implementation. • The activity’s M&E structure was too lean. In addition, there was no Knowledge Management lead to ensure the documentation and dissemination of successes and lessons learned. Too few M&E staff created challenges with ensuring the timely collection and reporting of data (though the project was broadly successful in ensuring that reports met USAID deadlines). The lack of a Knowledge Management function created challenges in ensuring the effective utilization of institutional memory. From a partnerships standpoint, the fact that ETB did not develop, from the outset of implementation, a clear strategy for engaging with the private sector, meant that deep partnerships were not established. Private sector support was provided to one-off events (e.g., Barclays funding of a World TB Day event) but private investments with a long-term commitment to the TB space were not forthcoming. ETB’s creation of a clear private sector engagement strategy towards the end of implementation does provide a foundation on which future TB activities can build, provided such a strategy is effectively operationalized. 1 Source: KII with with two members of ETB senior leadership; KII with USAID senior leadership. 4 EVALUATION QUESTION 3: TO WHAT EXTENT ARE ETB METHODOLOGIES, INTERVENTIONS, AND MANAGEMENT SETTING THE STAGE FOR FUTURE SUSTAINABILITY OF PROJECT OUTPUTS AND OUTCOMES? The long-term sustainability of many of ETB’s components remains open to debate. However, two components have gained traction and have been (or are seemingly likely to be) institutionalized within MOH/NTLP policy and practice: • Site activation – the demand for an “all hands-on deck” approach to TB screening at health facilities. By ensuring comprehensive screening of all patients, in all facility departments, and at all points of entry, the improved rates of TB screening documented by ETB should be maintained. • Weekly TB Situation Room with the NTLP – a broad consensus exists on the value of having a collaborative forum where the NTLP, implementors, and others in the TB space can regularly engage, peer review existing data, and critically reflect on evident trends to inform policy and practice. Where sustainability may be tougher to sustain, it is in relation to: • The technical capacitation of district-level facility personnel – the high rates of attrition among district health care workers, as well as the high likelihood of personnel being transferred, makes it difficult for health facilities to maintain institutional memory and awareness of “best practices” in TB screening, diagnosis, and treatment that were imparted by ETB. • The CBV model – as with any volunteer-focused approach, ensuring the long-term incentivization of CBVs, in the absence of regular USAID activity engagement, is a challenge. While enthusiasm has been generated within the MOH/NTLP for maintaining and even upscaling the CBV model to ensure improved patient care, it is not clear that a clear strategy exists for CBV retention or for the training of new CBVs – though the MOH should have such a resource that future USAID activities can draw on. EVALUATION QUESTION 4: WHAT INSTITUTIONAL CAPACITY HAS ETB BUILT WITHIN THE MOH AND OTHER STAKEHOLDERS TO GENERATE SUSTAINABILITY FOR THE PROCESSES AND OUTPUTS ACHIEVED? ETB contributed to the building of institutional capacity within the MOH, albeit to varying degrees and with different likelihoods of long-term sustainability. As the activity concludes, there are several areas for which capacity seems to have been effectively developed. The unknown factor is whether the attrition of health care workers at district-level facilities will, over time, dilute the institutional knowledge built up by the activity. At present, key capacity gains identified by the evaluation are: • Improved knowledge and practice across an array of facility health care workers on how to screen patients for TB. Site activation should be a lasting institutional practice to emerge from ETB. Rather than focusing solely on capacitating TB clinicians, the activity worked to sensitize health care workers in all facility departments on how to undertake screening. This should continue allowing for increased screening and case identification. • Improved knowledge on the use of the GeneXpert diagnostic tool. District and provincial laboratory staff were given intensive training by ETB on how to utilize and maintain the GeneXpert machines for improved case diagnosis. 5 • Greater capacity within the MOH/NTLP – particularly at national and provincial levels – to understand, discuss, and utilize data in decision-making. The national TB TWG and weekly NTLP Situation Room are at the forefront of this improved capacity. RECOMMENDATIONS For USAID/Zambia • Future TB activities should continue dedicating resources to the expansion of provincial TB elimination committees. • Ongoing and future TB activities (and other Health activities) should adopt/ continue utilizing a zonal approach to capacity strengthening. • Ongoing and future USAID/Zambia TB activities – as well as USAID/Zambia Health activities in the HIV/AIDS space – should continue co-investing in the maintenance of the integrated courier system for sample collection and delivery. • Future USAID/Zambia TB activities should develop private sector engagement strategies at the outset of implementation. • Future USAID/Zambia TB activities should ensure the adequate allocation of sufficiently trained activity technical staff at district-level. • Future USAID/Zambia TB activities should consider the professional profiles and experience levels of sub-national staff assigned to work with the MOH at district and facility-levels. • Future USAID/Zambia TB activities should allocate more staffing resources towards M&E and should put in place a Knowledge Management lead. For the MOH/NTLP • The MOH should continue emphasizing site activation as a central component of its approach to increasing TB screening. This means establishing clear expectations for health facilities and putting in place a monitoring mechanism to ensure that facilities transition to a fully site activation approach. • The MOH should ensure that if district-level health facilities are going to be at the forefront of data collection and reporting, and if public health policy is to be made based on these data, that facilities are equipped with the resources they require to play this role. 6 ACRONYMS ADS Automated Directives System ART Antiretroviral Therapy CBV Community-Based Volunteer CIDRZ Centre for Infectious Disease Research in Zambia CITAM+ Community Initiative for Tuberculosis, HIV/AIDS, and Malaria Plus Related Diseases CLA Collaborating, Learning, and Adapting DOTS Directly Observed Therapy, Short-Course DR-TB Drug-Resistant Tuberculosis ETB Eradicate Tuberculosis Activity FGD Focus Group Discussion FY Fiscal Year GRZ Government of the Republic of Zambia HIV Human Immunodeficiency Virus HMIS Health Management Information System ICF Intensified Case Finding IP Implementing Partner IRB Institutional Review Board KI Key Informant KII Key Informant Interview LAM Lipoarabinomannan Assay LOP Life of Project M&E Monitoring and Evaluation MDR-TB Multi-Drug-Resistant Tuberculosis MOH Ministry of Health MTEF Medium-term Expenditure Framework NSP National Strategic Plan for Tuberculosis Prevention, Care, and Control NTLP National Tuberculosis and Leprosy Program NTRL National Tuberculosis Reference Laboratory OECD Organisation for Economic Co-operation and Development OR Operational Research 7 PHO Provincial Health Office PLHIV People Living with HIV PSE Private Sector Engagement TB Tuberculosis TOC Theory of Change TWG Technical Working Group UNAIDS Joint United Nations Programme on HIV/AIDS USAID United States Agency for International Development USG United States Government WHO World Health Organization XDR-TB Extensively Drug-Resistant Tuberculosis ZACCI Zambia Association of Chambers of Commerce and Industry ZAM Zambia Association of Manufacturers ZAMBART Zambia AIDS Related Tuberculosis Project Z-MELP USAID/Zambia Monitoring, Evaluation & Learning Platform ZNFU Zambia National Farmers Union 8 TABLE OF CONTENTS ACRONYMS.............................................................................................................................6 TABLE OF CONTENTS .........................................................................................................8 1. EVALUATION PURPOSE & QUESTIONS...................................................................9 2. PROJECT BACKGROUND .............................................................................................9 3. METHODS AND LIMITATIONS..................................................................................13 4. FINDINGS & CONCLUSIONS .....................................................................................16 4.1 EVALUATION QUESTION 1: WHAT LESSONS LEARNED AND/OR BEST PRACTICES SHOULD BE TAKEN FROM ETB TO INFORM THE IMPLEMENTATION OF OTHER (EXISTING) TB-FOCUSED ACTIVITIES AND THE DESIGN OF NEW ACTIVITIES FOCUSED ON TB ERADICATION? ............................................................16 4.1.1 SUMMARY RESPONSE TO EVALUATION QUESTION 1........................................................16 4.1.2 SUCCESSES .............................................................................................................................................17 4.1.3 CHALLENGES ........................................................................................................................................22 4.2 EVALUATION QUESTION 2: WERE ETB STAFFING AND PARTNERSHIP STRUCTURES OPTIMAL FOR ACHIEVING PROJECT OBJECTIVES?.........................23 4.2.1 SUMMARY RESPONSE TO EVALUATION QUESTION 2........................................................23 4.2.2 SUCCESSES .............................................................................................................................................24 4.2.3 CHALLENGES ........................................................................................................................................25 4.3 EVALUATION QUESTION 3: TO WHAT EXTENT ARE ETB METHODOLOGIES, INTERVENTIONS, AND MANAGEMENT SETTING THE STAGE FOR FUTURE SUSTAINABILITY OF PROJECT OUTPUTS AND OUTCOMES? .................................27 4.3.1 SUMMARY RESPONSE TO EVALUATION QUESTION 3........................................................27 4.3.2 SUCCESSES .............................................................................................................................................28 4.3.3 CHALLENGES ........................................................................................................................................29 4.4 EVALUATION QUESTION 4: WHAT INSTITUTIONAL CAPACITY HAS ETB BUILT WITHIN THE MOH AND OTHER STAKEHOLDERS TO GENERATE SUSTAINABILITY FOR THE PROCESSES AND OUTPUTS ACHIEVED?...................31 4.4.1 SUMMARY RESPONSE TO EVALUATION QUESTION 4........................................................31 5. RECOMMENDATIONS .................................................................................................31 ANNEX 1: EVALUATION STATEMENT OF WORK.......................................................34 ANNEX 2: EVALUATION FRAMEWORK .........................................................................44 ANNEX 3: DATA COLLECTION TOOLS .........................................................................53 9 1. EVALUATION PURPOSE & QUESTIONS This report presents the main findings, conclusions, and recommendations emerging from an end-of￾project process evaluation of USAID/Zambia’s Eradicate Tuberculosis (ETB) activity. The evaluation was implemented by the USAID/Zambia Monitoring, Evaluation, and Learning Platform (Z-MELP) from August to November 2022. The evaluation, required under Automated Directives System (ADS) 201, serves USAID/Zambia and its implementing partners (IPs) by informing the continuation of interventions under an existing Tuberculosis (TB) activity, as well as informing the design of a new activity, scaled-up to all provinces, with the goal of eradicating TB. USAID/Zambia will share the evaluation publicly to account for the public funding used by the activity. The Zambian Ministry of Health (MOH), and particularly the National Tuberculosis and Leprosy Program (NTLP), are also expected to use the evaluation findings to inform long-term policy change. The evaluation addressed four questions: 5) What lessons learned and/or best practices should be taken from ETB to inform the implementation of other (existing) TB-focused activities and the design of new activities focused on TB eradication? 6) Were the ETB staffing and partnership structures optimal for achieving project objectives? 7) To what extent are ETB methodologies, interventions, and management setting the stage for future sustainability of project outputs and outcomes? 8) What institutional capacity has ETB built within the MOH and other stakeholders (private sector, Zambian research institutions) to generate sustainability for the processes and outputs achieved? To the extent possible, the evaluation answered these questions with consideration of three cross￾cutting issues: 1) equity (including gender); 2) localization; and 3) anti-corruption. 2. PROJECT BACKGROUND2 USAID/Zambia’s ETB activity, implemented by PATH alongside three local organizations – Afya Mzuri, the Community Initiative for Tuberculosis, HIV/AIDS, and Malaria Plus Related Diseases (CITAM+), and the Zambia AIDS Related Tuberculosis Project (ZAMBART) – was implemented from May 2017 to September 2022. Working in partnership with the MOH’s NTLP, the private sector, and other partners at the community, facility, district, provincial, and national levels, ETB sought to strengthen the implementation of key components of the national response to control TB across Zambia. Present in six provinces – Central, Copperbelt, Luapula, Muchinga, Northern, and North-Western – and covering 63 districts and 341 health facilities, ETB’s goal was to reduce TB-related mortality by 50 percent from an established 2016 baseline of 5 percent. To meet this goal, ETB pursued two objectives, both of which were to strengthen the NTLP’s efforts to achieve TB elimination by 2030: • Objective 1: Increase the number of individuals screened for TB in the target provinces by 50 percent as compared to 2016 baseline levels. • Objective 2: Increase the TB treatment success rate to at least 85 percent in the target provinces. The rationale for ETB was rooted in Zambia’s high rate of TB infection and TB/Human Immunodeficiency Virus (HIV) co-infection. According to the World Health Organization (WHO)’s 2019 Global Tuberculosis Report, Zambia ranks among the top thirty countries worldwide on both measures. The WHO estimated Zambia’s TB incidence in 2019 to be 333 per 100,000 people (equal to 60,000 TB patients). Meanwhile, the WHO’s 2019 data indicated that TB mortality was 33 per 2 The information in this section is taken from the ETB Annual Progress Report (FY 2021), as well as the project’s Q2 and Q3 Quarterly Progress Reports (FY 2022). The evaluation team was unable to review the activity proposal, which was not held on file by any of the study’s Key Informants. 10 100,000 in HIV-negative patients and 53 per 100,000 in HIV-positive TB patients. Although Zambia is still considered a low-burden country for drug-resistant TB (DR-TB), in 2019 the country recorded its first extensively drug-resistant (XDR-TB) TB patient, as well as 515 confirmed DR-TB patients, of whom 97 percent initiated treatment. There have been successes in Zambia around TB outcomes. According to WHO (2019), the TB treatment success rate at the national level is 90 percent. Most TB patients know their HIV status and 97 percent of co-infected patients start antiretroviral therapy (ART). The treatment success rate for DR-TB patients has increased – from 71 percent in 2018 to 76 percent in 2019. Nevertheless, the NTLP has outlined the urgency of a continued focus on national TB control, stating in its National Strategic Plan (NSP) for Tuberculosis Prevention, Care, and Control, an objective of moving “towards elimination”. The NSP takes a pro-active approach, also applied by ETB, to identifying missing TB cases. This marks a move away from the passive case-finding approach that was previously applied. ETB THEORY OF CHANGE & APPROACH ETB’s Theory of Change (TOC), represented in Figure 1, held that to contribute to the elimination of TB in Zambia through increased screening and improved treatment, it is necessary to strengthen collaboration and coordination, as well as harness existing technical skills among Government of the Republic of Zambia (GRZ) stakeholders, United States Government (USG) partners and other donors, the private sector, civil society, medical professionals, and community leaders and volunteers. Doing so would maximize investments, promote innovation, and produce feasible, gender-equitable, and evidence-based interventions. These interventions would focus on finding missing cases, improving diagnostic networks, and improving outcomes for all persons with all forms of TB – with a specific focus on populations with high TB burdens, including underserved communities, People Living with HIV (PLHIV), children, and people in congregate settings such as prisons. To increase the number of individuals screened by TB by 50 percent from 2016 baseline levels, ETB focused on finding missing cases through improving diagnostic networks, expanding active and targeted community- and facility-based case finding, supporting the strengthening of NTLP’s Monitoring and Evaluation (M&E) systems, and facilitating multi-sector involvement in TB control. This was done through three tasks: • Task 1: Pursue high-quality directly observed therapy, short-course (DOTS) expansion and enhancement. • Task 2: Increase access to TB/HIV and Multidrug-resistant-TB (MDR-TB) services for poor and vulnerable populations in target provinces. • Task 3: Engage all categories of care providers to improve the quality and outreach of TB screening and treatment. To increase the TB treatment success rate to at least 85 percent in all target provinces, ETB focused on standardizing treatment, support, and care; increasing capacity for treatment of drug-susceptible and drug-resistant TB and TB/HIV co-morbidity; improving the effectiveness and quality of pediatric TB services; and supporting all categories of care providers to adhere to national guidelines. This was done through four tasks: • Task 1: Pursue high-quality DOTS expansion and enhancement. • Task 2: Increase access to TB/HIV and MDR-TB services for poor and vulnerable populations in target provinces. • Task 3: Engage all categories of care providers. 11 • Task 4: Intensify research and innovation. The purpose of this evaluation is not to assess ETB’s performance against targets. Indicator-specific performance data were validated with USAID/Zambia in the activity’s July 2022 Project Performance Review. However, before delving into the four evaluation questions that are the focus of this study, it is worth providing a summary of ETB’s achievements3: • Over 630,000 individuals tested for TB in a laboratory setting –178 percent of the life-of￾project (LOP) target and exceeding the objective of increasing the number of individuals screened for TB in the target provinces by 50 percent as compared to 2016 baseline levels. • Over 89,000 individuals notified of their TB status – 103 percent of the LOP target. • 830 individuals notified of their DR-TB status – 53 percent of the LOP target and the only indicator for which ETB significantly fell short of its objective. • A TB treatment success rate of 92 percent – exceeding the objective of increasing the TB treatment success rate to at least 85 percent in the target provinces. • 98 percent of TB patients in supported facilities tested for HIV – just under the LOP target of 100 percent. • A TB notification rate among children of 7.2 percent – below the LOP target of 10 percent and well-below the recommended WHO standard of at least 15 percent. • A TB mortality rate of 2.7 percent – just above the LOP target of reducing the TB mortality rate to 2.5 percent. ETB PARTNERSHIP ROLES & RESPONSIBILITIES The roles and responsibilities of each member of ETB’s implementing consortium were: • PATH provided overall management, financial, and technical oversight in addition to M&E and strategic information support. PATH led corporate engagement, gender-related, public-private mix, DR-TB, TB/HIV co-infection, and pediatric TB capacity-building activities. PATH also worked to increase case detection through quality-assured microscopy and other diagnostic methods in addition to the provision of technical assistance to the National Tuberculosis Reference Laboratory (NTRL) to strengthen its capacity to process specimens and improve turnaround time. • Afya Mzuri was responsible for community TB training, mobilization, awareness-raising activities, and community-level case finding and treatment support. • CITAM+ was responsible for advocacy and activities aimed at increasing financing for TB control efforts from the GRZ, private sector, and international organizations. CITAM+ also supported capacity-building for community-based volunteers (CBVs) to improve treatment literacy, including treatment counselling, adherence support, and stigma reduction. • ZAMBART promoted research to optimize implementation, impact, and innovation through training and mentoring staff in operational research (OR). ZAMBART also leveraged TB research for strategic partnerships with the private sector, GRZ, and Zambian research institutions. 3 ETB results against targets are taken from the Project Performance Review (presentation to USAID on July 7, 2022). 12 Figure 1: ETB Theory of Change DISCOVER-Health SmartCare Clinical care teams DHIS2 Patient-centered approach TB-related mortality reduced by 50 percent in the intervention provinces by 2022 Internalize and apply emerging lessons to address challenges Knowledge-sharing fora Community dialogue Multi-channel communication Documentation and dissemination of learnings Promotion of local ownership Number of individuals screened for TB in target provinces increased by 50 percent TB treatment success increased to >85 percent GRZ at all levels; USG partners; The Global Fund to Fight AIDS, Tuberculosis, and Malaria; World Bank; private sector; civil society; pharmacists; private providers; traditional leaders; community volunteers SAFE Neighborhood health committees EQUIP World Bank CIDRZ Leverage existing platforms to sustain results Engage all categories of care providers Engage private medical practitioners Engage pharmacists Engage community-based volunteers Engage traditional leaders Intensify research and innovation Build capacity in operational research and establish national TB operational research priorities Develop strategic partnerships for funding of operational research Pursue high-quality DOTS expansion and treatment Increase political commitment Increase TB case detection through improved diagnostics Standardize TB treatment, support, and care Support the existing NTLP monitoring and evaluation system Increase access to TB/HIV and MDR-TB services for poor and vulnerable populations Reduce the burden of TB/HIV comorbidity Strengthen systems to halt DR-TB expansion Detect and treat TB among children and other vulnerable groups Provide support to increase TB detection and treatment among prisoners Achieve gender-equitable access to quality-assured TB services Monitor, evaluate, and use data for decision-making and continuous improvement Strengthen collaboration and coordination Maximize investments and address challenges Target evidence-based interventions Achieve results USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 13 | P a g e 3. METHODS AND LIMITATIONS4 This end-of-project process evaluation of ETB involved the collection and analysis of both primary and secondary data. The latter were sourced from an extensive document review, while the former were collected through three qualitative data collection methods: 1) Key Informant Interviews (KIIs) (41 KIIs in total) with individuals who had specialized activity or contextual knowledge that was essential to answering the evaluation questions. 2) Focus Group Discussions (FGDs) (6 FGDs, 22 participants) with CBVs (TB Treatment Supporters) who were the focus of ETB’s community-level efforts to improve TB case identification and support for TB patients. 3) A limited amount of semi-structured observation was undertaken at six district-level health volume facilities to observe the quality of ETB’s capacity strengthening support to laboratory technicians, TB focal points, and district TB coordinators. Each of these methods are briefly detailed below. All data collection protocols and tools underwent an ethical review by a Lusaka-based Institutional Review Board (IRB). 5 Two points are worth emphasizing at the outset of this section: 1) In consultation with ETB and USAID/Zambia, a decision was made to forego a quantitative approach to data collection. This was due to the attrition of ETB, partner, and MOH (facility-level) staff attached to the activity. Many staff left the activity in the close-out period preceding ETB’s formal completion in September 2022 (and before the evaluation fieldwork could begin). Since a representative sample of staff would not be possible to obtain, it was deemed more useful to focus efforts on capturing rich, descriptive qualitative data from those individuals the evaluation could successfully reach. It was also agreed during the evaluation’s inception phase that no data – qualitative or quantitative –would be collected from those individuals screened, diagnosed, and/or treated for TB. See the study limitations (Sub-Section 3.5) for further reflection on these points. 2) The evaluation team agreed with ETB and USAID/Zambia to focus the evaluation on three of the six provinces where ETB was implemented. Doing so was seen to aid the efficiency of data collection. Moreover, it was determined that the evaluation would be of greatest utility in focusing on provinces where activity implementation was most intensive (Copperbelt Province), and where TB treatment success rates were highest (Luapula Province – 93 percent) and lowest (Central Province – 90 percent) in case there were meaningful differences in the quality of ETB’s implementation that could help explain these differences. Within each province, three districts were purposively chosen as locations for data collection. The districts, selected on the advice of ETB and in consultation with USAID/Zambia, were those where the activity had rolled-out comprehensive TB service delivery, signified by the presence of the GeneXpert TB diagnostic tool in district health facilities. By focusing the evaluation’s limited time and other resources on districts where ETB had supported the rollout of all three TB testing methods – GeneXpert and more traditional microscopy and lipoarabinomannan assay (LAM) – it was posited that the evaluation would be best placed to gain insights into the full scope of the activity’s clinical interventions. Within each district, the facilities to visit for observation, the facility staff to be interviewed, and the CBVs to be included in FGDs were determined purposively, based on discussions with 4 See Annex 2 for a more detailed overview of the study design, data collection and analysis methods, and IRB submission. This annex contains a list of the provinces, districts, and facilities visited during the evaluation’s fieldwork. Copies of the evaluation’s data collection tools can be found in Annex 3. Annex 4 contains a list of the evaluation’s evidence sources. 5 ERES Converge was the IRB engaged for this evaluation. USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 14 | P a g e ETB, USAID/Zambia and those national-level MOH/NTLP contacts the evaluation had engaged prior to finalizing its district data collection plans. At each facility visited, the evaluation prioritized KIIs with TB focal points and their immediate clinical support staff. 3.1 DOCUMENT REVIEW Prior to the start of primary data collection and continuing over the course of the evaluation as additional resources were identified and accessed, the evaluation team conducted a comprehensive document review. The documents accessed included both ETB-specific documentation as well as other resources which provided important contextual information relevant to the activity (e.g., on GRZ/MOH policy developments in the TB space). The initial document review was particularly important in informing how the evaluation would go about framing and addressing the evaluation questions, as well as developing the study’s data collection tools. It also served to identify who should be included as participants in KIIs and FGDs, and it enabled a review of the key activity-level information (e.g., around implementation processes, stakeholder coordination, etc.) that was essential to informing a process-focused evaluative approach. A full list of reviewed documents is provided in Annex 4. However, the list below provides an indication of the main types of documents accessed and reviewed for the evaluation. • ETB annual reports and work plans. • ETB financial records. • ETB M&E reports and monitoring data. • Training records – including for ETB’s training of CBVs, district laboratory staff, district TB focal points, and district-level TB coordinators. • ETB staffing information. • Meeting minutes (e.g., of coordination meetings between project partner organizations, minutes of meetings). • Operational research reports. • MOH bulletins and other TB-focused reports/strategy documents. • GRZ policy documents (e.g., produced by the Parliamentary Caucus on TB). 3.2 KEY INFORMANT INTERVIEWS KIIs formed the foundation for the evaluation’s collection of primary data. A total of 41 KIIs were carried￾out by the evaluation team. These KIIs were drawn from the following groups: • Senior ETB staff from PATH Zambia. • Staff from organizations in the ETB implementing consortium. • Ministry of Health (TB Advisor, NTLP representatives, Provincial Health Directors, Provincial TB Coordinators, District Health Directors and TB Coordinators, as well as health facility staff – including TB focal points. A full list of KIs (identified by institution rather than by name) is provided in Annex 4. As KIIs are used to generate insights from those individuals who have specialized expertise on a particular issue or set of issues (Carter and Beaulieu, 1992; Rowley, 2012), it is important to reiterate that the KIs for this evaluation were selected purposively. The inclusion of KIs was based on their established level of knowledge/involvement in ETB and/or familiarity with the Zambian TB context. The evaluation team engaged with both ETB and with USAID/Zambia in the identification of KIs. The evaluation team developed semi-structured KII guides to use with the different informant types (see Annex 3). The USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 15 | P a g e interview guides were designed to elicit information directly relevant to the four evaluation questions, while also allowing for flexibility in the parameters of the topics being discussed. Upon completion, each KII was transcribed and a first round of data analysis was conducted on the interview transcripts in order to inductively code the data (using a code book) into broad themes. These themes were then interrogated further and refined, including with data being categorized – based on the development of new codes – into relevant sub-themes deemed useful for addressing the evaluation questions. A content analysis was carried-out on these re-coded and grouped data to pull out the relevant findings/themes that are presented in the remainder of the report. 3.3 FOCUS GROUP DISCUSSIONS FGDs served as the evaluation’s key data collection method with the community-level actors engaged by ETB. A total of six FGDs – inclusive of 22 participants – were conducted with CBVs trained by ETB to assist in the pro-active identification of cases, as well as to support TB patients to maintain their treatment regimen. FGDs serve as moderated discussions among small groups of individuals, which are done with the aim of eliciting their views on a topic or set of topics (Gray, 2014). By providing a safe environment for discussion, FGDs aim to generate interactive dynamics that promote “self-disclosure” around shared and unshared attitudes and experiences (Freeman, 2006). For this evaluation, FGDs (done in mixed groups of women and men) provided a platform for ETB-trained CBVs to reflect on their experiences being trained by ETB and their experiences utilizing this training while serving as frontline community health workers tasked with supporting the local-level TB response. As with KIs, FGD participants were selected purposively based on consultations with ETB, but with the evaluation team making the final selection of participants. Each FGD was held in a private location, with one member of the evaluation team serving as a discussion facilitator and another team member serving as a notetaker. Analysis of the qualitative FGD data was done in the same manner as the analysis of the KII data. 3.4 SEMI-STRUCTURED OBSERVATION The evaluation team carried-out a limited amount of observation at ETB-supported district health facilities. Observation was intended to be a larger part of the evaluation’s methodology, but the timing of the evaluation – taking place as most ETB activities were beign wound down – limited this method’s utility. Nevertheless, the evaluation team utilized a semi-structured observation checklist (see Annex 3) to review a small number of ongoing activity refresher trainings being provided to district facility TB focal points and their support staff. The observations made are detailed, where relevant, in the study findings. 3.5 LIMITATIONS A couple methodological limitations are worth noting: 1) The lack of a quantitative/survey component to the evaluation is a missed opportunity. While the timing of the evaluation, occurring as many ETB and other partner staff had already left the activity, means that it may never have been possible to obtain a representative sample of ETB staff or other stakeholders, there would have still been value in administering a small survey to KIIs. Likely focused on a limited set of structured questions on perceptions of and/or satisfaction with different components of ETB implementation, the survey would have complemented the descriptive qualitative data that the evaluation did collect while also providing an additional source of data with which to triangulate the study’s findings. 2) Fewer CBVs than hoped were engaged in the evaluation’s FGDs. The fact that the evaluation took place as activity close-out was already well underway, presented a number of challenges. While the evaluation was successful in interviewing most of its intended KIs (using the inception phase to track down those who had already departed the activity), it was more difficult to track down CBVs without having USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 16 | P a g e activity staff readily available to help identify and facilitate introductions to these volunteers. As such, only 22 CBVs were included in the six FGDs. Efforts were made to ensure that a diverse and reasonably representative cross-section of CBVs were included in the study (e.g., CBVs of different sexes, ages, and geographies), but FGDs were composed of fewer than the preferred 8-12 participants, which limited some of the dynamism that the FGD approach is intended to generate. 4. FINDINGS & CONCLUSIONS This section presents a summary of the evaluation’s key findings and conclusions for each of the four evaluation questions. The report elaborates on the implications of these findings and identifies optimal approaches for responding to them in the Recommendations section that follows. 4.1 EVALUATION QUESTION 1: WHAT LESSONS LEARNED AND/OR BEST PRACTICES SHOULD BE TAKEN FROM ETB TO INFORM THE IMPLEMENTATION OF OTHER (EXISTING) TB-FOCUSED ACTIVITIES AND THE DESIGN OF NEW ACTIVITIES FOCUSED ON TB ERADICATION? In answering this evaluation question, the study chose to adopt the well-known definition of lessons learned put forward by the Development Assistance Committee of the Organisation for Economic Co-operation and Development (OECD): “Generalizations based on evaluation experiences with projects, programs, or policies that abstract from the specific circumstances to broader situations. Frequently, lessons highlight strengths or weaknesses in preparation, design, and implementation that affect performance, outcome, and impact.” (OECD, 2002: p. 26) This definition is certainly appropriate for a process evaluation focused on identifying key strengths and weaknesses in ETB’s preparation, design, and implementation. Moreover, by noting the importance of identifying “generalizations” with a utility beyond the initiative being directly evaluated, the OECD definition helpfully articulates the objective of this evaluation question – to focus less on the minutiae of ETB’s results and to instead provide generalized learnings that can be applied by USAID/Zambia and its partners in the context of a broadly focused TB programming portfolio. In assessing best practices, meanwhile, the evaluation draws on UNAIDS’ (1999) sensible maxim that defines best practices not as “ultimate truths” or “gold standards”, but as apparent “keys to success of any given project, program, or policy” (UNAIDS, 1999: p. 5). This report’s presentation of best practices takes a similar track, discussing those factors that the evaluation’s evidence base indicates are important, but also presenting them in context and with qualifications where necessary. 4.1.1 SUMMARY RESPONSE TO EVALUATION QUESTION 1 The evaluation identified great value in a number of the seemingly simple, yet vital, reforms that ETB facilitated. A number of these should be seized upon as best practices for ongoing and future USAID/Zambia TB programming: • The importance of adopting a decentralized zonal-focused model of capacity strengthening for district-level TB clinicians. • A holistic approach to TB case identification, in which site activation places a responsibility on all health care workers in a facility – and all facility departments – to screen patients for TB. USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 17 | P a g e • The improved use of simple information communications technologies (SMS, WhatsApp) to improve the flow of information between clinicians at facilities (often provincial) where patients are diagnosed and facilities (district-level) where many patients will seek subsequent care. • The value of opportunistically “piggybacking” on existing HIV partners’ sample courier systems to ensure the timely delivery of sputum samples for quicker TB case identification. The document review and KIIs identified these four reforms as being among the most significant contributors to improved TB case identification (including among children) and patient care. Some of these reforms, such as site activation, are already becoming institutionalized within the NTLP’s new NSP covering the period 2022 to 2026. Others will require ongoing efforts, including through new USAID TB programming, to institutionalize and sustain. Also important to consider for future activities is the important role that ETB played as a convening force within the wider TB ecosystem. ETB provided considerable support to the NTLP in convening both the Technical Working Group (TWG) on TB and in fostering the creation of provincial TB elimination committees. These groups, in turn, have emerged as significant forums for dialogue on how best to identify and remove roadblocks to TB elimination. In the absence of ETB playing this convening role, it cannot be taken for granted that these groups will survive. Ongoing and future USAID/Zambia TB programs may need to identify their own potential entry points to providing this same convening assistance. Finally, ETB’s facilitation of virtual “situation rooms” – an unintended outcome of COVID-19 – to foster discussion of TB data trends and associated action points, is highly regarded by the NTLP and by USAID/non-USAID implementors of TB activities. The “situation rooms” seem to have unblocked a significant barrier to data-driven decision-making around TB by allowing for real-time reflections on available evidence from multiple sources. These innovations should continue and become institutionalized by the NTLP. Where lessons learned provide opportunities for reflection and improvement, the following conclusions emerged from the evaluation: • Partnerships cannot be established ad hoc but must be planned for from the outset of activity implementation. This is true of private sector partnerships, which ETB did not foster to the extent intended. However, it is also true of partnerships with the GRZ, where ETB’s efforts to engage the Parliamentary Caucus on TB, while by no means ineffective, have produced few tangible gains. • Any activity that requires the collection and reporting of data by health facilities must ensure that these facilities have the computers (or tablets) and connectivity required to play this role. ETB’s notable efforts to capacitate personnel at district-level facilities to collect quality data were somewhat undermined by the activity’s failure to ensure that facilities were equipped with the computers/tablets and internet connectivity required to submit these data in a timely manner. Moving forward, any TB-focused activity that relies on facilities to provide the majority of data need to consider a cost-effective way to ensure that facilities have the digital infrastructure required to realistically play this role. This could include training facilities on using smart phones and relevant apps for data collection, as well as ensuring that provision exists for offline reporting. 4.1.2 SUCCESSES 1) Finding: ETB played an important convening role within the Zambian TB space, providing structure to the NTLP’s efforts to institutionalize multi-stakeholder collaboration. The NTLP USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 18 | P a g e serves as the GRZ’s leading agency in planning, coordinating, and implementing the national TB response, primarily through the operationalization of the NSP (NSP, 2017, 2022). A notable criticism of the NTLP has been its perceived weakness in establishing and consistently leading a structured approach to collaboration among sub-national officials in the MOH, and among agencies engaged in the fight against TB – including (among others) USAID, Challenge TB, the Stop TB Partnership, the Centre for Infectious Disease Research in Zambia (CIDRZ), and the World Bank (WHO, 2019). ETB shifted this dynamic, both by creating new coordination spaces (provincial TB elimination committees), and by taking on convening responsibilities that the NTLP lacked the manpower to take on itself. Indeed, a common point to emerge from the KIIs was the importance of ETB adopting select “secretarial” duties on behalf of the NTLP, including scheduling meetings of the national TWG on TB, supporting the NTLP in the co-creation of TWG meeting agendas, ensuring the proper documentation of TWG discussions, and following-up with partners to obtain progress updates on action items identified in the TWG sessions. 6 Similarly, at the provincial level, the activity’s creation of provincial TB elimination committees – bringing together ETB, PHO staff, and staff from teaching hospitals and other health facilities – was viewed as important in ensuring the availability of an active platform for sub-national discussions of strategies aimed at enhancing TB case identification and management. “In the long-term, I think the provincial TB elimination committees will be an important contribution [of ETB]. For TB control [to be effective] the planning and discussion needs to be decentralized so that responses are appropriate to context. The [TB elimination] committees move us in this direction.” (KII: Provincial TB Coordinator, Copperbelt Province) The upshot of this convening support was the gradual emergence of a more consistent and sustained national dialogue on TB over the course of activity implementation (NTLP, 2021; WHO, 2021).7 Conclusion: ETB’s contributions to improved multi-stakeholder collaboration at national-level sits somewhat outside of the activity’s main objectives. However, the perceived value of the activity’s role as a convening force in the national TB space, points to a function that current/future USAID/Zambia TB activities should actively seek to play. This is particularly true as the NTLP is likely to remain overstretched and seems unlikely to fully play this convening role (at national-level) in the immediate future (WHO, 2021). While steps have been taken to institutionalize the provincial TB elimination committees, most remain new and do not yet exist in all provinces. As such, it is likely that their immediate sustainability will also depend on active and ongoing USAID/Zambia support. 2) Finding: ETB’s decentralized approach to capacity strengthening, focused on zonal-led sensitizations, trainings, and mentorships, was critically important to activity success. This was especially true in strengthening approaches to pediatric TB care. A notable challenge identified within the TB literature in Zambia (Bwalya, 2014), and by several of this study’s KIs, was the poor detection of TB in children. In most cases, this was the result of limited knowledge and experience of TB diagnoses among pediatricians working at district (and even many provincial) health facilities. Poor knowledge on the part of these health care workers led to worsening symptoms, delayed referrals, and a high mortality rate among children with TB (Bwalya, 2014; Kapata et al., 2014).8 ETB’s decentralized approach to capacity strengthening saw health care workers in select provinces (particularly from high-volume facilities in Central and Copperbelt Provinces) be selected for intensive training. This training, provided by consultant pediatricians, focused on the identification and treatment of child TB. These trained personnel were then assigned to serve as pediatric TB focal points for their facilities and for the satellite health facilities within their facilities’ respective coverage zones (ETB, 2021). 6 Source: KII with member of ETB senior staff; KII with member of NTLP national leadership. 7 Source: KII with member of NTLP national leadership. 8 Source: KII with member of USAID/Zambia senior leadership; KII with TB Hospital Focal Point in Central Province. USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 19 | P a g e Available documentation, as well as several KIIs, identified the capacitation of these “zonal leads”, as well as the further training they then provided to other lower-level facility personnel, as a major contributor to improved processes around the identification of pediatric TB. This included improved contact tracing, improved TB screening in outpatient departments and other hospital wards, the appropriate use and understanding of diagnostic tools (sputum, stool, LAM), and chest x-ray interpretation (ETB, 2021, 2022).9 While the TB notification rate among children is still below the 10 percent LOP target, ETB’s decentralized approach is seen as having put in place an important foundation for improving the quality of pediatric TB care in the future: “This issue of childhood TB has been a real challenge. Most [health care] workers in the districts are unfamiliar with how to correctly diagnose TB. I think ETB has created a zonal training and supervision model that should be maintained, and which could be applied across the country and even to address other health challenges like HIV/AIDS.” (KII: Senior Representative from the NTLP) Conclusion: ETB’s zonal approach to capacity strengthening is a key innovation that should be maintained and/or scaled-up for current and future USAID/Zambia TB programming. Where it has been rolled-out (in Central and Copperbelt Provinces), this approach appears to have played a significant role in strengthening the capabilities of health care workers at both provincial and district levels to effectively screen and refer cases of child TB. The fact that child TB identification has been less successful in other ETB focus provinces10, suggests that there may be a correlation between the implementation of the zonal approach and case identification outcomes. 3) Finding: Site activation and an intensified case finding (ICF) model proved to be game￾changers in terms of improving TB case identification. As noted in Section 1 of this report, ETB￾supported facilities tested over 630,000 individuals for TB, a figure that is 178 percent of the LOP target and far exceeding the activity’s objective of increasing the number of individuals screened for TB in target provinces by 50 percent as compared to 2016 baseline levels. Central to this was a shift away from what the literature identifies as a “passive” case finding approach, in which health care workers would only screen for TB at the request of the patient or in case of obvious symptoms (Kapata et al., 2014). In its place, ETB promoted a pro-active approach, in which facility in-charges and health care staff from every facility department undertake TB screening of all patients accessing services and at all facility points-of￾entry (ETB, 2021; NTLP, 2020, 2022). “ETB’s focus on training and mentoring staff at hospitals, even if they are not TB specialists, to play a role in TB screening, has helped identify many TB cases that probably would have gone undetected. Even [our] physiotherapy unit has identified six patients with spinal TB.” (KII: Facility TB Focal Point, Central Province) Conclusion: While seemingly a simple reform to institute, site activation and the associated capacitation of non-TB specialists to screen patients for TB, combined with the promotion of a shift in institutional practice towards a pro-active approach to case identification, have clearly paid dividends. Indeed, the success of this approach can be seen in the fact that site activation was included as a central component of the NTLP’s new NSP covering the period 2022 to 2026 (NTLP, 2022).11 4) Finding: ETB successfully utilized basic information communications technologies (SMS and WhatsApp) to improve the flow of clinical information between facilities and between 9 Source: KII with member of USAID/Zambia senior leadership; KII with two members of ETB senior staff; KII with member of NTLP national leadership; KII with two health facility TB focal points in Central and Copperbelt Provinces. 10 Source: KII with member of USAID/Zambia senior leadership. 11 Source: KII with member of NTLP national leadership. USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 20 | P a g e clinicians. A key outcome of this was the rollout of the “trigger system” to track patient care requirements. The literature on TB in Zambia identifies a central challenge when it comes to ensuring long-term patient care – the lack of a dedicated system to ensure that patients, once diagnosed, are consistently tracked regardless of their movements. In several cases, patients will receive diagnoses at a centrally-located hospital and then return to their communities. However, patient information (e.g., details of their treatment regimen) is seldom transmitted from the facility providing the initial diagnosis to the district-level facilities where the patient is likeliest to access continuing care (Kapata et al., 2014; WHO, 2018). The trigger system facilitated by ETB allows TB clinicians at diagnostic facilities to note important details (appointment dates, reminders of the need for sputum sample collection, required treatments) and communicate these through SMS or WhatsApp to counterparts at district-level. This, in turn, allows district-level care providers to pro-actively engage with patients and/or with CBVs who can engage with patients and ensure that they adhere to their appointments and treatments.12 “The trigger system is so simple, relying on SMS, but has helped improve TB treatment. With a better flow of information [between health facilities] I don’t think we will see so many patients drop-off after their diagnosis. This means fewer will die.” (KII: Facility TB District Coordinator, Copperbelt Province) “One of the good things about the trigger system is it allows our local facilities to contact us [the CBVs] and assign us as buddies to [TB] patients. We can then help patients go to their appointments and stick to their treatment.” (FGD: Female CBV, Central Province) Conclusion: Simple adaptations to how TB clinicians communicate and share information can have a profound effect on TB treatment outcomes. The long-term sustainability of the trigger system – and of the network of CBVs that complement and reinforce patient’s access to community care – does remain a concern. The large amount of personnel turnover in health facilities (due to transfers or simple attrition) means that this system needs to be institutionalized and cannot simply rely on the willingness of ETB-supported clinicians, who may not be around in the long-term. Similarly, it was not clear from a review of ETB documentation, or through engagement with the KIIs or FGDs, that there are mechanisms in place to incentivize the CBVs over the long-term (or to replace CBVs who quit their roles). However, a model has been tested that suggests that patient care – and by association – TB mortality – can be affected if small improvements to inter-facility communication remain a core USAID/Zambia and partner focus. 5) Finding: ETB’s support for the development of an integrated inter- and intra-district courier system for sputum specimen delivery, has increased efficiencies and reduced costs for sample transport. A key challenge constraining the efficient and cost-effective delivery of sputum specimens, and by extension reducing the utilization of the GeneXpert diagnostic tool allowing for quicker TB case identification, has been the lack of reliable courier systems at district-level (MOH, 2016; USAID, 2019a, 2019b). The MOH’s courier system, understaffed and relying on motorcycles lacking parts and ready access to fuel, provided an insufficient basis on which ETB could strive to improve case identification metrics, particularly as GeneXpert machines are available at only 141 of ETB’s 341 supported health facilities (ETB, 2021). As a response, the activity adapted by engaging with partners working in the HIV space to “piggyback” on their comparatively well-established (and donor-funded) courier networks to ensure the inclusion of sputum specimens alongside blood samples being transported for HIV testing (ETB, 2021; WHO, 2021). 12 Source: KII with member of USAID/Zambia senior leadership; KII with TB/HIV Coordinator, Luapula Province; KIIs with District TB Coordinators in Central and Luapula Provinces. USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 21 | P a g e ETB’s contribution was typically a cash payment for each delivery to cover partial fuel and maintenance costs (ETB, 2021, 2022).13 The upshot of this approach has been a significant increase in timely and completed specimen delivery in ETB’s target districts (Anderson et al., 2022; ETB 2022, NTLP, 2022). Three KIs cited the efficiencies of the courier system as a contributor to significant year-on-year increases in the number of samples being successfully delivered (from 40,902 in FY 2020 to 55,323 samples in FY 2021) (ETB, 2021).14 Conclusion: The maintenance of this HIV-TB integrated courier system should be a priority for ongoing and future USAID/Zambia TB programming. Efforts are being made to generate this system’s sustainability – e.g., ETB’s engagement of PTOs and the NTLP to include the fuel and motorbike maintenance costs in the MOH’s Medium-term Expenditure Framework (MTEF) (detailed further in Section 4.3). However, it can be expected that donor funding for this integrated courier system will still be necessary in the short-term. 6) Finding: ETB demonstrated effective adaptive management in its approach to M&E, particularly through the rollout of its collaborative “Situation Rooms” for regular virtual data reviews. The COVID-19 pandemic had a notable effect on the ability of ETB staff, MOH/NTLP officials, USAID/Zambia, and other implementors to regularly meet in-person and review progress. Connected to the point on ETB’s convening role, noted above, the activity supported the NTLP, early in the pandemic, to develop a weekly (virtual) TB Situation Room. Bringing together implementors across Zambia engaged in the TB space, the Situation Room provided a platform for regular meetings with NTLP officials to specifically discuss and critically reflect on key TB indicator data in order to identify bottlenecks in TB case identification and/or patient care (ETB, 2021). Crucially, these weekly sessions provided opportunities for the quick dissemination of data and identification of challenges, thus addressing a longstanding criticism of Zambian health programs being too slow to ensure the exchange of data between implementors and the MOH (Anderson et al., 2022).15 According to several KIs, the TB Situation Room also provided a collaborative space for discussions around improving data visualizations, as well as on the development of data-focused trainings for district and facility-level health care workers.16 ETB played a leading role in also supporting the development of a bi-weekly USAID TB IP’s Situation Room, which served much the same purpose as the NTLP Situation Room, but with discussion focused among USAID/Zambia IPs. Both Situation Rooms were reported by multiple KIs as contributing to enhanced skills for collaboration and data-driven decision-making: “The Situation Rooms, particularly the one with the NTLP, were useful platforms for collaboration. I had the opportunity to present our [ETB] data and obtain feedback from the Ministry [of Health] and other implementors in real-time.” (KII: Senior ETB staff member) “The Situation Room with the NTLP was great in allowing us to share data on the spot. Before, there were meetings but a lot of the discussion on data would not happen until someone [at the MOH] reviewed our reports. It took a long time. With [the Situation Room], we obtained feedback right away, which made our decision-making faster.” (KII: Senior ETB staff member) Conclusion: Given the benefits noted above, the maintenance of the Situation Rooms as a “best practice” for ongoing and future TB programming is clearly of value. This is especially true if 13 Source: KII with member of ETB senior leadership. 14 Source: KII with member of NTLP national leadership. 15 Source: KII with member of ETB senior leadership; KII with member of ETB implementing consortium. 16 Source: KII with member of ETB senior leadership; KII with USAID senior leadership; KII with member of NTLP national leadership; KII with MOH TB technical official, Luapula Province. USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 22 | P a g e they continue to provide an avenue by which data can be presented, interrogated through a de facto peer review, and used to identify bottlenecks and inform rapid decision-making, whether on the part of USAID/Zambia, the NTLP, or other program implementors. 4.1.3 CHALLENGES 7) Finding: ETB’s approach to partnerships, while affected by a number of contextual factors outside the activity’s control, seldom demonstrated clear strategic intent. Two of ETB’s key partnerships were with the Select Parliamentary Caucus on TB and with an intended array of private sector actors. Relationship-building with the former was interrupted by the Zambian national election in August 2021, while partnership building with the private sector was made more difficult due to the advent of COVID-19 in early 2020 and the resulting interruption of face-to-face meetings (ETB 2020, 2021, 2022).17 However, the evaluation found little evidence in its document review, or in its engagement with KIIs, that ETB had a clear and dedicated focus when it came to partnership building. Based on a review of minutes from ETB coordination meetings, private sector engagement was discussed at various points early in project implementation (ETB, 2017a, 2017b). However, in part due to the turnover in the activity’s senior leadership (ETB had two Chiefs of Party and three Deputy Chiefs of Party over its implementation – see Section 4.2 for further discussion), no private sector engagement strategy was developed until FY 2020 and detailed sector-focused engagements were not begun with private sector actors until FY 2021, after ETB’s second COP was onboarded (ETB, 2020). ETB played a more active role in supporting the creation of the Parliamentary Caucus on TB, but the evaluation found no evidence of a dedicated engagement strategy to guide the activity’s interactions with the Caucus or to spell out what the activity hoped to engage (on its behalf or on behalf of the NTLP) through these interactions. Indeed, while the project did develop constituency profiles to provide context analyses for select parliamentarians (ETB, 2021), it is not clear from the document review of KIIs what else this intended partnership has achieved. Conclusion: Partnerships cannot be established ad hoc but must be planned for from the outset of activity implementation. Unless clear partner engagement plans are established early in an activity, and ample time is provided to operationalize them, such engagements are unlikely to be successful. In the case of ETB, private sector and Parliamentary Caucus partnership-building, while hardly neglected, did not emerge (at least based on a review of available evidence) as clear topline priorities, and certainly not priorities where ETB staff had clear tangible outcomes against which to guide their actions. 8) Finding: For data to effectively guide programming, more focus needs to be placed on ensuring that district-level focal points have the capacity and resources they require to collect and report their data. As noted earlier, ETB introduced an important M&E innovation at the national-level in its facilitation of “situation rooms” to serve as platforms for detailed discussions of emerging data. At the district-level, activity M&E was compromised by a simple problem, namely a lack of computers through which district and facility focal points could easily submit required data.18 “We need all facilities to have computers so they can report their data. Many facilities didn’t have computers and we were forced to try and secure support from other donors to see if they could provide computers. This usually wasn’t possible. Even when computers were present in facilities, connectivity was so bad that data couldn’t be sent anyway.” (KII: Senior ETB staff member) 17 Source: KII with member of ETB senior leadership; KII with member of ETB implementing consortium. 18 Source: KII with member of ETB M&E team; KII with two District TB Coordinators in Luapula Province. USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 23 | P a g e The upshot was that data submissions from facilities were often late and/or ETB staff based at PHOs were obligated to travel to district facilities and physically collect manually-entered data.19 In addition to marking an inefficient use of activity resources, this diminished the utility of those data collected. Data from more physically remote facilities, in particular, were liable to be weeks old before they were analyzed and officially reported at provincial level or by ETB in Lusaka. This, in turn, delayed the activity’s feedback to the facilities highlighting any suggested action points emerging from the data. Conclusion: Any activity that requires the collection and reporting of data by health facilities must ensure that these facilities have the computers (or tablets) and connectivity required to play this role. It is apparent that ETB’s M&E demands were not cognizant of the actual capacity that health facilities had on the ground to report their data. ETB contributed significantly to capacitating relevant district health staff to effectively collect data. What the activity did not resolve was how to ensure the timely submission of these data – both for ETB’s reporting purposes to USAID/Zambia and for active decision-making/adaptive management. If future TB programs are going to work in large numbers of district-level facilities, and if they plan to rely on these facilities to serve as the main providers of data, then thought will have to be given as to the most cost-effective way to provide facilities with computers, tablets, and quality connectivity. 4.2 EVALUATION QUESTION 2: WERE ETB STAFFING AND PARTNERSHIP STRUCTURES OPTIMAL FOR ACHIEVING PROJECT OBJECTIVES? 4.2.1 SUMMARY RESPONSE TO EVALUATION QUESTION 2 ETB’s staffing structure was broadly, though not entirely, fit for purpose. The activity experienced significant turnover in its senior leadership – having in place two Chiefs of Party and three Deputy Chiefs of Party over the course of implementation. This created challenges in terms of the continuity of technical leadership and the generation of institutional knowledge, though the evaluation did not discern a negative impact on activity performance caused by this turnover. The partnership between PATH and its consortium partners was deemed to be strong, with the embedding of partner organization staff within the PATH office seen as helping to ensure effective awareness and collaboration.20 ETB represented a new approach by a USAID/Zambia activity to embed field technical staff directly with the MOH, primarily in PHOs. This engagement with the MOH was viewed as a success, not least by the Ministry itself, which valued day-to-day exposure to ETB activities and the opportunities for learning and capacitation that this provided. For ETB, a close working relationship with the MOH was productive yet challenging. Important relationships were established and MOH technical capacity built. However, near constant turnover in MOH staff (including facility in-charges and provincial and district-level TB coordinators) created problems with technical continuity and placed a burden on ETB staff to provide almost non-stop orientations and repeated technical trainings to new or relocated MOH personnel. It is to ETB’s credit that the activity took steps to address this need by standardizing as much of this orientation/training content as possible so as to ensure consistency and reduce the time demand placed on staff to devise orientation/training materials. Where ETB’s staffing structure was inadequate, it was for the following reasons: • Sufficient human resources were not allocated to the district-level. ETB placed a limited compliment of staff (clinician, M&E staff, driver) at PHOs, but these staff could not provide 19 Source: KII with member of ETB M&E team. 20 Source: KII with with two members of ETB senior leadership; KII with USAID senior leadership. USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 24 | P a g e timely technical assistance or data collection to all districts given the expansive geography they had to cover. • ETB staff working with district TB coordinators and other facility staff did not always have the competencies required to provide needed capacity strengthening support. When ETB staff lacked skillsets that were fundamentally different from the facility staff they were engaging with, they were unable to troubleshoot and often replicated coordination functions already being played by the facility staff instead of providing clear technical leadership on activity implementation. • The activity’s M&E structure was too lean. In addition, there was no Knowledge Management lead to ensure the documentation and dissemination of successes and lessons learned. Too few M&E staff created challenges with ensuring the timely collection and reporting of data (though the project was broadly successful in ensuring that reports met USAID deadlines). The lack of a Knowledge Management function created challenges in ensuring the effective utilization of institutional memory. From a partnerships standpoint, the fact that ETB did not develop, from the outset of implementation, a clear strategy for engaging with the private sector, meant that deep partnerships were not established. Private sector support was provided to one-off events (e.g., Barclays funding of a World TB Day event) but private investments with a long-term commitment to the TB space were not forthcoming. ETB’s creation of a clear private sector engagement strategy towards the end of implementation does provide a foundation on which future TB activities can build, provided such a strategy is effectively operationalized. 4.2.2 SUCCESSES 1) Finding: While not without challenges, ETB established an effective working relationship with the MOH at all levels. ETB also demonstrated effective adaptive management in finding ways to cope with the high levels of turnover among MOH personnel at facility-level. ETB represented the piloting of a new approach to engaging the MOH on TB programming – by seconding staff directly to work alongside MOH personnel – housed at PHOs and tasked with delivering technical and M&E support to district-level health facilities. This engagement with the MOH was viewed as a success, not least by the Ministry itself, which valued day-to-day exposure to ETB activities and the opportunities for learning and capacitation that this provided (NTLP, 2021, 2022).21 Key staffing challenges did emerge from the MOH in the form of high turnover (among facility in-charges, district TB coordinators, lab technicians, and other staff responsible for assisting the project with facility￾level reporting). “There was no stability in terms of MOH staff working on TB in the districts. It was actually very frustrating because often when we went to provide technical support, we found new staff we had not dealt with previously. We then had to get them up to speed before moving ahead with technical work we came to do.” (KII: ETB clinician) While this constant churn of facility-level personnel was a challenge, ETB took pro-active steps to minimize the disruptiveness of this turnover by co-creating with the MOH standardized orientation and training plans. Several KIIs reported that these plans served to shorten the time it required for new recruits (or 21 Source: KII with member of NTLP national leadership. USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 25 | P a g e personnel relocated from health facilities elsewhere in Zambia) to be onboarded and begin engaging with the activity. 22 Conclusion: ETB’s model of embedding activity technical staff in MOH offices is an effective approach to encouraging exchanges of knowledge and experience. However, there will always be a need to cope with a potential lack in continuity among MOH personnel being engaged with. Activities need to incorporate orientations and re-trainings for new/relocated facility-level personnel into their work plans but should draw on a key lesson from ETB by ensuring the availability of standardized orientation/training materials to create efficiencies in the onboarding of MOH counterparts. 4.2.3 CHALLENGES 2) Finding: ETB’s technical staffing structure did not allocate sufficient human resources at the district-level. In each of the six provinces where ETB implemented activities, only three project staff (a clinical officer, a M&E officer, and a driver) were allocated, working out of the PHOs. This staff compliment was insufficient to ensure adequate coverage – whether in the timely provision of technical assistance or the collection of data – across the 63 districts and 341 health facilities where the project was present.23 The decision to place staff in the PHOs was also questionable given that MOH personnel working at the provincial level provide technical oversight but are seldom responsible for implementing activities. This created a mismatch whereby ETB staff in the PHOs were not paired with MOH counterparts with whom they could align their activity-focused work plans.24 It was at the district-level where ETB would have benefitted from allocating most of its clinical staff. While ETB staff were present in every province, staff working at the district-level were only present in three “high volume” districts in Copperbelt Province (Kabwe, Kitwe, and Ndola) (ETB, 2021). Outside of these districts, the lack of permanent district staff meant that ETB was unable to ensure consistent engagement with the MOH’s district TB coordinators.25 This lack of regular engagement did not dramatically affect activity results but did lead to delays in the rollout and monitoring of activities since MOH staff were often busy with other tasks and would not necessarily prioritize ETB activities in the absence of ETB clinicians. “The staff were not adequate [at district-level] because the project was designed to work in all the districts of the province and the project had only two technical staff and a driver. You can imagine a province of 10 districts with only two staff… that is a lot of work.” (KII: Senior ETB staff member) “ETB did not have any staff at district or facility level, only Copperbelt and Central provinces had ETB staff [at these levels]. The absence of staff at the district level proved to be a challenge in the coordination of activities since MOH staff [District TB Coordinators] had other activities.” (KII: Senior ETB staff member) Conclusion: For TB activities focusing mainly on district and facility-level interventions, there is a need to re-think activity staffing structures to ensure that adequate and timely technical support can be provided. ETB’s decision to allocate its technical staff at provincial level and within only a small number of “high volume” districts, was a pragmatic reflection of the need to control staffing costs in a geographically expansive project. Such considerations will undoubtedly continue to be important. However, while placing technical and M&E staff in each district may not be possible, grouping districts and having staff cover 3-4 districts (rather than covering all districts from a provincial base) may be feasible. 22 Source: KII with member of NTLP national leadership; KII with three District TB Coordinators in Central and Luapula Provinces. 23 Source: KII with Provincial TB Coordinator, Central Province; KII with ETB Laboratory staff member, Copperbelt Province. 24 Source: KII with three District TB Coordinators in Central and Luapula Provinces. 25 Source: KII with two District TB Coordinators in Luapula Province; KII with Provincial TB Coordinator in Central Province. USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 26 | P a g e 3) Finding: ETB staff working with the district TB coordinators did not always have the competencies required to provide needed capacity strengthening support. A recurring issue raised by a few of the evaluation’s KIIs was that, in some provinces, ETB’s staff had similar professional profiles and levels of experience to the district TB coordinators and facility staff they were intended to work with. The upshot was that these ETB staff were not always able to come in and provide troubleshooting support to the district TB coordinators, or impart new knowledge, since they themselves did not have different or more sophisticated skill sets. One consequence of this was that the ETB sub￾national staff were sometimes seen as replicating rather than complementing the work of district TB coordinators, focusing on issues of coordination (which the MOH staff were already doing) rather than providing leadership and guidance on technical implementation.26 Conclusion: It is essential to ensure that activity staff working sub-nationally have the skill sets required to provide leadership and guidance on technical implementation. It can be challenging to find technically strong staff willing to work sub-nationally. However, to the extent possible, attempts should be made to avoid hiring staff who will be unable to provide technical guidance to those they are intended to capacitate. 4) Finding: For a data-driven activity, ETB’s M&E structure was too lean. There was also no Knowledge Management position staffed, which prevented the effective documentation and dissemination of successes and lessons learned. For the first two years of implementation, ETB only had in place four M&E staff (ETB, 2020). These staff had responsibility for engaging with provincial and district data collectors across the entirety of the activity’s six implementation provinces. This was problematic, with the existing staff being placed under pressure to provide quality control and ensure the timely reporting of data. The fact that ETB’s main data collection tools – TB registers – were not integrated into the national Health Management Information System (HMIS) created an added data entry burden for district TB coordinators. This, in turn, created a further burden for ETB M&E staff who had to engage in ongoing training with coordinators on an unfamiliar data collection tool/system (NTLP, 2022).27 KIIs reported that ETB M&E staff were more engaged in the need to travel and follow-up on late and/or missing data than they were in analyzing existing data and feeding these data into management decision-making.28 As such, decisions on adaptive management were seldom made with the most up-to-date data from the field. “The [ETB] M&E team did their best, but they were under a lot of pressure, especially at the beginning [of the project]. Eventually, more M&E staff were hired to work in Central and Copperbelt [provinces] and we hired interns to work in all the provinces. They [interns] helped with a lot with improving ETB’s data management.” (KII: Senior ETB staff member) ETB did have in place a Strategic Information lead who had overall responsibility for operational research. However, because the activity did not put in place a dedicated Knowledge Management or Collaboration, Learning, and Adapting (CLA) specialist, ETB did not have clear documentation of its learning over the course of the activity. Nor did the evaluation find evidence that the activity developed specific learning products. This was problematic given the significant turnover in ETB’s senior leadership, with the evaluation finding that staff hired in the latter half of the project, including its Chief of Party, struggled to gain familiarity with what had happened in the early years of the project (outside what could be gleaned from quarterly and annual progress reports).29 Conclusion: Ensure that adequate resources are allocated to staffing activity M&E. Ideally, M&E staff should be dedicated mainly to the analysis and interpretation of activity data rather than spending the 26 Source: KII with member of ETB senior leadership. 27 Source: KII with member of ETB M&E team; KII with two District TB Coordinators in Luapula Province. 28 Source: KIIs with four members of ETB senior leadership; KII with member of ETB M&E team; KII with member of NTLP national leadership. 29 Source: KII with member of ETB senior leadership; KII with member of USAID senior leadership. USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 27 | P a g e majority of their time chasing absent or late data from the field. In practice, there will always be a requirement for M&E staff to engage in regular facility engagement and follow-up. As with the broader staffing issues noted earlier, there is a need to balance what is realistic in terms of staffing levels in a geographically expansive activity, versus what is practically needed to ensure the functioning of an effective M&E system with minimal disruptions to data flow. 5) Finding: From a partnership standpoint, ETB’s efforts to engage the private sector were inconsistent. This is evidenced by the fact that a structured private sector engagement strategy was not developed until late in the activity. As such, ETB was unable to generate the private sector buy-in required to develop and sustain potential partnerships. ETB set out with the objective of engaging the Zambian private sector, based on firms’ stated commitments to corporate social responsibility, to create a pool of additional funding that could be used – over the long￾term – to support TB-focused initiatives. Initial engagements with Barclays Bank PLC did bear fruit in generating funding for a World TB Day lunch, which allowed for multi-stakeholder consultations between ETB, the MOH, and other organizations implementing TB programs (ETB, 2021). However, the withdrawal of Barclays from the Southern African region and its subsequent takeover by Absa, placed a prolonged hold on any further funding commitments from this institution (ETB, 2021). Similarly, COVID￾19 and the restrictions it placed on face-to-face meetings, made it difficult for the project to build a rapport with potential partners. At the same time, the project’s failure to develop a private sector engagement strategy at the outset meant that when discussions were held with firms or with private sector associations – as took place with the Zambia National Farmers Union (ZNFU), Zambia Association of Manufacturers (ZAM), and the Zambia Association of Chambers of Commerce and Industry (ZACCI) – the objectives of these engagements were not clearly laid out or understood by ETB or partner staff. Nor did the project have in place clearly articulated engagement strategies tailored to each firm/association. The result was the project engaged in ad hoc arrangements – such as Barclays support for the World TB Day lunch – that were not representative of the types of deep and sustained private sector commitments that were intended – and which USAID itself articulated to ETB in sharing its own Private Sector Engagement (PSE) Strategy. By the time an external consultant was contracted to develop an ETB PSE strategy in late FY2020, which did map out sector-specific approaches to engaging with firms operating in the mining, banking, agriculture, telecommunications, and manufacturing sectors, there was little time available to operationalize it. Conclusion: It is necessary to develop activity-level PSE strategies at the start of implementation. For PSE to bear fruit, engagements with firms need to be undertaken consistently over time and with clear expectations being established on both sides. When PSE becomes a secondary focus and a clear strategy for engagement not developed until late in an activity’s implementation, it is not realistic to expect this engagement to provide anything other than the occasional – often ad hoc – benefit (e.g., the funding of one-off events rather than sustained investments). 4.3 EVALUATION QUESTION 3: TO WHAT EXTENT ARE ETB METHODOLOGIES, INTERVENTIONS, AND MANAGEMENT SETTING THE STAGE FOR FUTURE SUSTAINABILITY OF PROJECT OUTPUTS AND OUTCOMES? 4.3.1 SUMMARY RESPONSE TO EVALUATION QUESTION 3 The long-term sustainability of many of ETB’s components remains open to debate. However, two components have gained particular traction and have been (or are seemingly likely to be) institutionalized within MOH/NTLP policy and practice: USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 28 | P a g e • Site activation – the demand for an “all hands-on deck” approach to TB screening at health facilities. By ensuring comprehensive screening of all patients, in all facility departments, and at all points of entry, the improved rates of TB screening documented by ETB should be maintained. • Weekly TB Situation Room with the NTLP – a broad consensus exists on the value of having a collaborative forum where the NTLP, implementors, and others in the TB space can regularly engage, peer review existing data, and critically reflect on evident trends to inform policy and practice. Where sustainability may be tougher to sustain, it is in relation to: • The technical capacitation of district-level facility personnel – the high rates of attrition among district health care workers, as well as the high likelihood of personnel being transferred, makes it difficult for health facilities to maintain institutional memory and awareness of “best practices” in TB screening, diagnosis, and treatment that were imparted by ETB. • The CBV model – as with any volunteer-focused approach, ensuring the long-term incentivization of CBVs, in the absence of regular USAID activity engagement, is a challenge. While enthusiasm has been generated within the MOH/NTLP for maintaining and even upscaling the CBV model to ensure improved patient care, it is not clear that a clear strategy exists for CBV retention or for the training of new CBVs - though the MOH should have such a resource that future USAID activities can draw on. 4.3.2 SUCCESSES 1) Finding: Consistent engagement with the MOH/NTLP has contributed to several activity approaches/priorities being incorporated into the new National Strategic Plan for TB Prevention, Care, and Elimination (2022-2026). Site activation and its demand for all facility staff (and not just TB clinicians) to screen patients for TB at all facility points-of-entry, has been incorporated as a central pillar of the new NSP (NTLP, 2022). As such, this approach should now be adopted by all implementors (USAID or non-USAID) engaging in facility-level support around TB. It also implies a dedicated commitment by the MOH to train a wide swathe of health care workers on correct practices for TB screening, which should ensure that the significant increases in the numbers of people screened for TB, documented over the course of ETB’s implementation, will be maintained. “The new NSP is already adopting some of the project’s [ETB’s] best practices. Site activation is probably the most important. This will be a guiding approach [for the MOH] moving forward. I think we will also see the incorporation of the CBV model. We [at the MOH/NTLP] have seen how successful these approaches can be since we were part of implementing [ETB]” (KII: Member of NTLP senior leadership) A number of KIIs, as well as FGD participants, pointed to enthusiasm by the MOH for maintaining CBVs as a linchpin for community-focused TB patient care and follow-up.30 While not explicitly mentioned in the NSP, a review of meeting minutes of the national TB TWG suggests recognition of the importance of utilizing a community-based volunteer model to strengthen care and prevent patient drop-off during treatment (NTLP TWG, 2021a, 2021b). Finally, ETB has successfully had the financing costs of the integrated sputum sample courier system into the MOH’s MTEF31, which indicates that more commitment 30 Source: KII with two members of ETB senior leadership; KII with member of NTLP national leadership; FGDs with CBVs in Central, Copperbelt, and Luapula Provinces. 31 Source: KII with member of NTLP national leadership. USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 29 | P a g e may be forthcoming around funding (or at least cost-sharing) for an integrated courier system that addresses both TB and HIV/AIDS clinical needs. Conclusion: Engaging the MOH/NTLP on a day-to-day basis and ensuring that opportunities are provided for officials to observe and interact with “what works” in the field, is a foundation for sustainability. Site activation, the utilization of CBVs, and the opportunistic development of an integrated TB-HIV/AIDS courier system for sample collection and delivery, are all innovations that have been established as central to ETB’s success. These are all innovations that have now been observed and engaged with directly by MOH officials at all levels and there is now momentum to maintain them as a result. 2) Finding: With the facilitation of the consultative “situation rooms”, ETB has contributed to an institutionalization of data-focused peer reviews and collaborative decision-making. Available evidence indicates that the weekly TB Situation Room, in which implementors and other stakehodlers in the TB space meet virtually with the NTLP and review available data to identify and discuss emerging trends, bottlenecks, and concerns, will be maintained (NTLP, 2022; WHO, 2022).32 A clear benefit of the weekly situation room has been the opportunities it has provided to MOH officials, particularly those at the provincial level who have core responsibilities for technical oversight, to interrogate data, to speak about data, and to gain practice in utilizing data to feed into policy and program adaptations (MOH, 2020, 2022; WHO, 2022). “The Situation Room has demonstrated real value. It has demonstrated the value of collaboration and of using data in real-time to make decisions. These consultations allow us to plan and not just be reactive to events. We will keep this going” (KII: Member of NTLP senior leadership) “One benefit of consultative meetings [situation rooms] is that they provide people with the opportunity to speak and learn. Maybe not yet at the districts, but at the provincial level, I think we’ve seen real growth in officials’ comfort level talking about very technical TB matters. That should create better approaches and better policy.” (KII: Senior ETB staff member) Conclusion: Improved collaboration, discussion, and data-driven decision-making appear to be positive legacies for ETB. The true sustainability of the weekly TB Situation Room will become clearer in the future. At the moment, however, there appears to be considerable enthusiasm for maintaining this forum, which at least increases the likelihood that policy and practice will be informed by multi-stakeholder consultations and analytical considerations that are based on quality evidence. 4.3.3 CHALLENGES 1) Finding: A notable challenge to the sustainability of ETB’s interventions is the near￾constant turnover of personnel at the district facility-level. Even during activity implementation, this turnover served as a roadblock since it necessitated engaging regularly with new facility in-charges, TB focal points, and district TB coordinators, often to review approaches (e.g., on the use of TB diagnostic tools or on the collection and reporting of data) that had already been discussed at these same facilities with now-departed personnel. With ETB’s completion, it is unclear how much of the technical capacitation provided at district-level (or by provincial-level MOH officials with responsibility for engaging with district counterparts) will be maintained.33 32 Source: KII with member of NTLP national leadership. 33 Source: KII with two members of ETB senior leadership; KII with member of USAID senior leadership; KIIs with 2 provincial TB coordinators in Central and Copperbelt Provinces. USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 30 | P a g e “The problem with building capacity is that the MOH, at facilities, does not make anyone available to build capacity with.” (KII: Member of USAID senior leadership) “There is such a constant churn of personnel at the health facilities. Maybe some of the TA [technical assistance] provided will still be useful if people get transferred to other facilities elsewhere in Zambia. However, there’s been a lot of capacity building with people who might just drop out of the [health care] profession altogether.” (KII: Senior ETB staff member) Conclusion: For activities seeking to generate improved outcomes at district and facility￾levels, sustainability will always be challenged by a high likelihood of attrition/relocation among health care workers. As such, ETB’s aforementioned development of standardized orientation and training guides for new/relocated personnel, is an important innovation. If activities have little choice but to provide almost constantly repeating orientations/trainings, then doing so from a standardized package/approach at least reduces the time burden that would otherwise be required to continuously reinvent the wheel. Over the longer-term, however, new approaches need to be considered to improving staff retention at facilities. 2) Finding: It is unclear how ETB’s well-regarded CBV model will be maintained over the long-term. Both the document review and KIIs pointed to a high level of regard for the CBV model introduced by ETB to facilitate improved patient follow-up and care at community-level (CARE Zambia, 2021; ETB, 2020b, 2021; WHO, 2022).34 The FGDs with CBVs, meanwhile, reflected both a high level of enthusiasm by CBVs for the role they play – which should sustain most in their roles over the short-term – but also a lack of clarity on how they will be supported (if at all) by future USAID or other donor activities. “The CBVs play a very important role in our communities. We make sure patients stick to their treatment regimen, go to their appointments, but we also check in on them to make sure they are doing okay physically and mentally. We take pride in this role.” (FGD: Female CBV, Copperbelt Province) “We will continue in our role. We are enthusiastic. But there aren’t enough of us to provide support for everyone [TB patients] who need it. We will try to recruit others to join us, but we also under if USAID will recruit more [CBVs]? (FGD: Male CBV, Copperbelt Province) Conclusion: As with any initiative focused on mobilizing community volunteers, ways will need to be found to continue incentivizing CBV participation and the recruitment of new members. In the short-term, it is clear that the MOH/NTLP does not have the resources needed to support extensive community-based patient support.35 As such, it will likely fall to future TB programs – whether funded by USAID/Zambia or other donors – to continue managing the CBVs and identifying optimal ways to incentivize existing members while also upping recruitment levels. 34 Source: KII with member of NTLP national leadership; KIIs with Provincial TB Coordinators in Central, Copperbelt, and Luapula Provinces. 35 Source: KII with member of NTLP national leadership; KII with Provincial TB Focal Point, Luapula Province. USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 31 | P a g e 4.4 EVALUATION QUESTION 4: WHAT INSTITUTIONAL CAPACITY HAS ETB BUILT WITHIN THE MOH AND OTHER STAKEHOLDERS TO GENERATE SUSTAINABILITY FOR THE PROCESSES AND OUTPUTS ACHIEVED? 4.4.1 SUMMARY RESPONSE TO EVALUATION QUESTION 4 ETB contributed to the building of institutional capacity within the MOH, albeit to varying degrees and with different likelihoods of long-term sustainability. As the activity concludes, there are a number of areas for which capacity seems to have been effectively developed. The unknown factor is whether the attrition of health care workers at district-level facilities will, over time, dilute the institutional knowledge built up by the activity. At present, key capacity gains identified by the evaluation are: • Improved knowledge and practice across an array of facility health care workers on how to screen patients for TB. Site activation should be a lasting institutional practice to emerge from ETB. Rather than focusing solely on capacitating TB clinicians, the activity worked to sensitize health care workers in all facility departments on how to undertake screening. This should continue allowing for increased screening and case identification. • Improved knowledge on the use of the GeneXpert diagnostic tool. District and provincial laboratory staff were given intensive training by ETB on how to utilize and maintain the GeneXpert machines for improved case diagnosis. • Greater capacity within the MOH/NTLP – particularly at national and provincial levels – to understand, discuss, and utilize data in decision-making. The national TB TWG and weekly NTLP Situation Room are at the forefront of this improved capacity. 5. RECOMMENDATIONS Below are preliminary recommendations emerging from the evaluation: For USAID/Zambia 1) Future TB activities should continue dedicating resources to the expansion of provincial TB elimination committees. These committees are valued by a wide range of stakeholders for the opportunity they provide to develop a decentralized and context-aware TB response. These committees are not yet active in each province, but ETB has provided sufficient “proof of concept” to justify USAID/Zambia dedicating further financial and technical support towards establishing and strengthening these bodies (this recommendation addresses the conclusion for Evaluation Question #1, Finding #1). 2) Ongoing and future TB activities (and other Health activities) should adopt/ continue utilizing a zonal approach to capacity strengthening. ETB achieved a notable success in introducing a trainer-of-trainers (ToT) model in which select facility-based personnel would be trained on improved practices around pediatric TB – and then go on to serve as trainers for other staff in their facilities and other satellite facilities in their established “zones”. This approach has had a clear effect in allowing technical assistance to reach as broad an audience as possible. In the context of limited resources being available to ensure the provision of direct technical assistance to lower-level facilities, a zone-focused ToT is an approach worth continuing and worth utilizing in other program contexts (this recommendation addresses the conclusion for Evaluation Question #1, Finding #2). USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 32 | P a g e 3) Ongoing and future USAID/Zambia TB activities – as well as USAID/Zambia Health activities in the HIV/AIDS space – should continue co-investing in the maintenance of the integrated courier system for sample collection and delivery. Eventually, an effective courier system will need to be institutionalized within the MOH rather than being donor-funded. The inclusion of motorcycle maintenance and fuel costs in the MTEF may be a first step towards this objective. For the time being, however, TB activities’ ability to access an effective courier system will continue to depend on their ability to “piggyback” on systems utilized by HIV/AIDS activities. This is an important area of coordination for USAID/Zambia’s Health Office (this recommendation addresses the conclusion for Evaluation Question #1, Finding #5). 4) Future USAID/Zambia TB activities should develop private sector engagement strategies at the outset of implementation. These plans should be based on a clear understanding of the private sector landscape (types of companies, their existing involvement in public health/TB initiatives, and their corporate social responsibility objectives). They should identify clear medium and long-term objectives, be aligned (as appropriate) with USAID’s own PSE strategy and should be tailored to identifying clear sector-specific (and even firm/association-specific) engagement approaches. By establishing these plans early in project implementation, they can provide a consistent guide to PSE that can be maintained in the event of staff turnover and/or adapted in the face of unexpected shocks (e.g., COVID-19). ETB’s PSE strategy, while developed too late to be of much practical value for the project, did provide a thorough overview of potential private sector partners across multiple focus areas and at province and even district-levels. As such, this strategy should serve as a strong foundation to support future TB programs develop their PSE strategies (this recommendation addresses the conclusion for Evaluation Question #1, Finding #7, and Evaluation Question #2, Finding # 5). 5) Ongoing and future USAID/Zambia TB activities should work with the MOH to develop a facility-focused staff retention strategy. High levels of attrition at district-level health facilities serves as a notable risk to the sustainability of improved TB practices and outcomes. Indeed, improved capacity on several fronts – whether in the use of the GeneXpert diagnostic tools, the application of appropriate TB screening processes, or in methods to collect and report quality data, can be lost if trained staff simply drop out of the health care system. Future TB activities should focus on working with the MOH on how best to retain trained staff with TB expertise (this recommendation addresses the conclusion for Evaluation Question #2, Finding #1). 6) Future USAID/Zambia TB activities should ensure the adequate allocation of sufficiently trained activity technical staff at district-level. ETB’s decision to allocate its technical staff at provincial level and within only a small number of “high volume” districts, was a pragmatic reflection of the need to control staffing costs in a geographically expansive activity. However, USAID/Zambia’s future TB activities, recognizing that the bulk of activity implementation takes place at the district (and community) levels, should ensure that enough staff (particularly, clinical officers) are allocated to the districts to allow for consistent engagement with the MOH’s district TB coordinators. In defining their idealized staffing structure, senior ETB personnel noted a preference for a clinical officer to be present in each district, likely being housed in DHOs or (preferably) in key district health facilities. Provincial-level coordinators, ideally medical doctors providing technical oversight, could then be housed in PHOs or main provincial health facilities, as could 1-3 M&E staff (depending on the size of the province). Clearly, this preferred staffing structure was not practical for ETB since it would have necessitated the hiring of technical personnel to cover 63 districts. A prioritization of district-level staffing may USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 33 | P a g e thus necessitate a trade-off in the geographic scope of any future TB project. Focusing on all districts within fewer provinces, or fewer districts within a larger number of provinces, could be worthwhile if doing so improves day-to-day coordination with the MoH and improves the quality of interventions (this recommendation addresses the conclusion for Evaluation Question #2, Finding #2). 7) Future USAID/Zambia TB activities should consider the professional profiles and experience levels of sub-national staff assigned to work with the MOH at district and facility-levels. The evaluation’s KIIs clearly noted that the MOH effectively plays a coordinating role at the district and facility-levels. Where MOH expertise is sometimes lacking is in the technical implementation of select activities. As such, project staff should ideally have levels of technical knowledge and expertise that allow them to be effective in building – rather than simply reinforcing – district-level capacity. This may require a more judicious approach to project hiring and/or a careful consideration of project salary structures if it is necessary to hire more experienced clinical staff to be based at provincial-level and/or be allocated to district-level work (this recommendation addresses the conclusion for Evaluation Question #2, Finding #3). 8) Future USAID/Zambia TB activities should allocate more staffing resources towards M&E and should put in place a Knowledge Management lead. Ideally, M&E staff should be dedicated mainly to the analysis and interpretation of activity data rather than spending the majority of their time chasing absent or late data from the field. For this to be possible, there needs to be sufficient M&E coverage to provide consistent support at district-level. This likely does not mean an M&E staff member being seconded to each district but potentially M&E staff being assigned to “zones” (clusters of districts) rather than being primarily Lusaka or provincially based (this recommendation addresses the conclusion for Evaluation Question #2, Finding #4). For the MOH/NTLP 9) The MOH should continue emphasizing site activation as a central component of its approach to increasing TB screening. This means establishing clear expectations for health facilities and putting in place a monitoring mechanism to ensure that facilities transition to a fully site activation approach. Site activation has become institutionalized in the new NSP covering the period 2022 to 2026. In theory, this should guarantee that this approach becomes central to the policies of the MOH/NTLP and to the implementing approaches of USAID/Zambia IPs and other implementors. However, the translation of site activation from being a desired policy to becoming an institutionalized practice is likely to be fraught. The MOH/NTLP should develop clear guidelines for how site activation will be pursued, a timeline, as well as a monitoring mechanism to track facility performance in transitioning to a system in which all facility staff are trained to screen patients for TB (this recommendation addresses the conclusion for Evaluation Question #1, Finding #3). 10) The MOH should ensure that if district-level health facilities are going to be at the forefront of data collection and reporting, and if public health policy is to be made based on these data, that facilities are equipped with the resources they require to play this role. Computers, tablets, and quality internet connectivity are costly and challenging to deliver, particularly in remote areas. However, without these tools, health facilities cannot realistically be expected to serve as the main cog in the timely reporting of health data. Discussions should be held between the MOH and USAID/Zambia and opportunities for direct financial support in this area (this recommendation addresses the conclusion for Evaluation Question #1, Finding #8). USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 34 | P a g e ANNEX 1: EVALUATION STATEMENT OF WORK A list of acronyms can be found on the last page of the SOW. BACKGROUND USAID/Zambia’s Eradicate Tuberculosis (ETB) Activity works in partnership with the Ministry of Health’s (MOH’s) National Tuberculosis and Leprosy Program (NTLP), the private sector, and other partners at the community, facility, district, provincial, and national levels to strengthen the implementation of key components of the national response to control Tuberculosis (TB) across Zambia. Working in six provinces – Central, Copperbelt, Luapula, Muchinga, Northern, and North-Western – and covering 63 districts and 341 health facilities, ETB’s goal is to reduce TB-related mortality by 50% from an established 2016 baseline of 5%. To meet this goal, ETB has two key objectives, both of which are expected to strengthen NTLP’s efforts to achieve TB elimination by 2030: 1) To increase the number of individuals screened by TB by 50% from 2016 baseline levels. 2) To increase the TB treatment success rate to at least 85% in all target provinces. The rationale for ETB is rooted in Zambia’s high rate of TB infection and TB/HIV co-infection. According to the World Health Organization’s (WHO) 2019 Global Tuberculosis Report, Zambia ranks among the top 30 countries in the world on both measures. The WHO estimated Zambia’s TB incidence in 2019 to be 333 per 100,000 people (equal to 60,000 TB patients). Meanwhile, the WHO’s 2019 data indicated SUMMARY EVALUATION OBJECT USAID/Zambia Eradicate Tuberculosis Activity (ETB) CONTRACT NUMBER Eradicate TB Activity_AID-611-C-17-00003 TYPE OF EVALUATION Performance Evaluation ACTIVITY START & END DATE September 29, 2017 – September 30, 2022 ACTIVITY BUDGET USD 21,467,448 PRIME IMPLEMENTING PARTNER PATH NAME OF SUBCONTRACTORS: Afya Mzuri; Community Initiative for Tuberculosis, HIV/AIDS, and Malaria Plus Related Diseases (CITAM+); and the Zambia AIDS Related Tuberculosis (ZAMBART) project MAJOR COUNTERPART ORGANIZATIONS Government of the Republic of Zambia (GRZ) (Ministry of Health/National Tuberculosis and Leprosy Program) GEOGRAPHIC COVERAGE Central, Copperbelt, Luapula, Muchinga, Northern, North￾Western provinces (63 districts across these provinces) EVALUATION TIMELINE March 2022 – August 2022 USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 35 | P a g e that TB mortality was 33 per 100,000 in HIV-negative TB patients and 53 per 100,000 in HIV-positive TB patients. Although Zambia is still considered a low-burden country for drug-resistant TB (DR-TB), in 2019 the country recorded its first extensively drug-resistant (XDR-TB) TB patient, as well as 515 confirmed DR-TB patients, of whom 97% initiated treatment. There have been some successes in Zambia around TB outcomes. The TB treatment success rate at the national level is 90%. Most TB patients know their HIV status and 97% of co-infected patients start antiretroviral therapy (ART). The treatment success rate for DR-TB patients has increased – from 71% in 2018 to 76% in 2019. Nevertheless, NTLP has outlined the urgency of a continued focus on national TB control, stating in its National Strategic Plan (NSP) for Tuberculosis Prevention, Care and Control (2017-2021) an objective of moving “towards elimination”. The NSP takes a pro-active approach, also applied by ETB, to identifying missing TB cases. This marks a move away from the more passive case-finding approach that was previously applied. ETB THEORY OF CHANGE & APPROACH ETB’s Theory of Change (ToC), represented in Figure 1, holds that to contribute to the eradication of TB in Zambia through increased screening and improved treatment, it is necessary to strengthen collaboration and coordination, as well as harness existing technical skills among Government of the Republic of Zambia (GRZ) stakeholders, United States Government (USG) partners and other donors, the private sector, civil society, medical professionals, and community leaders and volunteers. Doing so will maximize investments, promote innovation, and produce feasible, gender-equitable, and evidence-based interventions. These interventions will focus on finding missing cases, improving diagnostic networks, and improving outcomes for all persons with all forms of TB – with a specific focus on populations with high TB burdens, including underserved communities, People Living with HIV (PLHIV), children, and people in congregate settings such as prisons. To increase the number of individuals screened by TB by 50% from 2016 baseline levels, ETB focuses on finding missing cases through improving diagnostic networks, expanding active and targeted community￾and facility-based case finding, supporting the strengthening of NTLP’s Monitoring and Evaluation (M&E) systems, and facilitating multi-sector involvement in TB control. This is done through three tasks: • Task 1: Pursue high-quality directly observed therapy, short-course (DOTS) expansion and enhancement. • Task 2: Increase access to TB/HIV and Multidrug-resistant-TB (MDR-TB) services for poor and vulnerable populations in target provinces. • Task 3: Engage all categories of care providers. To increase the TB treatment success rate to at least 85% in all target provinces, ETB focuses on standardizing treatment, support, and care; increasing capacity for treatment of drug-susceptible and drug￾resistant TB and TB/HIV co-morbidity; improving the effectiveness and quality of pediatric TB services; and supporting all categories of care providers to adhere to national guidelines. This is done through four tasks: • Task 1: Pursue high-quality DOTS expansion and enhancement. • Task 2: Increase access to TB/HIV and MDR-TB services for poor and vulnerable populations in target provinces. USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 36 | P a g e • Task 3: Engage all categories of care providers. • Task 4: Intensify research and innovation. At the end of FY21, ETB reported having successfully met its first life-of-project (LOP) objective to increase screening of individuals for TB by 50% from the 2016 baseline level, achieving 1,543% coupled with an increase of 226% in TB notifications in FY21. Regarding the LOP objective to increase the TB treatment success rate above 85% in all target provinces, ETB in FY21 surpassed the target, with all six provinces reporting a minimum of 90%. Of significance were the cure rates, which were above 80% in four of the six target provinces, reflecting the quality of treatment success. The goal of the ETB project is to reduce TB mortality by 50% from the baseline of 5% in 2016 to 2.5% at the end of the project in May 2022. Through the interventions highlighted above, the project, in collaboration with NTLP at all levels, have, by the end of FY21, reduced the TB mortality in the six provinces to 3.6%; this translates to a 28% reduction. ETB PARTNERSHIP ROLES AND RESPONSIBILITIES The ETB consortium consists of four organizations, led by PATH: Afya Mzuri; Community Initiative for Tuberculosis, HIV/AIDS, and Malaria Plus Related Diseases (CITAM+); and the Zambia AIDS Related Tuberculosis (ZAMBART) project. The roles and responsibilities of each organization are summarized below: • PATH provides overall management, financial, and technical oversight in addition to M&E and strategic information support. Moreover, PATH leads corporate engagement, gender-related, public-private mix, DR-TB, TB/HIV co-infection, and pediatric TB capacity-building activities. PATH also works to increase case detection through quality-assured microscopy and other diagnostic methods in addition to the provision of technical assistance to the National Tuberculosis Reference Laboratory (NTRL) to strengthen its capacity to process specimens and improve turnaround time. • AFYA MZURI is responsible for community TB training, mobilization, awareness-raising activities, and community-level case finding and treatment support. • CITAM+ is responsible for advocacy and activities aimed at increasing financing for TB control efforts from the GRZ, private sector, and international organizations. CITAM+ also supports capacity￾building for community-based volunteers (CBVs) to improve treatment literacy, including treatment counselling, adherence support, and stigma reduction. • ZAMBART promotes research to optimize implementation, impact, and innovation through training and mentoring staff in operational research (OR). ZAMBART also leverages TB research for strategic partnerships with the private sector, the GRZ, and Zambian research institutions. USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 37 | P a g e Figure 1: ETB Logic Model DISCOVER-Health SmartCare Clinical care teams DHIS2 Patient-centered approach TB-related mortality reduced by 50 percent in the intervention provinces by 2022 Internalize and apply emerging lessons to address challenges Knowledge-sharing fora Community dialogue Multi-channel communication Documentation and dissemination of learnings Promotion of local ownership Number of individuals screened for TB in target provinces increased by 50 percent TB treatment success increased to >85 percent GRZ at all levels; USG partners; The Global Fund to Fight AIDS, Tuberculosis, and Malaria; World Bank; private sector; civil society; pharmacists; private providers; traditional leaders; community volunteers SAFE Neighborhood health committees EQUIP World Bank CIDRZ Leverage existing platforms to sustain results Engage all categories of care providers Engage private medical practitioners Engage pharmacists Engage community-based volunteers Engage traditional leaders Intensify research and innovation Build capacity in operational research and establish national TB operational research priorities Develop strategic partnerships for funding of operational research Pursue high-quality DOTS expansion and treatment Increase political commitment Increase TB case detection through improved diagnostics Standardize TB treatment, support, and care Support the existing NTLP monitoring and evaluation system Increase access to TB/HIV and MDR-TB services for poor and vulnerable populations Reduce the burden of TB/HIV comorbidity Strengthen systems to halt DR-TB expansion Detect and treat TB among children and other vulnerable groups Provide support to increase TB detection and treatment among prisoners Achieve gender-equitable access to quality-assured TB services Monitor, evaluate, and use data for decision-making and continuous improvement Strengthen collaboration and coordination Maximize investments and address challenges Target evidence-based interventions Achieve results USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 38 | P a g e PURPOSE The evaluation is required, as per ADS 201. The performance evaluation will primarily serve USAID/Zambia and its implementing partners (IPs) by informing the continuation of interventions under an existing USAID activity, as well as informing the subsequent design of a new activity, scaled￾up to all provinces, with the goal of eradicating TB. USAID/Zambia will also share the evaluation publicly to account for the public funding used by the activity. The MOH, and particularly the NTLP, may become a secondary user of evaluation findings to inform longer-term policy change. EVALUATION QUESTIONS The following are an initial set of questions that must be addressed if the evaluation is to meet its intended purpose. They will inform clear and actionable recommendations to serve the intended use of the evaluation. The evaluation team is expected to finalize initial evaluation questions and break them down into an evaluation matrix during the inception phase of the assignment. Time allowing, a co-creation process may also be introduced to allow all potential users of the evaluation, including the MOH, to feed into refining these initial evaluation questions. 1) What lessons learned and/or best practices should be taken from ETB to inform the implementation of other (existing) TB-focused activities and the design of new activities focused on TB eradication? 2) Were the ETB staffing and partnership structures optimal for achieving project objectives? 3) To what extent are ETB methodologies, interventions, and management setting the stage for future sustainability of project outputs and outcomes? 4) What institutional capacity has ETB built within the MOH and other stakeholders (private sector, Zambian research institutions) to generate sustainability for the processes and outputs achieved? All questions listed above should need be reflected against three cross-cutting issues, namely: i) equity (incl. gender), ii) localization, and iii) anti-corruption. METHODS During the inception phase of the assignment, the evaluation team will use the evaluation matrix to outline the specific data collection methods that will be employed to address each of the evaluation questions and sub-questions. It is also expected that the evaluation team will detail the strengths and weaknesses of the methods proposed, including around sampling and/or the selection of study participants (e.g., as K\key informants), as well as the implications of these on how the evaluation findings can be used. Further, the evaluation team will produce a data analysis plan that clearly demonstrates how the data collected through each method will be analyzed, how different analysis methods can be used for triangulation, and the types of data visualizations that will be used to represent findings. With the above in mind, it is expected that the evaluation will use a mix of qualitative and qualitative methods collection methods to balance strengths and weaknesses. The methods selected should enable the evaluation to effectively engage with a wide range of ETB stakeholders. Suggested methods include: USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 39 | P a g e Desk Review: The evaluation team will conduct a comprehensive review of ETB Activity documentation. A full list of relevant documents will be identified in collaboration with USAID in the inception phase of evaluation. An initial list includes: • Activity annual reports and work plans. • Activity financial records. • Activity M&E reports and monitoring data. • Training records. • Meeting minutes (e.g., of coordination meetings between consortium partners). • Operational research reports. • MOH bulletins and other TB-focused reports. • GRZ policy documents (e.g., produced by the Parliamentary Caucus on TB). In addition to identifying the processes and outputs achieved by ETB, it is expected that the findings from the document review will feed into both the potential refining of the evaluation questions, as well as the final selection of methods outlined in the evaluation matrix. Key Informant Interviews (KIIs) – in-depth KIIs, carried out with several ETB stakeholders, are expected to form the foundation of the evaluation’s collection of primary data. Stakeholders to be engaged include (but are not necessarily limited to) representatives from: • PATH and other members of the implementing consortium. • MOH (at the national, provincial and district-levels). • NTLP – including members of the National TB technical working group. • District-level health facilities (those healthcare workers directly involved in testing and screening TB patients, as well as TB focal point persons). • National Tuberculosis Reference Laboratory. • Community-based volunteers (CBVs). • Private pharmacies/drug stores providing TB case identification and referral services. • Private health facilities engaged in TB case detection and management. • Private sector firms and national-level sector organizations (e.g., Zambia National Farmers Union, Zambia Chamber of Mines) engaged in the public-private partnership for TB elimination. • The Parliamentary Caucus on TB. Focus Group Discussions (FGDs) – the evaluation team is expected to consider utilizing FGDs as a data collection method with select types of respondents. These include (but are not necessarily limited to): • Community-based volunteers (CBVs). • Healthcare workers at district-level health facilities. • TB focal point persons at district-level health facilities. Structured Observation – to the extent practicable, the evaluation team will be expected to engage in site visits to project-supported health facilities, pharmacies/drug stores, and laboratories (as well as other identified sites as relevant). Structured Observation will focus on components of ETB such as (but not limited to): 1) project mentoring of health facility workers to carry out TB screening and pediatric TB capacity-building; 2) community TB training, mobilization, awareness-raising activities; and 3) capacity-building for CBVs. USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 40 | P a g e Quantitative Survey: To strengthen representativity and maximize opportunities for stakeholders to lend their voice to the evaluation, a quantitative survey will complement the qualitative data collection methods. The survey will target service and care providers. As such, it is expected that the survey can be administered either via computer-assisted phone interviews or an online tool; or a combination of these modes. This assumes the ETB Activity does maintain a relevant contact list. It is required that the evaluation team upholds the highest standards of ethical conduct in the design and implementation of the evaluation. This includes working with USAID/Zambia to obtain required approvals, including bio-ethical clearance from a relevant ethical committee, prior to starting data collection. It is also expected that the evaluation will obtain informed consent from all participants to both collect and use data, and that the evaluation will ensure the anonymity and confidentiality of respondent data unless informed consent is provided by study participants to identify them by name. DELIVERABLES AND TIMELINES The following deliverables will be submitted to USAID/Zambia following the assigned timelines in the schedule presented at the end of this SOW. 1. D1: A theory of change workshop in the inception phase will help elicit causal hypotheses and context assumptions to inform the evaluation. 2. D2: An inception report that fully details the proposed evaluation approach, inclusive of an evaluation matrix, data collection tools, and data analysis plan that clearly details how each evaluation question will be addressed. The inception report will also clearly highlight the strengths and limitations of the proposed methods, as well as present a dissemination plan showing how the final evaluation outputs will be accessed and utilized by the evaluation’s users. The inception report may draw upon preliminary interviews with these users to further refine the evaluation questions and methods. 3. D3. A preliminary findings presentation will be held after the initial analysis of the data. The discussion will inform the draft evaluation report. 4. D4a & D4b. A draft and final evaluation report that fully addresses each of the evaluation question, outlines a dissemination plan, and which provides clear recommendations for how the evaluation findings can be used by USAID/Zambia and its implementing partners. 5. D5. A series of recommendation briefers (2-5 pages) that complement the evaluation report will be developed to support evaluation use by providing concise and action-driven overviews of individual recommendations. A maximum of 5 briefers will be produced for selected key recommendations. 6. D6a & D6b. A post-evaluation action plan that details clear and practical steps that USAID/Zambia and its implementing partners can take to action the recommendations of the evaluation. It is expected that the action plan will be discussed and validated at an in-person workshop with ETB and a broader USAID audience. The final version of the evaluation report, not exceeding 35 pages, must be submitted to USAID/Zambia electronically in Microsoft Word and PDF. The report must be written in English, single-spaced in Gills Sans MT, size 11 type fonts. The structuring of the report should generally align with the structure outlined below, though this will be finalized during the evaluation’s inception phase. USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 41 | P a g e • Executive summary (2-5 pages) that summarizes key points that will include purpose and background, evaluation questions, methods, findings, and conclusions. • List of acronyms • Table of contents • Evaluation purpose and questions (1-2 pages) • Project background (1-2 pages) • Methods and limitations (2 pages) • Findings (15 pages) • Conclusions (1 page) • Recommendations (1-2 pages) • References (1-2 pages) • Annex, which will include but is not limited to the evaluation statement of work, detailed description of the evaluation design and methods, copies of data collection tools, a list of information sources (including documents reviewed, sites visited, and key informants), and disclosure of any conflict of interest by evaluation team members. Upon completion of the evaluation, all data and materials are to be surrendered to and will remain the property of USAID. TEAM COMPOSITION A Team Lead will have overall responsibility for overseeing the implementation of the evaluation, as well as providing quality assurance at each stage of the design, roll-out, analysis, and reporting phases. The Team Lead will be expected to coordinate the timely submission of all expected deliverables and to be the point of contact for USAID/Zambia and Z-MELP throughout the evaluation. The Team Lead will have the public health and evaluation expertise required to lead the design of the evaluation, including the final selection of methods and recommendation of these to USAID/Zambia and Z-MELP. The Team Lead will also have expertise in analyzing findings pertaining to the implementation of public health programs, and TB programs in particular, and will be responsible for drafting the final report with an eye towards providing practical recommendations to inform ongoing and future TB interventions. The Team Lead will work alongside a Methods Specialist and Local Health Evaluation Expert, a Zambian national with extensive evaluation and programming experience in the country’s public health sector. The Local Expert will have the responsibility for collecting and analyzing primary data, as well as contributing to the drafting of the evaluation report. Below are the minimum qualifications and experience required of the Team Lead and the Local Expert: TEAM LEAD • The role is open to international applicants. • Advanced degree in Public Health or a related field. USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 42 | P a g e • Demonstrated experience in the evaluation of health programs, with direct experience on TB￾focused evaluations being preferred. • Demonstrated technical expertise in health systems programming/M&E in sub-Saharan Africa. • Strong qualitative data collection and analysis skills, with proven experience carrying-out in￾depth KIIs. Proficiency in relevant data analysis software, such as NVivo, is preferred. • Previous experience undertaking evaluations for USAID is preferred. • Knowledge of USAID program cycle and evaluation processes and policy. • Excellent communication, writing, facilitation, and presentation skills. • Excellent report writing skills. • Fluency in spoken and written English required. LOCAL HEALTH EVALUATION EXPERT • Must be a Zambian national with an advanced degree in a field related to Public Health. • Demonstrated experience in undertaking evaluations of public health programs, with direct experience related to TB programming being preferred. • Demonstrated technical expertise in health systems or TB programming in Zambia is preferred. • Strong qualitative data collection and analysis skills, including with proficiency in relevant software such as NVivo. • Previous experience undertaking evaluations for USAID is preferred. • Excellent communication, writing, facilitation, and presentation skills. • Fluency in spoken and written English required and at least any two local languages spoken in Lusaka, Central and Copperbelt i.e., Bemba and Nyanja. The locally based Z-MELP team will provide technical and logistical support, and quality assurance throughout the evaluation process. The Team Lead and Local Expert will liaise regularly with Z￾MELP’s Chief of Party, Deputy Chief of Party, and Survey Manager. They will also be in regular communication with USAID/Zambia staff. SCHEDULING AND LOGISTICS The table below outlines the tentative timeline for completing all stages of the evaluation. It is expected that the timeline will be finalized during the inception phase of the assignment. W2 W3 W4 W1 W2 W3 W4 W1 W2 W3 W4 W1 W2 W3 W4 W1 W2 W3 W4 W1 W2 W3 W4 Hire of consultants Theory of Change workshop D1 Inception Report D2 Ethical clearance (IRB) Pretest & Revise tools Data collection Data cleaning, analysis Preliminary findings presentation D3 Develop report D4a D4b Series of recommendation briefers D5 Post evaluation validation workshop D6a Post-evaluation action plan D6b June July August Task March April May Evaluation Schedule USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 43 | P a g e BUDGET The budget below presents initial cost estimates only. A final budget will be prepared once the technical SOW has been agreed upon. INITIAL BUDGET ESTIMATES ITEM DESCRIPTION COST (USD) Labor – SoCha Home Office Covered by Z-MELP core funds $ 0.00 Labor – Z-MELP locally based team Covered by Z-MELP core funds $ 0.00 Labor – Consultants Team Leader, Methods Specialist, National Health Evaluation Expert, Enumerators $ 81,000.00 Travel & Per diem Travel to Zambia for team leader, in-country travel team leader, nat. health evaluation expert, Z-MELP staff $ 40,000.00 Other Direct Cost Workshops, car hire, travel allowance graphic design $ 25,000.00 Indirect Cost As per NICRA $ 52,000.00 TOTAL $ 198,000.00 USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 44 | P a g e ANNEX 2: EVALUATION FRAMEWORK EVALUATION FRAMEWORK Design Level / Evaluation Objective (EO) Broad Evaluation Question Sub-Questions Data Type Measure of Indicator Data Source Potential Informant Relevance 1. What lessons learned and/or best practices should be taken from ETB to inform the implementation of other (existing) TB￾focused activities and the design of new activities focused on TB eradication? 1.1. What was the role of your organization in the implementation of the ETB project? (Probe: • What in your opinion worked well for you? • Are there aspects of this model that you think could have worked better had they been planned differently?) Qualitative Stakeholders roles in the implementation of the ETB project Project reports; KII; FGDs Partners 1.2.Did the new responsibility add an extra challenge towards workload? (Probe: Was there support given to you towards the implementation of the activities from the project?) Qualitative Stakeholders engagement and support during implementation of the ETB project Project reports; KII; FGDs Partners 1.3.What key results would you point out resulting from implementation of ETB project? Qualitative Stakeholders opinions on the results of ETB project Project reports; KII; FGDs Project staff HF staff MOH officials Partners 1.4.What lessons can be drawn in terms of your roles on the ETB project? (Probe: Lessons in capacity building, approaches to TB case finding and management; acceptance; Qualitative Overview of stakeholders’ opinions on lessons learned from implementation of the ETB project Project reports; KII; FGDs Project staff HF staff MOH officials Partners USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 45 | P a g e Design Level / Evaluation Objective (EO) Broad Evaluation Question Sub-Questions Data Type Measure of Indicator Data Source Potential Informant sustainability; institutional development etc.) 1.5.What best practices can be drawn during implementation ETB project? Qualitative Overview of stakeholders’ opinions on best practices from implementation of the ETB project Project reports; KII; FGDs Project staff HF staff MOH officials Partners 1.6.What major challenges did you encounter during the implementation of this program? Qualitative Overview of stakeholders’ opinions on challenges encountered during implementation of the ETB project Project reports; KII; FGDs Project staff HF staff 1.7.Where there any unintended outcomes from the USAID assistance? Qualitative Overview of stakeholders’ opinions on unintended outcomes from implementation of the ETB project Project reports; KII; FGDs MOH officials Efficiency & Effectiveness 2. Were the ETB staffing and partnership structures optimal for achieving 2.1. How optimal were the staffing in implementation of the ETB project? (Probe: • Were the numbers of staffing adequate? • Did the staff have adequate capacity to implement the ETB project? Qualitative Overview of staffing and their capacity in implementation of the ETB project, limitations and recommendations for optimal staffing for Project reports; KII; FGDs Project staff MOH officials Partners USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 46 | P a g e Design Level / Evaluation Objective (EO) Broad Evaluation Question Sub-Questions Data Type Measure of Indicator Data Source Potential Informant project objectives? • What mechanism was in place to ensure capacity assessment is done and feedback provided to ETB project staff? • Were there any challenges/limitations in staffing and their capacity? If Yes, what could have been the cause? • What in your opinion should be the optimal staffing and their capacity for achieving success in ETB project?) achieving success in ETB project 2.2. How optimal were the partnerships in implementation of the ETB project? (Probe: • What kind of partnerships were formed? • How useful were these different kinds of partnerships in the implementation of programs such as ETB? • How useful were the partnerships between government and donors in the implementation of programs such as ETB?) • Were there any challenges/limitations in forging partnerships? If Yes, what could have been the cause? • What in your opinion should be the optimal partnerships for achieving success in ETB project?) Qualitative Overview of forged partnerships in implementation of the ETB project, limitations and recommendations for optimal partnerships for achieving success in ETB project Project reports; KII; FGDs Project staff MOH officials Partners 2.3. Did the ETB project integrate gender dimensions and contribute to the advancement of gender equality? (Probe: Qualitative Integration of gender dimensions and how the project contributed to the Project reports; KII; FGDs Project staff USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 47 | P a g e Design Level / Evaluation Objective (EO) Broad Evaluation Question Sub-Questions Data Type Measure of Indicator Data Source Potential Informant • Did the project ensure equal opportunity for women and men in the management and implementation arrangements of the project? • Was the capacity building activities accommodate the different education/skill levels that may exist between women and men? How? advancement of gender equality Effectiveness & Sustainability 3. To what extent are ETB methodologies, interventions, and management setting the stage for future sustainability of project outputs and outcomes? 3.1. Kindly describe the current TB landscape in Zambia. (Probe: • How much of a burden is TB and who is most affected? • How many donor projects are currently working on TB in the country? • Which donor communities currently fund TB programs to the Zambian government? • To what extent has donor-implemented projects addressed TB eradication?) Qualitative Quantitative TB landscape in Zambia Project reports; KII; FGDs MOH officials 3.2. What government plans/polices/strategies exist to address TB in Zambia? (Probe: • Do you have officers at lower levels to actualize these polices and if so, how does this happen? • How do the lower organs of the government get to actualize the government plans? Qualitative An overview of Government plans/polices/strategies exist to address TB in Zambia Literature review, Project reports; KII; FGDs MOH officials USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 48 | P a g e Design Level / Evaluation Objective (EO) Broad Evaluation Question Sub-Questions Data Type Measure of Indicator Data Source Potential Informant • Did the project contribute sufficiently to government strategic policy goals? • What were the particular features of the Strategy that made a difference?) 3.3. What is the TB eradication structure for the ministry and how did it fit in the ETB project? (Probe: • Can the collaboration be better improved to achieve results and for sustainability? If yes, how? • Were the ministry staff sufficiently trained and supported to achieve project implementation? • Are there factors that limited implementation? • What recommendations do you have for this particular aspect?) Qualitative Overview of TB eradication structure for the ministry and how it fits in the ETB project Literature review, Project reports; KII; FGDs MOH officials 3.4.How much involved is the government in the design and implementation of Non￾government Organizations projects? (Probe: What does this mean in terms of the success of the project?) Qualitative Involvement of the Government in the design and implementation of NGO’s projects KII; FGDs MOH officials 3.5.To what extent was the ETB project aligned to the TB NSP? Qualitative Alignment of the ETB project to the TB NSP Literature review, Project reports; KII; FGDs Project staff USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 49 | P a g e Design Level / Evaluation Objective (EO) Broad Evaluation Question Sub-Questions Data Type Measure of Indicator Data Source Potential Informant 3.6.What new responsibilities were added to you after being identified as the focal point person for ETB activity? (Probe: • Where were you trained to perform these responsibilities? • How differently are you managing TB cases now when compared to how you used to manage them before this project? • How sustainable are the newly introduced interventions beyond the life of this project?) Qualitative Engagement of ministry staff in implementation ETB activities Project reports; KII; FGDs HF staff 3.7. If given the mandate to plan the implementation of your specialty on TB eradication (Probe: • How differently would you plan and implement the activities? • What interventions would you recommend that we focus on to achieve greater results?) Qualitative Recommendations from ministry staff on planning of TB eradication projects KII; FGDs HF staff MOH officials 3.8.If we were to plan a program intervention targeted at TB eradication, what interventions would you recommend that we focus on to achieve greater results? (Probe: How would these interventions differ from ETB intervention Qualitative Recommendations from project staff on planning of TB eradication projects KII Project staff Partners USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 50 | P a g e Design Level / Evaluation Objective (EO) Broad Evaluation Question Sub-Questions Data Type Measure of Indicator Data Source Potential Informant or interventions implemented by the NTP?) 3.9.To what extent did ETB project management structures contribute towards realization of the project objectives? Qualitative Contribution of management structures towards achievement of objectives. Project reports; KII; FGDs; Success Stories. Project staff 3.10. What systems were put in place to ensure sustainability of the project achievements post project period? Qualitative Availability of Sustainability Plan Sustainability Plan, KII; FGDs. Project staff 3.11. Would different sites (i.e., facilities, districts, province) require different perspectives for program integration? (Probe: How?) Qualitative Localization of approaches for program integration KII; FGDs. Project staff MOH officials 3.12. Did the project develop gender-specific targets or performance indicators that track gender results and impact? (Probe: List of the indicators?) Qualitative Inclusion of gender￾specific targets or performance indicators Gender mainstreaming policy; Project reports; KII Project staff 3.13. How did the ETB project safeguard against corruption? (Probe: Anti-corruption measures such as transparency, Accountability, Prevention and Education). Qualitative Inclusion of measures to safeguard the project against corruption Anti-corruption policy; Project reports; KII Project staff USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 51 | P a g e Design Level / Evaluation Objective (EO) Broad Evaluation Question Sub-Questions Data Type Measure of Indicator Data Source Potential Informant Sustainability 4. What institutional capacity has ETB built within the MOH and other stakeholders (private sector, Zambian research institutions) to generate sustainability for the processes and outputs achieved? 4.1. Did your work involve working with MoH staff and other stakeholders? (Probe: • Were there tasks/roles you think where not being performed and only got to be done after your coming to this province around ETB? • What are some of those tasks/roles? • Do you think they will be continued if this project comes to an end?) Qualitative Overview of new tasks/roles that were introduced by the ETB project within the MOH and other stakeholders Sustainability Plan, KII Project staff Partners 4.2. What institutional capacity has ETB built within the facility, district, province, and national level to generate continuation of project activities? (Probe: • Will the new capacities continue to generate the kinds of results the project intended? • Are the new capacities likely to be self-sustaining? • Any evidence for sustainability?) Qualitative Overview of institutional capacity that ETB built within the facility, district, province, and national level to generate continuation of project activities Sustainability Plan, KII, FGDs Project staff MOH officials USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 52 | P a g e Design Level / Evaluation Objective (EO) Broad Evaluation Question Sub-Questions Data Type Measure of Indicator Data Source Potential Informant 4.3. What institutional networks has ETB built within the facility, district, province, and national level to generate continuation of project activities? (Probe: • Will the new networks continue to generate the kinds of results the project intended? • Are the new networks likely to be self-sustaining? • Any evidence for sustainability?) Qualitative Overview of institutional networks that ETB built within the facility, district, province, and national level to generate continuation of project activities Sustainability Plan, KII, FGDs Project staff MOH officials USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 53 | P a g e ANNEX 3: DATA COLLECTION TOOLS Tool 1: Project Staff FGD Guide Focus Group Discussion Guides (PATH Provincial: TB, Lab and M&E Specialists) Date of FGD: _____________ (dd/mm/yyyy) Location of FGD: _____________ Name of Facilitator: _____________ Signature of Facilitator: _____________ Name of Notetaker: _____________ Signature of Notetaker: _____________ FGD Start-Time: _____: _____ FGD End-Time: _____: _____ Welcome Good morning/afternoon and welcome to our discussion. Thank you for taking the time to join us today. My name is ________________ and I will be facilitating today’s discussion. Assisting me is ________________, who will be taking notes. We are from the USAID/Zambia Monitoring, Evaluation and Learning Platform (Z-MELP). Z-MELP has been requested by USAID/Zambia to provide independent evaluation services for the USAID/Zambia Eradicate Tuberculosis Activity (ETB) [show participants the introduction letter]. General questions Evaluation Question 1: What lessons learned and/or best practices should be taken from ETB to inform the implementation of other (existing) TB-focused activities and the design of new activities focused on TB eradication? 1. What key results would you point out resulting from implementation of ETB project? 2. What lessons can be drawn from implementation of the ETB project? Probe: Lessons in capacity building, approaches to TB case finding and management; acceptance; sustainability; institutional development etc. 3. What best practices can be drawn during implementation ETB project? 4. What major challenges did you encounter during the implementation of this program? Evaluation Question 2: Were the ETB staffing and partnership structures optimal for achieving project objectives? 5. How optimal were the staffing in implementation of the ETB project? Probe: • Were the numbers of staffing adequate? • Did the staff have adequate capacity to implement the ETB project? • What mechanism was in place to ensure capacity assessment is done and feedback provided to ETB project staff? • Were there any challenges/limitations in staffing and their capacity? If yes, what could have been the cause? USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 54 | P a g e • What in your opinion should be the optimal staffing and their capacity for achieving success in ETB project?) 6. How optimal were the partnerships in implementation of the ETB project? Probe: • What kind of partnerships were formed? • How useful were these different kinds of partnerships in the implementation of programs such as ETB? • How useful were the partnerships between government and donors in the implementation of programs such as ETB?) • Were there any challenges/limitations in forging partnerships? If yes, what could have been the cause? • What in your opinion should be the optimal partnerships for achieving success in ETB project?) 7. Did the ETB project integrate gender dimensions and contribute to the advancement of gender equality? (Probe: • Did the project ensure equal opportunity for women and men in the management and implementation arrangements of the project? Evaluation Question 3: To what extent are ETB methodologies, interventions, and management setting the stage for future sustainability of project outputs and outcomes? 8. To what extent was the ETB project aligned to the TB NSP? 9. If we were to plan a program intervention targeted at TB eradication, what interventions would you recommend that we focus on to achieve greater results? Probe: How would these interventions differ from ETB intervention or interventions implemented by the NTP?) 10. To what extent did ETB project management structures contribute towards realization of the project objectives? 11. What systems were put in place to ensure sustainability of the project achievements post project period? 12. Would different sites (i.e., facilities, districts, province) require different perspectives for program integration? Probe: How? 13. Did the project develop gender-specific targets or performance indicators that track gender results and impact? Probe: List of the indicators?) 14. How did the ETB project safeguard against corruption? Probe: Anti-corruption measures such as transparency, Accountability, Prevention and Education). Evaluation Question 4: What institutional capacity has ETB built within the MOH and other stakeholders (private sector, Zambian research institutions) to generate sustainability for the processes and outputs achieved? 15. Did your work involve working with MoH staff and other stakeholders? Probe: • Were there tasks/roles you think where not being performed and only got to be done after your coming to this province around ETB? • What are some of those tasks/roles? • Do you think they will be continued if this project comes to an end? 16. What institutional capacity has ETB built within the facility, district, province, and national level to generate continuation of project activities? Probe: • Will the new capacities continue to generate the kinds of results the project intended? • Are the new capacities likely to be self-sustaining? • Any evidence for sustainability? USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 55 | P a g e 17. What institutional networks has ETB built within the facility, district, province, and national level to generate continuation of project activities? Probe: • Will the new networks continue to generate the kinds of results the project intended? • Are the new networks likely to be self-sustaining? • Any evidence for sustainability? USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 56 | P a g e Tool 2: Partners Interview Guide Focus Group Discussion Guides (Partners Interview Guide) Date of FGD: _____________ (dd/mm/yyyy) Location of FGD: _____________ Name of Facilitator: _____________ Signature of Facilitator: _____________ Name of Notetaker: _____________ Signature of Notetaker: _____________ FGD Start-Time: _____: _____ FGD End-Time: _____: _____ Welcome Good morning/afternoon and welcome to our discussion. Thank you for taking the time to join us today. My name is ________________ and I will be facilitating today’s discussion. Assisting me is ________________, who will be taking notes. We are from the USAID/Zambia Monitoring, Evaluation and Learning Platform (Z-MELP). Z-MELP has been requested by USAID/Zambia to provide independent evaluation services for the USAID/Zambia Eradicate Tuberculosis Activity (ETB) [show participants the introduction letter]. General questions Evaluation Question 1: What lessons learned and/or best practices should be taken from ETB to inform the implementation of other (existing) TB-focused activities and the design of new activities focused on TB eradication? 1. What was the role of your organization in the implementation of the ETB project? Probe: What in your opinion worked well for you? Are there aspects of this model that you think could have worked better had they been planned differently? 2. Did the new responsibility add an extra challenge towards workload? Probe: Was there support given to you towards the implementation of the activities from the project? 3. What key results would you point out resulting from implementation of ETB project? 4. What lessons can be drawn in terms of your roles on the ETB project? Probe: Lessons in capacity building, approaches to TB case finding and management; acceptance; sustainability; institutional development etc. 5. What best practices can be drawn during implementation ETB project? Evaluation Question 2: Were the ETB staffing and partnership structures optimal for achieving project objectives? 6. How optimal were the staffing in implementation of the ETB project? Probe: • Were the numbers of staffing adequate? • Did the staff have adequate capacity to implement the ETB project? • What mechanism was in place to ensure capacity assessment is done and feedback provided to ETB project staff? USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 57 | P a g e • Were there any challenges/limitations in staffing and their capacity? If yes, what could have been the cause? • What in your opinion should be the optimal staffing and their capacity for achieving success in ETB project?) 7. How optimal were the partnerships in implementation of the ETB project? Probe: • What kind of partnerships were formed? • How useful were these different kinds of partnerships in the implementation of programs such as ETB? • How useful were the partnerships between government and donors in the implementation of programs such as ETB?) • Were there any challenges/limitations in forging partnerships? If yes, what could have been the cause? • What in your opinion should be the optimal partnerships for achieving success in ETB project?) Evaluation Question 3: To what extent are ETB methodologies, interventions, and management setting the stage for future sustainability of project outputs and outcomes? 8. If we were to plan a program intervention targeted at TB eradication, what interventions would you recommend that we focus on to achieve greater results? Probe: How would these interventions differ from ETB intervention or interventions implemented by the NTP?) Evaluation Question 4: What institutional capacity has ETB built within the MOH and other stakeholders (private sector, Zambian research institutions) to generate sustainability for the processes and outputs achieved? 9. Did your work involve working with MoH staff and other stakeholders? Probe: • Were there tasks/roles you think where not being performed and only got to be done after your coming to this province around ETB? • What are some of those tasks/roles? • Do you think they will be continued if this project comes to an end? USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 58 | P a g e Tool 3: MOH officials Interview Guide Key Informant Interview Guide (MoH TB Principal/ Provincial TB coordinator/ DHD, PHD) Date of FGD: _____________ (dd/mm/yyyy) Location of FGD: _____________ Name of Facilitator: _____________ Signature of Facilitator: _____________ Name of Notetaker: _____________ Signature of Notetaker: _____________ FGD Start-Time: _____: _____ FGD End-Time: _____: _____ Welcome Good morning/afternoon and welcome to our discussion. Thank you for taking the time to join us today. My name is ________________ and I will be facilitating today’s discussion. Assisting me is ________________, who will be taking notes. We are from the USAID/Zambia Monitoring, Evaluation and Learning Platform (Z-MELP). Z-MELP has been requested by USAID/Zambia to provide independent evaluation services for the USAID/Zambia Eradicate Tuberculosis Activity (ETB) [show participants the introduction letter]. General questions Evaluation Question 1: What lessons learned and/or best practices should be taken from ETB to inform the implementation of other (existing) TB-focused activities and the design of new activities focused on TB eradication? 1. What key results would you point out resulting from implementation of ETB project? 2. What lessons can be drawn from implementation of the ETB project? Probe: Lessons in capacity building, approaches to TB case finding and management; acceptance; sustainability; institutional development etc. 3. What best practices can be drawn during implementation ETB project? 4. Where there any unintended outcomes from the USAID assistance? Evaluation Question 2: Were the ETB staffing and partnership structures optimal for achieving project objectives? 5. How optimal were the staffing in implementation of the ETB project? Probe: • Were the numbers of staffing adequate? • Did the staff have adequate capacity to implement the ETB project? • What mechanism was in place to ensure capacity assessment is done and feedback provided to ETB project staff? • Were there any challenges/limitations in staffing and their capacity? If yes, what could have been the cause? • What in your opinion should be the optimal staffing and their capacity for achieving success in ETB project?) 6. How optimal were the partnerships in implementation of the ETB project? Probe: USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 59 | P a g e • What kind of partnerships were formed? • How useful were these different kinds of partnerships in the implementation of programs such as ETB? • How useful were the partnerships between government and donors in the implementation of programs such as ETB?) • Were there any challenges/limitations in forging partnerships? If yes, what could have been the cause? • What in your opinion should be the optimal partnerships for achieving success in ETB project?) Evaluation Question 3: To what extent are ETB methodologies, interventions, and management setting the stage for future sustainability of project outputs and outcomes? 7. Kindly describe the current TB landscape in Zambia. Probe: • How much of a burden is TB and who is most affected? • How many donor projects are currently working on TB in the country? • Which donor communities currently fund TB programs to the Zambian government? • To what extent has donor-implemented projects addressed TB eradication?) 8. What government plans/polices/strategies exist to address TB in Zambia? Probe: • Do you have officers at lower levels to actualize these polices and if so, how does this happen? • How do the lower organs of the government get to actualize the government plans? • Did the project contribute sufficiently to government strategic policy goals? • What were the particular features of the Strategy that made a difference?) 9. What is the TB eradication structure for the ministry and how did it fit in the ETB project? Probe: • Can the collaboration be better improved to achieve results and for sustainability? If yes, how? • Were the ministry staff sufficiently trained and supported to achieve project implementation? • Are there factors that limited implementation? • What recommendations do you have for this particular aspect?) 10. How much involved is the government in the design and implementation of Non-government Organizations projects? Probe: What does this mean in terms of the success of the project? 11. If given the mandate to plan the implementation of your specialty on TB eradication Probe: • How differently would you plan and implement the activities? • What interventions would you recommend that we focus on to achieve greater results?) 12. Would different sites (i.e., facilities, districts, province) require different perspectives for program integration? Probe: How? Evaluation Question 4: What institutional capacity has ETB built within the MOH and other stakeholders (private sector, Zambian research institutions) to generate sustainability for the processes and outputs achieved? 13. What institutional capacity has ETB built within the facility, district, province, and national level to generate continuation of project activities? Probe: • Will the new capacities continue to generate the kinds of results the project intended? • Are the new capacities likely to be self-sustaining? • Any evidence for sustainability? 14. What institutional networks has ETB built within the facility, district, province, and national level to generate continuation of project activities? USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 60 | P a g e Probe: • Will the new networks continue to generate the kinds of results the project intended? • Are the new networks likely to be self-sustaining? • Any evidence for sustainability?) USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 61 | P a g e Tool 4: Health Facility Staff Interview Guide KIIs Guides (Health facility staff) Date of FGD: _____________ (dd/mm/yyyy) Location of FGD: _____________ Name of Facilitator: _____________ Signature of Facilitator: _____________ Name of Notetaker: _____________ Signature of Notetaker: _____________ FGD Start-Time: _____: _____ FGD End-Time: _____: _____ Welcome Good morning/afternoon and welcome to our discussion. Thank you for taking the time to join us today. My name is ________________ and I will be facilitating today’s discussion. Assisting me is ________________, who will be taking notes. We are from the USAID/Zambia Monitoring, Evaluation and Learning Platform (Z-MELP). Z-MELP has been requested by USAID/Zambia to provide independent evaluation services for the USAID/Zambia Eradicate Tuberculosis Activity (ETB) [show General questions Evaluation Question 1: What lessons learned and/or best practices should be taken from ETB to inform the implementation of other (existing) TB-focused activities and the design of new activities focused on TB eradication? 1. What key results would you point out resulting from implementation of ETB project? 2. What lessons can be drawn from implementation of the ETB project? Probe: Lessons in capacity building, approaches to TB case finding and management; acceptance; sustainability; institutional development etc. 3. What best practices can be drawn during implementation ETB project? 4. What major challenges did you encounter during the implementation of this program? Evaluation Question 3: To what extent are ETB methodologies, interventions, and management setting the stage for future sustainability of project outputs and outcomes? 5. What new responsibilities were added to you after being identified as the focal point person for ETB activity? Probe: • Where were you trained to perform these responsibilities? • How differently are you managing TB cases now when compared to how you used to manage them before this project? • How sustainable are the newly introduced interventions beyond the life of this project?) 6. If given the mandate to plan the implementation of your specialty on TB eradication Probe: • How differently would you plan and implement the activities? • What interventions would you recommend that we focus on to achieve greater results?) USAID/Zambia Monitoring, Learning & Evaluation Platform (Z-MELP) 62 | P a g e Evaluation Question 4: What institutional capacity has ETB built within the MOH and other stakeholders (private sector, Zambian research institutions) to generate sustainability for the processes and outputs achieved? 7. What institutional capacity has ETB built within the facility to generate continuation of project activities? Probe: • Will the new capacities continue to generate the kinds of results the project intended? • Are the new capacities likely to be self-sustaining? • Any evidence for sustainability? 8. What institutional networks has ETB built within the facility to generate continuation of project activities? Probe: • Will the new networks continue to generate the kinds of results the project intended? • Are the new networks likely to be self-sustaining? • Any evidence for sustainability?